Build from source fails in Fedora 37

Last Friday (2022-11-11), I successfully compiled Tensorflow from source, based on the instructions I found in Tensorflow.
Before compiling under Fedora 36, I executed these commands:

> sudo dnf install python3-pip python3-devel python3-virtualenv
> pip install -U --user pip numpy wheel packaging requests opt_einsum
> pip install -U --user keras_preprocessing --no-deps

Bazel was installed by downloading and executing the installer (https://github.com/bazelbuild/bazel/releases/download/5.3.2/bazel-5.3.2-installer-linux-x86_64.sh). Before executing ./configure, I had to change the number in ~/Git/tensorflow/.bazelversion to 5.3.2.
./configure was executed with default options whereafter

 bazel build //tensorflow/tools/pip_package:build_pip_package

resulted in a build_pip_package that could be installed (after installing wheel).

Yesterday, I upgraded to Fedora 37. There were no problems but my tensorflow-installation was gone :pensive:. I executed the same commands as above. But now, after compiling approximately 10700 files, the process aborted.

[bengbers@fedora tensorflow]$ bazel build //tensorflow/tools/pip_package:build_pip_package
WARNING: Ignoring JAVA_HOME, because it must point to a JDK, not a JRE.
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=238
INFO: Reading rc options for 'build' from /home/bengbers/Git/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/bengbers/Git/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false
INFO: Reading rc options for 'build' from /home/bengbers/Git/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env PYTHON_LIB_PATH=/usr/lib/python3.11/site-packages --python_path=/usr/bin/python3
INFO: Reading rc options for 'build' from /home/bengbers/Git/tensorflow/.bazelrc:
  'build' options: --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/common,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
INFO: Found applicable config definition build:short_logs in file /home/bengbers/Git/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/bengbers/Git/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:linux in file /home/bengbers/Git/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-unknown-warning --copt=-Wno-array-parameter --copt=-Wno-stringop-overflow --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /home/bengbers/Git/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/runtime/archive/f027b6182beb76f0f73c63208c54d96ce33d0d12.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (566 packages loaded, 31655 targets configured).
INFO: Found 1 target...
ERROR: /home/bengbers/.cache/bazel/_bazel_bengbers/d5e1513df739af824db2721edbf62765/external/com_google_protobuf/BUILD:812:10: Compiling python/google/protobuf/pyext/descriptor.cc failed: (Exit 1): gcc failed: error executing command /usr/lib64/ccache/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 71 arguments skipped)
external/com_google_protobuf/python/google/protobuf/pyext/descriptor.cc: In function 'bool google::protobuf::python::_CalledFromGeneratedFile(int)':
external/com_google_protobuf/python/google/protobuf/pyext/descriptor.cc:108:18: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
  108 |     frame = frame->f_back;
      |                  ^~
In file included from bazel-out/k8-opt/bin/external/local_config_python/python_include/Python.h:42,
                 from external/com_google_protobuf/python/google/protobuf/pyext/descriptor.cc:33:
bazel-out/k8-opt/bin/external/local_config_python/python_include/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
external/com_google_protobuf/python/google/protobuf/pyext/descriptor.cc:114:12: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
  114 |   if (frame->f_code->co_filename == NULL) {
      |            ^~
bazel-out/k8-opt/bin/external/local_config_python/python_include/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
In file included from bazel-out/k8-opt/bin/external/local_config_python/python_include/Python.h:44:
external/com_google_protobuf/python/google/protobuf/pyext/descriptor.cc:119:37: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
  119 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
      |                                     ^~
external/com_google_protobuf/python/google/protobuf/pyext/descriptor.cc:119:7: note: in expansion of macro 'PyString_AsStringAndSize'
  119 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
      |       ^~~~~~~~~~~~~~~~~~~~~~~~
bazel-out/k8-opt/bin/external/local_config_python/python_include/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
external/com_google_protobuf/python/google/protobuf/pyext/descriptor.cc:119:37: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
  119 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
      |                                     ^~
external/com_google_protobuf/python/google/protobuf/pyext/descriptor.cc:60:56: note: in definition of macro 'PyString_AsStringAndSize'
   60 |                                PyUnicode_AsUTF8AndSize(ob, (sizep)))) == NULL \
      |                                                        ^~
bazel-out/k8-opt/bin/external/local_config_python/python_include/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
external/com_google_protobuf/python/google/protobuf/pyext/descriptor.cc:119:37: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
  119 |   if (PyString_AsStringAndSize(frame->f_code->co_filename,
      |                                     ^~
external/com_google_protobuf/python/google/protobuf/pyext/descriptor.cc:63:50: note: in definition of macro 'PyString_AsStringAndSize'
   63 |                        : PyBytes_AsStringAndSize(ob, (charpp), (sizep)))
      |                                                  ^~
bazel-out/k8-opt/bin/external/local_config_python/python_include/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
external/com_google_protobuf/python/google/protobuf/pyext/descriptor.cc:140:12: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
  140 |   if (frame->f_globals != frame->f_locals) {
      |            ^~
bazel-out/k8-opt/bin/external/local_config_python/python_include/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
external/com_google_protobuf/python/google/protobuf/pyext/descriptor.cc:140:32: error: invalid use of incomplete type 'PyFrameObject' {aka 'struct _frame'}
  140 |   if (frame->f_globals != frame->f_locals) {
      |                                ^~
bazel-out/k8-opt/bin/external/local_config_python/python_include/pytypedefs.h:22:16: note: forward declaration of 'PyFrameObject' {aka 'struct _frame'}
   22 | typedef struct _frame PyFrameObject;
      |                ^~~~~~
At global scope:
cc1plus: note: unrecognized command-line option '-Wno-unknown-warning' may have been intended to silence earlier diagnostics
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 18082.085s, Critical Path: 465.17s
INFO: 7008 processes: 220 internal, 6788 local.
FAILED: Build did NOT complete successfully

I now managed to install tensorflow with pip but I would prefer to install from source.
How can I build tensorflow under Fedora 37?

Ben

I cross-posted this question to the Fedora Forum. Yesterday I received a message, telling that my problem is caused by changes in Python.

Apparantly there is some work to be done by the Tensorflow team :smirk:

I just did a web search on PyFrameObject.
The results were full of similar problems that arise from a change in Python 3.11 from 3.10.
Apparently PyFrameObject has been made into an opaque type (which it always was suppose to be), so access to its internals is no longer possible.

To get TensorFlow to build you will need to revert your Python installation back to 3.10.

In the long run it will need changes to TensorFlow so that it uses the latest version of Python. 

Unfortunately we don’t provide build support for Fedora 37. Currently we are supporting the following OS

  • Ubuntu 16.04 or higher (64-bit)
  • macOS 10.12.6 (Sierra) or higher (64-bit) (no GPU support)
  • Windows Native - Windows 7 or higher (64-bit) (no GPU support after TF 2.10)
  • Windows WSL2 - Windows 10 19044 or higher (64-bit)

Thank you!