Unit Tests: FAILED: Build did NOT complete successfully

General Discussion
, ,
#1

Hello all,

When trying to run unit tests com docker, when the build ended I got this message. There were about 5 errors regarding gcc. Namely:
gcc: error: unrecognized command line option ‘-mamx-int8’
gcc: error: unrecognized command line option ‘-mavxvnni’; did you mean ‘-mavx512vnni’?
While the others just said:
gcc: fatal error: Killed signal terminated program cc1plus
compilation terminated.

Could this be the reason for the build to fail? If so, how do I fix this?

Regarding my setup, I ran the command on docker ( docker run -it --rm -v $PWD:/tmp -w /tmp tensorflow/build:2.17-python3.12) and
gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0

On the same subject, when I re-ran the docker command followed by the bazel command, it restarted downloading and compiling about 15k files. Do I need to do this every time I want to run unit tests?

Thank you

1 Like
#3

I have the same issue. I tested with the 2.12-python3.10, 2.15-python3.10, 2.15-python3.11, 2.16-python3.10 images and had the same or similar errors every time.

Steps:

docker run -it --rm -v $PWD:/tmp -w /tmp tensorflow/build:2.15-python3.10

tf-docker/tmp > export TF_PYTHON_VERSION=3.10
tf-docker/tmp > export flags="-c opt -k"  
tf-docker/tmp > bazel test ${flags} tensorflow/python/kernel_tests/nn_ops:softmax_op_test

Output:

2024/03/14 14:33:05 Downloading https://releases.bazel.build/6.5.0/release/bazel-6.5.0-linux-x86_64...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Reading 'startup' options from /tmp/.bazelrc: --windows_enable_symlinks
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=206
INFO: Reading rc options for 'test' from /tmp/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'test' from /etc/bazel.bazelrc:
  Inherited 'build' options: --action_env=DOCKER_CACHEBUSTER=1696724024742236250 --host_action_env=DOCKER_HOST_CACHEBUSTER=1696724024842064639
INFO: Reading rc options for 'test' from /tmp/.bazelrc:
  Inherited 'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
INFO: Found applicable config definition build:short_logs in file /tmp/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /tmp/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:linux in file /tmp/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /tmp/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
INFO: Analyzed 2 targets (624 packages loaded, 41610 targets configured).
INFO: Found 2 test targets...
ERROR: /root/.cache/bazel/_bazel_root/d42b9c57d24cf5db3bd8d332dc35437f/external/XNNPACK/BUILD.bazel:2683:19: Compiling src/amalgam/gen/avx512amx.c failed: (Exit 1): gcc failed: error executing command (from target @XNNPACK//:avx512amx_prod_microkernels) /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 54 arguments skipped)
gcc: error: unrecognized command line option '-mamx-int8'
ERROR: /root/.cache/bazel/_bazel_root/d42b9c57d24cf5db3bd8d332dc35437f/external/XNNPACK/BUILD.bazel:1873:19: Compiling src/amalgam/gen/avxvnni.c failed: (Exit 1): gcc failed: error executing command (from target @XNNPACK//:avxvnni_prod_microkernels) /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 48 arguments skipped)
gcc: error: unrecognized command line option '-mavxvnni'; did you mean '-mavx512vnni'?

Any help would be greatly appreciated, thank you!

#5

I also was facing the same. A possible workaround is to enable llvm+ clang support while running ./configure as is shown in
https://www.tensorflow.org/install/source
With clang support enabled the compilation seems to take more time though.

#7

Hi @T_F

Please refer to this link Docker Linux builds and follow the commands to build Linux packages from source. These images already contain the source code and dependencies required to build TensorFlow.

Let us know if the issue still persists. Thank you.

#9

I built from the tf docker image but have the same error.