Build TF with `asan` flag

Stonepia · July 22, 2021, 10:06am

I am trying to build TF with ASAN flag enabled , using the command :

bazel build --config=dbg --config=asan --copt -fsanitize=address --linkopt -fsanitize=address //tensorflow/tools/pip_package:build_pip_package --verbose_failures

but this would give me an error about the third party library:

ERROR: /home/user/.cache/bazel/_bazel_user/7562dd0a46bcb6cada6ae19650738275/external/libjpeg_turbo/BUILD.bazel:238:8: Executing genrule @libjpeg_turbo//:simd_x86_64_assemblage23 failed (Exit 1): bash failed: error executing command
  (cd /home/user/.cache/bazel/_bazel_user/7562dd0a46bcb6cada6ae19650738275/execroot/org_tensorflow && \
  exec env - \
    LD_LIBRARY_PATH=/usr/local/cuda-11.3/lib64 \
    PATH=/home/user/bin:/usr/local/cuda-11.3/bin:/home/user/anaconda3/envs/tfenv/bin:/home/user/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; for out in bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jccolor-avx2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jccolor-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jcgray-avx2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jcgray-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jchuff-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jcphuff-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jcsample-avx2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jcsample-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jdcolor-avx2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jdcolor-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jdmerge-avx2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jdmerge-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jdsample-avx2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jdsample-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jfdctflt-sse.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jfdctfst-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jfdctint-avx2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jfdctint-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jidctflt-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jidctfst-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jidctint-avx2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jidctint-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jidctred-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jquantf-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jquanti-avx2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jquanti-sse2.o bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/simd/x86_64/jsimdcpu.o; do
  bazel-out/k8-dbg/bin/external/nasm/nasm -f elf64    -DELF -DPIC -D__x86_64__    -I $(dirname bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/jconfig.h)/    -I $(dirname bazel-out/k8-opt-exec-50AE0418/bin/external/libjpeg_turbo/jconfigint.h)/    -I $(dirname external/libjpeg_turbo/simd/nasm/jsimdcfg.inc.h)/    -I $(dirname external/libjpeg_turbo/simd/x86_64/jccolext-sse2.asm)/    -o $out    $(dirname external/libjpeg_turbo/simd/x86_64/jccolext-sse2.asm)/$(basename ${out%.o}.asm)
done')
Execution platform: @local_execution_config_platform//:platform

=================================================================
==479927==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 5888 byte(s) in 32 object(s) allocated from:
    #0 0x7f0adb6bcbc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x55d56e9d76d1 in nasm_malloc (/home/user/.cache/bazel/_bazel_user/7562dd0a46bcb6cada6ae19650738275/execroot/org_tensorflow/bazel-out/k8-dbg/bin/external/nasm/nasm+0x1d96d1)
    #2 0x55d56e9b9c2e in do_directive (/home/user/.cache/bazel/_bazel_user/7562dd0a46bcb6cada6ae19650738275/execroot/org_tensorflow/bazel-out/k8-dbg/bin/external/nasm/nasm+0x1bbc2e)
    #3 0x55d56e9c5d60 in pp_getline (/home/user/.cache/bazel/_bazel_user/7562dd0a46bcb6cada6ae19650738275/execroot/org_tensorflow/bazel-out/k8-dbg/bin/external/nasm/nasm+0x1c7d60)
    #4 0x55d56e99d13f in assemble_file.constprop.8 (/home/user/.cache/bazel/_bazel_user/7562dd0a46bcb6cada6ae19650738275/execroot/org_tensorflow/bazel-out/k8-dbg/bin/external/nasm/nasm+0x19f13f)
    #5 0x55d56e97b275 in main (/home/user/.cache/bazel/_bazel_user/7562dd0a46bcb6cada6ae19650738275/execroot/org_tensorflow/bazel-out/k8-dbg/bin/external/nasm/nasm+0x17d275)
    #6 0x7f0adb2020b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)

Direct leak of 768 byte(s) in 192 object(s) allocated from:
    #0 0x7f0adb6bcbc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x55d56e9d76d1 in nasm_malloc (/home/user/.cache/bazel/_bazel_user/7562dd0a46bcb6cada6ae19650738275/execroot/org_tensorflow/bazel-out/k8-dbg/bin/external/nasm/nasm+0x1d96d1)
    #2 0x55d56e9ac010 in new_Token.constprop.9 (/home/user/.cache/bazel/_bazel_user/7562dd0a46bcb6cada6ae19650738275/execroot/org_tensorflow/bazel-out/k8-dbg/bin/external/nasm/nasm+0x1ae010)
    #3 0x55d56e9c706d in pp_getline (/home/user/.cache/bazel/_bazel_user/7562dd0a46bcb6cada6ae19650738275/execroot/org_tensorflow/bazel-out/k8-dbg/bin/external/nasm/nasm+0x1c906d)
    #4 0x55d56e99d13f in assemble_file.constprop.8 (/home/user/.cache/bazel/_bazel_user/7562dd0a46bcb6cada6ae19650738275/execroot/org_tensorflow/bazel-out/k8-dbg/bin/external/nasm/nasm+0x19f13f)
    #5 0x55d56e97b275 in main (/home/user/.cache/bazel/_bazel_user/7562dd0a46bcb6cada6ae19650738275/execroot/org_tensorflow/bazel-out/k8-dbg/bin/external/nasm/nasm+0x17d275)
    #6 0x7f0adb2020b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)

So I tried to add the following to the third_party/jpec/BUILD.bazel :

# line 238
genrule(
    name = "simd_x86_64_assemblage23",
+    tags = [
+       "noasan",
+        "no_cuda_asan",
+       "nomsan",
+        "notsan",
+    ],

This still does not help.

Is there tips of how to deal with this?

Bhack · July 22, 2021, 10:34am

Just to link your ticket to the thread:

github.com/tensorflow/tensorflow

Tensorflow could not build with `config=asan` flag

opened 04:47AM - 22 Jul 21 UTC

closed 05:41AM - 04 Aug 21 UTC

Stonepia

stat:awaiting response type:build/install subtype: ubuntu/linux subtype:bazel

<em>Please make sure that this is a build/installation issue. As per our [GitHub… Policy](https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md), we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template</em> **System information** OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04 Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: TensorFlow installed from (source or binary): source TensorFlow version: master Python version: 3.8.8 Installed using virtualenv? pip? conda?: cinda Bazel version (if compiling from source): 3.7.2 GCC/Compiler version (if compiling from source): clang-11 CUDA/cuDNN version: CUDA 11.3, CuDNN 8 GPU model and memory: GTX 1080 Ti **Describe the problem** **Provide the exact sequence of commands / steps that you executed before running into the problem** I am trying to build TF with Address Sanitizer enabled. TF could not built with the flag `--config asan`. I tried to use the following command: ```Batch CC=/usr/lib/llvm-11/bin/clang TMP=~/tmp bazel build --config asan --config=dbg --per_file_copt=//tensorflow/core/grappler/./.*\.cc@-g //tensorflow/tools/pip_package:build_pip_package --verbose_failures ``` and it failed with the following ```Log tensorflow git:(master) ✗ CC=/usr/lib/llvm-11/bin/clang TMP=~/tmp bazel build --config asan --config=dbg --per_file_copt=//tensorflow/core/grappler/./.*\.cc@-g //tensorflow/tools/pip_package:build_pip_package --verbose_failures WARNING: The following configs were expanded more than once: [cuda_clang, cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior. INFO: Options provided by the client: Inherited 'common' options: --isatty=1 --terminal_columns=149 INFO: Reading rc options for 'build' from /home/user/tensorflow/.bazelrc: Inherited 'common' options: --experimental_repo_remote_exec INFO: Reading rc options for 'build' from /home/user/tensorflow/.bazelrc: 'build' options: --define framework_shared_object=true --java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --host_java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true INFO: Reading rc options for 'build' from /home/user/tensorflow/.tf_configure.bazelrc: 'build' options: --action_env PYTHON_BIN_PATH=/home/user/anaconda3/envs/tfenv/bin/python3 --action_env PYTHON_LIB_PATH=/home/user/anaconda3/envs/tfenv/lib/python3.8/site-packages --python_path=/home/user/anaconda3/envs/tfenv/bin/python3 --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda-11.3 --action_env TF_CUDA_COMPUTE_CAPABILITIES=6.1,6.1,6.1,6.1 --action_env LD_LIBRARY_PATH=/usr/local/cuda-11.3/lib64 --config=cuda_clang --action_env CLANG_CUDA_COMPILER_PATH=/usr/lib/llvm-11/bin/clang --config=cuda_clang INFO: Reading rc options for 'build' from /home/user/tensorflow/.bazelrc: 'build' options: --verbose_failures INFO: Found applicable config definition build:short_logs in file /home/user/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING INFO: Found applicable config definition build:v2 in file /home/user/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1 INFO: Found applicable config definition build:cuda_clang in file /home/user/tensorflow/.bazelrc: --config=cuda --repo_env TF_CUDA_CLANG=1 --@local_config_cuda//:cuda_compiler=clang INFO: Found applicable config definition build:cuda in file /home/user/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda INFO: Found applicable config definition build:cuda_clang in file /home/user/tensorflow/.bazelrc: --config=cuda --repo_env TF_CUDA_CLANG=1 --@local_config_cuda//:cuda_compiler=clang INFO: Found applicable config definition build:cuda in file /home/user/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda INFO: Found applicable config definition build:asan in file /home/user/tensorflow/.bazelrc: --strip=never --copt -fsanitize=address --copt -DADDRESS_SANITIZER --copt -g --copt -O3 --copt -fno-omit-frame-pointer --linkopt -fsanitize=address INFO: Found applicable config definition build:dbg in file /home/user/tensorflow/.bazelrc: -c dbg --per_file_copt=+.*,-tensorflow.*@-g0 --per_file_copt=+tensorflow/core/kernels.*@-g0 --cxxopt -DTF_LITE_DISABLE_X86_NEON --copt -DDEBUG_BUILD INFO: Found applicable config definition build:linux in file /home/user/tensorflow/.bazelrc: --copt=-w --host_copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes INFO: Found applicable config definition build:dynamic_kernels in file /home/user/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS WARNING: Download from http://mirror.tensorflow.org/github.com/tensorflow/toolchains/archive/d781e89e2ee797ea7afd0c8391e761616fc5d50d.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found WARNING: Download from http://mirror.tensorflow.org/github.com/tensorflow/runtime/archive/c362c993e4cca885ba6afef155d864a2dfd21f85.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/bcf6f641acdbeb208ea07a9e8ded37cd5b796d26.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (0 packages loaded, 0 targets configured). INFO: Found 1 target... ERROR: /home/user/.cache/bazel/_bazel_user/7562dd0a46bcb6cada6ae19650738275/external/highwayhash/BUILD.bazel:23:11: undeclared inclusion(s) in rule '@highwayhash//:arch_specific': this rule is missing dependency declarations for the following files included by 'highwayhash/highwayhash/arch_specific.cc': '/usr/lib/llvm-11/lib/clang/11.1.0/share/asan_blacklist.txt' Target //tensorflow/tools/pip_package:build_pip_package failed to build INFO: Elapsed time: 0.680s, Critical Path: 0.40s INFO: 17 processes: 17 internal. FAILED: Build did NOT complete successfully ``` Also tried the command of `--copt -fsanitize=address --linkopt -fsanitize=address`: ```Batch CC=/usr/lib/llvm-11/bin/clang TMP=~/tmp bazel build --copt -fsanitize=address --linkopt -fsanitize=address --config=dbg --per_file_copt=//tensorflow/core/grappler/./.*\.cc@-g //tensorflow/tools/pip_package:build_pip_package --verbose_failures ``` and get the similar error. **Any other info / logs** Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. ```Batch $env USER=user LOGNAME=user HOME=/home/user PATH=/home/user/bin:/usr/local/cuda-11.3/bin:/home/user/anaconda3/envs/tfenv/bin:/home/user/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin SHELL=/usr/bin/zsh TERM=xterm-256color XDG_SESSION_ID=99 XDG_RUNTIME_DIR=/run/user/1000 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus XDG_SESSION_TYPE=tty XDG_SESSION_CLASS=user MOTD_SHOWN=pam LANG=en_US.UTF-8 LC_NUMERIC=zh_CN.UTF-8 LC_TIME=zh_CN.UTF-8 LC_MONETARY=zh_CN.UTF-8 LC_PAPER=zh_CN.UTF-8 LC_NAME=zh_CN.UTF-8 LC_ADDRESS=zh_CN.UTF-8 LC_TELEPHONE=zh_CN.UTF-8 LC_MEASUREMENT=zh_CN.UTF-8 LC_IDENTIFICATION=zh_CN.UTF-8 SSH_CLIENT=10.249.174.114 49269 22 SSH_CONNECTION=10.249.174.114 49269 10.239.44.39 22 SSH_TTY=/dev/pts/3 SHLVL=1 PWD=/home/user/tensorflow OLDPWD=/home/user ZSH=/home/user/.oh-my-zsh PAGER=less LESS=-R LSCOLORS=Gxfxcxdxbxegedabagacad LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36: LD_PRELOAD=/home/user/anaconda3/envs/dlrm/lib/libiomp5.so CONDA_EXE=/home/user/anaconda3/bin/conda _CE_M= _CE_CONDA= CONDA_PYTHON_EXE=/home/user/anaconda3/bin/python CONDA_SHLVL=2 CONDA_PREFIX=/home/user/anaconda3/envs/tfenv CONDA_DEFAULT_ENV=tfenv CONDA_PROMPT_MODIFIER=(tfenv) LD_LIBRARY_PATH=/usr/local/cuda-11.3/lib64 CONDA_PREFIX_1=/home/user/anaconda3 _=/usr/bin/env ```