Building tensorFlow windows x86 c++ library

[smaraux]

Hello,

I am new to tensorflow community. I need to build a c++ lib, I am following this page Build from source on Windows  |  TensorFlow

Using bazel 4.2.2 and tensorflow 2.7, MSys setup, etc.

Also tried with bazel 3.7 and tensorflow 2.6.

I Did all the steps and launched the build as administrator (it seems that not doing so produces other issues)

after configure, using the build line

bazel build --config=opt //tensorflow:tensorflow_cc tensorflow:tensorflow_cc_dll_import_lib //tensorflow:install_headers >buildSeb.log 2>&1

got the following log :
INFO: Options provided by the client:
Inherited ‘common’ options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for ‘build’ from c:\git\tensorflow.bazelrc:
Inherited ‘common’ options: --experimental_repo_remote_exec
INFO: Options provided by the client:
‘build’ options: --python_path=C:/python3/python.exe
INFO: Reading rc options for ‘build’ from c:\git\tensorflow.bazelrc:
‘build’ options: --define framework_shared_object=true --java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --host_java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true
INFO: Reading rc options for ‘build’ from c:\git\tensorflow.tf_configure.bazelrc:
‘build’ options: --action_env PYTHON_BIN_PATH=C:/Python3/python.exe --action_env PYTHON_LIB_PATH=C:/Python3/lib/site-packages --python_path=C:/Python3/python.exe --copt=/d2ReducedOptimizeHugeFunctions --host_copt=/d2ReducedOptimizeHugeFunctions --define=override_eigen_strong_inline=true
INFO: Reading rc options for ‘build’ from c:\git\tensorflow.bazelrc:
‘build’ options: --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/common,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/fallback,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
INFO: Found applicable config definition build:short_logs in file c:\git\tensorflow.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file c:\git\tensorflow.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:opt in file c:\git\tensorflow.tf_configure.bazelrc: --copt=/arch:AVX --host_copt=/arch:AVX
INFO: Found applicable config definition build:windows in file c:\git\tensorflow.bazelrc: --copt=/W0 --copt=/D_USE_MATH_DEFINES --host_copt=/D_USE_MATH_DEFINES --cxxopt=/std:c++14 --host_cxxopt=/std:c++14 --config=monolithic --copt=-DWIN32_LEAN_AND_MEAN --host_copt=-DWIN32_LEAN_AND_MEAN --copt=-DNOGDI --host_copt=-DNOGDI --copt=/experimental:preprocessor --host_copt=/experimental:preprocessor --linkopt=/DEBUG --host_linkopt=/DEBUG --linkopt=/OPT:REF --host_linkopt=/OPT:REF --linkopt=/OPT:ICF --host_linkopt=/OPT:ICF --verbose_failures --features=compiler_param_file --distinct_host_configuration=false
INFO: Found applicable config definition build:monolithic in file c:\git\tensorflow.bazelrc: --define framework_shared_object=false
Loading:
Loading: 0 packages loaded
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/43d6991c2a4cc2ac374e68c029634f2b59ffdfdf.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
WARNING: Download from http://mirror.tensorflow.org/github.com/tensorflow/runtime/archive/64c92c8013b557087351c91b5423b6046d10f206.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
Analyzing: 3 targets (0 packages loaded, 0 targets configured)
Analyzing: 3 targets (0 packages loaded, 0 targets configured)
Analyzing: 3 targets (0 packages loaded, 0 targets configured)
Analyzing: 3 targets (0 packages loaded, 0 targets configured)
Analyzing: 3 targets (0 packages loaded, 0 targets configured)
Analyzing: 3 targets (0 packages loaded, 0 targets configured)
INFO: Repository io_bazel_rules_docker instantiated at:
C:/git/tensorflow/WORKSPACE:23:14: in
C:/git/tensorflow/tensorflow/workspace0.bzl:108:34: in workspace
C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_toolchains/repositories/repositories.bzl:35:23: in repositories
Repository rule git_repository defined at:
C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in
ERROR: An error occurred during the fetch of repository ‘io_bazel_rules_docker’:
Traceback (most recent call last):
File “C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_tools/tools/build_defs/repo/git.bzl”, line 181, column 30, in _git_repository_implementation
update = _clone_or_update(ctx)
File “C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_tools/tools/build_defs/repo/git.bzl”, line 36, column 20, in clone_or_update
git = git_repo(ctx, directory)
File “C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_tools/tools/build_defs/repo/git_worker.bzl”, line 91, column 12, in git_repo
_update(ctx, git_repo)
File “C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_tools/tools/build_defs/repo/git_worker.bzl”, line 103, column 10, in _update
fetch(ctx, git_repo)
File “C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_tools/tools/build_defs/repo/git_worker.bzl”, line 136, column 13, in fetch
_git(
File “C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_tools/tools/build_defs/repo/git_worker.bzl”, line 169, column 15, in _git
_error(ctx.name, start + list(args), st.stderr)
File “C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_tools/tools/build_defs/repo/git_worker.bzl”, line 190, column 9, in _error
fail(“error running ‘%s’ while working with @%s:\n%s” % (command_text, name, stderr))
Error in fail: error running ‘git fetch origin refs/heads/:refs/remotes/origin/ refs/tags/:refs/tags/’ while working with @io_bazel_rules_docker:
BUG: run-command.c:521: disabling cancellation: Invalid argument
0 [main] git 1485 cygwin_exception::open_stackdumpfile: Dumping stack trace to git.exe.stackdump
ERROR: Error fetching repository: Traceback (most recent call last):
File “C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_tools/tools/build_defs/repo/git.bzl”, line 181, column 30, in _git_repository_implementation
update = _clone_or_update(ctx)
File “C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_tools/tools/build_defs/repo/git.bzl”, line 36, column 20, in clone_or_update
git = git_repo(ctx, directory)
File “C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_tools/tools/build_defs/repo/git_worker.bzl”, line 91, column 12, in git_repo
_update(ctx, git_repo)
File “C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_tools/tools/build_defs/repo/git_worker.bzl”, line 103, column 10, in _update
fetch(ctx, git_repo)
File “C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_tools/tools/build_defs/repo/git_worker.bzl”, line 136, column 13, in fetch
_git(
File “C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_tools/tools/build_defs/repo/git_worker.bzl”, line 169, column 15, in _git
_error(ctx.name, start + list(args), st.stderr)
File “C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/bazel_tools/tools/build_defs/repo/git_worker.bzl”, line 190, column 9, in _error
fail(“error running ‘%s’ while working with @%s:\n%s” % (command_text, name, stderr))
Error in fail: error running ‘git fetch origin refs/heads/:refs/remotes/origin/ refs/tags/:refs/tags/’ while working with @io_bazel_rules_docker:
BUG: run-command.c:521: disabling cancellation: Invalid argument
0 [main] git 1485 cygwin_exception::open_stackdumpfile: Dumping stack trace to git.exe.stackdump
INFO: Repository go_sdk instantiated at:
C:/git/tensorflow/WORKSPACE:23:14: in
C:/git/tensorflow/tensorflow/workspace0.bzl:120:20: in workspace
C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/com_github_grpc_grpc/bazel/grpc_extra_deps.bzl:36:27: in grpc_extra_deps
C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/io_bazel_rules_go/go/toolchain/toolchains.bzl:379:28: in go_register_toolchains
C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/io_bazel_rules_go/go/private/sdk.bzl:65:21: in go_download_sdk
Repository rule _go_download_sdk defined at:
C:/users/smaraux.golaem/_bazel_smaraux/btxde6oq/external/io_bazel_rules_go/go/private/sdk.bzl:53:35: in
ERROR: Analysis of target ‘//tensorflow:tensorflow_cc’ failed; build aborted: error running ‘git fetch origin refs/heads/:refs/remotes/origin/ refs/tags/:refs/tags/’ while working with @io_bazel_rules_docker:
BUG: run-command.c:521: disabling cancellation: Invalid argument
0 [main] git 1485 cygwin_exception::open_stackdumpfile: Dumping stack trace to git.exe.stackdump
INFO: Elapsed time: 5.780s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)

Would anybody help to achieve this ?

Is the methode described on the page Build from source on Windows  |  TensorFlow still up to date ?

Best

Sebastien

[cesarpgouveia]

Hi,

I have the same issue. @smaraux did you manage to overcome this problem?

Thanks,
César.

[smaraux]

Actually, we did not expect to meet so much difficulties to switch between python and c++.
We tried several frameworks and dropped Tensorflow due to the (clearly mentionned) unmaintained / unwarrantied C++ API. For inference part, we once expected ONNX to be our safety belt, but we need some model features to enable dropout, to get an epistemic error estimation, and ONNX have no such feature.

In fact, no current framework properly allows to build a model in python, and use it for training / with dropout in c++. Tensorflow claims that c++ side is not maintained, pytorch has no complete model interactions between python and c++ (makes error when loading jit model in c++), ONNX is not sufficient for our needs, and we did not try others but expect same kind of issues.

We finally felt back to torch for complete c++ support, even if it does not interact well with python for complete models (sadly).

It does not seem safe to go c++ way with Tensorflow as it is not warrantied to be maintained and will probably require tweaks when building each version.

[cesarpgouveia]

@smaraux

It does not seem safe to go c++ way with Tensorflow as it is not warrantied to be maintained and will probably require tweaks when building each version.

Unfortunately I think this is the case yes…

Thanks for sharing your view and your issues.

[Bhack]

I saw a thread recently about libtensorflow building in the SIG build chat

/cc @angerson