Are they any plans to upgrade TF to a newer version in TensorFlow Java?
Yes, we have a branch with 2.15 in it, but we’re having some difficulty with the Windows builds so we can’t release it yet.
Thanks @craigacp. Can you provide a link to the branch with 2.15?
Also, Intel now builds and releases the TF Windows bits. If the issues are TF related, we can take a look it.
The branch is bazelcism as we’re working on using the TF binaries from either the Python wheels or C API builds so we can avoid building TF ourselves. We fixed all the issues with macOS and Linux, however the Windows builds have a number of issues. When trying to build 2.15 on Windows in Github Actions (and locally on my desktop) we hit path length issues, even after setting all the bazel paths to point at “C:\tmp”, you can see an example of that here - Fixes for Windows build. · tensorflow/java@4417c0e · GitHub. When trying to use the tensorflow-intel wheels we hit an issue where Windows builds use pyd files not dlls and we can’t load those from Java. When trying to use the C API builds for Windows we hit symbol not exported issues - Missing symbols in official Windows TensorFlow native builds · Issue #62579 · tensorflow/tensorflow · GitHub.
We raised the latter two issues with SIG-BUILD, but not had any response from people who control those builds. The first issue we narrowed down this weekend so we’ve not reported it yet.
Hi @craigacp,
To resolve long path issues while building with Bazel, set --output_user_root=c:\
e.g. bazel --output_user_root=c:\ build --config=opt --repo_env=TF_PYTHON_VERSION=3.10 --define=no_tensorflow_py_deps=true //tensorflow/tools/pip_package:build_pip_package
Apart from this, also consider looking into the system settings to ensure the long path is enabled.
Thanks @craigacp for the info. The Intel team is working on the issues you reported. Pls update the github issues if you have more info.
Bazel is refusing to respect that argument, it either writes things to C:\tmp (after exporting various tmpdir params) or to c:\Users<username>\AppData\Local\Temp. I have long paths set, however MSVC doesn’t support them. Our buildscript is here - https://github.com/tensorflow/java/blob/bazelcism/tensorflow-core/tensorflow-core-native/scripts/bazel_build.sh, and I tried numerous places to add that argument. The one which seemed to do something (in that the bazel install was written to root) was adding it to the top of our tensorflow.bazelrc.
Hi @craigacp, could you please share the steps to build the TensorFlow Java package, or if you have any logs, please share them? We are unable to reproduce the error on our end.
Running mvn clean install -Dnative.build -B -U -e -Djavacpp.platform=windows-x86_64 -Djavacpp.platform.extension="" will build it for Windows. You can see how we configure it in the CI here - https://github.com/tensorflow/java/blob/bazelcism/.github/workflows/build.yml#L157.
Hi @craigacp to fix the long path issue, we can control the path by making the change below in bazel_common.sh.
export BAZEL_VC=“C:\Program Files\Microsoft Visual Studio\2022\Community\VC”
TMP='D:'
if [[ -d $BAZEL_VC ]]; then
export BUILD_FLAGS=“–output_user_root=$(cygpath -w $TMP) build”
export PYTHON_BIN_PATH=$(which python.exe)
else
export BUILD_FLAGS=“build”
export PYTHON_BIN_PATH=$(which python3)
fi
Please let me know if it is still an issue of the long path. There is a linking error with target //tensorflow:tensorflow_cc
@craigacp we have been testing GitHub - tensorflow/java at bazelcism branch on Linux with Java 11 and it looks fine.
However, there is a need for us to build this branch with Java 8. We are running into issues building. Do you think it is possible to build this branch with Java 8?
We’ve merged that branch into master now and plan to upgrade to TF 2.16.1 in the near future.
We don’t support Java 8 anymore, but it shouldn’t be too hard to compile with it, we’ve not diverged too far. What’s causing the problem?
Hi @craigacp,
Thanks for your help. I’m helping Ramesh try to compile TF Java with JDK-8. What I get is a LifecycleExecutionException caused by a CompilationFailureException during the compile of the “TF Native Library” subproject. The only changes I made to pom files were to remove uses of the javac “release” arg and specify “source” and “target” as JDK-8.
I see 2 warnings during the compile:
[WARNING] Can’t extract module name from javacpp-1.5.10.jar: null
[WARNING] Can’t extract module name from classes: null
I pasted the console output / stack traces below. I have Java experience but only a little experience with Maven.
Steve
steve@steve-xps:~/TFJAVA/java$ mvn -e compile
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects…
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] TensorFlow Java Parent [pom]
[INFO] TensorFlow Core Parent [pom]
[INFO] TensorFlow Native Library [jar]
[INFO] TensorFlow Generators [jar]
[INFO] TensorFlow API [jar]
[INFO] TensorFlow API Platform [jar]
[INFO] TensorFlow Framework Library [jar]
[INFO]
[INFO] -------------------< org.tensorflow:tensorflow-java >-------------------
[INFO] Building TensorFlow Java Parent 0.6.0-SNAPSHOT [1/7]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] — maven-enforcer-plugin:3.4.1:enforce (enforce) @ tensorflow-java —
[INFO] Rule 0: org.apache.maven.enforcer.rules.dependency.DependencyConvergence passed
[INFO] Rule 1: org.apache.maven.enforcer.rules.version.RequireMavenVersion passed
[INFO]
[INFO] -------------------< org.tensorflow:tensorflow-core >-------------------
[INFO] Building TensorFlow Core Parent 0.6.0-SNAPSHOT [2/7]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] — maven-enforcer-plugin:3.4.1:enforce (enforce) @ tensorflow-core —
[INFO] Rule 0: org.apache.maven.enforcer.rules.dependency.DependencyConvergence passed
[INFO]
[INFO] ---------------< org.tensorflow:tensorflow-core-native >----------------
[INFO] Building TensorFlow Native Library 0.6.0-SNAPSHOT [3/7]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] — javacpp:1.5.10:build (javacpp-validate) @ tensorflow-core-native —
[INFO] Detected platform “linux-x86_64”
[INFO] Building platform “linux-x86_64”
[INFO]
[INFO] — maven-enforcer-plugin:3.4.1:enforce (enforce) @ tensorflow-core-native —
[INFO] Rule 0: org.apache.maven.enforcer.rules.dependency.DependencyConvergence passed
[INFO]
[INFO] — exec-maven-plugin:3.1.0:exec (bazel-build) @ tensorflow-core-native —
[INFO] skipping execute as per configuration
[INFO]
[INFO] — exec-maven-plugin:3.1.0:exec (dist-download) @ tensorflow-core-native —
Downloading https://files.pythonhosted.org/packages/fa/44/a1698c62942d20cab378ba201a6cbfcce579418351a0c6e4ea9d66c9adf2/tensorflow_cpu-2.15.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
total 599952
drwxr-xr-x 3 steve steve 4096 Nov 11 05:03 _api
drwxr-xr-x 7 steve steve 4096 Nov 11 05:03 compiler
drwxr-xr-x 14 steve steve 4096 Nov 11 05:03 core
drwxr-xr-x 3 steve steve 4096 Nov 11 05:03 distribute
drwxr-xr-x 4 steve steve 4096 Nov 11 05:03 dtensor
drwxr-xr-x 13 steve steve 4096 Nov 11 05:03 include
-rw-r–r-- 1 steve steve 29449 Nov 11 05:03 init.py
lrwxrwxrwx 1 steve steve 21 Mar 19 12:05 libtensorflow_cc.so → libtensorflow_cc.so.2
-rwxr-xr-x 1 steve steve 564733624 Nov 11 05:03 libtensorflow_cc.so.2
lrwxrwxrwx 1 steve steve 28 Mar 19 12:05 libtensorflow_framework.so → libtensorflow_framework.so.2
-rwxr-xr-x 1 steve steve 49042544 Nov 11 05:03 libtensorflow_framework.so.2
drwxr-xr-x 6 steve steve 4096 Nov 11 05:03 lite
drwxr-xr-x 33 steve steve 4096 Nov 11 05:03 python
drwxr-xr-x 3 steve steve 4096 Nov 11 05:03 security
-rw-r–r-- 1 steve steve 476563 Nov 11 05:03 THIRD_PARTY_NOTICES.txt
drwxr-xr-x 6 steve steve 4096 Nov 11 05:03 tools
drwxr-xr-x 4 steve steve 4096 Nov 11 05:03 tsl
drwxr-xr-x 5 steve steve 4096 Nov 11 05:03 xla_aot_runtime_src
[INFO]
[INFO] — maven-resources-plugin:3.3.1:resources (copy-resources) @ tensorflow-core-native —
[INFO] skip non existing resourceDirectory /home/steve/TFJAVA/java/tensorflow-core/tensorflow-core-native/src/main/resources
[INFO]
[INFO] — maven-compiler-plugin:3.11.0:compile (javacpp-compiler) @ tensorflow-core-native —
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] — exec-maven-plugin:3.1.0:exec (bazel-generate) @ tensorflow-core-native —
[INFO] skipping execute as per configuration
[INFO]
[INFO] — javacpp:1.5.10:parse (javacpp-parser) @ tensorflow-core-native —
[INFO] Skipping execution of JavaCPP Builder
[INFO]
[INFO] — build-helper-maven-plugin:3.4.0:add-source (add-gen-sources) @ tensorflow-core-native —
[INFO] Source directory: /home/steve/TFJAVA/java/tensorflow-core/tensorflow-core-native/src/gen/java added.
[INFO]
[INFO] — maven-resources-plugin:3.3.1:resources (default-resources) @ tensorflow-core-native —
[INFO] skip non existing resourceDirectory /home/steve/TFJAVA/java/tensorflow-core/tensorflow-core-native/src/main/resources
[INFO]
[INFO] — maven-resources-plugin:3.3.1:copy-resources (generated-resources-copy) @ tensorflow-core-native —
[INFO] Copying 1510 resources from src/gen/resources to target/defs-classes
[INFO]
[INFO] — maven-compiler-plugin:3.11.0:compile (default-compile) @ tensorflow-core-native —
[WARNING] Can’t extract module name from javacpp-1.5.10.jar: null
[WARNING] Can’t extract module name from classes: null
[INFO] Changes detected - recompiling the module! :source
[INFO] Compiling 493 source files with javac [forked debug target 8 module-path] to target/classes
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for TensorFlow Java Parent 0.6.0-SNAPSHOT:
[INFO]
[INFO] TensorFlow Java Parent … SUCCESS [ 0.361 s]
[INFO] TensorFlow Core Parent … SUCCESS [ 0.005 s]
[INFO] TensorFlow Native Library … FAILURE [ 0.896 s]
[INFO] TensorFlow Generators … SKIPPED
[INFO] TensorFlow API … SKIPPED
[INFO] TensorFlow API Platform … SKIPPED
[INFO] TensorFlow Framework Library … SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.382 s
[INFO] Finished at: 2024-03-19T12:05:25-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile (default-compile) on project tensorflow-core-native: Compilation failure → [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile (default-compile) on project tensorflow-core-native: Compilation failure
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:972)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.compiler.CompilationFailureException: Compilation failure
at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute (AbstractCompilerMojo.java:1214)
at org.apache.maven.plugin.compiler.CompilerMojo.execute (CompilerMojo.java:193)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:972)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
I’m not sure what the root cause of that exception is, it’s odd that it’s not reporting it properly, but you’ll need to remove the module-info.java files from the code base as well as setting source and target appropriately.
Thanks. I had set the source and target to 8 in all poms but had not gotten rid of the module-info files. I did that just now and the two warnings mentioned above went away. Unfortunately, compiling still produces the same failure with the same stack traces.
That might be spotless, google-java-format and maybe error-prone which I think now require Java 11 or newer. You can turn those off by hacking at the pom file some more.
Thanks very much Craig for the suggestions. I disabled spotless and error-prone and started getting meaningful stack traces. I worked through needed jdk-11 → jdk-8 source changes and got the TensorFlow Native Library subproject to compile.
I am now getting compile failure with the TensorFlow API subproject. I has to do with the ndarray dependency. It needs jdk-8 class files (ver. 52) but the downloaded jar is compiled with jdk-11 (ver. 55). I cloned the ndarray project to compile it with jdk-8. Maven in ndarray is now giving me the same non-specific stack trace I had with the TF-Java native subproject. I looked at the ndarray pom, specifically the entries but they don’t seem optional.
I realize ndarry is no longer part of the TensorFlow project. Do you have any suggestions for compiling ndarray with jdk-8?
Same kind of thing, check out the project and replace the release 11 with release 8 in the pom file and turn off the code checkers - GitHub - tensorflow/java-ndarray.