How to link TensorFlow_Text ops into Android/iOS binaries?

cpoohee · July 26, 2021, 12:38pm

I have a tensorflow-lite model that requires some TensorFlow_Text ops (CaseFoldUTF8 & RegexSplitWithOffsets).

While using python’s tflite interpreter, I can use

import tensorflow_text as tf_text

to load the necessary tensorflow_text binaries for inference.

But by deploying on Android, the monolithic AARs libraries from Maven Central (both lite and select-ops) do not include tensorflow_text ops, which gives me errors such as:

Op type not registered 'CaseFoldUTF8' in binary running on localhost. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.

Looking at the tflite’s guide from Supported Select TensorFlow operators | TensorFlow Lite , it is stated that, ‘On the runtime side, it is also required to link the TensorFlow Text or SentencePiece library into the final app or binary.’

The question is, how do I build AARs that includes TensorFlow-Text ops?
I have tried using using the tensorflow/lite/tools/build_arr.sh to build the AAR to my model, and it fails because of similar errors like “Op CaseFoldUTF8 not found”.

Also, while I haven’t tried on iOS binaries, do I need to specially build the iOS frameworks that include TensorFlow Text ops too? If so, how do I do that?

George_Soloupis · July 28, 2021, 4:21am

Hi @cpoohee

So you have added:

implementation ‘org.tensorflow:tensorflow-lite-select-tf-ops:2.5.0’

inside android project and it still throws an error?

Also can you provide a link to download the .tflite file to do an experiment?

cpoohee · July 28, 2021, 7:50pm

Hi, @George_Soloupis ,
Thank you for responding to my help request.
Well, I was running on flutter’s tflite plugin with my modification to include flex delegate. I have included both binaries from extracted from the those AARs.
https://search.maven.org/artifact/org.tensorflow/tensorflow-lite/2.5.0/aar and https://search.maven.org/artifact/org.tensorflow/tensorflow-lite-select-tf-ops/2.5.0/aar.
This should have a similar effect to having these on gradle:

implementation ‘org.tensorflow:tensorflow-lite:2.5.0’
implementation ‘org.tensorflow:tensorflow-lite-select-tf-ops:2.5.0’

My errors of my current model is showing those error messages:

I/tflite  (27736): Created TensorFlow Lite delegate for select TF ops.
I/tflite  (27736): Initialized TensorFlow Lite runtime.
I/native  (27736): core/platform/cpu_feature_guard.cc:142 This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2
I/native  (27736): To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
I/tflite  (27736): TfLiteFlexDelegate delegate: 71 nodes delegated out of 512 nodes with 25 partitions.
I/tflite  (27736): Replacing 71 node(s) with delegate (TfLiteFlexDelegate) node, yielding 50 partitions.
E/tflite  (27736): Op type not registered 'CaseFoldUTF8' in binary running on localhost. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
E/tflite  (27736): Op type not registered 'RegexSplitWithOffsets' in binary running on localhost. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 2 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 2 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 2 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 2 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 2 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 5 nodes delegated out of 15 nodes with 2 partitions.
I/tflite  (27736): Replacing 5 node(s) with delegate (TfLiteFlexDelegate) node, yielding 4 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 8 nodes delegated out of 20 nodes with 2 partitions.
I/tflite  (27736): Replacing 8 node(s) with delegate (TfLiteFlexDelegate) node, yielding 4 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 2 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 2 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 2 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 2 nodes delegated out of 4 nodes with 1 partitions.
I/tflite  (27736): Replacing 2 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 2 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 2 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 2 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 4 nodes delegated out of 10 nodes with 2 partitions.
I/tflite  (27736): Replacing 4 node(s) with delegate (TfLiteFlexDelegate) node, yielding 4 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 1 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/chatty  (27736): uid=10154(app.pockett.mlkit_test) 1.ui identical 2 lines
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 1 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 1 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 1 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 1 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 2 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 2 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 2 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 2 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 0 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 8 nodes delegated out of 20 nodes with 2 partitions.
I/tflite  (27736): Replacing 8 node(s) with delegate (TfLiteFlexDelegate) node, yielding 4 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 1 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 2 nodes delegated out of 40 nodes with 2 partitions.
I/tflite  (27736): Replacing 2 node(s) with delegate (TfLiteFlexDelegate) node, yielding 5 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 1 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 9 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 2 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 3 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 5 nodes delegated out of 10 nodes with 2 partitions.
I/tflite  (27736): Replacing 5 node(s) with delegate (TfLiteFlexDelegate) node, yielding 4 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 1 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 4 nodes delegated out of 91 nodes with 2 partitions.
I/tflite  (27736): Replacing 4 node(s) with delegate (TfLiteFlexDelegate) node, yielding 4 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 1 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 2 nodes delegated out of 9 nodes with 2 partitions.
I/tflite  (27736): Replacing 2 node(s) with delegate (TfLiteFlexDelegate) node, yielding 4 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 1 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 2 nodes delegated out of 18 nodes with 2 partitions.
I/tflite  (27736): Replacing 2 node(s) with delegate (TfLiteFlexDelegate) node, yielding 4 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 1 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 3 nodes delegated out of 8 nodes with 2 partitions.
I/tflite  (27736): Replacing 3 node(s) with delegate (TfLiteFlexDelegate) node, yielding 4 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 1 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 3 nodes delegated out of 8 nodes with 2 partitions.
I/tflite  (27736): Replacing 3 node(s) with delegate (TfLiteFlexDelegate) node, yielding 4 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 1 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 3 nodes delegated out of 8 nodes with 2 partitions.
I/tflite  (27736): Replacing 3 node(s) with delegate (TfLiteFlexDelegate) node, yielding 4 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 2 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 1 nodes delegated out of 17 nodes with 1 partitions.
I/tflite  (27736): Replacing 1 node(s) with delegate (TfLiteFlexDelegate) node, yielding 3 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 0 nodes delegated out of 2 nodes with 0 partitions.
I/tflite  (27736): TfLiteFlexDelegate delegate: 3 nodes delegated out of 11 nodes with 2 partitions.
I/tflite  (27736): Replacing 3 node(s) with delegate (TfLiteFlexDelegate) node, yielding 4 partitions.
E/tflite  (27736): Delegate kernel was not initialized
E/tflite  (27736): Node number 512 (TfLiteFlexDelegate) failed to prepare.

Apologies, for such a long log, the full model has many flex ops. I will try to replicate the issue on just the Android project with a smaller gist tflite model. I will share that with you when I’m ready.

George_Soloupis · July 29, 2021, 3:53am

cpoohee:

E/tflite  (27736): Op type not registered 'CaseFoldUTF8' in binary running on localhost. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
E/tflite  (27736): Op type not registered 'RegexSplitWithOffsets' in binary running on localhost. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.

So you have two operators that need to be inserted. Currently, tensorflow-lite-select-tf-ops:2.5.0 only contains a subset of selected TF ops. You can refer to this page for detailed ops.
There is also this web link when there are instructions about TensorFlow Text and SentencePiece operators where you can see these two ones that you use.
I asked you to send me the file so I can give a try to this Colab notebook where you can build select-tf-ops.aar file from a .tflite model with special ops. If you give it a try inform us of the result when you import it inside android or flutter. If you cannot make it work ping me again. Pay attention to the last info where you have to download and place a custom .sh file inside tensorflow directory. Also use less target_arch let’s say ‘arm64-v8a,armeabi-v7a’ as this procedure takes a lot of time. (Of course do not use the other build bazel commands that create Support libraries ).

Best

cpoohee · August 5, 2021, 12:23pm

Hi George,

I had tried to run your notebook on colab, unfortunately i got disconnected before it’s completed. Nevertheless, I tried to copy your build_select_tf_ops_aar.sh on my machine to compile from source (v2.5.0). I have a slightly different android configurations, ndk version 19c, sdk version 30.0.3. I only attempted to build x86_64. Near the end of the compilation, i still get the following errors:

ERROR: /home/cph/Documents/tensorflowRep/tensorflow/tmp/BUILD:19:28: Listing flex ops from savedTokenizerModel.tflite... //tmp:custom_tensorflowlite_flex_flex_delegate_tf_op_headers_list_flex_ops failed (Aborted): bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)
2021-08-05 16:40:20.507129: F tensorflow/lite/tools/list_flex_ops.cc:58] Op CaseFoldUTF8 not found: Not found: Op type not registered 'CaseFoldUTF8' in binary running on cph. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
/bin/bash: line 1: 36001 Aborted                 (core dumped) bazel-out/host/bin/tensorflow/lite/tools/list_flex_ops_main --graphs=tmp/savedTokenizerModel.tflite > "bazel-out/android-x86_64-opt/bin/tmp/custom_tensorflowlite_flex_flex_delegate_tf_op_headers_tf_generated_kernel_header/list_flex_ops"
Target //tmp:tensorflow-lite-select-tf-ops failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 6.863s, Critical Path: 1.92s
INFO: 5 processes: 3 internal, 2 local.
FAILED: Build did NOT complete successfully

I am starting to wonder if I need to edit the BUILD files to include tensorflow_text dependancies. In another recent post, thisisjoshstone was attempting to compile tf_text ops onto the benchmark tool. There were some mentions on adding @org_tensorflow_text//tensorflow_text:ops_lib as a dependency. However, I am unfamiliar with the inner workings of how bazel compiles TF libraries.

Anyway, I’m providing a gist model definition in colab, as well as the its converted tflite model in the links below.

The gist model in colab .
The gist’s tflite model.

Also, I’m providing my tflite model that I am working on.

Full tflite model.

George_Soloupis · August 5, 2021, 2:45pm

Thank you for providing the links. I will give it a try and get back to you. I am really interested in this

George_Soloupis · August 6, 2021, 2:08pm

Hi @cpoohee

Unfortunately Colab method is really slow and gave me a timeout after 12 hours. It seems though that it was near the end and it was not giving any errors until the timeout. We cannot use this method right now. We have to go with this that explains how to build select_tf_ops.aar file with Docker.

Have you used Docker before? Have you tried this method I wrote above?

cpoohee · August 6, 2021, 7:52pm

Thank you for giving a try earlier. I haven’t tried Docker method before. I will give it a go and will report back my findings.

Meanwhile, I was trying another approach by attaching the tf_text dependancy when building flex_delegate by adding it on the /tensorflow/lite/delegate/flex/BUILD file. I was trying to replicate what thisisjoshstone did for building the benchmark tool.

tflite_flex_cc_library(
    name = "delegate",
    additional_deps = ["@org_tensorflow_text//tensorflow_text:ops_lib"],
    visibility = ["//visibility:public"],
)

This is along with copying some other tf_text dependent libs into the ‘/third_party’ folder.
I got a compiled aar (monolithic) from this way. However, I got the following errors when android is trying to load the library.

2021-08-07 03:36:58.736 4845-4845/com.example.android_tflitetest E/AndroidRuntime: FATAL EXCEPTION: main
    Process: com.example.android_tflitetest, PID: 4845
    java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "_ZN10tensorflow14ResourceHandle14ANONYMOUS_NAMEE" referenced by "/data/app/~~2bQNIzfI7nLL_ZRsoniNzw==/com.example.android_tflitetest-cQfuuYwEt3pqiLamMpM8Sg==/lib/x86_64/libtensorflowlite_flex_jni.so"...
at java.lang.Runtime.loadLibrary0(Runtime.java:1087)
        at java.lang.Runtime.loadLibrary0(Runtime.java:1008)
        at java.lang.System.loadLibrary(System.java:1664)
        at org.tensorflow.lite.flex.FlexDelegate.<clinit>(FlexDelegate.java:61)
        at java.lang.Class.classForName(Native Method)
        at java.lang.Class.forName(Class.java:454)
        at java.lang.Class.forName(Class.java:379)
        at org.tensorflow.lite.NativeInterpreterWrapper.maybeCreateFlexDelegate(NativeInterpreterWrapper.java:511)
        at org.tensorflow.lite.NativeInterpreterWrapper.applyDelegates(NativeInterpreterWrapper.java:476)
        at org.tensorflow.lite.NativeInterpreterWrapper.init(NativeInterpreterWrapper.java:88)
        at org.tensorflow.lite.NativeInterpreterWrapper.<init>(NativeInterpreterWrapper.java:66)
        at org.tensorflow.lite.Interpreter.<init>(Interpreter.java:287)
        at com.example.android_tflitetest.MainActivity.runTF(MainActivity.java:74)


... and more ...

A check on my test android project’s minSdkVersion is 21 and targetSdkVersion is 30 , which is consistent to my configs done during ./configure.
Using nm command on the .so file reveals the _ZN10tensorflow14ResourceHandle14ANONYMOUS_NAMEE is an Undefined symbol. I’m currently stuck down here, pretty much trying without knowing entirely what i’m doing . I’m still working out what’s missing.

George_Soloupis · August 7, 2021, 3:15am

If I find the time and work on the Docker method, should I use the gist or the full tflite model??

cpoohee · August 7, 2021, 11:54am

Hi, George,

I have tried the docker method using build_arr.sh on my full tflite model. It still gives me the error of Ops type not registered 'CaseFoldUTF8' in binary … similar to what I had compiled on my machine without any modifications to BUILD files.

As for which model to test, I think we should try on the smaller gist model and make it work first. The smaller gist model has a simpler input/output to verify if it’s working or not.

George_Soloupis · August 7, 2021, 3:00pm

It seems that we are in a dead end here. If Docker method could not provide a custom select-tf-ops.aar file to help you with the project there is not much we can do.

But I wonder how you have managed to create the .tflite file from the SavedModel? Do you have a colab notebook to check?

cpoohee · August 7, 2021, 6:42pm

Honestly, my current full model is a little complex to refactor into colab now as it is written into bits of classes. It was saved into SavedModels, loaded back for outputs verification, then converted using tf.lite.TFLiteConverter.from_saved_model(). There were some issues regarding space optimisation/quantisation as well, but I will probably explore after I at least get the model running on mobile.

I was trying to get around the issue of tflite’s currently unsupported initialisation of vocab.txt into lookup tables specifically for ‘hashtablev2’ ops. This is required for Bert’s wordpiece tokeniser. So i went and rewritten the lookup table ops parts, replace it with tflite conversion friendly code. It’s a python dict that initialises vocabs into a normal TensorArray that tflite is able to convert and operate on.

For the Bert’s wordpiece tokeniser, there is a preprocessing step which they referred as Basic Tokeniser. This BasicTokeniser is largely what’s on the gist model i shared earlier, apart from a missing ‘nfd’ normalisation. I have thought about it if i can rewrite the tf_text ops into normal tf ops or even on mobile before it enters the model (for ndf case.). But i found that it’s non-trivial to rewrite some regex delimited splits ops. That’s why i’m stuck with using some of the tf_text ops for now.

I did briefly considered if tflite support library would help, but I’m looking for an eventual solution that works with iOS as well, so i didn’t dig further.

I felt i’m so close to making it work on mobile by just having a working binary, that’s why i’m still working around this approach.