Tensorflow lite model works in python but not in java app for Android

General Discussion
, ,
#1

Hi,

I have successfully converted my tensorflow model from h5 to tflite and was able to use it in python to perform inference on a random test image (see tflite_model_test.ipynb under Colab Notebooks - Google Drive)

I tried to port it to Android Studio (Arctic Fox), loading the model to assets, and attempted two code paths to perform inference and both returned NAN values (see code below). I have been looking for answers in the last few days and couldn’t find anything that would work. I would really appreciate it if someone can let me know what I may have missed.

Code:

    String path = "/storage/emulated/0/Pictures/test_img.png"
    try {
        if (!OpenCVLoader.initDebug())
            Log.e("OpenCV", "Unable to load OpenCV!");
        else
            Log.d("OpenCV", "OpenCV loaded Successfully!");

        // Create inputs for reference.
        TensorBuffer inputFeature = TensorBuffer.createFixedSize(new int[]{1, 256, 256, 1}, DataType.FLOAT32);
        Mat image = Imgcodecs.imread(path);
        Imgproc.cvtColor(image, image, Imgproc.COLOR_RGB2GRAY);
        Imgproc.resize(image, image, new Size(256, 256));

        // Check if the image is loaded properly
        Bitmap testBmp = Bitmap.createBitmap(image.cols(), image.rows(), Bitmap.Config.ARGB_8888);
        Utils.matToBitmap(image, testBmp);

        // normalize first
        image.convertTo(image, CvType.CV_32FC(1));
        Mat dstFrame = new Mat();
        Core.divide(image, new Scalar(255.0), dstFrame);

        // load content of image into inputFeature
        float[] floats = new float[dstFrame.rows() * dstFrame.cols()];
        dstFrame.get(0,0, floats);
        byte[] bytes = floatToByte(floats);
        ByteBuffer byteBuffer = ByteBuffer.wrap(bytes);
        inputFeature.loadBuffer(byteBuffer);

        // sanity check to see if retBmp is the same as testBmp, using the Android Studio debug feature,
        // I was able to confirm this was the case, the image also matches the input used in colab
        float[] floats1 = inputFeature.getFloatArray();
        Mat retFrame = new Mat(256, 256, CvType.CV_32FC(1));
        retFrame.put(0, 0, floats1);
        Core.multiply(retFrame, new Scalar(255.0), retFrame);
        Bitmap retBmp = convertMat2Bitmap(retFrame);

        // Method 1: using the default path suggested by sample code in Android Studio for test_model.tflite
        TestModel model = TestModel.newInstance(this);
        TestModel.Outputs outputs = model.process(inputFeature);
        TensorBuffer outputFeature = outputs.getOutputFeature0AsTensorBuffer();
        float[] data = outputFeature.getFloatArray();
        // data is populated with NAN values

        int[] bbox = new int[4];
        for (int i=0; i<4; i++)
            bbox[i] = (int) (data[i] * 256);

        // Releases model resources if no longer used.
        model.close();

        // Method 2: Use Interpreter, and this yields the same result of NAN values
        MappedByteBuffer tfliteModel = FileUtil.loadMappedFile(this, "test_model.tflite");
        Interpreter tflite = new Interpreter(tfliteModel);
        TensorBuffer outputBuffer = TensorBuffer.createFixedSize(new int[]{1, 4}, DataType.FLOAT32);
        Mat retFrame2 = new Mat(256, 256, CvType.CV_32FC(1));
        tflite.run(inputFeature.getBuffer(), outputBuffer.getBuffer());
        float[] data2 = outputBuffer.getFloatArray();
        // data2 is populated with NAN values

        int[] bbox2 = new int[4];
        for (int i=0; i<4; i++)
            bbox2[i] = (int) (data2[i] * 256.0);

        Log.d("The end", "last line of the code");
    } catch (Throwable t) {
        // TODO Handle the exception
        Log.d("ERROR", t.getMessage());
    }
1 Like
#2

Hi @George_Zheng

Is it necessary to do that using OpenCV? Do you have restrictions for using that?
Wouldn’t be simpler to load directly the Bitmap from assets folder, resize it and create the ByteBuffer?

Like:

fun loadBitmapFromResources(context: Context, path: String): Bitmap {
val inputStream = context.assets.open(path)
return BitmapFactory.decodeStream(inputStream)
}

var loadedBitmap = loadBitmapFromResources(context, “woman.png”)

val inputBitmap = Bitmap.createScaledBitmap(
loadedBitmap,
width,
height,
true
)

var tensorImage = TensorImage(DataType.FLOAT32)
tensorImage.load(inputBitmap)

…

TensorBuffer outputBuffer = TensorBuffer.createFixedSize(new int[]{1, 4}, DataType.FLOAT32);
tflite.run(tensorImage.getBuffer(), outputBuffer)
val data2 = outputBuffer. getFloatArray()

and log to see the values of data2.

OR if your model contains metadata you can follow the example here where the ML Model Binding does the preprocessing also:

check the generated code for code sample.

Ping me if you need more help

#3

Hi @George_Soloupis ,

Thanks for your response. We use OpenCV extensively in Python for a multistage classification project (using prerecorded videos as input) to take advantage of its simplicity in matrix manipulation. We are in the phase of porting the code to an Android device. The example I put together is a simplified version for the first stage, and the algorithm does require each input video frame to be turned into gray scale with a single channel and then normalized so pixel values are between 0.0 and 1.0. I see your code example was written in Kotlin, which I am not familiar with, but I was able to turn that code into Java (included below so you can review).

String path = "/storage/emulated/0/Pictures/test_img.png"
try {
        InputStream inputStream = new FileInputStream(path);
        Bitmap loadedBitmap = BitmapFactory.decodeStream(inputStream);
        Bitmap scaledBitmap = Bitmap.createScaledBitmap(loadedBitmap, 256, 256, true);
        TensorImage tensorImage = new TensorImage(DataType.FLOAT32);
        MappedByteBuffer tfliteModel = FileUtil.loadMappedFile(this, "test_model.tflite");
        Interpreter tflite = new Interpreter(tfliteModel);
        TensorBuffer outputBuffer = TensorBuffer.createFixedSize(new int[]{1, 4}, DataType.FLOAT32);
        tflite.run(tensorImage.getBuffer(), outputBuffer.getBuffer());
        ...

Two follow-up questions:

  1. tensorImage contains 3 channels, but I need to turn that into grayscale and get just one channel before feeding the input to the model. What’s the correct way of doing that? The execution of the last line as is currently results in the following exception:
    java.lang.IllegalArgumentException: Cannot convert between a TensorFlowLite buffer with 262144 bytes and a Java Buffer with 786432 bytes.
  2. Note that we haven’t divided the pixel values by 255.0 before feeding that into the last line. How can I do that as required by the model, once the above error is fixed?

Thanks again and look forward to your followup.

George

#4

There is also a solution for that!

Add to your project a dependency for Support Library:

implementation ‘org.tensorflow:tensorflow-lite-support:0.2.0’

Then check documentation for ImageProcessor:

There you can directly resize, normalize and get the buffer from one channel like:

fun loadBitmapFromResources(context: Context, path: String): Bitmap {
val inputStream = context.assets.open(path)
return BitmapFactory.decodeStream(inputStream)
}
var loadedBitmap = loadBitmapFromResources(context, “woman.png”)

val imageProcessor = ImageProcessor.Builder()
.add(ResizeOp(width, height, ResizeOp.ResizeMethod.BILINEAR))
.add(TransformToGrayscaleOp())
.add(NormalizeOp(0.0F, 255.0F))
.build()

var tImage = TensorImage(DataType.FLOAT32)

tImage.load(loadedBitmap)
tImage = imageProcessor.process(tImage)

TensorBuffer outputBuffer = TensorBuffer.createFixedSize(new int[]{1, 4}, DataType.FLOAT32);
tflite.run(tImage.getBuffer(), outputBuffer)
val data2 = outputBuffer. getFloatArray()

So the basic operator that will give you one buffer out of three is TransformToGrayscaleOp() . There at this function you can see all the procedure for transforming the bitmap to grayscale and get buffer from one channel. This is an operator I and @Xunkai_Zhang created about 9 months ago :slight_smile:

If you do not want to use TensorFlow Lite Support Library then take a look at this file where there is custom manipulation of the above procedure with plain functions. Convert bitmap to grayscale:

and get buffer from one channel:

Get back if you have more issues.

Thanks

#5

Hi @George_Soloupis , just got the chance to give this a quick try and I am getting the output I was expecting. Thank you so much for the prompt guidance!

George

1 Like
#6

Hi @George_Soloupis, upon closer comparison between the colab result (see Google Colab) and the one from the Android app using code here, I noticed a slight difference. By the way, I updated the Java code to round the value:

        float[] data = outputBuffer.getFloatArray();

        int[] bbox = new int[4];
        for (int i=0; i<4; i++)
            bbox[i] = Math.round(data[i] * 256);

  • The colab inference bounding box is [0.05095369, 0.01484564, 0.96664214, 0.42464137] times 256, which leads to [ 13, 3, 247, 108]

  • The inference result using the above code is [0.04957533, 0.013957918, 0.9678618, 0.4281884] times 256, which leads to [13, 4, 248, 110]

I tried the same code on some other input images and the difference can be 3 to 4 pixels. Do you have any insight into this difference and what I may need to do to minimize it?

#7

I presume that inside colab you are using OpenCV and inside android the standard libraries for loading and resizing. There are some slight difeerences doing so. Check an article I have written about a year ago about those differences:

Best

#8

@George_Soloupis Thanks for the followup. This is very helpful!