TFLite Classification Model Grayscale Input in Java

[ahmadchalhoub99]

Hello,

I trained a Keras CNN model for a classification task using grayscale images (48x48x1). After training and saving the model, I now would like to use the model in an Android application, so I converted the trained model to a TFLite model.

After conversion, I did two things in order to confirm that the converted model works properly:

1. I tested running inferences on some grayscale test images using the TFLite model in Python, and got proper and expected results (using the same dimensions for the input as I used for training).

2. I inspected the TFLite model in Netron and confirmed that it expects an input of the following format: float32[1, 48, 48, 1].

The issue I am facing now is with loading input images to the TFLite model using Java in Android Studio. Since my model expects grayscale input images (1 channel), the total input bytes for one frame are supposed to be 48x48x1x4= 9216 bytes (the last 4 multiplied is because of the input being of FLOAT32 format). The problem is that the TensorImage input format, which is what the TFLite model supports, only supports RGB input images, which isn’t compatible with single-channel input images. In addition, Bitmap doesn’t even support storing grayscale images, so I wouldn’t know how to store any grayscale input images into a Bitmap and not have an extra number of bytes that would later cause issues.

I am not an expert in Android development and have slowly been learning and building things, so I understand there might be easy solutions that I’m overlooking. I would appreciate any help that comes my way!

Thanks,
Ahmad

[George_Soloupis]

Hi @ahmadchalhoub99 ,

Thanks for the detailed explanation. This issue has been taken care of about a year ago with the introduction of TensorFlow Lite Support TransformToGrayScaleOp.java operator. Check inside the file to see how this is handled.

So inside your code you can use the TensorFlow Lite Support library using the Basic image manipulation and conversion adding the operator to the ImageProcessor.

I hope it helps.
Tag me if you need something more.

Regards

[ahmadchalhoub99]

Hi @George_Soloupis

Thank you so much for your response!

After finding out about the ImageProcessor support in TFLite, I tried to check if there was any op that would take care of the conversion to grayscale. However, I couldn’t find any, and I just learned that this was due to my using ‘org.tensorflow:tensorflow-lite-support:0.1.0’ in my build.gradle file instead of ‘org.tensorflow:tensorflow-lite-support:0.2.0’ - only the latter supports the TransformToGrayscaleOp.

One comment that I do have is that no matter how long and hard I searched for a solution online, I was not able to come across any documentation for the TransformToGrayscaleOp online. I’m not sure if it is anything to do with my way of searching, but it would definitely be more convenient if the documentation for things like these were more easily ‘findable’.

But again, I really appreciate your help! I was stuck on this for a few days and your answer solves my issue!

Thanks,
Ahmad

[macd]

One comment that I do have is that no matter how long and hard I searched for a solution online, I was not able to come across any documentation for the TransformToGrayscaleOp online

Hey @ahmadchalhoub99 - Mark from the TF Docs team here. Sorry this wasn’t optimal for you - can you provide some examples of how you tried to find it (e.g. “did a google search for X, Y, Z”, “browsed this repo: …”)? Be as verbose as you would like

There are many paths to the many docs, and hearing your experiences, even anecdotally, helps us with our content strategy, whether it be adding the right metadata, or building better supplementary content (codelabs, etc).

[George_Soloupis]

@ahmadchalhoub99 You can also do that manually. First you can turn your Bitmap to grayscale:

private fun androidGrayScale(bmpOriginal: Bitmap): Bitmap {
        val width: Int
        val height: Int
        height = bmpOriginal.height
        width = bmpOriginal.width
        val bmpGrayscale = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
        val canvas = Canvas(bmpGrayscale)
        val paint = Paint()
        val colorMatrix = ColorMatrix()
        colorMatrix.setSaturation(0f)
        val colorMatrixFilter = ColorMatrixColorFilter(colorMatrix)
        paint.colorFilter = colorMatrixFilter
        canvas.drawBitmap(bmpOriginal, 0f, 0f, paint)
        return bmpGrayscale
    }

and then take the byteBuffer from one channel of the above:

private fun getByteBufferNormalized(bitmapIn: Bitmap): ByteBuffer {
        val bitmap = Bitmap.createScaledBitmap(
            bitmapIn,
            CONTENT_IMAGE_WIDTH,
            CONTENT_IMAGE_HEIGHT,
            true
        )
        val width = bitmap.width
        val height = bitmap.height
        val mImgData: ByteBuffer = ByteBuffer.allocateDirect(4 * width * height)
        mImgData.order(ByteOrder.nativeOrder())
        val pixels = IntArray(width * height)
        bitmap.getPixels(pixels, 0, width, 0, 0, width, height)
        for (pixel in pixels) {
            mImgData.putFloat(Color.blue(pixel).toFloat() / 255.0f)
        }
        return mImgData
    }

but the TransformToGrayscaleOp uses more sofisticated method to grayscale the image with a matrix that is suggested from OpenCV library.

You can see this inside the Operator.

Regards

[ahmadchalhoub99]

Hi Mark,

Thank you for taking the time to respond.

I wouldn’t be able to remember specifically what phrases I used to search, but it was things along the lines of: ‘load grayscale image into tensorflow lite model java android’, ‘create grayscale TensorImage’, ‘convert bitmap to grayscale TensorImage’, and things like that.

Apologies for the late response on my part!

Best,
Ahmad

[ahmadchalhoub99]

Thank you so much! It would be really beneficial to also fully understand all the steps of conversion by doing it manually, so I appreciate your very useful response!

Just as a thought, since I only recently started working with Android development (Java), would you recommend that I stick to Java and improve in it, or do you think switching over to Kotlin would be a better thing to do?

[George_Soloupis]

@ahmadchalhoub99
Well generally in the future you will get involved into projects that will require both skills. Surely Kotlin is better, with easier functions , null handling and more but you will always get into a situation where you will try to understand a block of code in Java that happens generally in the documentation. Also in the future you will like to contribute to a github library that probably it is going to be in Java. Android Studio has an easy way to insert Java code in Kotlin files by interpreting it and 99% of the cases it does a pretty good job.
So I believe that it is necessary to continue now with Java, have a good understanding of the language and when you feel ready switch to Kotlin… Try to get in touch once in a while with Java though in the future… Java will always be there in some projects and documentation.

Best

[ahmadchalhoub99]

Thank you for your detailed respose - I appreciate it!