TFLite Classification Model Grayscale Input in Java

Hello,

I trained a Keras CNN model for a classification task using grayscale images (48x48x1). After training and saving the model, I now would like to use the model in an Android application, so I converted the trained model to a TFLite model.

After conversion, I did two things in order to confirm that the converted model works properly:

  1. I tested running inferences on some grayscale test images using the TFLite model in Python, and got proper and expected results (using the same dimensions for the input as I used for training).

  2. I inspected the TFLite model in Netron and confirmed that it expects an input of the following format: float32[1, 48, 48, 1].

The issue I am facing now is with loading input images to the TFLite model using Java in Android Studio. Since my model expects grayscale input images (1 channel), the total input bytes for one frame are supposed to be 48x48x1x4= 9216 bytes (the last 4 multiplied is because of the input being of FLOAT32 format). The problem is that the TensorImage input format, which is what the TFLite model supports, only supports RGB input images, which isn’t compatible with single-channel input images. In addition, Bitmap doesn’t even support storing grayscale images, so I wouldn’t know how to store any grayscale input images into a Bitmap and not have an extra number of bytes that would later cause issues.

I am not an expert in Android development and have slowly been learning and building things, so I understand there might be easy solutions that I’m overlooking. I would appreciate any help that comes my way!

Thanks,
Ahmad

1 Like

Hi @ahmadchalhoub99 ,

Thanks for the detailed explanation. This issue has been taken care of about a year ago with the introduction of TensorFlow Lite Support TransformToGrayScaleOp.java operator. Check inside the file to see how this is handled.

So inside your code you can use the TensorFlow Lite Support library using the Basic image manipulation and conversion adding the operator to the ImageProcessor.

I hope it helps.
Tag me if you need something more.

Regards

2 Likes

Hi @George_Soloupis

Thank you so much for your response!

After finding out about the ImageProcessor support in TFLite, I tried to check if there was any op that would take care of the conversion to grayscale. However, I couldn’t find any, and I just learned that this was due to my using ‘org.tensorflow:tensorflow-lite-support:0.1.0’ in my build.gradle file instead of ‘org.tensorflow:tensorflow-lite-support:0.2.0’ - only the latter supports the TransformToGrayscaleOp.

One comment that I do have is that no matter how long and hard I searched for a solution online, I was not able to come across any documentation for the TransformToGrayscaleOp online. I’m not sure if it is anything to do with my way of searching, but it would definitely be more convenient if the documentation for things like these were more easily ‘findable’.

But again, I really appreciate your help! I was stuck on this for a few days and your answer solves my issue!

Thanks,
Ahmad

1 Like

One comment that I do have is that no matter how long and hard I searched for a solution online, I was not able to come across any documentation for the TransformToGrayscaleOp online

Hey @ahmadchalhoub99 - Mark from the TF Docs team here. Sorry this wasn’t optimal for you - can you provide some examples of how you tried to find it (e.g. “did a google search for X, Y, Z”, “browsed this repo: …”)? Be as verbose as you would like :slight_smile:

There are many paths to the many docs, and hearing your experiences, even anecdotally, helps us with our content strategy, whether it be adding the right metadata, or building better supplementary content (codelabs, etc).

2 Likes

@ahmadchalhoub99 You can also do that manually. First you can turn your Bitmap to grayscale:

private fun androidGrayScale(bmpOriginal: Bitmap): Bitmap {
        val width: Int
        val height: Int
        height = bmpOriginal.height
        width = bmpOriginal.width
        val bmpGrayscale = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
        val canvas = Canvas(bmpGrayscale)
        val paint = Paint()
        val colorMatrix = ColorMatrix()
        colorMatrix.setSaturation(0f)
        val colorMatrixFilter = ColorMatrixColorFilter(colorMatrix)
        paint.colorFilter = colorMatrixFilter
        canvas.drawBitmap(bmpOriginal, 0f, 0f, paint)
        return bmpGrayscale
    }

and then take the byteBuffer from one channel of the above:

private fun getByteBufferNormalized(bitmapIn: Bitmap): ByteBuffer {
        val bitmap = Bitmap.createScaledBitmap(
            bitmapIn,
            CONTENT_IMAGE_WIDTH,
            CONTENT_IMAGE_HEIGHT,
            true
        )
        val width = bitmap.width
        val height = bitmap.height
        val mImgData: ByteBuffer = ByteBuffer.allocateDirect(4 * width * height)
        mImgData.order(ByteOrder.nativeOrder())
        val pixels = IntArray(width * height)
        bitmap.getPixels(pixels, 0, width, 0, 0, width, height)
        for (pixel in pixels) {
            mImgData.putFloat(Color.blue(pixel).toFloat() / 255.0f)
        }
        return mImgData
    }

but the TransformToGrayscaleOp uses more sofisticated method to grayscale the image with a matrix that is suggested from OpenCV library.

You can see this inside the Operator.

Regards