Converting keras to .tflite - unreal classifying results

Dear all,

I’m working on a university project aiming to create an image classifier app on Android. Apart from using Tflite Model Maker, I wanted to try Keras as well - Sequential model, converted to .tflite afterwards. I get strange classification results however on my app and cannot really understand why:

import numpy as np
import PIL
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras import preprocessing
from tensorflow.keras.models import Sequential


import pathlib

dataset_url = "/content/gdrive/MyDrive/test_set"
data_dir = pathlib.Path(dataset_url)

batch_size = 16
img_height = 400
img_width = 300

train_X = tf.keras.preprocessing.image_dataset_from_directory(
  data_dir,
  validation_split=0.2,
  subset="training",
  seed=123,
  image_size=(img_height, img_width),
  batch_size=batch_size)

val_X = tf.keras.preprocessing.image_dataset_from_directory(
  data_dir,
  validation_split=0.2,
  subset="validation",
  seed=123,
  image_size=(img_height, img_width),
  batch_size=batch_size)

#val_X, test_X = tf.keras.utils.split_dataset(val_X, left_size=0.5)

class_names = train_X.class_names
print(class_names)

data_augmentation = keras.Sequential(
  [  
    layers.RandomFlip("horizontal",
                      input_shape=(img_height,
                                  img_width,
                                  3)),
    layers.RandomRotation(0.2),
    layers.RandomZoom(0.2),
   # layers.RandomBrightness([-0.2,0.2]),
    layers.RandomContrast(0.2),
  ]
)


normalization_layer = layers.Rescaling(1./255)

num_classes = len(class_names)
rescaler = layers.Rescaling(1./255)

model1 = Sequential([ # Accuracy ~0.9
  data_augmentation,
  rescaler,
  layers.Conv2D(16, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(32, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Conv2D(64, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Dropout(0.2),
  layers.Flatten(),
  layers.Dense(128, activation='relu'),
  layers.Dense(num_classes, name="outputs")
])

model1.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

#model.summary()

epochs=1
history = model1.fit(
  train_X,
  validation_data=val_X,
  epochs=epochs,
  verbose='auto',
  validation_freq=[5,10,15,20,25,30,35,40,45,50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100]
)

Here is a simple example, based mostly on official tutorial (assume that path to dataset contains directory with 8 subdirectories containing photos of respective objects - .jpg extensions, 400px height, 300px width).

My Android app uses Kotlin API just like here, in official example.

This API demands .tflite to provide metadata - for normalization. If I understood documentation, for my case mean = 0,.0 std = 255.0 so that (x - mean)/std results in range [0, 1]. Therefore I use official TF tutorial to add metadata to the model .tflite file:

converter = tf.lite.TFLiteConverter.from_keras_model(model1)
tflite_model = converter.convert()

with open('model.tflite', 'wb') as f:
  f.write(tflite_model)

I will omit desciption lines for clarity.

from tflite_support import flatbuffers
from tflite_support import metadata as _metadata
from tflite_support import metadata_schema_py_generated as _metadata_fb

model_meta = _metadata_fb.ModelMetadataT()

input_meta.content = _metadata_fb.ContentT()
input_meta.content.contentProperties = _metadata_fb.ImagePropertiesT()
input_meta.content.contentProperties.colorSpace = (
    _metadata_fb.ColorSpaceType.RGB)
input_meta.content.contentPropertiesType = (
    _metadata_fb.ContentProperties.ImageProperties)
input_normalization = _metadata_fb.ProcessUnitT()
input_normalization.optionsType = (
    _metadata_fb.ProcessUnitOptions.NormalizationOptions)
input_normalization.options = _metadata_fb.NormalizationOptionsT()
input_normalization.options.mean = [0.0]
input_normalization.options.std = [255.0]
input_meta.processUnits = [input_normalization]
input_stats = _metadata_fb.StatsT()
input_stats.max = [255]
input_stats.min = [0]
input_meta.stats = input_stats

output_meta = _metadata_fb.TensorMetadataT()
output_meta.content = _metadata_fb.ContentT()
output_meta.content.content_properties = _metadata_fb.FeaturePropertiesT()
output_meta.content.contentPropertiesType = (
    _metadata_fb.ContentProperties.FeatureProperties)
output_stats = _metadata_fb.StatsT()
output_stats.max = [1.0]
output_stats.min = [0.0]
output_meta.stats = output_stats
label_file = _metadata_fb.AssociatedFileT()
label_file.name = os.path.basename("/content/labels")
label_file.description = "Labels for objects that the model can recognize."
label_file.type = _metadata_fb.AssociatedFileType.TENSOR_AXIS_LABELS
output_meta.associatedFiles = [label_file]

subgraph = _metadata_fb.SubGraphMetadataT()
subgraph.inputTensorMetadata = [input_meta]
subgraph.outputTensorMetadata = [output_meta]
model_meta.subgraphMetadata = [subgraph]

b = flatbuffers.Builder(0)
b.Finish(
    model_meta.Pack(b),
    _metadata.MetadataPopulator.METADATA_FILE_IDENTIFIER)
metadata_buf = b.Output()

populator = _metadata.MetadataPopulator.with_model_file("/content/model.tflite")
populator.load_metadata_buffer(metadata_buf)
populator.load_associated_files(["/content/labels"])
populator.populate()

Once again please assume that /content/labels is a path for object type labels in appropriate format.

I’m nearly sure that something is wrong with conversion process, because:
→ models made by Tflite model maker work just fine on my app
→ my .tflite models achieved in process described above fail on my app and on official app as well
→ for sure as you know, the Sequential model is in general fine (I mean here that its form is taken from official documentation)

What really goes wrong with my .tflite models:
After equipping .tflite model with metadata in the way I described, the android app will not throw any errors anymore - it gives prediction on images, but:
The predictions seem to be sort of constant and out of the world: they do not change much among photos and .score is always way above 1 for each photo. Let’s say I take three photos and have three candidate labels. The .score attributes might be:
First photo:
→ label1 - .score = 14.32
→ label2 - .score = 5.23
→ label3 - .score = 3.21
Second photo:
→ label1 - .score = 14.58
→ label2 - .score = 5.64
→ label3 - .score = 3.12
Third photo:
→ label1 - .score = 14.71
→ label2 - .score = 5.17
→ label3 - .score = 3.27

The .score field is for every label around the very same level more or less between two consecutive integers (label1 - [14, 15] and so on). The range does not vary among photos. It looks like as if the order of probabilities would always be the same, no matter what photo is being processed.
(What worries me as well is the violation of .score being under 1 - if it describes probability - how is it even possible to give such outcomes
Third issue is that some labels seem to be missing in app - the size of List is below number of labels, despite setting .setMaxResults() properly).

I would be extremly grateful for a feedback of yours