ModelCheckpoint saves a model that only returns ones ( Keras )

General Discussion
,
#1

In my model.fit() I’m using this callback:

callbacks.ModelCheckpoint(f"checkpoint/{name}.h5", monitor='val_loss', save_best_only=True, save_freq='epoch')

Later after training when I tried using the model with model.predict([img]) it always returns a one. This doesn’t happen with a model saved with model.save(). Can anyone help me get modelcheckpoint working?

#3

It hard to debug only with these few details.

Can you prepare/share a minimized (few lines of code) standalone example or Colab to reproduce your issue?

#5

Sorry for the long response time, I was busy with other stuff and couldn’t work on this project. My code minimized that produces the same result above on my computer even after python reinstall:

from datetime import datetime as dt

import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout, Input, BatchNormalization
from tensorflow.keras import callbacks
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.preprocessing.image import ImageDataGenerator

name = "test"
batch_size = 32

name = f"{name}-{dt.now():%d_%H_%M}"

img_inputs = Input(shape=(256, 256, 3))

x = Conv2D(32, (3, 3), padding="same", activation='relu')(img_inputs)
x = Conv2D(32, (3, 3), padding="same", activation='relu')(x)
x = BatchNormalization()(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = MaxPooling2D(pool_size=(2, 2))(x) #these extra are here so I don't have 18mil perams is the smaller model

x = Flatten()(x)
x = BatchNormalization()(x)
x = Dropout(0.5)(x)

x = Dense(units=32, activation='relu')(x)
x = Dense(units=32, activation='relu')(x)
x = Dropout(0.2)(x)
x = BatchNormalization()(x)

output = Dense(units=1, activation='sigmoid')(x)

model = Model(inputs=img_inputs, outputs=output, name=name)
model.compile(optimizer=Adam(), loss='binary_crossentropy', metrics=['accuracy'])

train_datagen = ImageDataGenerator(
    rescale = 1 / 255,
    validation_split=0.2,
)

training_set = train_datagen.flow_from_directory(
    'data',
    target_size=(256, 256),
    batch_size=batch_size,
    class_mode='binary',
    subset='training')

testing_set = train_datagen.flow_from_directory(
    'data',
    target_size=(256, 256),
    batch_size=batch_size,
    class_mode='binary',
    subset='validation')

mc = callbacks.ModelCheckpoint(f"checkpoint/{name}.h5", monitor='val_loss', save_best_only=True, save_freq='epoch')

model.fit(
        training_set,
        steps_per_epoch = training_set.samples // batch_size,
        epochs=200,
        callbacks=[mc],
        batch_size=batch_size,
        validation_data = testing_set, 
        validation_steps = testing_set.samples // batch_size,
    )

model.save("model.h5")
#6

Can you minimize to use dummy data as we need something to copy, paste and run.

model.predict([img]) it always returns a one.

Also please add the lines to verify/reproduce this effect.

#7

Well, the effect is in the models saved by model checkpoint, so I have to use a separate file. I’m also not sure how to minimize data it’s a dataset of 50k 256x256 images. The fact that it is an image might be a key reason I’m not sure. I guess I’ll have to test that too.

Heres the testing script:

import cv2
import tensorflow
import os

img_size = 256

model = tensorflow.keras.models.load_model("checkpoint/test-19_15_29.h5")

category = "dogs"
for file in os.listdir(f"./data/{category}"):
    img_array = cv2.imread(f"./data/{category}/{file}", cv2.IMREAD_COLOR)
    new_array = cv2.resize(img_array, (img_size, img_size))
    img = new_array.reshape(-1, img_size, img_size, 3)
    
    prediction = model.predict([img])
    print(prediction)

Edit: I forgot to add, I’m running python 3.9.

#9

Are you going to apply on prediction the same preprocessing you do in training with ImageDataGenerator?

#11

You know, I suddenly feel unbelievably dumb. I do find it odd that the fully trained model was working fine. Well thanks though, you saved me a lot of pain.

1 Like