Warnings while training

Muhammed_Magdy · May 16, 2022, 11:23am

I got these warnings while training movinet on my custom data following the movinet_tutorial.ipynb
Is this normal?
and does this affect training and accuracy? because I got small accuracy
my code is here Google Colab

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.SpatialAveragePool3D object at 0x7fa760443e90>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.SpatialAveragePool3D object at 0x7fa7604064d0>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.SpatialAveragePool3D object at 0x7fa7601007d0>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.SpatialAveragePool3D object at 0x7fa760109b50>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.SpatialAveragePool3D object at 0x7fa760092d90>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.SpatialAveragePool3D object at 0x7fa6fa6bea90>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.SpatialAveragePool3D object at 0x7fa6fa5ec810>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.SpatialAveragePool3D object at 0x7fa6fa5177d0>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.SpatialAveragePool3D object at 0x7fa6fa4cc9d0>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.SpatialAveragePool3D object at 0x7fa6fa401d50>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.SpatialAveragePool3D object at 0x7fa6fa33f350>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.SpatialAveragePool3D object at 0x7fa6fa289ad0>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.SpatialAveragePool3D object at 0x7fa6fa15a950>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.SpatialAveragePool3D object at 0x7fa6fa09c5d0>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.SpatialAveragePool3D object at 0x7fa6e23b5450>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <keras.layers.core.dropout.Dropout object at 0x7fa7604c1ad0>, because it is not built.

WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.modeling.layers.nn_layers.TemporalSoftmaxPool object at 0x7fa6e15dfc10>, because it is not built.

Bhack · May 16, 2022, 12:13pm

Check:

github.com/tensorflow/models

movinet a0 model fine-tuning and save weight/model issues

opened 07:47PM - 16 Nov 21 UTC

closed 06:38AM - 19 Nov 21 UTC

liveinfootball

type:support models:official

Hi all! Appreciate your support in advance! I successfully downloaded the …a0 model from tf hub and trained with my custom dataset. The result was great but saved weights/model is empty when it was loaded. Is there any specific way to save model/weight for model with custom layers? <code> model_id = 'a0' tf.keras.backend.clear_session() backbone = movinet.Movinet(model_id=model_id) model = movinet_model.MovinetClassifier( backbone=backbone, num_classes=600) model.build([batch_size, num_frames, resolution, resolution, 3]) movinet_hub_url = f'https://tfhub.dev/tensorflow/movinet/{model_id}/base/kinetics-600/classification/3' movinet_hub_model = hub.KerasLayer(movinet_hub_url, trainable=True) pretrained_weights = {w.name: w for w in movinet_hub_model.weights} model_weights = {w.name: w for w in model.weights} for name in pretrained_weights: model_weights[name].assign(pretrained_weights[name]) model = movinet_model.MovinetClassifier( backbone=backbone, num_classes=num_classes) model.build([batch_size, num_frames, resolution, resolution, 3]) for layer in model.layers[:-2]: layer.trainable = True train_steps = num_examples['train'] // batch_size total_train_steps = train_steps * num_epochs test_steps = num_examples['test'] // batch_size loss_obj = tf.keras.losses.CategoricalCrossentropy( from_logits=True, label_smoothing=0.1) metrics = [ tf.keras.metrics.TopKCategoricalAccuracy( k=1, name='top_1', dtype=tf.float32), tf.keras.metrics.TopKCategoricalAccuracy( k=2, name='top_2', dtype=tf.float32), ] initial_learning_rate = lr learning_rate = tf.keras.optimizers.schedules.CosineDecay( initial_learning_rate, decay_steps=total_train_steps, ) optimizer = tf.keras.optimizers.RMSprop( learning_rate, rho=0.9, momentum=0.9, epsilon=1.0, clipnorm=1.0) model.compile(loss=loss_obj, optimizer=optimizer, metrics=metrics) checkpoint_filepath = os.path.join('chkpnts', 'weights.{epoch:02d}-{val_loss:.2f}.hdf5') model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint( filepath=checkpoint_filepath, save_weights_only=False, monitor='val_accuracy', mode='max', save_best_only=False, verbose=1) callbacks = [ tf.keras.callbacks.TensorBoard(log_dir='./logs'), model_checkpoint_callback, ] model.summary() </code> Model: movinet_classifier_1 _________________________________________________________________ Layer (type) Output Shape Param \# _________________________________________________________________ image (InputLayer) [(None, None, None, None, 0 _________________________________________________________________ movinet (Movinet) ({'stem': (None, None, No 911583 _________________________________________________________________ classifier_head_1 (Classifie (None, 4) 993284 _________________________________________________________________ Total params: 1,904,867 Trainable params: 1,890,595 Non-trainable params: 14,272 <code> results = model.fit( train_dataset, validation_data=test_dataset, epochs=num_epochs, steps_per_epoch=train_steps, validation_steps=test_steps, callbacks=callbacks, validation_freq=1, verbose=1) </code> Epoch 1/2 1318/1318 [==============================] - 865s 636ms/step - loss: 1.0455 - top_1: 0.5992 - top_2: 0.8443 - val_loss: 0.9499 - val_top_1: 0.6970 - val_top_2: 0.8970 Epoch 00001: saving model to chkpnts/weights.01-0.95.hdf5 Epoch 2/2 1318/1318 [==============================] - 835s 634ms/step - loss: 0.7417 - top_1: 0.8024 - top_2: 0.9454 - val_loss: 0.7451 - val_top_1: 0.7992 - val_top_2: 0.9371 Epoch 00002: saving model to chkpnts/weights.02-0.75.hdf5 - check point was generated but it didn't have the trained weights. <code> tf.keras.models.save_model( model, './chkpnts/model_v8', options=tf.saved_model.SaveOptions(experimental_custom_gradients=True)) </code> WARNING:tensorflow:Skipping full serialization of Keras layer <keras.layers.core.Dropout object at 0x7f4c5f7e1080>, because it is not built. WARNING:tensorflow:Skipping full serialization of Keras layer <keras.layers.core.Dropout object at 0x7f4c5f7e1080>, because it is not built. WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.beta.modeling.layers.nn_layers.TemporalSoftmaxPool object at 0x7f4c5f7e1cc0>, because it is not built. WARNING:tensorflow:Skipping full serialization of Keras layer <official.vision.beta.modeling.layers.nn_layers.TemporalSoftmaxPool object at 0x7f4c5f7e1cc0>, because it is not built. WARNING:absl:Found untraced functions such as init_states, head_layer_call_fn, head_layer_call_and_return_conditional_losses, classifier_layer_call_fn, classifier_layer_call_and_return_conditional_losses while saving (showing 5 of 1556). These functions will not be directly callable after loading. INFO:tensorflow:Assets written to: ./chkpnts/model_v8/assets INFO:tensorflow:Assets written to: ./chkpnts/model_v8/assets - model_v8 was generated with errors and it is empty.

Muhammed_Magdy · May 16, 2022, 5:56pm

I can’t find a solution in the link you sent
Is there a thing I am missing ?

Bhack · May 16, 2022, 6:17pm

Yes sorry wrong link I’ve fixed it.
It was already triaged a similar issue on Movinet. Can you check the proposed solution about the saved_model format?