I am using the Sharpness Aware Minimization to subclass model, as shown here:
The problem I am facing is although it appears to dramatically improve model convergence (and, hopefully but TBD to improve generalization), the evaluation loss reported on each epoch appears to be incorrect. Here is an example:
Epoch 72/100
1760/1760 [==============================] - ETA: 0s - loss: 0.2741 - accuracy: 0.9187
Epoch 72: val_accuracy improved from 0.88879 to 0.89052, saving model to /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf
WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 5). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf/assets
INFO:tensorflow:Assets written to: /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf/assets
1760/1760 [==============================] - 44s 25ms/step - loss: 0.2741 - accuracy: 0.9187 - val_loss: 50.3961 - val_accuracy: 0.8905
Epoch 73/100
1760/1760 [==============================] - ETA: 0s - loss: 0.2696 - accuracy: 0.9199
Epoch 73: val_accuracy improved from 0.89052 to 0.89269, saving model to /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf
WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 5). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf/assets
INFO:tensorflow:Assets written to: /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf/assets
1760/1760 [==============================] - 44s 25ms/step - loss: 0.2696 - accuracy: 0.9199 - val_loss: 50.4510 - val_accuracy: 0.8927
Epoch 74/100
1758/1760 [============================>.] - ETA: 0s - loss: 0.2653 - accuracy: 0.9211
Epoch 74: val_accuracy improved from 0.89269 to 0.89468, saving model to /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf
WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 5). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf/assets
INFO:tensorflow:Assets written to: /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf/assets
1760/1760 [==============================] - 45s 25ms/step - loss: 0.2653 - accuracy: 0.9211 - val_loss: 50.5057 - val_accuracy: 0.8947
Epoch 75/100
1760/1760 [==============================] - ETA: 0s - loss: 0.2611 - accuracy: 0.9221
Epoch 75: val_accuracy improved from 0.89468 to 0.89627, saving model to /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf
WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 5). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf/assets
INFO:tensorflow:Assets written to: /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf/assets
1760/1760 [==============================] - 44s 25ms/step - loss: 0.2611 - accuracy: 0.9221 - val_loss: 50.5603 - val_accuracy: 0.8963
Epoch 76/100
1758/1760 [============================>.] - ETA: 0s - loss: 0.2570 - accuracy: 0.9232
Epoch 76: val_accuracy improved from 0.89627 to 0.89790, saving model to /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf
WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 5). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf/assets
INFO:tensorflow:Assets written to: /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf/assets
1760/1760 [==============================] - 44s 25ms/step - loss: 0.2569 - accuracy: 0.9232 - val_loss: 50.6153 - val_accuracy: 0.8979
Epoch 77/100
1760/1760 [==============================] - ETA: 0s - loss: 0.2531 - accuracy: 0.9244
Epoch 77: val_accuracy improved from 0.89790 to 0.89966, saving model to /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf
WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 5). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf/assets
INFO:tensorflow:Assets written to: /mnt/c/eaf_llc/aa-analytics_and_bi/alliance_molds/radome_quality/image_analysis/models/2023-01-24_11_10_47_best_model.tf/assets
You can see the train loss decreases and both accuracies are increasing, but it reports the val loss as actually increasing.