I read the tutorial on fine tuning a model https://www.tensorflow.org/tutorials/images/transfer_learning#fine_tuning.
There it is said that the
base_model should be set to inference mode (
training = False when calling the
base_model) for fine tuning, because of the
BatchNormalization layers. As far as I saw and understood inference mode will also affect
Dropout layers, if these are present in the
base_model. Because of the inference mode the dropout would not get applied during fine tuning.
Is this the intended behavior for fine tuning or is it just a side effect, because the
BatchNormalization layer require inference mode and actually it would be preferred to keep
Dropout in training mode?
I would appreciate if someone could explain this further to me.