Which checkpoint to use?

General Discussion
, ,
#1

When starting training object detection, I get the following output:

Step 100 per-step time 2.710s
INFO:tensorflow:{'Loss/classification_loss': 0.039203472,
 'Loss/localization_loss': 0.0006053609,
 'Loss/regularization_loss': 0.028673949,
 'Loss/total_loss': 0.06848278,
 'learning_rate': 0.00416}

and a checkpoint file is created. I have set the training to create a checkpoint every 500 step.
So, at step 500 it creates a new checkpoint saying

Step 500 per-step time 1.811s
INFO:tensorflow:{'Loss/classification_loss': 0.027229222,
 'Loss/localization_loss': 0.00049498235,
 'Loss/regularization_loss': 0.02863204,
 'Loss/total_loss': 0.056356244,
 'learning_rate': 0.016800001}
I0309 08:09:52.685846 139872661055296 model_lib_v2.py:707] {'Loss/classification_loss': 0.027229222,
 'Loss/localization_loss': 0.00049498235,
 'Loss/regularization_loss': 0.02863204,
 'Loss/total_loss': 0.056356244,
 'learning_rate': 0.016800001}

And I now have 2 checkpoints (ckpt-1 and ckpt-2).
Let’s say I would like the weights from the step 500. Should I then load ckpt-1 or ckpt-2? I find it confusing since I do not expect it to create ckpt-1 when the training start, as I expect this at step 500.

Thanks for any help!

#3

Hi @TensorOverflow May i know for how many epochs have you trained your model. Thank you.