How can I fine-tune EfficientNetB3 model and retain some of its exisiting labels?

Hi guys,

I’ve tested EfficientNetB3 model (trained on ImageNet) on my large image set and it recognizes some classes of images that I have with varying accuracy, the others are not recognized at all.

For example, it does a great job for school buses: ('n04146614', 'school_bus') and a decent job for ('n04487081', 'trolleybus'), ('n02701002', 'ambulance'), ('n03977966', 'police_van').

So I would like to keep these labels and feed more images to the model to improve their detection rate. At the same time, while it detects police vans, it completely misses other police vehicles, so I would have to create new labels for them.

How should I approach? Is this possible in one training session?