Tensorflow HUB - transfer learning for object detection

Fraccalo · October 11, 2021, 2:43am

Hi all,

I am having some trouble with applying transfer learning in object detection models.
The documentation provides a nice tutorial for transfer learning in classification models (إعادة تدريب مصنف الصور | TensorFlow Hub). However, I couldn’t find anything similar for the object detection case.

I tried to readapt the object detection tutorial (TensorFlow Hub Object Detection Colab) according to the classification tutorial linked above, but I didn’t manage to progress much and got a bunch of errors that couldn’t solve.
Does anyone have any suggestion in this sense (or can suggest clear tutorials that explain this stuff)?

Thanks

Bhack · October 11, 2021, 11:03am

Bhack · October 11, 2021, 11:06am

Please note that we are also migrating the detection folder. Read the disclaimer at:

https://github.com/tensorflow/models/tree/master/official/vision/detection#object-detection-models-on-tensorflow-2

Fraccalo · October 11, 2021, 11:44am

Hi @Bhack, thanks a lot for your reply!

I have been using the tensorflow Object Detection API lately.
There are a few things that I don’t particularly like (absence of Callbacks, for example), and some bugs that arise when trying more custom usages (see for example the bug issue I opened last week: Object detection API - error when finetuning centernet and faster_rcnn models: RuntimeError: Groundtruth tensor weights has not been provided · Issue #10295 · tensorflow/models (github.com)). I also find the code for the models, meta architectures, training, etc. quite obscure (I am not a TF expert), so I was looking for an alternative approach.

I also noticed that the detection folder on the official repo is being deprecated as you are migrating it, and that’s why I was curious to know if there is an alternative approach to transfer learning in object detection that could be used in a similar way to the TF HUB transfer learning for image classification (that I linked in the OP).

If that’s not the case, I guess I’ll have to wait for the official release of object detection

Bhack · October 11, 2021, 11:51am

We had many issues about a pure TFHub approach but I don’t know if we have something in the TFHub roadmap:

github.com/tensorflow/hub

Make Object detection models fine-tunable

opened 10:51AM - 23 Sep 20 UTC

closed 01:46PM - 09 Jun 21 UTC

RobinBaumann

type:module request type:feature

As I understand from #473, it is not possible to fine-tune object detection mode…ls fetched from TF Hub directly. It would be nice to have this as a feature. E.g., with the already existing Keras integration via hub.KerasLayer, it would be nice to enable a workflow similar to: 1. Load Model (e.g. https://tfhub.dev/tensorflow/ssd_mobilenet_v2/fpnlite_640x640/1 via TF Hub as hub.KerasLayer) 2. Add new input and output layers (In the [Retrain an Image classifier guide](https://www.tensorflow.org/hub/tutorials/tf2_image_retraining), this is done in a Sequential model, I think we need the functional API because the resulting hub.KerasLayer will have multiple outputs). 3. Build a new Keras Model with the desired inputs and outputs and fine tune it on custom dataset. Please correct me if this is already possible. I played around with TF Hub the past few days and this is what I tried until now: ```python import tensorflow as tf import tensorflow_hub as hub MODULE_HANDLE = "https://tfhub.dev/tensorflow/ssd_mobilenet_v2/fpnlite_640x640/1" # Try to set shapes for new dataset, which contains 10 foreground classes, note that this somehow does not work shapes = { 'num_detections': (None), 'detection_boxes': (None, 4), 'detection_classes': (None), 'detection_scores': (None), 'raw_detection_boxes': (None, 4), 'raw_detection_scores': (None, 10), 'detection_anchor_indices': (None), 'detection_multiclass_scores': (None, 11) } module = hub.KerasLayer(MODULE_HANDLE, trainable=True, output_shape=shapes) ``` Results in > ERROR:absl:hub.KerasLayer is trainable but has zero trainable weights. > ValueError: Tensor's shape (1, 100, 91) is not compatible with supplied shape (1, None, 11) **Expected behavior (Feature Request):** Regarding the workflow described above, in (pseudo-ish) code something like the following would be nice: ```python MODULE_HANDLE = "https://tfhub.dev/tensorflow/ssd_mobilenet_v2/fpnlite_640x640/1" input_layer = tf.keras.layers.InputLayer(input_shape=(None, None, 3), dtype=tf.float32) # instead of output_shape, it would be nice if the option exists to only provide the number of predicted classes of the new dataset module = hub.KerasLayer(MODULE_HANDLE, trainable=True, num_classes=NEW_CLASSES)(input_layer) model = tf.keras.Model(inputs=input_layer, outputs=[module['detection_boxes'], module['detection_scores']]) model.compile(...) ```

github.com/tensorflow/hub

Finetune tfhub object detection model on Custom data (TensorFlow 2)

opened 05:57AM - 17 Apr 21 UTC

closed 10:57AM - 03 May 21 UTC

ameyparanjape

type:support type:docs stat:awaiting response subtype:Image Retraining subtype:image-feature-vector hub

I am interested in training object detectors using efficientdet models. I was ab…le to train the model using the official repo, but I could not train the efficientdet-lite models using the official repo. So I'm interested in using tfhub lite model/feature vector to finetune/retrain on custom data. It seems possible, but I could not find any documentation/notes on how to retrain tfhub object detector. All the docs show the inference only. Any help on this would be greatly appreciated!

Fraccalo · October 11, 2021, 12:18pm

@Bhack thanks again!
I somehow missed those issues when I was looking for fine-tuning object detection models in TF Hub.
Do you know if there is any ETA for the new official object detection folder (I suspect the answer is no, but it’s worth asking :P)?

lgusm · October 11, 2021, 1:47pm

Hi, sorry for being late but as @Bhack mentioned, the Object Detection fine tuning story on TFHub is not perfect. Most of the OD models that were published don’t have the proper api to be fine tuned easily (like the image classification ones you mentioned)

Depending on the project you’re building, I’d suggest you try Model Maker (TensorFlow Lite Model Maker를 사용한 객체 감지) as it’s the simplest solution you’ll find

Winton_Cape · October 12, 2021, 12:55am

I have tried to use this example for custom object detection… and I get the following error…
—> 60 inputs, attrs, num_outputs)
61 except core._NotOkStatusException as e:
62 if name is not None:

2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm.

It the notebook only seem to work if you set it to CPU mode ( Edit=>Notebook settings to none) which takes a very very long time

lgusm · October 12, 2021, 11:34am

Can you try it with GPU but removing the TF instalation on the first line? The training will be much faster

Winton_Cape · October 12, 2021, 5:28pm

Hi,
Thanks for the tip. I did try it and it ran much faster; I was able to move beyond the training cell. However, once I got to the evaluate cell… model.evaluate_tflite(“model.tflite”, test_data), I again got an error…
InvalidArgumentError: Incompatible shapes: [1,25] vs. [4] [Op:Mul]

I think that I read somewhere that the order of the outputs were changed??? Do you know how to get past this error?

lgusm · October 13, 2021, 10:17am

Sorry for that Winton,
I tried here and got the same issue. Raised a bug and it’ll be fixed soon. I’ll keep you posted!

Winton_Cape · October 13, 2021, 5:33pm

Thanks… looking forward to a solution.

santhoshnumberone · April 25, 2022, 4:01am

Hey @all

After going through extensively through each and every like that has been posted here.

I can come to the conclusion that Training Custom Object Detector with tensorflow 2 Object Detection API tutorialhttps://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html is of no use.

As training the entire network on new data has been shown here by one of the user to yield worst results than the pre trained network used for transfer learning before training sourcehttps://github.com/tensorflow/models/issues/2203?#issuecomment-883248328
Have no idea if the user has played around changing the learning rate either case. Could love to preserve the base feature extraction layer and train the prediction final layer

Adding freeze_variable forcefully is not being detected being training also sourcehttps://github.com/tensorflow/models/issues/10167

Can we help each other replicate Transfer learning and fine-tuning on tensorflow where they are using keras for object detection the example on the post is a classification network?

I am willing to contribute for getting this thing working, if anyone can pitch in.

I am having same issue which I have posted on tensorflow/models repo Transfer learning and fine tuning of hidden layers and the final layer by looking at the history of issue comments on that repo I seriously doubt anyone will even reply to the issue.