TFX Object Detection

@tensorflow_team

I’ve been reading up on TFX and wanted to make a pipeline for object detection, using an “off-the-shelf” model from Model Hub or the object detection API. I’ve also had a look at the TF Lite model builder.

But (as far as I can see) all of these options has a snag that makes it unusable in the context of a TFX pipeline.

The closest I’ve gotten is with a Efficientdet feature-vector from Model Hub:

My question:
Is there an easy way to retrain a Tensorflow object detection model in a TFX pipeline at this moment in time?

regards,
Anders

Here is the sample example on image classification for TFX.

Train and serve a TensorFlow model with TensorFlow Serving

Yes, I’m aware of that. This question is about object detection.

Which part of object detection is sufficiently different from image classification that you’re having problems?

Hi @Robert_Crowe,
Thanks for reaching out on this issue.

My main problem is that I have not succeeded in taking a pre-built model from either Tensorflow Hub or Tensorflow Object Detection API, or anyother place.

As stated on another thread on this forum:
https://tensorflow-prod.ospodiscourse.com/t/tensorflow-hub-transfer-learning-for-object-detection/4952/8

the Object Detection fine tuning story on TFHub is not perfect. Most of the OD models that were published don’t have the proper api to be fine tuned easily (like the image classification ones you mentioned).

As you know the TFX trainer module expects the function _build_keras_model() to return a compiled Keras model.

Model HUB approach

If I’m not mistaken, I can’t use tf.keras.Sequential() (as in the above tutorial) because there is more than one output from object detection models. The class, and the box.

I tried a naive approach with the functional API:

    img_inputs = keras.Input(shape=IMAGE_SIZE + (3,))
    hub_model = hub.KerasLayer(model_handle, trainable=do_fine_tuning)
    outputs = hub_model(img_inputs)
    model = keras.Model(inputs=img_inputs, outputs=outputs, name="obj_det_model")

    loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
    model.compile(optimizer="adam", loss=loss_fn, metrics=["accuracy"])

This got me a bit further, but will not work because of the loss function. I need a combination of the Huber and Focal functions. But implementing the loss function is a bit over my level. Even if I felt comfortable doing it, it’s not quite an “of-the-shelf” solution.

Object Detection API Approach

The Tensorflow object detection API has some very good notebook tutorials but the model that is built is of type object_detection.meta_arcitectures.ssd_mera_arch.<MODEL>Arch. I have not found a way to get from that to a Keras model.

Trying to load a saved model using tf.keras.models.load_model() gives me a <tensorflow.python.saved_model.load.Loader._recreate_base_user_object.<locals>._UserObject not a keras model.

Related(?) Github issue github.com/tensorflow/tensorflow/issues/33870

Training from scratch

I’m currently following the instructions on training a model from scratch with the object detection API.

Recap

So I have tried quite a few things. I don’t see an easy way to take a predefined object detection model and use it in a TFX pipeline. I would be happy to be proven wrong with an example.

Regards,
Anders

Update

I finished training a version of Centerdet with the Object Detection API. It produces checkpoints, no saved model. I don’t know how to convert from checkpoints to the TF >2.5 version of saved_model.

AutoML

I’ve also tried the AutoML repository. But it also produces the old version of saved_model. (there is no keras_metadata.pb file)

After training, are you calling model.save()?

AutoML uses the TF v1 method b = tf.saved_model.Builder(output_dir)

https://github.com/google/automl/blob/39c39e5b635f73e86082bff3deddefecfbc0a123/efficientdet/model_inspect.py#L338

Object Detection API
Also seems to use the deprecated tf.saved_model.save

github.com/tensorflow/models/research/object_detection/exporter_lib_v2.py#L278

Model Hub

I have not come so far the I could call model.save() on a trained model.

Hello. I’m also interested in integrate Object Detection API in TFX.
More in general, is it possible using tfx trainer with custom trainer loop where there is not a compiled keras model?? ( Writing a training loop from scratch | TensorFlow Core)

Thanks!!

@smedegaard As you have found out exporter_main_v2 is the script we recommend for exporting your model.

Also, all the meta arch (like SSD etc) inherit from a keras Layer. This may be of use to you

Also note that an OD API model can be fine tuned using any of the checkpoints from model zoo, does that work for you ?

I was not aware of tf.saved_model.save being deprecated ? Can you send me a pointer to that ?

Unfortunately an object detection models will always have to output multiple values, but we can make some changes to the exporter if it can help your use case.

Thanks
Vighnesh

@vighneshbirodkar thank you for your reply. I’ll look into this. I might have been confused about tf.savedmodel.save(). But I’ve run into errors messages complaining about a missingkeras_metadata.pb file, that apparently is only created by the new API?

Thruth be told, I just wanted to make a working TFX pipeline for object detection, but it’s proven to be much more difficult than I expected.

Could you share the following with us?

  1. The code you used for your TFX pipeline.
  2. The full error log of your program?

Thanks
Vighnesh

@vighneshbirodkar thank you for your help.
I’ve had to put this project on pause for now.

If you are interested, the code can be found here. Anders Smedegaard Pedersen / objdet_ml_pipeline · GitLab
But it shows that I’ve been trying a lot of different ways, and I’m not finished yet.

As Crowe hinted to earlier in this thread, it should be trivial to take one of the classifications examples from the Tensorflow website and convert it to an object detection pipeline. But I have to found that to be true. So any example of taking an “off-the-shelve” object det. model and using it in a TFX pipeline would greatly appreciated.

Hi Smedegaard,

Last year I have tried to do what you are doing. To build a TFX pipeline for object detection.
Thinking “how hard can it be”. It is very hard. Be aware that Object Detection can not be tested
in the same way as Classification. Classification is easy: Image in - label out.
Object detection output is much more complex: Image in - N x (label + coordinates) out.
So your model has a very different output signature than for classification problems.

You’ll have to change the way you interact with the tensorflow-serving instance.
You’ll probably want a tensorflow-serving instance with GPU support. Even firing that up
is quite a challenge. TFX does not support it out of the box.

To my knowledge there is not a single code example for what you are getting into.
I remember wasting a lot of time trying to make it work and in the end I failed to export
my model with the right signature to make it work with my tensorflow serving instance.