TCAV use in other applications beside image classification and text

General Discussion
, ,
#1

Hi, I’m trying to use TCAV for Object detection and semantic segmentation models, but I am having a hard time figuring out how to do it. For me is especially hard on the Object detection models that TensorFlow Object Detection API provides. The SSDMetaArch architecture is a weird combination of Keras and Graphs. Hence I’m totally lost. If anyone has a clue what the right approach to this problem is, will be awesome :smile:. I really appreciate any help you can provide.

1 Like
#2

Hi, to update my recent findings and issues.

Findings:

  • tracing back to the objectdetection class, I found out how to get the prediction layer of the SSD.

  • Also, I successfully recreate the loss function for this layer that works in the same fashion as KerasWrapper
    Issues:

  • I copy the KerasWrapper class to build a new KerasWrapperSSD (the SSD layer are Keras layers).

  • This class is also on model.py of the TCAV repository

  • I load the model as I did to found the model’s layers, no error in that, except that the model seems that is no loading the backbone or any other layer. Hence I cannot extract the bottlenecks.

  def import_keras_model(self, saved_path):
    """Loads keras model, fetching bottlenecks, inputs and outputs."""
    self.ends = {}
    PATH_TO_CFG=saved_path + "/pipeline.config"
    PATH_TO_CKPT = saved_path + "/checkpoint"
    configs = config_util.get_configs_from_pipeline_file(PATH_TO_CFG)
    model_config = configs['model']
    detection_model = model_builder.build(model_config=model_config, is_training=False)
    ckpt = tf.compat.v2.train.Checkpoint(model=detection_model)
    ckpt.restore(os.path.join(PATH_TO_CKPT, 'ckpt-0')).expect_partial()
    self.model = detection_model
    print(self.model)
    print(self.model.feature_extractor.classification_backbone)
    self.get_bottleneck_tensors()
    self.get_inputs_and_outputs_and_ends()
    self.get_image_shape()

If I the load model in this fashion on other script works, but as a class of TCAV, the model is just a simple shell
I don’t know what is missing or what I’m doing wrong. I really, really appreciate any help.
Bests

1 Like
#3

Hello,
I am facing with the same issue trying to load Object Detection model for tcav. Did you by any chance find solution to this issue ?