Applying Grad-Cam to a model that was converted to Tensorflow Lite

General Discussion
, , ,
#1

I have a TensorFlow model converted to TensorFlow Lite, where my objective is to run it on Raspberry Pi. My model is a classification model that uses quantization aware training before conversion to the TFLite model. According to this example: Grad-CAM class activation visualization. I successfully applied grad-cam to an actual Tensorflow model. However, I did not find the methods to compute the gradients for Tensorflow Lite.

How can I replicate the TensorFlow grad-cam but for TensorFlow lite?

#3

Please export the generated TensorFlow model as a saved model and convert it through the TFLite converter API.

#5

Hi, my model is already in Tensorflow Lite format. Here’s the tflite file.

tflite_interpreter = tf.lite.Interpreter(model_path=model_path)
tflite_interpreter.allocate_tensors()

input_details = tflite_interpreter.get_input_details()
output_details = tflite_interpreter.get_output_details()

print("== Input details ==")
print(“name:”, input_details[0][‘name’])
print(“shape:”, input_details[0][‘shape’])
print(“type:”, input_details[0][‘dtype’])
print("\n== Output details ==")
print(“name:”, output_details[0][‘name’])
print(“shape:”, output_details[0][‘shape’])
print(“type:”, output_details[0][‘dtype’])

I am loading the details of my model, but cannot proceed with the required operations to execute grad-cam

#6

It is really hard to understand the problem. To get supports from this thread, please consider adding more relevant information and uploading reproducible steps as a gist.

#7

I would like to know which operations I need to perform on a TF Lite model to achieve the class activation maps for a given image during classification. I have a separate classify.py file that loads my functions for TF Lite model inference.

import collections
import operator
import numpy as np

Class = collections.namedtuple('Class', ['id', 'score'])


def input_details(interpreter, key):
  return interpreter.get_input_details()[0][key]


def input_size(interpreter):
  _, height, width, _ = input_details(interpreter, 'shape')
  return width, height


def input_tensor(interpreter):
  tensor_index = input_details(interpreter, 'index')
  return interpreter.tensor(tensor_index)()[0]


def output_tensor(interpreter, dequantize=True):
  output_details = interpreter.get_output_details()[0]
  output_data = np.squeeze(interpreter.tensor(output_details['index'])())

  if dequantize and np.issubdtype(output_details['dtype'], np.integer):
    scale, zero_point = output_details['quantization']
    return scale * (output_data - zero_point)

  return output_data

def set_input(interpreter, data):
  input_tensor(interpreter)[:, :] = data

def get_output(interpreter, top_k=1, score_threshold=0.0):
  scores = output_tensor(interpreter)
  classes = [
      Class(i, scores[i])
      for i in np.argpartition(scores, -top_k)[-top_k:]
      if scores[i] >= score_threshold
  ]
  return sorted(classes, key=operator.itemgetter(1), reverse=True)

Then, my image_classify.py performs the model inference for the given image as input to the tensor.

import argparse
import time
from PIL import Image
import classify
import tflite_runtime.interpreter as tflite
import platform

EDGETPU_SHARED_LIB = {
‘Linux’: ‘libedgetpu.so.1’,
‘Darwin’: ‘libedgetpu.1.dylib’,
‘Windows’: ‘edgetpu.dll’
}[platform.system()]

def load_labels(path, encoding=‘utf-8’):
with open(path, ‘r’, encoding=encoding) as f:
lines = f.readlines()
if not lines:
return {}

if lines[0].split(' ', maxsplit=1)[0].isdigit():
  pairs = [line.split(' ', maxsplit=1) for line in lines]
  return {int(index): label.strip() for index, label in pairs}
else:
  return {index: line.strip() for index, line in enumerate(lines)}

def make_interpreter(model_file):
model_file, *device = model_file.split(’@’)
return tflite.Interpreter(
model_path=model_file,
experimental_delegates=[
tflite.load_delegate(EDGETPU_SHARED_LIB,
{‘device’: device[0]} if device else {})
])

def main():
parser = argparse.ArgumentParser(
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument(
‘-m’, ‘–model’, required=True, help=‘File path of .tflite file.’)
parser.add_argument(
‘-i’, ‘–input’, required=True, help=‘Image to be classified.’)
parser.add_argument(
‘-l’, ‘–labels’, help=‘File path of labels file.’)
parser.add_argument(
‘-k’, ‘–top_k’, type=int, default=1,
help=‘Max number of classification results’)
parser.add_argument(
‘-t’, ‘–threshold’, type=float, default=0.0,
help=‘Classification score threshold’)
parser.add_argument(
‘-c’, ‘–count’, type=int, default=5,
help=‘Number of times to run inference’)
args = parser.parse_args()

labels = load_labels(args.labels) if args.labels else {}

interpreter = make_interpreter(args.model)
interpreter.allocate_tensors()

size = classify.input_size(interpreter)
image = Image.open(args.input).convert(‘RGB’).resize(size, Image.ANTIALIAS)
classify.set_input(interpreter, image)

for _ in range(args.count):
start = time.perf_counter()
interpreter.invoke()
inference_time = time.perf_counter() - start
classes = classify.get_output(interpreter, args.top_k, args.threshold)
print(’%.1fms’ % (inference_time * 1000))

for klass in classes:
print(’%s: %.5f’ % (labels.get(klass.id, klass.id), klass.score))

What I would like to do is to perform the same operations such as retrieving the layers, computing the gradients, performing the channel multiplications that are usually done for a standard tensorflow model, but instead, for a tensorflow lite model and show the grad-cam heat map for the image input

#8

As I mentioned in a comment on the SO thread, computing gradients with TFLite models isn’t possible currently. The best you can do is to compute the gradients of the Keras model / SavedModel and produce the CAMs.

#9

If you are interested in this you could take a look at what’s cooking in:

#11

You can also refer to the following colab for getting the gradients:

#12

Can we access to the gradients with this?

#13

The gradients won’t be computed unless the model exposes the gradient computation tasks as a model structure. The above example shows how to include the gradient computations into a model signature and compute the gradients on device.

#14

Ok, so you can get it back on the device with:

train = interpreter.get_signature_runner("train")

#15

Noted that as described at the colab, the original TF model should export a signature for the gradient computations.

#16

So, I need to retrain the model on my Raspberry Pi and use the “train = interpreter.get_signature_runner(“train”)” to get the gradients??

#17

It is not a simple task. You need to understand how the above colab works first especially for the train signature.

Train signature is not automatically an enabled feature and is explicitly implemented in the original TensorFlow model as a signature of the TensorFlow model in the colab.

It requires to write down a Python code to create a grad. computations like the below example for your model:

  @tf.function(input_signature=[
      tf.TensorSpec([None, IMG_SIZE, IMG_SIZE], tf.float32),
      tf.TensorSpec([None, 10], tf.float32),
  ])
  def train(self, x, y):
    with tf.GradientTape() as tape:
      prediction = self.model(x)
      loss = self._LOSS_FN(prediction, y)
    gradients = tape.gradient(loss, self.model.trainable_variables)
    self._OPTIM.apply_gradients(
        zip(gradients, self.model.trainable_variables))
    result = {"loss": loss}
    for grad in gradients:
      result[grad.name] = grad
    return result
#18

Yes as manipulating these objects on the device directly It is part of the mentioned RFC.

#20

Hi @puelon . Have you been able to solve your problem? I am also trying to get the gradients of the loss function wrt input image in a TFLite model.