Model zoo TF1 frozen graph conversion fails or creates a .tflite file that cannot be used inside android

General Discussion
, ,
#1

Hi community,

I have an issue with tf1 object detection models at model zoo. Especially I want to convert faster_rcnn_inception_v2_coco. Inside the folder if you download it there is a saved_model and a frozen_inference_graph.pb

  1. With the saved model I use a concrete function to convert it:
model = tf.saved_model.load('/content/faster_rcnn_inception_v2_coco_2018_01_28/saved_model')
concrete_func = model.signatures[
tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY]
concrete_func.inputs[0].set_shape([1, 300, 300, 3])

converter = tf.lite.TFLiteConverter.from_concrete_functions([concrete_func])
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]

tflite_model = converter.convert()

with open('detect2.tflite', 'wb') as f:
  f.write(tflite_model)

the .tflite file is generated without errors but it seems that it throws an error inside android:
java.lang.RuntimeException: java.lang.IllegalArgumentException: Internal error: Cannot create interpreter: Did not get operators or tensors in subgraph 1.

  1. With the frozen graph the conversion procedure fails:
converter = tf.compat.v1.lite.TFLiteConverter.from_frozen_graph(
    graph_def_file = '/content/faster_rcnn_inception_v2_coco_2018_01_28/frozen_inference_graph.pb', 
    input_arrays = ['image_tensor'],
    output_arrays = ['detection_boxes', 'detection_classes', 'detection_scores', 'num_detections']
)

converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]

tflite_model = converter.convert()

with open('detect.tflite', 'wb') as f:
  f.write(tflite_model)

ConverterError: Merge of two inputs that differ on more than one predicate {s(SecondStagePostprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/Greater:0,else), s(SecondStagePostprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/cond/cond/pred_id/_390__cf__396:0,then), s(SecondStagePostprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/cond/Greater/_389__cf__395:0,then)} and {s(SecondStagePostprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/Greater:0,else), s(SecondStagePostprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/cond/cond/pred_id/_390__cf__396:0,else), s(SecondStagePostprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/cond/Greater/_389__cf__395:0,else)} for node {{node SecondStagePostprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/cond/cond/Merge}} Failed to functionalize Control Flow V1 ops. Consider using Control Flow V2 ops instead.

I use colab and TensorFlow version is 2.7.0

Does anyone know how to overcome the above error or a different procedure for successful conversion of the tf1 files of model zoo?

Thanks in advance.

#3

UPDATE

I have managed to overcome 2nd error: Failed to functionalize Control Flow V1 ops. Consider using Control Flow V2 ops instead …by using enable_control_flow_V2 during loading the checkpoint files in TensorFlow version 1.15 and converting to frozen_graph.pb as below:

import os, argparse

# The original freeze_graph function
# from tensorflow.python.tools.freeze_graph import freeze_graph 

# dir = os.path.dirname(os.path.realpath(__file__))

def freeze_graph(model_dir, output_node_names):
    """Extract the sub graph defined by the output nodes and convert 
    all its variables into constant 
    Args:
        model_dir: the root folder containing the checkpoint state file
        output_node_names: a string, containing all the output node's names, 
                            comma separated
    """
    if not tf.gfile.Exists(model_dir):
        raise AssertionError(
            "Export directory doesn't exists. Please specify an export "
            "directory: %s" % model_dir)

    if not output_node_names:
        print("You need to supply the name of a node to --output_node_names.")
        return -1

    # We retrieve our checkpoint fullpath
    checkpoint = tf.train.get_checkpoint_state(model_dir)
    input_checkpoint = checkpoint.model_checkpoint_path
    
    # We precise the file fullname of our freezed graph
    absolute_model_dir = "/".join(input_checkpoint.split('/')[:-1])
    output_graph = absolute_model_dir + "/frozen_model_V2.pb"

    # We clear devices to allow TensorFlow to control on which device it will load operations
    clear_devices = True

    # We start a session using a temporary fresh Graph
    with tf.Session(graph=tf.Graph()) as sess:
        # We import the meta graph in the current default Graph
        saver = tf.train.import_meta_graph(input_checkpoint + '.meta', clear_devices=clear_devices)

        # We restore the weights
        saver.restore(sess, input_checkpoint)

        tf.enable_control_flow_v2()

        # We use a built-in TF helper to export variables to constants
        output_graph_def = tf.graph_util.convert_variables_to_constants(
            sess, # The session is used to retrieve the weights
            tf.get_default_graph().as_graph_def(), # The graph_def is used to retrieve the nodes 
            output_node_names.split(",") # The output node names are used to select the usefull nodes
        ) 

        # Finally we serialize and dump the output graph to the filesystem
        with tf.gfile.GFile(output_graph, "wb") as f:
            f.write(output_graph_def.SerializeToString())
        print("%d ops in the final graph." % len(output_graph_def.node))

    return output_graph_def



freeze_graph('/content/faster_rcnn_inception_v2_coco_2018_01_28', 'detection_boxes')

Then switching to TF version 2.7.0 and converting to .tflite:

converter = tf.compat.v1.lite.TFLiteConverter.from_frozen_graph(
    graph_def_file='/content/faster_rcnn_inception_v2_coco_2018_01_28/frozen_model_V2.pb',
    input_shapes = {'image_tensor':[1,300,300,3]},
    input_arrays = ['image_tensor'],
    output_arrays = ['detection_boxes']
)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]
tflite_model = converter.convert()

with open('detectV2.tflite', 'wb') as f:
  f.write(tflite_model)

BUT the first error still persists…Inside android application you get:
java.lang.RuntimeException: java.lang.IllegalArgumentException: Internal error: Cannot create interpreter: Did not get operators or tensors in subgraph 1.

Any thoughts?

#4

UPDATE 2

The error java.lang.RuntimeException: java.lang.IllegalArgumentException: Internal error: Cannot create interpreter: Did not get operators or tensors in subgraph 1. it is due to:

converter.optimizations = [tf.lite.Optimize.DEFAULT]

so by commenting this out the above error dissapears inside android. Tagging @khanhlvg to add people to check this error.

#6

There are several issues with the approach you took:

  • TensorFlow Object Detection API models need to be preprocessed before you can convert them to TFLite. You need to follow the steps listed here before the conversion. @Sayak_Paul also wrote a blog post about it.
  • TFLite only supports SSD models, so you won’t be able to convert the Faster RCNN model to TFLite even if you follow the steps above.

So if you want to do object detection with TFLite, my recommendation would be using EfficientDet-Lite via Model Maker, or use the MobileDets models in TF Object Detection API if you want to customize the model architecture.

2 Likes