Android GPU delegate gives same output from intermediate layers outputand

General Discussion
, , , ,
#1

Hi Team

I have trained a mobilnetv2 model using the tfod API. Along with the normal detector output (bbox, score,num_detection, confidences), I also wanted an intermediate layers output, so while converting the model to tflite i specified the intemediate node from where i needed the output (‘FeatureExtractor/MobilenetV2/expanded_conv_9/add’) as illustrated below :

tflite_convert --graph_def_file=$1/tflite_graph/tflite_graph.pb --output_file=$1/tflite_graph/detect.tflite --input_shapes=1,300,300,3 --input_arrays=normalized_input_image_tensor --output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3','FeatureExtractor/MobilenetV2/expanded_conv_9/add' --inference_type=FLOAT --allow_custom_ops

When I run the tflite model on the android with the XNNPack or CPU, I am able to get the different output of the intermediate layer which is coming at index 4
code snippet :

try {
      Interpreter.Options options = new Interpreter.Options();
      options.setNumThreads(NUM_THREADS);
      options.setUseXNNPACK(true);
      d.tfLite = new Interpreter(modelFile, options);
      // 4th index for intermediate output
      int[] shape = d.tfLite.getOutputTensor(4).shape();
      d.tfLiteOptions = options;
    } catch (Exception e) {
      throw new RuntimeException(e);
    }

But when I run the same model on GPU delegate, I am getting the same output for every frame I am passing. The other output like (bbox, score,num_detection, confidences) changes but the intermediate output remains the same

try {
      Interpreter.Options options = new Interpreter.Options();
      CompatibilityList compatList = new CompatibilityList();
      GpuDelegate.Options delegateOptions = compatList.getBestOptionsForThisDevice();
      GpuDelegate gpuDelegate = new GpuDelegate(delegateOptions);
      options.addDelegate(gpuDelegate);

      d.tfLite = new Interpreter(modelFile, options);
      // 4th index for intermediate output
      int[] shape = d.tfLite.getOutputTensor(4).shape();
      d.tfLiteOptions = options;
    } catch (Exception e) {
      throw new RuntimeException(e);
    }

System information

  • Mobile device: OnePlus 6T, Mi Note5 pro
  • TFOD Api TensorFlow version : 1.15
  • Android Tensorflow Lite version: 2.4.0

Need help to figure out what may be going wrong
more info on this github issue link : Android GPU delegate gives same output from intermediate layers output · Issue #51800 · tensorflow/tensorflow · GitHub

#3

@letdive_deep Welcome to Tensorflow Forum!

Here’s a breakdown of possible causes and solutions for the above issue :-

1.The GPU delegate might cache intermediate results for performance, leading to identical outputs. Certain model structures might not be fully optimized for GPU acceleration, causing unexpected behavior.

Experiment with delegate options to potentially disable caching. Try Different GPU Delegate Versions:** Compatibility and optimization can vary across versions. Restructure Model (if possible):** Explore model restructuring to improve GPU compatibility.

2.The --allow_custom_ops flag might not fully convert necessary operations for GPU execution.

Verify Conversion:** Ensure all model operations are supported for GPU acceleration.

  1. Insufficient Memory:** Memory constraints could lead to incorrect results.

Monitor GPU Memory:** Check GPU memory usage during inference. Reduce Batch Size:** Lower input batch size to reduce memory footprint.

  1. Improper configuration might cause unexpected behavior.

Double-check TFLite interpreter configuration for GPU usage.

Let us know if you need more assistance !

Let us know!