Exporter_main_v2.py 'skipping full serialization of Keras layer' error

ahmadchalhoub · July 2, 2021, 5:53pm

Hello,

Whenever I try to convert a trained checkpoint to a SavedModel using ‘exporter_main_v2.py’, I get a warning that says WARNING:tensorflow:Skipping full serialization of Keras layer

This is causing trouble for later operations that I want to perform with the trained model.

Any thoughts on why this might be happening? Is there any certain arguments that I need to pass to this function for it to convert properly?

Thanks.

markdaoust · July 6, 2021, 2:17pm

It’s impossible to answer this without more details.

Can you figure out which layer it’s complaining about? It might be a custom layer missing the .get_config/.from_config methods.

If your model can’t be loaded as a keras model (tf.keras.models.load_model) it can probably still be loaded as a saved_model with tf.saved_model.load. That doesn’t give you access to the layers, but it will let you use the reloaded model, if that’s what you’re trying to do.

ahmadchalhoub · July 6, 2021, 2:30pm

Hello,

I am trying to export the trained checkpoint of the SSD MobilenetV2 320x320 model found on TF2 Zoo ‘http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v2_320x320_coco17_tpu-8.tar.gz’.

The command I am using is the following:
python ~/models/research/object_detection/exporter_main_v2.py --input_type image_tensor --pipeline_config_path pipeline.config --trained_checkpoint_dir ./checkpoint/ --output_directory exported_model/

The following attached screenshot shows the warning message that I get:

I am using tf-nightly.

Bhack · July 6, 2021, 4:46pm

github.com/tensorflow/models

ssd_mobilenet_v2_320x320_coco17_tpu-8 for object detection on GPU using Tensorflow 2 "INFO:tensorflow:depth of additional conv before box predictor: 0" and "WARNING:tensorflow:Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7f51187a02b0>, because it is not built. W0423 10:47:57.954887 139987999496000 save_impl.py:77] Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7f51187a02b0>, because it is not built."

opened 04:53AM - 23 Apr 21 UTC

thongvhoang

type:bug models:research:odapi

# Prerequisites - [x] I am using the ssd_mobilenet_v2_320x320_coco17_tpu-8 an…d TensorFlow 2: 2.3.0 - [x] I am reporting the issue to the correct repository. (research directory) https://github.com/tensorflow/models/blob/master/research/object_detection/configs/tf2/ssd_mobilenet_v2_320x320_coco17_tpu-8.config - [x] I checked to make sure that this issue has not already been filed. ## 1. The entire URL of the file you are using https://github.com/tensorflow/models/tree/master/research/object_detection ## 2. Describe the bug Was the error caused by INFO:tensorflow:depth of additional conv before box predictor: 0 and the model after being exported does not use the latest checkpoint (checkpoint3)? **Error messages:** 2021-04-23 11:14:58.273174: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2021-04-23 11:14:59.464301: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1 2021-04-23 11:14:59.486935: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n ode zero 2021-04-23 11:14:59.487311: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1 coreClock: 1.7085GHz coreCount: 15 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 238.66GiB/s 2021-04-23 11:14:59.487332: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2021-04-23 11:14:59.488443: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10 2021-04-23 11:14:59.489511: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10 2021-04-23 11:14:59.489687: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10 2021-04-23 11:14:59.490804: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10 2021-04-23 11:14:59.491509: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10 2021-04-23 11:14:59.494130: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7 2021-04-23 11:14:59.494262: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returni[38/1972] ode zero 2021-04-23 11:14:59.494699: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n ode zero 2021-04-23 11:14:59.495014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0 2021-04-23 11:14:59.495262: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in perfo rmance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-04-23 11:14:59.499752: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3600000000 Hz 2021-04-23 11:14:59.500040: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x54c78b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-04-23 11:14:59.500051: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-04-23 11:14:59.561614: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n ode zero 2021-04-23 11:14:59.561986: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x555b8e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2021-04-23 11:14:59.562000: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1070, Compute Capability 6.1 2021-04-23 11:14:59.562205: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n ode zero 2021-04-23 11:14:59.562522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1 coreClock: 1.7085GHz coreCount: 15 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 238.66GiB/s 2021-04-23 11:14:59.562539: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 [18/1972] 2021-04-23 11:14:59.562557: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10 2021-04-23 11:14:59.562583: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10 2021-04-23 11:14:59.562593: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10 2021-04-23 11:14:59.562615: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10 2021-04-23 11:14:59.562639: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10 2021-04-23 11:14:59.562650: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7 2021-04-23 11:14:59.562718: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n ode zero 2021-04-23 11:14:59.563031: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n ode zero 2021-04-23 11:14:59.563370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0 2021-04-23 11:14:59.563389: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2021-04-23 11:14:59.921326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-04-23 11:14:59.921355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0 2021-04-23 11:14:59.921361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N 2021-04-23 11:14:59.921533: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n ode zero 2021-04-23 11:14:59.921864: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n ode zero 2021-04-23 11:14:59.922150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7014 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1) WARNING:tensorflow:From /home/nguyen_truong/.local/lib/python3.8/site-packages/object_detection/exporter_lib_v2.py:106: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with back_prop=False is deprecated an d will be removed in a future version. Instructions for updating: back_prop=False is deprecated. Consider using tf.stop_gradient instead. Instead of: results = tf.map_fn(fn, elems, back_prop=False) Use: results = tf.nest.map_structure(tf.stop_gradient, tf.map_fn(fn, elems)) W0423 11:15:00.141795 140359799744320 deprecation.py:568] From /home/nguyen_truong/.local/lib/python3.8/site-packages/object_detection/exporter_lib_v2.py:106: calling map_fn_v2 (from tensorflow.python.ops.map_fn ) with back_prop=False is deprecated and will be removed in a future version. Instructions for updating: back_prop=False is deprecated. Consider using tf.stop_gradient instead. Instead of: results = tf.map_fn(fn, elems, back_prop=False) Use: results = tf.nest.map_structure(tf.stop_gradient, tf.map_fn(fn, elems)) INFO:tensorflow:depth of additional conv before box predictor: 0 [1/1993] I0423 11:15:03.574930 140359799744320 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0423 11:15:03.575145 140359799744320 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0423 11:15:03.575302 140359799744320 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0423 11:15:03.575448 140359799744320 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0423 11:15:03.575573 140359799744320 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0423 11:15:03.575695 140359799744320 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0 WARNING:tensorflow:Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7fa7b84d5580>, because it is not built. W0423 11:15:08.253181 140359799744320 save_impl.py:77] Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7fa7b84d5580>, because it is not built . 2021-04-23 11:15:15.046676: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them. INFO:tensorflow:Assets written to: inference_graph/saved_model/assets I0423 11:15:26.415863 140359799744320 builder_impl.py:774] Assets written to: inference_graph/saved_model/assets INFO:tensorflow:Writing pipeline config file to inference_graph/pipeline.config I0423 11:15:26.901574 140359799744320 config_util.py:253] Writing pipeline config file to inference_graph/pipeline.config ## 3. Steps to reproduce Command 1: python model_main_tf2.py --model_dir=ssd_mobilenet_v2_320x320_coco17_tpu-8 --pipeline_config_path=ssd_mobilenet_v2_320x320_coco17_tpu-8/pipeline.config Command 2: python models/research/object_detection/exporter_main_v2.py --trained_checkpoint_dir ssd_mobilenet_v2_320x320_coco17_tpu-8 --output_directory inference_graph --pipeline_config_path ssd_mobilenet_v2_320x320_coco17_tpu-8/ssd_mobilenet_v2_320x320_coco17_tpu-8.config ## 4. Expected behavior Folder inference_graph contained model exported lasted with checkpoint3 and detected images clearly. But the bug exported the model using checkpoint1 has not updated so can not detect images with bounding box. ## 5. Additional context My ssd mobilenet v2 configure: model { ssd { num_classes: 1 image_resizer { fixed_shape_resizer { height: 300 width: 300 } } feature_extractor { type: "ssd_mobilenet_v2_keras" depth_multiplier: 1.0 min_depth: 16 conv_hyperparams { regularizer { l2_regularizer { weight: 3.9999998989515007e-05 } } initializer { truncated_normal_initializer { mean: 0.0 stddev: 0.029999999329447746 } } activation: RELU_6 batch_norm { decay: 0.9700000286102295 center: true scale: true epsilon: 0.0010000000474974513 train: true } } override_base_feature_extractor_hyperparams: true } box_coder { faster_rcnn_box_coder { y_scale: 10.0 x_scale: 10.0 height_scale: 5.0 width_scale: 5.0 } } matcher { argmax_matcher { matched_threshold: 0.5 unmatched_threshold: 0.5 ignore_thresholds: false negatives_lower_than_unmatched: true force_match_for_each_row: true use_matmul_gather: true } } similarity_calculator { iou_similarity { } } box_predictor { convolutional_box_predictor { conv_hyperparams { regularizer { l2_regularizer { weight: 3.9999998989515007e-05 } } initializer { random_normal_initializer { mean: 0.0 stddev: 0.009999999776482582 } } activation: RELU_6 batch_norm { decay: 0.9700000286102295 center: true scale: true epsilon: 0.0010000000474974513 train: true } } min_depth: 0 max_depth: 0 num_layers_before_predictor: 0 use_dropout: false dropout_keep_probability: 0.800000011920929 kernel_size: 1 box_code_size: 4 apply_sigmoid_to_scores: false class_prediction_bias_init: -4.599999904632568 } } anchor_generator { ssd_anchor_generator { num_layers: 6 min_scale: 0.20000000298023224 max_scale: 0.949999988079071 aspect_ratios: 1.0 aspect_ratios: 2.0 aspect_ratios: 0.5 aspect_ratios: 3.0 aspect_ratios: 0.33329999446868896 } } post_processing { batch_non_max_suppression { score_threshold: 9.99999993922529e-09 iou_threshold: 0.6000000238418579 max_detections_per_class: 100 max_total_detections: 100 use_static_shapes: false } score_converter: SIGMOID } normalize_loss_by_num_matches: true loss { localization_loss { weighted_smooth_l1 { delta: 1.0 } } classification_loss { weighted_sigmoid_focal { gamma: 2.0 alpha: 0.75 } } classification_weight: 1.0 localization_weight: 1.0 } encode_background_as_zeros: true normalize_loc_loss_by_codesize: true inplace_batchnorm_update: true freeze_batchnorm: false } } train_config { batch_size: 12 data_augmentation_options { random_horizontal_flip { } } data_augmentation_options { ssd_random_crop { } } sync_replicas: true optimizer { momentum_optimizer { learning_rate { cosine_decay_learning_rate { learning_rate_base: 0.03999999910593033 total_steps: 2000 warmup_learning_rate: 0.013333000242710114 warmup_steps: 500 } } momentum_optimizer_value: 0.8999999761581421 } use_moving_average: false } fine_tune_checkpoint: "ssd_mobilenet_v2_320x320_coco17_tpu-8/checkpoint0/ckpt-0" num_steps: 2000 startup_delay_steps: 0.0 replicas_to_aggregate: 8 max_number_of_boxes: 100 unpad_groundtruth_tensors: false fine_tune_checkpoint_type: "detection" fine_tune_checkpoint_version: V2 } train_input_reader { label_map_path: "data/object-detection.pbxt" tf_record_input_reader { input_path: "data/train.record" } } eval_config { metrics_set: "coco_detection_metrics" use_moving_averages: false } eval_input_reader { label_map_path: "data/object-detection.pbxt" shuffle: false num_epochs: 1 tf_record_input_reader { input_path: "data/test.record" } } ## 6. System information - OS Platform and Distribution: Linux Ubuntu 20.04 - TensorFlow installed from (source or binary): pip install - TensorFlow version (use command below): 2.3.0 - Python version: 3.8.5 - CUDA/cuDNN version: Cuda release 10.1, V10.1.243 - GPU model and memory: GeForce GTX 1070 TensorFlow 2.0 `python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"` --> 2.3.0

ahmadchalhoub · July 6, 2021, 4:54pm

Thank you for the reference. I am trying to find solutions through the link that you have sent but there seems to be no responses yet. Am I missing anything?

Bhack · July 6, 2021, 5:33pm

It is just a reference to the same topic on the github repository.

Have you tried with something like:

github.com/tensorflow/models

Type Error when exporting TF2 OD API to SavedModel using efficientdet_d0

opened 05:57PM - 11 Jul 20 UTC

m-brgs

type:bug models:research:odapi

# Prerequisites Please answer the following questions for yourself before sub…mitting an issue. - [x] I am using the latest TensorFlow Model Garden release and TensorFlow 2. - [x] I am reporting the issue to the correct repository. (Model Garden official or research directory) - [x] I checked to make sure that this issue has not already been filed. ## 1. The entire URL of the file you are using https://github.com/tensorflow/models/blob/master/research/object_detection/exporter_main_v2.py ## 2. Describe the bug I'm trying to export a TF2 OD API efficientdet_d0 that I just trained (using pretrained weights from model zoo 2 and 2GPUs in the tf2 docker image from OD API repo) with exporter_main_v2.py in order to serve this model with Tensorflow Serving. When exporting, I'm getting a type error. ## 3. Steps to reproduce Take the result of training a ssd_efficientdet_d0_512x512 model. Inside TF2 container, (https://github.com/tensorflow/models/blob/master/research/object_detection/dockerfiles/tf2/Dockerfile), run : ```sh python object_detection/exporter_main_v2.py \ --input_type encoded_image_string_tensor \ --trained_checkpoint_dir /home/tensorflow/outputs/efficientdet_d0_10k_16b_512_2gpu/model_dir/ \ --output_directory /home/tensorflow/outputs/efficientdet_d0_10k_16b_512_2gpu/export/ \ --pipeline_config_path /home/tensorflow/pipeline.config ``` ## 4. Expected behavior The model is exported in /home/tensorflow/outputs/efficientdet_d0_10k_16b_512_2gpu/export/ as described in https://github.com/tensorflow/models/blob/master/research/object_detection/exporter_main_v2.py ## 5. Additional context Logs : ``` I0711 17:37:09.737181 139828628232000 ssd_efficientnet_bifpn_feature_extractor.py:144] EfficientDet EfficientNet backbone version: efficientnet-b0 I0711 17:37:09.737298 139828628232000 ssd_efficientnet_bifpn_feature_extractor.py:145] EfficientDet BiFPN num filters: 64 I0711 17:37:09.737352 139828628232000 ssd_efficientnet_bifpn_feature_extractor.py:147] EfficientDet BiFPN num iterations: 3 I0711 17:37:09.745369 139828628232000 efficientnet_model.py:146] round_filter input=32 output=32 I0711 17:37:10.747392 139828628232000 efficientnet_model.py:146] round_filter input=32 output=32 I0711 17:37:10.747502 139828628232000 efficientnet_model.py:146] round_filter input=16 output=16 I0711 17:37:10.829591 139828628232000 efficientnet_model.py:146] round_filter input=16 output=16 I0711 17:37:10.829701 139828628232000 efficientnet_model.py:146] round_filter input=24 output=24 I0711 17:37:11.092894 139828628232000 efficientnet_model.py:146] round_filter input=24 output=24 I0711 17:37:11.093004 139828628232000 efficientnet_model.py:146] round_filter input=40 output=40 I0711 17:37:11.314886 139828628232000 efficientnet_model.py:146] round_filter input=40 output=40 I0711 17:37:11.314993 139828628232000 efficientnet_model.py:146] round_filter input=80 output=80 I0711 17:37:11.654193 139828628232000 efficientnet_model.py:146] round_filter input=80 output=80 I0711 17:37:11.654304 139828628232000 efficientnet_model.py:146] round_filter input=112 output=112 I0711 17:37:11.993307 139828628232000 efficientnet_model.py:146] round_filter input=112 output=112 I0711 17:37:11.993419 139828628232000 efficientnet_model.py:146] round_filter input=192 output=192 I0711 17:37:12.449521 139828628232000 efficientnet_model.py:146] round_filter input=192 output=192 I0711 17:37:12.449632 139828628232000 efficientnet_model.py:146] round_filter input=320 output=320 I0711 17:37:12.555558 139828628232000 efficientnet_model.py:146] round_filter input=1280 output=1280 I0711 17:37:12.599392 139828628232000 efficientnet_model.py:459] Building model efficientnet with params ModelConfig(width_coefficient=1.0, depth_coefficient=1.0, resolution=224, dropout_rate=0.2, blocks=(BlockConfig(input_filters=32, output_filters=16, kernel_size=3, num_repeat=1, expand_ratio=1, strides=(1, 1), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=16, output_filters=24, kernel_size=3, num_repeat=2, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=24, output_filters=40, kernel_size=5, num_repeat=2, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=40, output_filters=80, kernel_size=3, num_repeat=3, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=80, output_filters=112, kernel_size=5, num_repeat=3, expand_ratio=6, strides=(1, 1), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=112, output_filters=192, kernel_size=5, num_repeat=4, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=192, output_filters=320, kernel_size=3, num_repeat=1, expand_ratio=6, strides=(1, 1), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise')), stem_base_filters=32, top_base_filters=1280, activation='simple_swish', batch_norm='default', bn_momentum=0.99, bn_epsilon=0.001, weight_decay=5e-06, drop_connect_rate=0.2, depth_divisor=8, min_depth=None, use_se=True, input_channels=3, num_classes=1000, model_name='efficientnet', rescale_input=False, data_format='channels_last', dtype='float32') WARNING:tensorflow:From /home/tensorflow/.local/lib/python3.6/site-packages/object_detection/exporter_lib_v2.py:101: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with back_prop=False is deprecated and will be removed in a future version. Instructions for updating: back_prop=False is deprecated. Consider using tf.stop_gradient instead. Instead of: results = tf.map_fn(fn, elems, back_prop=False) Use: results = tf.nest.map_structure(tf.stop_gradient, tf.map_fn(fn, elems)) W0711 17:37:16.879535 139828628232000 deprecation.py:573] From /home/tensorflow/.local/lib/python3.6/site-packages/object_detection/exporter_lib_v2.py:101: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with back_prop=False is deprecated and will be removed in a future version. Instructions for updating: back_prop=False is deprecated. Consider using tf.stop_gradient instead. Instead of: results = tf.map_fn(fn, elems, back_prop=False) Use: results = tf.nest.map_structure(tf.stop_gradient, tf.map_fn(fn, elems)) WARNING:tensorflow:Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7f2b7c1ac5f8>, because it is not built. W0711 17:37:39.926661 139828628232000 save_impl.py:76] Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7f2b7c1ac5f8>, because it is not built. Traceback (most recent call last): File "object_detection/exporter_main_v2.py", line 126, in <module> app.run(main) File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run _run_main(main, args) File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main sys.exit(main(argv)) File "object_detection/exporter_main_v2.py", line 122, in main FLAGS.output_directory) File "/home/tensorflow/.local/lib/python3.6/site-packages/object_detection/exporter_lib_v2.py", line 180, in export_inference_graph signatures=concrete_function) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 951, in save obj, export_dir, signatures, options, meta_graph_def) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 1022, in _build_meta_graph _ = _SaveableView(checkpoint_graph_view) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 173, in __init__ self.checkpoint_view.objects_ids_and_slot_variables()) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/training/tracking/graph_view.py", line 415, in objects_ids_and_slot_variables trackable_objects, path_to_root = self._breadth_first_traversal() File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/training/tracking/graph_view.py", line 199, in _breadth_first_traversal for name, dependency in self.list_dependencies(current_trackable): File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 108, in list_dependencies extra_dependencies = self.list_extra_dependencies(obj) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 137, in list_extra_dependencies self._serialization_cache) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 2746, in _list_extra_dependencies_for_serialization .list_extra_dependencies_for_serialization(serialization_cache)) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/base_serialization.py", line 74, in list_extra_dependencies_for_serialization return self.objects_to_serialize(serialization_cache) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py", line 73, in objects_to_serialize serialization_cache).objects_to_serialize) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py", line 92, in _get_serialized_attributes serialization_cache) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/model_serialization.py", line 53, in _get_serialized_attributes_internal serialization_cache)) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py", line 101, in _get_serialized_attributes_internal functions = save_impl.wrap_layer_functions(self.obj, serialization_cache) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 163, in wrap_layer_functions '{}_layer_call_and_return_conditional_losses'.format(layer.name)) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 503, in add_function self.add_trace(*self._input_signature) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 418, in add_trace trace_with_training(True) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 416, in trace_with_training fn.get_concrete_function(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 547, in get_concrete_function return super(LayerCall, self).get_concrete_function(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 959, in get_concrete_function concrete = self._get_concrete_function_garbage_collected(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 865, in _get_concrete_function_garbage_collected self._initialize(args, kwargs, add_initializers_to=initializers) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 506, in _initialize *args, **kwds)) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2446, in _get_concrete_function_internal_garbage_collected graph_function, _, _ = self._maybe_define_function(args, kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2777, in _maybe_define_function graph_function = self._create_graph_function(args, kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2667, in _create_graph_function capture_by_value=self._capture_by_value), File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 981, in func_graph_from_py_func func_outputs = python_func(*func_args, **func_kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 441, in wrapped_fn return weak_wrapped_fn().__wrapped__(*args, **kwds) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 524, in wrapper ret = method(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 170, in wrap_with_training_arg lambda: replace_training_and_call(False)) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 65, in smart_cond pred, true_fn=true_fn, false_fn=false_fn, name=name) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/smart_cond.py", line 54, in smart_cond return true_fn() File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 169, in <lambda> lambda: replace_training_and_call(True), File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 165, in replace_training_and_call return wrapped_call(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 566, in call_and_return_conditional_losses return layer_call(inputs, *args, **kwargs), layer.get_losses_for(inputs) File "/home/tensorflow/.local/lib/python3.6/site-packages/object_detection/meta_architectures/ssd_meta_arch.py", line 251, in call return self._extract_features(inputs) File "/home/tensorflow/.local/lib/python3.6/site-packages/object_detection/models/ssd_efficientnet_bifpn_feature_extractor.py", line 224, in _extract_features list(zip(self._output_layer_alias, base_feature_maps))) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 968, in __call__ outputs = self.call(cast_inputs, *args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 71, in return_outputs_and_add_losses outputs, losses = fn(inputs, *args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 170, in wrap_with_training_arg lambda: replace_training_and_call(False)) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 65, in smart_cond pred, true_fn=true_fn, false_fn=false_fn, name=name) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/smart_cond.py", line 54, in smart_cond return true_fn() File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 169, in <lambda> lambda: replace_training_and_call(True), File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 165, in replace_training_and_call return wrapped_call(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 541, in __call__ self.call_collection.add_trace(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 418, in add_trace trace_with_training(True) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 416, in trace_with_training fn.get_concrete_function(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 547, in get_concrete_function return super(LayerCall, self).get_concrete_function(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 959, in get_concrete_function concrete = self._get_concrete_function_garbage_collected(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 865, in _get_concrete_function_garbage_collected self._initialize(args, kwargs, add_initializers_to=initializers) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 506, in _initialize *args, **kwds)) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2446, in _get_concrete_function_internal_garbage_collected graph_function, _, _ = self._maybe_define_function(args, kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2777, in _maybe_define_function graph_function = self._create_graph_function(args, kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2667, in _create_graph_function capture_by_value=self._capture_by_value), File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 981, in func_graph_from_py_func func_outputs = python_func(*func_args, **func_kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 441, in wrapped_fn return weak_wrapped_fn().__wrapped__(*args, **kwds) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 524, in wrapper ret = method(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 484, in wrap_with_training_arg return call_fn(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 566, in call_and_return_conditional_losses return layer_call(inputs, *args, **kwargs), layer.get_losses_for(inputs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1627, in get_losses_for reachable = tf_utils.get_reachable_from_inputs(inputs, losses) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 140, in get_reachable_from_inputs raise TypeError('Expected Operation, Variable, or Tensor, got ' + str(x)) TypeError: Expected Operation, Variable, or Tensor, got level_5 ``` ## 6. System information - OS Platform and Distribution : Linux Ubuntu 16.04, docker 19.3, nvidia-docker - Mobile device name if the issue happens on a mobile device: N/A - TensorFlow installed from (source or binary): Docker image, binary - TensorFlow version (use command below): 2.2.0 - Python version: 3.6 - Bazel version (if compiling from source):N/A - GCC/Compiler version (if compiling from source):N/A - CUDA/cuDNN version: N/A - GPU model and memory: N/A v2.2.0-rc4-8-g2b96f3662b 2.2.0

ahmadchalhoub · July 6, 2021, 6:38pm

I have previously looked at this but it is a different error than the warning that I am working with. I am able to successfully export a SavedModel from a trained checkpoint using ‘exporter_main_v2.py’ but the trouble I’m facing is working with the exported saved model after that. I need to convert the exported saved model to another format using a command that calls the function 'meta_optimizer.cc, which then returns an empty graph with 0 nodes and 0 edges after ‘optimizing’ it. I have posted a question about that specifically here https://tensorflow-prod.ospodiscourse.com/t/function-optimizer-py-returns-empty-graph/2724

anon26514083 · November 1, 2021, 5:58pm

Infact I am also getting this error. Due to this I am unable to export my model into tflite.