Issue Calling Predict on Transfer-Learned Model [Docker, React]

General Discussion
#1

Hello, I am attempting to integrate a transfer-learned model with some html code over REACT by using the tensorflow-serving docker container for a university project. I believe I have the container properly set up (was usable with the test files) but am having issues when posting to predict over JSON. Whenever I try to post to the url, I get a 404 error, even though curling the url works fine.

I believe this is an issue to do with my SavedModel, since both the test files work in my tf-serving docker container and the model.predict() function works in colab.

I have been searching over the past few days for a solution to my issue, but was unable to find any topics covering my exact issue, though I feel like I have gotten close I just cannot find what is wrong.

My model is transfer-learned from MobileNetV2, with a few added layers.

This is what the model summary shows:

Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 mobilenetv2_1.00_224 (Funct  (None, 1280)             2257984   
 ional)                                                          
                                                                 
 dense_3 (Dense)             (None, 512)               655872    
                                                                 
 dropout_2 (Dropout)         (None, 512)               0         
                                                                 
 dense_4 (Dense)             (None, 256)               131328    
                                                                 
 dropout_3 (Dropout)         (None, 256)               0         
                                                                 
 dense_5 (Dense)             (None, 5)                 1285      
                                                                 
=================================================================
Total params: 3,046,469
Trainable params: 788,485
Non-trainable params: 2,257,984

And I have it saved (from a checkpoint) by calling this code:

MODEL_DIR = '/my_model'
version = 4
export_path = os.path.join(MODEL_DIR, str(version))
tf.keras.models.save_model(
    model,
    export_path,
    overwrite=True,
    include_optimizer=True,
    save_format=None,
    signatures=None,
    options=None
)

loaded = tf.saved_model.load(export_path)
print(list(loaded.signatures.keys()))  # ["serving_default"]

infer = loaded.signatures["serving_default"]

print(infer.structured_outputs)

I have also tried to use tf.saved_model.save(), however that produced the same results.

The output from the print statements above are so:

['serving_default']
{'dense_5': TensorSpec(shape=(None, 5), dtype=tf.float32, name='dense_5')}

I have noticed that in other implementations, where ‘dense_5’ is there is normally the output name ‘prediction’.

The image attached below is the saved_model_cli information
image

Please forgive me if this is a simple issue, as I am quite new to Tensorflow and tf-serving, Thank you in advance.

#3

@hideen,

Welcome to the Tensorflow Forum!

If you are getting prediction results using curling then model setup on serving should be fine.

Can you please share us the code on how you are sending JSON request on tensorflow serving model? I suspect the issue is with, how you are sending the request to tf-serving model?

Thank you!

#4

By curl the url I had actually meant that it didn’t give me a port error when curled, without the predict method, though I suppose I was a little unclear, i.e.

(tf) F:\VirtualBoxShared\CSE_155_Data>curl http://localhost:8501/1/models/color_finder/metadata
<HTML><HEAD>
<TITLE>404 Not Found</TITLE>
</HEAD><BODY>
<H1>Not Found</H1>
</BODY></HTML>

Though I will include the python code for the JSON request as well:

# Open and read image as bitstring
input_image = open("test_image2.jpg", "rb").read()

# Encode image in b64
encoded_input_string = base64.b64encode(input_image)
input_string = encoded_input_string.decode("utf-8")

# Wrap bitstring in JSON
instance = [{"b64": input_string}]
data = json.dumps({"instances": instance})
headers = {"content-type": "application/json"}

response = requests.post('http://localhost:8501/v1/models/color_finder:predict', data=data, headers=headers)
print(response.json().keys()) # Returns dict_keys(['error'])

result = int(response.json()['predictions'][0])
print(result)

This code isn’t mine, as I found it when looking up how to send images over JSON to an image classifier. The reason why I thought the error was the saved_model signature was due to the only dict_key being ‘error’ when printing the response keys. Thank you for your help.

#5

@hideen,

Please follow this guide and confirm that your saved model directory contains the serialised signatures and variables as shown in example format.

Once done, you can use saved_model_cli to get SignatureDef of your model using show command.

This will make sure your model is in right format to serve. If the above two steps works please follow this guide to serve your model using tensorflow serving with docker.

If serving fails please send us the log on starting tensorflow serving container(i.e docker run) to debug further?

Thank you!

#7

This was the original guide that I was following in order to set up the SavedModel, but was having the same issues, which is why I was trying different ways. I was under the impression that for most models you do not need to set custom signatures, as stated here on that guide:

Important: Unless you need to export your model to an environment other than TensorFlow 2.x with Python, you probably don't need to export signatures explicitly. If you're looking for a way of enforcing an input signature for a specific function, see the input_signature argument to tf.function.

Though I suppose tensorflow-serving counts as a different environment, so I may be misguided in that way of thought.

Currently my saving code looks like this:

MODEL_DIR = '/ColorFinder'
version = 1
export_path = os.path.join(MODEL_DIR, str(version))

tf.saved_model.save(model, export_path)

loaded = tf.saved_model.load(export_path)
print(list(loaded.signatures.keys()))  # ["serving_default"]

infer = loaded.signatures["serving_default"]

print(infer.structured_outputs)

With the print statements showing:

['serving_default']
{'dense_2': TensorSpec(shape=(None, 5), dtype=tf.float32, name='dense_2')}

After downloading and running the model over docker:

saved_model_cli show --dir ColorFinder/1 --tag_set serve

gives the output:

The given SavedModel MetaGraphDef contains SignatureDefs with the following keys:
SignatureDef key: "__saved_model_init_op"
SignatureDef key: "serving_default"

If I need to add custom serialized signatures how would I go about doing that? I’m not sure in how to set up signatures, is this the correct guide for that? It seemed to me like that was for if there wasn’t a serving_default signature.

Thank you for your assistance.

#8

I’ll also add my docker run command and its output here, if needed:

command:

docker run -p 8501:8501 \
  --mount type=bind,\
source=/mnt/f/VirtualBoxShared/CSE_155_Data/ColorFinder,\
target=/models/color_finder\
  -e MODEL_NAME=color_finder -t tensorflow/serving &

output:

2023-04-18 17:31:38.267161: I tensorflow_serving/model_servers/server.cc:74] Building single TensorFlow model file config:  model_name: color_finder model_base_path: /models/color_finder
2023-04-18 17:31:38.268028: I tensorflow_serving/model_servers/server_core.cc:465] Adding/updating models.
2023-04-18 17:31:38.268046: I tensorflow_serving/model_servers/server_core.cc:594]  (Re-)adding model: color_finder
2023-04-18 17:31:38.443053: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: color_finder version: 1}
2023-04-18 17:31:38.443102: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: color_finder version: 1}
2023-04-18 17:31:38.443126: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: color_finder version: 1}
2023-04-18 17:31:38.445757: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:45] Reading SavedModel from: /models/color_finder/1
2023-04-18 17:31:38.479796: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:89] Reading meta graph with tags { serve }
2023-04-18 17:31:38.479844: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:130] Reading SavedModel debug info (if present) from: /models/color_finder/1
2023-04-18 17:31:38.481518: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-18 17:31:38.574667: I external/org_tensorflow/tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:357] MLIR V1 optimization pass is not enabled
2023-04-18 17:31:38.600845: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:229] Restoring SavedModel bundle.
2023-04-18 17:31:38.925297: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:213] Running initialization op on SavedModel bundle at path: /models/color_finder/1
2023-04-18 17:31:39.036579: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:305] SavedModel load for tags { serve }; Status: success: OK. Took 590826 microseconds.
2023-04-18 17:31:39.061501: I tensorflow_serving/servables/tensorflow/saved_model_warmup_util.cc:62] No warmup data file found at /models/color_finder/1/assets.extra/tf_serving_warmup_requests
2023-04-18 17:31:39.143182: I tensorflow_serving/core/loader_harness.cc:95] Successfully loaded servable version {name: color_finder version: 1}
2023-04-18 17:31:39.145019: I tensorflow_serving/model_servers/server_core.cc:486] Finished adding/updating models
2023-04-18 17:31:39.145174: I tensorflow_serving/model_servers/server.cc:118] Using InsecureServerCredentials
2023-04-18 17:31:39.145209: I tensorflow_serving/model_servers/server.cc:383] Profiler service is enabled
2023-04-18 17:31:39.148224: I tensorflow_serving/model_servers/server.cc:409] Running gRPC ModelServer at 0.0.0.0:8500 ...
[warn] getaddrinfo: address family for nodename not supported
2023-04-18 17:31:39.151143: I tensorflow_serving/model_servers/server.cc:430] Exporting HTTP/REST API at:localhost:8501 ...
[evhttp_server.cc : 245] NET_LOG: Entering the event loop ...
#9

@hideen,

From the tensorflow serving docker run log, i can see your model is loaded successfully and your GRPC and REST API endpoints are generated successfully.

It might be a firewall issue. Can you check your firewall rules from the machine where you are posting the request and where serving docker image is?

Yes, you can follow the same document to save the custom signature in tensorflow model, how ever it is not currently supported in tensorflow serving.

Thank you!

#10

I have temporarily disabled my local firewall and restarted docker. After Initializing the container through the docker run command above, and running my json request, I still have the same output from my python test code:

(tf) F:\VirtualBoxShared\CSE_155_Data>python test.py
dict_keys(['error'])  # print statement from outputting the dict_keys
--------------
Traceback (most recent call last):
  File "F:\VirtualBoxShared\CSE_155_Data\test.py", line 46, in <module>
    result = int(response.json()['predictions'][0])
KeyError: 'predictions'
#12

Adding something that may or may not be useful, attached is the metadata from calling

curl http://localhost:8501/v1/models/color_finder/metadata

{"model_spec":{
 "name": "color_finder",
 "signature_name": "",
 "version": "1"},

"metadata": {"signature_def": {
 "signature_def": {
  "serving_default": {
   "inputs": {
    "mobilenetv2_1.00_224_input": {
     "dtype": "DT_FLOAT",
     "tensor_shape": {
      "dim": [
        {"size": "-1",
        "name": ""},
       {"size": "244",
        "name": ""},
       {"size": "244",
        "name": ""},
       {"size": "3",
        "name": ""}],
      "unknown_rank": false},
     "name": "serving_default_mobilenetv2_1.00_224_input:0"}},
   "outputs": {
    "dense_2": {
     "dtype": "DT_FLOAT",
     "tensor_shape": {
      "dim": [
       {"size": "-1",
        "name": ""},
       {"size": "5",
        "name": ""}],
      "unknown_rank": false},
     "name": "StatefulPartitionedCall:0"}},
   "method_name": "tensorflow/serving/predict"},

  "__saved_model_init_op": {
   "inputs": {},
   "outputs": {
    "__saved_model_init_op": {
     "dtype": "DT_INVALID",
     "tensor_shape": {
      "dim": [],
      "unknown_rank": true},
     "name": "NoOp"}},
   "method_name": ""}}}}}