How to do parallel batch inferencing?

General Discussion
, , ,
#1

Hi Team, I have built the tf2 model which does image classification. Currently, when I do batch inference for 50 images it’s takes 42secs. And I does sequential inference. I would like to do parallel inference and reduce the inference because I got 70K images to do inference as batch.

Can anyone please help me here to solve the problem?

Please find me script below,

import tensorflow as tf, numpy as np
from PIL import Image, ImageOps


model = tf.keras.models.load_model(‘export/model’)


image = ‘new_image.png’
image = Image.open(image).convert(‘RGB’)
image = ImageOps.exif_transpose(image)
image = np.array(image.resize((224,224)))
image = np.reshape(image,(1,224,224,3))
prediction_result = model.predict(image)

Can anyone please help me to do parallel inference.

Currently, I use for loop to iterate the prediction sequentiality.

#tf2 #keras #inference #tensorflow #batchinference

#3

You need to create a Tensorflow Dataset class that reads your images and yields them in a sequence. Keras models expect to iterate through thousands of images using an object that reads a batch of images, collects the batch into a sequence, then returns that batch, then later starts on the next batch.

Learn the Dataset abstraction to achieve what you are trying to do with 70k images.

https://www.tensorflow.org/api_docs/python/tf/data/Dataset

1 Like