Keras model.predict does not work with 2 inputs

General Discussion
, , ,
#1

I’m completely new to Keras and AI. I have Keras 2.9.0 with Python 3.8.10 under Ubuntu 20.04. I have a model trained using 2 X inputs and an Y, and technically the training runs. Now I wanted to predict the Y using 2 inputs, but it fails. The training is done using this code fragment (I think only input and output format is interesting here):

def generate(aBatchSize:int=32, aRepeatParameter:int=2, aPort:int=12345):
  dim = (512, 512)
  paraShape = (aRepeatParameter * 2,)

  def generator():
    while True:
    # fill variables
      yield ((xParameter, xImage), y)

  dataset = tensorflow.data.Dataset.from_generator(generator,
    output_signature=(
      (tensorflow.TensorSpec(shape=paraShape, dtype=tensorflow.float32),
      tensorflow.TensorSpec(shape=dim, dtype=tensorflow.float32)),
      tensorflow.TensorSpec(shape=(1), dtype=tensorflow.float32)
            ))
  dataset = dataset.batch(aBatchSize)
  return dataset

repeatParameter = 2
batchSize = 16
model.fit(landscapeGenerator.generate(batchSize, repeatParameter, port), validation_data=landscapeGenerator.generate(batchSize, repeatParameter, port),
  epochs=50, steps_per_epoch=math.ceil(sampleSize / batchSize), validation_steps=validationSize/batchSize )

Printing the model input and output from training code yields this:

model.input [<KerasTensor: shape=(None, 4) dtype=float32 (created by layer 'input_1')>, <KerasTensor: shape=(None, 512, 512, 1) dtype=float32 (created by layer 'input_2')>]
model.output KerasTensor(type_spec=TensorSpec(shape=(None, 1), dtype=tf.float32, name=None), name='dense_4/BiasAdd:0', description="created by layer 'dense_4'")

This is the failing inference code:

image = numpy.multiply(imageio.imread(filename), 1.0 / 255.0)
model = tensorflow.keras.models.load_model(modelDir)

repeatParameter = 2
paraShape = (repeatParameter * 2,)
parameter = numpy.empty(paraShape, dtype=float)
# fill parameters
tempDiff = 5.0 * model.predict((parameter, image))  # also tried here with list instead of tuple, no luck

It writes, because does not understand that the model has 2 inputs of different size: ValueError: Data cardinality is ambiguous: x sizes: 4, 512 Make sure all arrays contain the same number of samples.

I also wanted to make the prediction using a generator, because for that I know how to provide shape info, but no success:

def generate(aParameter, aParaShape, aImage):
  dim = (512, 512)
  def generator():
    while True:
      yield (aParameter, aImage)

  dataset = tensorflow.data.Dataset.from_generator(generator,
    output_signature=(
      tensorflow.TensorSpec(shape=paraShape, dtype=tensorflow.float32),
      tensorflow.TensorSpec(shape=dim, dtype=tensorflow.float32)
    ))
  dataset = dataset.batch(1)
  return dataset

image = numpy.multiply(imageio.imread(filename), 1.0 / 255.0)
model = tensorflow.keras.models.load_model(modelDir)
repeatParameter = 2
paraShape = (repeatParameter * 2,)
parameter = numpy.empty(paraShape, dtype=float)
# fill parameters
tempDiff = 5.0 * model.predict(generate(parameter, paraShape, image), batch_size=1, steps=1)

This one complains: ValueError: Layer "model_2" expects 2 input(s), but it received 1 input tensors. Inputs received: [<tf.Tensor 'IteratorGetNext:0' shape=(None, 4) dtype=float32>]

I’ve tried several ideas from Stack Overflow, but none helped. Perhaps the API has changed since they have been published. What do I do wrong?

#3

I’d check the dimensions of the data you’re feeding in for training and also for predictions. It looks like you have a shape of [None, 4] and [None, 512] for training (where None is replaced by your batch size, which could even be 1), and when predicting you have data shape of [4] and [512]. If you make the tensor shapes [None, 4] and [None, 512] for prediction you should be ok.
The way to test this is to print .shape, for example image.shape , both during training and prediction to see what’s different.

#5

Thank you for the hint. For both cases, the shapes are the following:

inf:
Parameter shape:(4,)
Image shape:(512, 512)
tr:
Parameter shape:(4,)
Image shape:(512, 512)

I’ve printed the shape from training in the generator, before yield. In prediction, I’ve printed it before prediction. I think I understand what is missing for prediction, I try to do it.

git diff
diff --git a/inferenceTemp.py b/inferenceTemp.py
index 956f281..aff3695 100644
--- a/inferenceTemp.py
+++ b/inferenceTemp.py
@@ -39,6 +39,8 @@ if(argC > 4):
     parameter[i] = parameter[0]
     parameter[repeatParameter + i] = parameter[repeatParameter]

+  print(f'Parameter shape:{parameter.shape}')
+  print(f'Image shape:{image.shape}')
   tempDiff = 5.0 * model.predict(generate(parameter, paraShape, image), batch_size=1, steps=1)
   print(f'Inferenced temperature difference is {tempDiff} Celsius.')

diff --git a/landscapeGenerator.py b/landscapeGenerator.py
index d7b1acc..96c2a08 100644
--- a/landscapeGenerator.py
+++ b/landscapeGenerator.py
@@ -35,6 +35,8 @@ def generate(aBatchSize:int=32, aRepeatParameter:int=2, aPort:int=12345):
       for i in range(0, 3):
         xImage[0, i]  = xImage[1, i]
 #      print(xParameter[0] * 0.04, 600 * (100 ** xParameter[aRepeatParameter]), y[0] * 5)
+      print(f'Parameter shape:{xParameter.shape}')
+      print(f'Image shape:{xImage.shape}')
       yield ((xParameter, xImage), y)
#6

ok, if you need any more help its probably worth sharing the complete code

#7

Here you are. Included my failed experiments to expand the dimension for the batch.

import sys
import math
import numpy
import imageio
import tensorflow.keras.models


def generate(aParameter, aParaShape, aImage):
  dim = (512, 512)
  def generator():
    while True:
      yield (aParameter, aImage)

  dataset = tensorflow.data.Dataset.from_generator(generator,
    output_signature=(
      tensorflow.TensorSpec(shape=paraShape, dtype=tensorflow.float32),
      tensorflow.TensorSpec(shape=dim, dtype=tensorflow.float32)
    ))
  dataset = dataset.batch(1)
  return dataset

argC = len(sys.argv)
if(argC > 4):
  modelDir    = sys.argv[1]
  filename    = sys.argv[2]
  fieldOfView = float(sys.argv[3])
  distance    = float(sys.argv[4])
  image = numpy.multiply(imageio.imread(filename), 1.0 / 255.0)  # TODO normalize instead of div

  model = tensorflow.keras.models.load_model(modelDir)
  print(model.summary())
  repeatParameter = 2
  paraShape = (repeatParameter * 2,)

  parameter = numpy.empty(paraShape, dtype=float)
  parameter[0] = fieldOfView / 0.04
  parameter[repeatParameter] = math.log(distance / 600.0) / math.log(100.0)
  for i in range(1, repeatParameter):
    parameter[i] = parameter[0]
    parameter[repeatParameter + i] = parameter[repeatParameter]

  #parmeter = numpy.expand_dims(parameter, -1)
  #image = numpy.expand_dims(image, -1)
  parmeter = numpy.reshape(parameter, (1, 4))
  image = numpy.reshape(image, (1, 512, 512))
  print(f'Parameter shape:{parameter.shape}')
  print(f'Image shape:{image.shape}')
  tempDiff = 5.0 * model.predict([parameter, image], batch_size=1, steps=1)
  # tempDiff = 5.0 * model.predict(generate(parameter, paraShape, image), batch_size=1, steps=1)
  print(f'Inferenced temperature difference is {tempDiff} Celsius.')

else:
  print(f"Usage: {sys.argv[0]} modelDir imageName HorizontalFieldOfViewInRadian distanceInMeters")
  print("The image must be 512*512 grayscale PNG.")

Here is a sample model (does nothing, but has the needed I/O).
model

Here is a sample image:
sample

#9

I’ve found the bug: there was a typo in this line:

parmeter = numpy.reshape(parameter, (1, 4))

The variable should be called parameter.