Concat 2 models with different size

Research & Models
#1

HI,
I have a dataset of videos (60); from this videos I extract images and audio,
from each video I extract 30 images, so in total 1800 images, the shape of x_train and y_train is
x_train.shape => (1800,224,224,3),
y_train.shape => (1800,5)

From audio i extract 15 signal (array of numbers), so in total 60*15=900, the shape of x_train and y_train is
x_train.shape => (900, 128)
y_train.shape => (900,5)

images are fed into VggFace (fine tuned ) model(1); the output shape is (1800,128)
audios are fed into a Vggish model (2), the output shape is (900, 128)

after training two model (1 and 2)
Both models are used as input for third model3; The problem I faced is that:

model3.fit([x_audio_train, x_video_train], y_train, …)
I got the error:

the input size should be the same.

I hope you got my issue;

How can I fix this ?
Thank you

#3

Hi @youb,

Merge the outputs from the image and audio models using a fusion layer

fusion_layer = tf.keras.layers.Concatenate() 
merged_outputs = fusion_layer([image_outputs, audio_outputs])

model3 = tf.keras.Sequential([
    # Add your layers here
])

model3.fit(merged_outputs, y_train, ...)

I think above workaround will solve your problem.

Please let me know if it solves your problem.

Thanks