Merge two models

How can I implement this architecture in image bellow.

I really appreciate your help;
Thank you

Hi @youb, Once you have defined your 2 models. For example

import tensorflow as tf
vgg_model = tf.keras.applications.VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
resnet = tf.keras.applications.ResNet50(include_top=False, weights='imagenet', input_shape=(224, 224, 3))

    tf.keras.layers.Dense(1024, activation='relu'),
    tf.keras.layers.Dense(512, activation='relu')

    tf.keras.layers.Dense(512, activation='relu')

Then to do average of the last layers of 2 models and adding the remaining layers, you can do it by

merged_model=tf.keras.layers.Average()([model1.output, model2.output])

model = tf.keras.models.Model(inputs=[model1.output, model2.output], outputs=merged_model)

Please refer to this gist for working code example. Thank You.

1 Like

Hi @Kiran_Sai_Ramineni
thank you for code;
please how to train this model; is I should train each model alone, I mean model1, then model2 and after merged_model;
I have tried to compile and train the merged_model; I got an error

Thank you

I have tried this but I’m not sure:
import tensorflow as tf

from tensorflow.keras.layers import Dense, Dropout, Input, GlobalAveragePooling2D, Average, TimeDistributed, Average, Flatten

from tensorflow.keras.models import Model

from tensorflow.keras.applications.resnet50 import ResNet50

import tensorflow_hub as hub

Load the ResNet50 model with pre-trained ImageNet weights

resnet = ResNet50(weights=“imagenet”, include_top=False, input_tensor=Input(shape=(224, 224, 3)))

resnet.trainable = False

Load the ViT-B16 model

vit16 = hub.KerasLayer(“TensorFlow Hub”, trainable=False)

Define the inputs

inputs = Input(shape=(10, 224, 224, 3))

Extract features from ResNet50 and ViT-B16 models

resnet_features = TimeDistributed(resnet)(inputs)

resnet_features = TimeDistributed(GlobalAveragePooling2D())(resnet_features)

resnet_features = Flatten()(resnet_features)

resnet_features = Dense(1024)(resnet_features)

resnet_features = Dense(512)(resnet_features)

vit_features = TimeDistributed(vit16)(inputs)

#vit_features = TimeDistributed(tf.keras.layers.LayerNormalization())(vit_features)

vit_features = Flatten()(vit_features)

vit_features = Dense(512)(vit_features)

Take the average of the features

averaged_features = Average()([resnet_features, vit_features])

Add Dense layer with Dropout

x = Dense(512, activation=‘relu’)(averaged_features)

x = Dropout(0.5)(x)

Add final Dense layer with sigmoid activation function

outputs = Dense(5, activation=‘sigmoid’)(x)

Define the model

model = Model(inputs=inputs, outputs=outputs)

Compile the model

model.compile(optimizer=‘adam’, loss=‘mse’, metrics=[‘mae’])


Hi @youb, first you have to train the vgg16 and resnet50 model then you have to pass the last layers outputs of both models to the merged model for training. You are getting the error because you are passing data to the merged model with shape (224,224,3) but the merged model input shape will be last layer output of vgg and resnet models which is (None,512). Thank You,

1 Like

Thank you so much

Remember to freeze everything but the last Dense(3) and Dense(512) when you train the combined model otherwise you will lose any pretraining of the two models and the stretch after them. You can unfreeze and fine-tune once the last two dense layers have converged.

Yes I did; freeze all pretrained layers.
Thank you so much.