How to visualize the embedding and weights in multimodal model

Research & Models
,
#1

How to visualize the individual branch embeddings from the below model

def branch(self):
    res_inp = Input(shape=(self.res_num, self.max_nbr))
    res_fea_len = Input(shape=(self.res_num, self.res_fea_len))
    res_type = Input(shape=(self.res_num,))
    res_fea = Embedding(input_dim=200, output_dim=self.res_fea_len)(res_type)
    for _ in range(self.ncov):
        res_fea = self.cov_layer(a_inp, res_inp, res_fea)

    res_fea = Activation('relu')(res_fea)
    ouput = Dense(self.num_target)(res_fea)

    branch = Model(inputs=[res_fea_len, res_inp, res_type], outputs=ouput)

    return branch

def multi_modal(self):
    
    res_inp_1 = Input(shape=(self.res_num, self.max_nbr))
    res_fea_len_1 = Input(shape=(self.res_num, self.res_fea_len))
    res_type = Input(shape=(self.res_num,))
    res_inp_2 = Input(shape=(self.res_num, self.max_nbr))
    res_fea_len_2 = Input(shape=(self.res_num, self.res_fea_len))
    
    
    model_brach = branch()

    branch_1 = model_branch(res_fea_len_1, res_inp_1, res_type)
    branch_2 = model_branch(res_fea_len_2, res_inp_2, res_type)

    combined = concatenate([branch_1, branch_2])

    self.model = Model(inputs=[res_fea_len_1, res_inp_1, res_type, res_fea_len_2, res_inp_2], outputs=combined)
    self.model.compile(optimizer=self.optimizer)
    self.losses = ['mse', 'mqse']

I was trying to save the weights but was not able to extract the embeddings from an individual branch.

#3

There are no “individual branch embeddings”. You have created one set of weights with this:

    model_branch = branch()

You then apply it to two different input sets. The full model uses the same set of weights for both branches. This design technique is called (I think) “tied weights”. If you want separate weights for each input branch, then this is the right code:

    model_branch1 = branch()
    model_branch2 = branch()

    branch_1 = model_branch1(res_fea_len_1, res_inp_1, res_type)
    branch_2 = model_branch2(res_fea_len_2, res_inp_2, res_type)
#4

@Lance_N
I don’t want to separate weights, I want the "tied_weights"functionality as it is. If I use your code, I think I could not retain the functionality of the model? Correct me if I am wrong.
However, I want to explore the embedding space.

#5

Apologies, I think I misread your first post.

You can run branch1.predict() on the input data to get the embedding created by the branch.