CNN combined with TF DF Random forest model

I have time series data and tabular data. I have developed the hybrid model which takes time series data as input to CNN architecture and tabular data to the TF DF random forest model, which is given to the FC layer for prediction. I would like to know the feature importance of the tabular data. When I try to use the following line to get feature importance it says:
inspector = rf_model_layer.make_inspector()
TypeError: the object of type ‘NoneType’ has no len()
I have attached the model summary : Model: “model_1”


Layer (type) Output Shape Param # Connected to

input_3 (InputLayer) [(None, 12, 3000, 1)] 0 []

conv2d_2 (Conv2D) (None, 1, 2876, 16) 24016 [‘input_3[0][0]’]

conv2d_3 (Conv2D) (None, 1, 2837, 32) 20512 [‘conv2d_2[0][0]’]

input_4 (InputLayer) [(None, 60)] 0 []

flatten_1 (Flatten) (None, 90784) 0 [‘conv2d_3[0][0]’]

random_forest_model_1 (Ran (None, 1) 1 [‘input_4[0][0]’]
domForestModel)

concatenate_1 (Concatenate (None, 90785) 0 [‘flatten_1[0][0]’,
) ‘random_forest_model_1[0][0]’
]

dense_2 (Dense) (None, 32) 2905152 [‘concatenate_1[0][0]’]

dense_3 (Dense) (None, 1) 33 [‘dense_2[0][0]’]

==================================================================================================
Total params: 2949714 (11.25 MB)
Trainable params: 2949713 (11.25 MB)
Non-trainable params: 1 (1.00 Byte)


None
Any suggestions would be appreciated.
Thank you

When you construct the model, are you specifying that it should compute out-of-bag (OOB) variable importances?

model = tfdf.keras.RandomForestModel(compute_oob_variable_importances=True)

More here.

Thank you very much. I did not specify variable importance to be true. Let me try. I also have the query that the below combined model does train both CNN and RF model when I use tuner_search on it? Here’s the sample model built.

class CombinedModel(HyperModel):
            def __init__(self, cnn_input_shape, rf_input_shape):
                self.cnn_input_shape = cnn_input_shape
                self.rf_input_shape = rf_input_shape
            
            def build(self, hp):
                            # CNN part
                    cnn_input = Input(shape=self.cnn_input_shape)
                    RawInputECG = Input(shape=(12,301,1))
                    cnn_output = tf.keras.layers.Conv2D(filters=16, kernel_size=(12, 125), activation='relu')(cnn_input)
                    cnn_output = tf.keras.layers.Conv2D(filters=32, kernel_size=(1, 40), activation='relu')(cnn_output)
                    cnn_output = tf.keras.layers.Flatten()(cnn_output) # here the parameters are fixed to test the model.
                   # RF part
                   rf_output = tfdf.keras.RandomForestModel(
                                      num_trees=fixed_hyperparameters['rf_num_trees'],
                                      max_depth=fixed_hyperparameters['rf_max_depth'],
                                      min_examples=fixed_hyperparameters['min_examples']
                                      )(rf_input)
  
                  # Combine CNN and RF outputs
                  combined_layer = concatenate([cnn_output,  rf_output])
                  # Fully Connected layer
                  fc_activation = hp.Choice('fc_activation', values=['relu', 'sigmoid'])
                  fc_layer = Dense(32, activation=fc_activation)(combined_layer)
                  # Output layer
                  output_layer = Dense(1, activation='relu')(fc_layer)
                  model = Model(inputs=[cnn_input, rf_input], outputs=output_layer)
                  model.compile(optimizer=optimizer, loss='mse',      metrics=[tf.keras.metrics.RootMeanSquaredError(), mae_error])
                  return model
  if __name__ == '__main__':
      cnn_input_shape = (12, 301, 1)
      rf_input_shape = ( 60,)
      combined_model = CombinedModel(cnn_input_shape, rf_input_shape)
   # loading the data for CNN and RF 
  # loading the tuner    
      tuner_bo = RandomSearch(
          combined_model,
          objective=keras_tuner.Objective("val_loss", direction="min"),
          max_trials=50,
          seed=16,
          executions_per_trial=1,
          overwrite=False,
          project_name="Hybrid_model")
      es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=20)
      mc = ModelCheckpoint(saveFN, monitor='val_loss', mode='min', verbose=1, save_best_only=True)
      tuner_bo.search([dataTrain,dataTrainRF], labelsTrain,
                            validation_data = ([dataVal, dataValRF], labelsVal))

In the above code, rf_model is one of the layers in the combined model. Does it get trained when I call tuner_bo.search? or do i have to extract the best hyperparameters and call the best_ model.fit() to make sure both the CNN and RF layer in the combined model are trained as shown below?

# Get the best hyperparameters
best_hyperparameters = keras_tuner.get_best_hyperparameters()[0]

# Build the model with the best hyperparameters
best_model = build_model(best_hyperparameters)

# Train the model
best_model.fit(train_ds, epochs=num_epochs, validation_data=valid_ds)

# Evaluate the model on the test dataset
test_loss, test_accuracy = best_model.evaluate(test_ds)

I also get the following warning when I use best_model.fit()
WARNING:absl:The model was called directly (i.e. using model(data) instead of using model.predict(data)) before being trained. The model will only return zeros until trained. The output shape might change after training Tensor(“inputs:0”, shape=(None, 60), dtype=float32).

Any suggestions would be appreciated.
thank you very much.

Yes, when you invoke search on the tuner, it should cause the combined model (actually a bunch of candidate models) to be trained in multiple trials as it determines the best combination of hyperparameters. Since the rf model is part of the combined model, it should be trained alongside the cnn model.

You should be able to get the best model (which was already trained in the tuning process):

best_model = tuner.get_best_models(num_models=1)[0]

You may use the best model as is, but as you have done, you can retrain it (examples suggest using the entire dataset (training and validation data combined) using the best hyperparameters.

The tutorial gives as an example:

hypermodel = MyHyperModel()
best_hp = tuner.get_best_hyperparameters()[0]
model = hypermodel.build(best_hp)
hypermodel.fit(best_hp, model, x_all, y_all, epochs=1)

So you might modify your code to match that example and see if it makes a difference.

Thank you @rcauvin. Let me try the approach you have suggested and update here if it works. Regarding my first question, it still gets the TypeError: the object of type ‘NoneType’ has no len() even after I specify variable_importance=True. Here’s the code of how I access the random forest model layer from the combined_model

rf_model_layer = model.layers[5]  # Assuming the random forest model is the 5th layer of the combined_model
inspector = rf_model_layer.make_inspector()

It works fine when I call the TFDF RF model alone (not including the CNN model) on tabular data, I get the feature importance.
Since the tfdf random forest model is one of the layers of the Keras model, can I not use make_inspector() on the model directly to get the feature importance? Or the RandomForest model is not trained so I couldn’t access the attribute make_inspector()

I apologize for any inconvenience.
Thank you very much

In

rf_model_layer = model.layers[5]  # Assuming the random forest model is the 5th layer of the combined_model

How are you getting model?

This is how i get the model :


cnn_input = tf.keras.Input(shape=(12, 301, 1))
rf_input = tf.keras.Input(shape=(60,))
cnn_output = tf.keras.layers.Conv2D(filters=16, kernel_size=(12, 125), activation='relu')(cnn_input)
cnn_output = tf.keras.layers.Conv2D(filters=32, kernel_size=(1, 40), activation='relu')(cnn_output)
cnn_output = tf.keras.layers.Flatten()(cnn_output)

rf_output = tfdf.keras.RandomForestModel(
    num_trees=fixed_hyperparameters['rf_num_trees'],
    max_depth=fixed_hyperparameters['rf_max_depth'],
    min_examples=fixed_hyperparameters['min_examples'],
    compute_oob_variable_importances=True
)(rf_input)

combined_output = tf.keras.layers.concatenate([cnn_output, rf_output])
fc_output = tf.keras.layers.Dense(32, activation='relu')(combined_output)
output = tf.keras.layers.Dense(1, activation='relu')(fc_output)

model = tf.keras.Model(inputs=[cnn_input, rf_input], outputs=output)

optimizer = tf.keras.optimizers.Adam(learning_rate=fixed_hyperparameters['learning_rate'])
model.compile(optimizer=optimizer, loss='mse')


trained_model=model.fit([dataTrain, dataTrainRF], labelsTrain, validation_data=([dataVal, dataValRF], labelsVal), epochs=5)

rf_model_layer = model.layers[5]  # Assuming the random forest model is the 5th layer
inspector = rf_model_layer.make_inspector()

This is the complete error :
Traceback (most recent call last):

Cell In[184], line 2
inspector = rf_model_layer.make_inspector()

File ~/anaconda3/lib/python3.11/site-packages/tensorflow_decision_forests/keras/core_inference.py:411 in make_inspector
path = self.yggdrasil_model_path_tensor().numpy().decode(“utf-8”)

File ~/anaconda3/lib/python3.11/site-packages/tensorflow/python/util/traceback_utils.py:153 in error_handler
raise e.with_traceback(filtered_tb) from None

File /tmp/autograph_generated_file0diouhww.py:38 in tf__yggdrasil_model_path_tensor
ag
.if_stmt(ag__.ld(multitask_model_index) >= ag__.converted_call(ag__.ld(len), (ag__.ld(self)._models,), None, fscope), if_body, else_body, get_state, set_state, (), 0)

TypeError: in user code:

File "/home/hybrid/anaconda3/lib/python3.11/site-packages/tensorflow_decision_forests/keras/core_inference.py", line 436, in yggdrasil_model_path_tensor  *
    if multitask_model_index >= len(self._models):

TypeError: object of type 'NoneType' has no len()

Here are the layers present in the model.

layers = model.layers
print(layers)

Output
[<keras.src.engine.input_layer.InputLayer object at 0x7f00587cf390>, <keras.src.layers.convolutional.conv2d.Conv2D object at 0x7f005840fed0>, <keras.src.layers.convolutional.conv2d.Conv2D object at 0x7f0058763b50>, <keras.src.engine.input_layer.InputLayer object at 0x7f00587095d0>, <keras.src.layers.reshaping.flatten.Flatten object at 0x7f00586242d0>, <tensorflow_decision_forests.keras.RandomForestModel object at 0x7f0058463710>, <keras.src.layers.merging.concatenate.Concatenate object at 0x7f00584604d0>, <keras.src.layers.core.dense.Dense object at 0x7f005845fe10>, <keras.src.layers.core.dense.Dense object at 0x7f00587b8910>]
Thank you

What is the output of print(trained_model.layers?

And have you tried retrieving rf_model_layer from trained_model instead of model?

I get this error when I try to print the trained_model.layers:

print(trained_model.layers)
Traceback (most recent call last):

  Cell In[214], line 1
    print(trained_model.layers)

AttributeError: 'History' object has no attribute 'layers'

Thank you very much for your prompt response

Sorry, I forgot that model.fit returns a History object.

This tutorial shows how to stitch together neural networks and decision forest models, but it trains the decision forest models separately before combining them into a single ensemble model.

Thank you very much for the reference link.

  1. What is happening in the following lines of code in the above reference link? The combined model ( NN and DF, ensemble_nn_and_df) has not been trained before. It’s just two DF models trained separately and how is it reflected in the ensemble_nn_and_df?
Let's train the two Decision Forest components (one after another).
%%time
train_dataset_with_preprocessing = train_dataset.map(lambda x,y: (preprocessor(x), y))
test_dataset_with_preprocessing = test_dataset.map(lambda x,y: (preprocessor(x), y))

model_3.fit(train_dataset_with_preprocessing)
model_4.fit(train_dataset_with_preprocessing)

mean_nn_and_df = tf.reduce_mean(
    tf.stack([m1_pred, m2_pred, m3_pred, m4_pred], axis=0), axis=0)
ensemble_nn_and_df = tf_keras.models.Model(raw_features, mean_nn_and_df)
ensemble_nn_and_df.compile(
    loss=tf_keras.losses.BinaryCrossentropy(), metrics=["accuracy"])
evaluation_nn_and_df = ensemble_nn_and_df.evaluate(
    test_dataset, return_dict=True)
  1. I have numpy array for the CNN model and tabular data (with categorical variables) for RF data. When I use tfdf.keras.pd_dataframe_to_tf_dataset for tabular data for the TFDF Random model (as it contains categorical variables) and tf. data.Dataset.from_tensor_slices for CNN model layers. It is not compatible. Do you have any suggestions on how to make two inputs to be fed compatible with two models? Or if we train models separately as mentioned in the reference link, the compatibility issue wouldn’t arise.

Any help is appreciated.
Thank you very much.

It looks like the tutorial creates and stitches the models together before compiling or training them. Then it compiles and trains the neural network and the decision forests separately, after which ensemble_nn_and_df is compiled and evaluated.

I think you have a few options for dealing with the input dataset and using it to train the different models:

  1. Train the models separately as in the tutorial (but with preprocessed input for the CNN model and tabular data input for the decision forest model).
  2. Add some feature preprocessing layers so that the CNN model receives preprocessed input while the decision forest model receives the tabular data.
  3. Use preprocessed input for both models.