LSTM model only predict 1 constant value

General Discussion
, ,
#1

Hi everyone
I am facing an issue with a model. Seem that the model itself have poor performances and need tune but it only predict single values.

Warning: Test predictions are constant. Model might not be learning effectively.
Warning: Validation predictions are constant. Model might not be learning effectively.

Sample Test Predictions: [0.05704502 0.05704502 0.05704502 0.05704502 0.05704502 0.05704502
0.05704502 0.05704502 0.05704502 0.05704502 0.05704502 0.05704502
0.05704502 0.05704502 0.05704502 0.05704502 0.05704502 0.05704502
0.05704502 0.05704502]
Sample Validation Predictions: [0.05704502 0.05704502 0.05704502 0.05704502 0.05704502 0.05704502
0.05704502 0.05704502 0.05704502 0.05704502 0.05704502 0.05704502
0.05704502 0.05704502 0.05704502 0.05704502 0.05704502 0.05704502
0.05704502 0.05704502]

def create_model(params):

#--------------------------GPU Setup--------------------------------------
# Calculate the global batch size and adjust the learning rate accordingly
#-------------------------------------------------------------------------
strategy = tf.distribute.MirroredStrategy()
num_gpus = strategy.num_replicas_in_sync  # Number of GPUs available
global_batch_size = params['batch_size'] * num_gpus
# params['learning_rate'] *= num_gpus  # Scale the learning rate

input_shape = (params['timesteps'], params['n_features'])
print('Number of devices: {}'.format(strategy.num_replicas_in_sync))

with strategy.scope():
#--------------------------GPU Setup--------------------------------------
# Calculate the global batch size and adjust the learning rate accordingly
#-------------------------------------------------------------------------
    
    # Regularization parameters
    l1_reg = 0.001  # L1 regularization factor 0.005 0.001 0.01 0.02
    l2_reg = 0.001  # L2 regularization factor

    model = Sequential()

    # First LSTM layer with more units
    model.add(LSTM(128, return_sequences=True, input_shape=input_shape,
                   kernel_regularizer=l1_l2(l1=l1_reg, l2=l2_reg)))
    model.add(Dropout(0.2))

    # Second LSTM layer
    model.add(LSTM(64, return_sequences=True,
                   kernel_regularizer=l1_l2(l1=l1_reg, l2=l2_reg)))
    model.add(Dropout(0.2))

    # Bidirectional LSTM layer
    model.add(Bidirectional(LSTM(32, return_sequences=True,
                                 kernel_regularizer=l1_l2(l1=l1_reg, l2=l2_reg))))
    model.add(Dropout(0.2))

    # Dense layer
    model.add(Dense(1, kernel_regularizer=l1_l2(l1=l1_reg, l2=l2_reg)))

==========================================================================

Model Structure Definition

==========================================================================

==========================================================================

Model Compile

==========================================================================

    optimizer = tf.keras.optimizers.Adam(learning_rate=params["learning_rate"], clipvalue=0.5)

    model.compile(optimizer=optimizer, 
                loss='mean_squared_error',
                metrics=[tf.keras.metrics.MeanAbsoluteError(), 
                        tf.keras.metrics.RootMeanSquaredError(), 
                        tf.keras.metrics.MeanAbsolutePercentageError(), 
                        r_squared, 'accuracy'])
                        # Add your custom metrics as needed

return model

==========================================================================

Model Compile

==========================================================================

Creating the model using hyperparameters

model = create_model(params)
model.summary()

U see something wrong in the model definition ?

Tnks !