LSTM using pandas 0.25 and tensorflow 1.15

I have raised an issue in StackOverflow that you can view in detail, but suffice to say I’m receiving an error I can’t make sense of.

KeyError: ‘No key found for either mapped or original key. Mapped Key: []; Original Key: []’

The intermediate goal for now is to simply get the LSTM model to forecast 1 period ahead using the current period’s data (only 1 timestep). I have two independent variables and one dependent variable. I think perhaps my understanding of how batches/timesteps are processed is a little muddy though. Can someone help me with the error and tell me if the summary() of the model makes sense for what I want to achieve?

Code (assume frame is a pandas dataframe that holds all of the variables):

X = frame[['optimal_long_log_return', 'optimal_short_log_return']][:-1]
Y = frame['equilibrium_log_return'].shift(-1)[:-1]
X_train, _X, y_train, _y = train_test_split(X, Y, test_size=0.5, shuffle=False, random_state=1)
X_validation, X_test, y_validation, y_test = train_test_split(_X, _y, test_size=0.5, shuffle=False, random_state=1)

train = TimeseriesGenerator(X_train, y_train, 1, batch_size=batch_size)
validation = TimeseriesGenerator(X_validation, y_validation, 1, batch_size=batch_size)
test = TimeseriesGenerator(X_test, y_test, 1, batch_size=batch_size)

model = Sequential(name='Expected Equilibrium Log Return')
model.add(LSTM(256, name='LSTM', stateful=True, batch_input_shape=(batch_size, 1, X_train.shape[1])))
model.add(Dense(1, name='Output'))
model.compile(loss='logcosh', metrics=['mean_absolute_percentage_error'], optimizer='Adam', sample_weight_mode='temporal')
model.fit_generator(train, epochs=10, validation_data=validation, verbose=1)
loss, accuracy = model.evaluate_generator(test, verbose=0)
print(f'Model Accuracy: {accuracy}')


Model: "Expected Equilibrium Log Return"
Layer (type)                 Output Shape              Param #   
LSTM (LSTM)                  (32, 256)                 265216    
Output (Dense)               (32, 1)                   257       
Total params: 265,473
Trainable params: 265,473
Non-trainable params: 0

I have created this question in detail on stackoverflow, but as a synopsis:

I want to create an LSTM which will take in a pandas dataframe of independent variables and optimize for the dependent variable. The error I’m receiving is as such:

KeyError: ‘No key found for either mapped or original key. Mapped Key: []; Original Key: []’

Unfortunately I cannot update tensorflow from 1.15 or pandas from 0.25 as the environment I’m developing in doesn’t allow for it. The error occurs in the fit_generator() method.

Turns out this specific issue relates to a patch that Quantconnect made to pandas dataframes which interfered with the older version of tensorflow/keras.