Can model.fit handle strided data?

General Discussion
,
#1

Hi,

X_train shape: (168708, 80, 20, 1)
y_train shape: (168708, 3)

The data is strided (sliding window). The problem is that when i call model.fit with the strided data it gives a segfault:

Epoch 1/160
Fatal Python error: Segmentation fault

Current thread 0x00007fa682bf04c0 (most recent call first):
File “/home/rarr/miniconda3/lib/python3.10/site-packages/keras/utils/generic_utils.py”, line 444 in
File “/home/rarr/miniconda3/lib/python3.10/site-packages/keras/utils/generic_utils.py”, line 444 in slice_arrays
File “/home/rarr/miniconda3/lib/python3.10/site-packages/keras/engine/training_arrays_v1.py”, line 397 in model_iteration
File “/home/rarr/miniconda3/lib/python3.10/site-packages/keras/engine/training_arrays_v1.py”, line 734 in fit
File “/home/rarr/miniconda3/lib/python3.10/site-packages/keras/engine/training_v1.py”, line 856 in fit
File “/home/rarr/DevelNeural/./Test_Generator.py”, line 490 in

Thanks!

#3

Hi @Rr_Tt, Generally this error is related to memory. This error occurs when the python program tries to access a memory beyond reach. could you please try to increase the stack that your operating system allocates for the python process. Also try to train the model on a smaller dataset to see if the error persists. Thank You.

#4

Thanks for the quick reply. I does seem to work with a training set that is Megabytes in size. Not sure where it exactly starts to fail. A few GB is already too much for it.

This is very weird, i have 32GB or RAM and 250GB swap, i was able to use 150GB datasets before adding striding to the data.