Train a neural network with input as sliding windows of a 2D matrix

I have a dataset which is a big matrix of shape (100 000, 2 000).

I would like to train the neural network with all the possible sliding windows/submatrices of shape (16, 2000) of this big matrix.

enter image description here

I use:

from skimage.util.shape import view_as_windows

A.shape  # (100000, 2000)  ie 100k x 2k matrix
X = view_as_windows(A, (16, 2000)).reshape((-1, 16, 2000, 1))
X.shape   # (99985, 16, 2000, 1)
...
model.fit(X, Y, batch_size=4, epochs=8)

Unfortunately, this leads to a memory problem:

W tensorflow/core/framework/allocator.cc:122] Allocation of … exceeds 10% of system memory.

This is normal, since X has ~ 100k * 16 * 2k coefficients, i.e. more than 3 billion coefficients!

But in fact, it is a waste of memory to load X in memory because it is highly redundant: it is made of sliding windows of shape (16, 2000) over A.

Question: how to train a neural network with input being all sliding windows of width 16 over a 100k x 2k matrix, without wasting memory?

The documentation of skimage.util.view_as_windows states indeed that it’s costly in memory:

One should be very careful with rolling views when it comes to memory usage. Indeed, although a ‘view’ has the same memory footprint as its base array, the actual array that emerges when this ‘view’ is used in a computation is generally a (much) larger array than the original, especially for 2-dimensional arrays and above.

For example, let us consider a 3 dimensional array of size (100, 100, 100) of float64. […] the hypothetical size of the rolling view (if one was to reshape the view for example) would be 8*(100-3+1)3*33 which is about 203 MB! The scaling becomes even worse as the dimension of the input array becomes larger.


Edit: timeseries_dataset_from_array is exactly what I’m looking for except that it works only for 1D sequences:

import tensorflow
import tensorflow.keras.preprocessing
x = list(range(100))
x2 = tensorflow.keras.preprocessing.timeseries_dataset_from_array(x, None, 10, sequence_stride=1, sampling_rate=1, batch_size=128, shuffle=False, seed=None, start_index=None, end_index=None)
for b in x2:
    print(b)

and it doesn’t work for 2D arrays:

x = np.array(range(90)).reshape(6, 15)
print(x)
x2 = tensorflow.keras.preprocessing.timeseries_dataset_from_array(x, None, (6, 3), sequence_stride=1, sampling_rate=1, batch_size=128, shuffle=False, seed=None, start_index=None, end_index=None)
# does not work

3 posts were split to a new topic: I can’t edit my post