Hi! I’m trying to train a model with the imagenet2012 using a cluster of three workers. This cluster has three nodes, all with at least one GPU, but the hard-drive size and memory is limited for two of the nodes, having a storage capacity of 100Gb in total, which makes it impossible to store the imagenet dataset there, so it is stored in the remaining node that has 2TB storage.
My problem is the following: I would like to train a model using the MultiWorkerMirroredStrategy involving the three nodes I mentioned, but the dataset is only available to one of the workers, which is a no-no for the other two nodes since the model.fit function requires a dataset to train the model with. Is it possible to execute this training strategy with these settings? Maybe I’m missing something regarding the configuration of the dataset distribution or the model fitting function? Thank you all in advance!