Auto shard policy for tf.keras.utils.Sequence

General Discussion
, ,
#1

Hello there,

So I am feeding a Sequence object to a recommender model. During training, I am getting the message:

AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_2

My question is how can I apply or disable the sharding policy? I understand that you have to create an options object and apply it to a tf.data.Dataset object, but I am not sure how does this work with a Sequence object instead.

Any help is greatly appreciated. Thanks.

#3

Hi @Andres_Toledo, You can disable the sharding using tf.data.experimental.AutoShardPolicy.OFF.

options = tf.data.Options()
options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF

For more details please refer to this documentation. Thank You.