TypeError("dataset length is unknown.") while reading tfrecord

Trying to read the tfrecord file and use it to train the model with .fit call but getting this error:

TypeError("dataset length is unknown.")

Here’s my tfrecord code:

    'lr': tf.io.FixedLenFeature([], tf.string),
    'hr': tf.io.FixedLenFeature([], tf.string),

def parser(example_proto):
    parsed_example = tf.io.parse_single_example(example_proto, FEATURE_DESCRIPTION)
    lr = tf.io.decode_jpeg(parsed_example['lr'])
    hr = tf.io.decode_jpeg(parsed_example['hr'])
    return lr, hr

train_data = tf.data.TFRecordDataset(TFRECORD_PATH)\
                    .batch(BATCH_SIZE, drop_remainder = True)\

And len(train_data) is giving error TypeError("dataset length is unknown.") because the cardinality is -2, or in other words the train_data is unable to capture the total number of samples because the dataset source is a file.
Is there any way we can tell the train_data how many samples/batches are there?

Check this thread:


1 Like

The solution is to manually set the cardinality as below:

# print(len(train_data)) gives error
train_data = train_data.apply(tf.data.experimental.assert_cardinality(NUM_BATCHES))
print(len(train_data)) # NUM_BATCHES
1 Like