Optimization loop failed: Cancelled: Operation was cancelled

Bhack · August 5, 2021, 8:58am

What TF version are you using?

Have you checked:

github.com/tensorflow/tensorflow

Error occurred when finalizing GeneratorDataset iterator

opened 10:06PM - 13 Dec 19 UTC

closed 12:22AM - 20 Dec 19 UTC

olk

type:bug comp:data TF 2.1

**System information** - OS Platform and Distribution: Arch Linux, 5.4.2-arch1-…1-ARCH - TensorFlow installed from: binary - TensorFlow version: 2.1.0rc0-1 - Keras version: 2.2.4-tf - Python version: 3.8 - GPU model and memory: 2x GTX 1080 Ti 11GB"` **Describe the current behavior** executing Tensorflow's MNIST handwriting example produces error: the error dissapears if the code doesn't use OneDeviceStrategy or MirroredStrategy > W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled **Code to reproduce the issue** ``` import tensorflow as tf import tensorflow_datasets as tfds import time from tensorflow.keras.optimizers import Adam def build_model(): filters = 48 units = 24 kernel_size = 7 learning_rate = 1e-4 model = tf.keras.Sequential([ tf.keras.layers.Conv2D(filters=filters, kernel_size=(kernel_size, kernel_size), activation='relu', input_shape=(28, 28, 1)), tf.keras.layers.MaxPooling2D(), tf.keras.layers.Flatten(), tf.keras.layers.Dense(units, activation='relu'), tf.keras.layers.Dense(10, activation='softmax') ]) model.compile(loss='sparse_categorical_crossentropy', optimizer=Adam(learning_rate), metrics=['accuracy']) return model datasets, info = tfds.load(name='mnist', with_info=True, as_supervised=True) mnist_train, mnist_test = datasets['train'], datasets['test'] num_train_examples = info.splits['train'].num_examples num_test_examples = info.splits['test'].num_examples strategy = tf.distribute.OneDeviceStrategy(device='/gpu:0') BUFFER_SIZE = 10000 BATCH_SIZE = 32 def scale(image, label): image = tf.cast(image, tf.float32) image /= 255 return image, label train_dataset = mnist_train.map(scale).shuffle(BUFFER_SIZE).repeat().batch(BATCH_SIZE).prefetch(buffer_size=tf.data.experimental.AUTOTUNE) eval_dataset = mnist_test.map(scale).repeat().batch(BATCH_SIZE).prefetch(buffer_size=tf.data.experimental.AUTOTUNE) with strategy.scope(): model = build_model() epochs=5 start = time.perf_counter() model.fit( train_dataset, validation_data=eval_dataset, steps_per_epoch=num_train_examples/epochs, validation_steps=num_test_examples/epochs, epochs=epochs) elapsed = time.perf_counter() - start print('elapsed: {:0.3f}'.format(elapsed)) ```

Farquhar13 · August 17, 2021, 6:48pm

I was using TF version 2.5 and have just upgraded to 2.6 where I am still getting the same issue. I am also using Python 3.6.

After I upgraded to TF 2.6, I am still getting the error, but the path is different.

In 2.6:
“W tensorflow/core/data/root_dataset.cc:167] Optimization loop failed: Cancelled: Operation was cancelled”

Whereas in 2.5:
“W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled”

I looked through that tread, but I wasn’t able to solve the problem. The error is that tread also has a different path:
“W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled”

Any help would be appreciated.

Anish_Josh · September 5, 2021, 10:13am

I was able to fix this, the reason why it was throwing an error was my batch_size was set greater than the size of the whole dataset.

FlyingMike · October 26, 2021, 5:07pm

This might be the case.
I encountered the same problem. As stated by @Farquhar13 , this warning information doesn’t always show up so this means the size of the batch should be fine? Please correct me if I am wrong.
Thanks!

Grant_Allan · December 14, 2021, 3:55am

I’m getting this same error.

2021-12-13 17:59:55.175834: W tensorflow/core/data/root_dataset.cc:163] Optimization loop failed: CANCELLED: Operation was cancelled

I know that my problem isn’t the batch size being larger than my dataset. I checked the size of the dataset each time I got the error, and it’s the same size as when I don’t get the error. Speaking of which, I don’t get this error every time I run the code. It seems almost random, but it obviously can’t be.

I can’t find a solution to this anywhere, so I’m hoping someone here can help me out.

Winston_Chang · January 2, 2022, 4:28am

I am running into a similar issue. I’m running TS v2.7 and python 3.7.4
I’ve checked the shapes passing and the batch size as referred to above and those are not the issues. However I get it on every iteration.

Any help would be appreciated!

user37 · March 14, 2022, 2:54am

I have the same problem as you, how did you solve it, I hope you can help me, thank you

sl7rf · March 28, 2022, 2:55am

TF 2.8 and python 3.8
I have the same issue when serving the model online, the error seems to show up for every model.predict() call.
My batch size and input sample is always 1, I dont think batch_size is the problem here.
Also I could not reproduce this warning in my local.

Syed_Nahin_Hossain · June 28, 2022, 2:46am

I am facing the same problem. My batch size and input sample is always 1, I don’t think batch size is the problem here.

Topspap · September 22, 2022, 11:38am

I’m having similar problem, with this Error:
“W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled”

My batch is not larger than the whole dataset
The exact code are taken from a YouTube tutorial:

you can get the code to reproduce the error from here (Although in the Video tutorial he didn’t get that error):

So what is the problem is it because i’m using a newer version of Tensorflow ?

Note:
i’m using Python 3.9 | Tensorflow 2.10.0 | Tensorflow-gpu 2.10.0 | keras 2.10.0 | Window 10

Timo · October 4, 2022, 9:21am

Hello I’m getting the same error with TF 2.7 (Python 3.9) when making predictions with the trained model.

My data is grouped in cycles and interestingly I only get the error with some cycles in a - what seems to be random - pattern. Batchsize is always 1 in my usecase:

INFO Processed cycle: 3/20
INFO Processed cycle: 4/20
2022-10-04 11:05:41.750979: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:41.869399: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:41.892579: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:46.490775: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:46.540553: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:46.587145: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:46.610908: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:46.636638: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:46.660311: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:46.755783: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:50.348900: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:50.466292: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:50.487418: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:50.510366: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:50.600544: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:56.583640: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:56.611070: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:56.689325: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:56.715945: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.150230: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.391989: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.466320: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.532687: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.594002: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.636137: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.660478: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.785125: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.836864: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:13.186727: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:13.292232: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:13.497294: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:31.738377: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:34.954275: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:38.405368: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:07:01.726403: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:07:01.776447: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
INFO Processed cycle: 5/20
2022-10-04 11:09:01.265304: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
INFO Processed cycle: 6/20
2022-10-04 11:10:16.427091: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:11:04.562144: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
INFO Processed cycle: 10/20

Edit: I restarted the script and got this result:

INFO Processed cycle: 3/20
INFO Processed cycle: 4/20
INFO Processed cycle: 5/20
INFO Processed cycle: 6/20
INFO Processed cycle: 10/20
2022-10-04 11:32:09.333830: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled

Also updating to TF 2.9 or 2.10 increases the amount of warnings.

James_Pung · November 10, 2022, 2:38am

I have the same issue, solved by adding

gpus = tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(gpu, True)

willnguyen22 · December 6, 2022, 9:21am

Have you tried
model.predict(input_data, training=False)

toncho11 · December 19, 2022, 3:47am

This solution worked for me!

ddavo · January 2, 2023, 4:03am

Tried it and I have the same problem. I’m solving the Lunar Lander problem from OpenAI’s GYM using a DQN approach in Kaggle. The problem appears both with GPU (P100) and without it

Timo · February 1, 2023, 11:09am

Added your lines and had no more problem since. Thank you!

Cesc_Zhong · March 8, 2023, 12:00pm

You are really a genius!

Pratik_Shah · April 11, 2023, 11:01am

Hi… I am facing the same issue when solving Lunar Lander… were you able to solve it?

ddavo · April 11, 2023, 3:32pm

I just ignored the warning

Ziwei_Yang · June 5, 2023, 3:34am

Hi James,

I have exactly same warning message for every training step by using tf.distributed.MirroredStrategy. Even adding your snippet of code, it still couldn’t be fixed. Do you have any idea where the issue could be?