Optimization loop failed: Cancelled: Operation was cancelled

I am getting the following error when training a custom model:

W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled

Has anyone seen this error before? My code is a bit of a mess right now so hard to share any specifics. So I am not looking for a concrete solution, just some pointers on what could cause this so I can debug further. Right now I don’t even know what could possibly cause this error/warning.

1 Like

Generally I suggest you to use tags on new thread cause specialized technical team members could be subscribed only on tag subset (e.g. in this case tfdata and help_request tags).

1 Like

I am getting the same warning. For me, this warning occurs frequently, but not on every training step.

Using the code below, I fit the model and print the shape of input data and target data.

    print(states.shape, targets_full.shape)
    self.model.fit(states, targets_full, epochs=1, verbose=1)

The output shows that the shapes are always the same, yet sometimes I get the warning and sometimes I do not. A random snippet of the output is pasted below.

(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2445
2021-06-15 13:02:19.379526: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1214
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1480
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2558
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1036
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1048
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2647
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1280
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1074
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2401
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1160
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1348
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2424
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1197
2021-06-15 13:02:20.594590: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1415
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2358
2021-06-15 13:02:20.790803: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1212
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1649
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2220
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1064
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1488
2021-06-15 13:02:21.299211: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled

Any help is understanding or resolving this warning would be appreciated.

1 Like

Hi , I am also facing the same issue. Any solution ?

What TF version are you using?

Have you checked:

I was using TF version 2.5 and have just upgraded to 2.6 where I am still getting the same issue. I am also using Python 3.6.

After I upgraded to TF 2.6, I am still getting the error, but the path is different.

In 2.6:
“W tensorflow/core/data/root_dataset.cc:167] Optimization loop failed: Cancelled: Operation was cancelled”

Whereas in 2.5:
“W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled”


I looked through that tread, but I wasn’t able to solve the problem. The error is that tread also has a different path:
“W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled”

Any help would be appreciated.

2 Likes

I was able to fix this, the reason why it was throwing an error was my batch_size was set greater than the size of the whole dataset.

1 Like

This might be the case.
I encountered the same problem. As stated by @Farquhar13 , this warning information doesn’t always show up so this means the size of the batch should be fine? Please correct me if I am wrong.
Thanks!

I’m getting this same error.

2021-12-13 17:59:55.175834: W tensorflow/core/data/root_dataset.cc:163] Optimization loop failed: CANCELLED: Operation was cancelled

I know that my problem isn’t the batch size being larger than my dataset. I checked the size of the dataset each time I got the error, and it’s the same size as when I don’t get the error. Speaking of which, I don’t get this error every time I run the code. It seems almost random, but it obviously can’t be.

I can’t find a solution to this anywhere, so I’m hoping someone here can help me out.

1 Like

I am running into a similar issue. I’m running TS v2.7 and python 3.7.4
I’ve checked the shapes passing and the batch size as referred to above and those are not the issues. However I get it on every iteration.

Any help would be appreciated!

I have the same problem as you, how did you solve it, I hope you can help me, thank you

TF 2.8 and python 3.8
I have the same issue when serving the model online, the error seems to show up for every model.predict() call.
My batch size and input sample is always 1, I dont think batch_size is the problem here.
Also I could not reproduce this warning in my local.

I am facing the same problem. My batch size and input sample is always 1, I don’t think batch size is the problem here.