Custom model trigerring retracing warning

General Discussion
#1

Hello!

Can someone please advise me how can I get rid of this warning received while training a custom model?
If I train several times a custom model with the same data, I receive the warning below once in a while.
It is strange also that the message says that only in 6 out of 8 tries it encountered a problem, although I use exactly the same data every time.
I use only tensorflow objects so I do not know what generates this message.

“WARNING:tensorflow:6 out of the last 8 calls to <function Model.make_predict_function..predict_function at 0x7f90184d5310> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to Better performance with tf.function  |  TensorFlow Core and tf.function  |  TensorFlow Core v2.8.0 for more details.”

Thank you,

#2

Are you using tf.data API?

#4

Training data is in ndarray format.

#5

What is your TF version?

#7

TF version Is: 2.5.0

#8

Do you have a Colab to reproduce It?

#10

same problem with 2.5.0

#12

hi, as the warning message says, you are decorating some code using tf.function that is leading parts of the computation getting traced multiple times (for example you are decorating the call of a layer, data transformation, etc). (its just a warning telling you that its not convenient to do that in terms of performance since tracing is expensive) , if you want to decorate functions with tf.function make sure you have it either outside of the loop or decorating an expensive operation (for example an update step function where tf.GradientTape is used).

one example of bad usage that happened to me when tf 2.0 came out: you have a custom layer where the call is decorated with tf.function… then the message will appear

#13

Hi, it is a custom layer but I am not using tf.function at all.

#15

have the the same issue here, wondering if you find a way to solve the problem