Urgent please help for training (deeplapv3+)

General Discussion
,
#1

We were working on this code in colab notebook

, we changed the loss function to binary-cross intropy because the original loss function was giving us a nan value for the (loss). But unfortunately the training cell gave us this error:

InvalidArgumentError: Graph execution error:

2 root error(s) found.
(0) INVALID_ARGUMENT: Input is empty.
[[{{node DecodePng_1}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_2]]
(1) INVALID_ARGUMENT: Input is empty.
[[{{node DecodePng_1}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored. [Op:__inference_test_function_41271]

We don’t know how to deal with this kind of error :frowning: can anyone please help us ? :cry:

Screen Shot 2022-03-09 at 7.22.33 PM
Screen Shot 2022-03-09 at 7.22.33 PM1920×968 102 KB

#3

Changing the loss function doesn’t work. To avoid nan loss ensure :

  1. Check that your training data is properly scaled and doesn’t contain nans
  2. Check that you are using the right optimizer and that your learning rate is not too large
  3. Check if you are facing the exploding gradient problem use gradient clipping

Thank you