I am trying to train on TPU using data from tensorflow_datasets by enalbing try_gcs=True. The training happens but the output logs are not in the format I desire. I have attached my notebook for your reference. And this is the format output I am getting
Exception ignored in: <function Executor.__del__ at 0x7f9ef205ff80> Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 46, in __del__ self.wait() File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 65, in wait pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle) tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence Exception ignored in: <function Executor.__del__ at 0x7f9ef205ff80> Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 46, in __del__ self.wait() File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 65, in wait pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle) tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence Exception ignored in: <function Executor.__del__ at 0x7f9ef205ff80> Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 46, in __del__ self.wait() File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 65, in wait pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle) tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence Exception ignored in: <function Executor.__del__ at 0x7f9ef205ff80> Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 46, in __del__ self.wait() File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 65, in wait pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle) tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence Exception ignored in: <function Executor.__del__ at 0x7f9ef205ff80> Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 46, in __del__ self.wait() File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 65, in wait pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle) tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence Exception ignored in: <function Executor.__del__ at 0x7f9ef205ff80> Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 46, in __del__ self.wait() File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 65, in wait pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle) tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence Exception ignored in: <function Executor.__del__ at 0x7f9ef205ff80> Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 46, in __del__ self.wait() File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 65, in wait pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle) tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence Exception ignored in: <function Executor.__del__ at 0x7f9ef205ff80> Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 46, in __del__ self.wait() File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 65, in wait pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle) tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence Exception ignored in: <function Executor.__del__ at 0x7f9ef205ff80> Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 46, in __del__ self.wait() File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 65, in wait pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle) tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence Epoch: 10 Loss: 0.007660247851163149 Accuracy: 0.9980666637420654 Exception ignored in: <function Executor.__del__ at 0x7f9ef205ff80> Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 46, in __del__ self.wait() File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/executor.py", line 65, in wait pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle) tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
Solved it. I had to add make sure the TPU had data to process at all times by adding repeat() to my data pipeline.
1 Like