Function call stack: train_function -> train_function
2021-07-02 14:23:52.766840: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated. [[{{node PyFunc}}]]
model/dense/MatMul (defined at rnn_flickr_fit.py:273) ]] (1) Invalid argument: In[0] mismatch In[1] shape: 1108 vs. 1120: [42,1108] [1120,256] 0 0
I’m not sure about the details, but this line is trying to tell you that you have a dense later somewhere that is getting an in put with an unexpected shape.
Since keras layers size their weights based on the first input they see, you’re probably feeding an input with a variable sized axis to that Dense layer. Maybe the first batch was (42, 1120), and this second batch is(42, 1108)
It appears that this happened because your last batch in your epoch has 1108 (pictures, ect) in it where your model is expecting 1120.
This is what ’ markdaoust’ was trying to explain, but I think it most often happens at your last batch of your epoch, since if you don’t divide out the right number of item it will not naturally be the 1120. take items //batch_size to only get an integer.