Metrics related [predictions must be <= 1] error

raft · November 26, 2021, 5:33pm

Hello,

I have the below error with TensorFlow 2.7. Same error happens with 2.6 but stack trace is different.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\hakan\AppData\Roaming\Python\Python39\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Users\hakan\AppData\Roaming\Python\Python39\site-packages\tensorflow\python\eager\execute.py", line 58, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError:  assertion failed: [predictions must be <= 1] [Condition x <= y did not hold element-wise:] [x (sequential_4/output/BiasAdd:0) = ] [[1.00585222][1.00123906][0.880351603]...] [y (Cast_3/x:0) = ] [1]
         [[node assert_less_equal/Assert/AssertGuard/Assert
 (defined at C:\Users\hakan\AppData\Roaming\Python\Python39\site-packages\keras\utils\metrics_utils.py:612)
]] [Op:__inference_train_function_29922]

Errors may have originated from an input operation.
Input Source operations connected to node assert_less_equal/Assert/AssertGuard/Assert:
In[0] assert_less_equal/Assert/AssertGuard/Assert/assert_less_equal/All:
In[1] assert_less_equal/Assert/AssertGuard/Assert/data_0:
In[2] assert_less_equal/Assert/AssertGuard/Assert/data_1:
In[3] assert_less_equal/Assert/AssertGuard/Assert/data_2:
In[4] assert_less_equal/Assert/AssertGuard/Assert/sequential_4/output/BiasAdd:
In[5] assert_less_equal/Assert/AssertGuard/Assert/data_4:
In[6] assert_less_equal/Assert/AssertGuard/Assert/Cast_3/x:

Operation defined at: (most recent call last)
>>>   File "<stdin>", line 1, in <module>
>>>
>>>   File "C:\Users\hakan\AppData\Roaming\Python\Python39\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>>
>>>   File "C:\Users\hakan\AppData\Roaming\Python\Python39\site-packages\keras\engine\training.py", line 1216, in fit
>>>     tmp_logs = self.train_function(iterator)
>>>
>>>   File "C:\Users\hakan\AppData\Roaming\Python\Python39\site-packages\keras\engine\training.py", line 878, in train_function
>>>     return step_function(self, iterator)
>>>
>>>   File "C:\Users\hakan\AppData\Roaming\Python\Python39\site-packages\keras\engine\training.py", line 867, in step_function
>>>     outputs = model.distribute_strategy.run(run_step, args=(data,))
>>>
>>>   File "C:\Users\hakan\AppData\Roaming\Python\Python39\site-packages\keras\engine\training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>>
>>>   File "C:\Users\hakan\AppData\Roaming\Python\Python39\site-packages\keras\engine\training.py", line 817, in train_step
>>>     self.compiled_metrics.update_state(y, y_pred, sample_weight)
>>>
>>>   File "C:\Users\hakan\AppData\Roaming\Python\Python39\site-packages\keras\engine\compile_utils.py", line 460, in update_state
>>>     metric_obj.update_state(y_t, y_p, sample_weight=mask)
>>>
>>>   File "C:\Users\hakan\AppData\Roaming\Python\Python39\site-packages\keras\utils\metrics_utils.py", line 73, in decorated
>>>     update_op = update_state_fn(*args, **kwargs)
>>>
>>>   File "C:\Users\hakan\AppData\Roaming\Python\Python39\site-packages\keras\metrics.py", line 177, in update_state_fn
>>>     return ag_update_state(*args, **kwargs)
>>>
>>>   File "C:\Users\hakan\AppData\Roaming\Python\Python39\site-packages\keras\metrics.py", line 1069, in update_state
>>>     return metrics_utils.update_confusion_matrix_variables(
>>>
>>>   File "C:\Users\hakan\AppData\Roaming\Python\Python39\site-packages\keras\utils\metrics_utils.py", line 612, in update_confusion_matrix_variables
>>>     tf.compat.v1.assert_less_equal(
>>>

Function call stack:
train_function -> assert_less_equal_Assert_AssertGuard_false_28872

Interesting thing is, this only happens when using BinaryCrossentropy(from_logits=True) loss and with metrics other than BinaryAccuracy, for example Precision or AUC metrics.

In other words, with BinaryCrossentropy(from_logits=False) loss it always works with any metrics, with BinaryCrossentropy(from_logits=True) loss it only works with BinaryAccuracy metrics.

Below is the model:

model = tf.keras.models.Sequential([
  tf.keras.layers.Dense(150, activation='relu', name='hidden_1', input_shape=(x_train.shape[1],)),
  tf.keras.layers.Dense(50, activation='relu', name='hidden_2'),
  tf.keras.layers.Dense(1, activation=None, name='output')
])

model.compile(optimizer=tf.keras.optimizers.Adam(),
  loss=tf.keras.losses.BinaryCrossentropy(from_logits=True), 
  metrics=METRICS)

model.fit(x_train, y_train, epochs=20, validation_data=(x_val, y_val), callbacks=[tensorboard_callback], verbose=1)

Any clues for how to fix this?

Thanks
raft

Bhack · November 27, 2021, 1:39am

Is the the cofusion Matrix update expect predicted values between 0 and 1:

github.com

tensorflow/tensorflow/blob/master/tensorflow/python/keras/utils/metrics_utils.py#L620-L629

      
        
            with ops.control_dependencies([
                check_ops.assert_greater_equal(
                    y_pred,
                    math_ops.cast(0.0, dtype=y_pred.dtype),
                    message='predictions must be >= 0'),
                check_ops.assert_less_equal(
                    y_pred,
                    math_ops.cast(1.0, dtype=y_pred.dtype),
                    message='predictions must be <= 1')
            ]):

raft · November 27, 2021, 1:20pm

not sure what this means. I’m not doing anything with the confusion matrix, at least not consciously

Bhack · November 27, 2021, 2:56pm

at least not consciously

That function update_confusion_matrix_variables is called internally by some metrics.

E.g. Precision

github.com

tensorflow/tensorflow/blob/master/tensorflow/python/keras/metrics.py#L1343

      
        
              y_true: The ground truth values, with the same dimensions as `y_pred`.
                Will be cast to `bool`.
              y_pred: The predicted values. Each element must be in the range `[0, 1]`.
              sample_weight: Optional weighting of each example. Defaults to 1. Can be a
                `Tensor` whose rank is either 0, or the same rank as `y_true`, and must
                be broadcastable to `y_true`.
            
            
Returns:
              Update op.
            """
            return metrics_utils.update_confusion_matrix_variables(
                {
                    metrics_utils.ConfusionMatrix.TRUE_POSITIVES: self.true_positives,
                    metrics_utils.ConfusionMatrix.FALSE_POSITIVES: self.false_positives
                },
                y_true,
                y_pred,
                thresholds=self.thresholds,
                thresholds_distributed_evenly=self._thresholds_distributed_evenly,
                top_k=self.top_k,
                class_id=self.class_id,

E.g. AUC

github.com

tensorflow/tensorflow/blob/master/tensorflow/python/keras/metrics.py#L2262

      
        
            
            
# Only forward label_weights to update_confusion_matrix_variables when
            # multi_label is False. Otherwise the averaging of individual label AUCs is
            # handled in AUC.result
            label_weights = None if self.multi_label else self.label_weights
            
            
if self._from_logits:
              y_pred = activations.sigmoid(y_pred)
            
            
with ops.control_dependencies(deps):
              return metrics_utils.update_confusion_matrix_variables(
                  {
                      metrics_utils.ConfusionMatrix.TRUE_POSITIVES:
                          self.true_positives,
                      metrics_utils.ConfusionMatrix.TRUE_NEGATIVES:
                          self.true_negatives,
                      metrics_utils.ConfusionMatrix.FALSE_POSITIVES:
                          self.false_positives,
                      metrics_utils.ConfusionMatrix.FALSE_NEGATIVES:
                          self.false_negatives,
                  },

raft · November 27, 2021, 3:38pm

I see. Can we say than this is a bug in some metrics? Or/else how to fix this?

Bhack · November 27, 2021, 3:53pm

As you can see from examples in:
https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy

When you use the non default from_logits=True you prediction are out of the 0-1 range.

Bhack · November 27, 2021, 3:59pm

Add support for more keras.metrics.* to accept from_logits argument.

opened 08:53PM - 26 Feb 20 UTC

closed 07:34AM - 27 Mar 20 UTC

MeghnaNatraj

stat:awaiting response type:bug comp:keras TF 2.1

**System information** - Have I written custom code: Yes - OS Platform and Di…stribution: Linux - CUDA/cuDNN version: CUDA Version: 10.1 - GPU model and memory: GeForce GTX 1080 **Describe the current behavior** Currently, in [keras.metrics](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/), all the metrics for binary classification only accept values in the range [0, 1]. This is an issue as all the models in TF2.x tutorials output logits (ranges from [-inf, +inf]) and AVOID using an activation in the last layer such as `sigmoid` (ranges from [0, 1]) Current (preferred way): `keras.layers.Dense(1) # Output range is [-inf, +inf]` Before (deprecated): `keras.layers.Dense(1, activation='sigmoid') # Output range is [0,1]` Now, if my model outputs values in the range [-inf, +inf] as per the current preferred way, I CANNOT use any of the following metrics as they all expect values in the range [0, 1] ``` 1. keras.metrics.Precision(name='precision') 2. keras.metrics.BinaryAccuracy(name='accuracy') 3. keras.metrics.Recall(name='recall') 4. keras.metrics.AUC(name='auc') 5. keras.metrics.TruePositives(name='tp') 6. keras.metrics.FalsePositives(name='fp') 7. keras.metrics.TrueNegatives(name='tn') 8. keras.metrics.FalseNegatives(name='fn') ``` However, `tf.keras.metrics.BinaryCrossentropy` is the only metric that accepts both range [-inf, inf] and range[0,1] and this is used in all tutorials. The issue is that I want to use all the 8 metrics mentioned above during training, but I cannot. **Describe the expected behavior** Provide an option to have a `from_logits=True` argument in the 8 metrics given above (and other metrics as well) **Standalone code to reproduce the issue** Pip Install: TensorFlow 2.x and Numpy ``` ! pip install tensorflow numpy ``` **Code** ``` # INITIALIZE import tensorflow as tf import numpy as np # define dataset dataset = np.array([[6,148,72,35,0,33.6,0.627,50,1], [1,85,66,29,0,26.6,0.351,31,0], [8,183,64,0,0,23.3,0.672,32,1], [1,89,66,23,94,28.1,0.167,21,0], [0,137,40,35,168,43.1,2.288,33,1]]) # split into input (X) and output (y) variables X = dataset[:,0:8] y = dataset[:,8] ``` ``` # WORKS: Model output is in range [0, 1] # define the keras model model = tf.keras.models.Sequential([tf.keras.layers.Dense(12, input_dim=8, activation='relu'), tf.keras.layers.Dense(1, activation='sigmoid')]) # compile the keras model model.compile(loss=tf.keras.losses.BinaryCrossentropy(), optimizer=tf.keras.optimizers.RMSprop(), metrics=[tf.keras.metrics.Accuracy(name='accuracy'), tf.keras.metrics.Precision(name='precision'), tf.keras.metrics.TruePositives(name='tp')]) # fit the keras model on the dataset model.fit(X, y, epochs=10, batch_size=10, verbose=0) ``` ``` # DOES NOT WORK: Model output is in range [-inf, inf] # define the keras model model = tf.keras.models.Sequential([tf.keras.layers.Dense(12, input_dim=8, activation='relu'), tf.keras.layers.Dense(1)]) # compile the keras model model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True), optimizer=tf.keras.optimizers.RMSprop(), metrics=[tf.keras.metrics.Accuracy(name='accuracy'), tf.keras.metrics.Precision(name='precision'), tf.keras.metrics.TruePositives(name='tp')]) # fit the keras model on the dataset model.fit(X, y, epochs=10, batch_size=10, verbose=0) . . . . . . . InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: assertion failed: [predictions must be <= 1] [Condition x <= y did not hold element-wise:] [x (sequential_8/dense_17/BiasAdd:0) = ] [[22.0911655][62.7297783][35.2793274]...] [y (metrics/precision/Cast_3/x:0) = ] [1] [[{{node metrics/precision/assert_less_equal/Assert/AssertGuard/else/_11/Assert}}]] [[metrics/tp/assert_greater_equal/Assert/AssertGuard/pivot_f/_31/_61]] (1) Invalid argument: assertion failed: [predictions must be <= 1] [Condition x <= y did not hold element-wise:] [x (sequential_8/dense_17/BiasAdd:0) = ] [[22.0911655][62.7297783][35.2793274]...] [y (metrics/precision/Cast_3/x:0) = ] [1] [[{{node metrics/precision/assert_less_equal/Assert/AssertGuard/else/_11/Assert}}]] 0 successful operations. 0 derived errors ignored. [Op:__inference_distributed_function_12253] Function call stack: distributed_function -> distributed_function ```

github.com/tensorflow/tensorflow

Not every keras.metrics.* accept from_logits=True

opened 09:09AM - 10 Aug 20 UTC

closed 11:12AM - 04 Jun 21 UTC

adamp87

stat:awaiting tensorflower type:bug comp:keras TF 2.3

**System information** - Have I written custom code: Yes - OS Platform and Dis…tribution: Linux Ubuntu 18.04.3 - TensorFlow installed from: binary (Docker Image tensorflow:2.3.0-gpu) - TensorFlow version: v2.3.0-rc2-23-gb36436b087 2.3.0 - Python version: 3.6.9 - CUDA/cuDNN version: CUDA10.2 - GPU model and memory: RTX 2080 8GB **Describe the current behavior** The issue is the same as described by MeghnaNatraj in ticket [37103](https://github.com/tensorflow/tensorflow/issues/37103). The mentioned ticket states the problem has been solved. I was requested to open a new ticket. If I have a model with an output layer of one neuron and 'sigmoid' activation function and using BinaryCrossentropy(from_logits=False), then the model.fit() function accepts any metrics (e.g. Precision) If I have the same model as before but without activation function and BinaryCrossentropy(from_logits=True), then model.fit() function will return an error. **Describe the expected behavior** - Having a model, which has an output layer without an activation function: keras.layers.Dense(1) # Output range is [-inf, +inf] - Loss function of the model working with logits: BinaryCrossentropy(from_logits=True) - Accepting metrics during fitting like keras.metrics.Precision **Standalone code to reproduce the issue** I have copied the same test code by MeghnaNatraj in ticket [37103](https://github.com/tensorflow/tensorflow/issues/37103) ``` # INITIALIZE import tensorflow as tf import numpy as np # define dataset dataset = np.array([[6,148,72,35,0,33.6,0.627,50,1], [1,85,66,29,0,26.6,0.351,31,0], [8,183,64,0,0,23.3,0.672,32,1], [1,89,66,23,94,28.1,0.167,21,0], [0,137,40,35,168,43.1,2.288,33,1]]) # split into input (X) and output (y) variables X = dataset[:,0:8] y = dataset[:,8] ``` ``` # WORKS: Model output is in range [0, 1] # define the keras model model = tf.keras.models.Sequential([tf.keras.layers.Dense(12, input_dim=8, activation='relu'), tf.keras.layers.Dense(1, activation='sigmoid')]) # compile the keras model model.compile(loss=tf.keras.losses.BinaryCrossentropy(), optimizer=tf.keras.optimizers.RMSprop(), metrics=[tf.keras.metrics.Accuracy(name='accuracy'), tf.keras.metrics.Precision(name='precision'), tf.keras.metrics.TruePositives(name='tp')]) # fit the keras model on the dataset model.fit(X, y, epochs=10, batch_size=10, verbose=0) ``` ``` # DOES NOT WORK: Model output is in range [-inf, inf] # define the keras model model = tf.keras.models.Sequential([tf.keras.layers.Dense(12, input_dim=8, activation='relu'), tf.keras.layers.Dense(1)]) # compile the keras model model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True), optimizer=tf.keras.optimizers.RMSprop(), metrics=[tf.keras.metrics.Accuracy(name='accuracy'), tf.keras.metrics.Precision(name='precision'), tf.keras.metrics.TruePositives(name='tp')]) # fit the keras model on the dataset model.fit(X, y, epochs=10, batch_size=10, verbose=0) InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: assertion failed: [predictions must be <= 1] [Condition x <= y did not hold element-wise:] [x (sequential_8/dense_17/BiasAdd:0) = ] [[22.0911655][62.7297783][35.2793274]...] [y (metrics/precision/Cast_3/x:0) = ] [1] [[{{node metrics/precision/assert_less_equal/Assert/AssertGuard/else/_11/Assert}}]] [[metrics/tp/assert_greater_equal/Assert/AssertGuard/pivot_f/_31/_61]] (1) Invalid argument: assertion failed: [predictions must be <= 1] [Condition x <= y did not hold element-wise:] [x (sequential_8/dense_17/BiasAdd:0) = ] [[22.0911655][62.7297783][35.2793274]...] [y (metrics/precision/Cast_3/x:0) = ] [1] [[{{node metrics/precision/assert_less_equal/Assert/AssertGuard/else/_11/Assert}}]] 0 successful operations. 0 derived errors ignored. [Op:__inference_distributed_function_12253] Function call stack: distributed_function -> distributed_function Traceback (most recent call last): File "<stdin>", line 2, in <module> File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 1098, in fit tmp_logs = train_function(iterator) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 780, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 840, in _call return self._stateless_fn(*args, **kwds) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2829, in __call__ return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1848, in _filtered_call cancellation_manager=cancellation_manager) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat ctx, args, cancellation_manager=cancellation_manager)) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 550, in call ctx=ctx) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute inputs, attrs, num_outputs) tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: assertion failed: [predictions must be <= 1] [Condition x <= y did not hold element-wise:] [x (sequential/dense_1/BiasAdd:0) = ] [[28.8497276][37.4658966][48.6209221]...] [y (Cast_5/x:0) = ] [1] [[{{node assert_less_equal/Assert/AssertGuard/else/_11/assert_less_equal/Assert/AssertGuard/Assert}}]] [[assert_less_equal_1/Assert/AssertGuard/pivot_f/_41/_81]] (1) Invalid argument: assertion failed: [predictions must be <= 1] [Condition x <= y did not hold element-wise:] [x (sequential/dense_1/BiasAdd:0) = ] [[28.8497276][37.4658966][48.6209221]...] [y (Cast_5/x:0) = ] [1] [[{{node assert_less_equal/Assert/AssertGuard/else/_11/assert_less_equal/Assert/AssertGuard/Assert}}]] 0 successful operations. 0 derived errors ignored. [Op:__inference_train_function_1165] ```

raft · November 27, 2021, 4:31pm

Thanks @Bhack!

I guess this is my workaround solution for now:
Not every keras.metrics.* accept from_logits=True · Issue #42182 · tensorflow/tensorflow · GitHub.

I will call this a bug since BinaryCrossentropy suggests using from_logits=True.

In the issue you had posted, they state this is fixed but I guess this is not the case. I will post there.

raft · November 27, 2021, 7:53pm

I confirm, the mentioned workaround effectively solved my issue:

github.com/tensorflow/tensorflow

Not every keras.metrics.* accept from_logits=True

opened 09:09AM - 10 Aug 20 UTC

closed 11:12AM - 04 Jun 21 UTC

adamp87

stat:awaiting tensorflower type:bug comp:keras TF 2.3

**System information** - Have I written custom code: Yes - OS Platform and Dis…tribution: Linux Ubuntu 18.04.3 - TensorFlow installed from: binary (Docker Image tensorflow:2.3.0-gpu) - TensorFlow version: v2.3.0-rc2-23-gb36436b087 2.3.0 - Python version: 3.6.9 - CUDA/cuDNN version: CUDA10.2 - GPU model and memory: RTX 2080 8GB **Describe the current behavior** The issue is the same as described by MeghnaNatraj in ticket [37103](https://github.com/tensorflow/tensorflow/issues/37103). The mentioned ticket states the problem has been solved. I was requested to open a new ticket. If I have a model with an output layer of one neuron and 'sigmoid' activation function and using BinaryCrossentropy(from_logits=False), then the model.fit() function accepts any metrics (e.g. Precision) If I have the same model as before but without activation function and BinaryCrossentropy(from_logits=True), then model.fit() function will return an error. **Describe the expected behavior** - Having a model, which has an output layer without an activation function: keras.layers.Dense(1) # Output range is [-inf, +inf] - Loss function of the model working with logits: BinaryCrossentropy(from_logits=True) - Accepting metrics during fitting like keras.metrics.Precision **Standalone code to reproduce the issue** I have copied the same test code by MeghnaNatraj in ticket [37103](https://github.com/tensorflow/tensorflow/issues/37103) ``` # INITIALIZE import tensorflow as tf import numpy as np # define dataset dataset = np.array([[6,148,72,35,0,33.6,0.627,50,1], [1,85,66,29,0,26.6,0.351,31,0], [8,183,64,0,0,23.3,0.672,32,1], [1,89,66,23,94,28.1,0.167,21,0], [0,137,40,35,168,43.1,2.288,33,1]]) # split into input (X) and output (y) variables X = dataset[:,0:8] y = dataset[:,8] ``` ``` # WORKS: Model output is in range [0, 1] # define the keras model model = tf.keras.models.Sequential([tf.keras.layers.Dense(12, input_dim=8, activation='relu'), tf.keras.layers.Dense(1, activation='sigmoid')]) # compile the keras model model.compile(loss=tf.keras.losses.BinaryCrossentropy(), optimizer=tf.keras.optimizers.RMSprop(), metrics=[tf.keras.metrics.Accuracy(name='accuracy'), tf.keras.metrics.Precision(name='precision'), tf.keras.metrics.TruePositives(name='tp')]) # fit the keras model on the dataset model.fit(X, y, epochs=10, batch_size=10, verbose=0) ``` ``` # DOES NOT WORK: Model output is in range [-inf, inf] # define the keras model model = tf.keras.models.Sequential([tf.keras.layers.Dense(12, input_dim=8, activation='relu'), tf.keras.layers.Dense(1)]) # compile the keras model model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True), optimizer=tf.keras.optimizers.RMSprop(), metrics=[tf.keras.metrics.Accuracy(name='accuracy'), tf.keras.metrics.Precision(name='precision'), tf.keras.metrics.TruePositives(name='tp')]) # fit the keras model on the dataset model.fit(X, y, epochs=10, batch_size=10, verbose=0) InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: assertion failed: [predictions must be <= 1] [Condition x <= y did not hold element-wise:] [x (sequential_8/dense_17/BiasAdd:0) = ] [[22.0911655][62.7297783][35.2793274]...] [y (metrics/precision/Cast_3/x:0) = ] [1] [[{{node metrics/precision/assert_less_equal/Assert/AssertGuard/else/_11/Assert}}]] [[metrics/tp/assert_greater_equal/Assert/AssertGuard/pivot_f/_31/_61]] (1) Invalid argument: assertion failed: [predictions must be <= 1] [Condition x <= y did not hold element-wise:] [x (sequential_8/dense_17/BiasAdd:0) = ] [[22.0911655][62.7297783][35.2793274]...] [y (metrics/precision/Cast_3/x:0) = ] [1] [[{{node metrics/precision/assert_less_equal/Assert/AssertGuard/else/_11/Assert}}]] 0 successful operations. 0 derived errors ignored. [Op:__inference_distributed_function_12253] Function call stack: distributed_function -> distributed_function Traceback (most recent call last): File "<stdin>", line 2, in <module> File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 1098, in fit tmp_logs = train_function(iterator) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 780, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 840, in _call return self._stateless_fn(*args, **kwds) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2829, in __call__ return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1848, in _filtered_call cancellation_manager=cancellation_manager) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat ctx, args, cancellation_manager=cancellation_manager)) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 550, in call ctx=ctx) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute inputs, attrs, num_outputs) tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: assertion failed: [predictions must be <= 1] [Condition x <= y did not hold element-wise:] [x (sequential/dense_1/BiasAdd:0) = ] [[28.8497276][37.4658966][48.6209221]...] [y (Cast_5/x:0) = ] [1] [[{{node assert_less_equal/Assert/AssertGuard/else/_11/assert_less_equal/Assert/AssertGuard/Assert}}]] [[assert_less_equal_1/Assert/AssertGuard/pivot_f/_41/_81]] (1) Invalid argument: assertion failed: [predictions must be <= 1] [Condition x <= y did not hold element-wise:] [x (sequential/dense_1/BiasAdd:0) = ] [[28.8497276][37.4658966][48.6209221]...] [y (Cast_5/x:0) = ] [1] [[{{node assert_less_equal/Assert/AssertGuard/else/_11/assert_less_equal/Assert/AssertGuard/Assert}}]] 0 successful operations. 0 derived errors ignored. [Op:__inference_train_function_1165] ```