From the documentation:

Bool. Defaults to `False`

. If `True`

, this `Model`

's logic will not be wrapped in a `tf.function`

. Recommended to leave this as `None`

unless your `Model`

cannot be run inside a `tf.function`

. `run_eagerly=True`

is not supported when using `tf.distribute.experimental.ParameterServerStrategy`

.

What is the significance of being wrapped in tf.function and what are the practical advantage/disadvantages of setting run_eagerly = True?

2 Likes

When something is wrapped inside `tf.function`

it has the advantage of being run in graph mode. All the backend compilation engineering is handled by TensorFlow itself in this case. The advantage is when all the operations are available as a graph we know much resources to allocate and how to best optimize the graph with the available resources. For more details refer to the following:

`run_eagerly=True`

lets figure out what exactly is going inside your model training loop. Letās say you have implemented a custom loop and put that inside the `train_step()`

method of a subclasses model. Setting `run_eagerly`

to `True`

will help you debug that loop if anything goes wrong. For practical applications of this, refer to the following guide:

3 Likes

Thanks very much! From the second link,

Thankfully, thereās an easy way to run your code in ādebug modeā, fully eagerly: pass `run_eagerly=True`

to `compile()`

. Your call to `fit()`

will now get executed line by line, without any optimization. Itās slower, but it makes it possible to print the value of intermediate tensors, or to use a Python debugger. Great for debugging.

does this mean that when run_eagerly = False, values of intermediate tensors in the train_step() cannot be saved? If I save some intermediate tensor(s) as an instance variable in a subclassed Model object, can I extract the values of these tensors after .fit() is complete as say a numpy array?

1 Like

Could you provide an example use-case?

1 Like

I slightly modified the first example from this tutorial by saving the y_pred from each epoch

```
class CustomModel(keras.Model):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.saved_pred = []
def train_step(self, data):
x, y = data
with tf.GradientTape() as tape:
y_pred = self(x, training=True) # Forward pass
loss = self.compiled_loss(y, y_pred, regularization_losses=self.losses)
# Compute gradients
trainable_vars = self.trainable_variables
gradients = tape.gradient(loss, trainable_vars)
# Update weights
self.optimizer.apply_gradients(zip(gradients, trainable_vars))
# Update metrics (includes the metric that tracks the loss)
self.compiled_metrics.update_state(y, y_pred)
# Return a dict mapping metric names to current value
self.saved_pred.append(y_pred)
return {m.name: m.result() for m in self.metrics}
```

import numpy as np

inputs = keras.Input(shape=(32,))

outputs = keras.layers.Dense(1)(inputs)

model = CustomModel(inputs = inputs, outputs = outputs)

model.compile(optimizer=āadamā, loss=āmseā, metrics=[āmaeā])

x = np.random.random((1000, 32))

y = np.random.random((1000, 1))

model.fit(x, y, epochs=3)

Hereās the output of model.saved_preds:

ListWrapper([<tf.Tensor ācustom_model_1/dense_5/BiasAdd:0ā shape=(None, 1) dtype=float32>, <tf.Tensor ācustom_model_1/dense_5/BiasAdd:0ā shape=(None, 1) dtype=float32>])

Thereās no numpy attribute for the Tensors in this list, so Iām wondering if itās possible to extract their values.

1 Like

Looks like a list. Can you do something like `model.saved_preds[0].numpy()`

? The model is supposed to be returning predictions i.e. some distribution or a set of values that must not contain any operations like `custom_model_1/dense_5/BiasAdd:0`

.

2 Likes

When trying .numpy() and tf.keras.backend.get_values(), I got an error saying that Tensor object has no attribute numpy. I also tried .eval which gave this error:

ValueError: Cannot use the given session to evaluate tensor: the tensorās graph is different from the sessionās graph.

1 Like

Since you are running with `run_eagerly=True`

now, you can print `self.saved_preds`

inside your `train_step()`

function directly for a better debugging experience. Could you do while calling `.fit()`

which I suppose you are doing currently.

2 Likes

But if I want to set `run_eagerly = False`

(for deployment), then thereās no way to extract the value of the Tensors from `self.saved_preds`

?

1 Like

Why would you wanna do that when you can simply call `.predict()`

or even the `model(x)`

directly? Just trying to better understand the situation here.

2 Likes

Sorry I gave that as a simplified example. I would like to see if itās possible to save intermediate Tensor results in the `train_step()`

, with `y_pred`

being a stand in for any arbitrary Tensor that was computed inside `train_step()`

in a subclassed Model class.

1 Like

Yeah, it should be possible. I have been able to save entire models with `train_step()`

. Hereās one such example:

2 Likes