How to implement a soft update for targetnetworks in tf?

General Discussion
,
#1

I am following a tutorial from a youtuber where the soft update of the target networks are perfromed by this function:

    def update_network_parameters(self, tau=None):
        if tau is None:
            tau = self.tau

        weights = []
        targets = self.target_actor.weights
        for i, weight in enumerate(self.actor.weights):
            weights.append(weight * tau + targets[i]*(1-tau))
        self.target_actor.set_weights(weights)

        weights = []
        targets = self.target_critic.weights
        for i, weight in enumerate(self.critic.weights):
            weights.append(weight * tau + targets[i]*(1-tau))
        self.target_critic.set_weights(weights)

But idk if that’s the proper way. What if there are NoisyDense layers or Normalization layers? Are they updated also using this approach? Is there somewhere a proper implementation? Maybe its already somewhere in the tf lib and idk it…

#3

Hi @Unnamed, welcome to the Forum.
yes, please feel free to have a look at the Soft Actor Critic agent tf_agents.agents.SacAgent
There’s a target_update_tau factor (+ many more) for soft updates of the target network.

Also used in this colab tutorial:

Lucky Rewards,
Dennis

1 Like