Retrain TF-Agents model with changed parameters (Checkpointer or PolicySaver)

General Discussion
#1

Hello everyone!

I am trying to load a deep q model and retrain it with different parameters (e.g. changed epsilon value). I am using a config file for setting various parameters of the environment and the agent.
What is the best way to go about this? Should i use Checkpointer, PolicySaver or something else entirely?

I would be very grateful for your help!
Thank you!

2 Likes
#2

I’d have to experiment with this too. Since Checkpointer allows you to save and load not just the policy network state, but also the training state, I’d give that a go. In addition, it seems like it also caches into a replay buffer, which would be useful for sampling in DQN’s case. (tf_agents.utils.common.Checkpointer  |  TensorFlow Agents)

Hope this helps a little. cc @yablak @markdaoust

1 Like
#3

@Sarah_Riedmann, spoke with tensorflow-agents team member and they advised PolicySaver - it’s TF-Agents specific for saving a policy (with the step, and other info etc). PolicySaver uses Checkpointer underneath. Hope this helps!

1 Like
#4

Thank you @8bitmp3! I will experiment with PolicySaver.

1 Like
#5

I have tried using PolicySaver and it works so far. However, I would like to set the loaded policy as the DQN agent’s policy. Is it possible to continue using the agent with the saved policy or do i have to use loaded_policy.action() manually?

Any help would be appreciated! Thank you.

1 Like