Question about advantage calculation

[leeacord]

Hi,

I have a question regarding the implementation of the advantage calculation. The code snippet is as follows:

dreamerv2/dreamerv2/agent.py

Lines 252 to 274 in 07d906e

	def actor_loss(self, seq, target):
	# Actions: 0 [a1] [a2] a3
	# ^ | ^ | ^ |
	# / v / v / v
	# States: [z0]->[z1]-> z2 -> z3
	# Targets: t0 [t1] [t2]
	# Baselines: [v0] [v1] v2 v3
	# Entropies: [e1] [e2]
	# Weights: [ 1] [w1] w2 w3
	# Loss: l1 l2
	metrics = {}
	# Two states are lost at the end of the trajectory, one for the boostrap
	# value prediction and one because the corresponding action does not lead
	# anywhere anymore. One target is lost at the start of the trajectory
	# because the initial state comes from the replay buffer.
	policy = self.actor(tf.stop_gradient(seq['feat'][:-2]))
	if self.config.actor_grad == 'dynamics':
	objective = target[1:]
	elif self.config.actor_grad == 'reinforce':
	baseline = self._target_critic(seq['feat'][:-2]).mode()
	advantage = tf.stop_gradient(target[1:] - baseline)
	action = tf.stop_gradient(seq['action'][1:-1])
	objective = policy.log_prob(action) * advantage

advantage = tf.stop_gradient(target[1:] - self._target_critic(seq['feat'][:-2]).mode())

Based on my understanding:

• seq['feat'] contains time steps from 0 to horizon.
• target contains time steps from 0 to horizon-1, since the value at the last step is used as a bootstrap for lambda_return.
• Therefore, baseline in Line 271 includes time steps from 0 to horizon-2, and target[1:] includes time steps from 1 to horizon-1.

If I understand correctly, the code uses $V_{t+1}^{\lambda} - v_\xi\left(\hat{z}_t\right)$ to calculate the advantage,

not $V_t^{\lambda} - v_{\xi}(\hat{z}_t)$ as stated in the paper?

I am very impressed with the Dreamer series of reinforcement learning algorithms. Thank you for your hard work!