- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies value mismatch when a shared actor trains across environments whose conditional returns differ for the same critic input. It compares zero, shared-value, and environment-conditional oracle baselines in deterministic bandits with a common optimal arm. The analysis establishes common convergence while distinguishing eventual samplewise improvement from recurrent drawdowns. The paper analyzes and evaluates established conditioning designs that provide a categorical environment identifier only to the critic, using FiLM or multiple value heads. PPO experiments cover controlled CartPole, mass-varied MuJoCo, BipedalWalker, and Procgen. Under the specified fixed, mixed-difficulty Procgen training protocol, the reported multihead aggregate normalized return on unseen levels improves by 40.8% over the shared critic.
Critical strengths
- The theoretical analysis provides a useful distinction between expected policy updates and realized learning paths. Propositions 1–3 show that the methods converge to the same optimum but differ in whether individual updates eventually improve the policy or continue to produce vanishing drawdowns. Appendix C further separates a variance-based improvement bound from actual finite-horizon behavior. Together, these results help explain how the choice of baseline affects learning.
- The experiments provide controlled evidence and practical support for conditioning the critic while retaining a shared actor. Figures 3–5 include identical and heterogeneous environments, constant-index and scalar-bias controls, and a capacity control. These comparisons help assess the contribution of conditioning. Table 1 also reports higher unseen-level test means for multihead on 14 of 16 Procgen games.
Critical weaknesses
- The methodological increment over multitask PopArt is limited, and the distinction in problem scope needs stronger justification. PopArt already combine a shared policy that does not directly receive the task index with task-indexed value heads. Their introduction also identifies reward sparsity and agent competence as sources of return-scale differences. Treating levels as tasks places the paper's multihead intervention within this existing design. Section 5 acknowledges the architectural precedent, but comparable reward scales alone do not establish a clear separation from the earlier setting. Propositions 1–3 provide distinct theoretical results. Nevertheless, I find the additional understanding established beyond the existing design insufficiently compelling for acceptance.
- The explanation for PopArt's lower aggregate performance lacks direct scale diagnostics and leaves open the possibility that the training setup limits its effectiveness. First, Section 4.5 relates the reduction in aggregate improvement from 40.8% to 33.3% in Table 1 to matched reward scales, but does not report per-level value-target scale measurements to support this premise. This matters because differences in difficulty, reward sparsity, and policy competence can produce different return scales even under the same reward convention. Second, the statistical update schedule provides a possible alternative explanation. The implementation notes in PopArt describe updates to normalization statistics per rollout. In contrast, this paper uses per-minibatch updates and three PPO epochs, as specified in G.1 and Table 6. It is unclear whether the same targets repeatedly enter the statistics across epochs. If they do, repeated statistical updates during optimization could affect PopArt's performance and contribute to the observed aggregate decline. Taken together, the missing scale measurements and the possible interaction with the training setup leave the attribution to comparable scales insufficiently supported.
Questions for authors
- Please clarify how differences in value-target scales relate to the benefit of PopArt in Table 1. Could you add a compact summary of the per-level target means and standard deviations at a specified existing checkpoint?
- Please test whether aligning the normalization update schedule with the original PopArt implementation changes the comparison. PopArt updates statistics per rollout, while this paper uses minibatches and multiple PPO epochs on the same data. To assess whether this difference contributes to the Table 1 result, could you run a controlled comparison on a small set of representative tasks, updating statistics once per newly collected rollout and holding them fixed during the subsequent PPO optimization epochs? This would align the statistical update unit with the original method while retaining the PPO training setup. If the performance difference persists, it would help rule out this scheduling difference as an explanation.
5 comments
Reply to Reviewer Y6Jg (2/3): Target scales and normalization updates (Weakness 2; Questions 1–2)
We added both requested checks: per level target statistics at existing checkpoints and a comparison that updates normalization statistics once per new rollout, holding them fixed during PPO optimization.
Target scale diagnostics. We evaluate MH (multihead) and MH+PA (PopArt normalization) at approximately 25M steps on four games, with five training seeds per game and variant. We freeze the actor, critic, PopArt buffers and saved global reward-normalization statistics, then collect 10 fresh rollouts, yielding 2,560 GAE targets per level across 200 training levels. Targets are measured before PopArt normalization, in each checkpoint’s frozen global reward normalization units. Each entry reports p10 / median / p90 across levels, taking the median of each quantile across seeds.
| Game | Critic | Level target mean | Level target SD |
|---|---|---|---|
| CaveFlyer | MH | 0.005 / 4.40 / 9.64 | 0.021 / 0.34 / 1.53 |
| CaveFlyer | +PA | 0.001 / 1.21 / 9.73 | 0.004 / 0.33 / 1.51 |
| CoinRun | MH | 5.92 / 7.70 / 7.94 | 0.067 / 0.22 / 1.26 |
| CoinRun | +PA | 5.17 / 7.91 / 8.07 | 0.051 / 0.22 / 1.37 |
| BigFish | MH | 0.041 / 2.03 / 8.77 | 0.092 / 1.00 / 2.48 |
| BigFish | +PA | 0.010 / 1.96 / 11.18 | 0.011 / 0.93 / 2.36 |
| Maze | MH | -0.026 / 2.94 / 2.97 | 0.002 / 0.011 / 0.038 |
| Maze | +PA | -0.001 / 3.61 / 3.66 | 0.001 / 0.007 / 0.084 |
Targets differ across levels within the common reward convention. Their heterogeneity alone does not determine normalization’s benefit: adding PopArt changes Table 1 mean return from 2.65 to 1.87 in CaveFlyer and from 2.99 to 3.56 in BigFish, although both have heterogeneous targets. MH already improves on Shared in both games.
Once per rollout control. In the original implementation, each target enters the statistics across three PPO epochs; eight minibatches per epoch permit up to 24 updates per head per rollout. The new control updates statistics once per newly collected rollout and freezes them throughout the subsequent PPO epochs.
The primary setting uses , the EMA coefficient for target statistics, following PopArt’s implementation notes; the optimizer learning rate is unchanged. A supplementary control uses , matching the cumulative retention of old statistics across 24 updates.
Both rollout settings train for approximately 25M steps with five seeds per game. We hold the PPO actor update and advantage normalization fixed and apply PopArt value normalization to the critic. This isolates the statistics schedule within our PPO setup; original multitask PopArt additionally rescales actor feedback by task scale (Eq. 12). Evaluation follows Table 1’s protocol on the same 600 unseen levels. Values are mean ± sd: the first three columns reproduce Table 1’s 10-seed results, while the new rollout columns use five seeds.
| Game | Shared | MH | PA: minibatch | PA: rollout, | PA: rollout, |
|---|---|---|---|---|---|
| CaveFlyer | 1.08 ± 0.16 | 2.65 ± 0.43 | 1.87 ± 0.31 | 1.66 ± 0.06 | 1.84 ± 0.45 |
| CoinRun | 5.76 ± 0.29 | 6.80 ± 0.37 | 6.46 ± 0.29 | 6.67 ± 0.19 | 6.56 ± 0.15 |
| BigFish | 1.87 ± 0.17 | 2.99 ± 0.61 | 3.56 ± 1.27 | 4.04 ± 0.20 | 4.28 ± 1.33 |
| Maze | 1.23 ± 0.13 | 1.49 ± 0.18 | 1.49 ± 0.15 | 1.48 ± 0.16 | 1.68 ± 0.11 |
Performance is mostly comparative among different schedules. MH retains higher means in CaveFlyer and CoinRun under both rollout settings; PopArt benefits BigFish, and the faster EMA also improves Maze over MH.
To sum up, these checks clarify the ablation: Shared versus multihead (MH) measures the benefit of conditional baselines; MH versus MH+PopArt measures additional normalization. MH has higher mean return than Shared in all four games, while normalization’s extra effect depends on the game and statistical adaptation rate. This directly identifies the benefit already obtained from separation and measures what normalization adds cannot contribute to stable increments.
To further explain why PopArt normalization behaves irregularly in our experiments, even within one game, per level normalization rescales critic errors differently across levels, changing how shared features are learned and thereby affecting subsequent value and advantage estimates; these changes can help or hinder training, so comparable reward scales do not guarantee a consistent additional benefit. In our single game setting, levels share a common reward convention, whereas the extreme cross game scale disparities studied by PopArt can dominate relative update magnitudes and make normalization especially consequential. Thus, our problem basically is very different from PopArt's multitask setting.
Manuscript revision (blue): Appendix J.4, “Value separation and target normalization,” reports both diagnostics and controls; Section 4.5 explains their role in the ablation.
Reply to Reviewer Y6Jg (3/3): Summary of the main concern
We again appreciate the request for a more thorough analysis of the PopArt ablation and have added the target scale diagnostics and rollout level statistics controls. These results clarify why we included this comparison.
First, the theoretical motivations differ. Multitask PopArt uses indexed value heads to support task specific normalization, addressing large differences in update magnitudes across tasks. Our theory studies how the value reference affects finite sample policy gradient dynamics within a common reward convention. Conditional baselines can therefore improve learning through value separation itself, even when the expected gradient at a fixed policy remains unchanged.
Second, the experiments distinguish these mechanisms. In our setting, multihead alone provides substantial gains, while normalization produces game dependent effects rather than consistent additional improvements, including under the revised statistics schedule. In PopArt’s own Atari ablation, additional heads alone provide no improvement, and the reported gains are attributed to adaptive rescaling.
The purpose of this ablation is therefore to distinguish the problems addressed and the sources of improvement behind similar architectures. It clarifies the theoretical and practical contribution of our work: understanding when and why separating value baselines itself changes learning.
Reply to Reviewer Y6Jg (1/3): Contribution and distinction from multitask PopArt (Weakness 1)
Thank you for acknowledging distinctness of our theories and providing questions to let us clarity our algorithms v.s. PopArt in detail.
We want to first emphasize that since our method and PopArt share a similar architecture, we make the ablation in the paper to distinguish the underlying problems they address and clarify the theoretical insights and practical value of conditional value baselines in our setting.
Our contribution explains why conditional value baselines changes learning within a single task. The shared actor, indexed critic architecture provides the intervention; the theory identifies the mechanism and the experiments test it.
The scale problem in PopArt is quantitatively substantial. In its Atari experiments, the running target mean exceeds 2,500 in Crazy Climber and stays below 15 in Bowling (Fig. 3 in its original paper). Such differences motivate controlling the magnitude of learning updates. At a fixed policy, the score function gradient obeys
(G_z,V_z)\mapsto(cG_z,cV_z)\ \Longrightarrow\ g_z\mapsto c g_z.$$ A large multiplicative return scale therefore directly amplifies a task's actor gradient contribution. Separate value heads leave this multiplier in place, so large scale tasks can dominate the shared update. PopArt's additional division by the estimated task scale addresses this imbalance. Its Atari ablation makes the distinction explicit: heads alone provide no improvement, with or without reward clipping; the authors conclude that “the performance boost of PopArt-IMPALA is indeed due to the adaptive rescaling.” [Hessel et al., Figs. 1–3 and Eq. 12](https://arxiv.org/html/1809.04474). **Our question concerns the value baseline working in one single task.** We train each Procgen game separately, using a common reward convention and the same global reward scaling and advantage normalization procedures. Within this setting, pooling values introduces an environment dependent offset: $$A_z^{\mathrm{shared}}=A_z^{\mathrm{cond}}+ \bigl(V_z^\pi(s)-\bar V^\pi(s)\bigr).$$ This offset can discourage a successful action in a lower value environment and alter subsequent sampling. Our propositions 1–3 establish the resulting pathwise distinction under the paper's finite bandit assumptions, exact baselines, and common strictly optimal action. For every fixed finite $\eta>0$, both processes converge to the same optimum almost surely; conditional centering eventually improves the optimal action probability on every update, whereas shared centering has recurrent, vanishing decreases when optimal rewards differ. This is the additional understanding supplied by our theory. The ablation directly separates the two questions: **Shared → multihead value functions tests the benefit of conditioning; multihead → multihead+PopArt tests additional target normalization (from our experiments there is no steadily and nontrivial increments with the additional normalization, thus our setting is completely different than PopArt's).** Table 1 already shows a 40.8% aggregate gain from multihead alone, compared with 33.3% after adding PopArt value normalization. The common architecture lets us isolate these different mechanisms: **PopArt's Atari gains rely on adaptive rescaling, while our study explains and demonstrates the role of conditioning baseline itself, and we are different problems with different explanations**. **Manuscript revision (blue):** Section 3 explains the sampled-update contribution; Section 4.5, “Target normalization,” states the purpose of the PopArt ablation and the distinction between the two mechanisms.