acceptodds
Under review as a conference paper at ICLR 2027

Shared Actors Need Not Share Critics: Effects of Value Mismatch in Parallel Reinforcement Learning

Abstract

When an actor-critic policy is trained in parallel across multiple environments in order to achieve strong generalization capability, implementations commonly use one critic throughout training by default. Yet different environments can assign different expected returns to the same state input to the critic. A critic without environment information must then reconcile distinct value targets, systematically shifting the sampled advantages within individual environments. Using illustrative bandit models with multiple environments and a common optimal arm, we characterize how this value mismatch redistributes sampled policy updates, reinforcing unhelpful actions while attenuating or even reversing useful ones. The oracle learning processes using no baseline, the shared value, or the value specific to the sampled environment have the same mean logit update at a fixed policy and converge to the same optimal policy, yet their realized learning paths can differ sharply under stochasticity. The analysis motivates a minimal intervention: give only a logged environment index to the critic so that it can separate the value targets. Controlled CartPole and MuJoCo experiments expose the predicted shifted values, advantages, and performance gaps. In the more complex BipedalWalker and Procgen settings, the same intervention yields more stable learning and higher returns. Across all Procgen games, the multihead conditional critic improves aggregate normalized return on unseen levels per game by . In conclusion, we present the theory identifying value mismatch as a direct mechanism through which critic sharing can degrade stochastic learning dynamics, not captured by scalar estimator variance alone, and conduct the experiments showing that conditioning on an index is broadly effective in parallel actor-critic RL.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Actor Critic Policy Gradient

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Y6JgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies value mismatch when a shared actor trains across environments whose conditional returns differ for the same critic input. It compares zero, shared-value, and environment-conditional oracle baselines in deterministic bandits with a common optimal arm. The analysis establishes common convergence while distinguishing eventual samplewise improvement from recurrent drawdowns. The paper analyzes and evaluates established conditioning designs that provide a categorical environment identifier only to the critic, using FiLM or multiple value heads. PPO experiments cover controlled CartPole, mass-varied MuJoCo, BipedalWalker, and Procgen. Under the specified fixed, mixed-difficulty Procgen training protocol, the reported multihead aggregate normalized return on unseen levels improves by 40.8% over the shared critic.

Critical strengths
  1. The theoretical analysis provides a useful distinction between expected policy updates and realized learning paths. Propositions 1–3 show that the methods converge to the same optimum but differ in whether individual updates eventually improve the policy or continue to produce vanishing drawdowns. Appendix C further separates a variance-based improvement bound from actual finite-horizon behavior. Together, these results help explain how the choice of baseline affects learning.
  2. The experiments provide controlled evidence and practical support for conditioning the critic while retaining a shared actor. Figures 3–5 include identical and heterogeneous environments, constant-index and scalar-bias controls, and a capacity control. These comparisons help assess the contribution of conditioning. Table 1 also reports higher unseen-level test means for multihead on 14 of 16 Procgen games.
Critical weaknesses
  1. The methodological increment over multitask PopArt is limited, and the distinction in problem scope needs stronger justification. PopArt already combine a shared policy that does not directly receive the task index with task-indexed value heads. Their introduction also identifies reward sparsity and agent competence as sources of return-scale differences. Treating levels as tasks places the paper's multihead intervention within this existing design. Section 5 acknowledges the architectural precedent, but comparable reward scales alone do not establish a clear separation from the earlier setting. Propositions 1–3 provide distinct theoretical results. Nevertheless, I find the additional understanding established beyond the existing design insufficiently compelling for acceptance.
  2. The explanation for PopArt's lower aggregate performance lacks direct scale diagnostics and leaves open the possibility that the training setup limits its effectiveness. First, Section 4.5 relates the reduction in aggregate improvement from 40.8% to 33.3% in Table 1 to matched reward scales, but does not report per-level value-target scale measurements to support this premise. This matters because differences in difficulty, reward sparsity, and policy competence can produce different return scales even under the same reward convention. Second, the statistical update schedule provides a possible alternative explanation. The implementation notes in PopArt describe updates to normalization statistics per rollout. In contrast, this paper uses per-minibatch updates and three PPO epochs, as specified in G.1 and Table 6. It is unclear whether the same targets repeatedly enter the statistics across epochs. If they do, repeated statistical updates during optimization could affect PopArt's performance and contribute to the observed aggregate decline. Taken together, the missing scale measurements and the possible interaction with the training setup leave the attribution to comparable scales insufficiently supported.
Questions for authors
  1. Please clarify how differences in value-target scales relate to the benefit of PopArt in Table 1. Could you add a compact summary of the per-level target means and standard deviations at a specified existing checkpoint?
  2. Please test whether aligning the normalization update schedule with the original PopArt implementation changes the comparison. PopArt updates statistics per rollout, while this paper uses minibatches and multiple PPO epochs on the same data. To assess whether this difference contributes to the Table 1 result, could you run a controlled comparison on a small set of representative tasks, updating statistics once per newly collected rollout and holding them fixed during the subsequent PPO optimization epochs? This would align the statistical update unit with the original method while retaining the PPO training setup. If the performance difference persists, it would help rule out this scheduling difference as an explanation.
5 comments
AuthorsOpenReview ↗

Reply to Reviewer Y6Jg (1/3): Contribution and distinction from multitask PopArt (Weakness 1)

Thank you for acknowledging distinctness of our theories and providing questions to let us clarity our algorithms v.s. PopArt in detail.

We want to first emphasize that since our method and PopArt share a similar architecture, we make the ablation in the paper to distinguish the underlying problems they address and clarify the theoretical insights and practical value of conditional value baselines in our setting.

Our contribution explains why conditional value baselines changes learning within a single task. The shared actor, indexed critic architecture provides the intervention; the theory identifies the mechanism and the experiments test it.

The scale problem in PopArt is quantitatively substantial. In its Atari experiments, the running target mean exceeds 2,500 in Crazy Climber and stays below 15 in Bowling (Fig. 3 in its original paper). Such differences motivate controlling the magnitude of learning updates. At a fixed policy, the score function gradient obeys

(G_z,V_z)\mapsto(cG_z,cV_z)\ \Longrightarrow\ g_z\mapsto c g_z.$$ A large multiplicative return scale therefore directly amplifies a task's actor gradient contribution. Separate value heads leave this multiplier in place, so large scale tasks can dominate the shared update. PopArt's additional division by the estimated task scale addresses this imbalance. Its Atari ablation makes the distinction explicit: heads alone provide no improvement, with or without reward clipping; the authors conclude that “the performance boost of PopArt-IMPALA is indeed due to the adaptive rescaling.” [Hessel et al., Figs. 1–3 and Eq. 12](https://arxiv.org/html/1809.04474). **Our question concerns the value baseline working in one single task.** We train each Procgen game separately, using a common reward convention and the same global reward scaling and advantage normalization procedures. Within this setting, pooling values introduces an environment dependent offset: $$A_z^{\mathrm{shared}}=A_z^{\mathrm{cond}}+ \bigl(V_z^\pi(s)-\bar V^\pi(s)\bigr).$$ This offset can discourage a successful action in a lower value environment and alter subsequent sampling. Our propositions 1–3 establish the resulting pathwise distinction under the paper's finite bandit assumptions, exact baselines, and common strictly optimal action. For every fixed finite $\eta>0$, both processes converge to the same optimum almost surely; conditional centering eventually improves the optimal action probability on every update, whereas shared centering has recurrent, vanishing decreases when optimal rewards differ. This is the additional understanding supplied by our theory. The ablation directly separates the two questions: **Shared → multihead value functions tests the benefit of conditioning; multihead → multihead+PopArt tests additional target normalization (from our experiments there is no steadily and nontrivial increments with the additional normalization, thus our setting is completely different than PopArt's).** Table 1 already shows a 40.8% aggregate gain from multihead alone, compared with 33.3% after adding PopArt value normalization. The common architecture lets us isolate these different mechanisms: **PopArt's Atari gains rely on adaptive rescaling, while our study explains and demonstrates the role of conditioning baseline itself, and we are different problems with different explanations**. **Manuscript revision (blue):** Section 3 explains the sampled-update contribution; Section 4.5, “Target normalization,” states the purpose of the PopArt ablation and the distinction between the two mechanisms.
AuthorsOpenReview ↗

Reply to Reviewer Y6Jg (2/3): Target scales and normalization updates (Weakness 2; Questions 1–2)

We added both requested checks: per level target statistics at existing checkpoints and a comparison that updates normalization statistics once per new rollout, holding them fixed during PPO optimization.

Target scale diagnostics. We evaluate MH (multihead) and MH+PA (PopArt normalization) at approximately 25M steps on four games, with five training seeds per game and variant. We freeze the actor, critic, PopArt buffers and saved global reward-normalization statistics, then collect 10 fresh rollouts, yielding 2,560 GAE targets per level across 200 training levels. Targets are measured before PopArt normalization, in each checkpoint’s frozen global reward normalization units. Each entry reports p10 / median / p90 across levels, taking the median of each quantile across seeds.

GameCriticLevel target meanLevel target SD
CaveFlyerMH0.005 / 4.40 / 9.640.021 / 0.34 / 1.53
CaveFlyer+PA0.001 / 1.21 / 9.730.004 / 0.33 / 1.51
CoinRunMH5.92 / 7.70 / 7.940.067 / 0.22 / 1.26
CoinRun+PA5.17 / 7.91 / 8.070.051 / 0.22 / 1.37
BigFishMH0.041 / 2.03 / 8.770.092 / 1.00 / 2.48
BigFish+PA0.010 / 1.96 / 11.180.011 / 0.93 / 2.36
MazeMH-0.026 / 2.94 / 2.970.002 / 0.011 / 0.038
Maze+PA-0.001 / 3.61 / 3.660.001 / 0.007 / 0.084

Targets differ across levels within the common reward convention. Their heterogeneity alone does not determine normalization’s benefit: adding PopArt changes Table 1 mean return from 2.65 to 1.87 in CaveFlyer and from 2.99 to 3.56 in BigFish, although both have heterogeneous targets. MH already improves on Shared in both games.

Once per rollout control. In the original implementation, each target enters the statistics across three PPO epochs; eight minibatches per epoch permit up to 24 updates per head per rollout. The new control updates statistics once per newly collected rollout and freezes them throughout the subsequent PPO epochs.

The primary setting uses β=0.0003\beta=0.0003, the EMA coefficient for target statistics, following PopArt’s implementation notes; the optimizer learning rate is unchanged. A supplementary control uses βeff=1−(1−0.0003)24≈0.0072\beta_{\mathrm{eff}}=1-(1-0.0003)^{24}\approx0.0072, matching the cumulative retention of old statistics across 24 updates.

Both rollout settings train for approximately 25M steps with five seeds per game. We hold the PPO actor update and advantage normalization fixed and apply PopArt value normalization to the critic. This isolates the statistics schedule within our PPO setup; original multitask PopArt additionally rescales actor feedback by task scale (Eq. 12). Evaluation follows Table 1’s protocol on the same 600 unseen levels. Values are mean ± sd: the first three columns reproduce Table 1’s 10-seed results, while the new rollout columns use five seeds.

GameSharedMHPA: minibatchPA: rollout, β=0.0003\beta=0.0003PA: rollout, β=0.0072\beta=0.0072
CaveFlyer1.08 ± 0.162.65 ± 0.431.87 ± 0.311.66 ± 0.061.84 ± 0.45
CoinRun5.76 ± 0.296.80 ± 0.376.46 ± 0.296.67 ± 0.196.56 ± 0.15
BigFish1.87 ± 0.172.99 ± 0.613.56 ± 1.274.04 ± 0.204.28 ± 1.33
Maze1.23 ± 0.131.49 ± 0.181.49 ± 0.151.48 ± 0.161.68 ± 0.11

Performance is mostly comparative among different schedules. MH retains higher means in CaveFlyer and CoinRun under both rollout settings; PopArt benefits BigFish, and the faster EMA also improves Maze over MH.

To sum up, these checks clarify the ablation: Shared versus multihead (MH) measures the benefit of conditional baselines; MH versus MH+PopArt measures additional normalization. MH has higher mean return than Shared in all four games, while normalization’s extra effect depends on the game and statistical adaptation rate. This directly identifies the benefit already obtained from separation and measures what normalization adds cannot contribute to stable increments.

To further explain why PopArt normalization behaves irregularly in our experiments, even within one game, per level normalization rescales critic errors differently across levels, changing how shared features are learned and thereby affecting subsequent value and advantage estimates; these changes can help or hinder training, so comparable reward scales do not guarantee a consistent additional benefit. In our single game setting, levels share a common reward convention, whereas the extreme cross game scale disparities studied by PopArt can dominate relative update magnitudes and make normalization especially consequential. Thus, our problem basically is very different from PopArt's multitask setting.

Manuscript revision (blue): Appendix J.4, “Value separation and target normalization,” reports both diagnostics and controls; Section 4.5 explains their role in the ablation.

AuthorsOpenReview ↗

Reply to Reviewer Y6Jg (3/3): Summary of the main concern

We again appreciate the request for a more thorough analysis of the PopArt ablation and have added the target scale diagnostics and rollout level statistics controls. These results clarify why we included this comparison.

First, the theoretical motivations differ. Multitask PopArt uses indexed value heads to support task specific normalization, addressing large differences in update magnitudes across tasks. Our theory studies how the value reference affects finite sample policy gradient dynamics within a common reward convention. Conditional baselines can therefore improve learning through value separation itself, even when the expected gradient at a fixed policy remains unchanged.

Second, the experiments distinguish these mechanisms. In our setting, multihead alone provides substantial gains, while normalization produces game dependent effects rather than consistent additional improvements, including under the revised statistics schedule. In PopArt’s own Atari ablation, additional heads alone provide no improvement, and the reported gains are attributed to adaptive rescaling.

The purpose of this ablation is therefore to distinguish the problems addressed and the sources of improvement behind similar architectures. It clarifies the theoretical and practical contribution of our work: understanding when and why separating value baselines itself changes learning.

Reviewer_VgXCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the authors study the problem of actor-critic learning scheme with multiple environment in reinforcement learning. Utilize the intuition obtained from the theoretical analysis conducted on a bandit problem, they provide an explain to the effect of value mismatching and argue that separating values changes the learning process and benefits the learning. Their arguments and proposed fix(proving the index label of environment) are put in test in a series numerical experiments with promising outcome.

Critical strengths

The theoretical analysis provided in the paper is innovative and sound. The view of monotonicity is providing a very reasonable explanation to the mismatch, indeed helps us in better understanding of thee basic learning dynamics. The explanation on variance provided in Section C is also very helpful for me to understand the problem, and I would recommend it to be included in the main text in some form.

Critical weaknesses
  1. The theoretical analysis is conducted on a stylish, simple and representative model, it is not immediate to see that they can be extended to general MDP problems.

  2. The numerical examples are representative but not convincingly extensive.

Questions for authors
  1. how would the main results (mainly propositions 2 and 3) change if variable learning rate is allowed?
  2. Could you talk about difficulties extending the theoretical results to general MDP?
5 comments
AuthorsOpenReview ↗

Reply to Reviewer VgXC (1/4): Variable learning rates and Propositions 2–3 (Q1)

Thank you for positive feedback over of our theories. We are glad to present our theories with novelty, accuracy, as well as its alignment on designing useful algorithms.

To start our replies, our choice of a constant learning rate is motivated by practical training with a fixed learning rate: we want to understand how baseline choice shapes the stochastic learning dynamics in this regime. Under the original finite bandit and oracle baseline assumptions, our results cover every fixed finite η>0\eta>0, without a small step restriction or learning rate decay. The proof therefore controls convergence and individual sampled updates while the step size remains fixed.

The qualitative conclusions of Propositions 2 and 3 also extend to positive, predictable scalar steps satisfying the classical Robbins-Monro conditions:

ηt>0,∑tηt=∞,∑tηt2<∞.\eta_t>0,\qquad \sum_t\eta_t=\infty,\qquad \sum_t\eta_t^2<\infty.

Here, the step is chosen before the current environment and action draws. This extension has a shorter noise control argument: square summability makes the accumulated martingale noise converge, and the bandit structure gives convergence and infinite sampling.

The intuition is simple: a positive step scales the update, while the baseline determines whether the sampled action is reinforced. Once the policy enters the original favorable region, the same margin calculation applies. Thus eventual conditional improvement and recurrent decreases under the contrasting baselines remain valid under the respective reward conditions of Propositions 2 and 3. Appendix E gives the proof sketch.

Manuscript revision (blue): Appendix E, “Variable learning rates,” states the Robbins–Monro conditions and gives the proof sketch extending Propositions 2–3.

AuthorsOpenReview ↗

Reply to Reviewer VgXC (2/4): Extending the theory to MDPs

To extend our results to the general MDP case, the main difficulty is that action values change with the continuation policy. Robertson et al. (2025) use independent parameters for each state and episode time stage, enabling backward induction from the last time stage. A stationary actor reuses state parameters across visits, coupling the continuation values. However, we can only adopt their methods in the single state MDP (bandit case) since our policy is more general type of policy π(a∣s)\pi (a|s) instead of πh(a∣s)\pi ^h(a|s) (conditional on horizon time hh), but we can still figure out one way to extend the result with additional assumption.

We added our extend result Proposition 12 and a proof sketch in Appendix H. The additional condition is an eventually stable optimal action gap: for fixed common actions a∗(s)a^*(s) and some δ>0\delta>0, each compared process eventually satisfies

Qzπt(s,a∗(s))−Qzπt(s,a)≥δ,a≠a∗(s).Q_z^{\pi_t}(s,a^{\ast}(s))-Q_z^{\pi_t}(s,a)\ge\delta,\qquad a\ne a^{\ast}(s).

uniformly over states and environments, almost surely. Both the values and the gap magnitudes may continue to change. This permits action dependent transitions and recurrent states.

The result uses finite discounted MDPs, independent tabular policy rows, a fixed full support distribution over state environment pairs, and any fixed positive learning rate. The update uses exact action value feedback Qzπt(s,a)−btQ_z^{\pi_t}(s,a)-b_t, with the corresponding oracle value baseline.

The original convergence argument extends using uniform gap bounds. Once the policy concentrates on the optimal action, conditional advantages have the signs required by the original margin calculation. This eventually increases the optimal action probability at every updated state. Shared centering produces recurrent decreases at states whose limiting values differ across environments. The extension preserves the central distinction: both policies reach the same limit, while their sampled paths differ.

Manuscript revision (blue): Appendix H, “Extending the mechanism to MDPs”.

AuthorsOpenReview ↗

Reply to Reviewer VgXC (3/4): Variance analysis in the main text

Following your suggestion, we added the key explanation from Appendix C after the opening paragraph of Section 3. For an unbiased gradient estimate with mean gg and total variance σ2\sigma^2, the standard smoothness bound is

E[ΔJ]≥η∥g∥2−LJη22(∥g∥2+σ2).\mathbb E[\Delta J]\ge\eta\|g\|^2-\frac{L_J\eta^2}{2}\bigl(\|g\|^2+\sigma^2\bigr).

This explains the classical motivation for variance reduction, which is to only increase the local lower bound and cannot predict the advantage of long term learning trajectory. Our contribution is to characterize how value baseline controls individual sampled updates and subsequent sampling. Propositions 2 and 3 show that baselines sharing the same asymptotic optimum can generate different learning paths: eventual samplewise improvement versus recurrent backward steps. Appendix C gives a complementary finite horizon example: the exact variance minimizing baseline yields a lower mean terminal optimal action probability than conditional value baseline.

Manuscript revision (blue): The paragraph immediately after the opening paragraph of Section 3, “Baselines change the online update dynamics,” brings this explanation into the main text. The complete variance analysis remains in Appendix C.

Reviewer_XqjcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies value mismatch when one actor is trained across environments but its critic lacks environment identity. It separates invariance of the expected policy-gradient update at a fixed policy from the stochastic learning paths induced by different baselines. In deterministic bandits with a common optimal arm and oracle baselines, the analysis establishes common asymptotic convergence while identifying eventual monotone improvement under conditional value centering and recurring drawdowns under shared centering. The resulting intervention supplies a logged environment index only to the critic, using FiLM or multihead conditioning. Experimental evaluations support practical benefits within recurring environment pools.

Critical strengths
  • A useful mechanism with appropriately delimited theory. Propositions 1–3 give concrete explanations for why separating environment-specific value targets can matter.

  • The experimental controls make the practical result persuasive.

Critical weaknesses

The authors explicitly acknowledge this limitation in Section 6. However, the practical applicability of the proposed intervention to highly diverse or continuously changing training environments remains unclear. In particular, the paper does not establish how environments should be adaptively grouped for critic conditioning, or how the value-mismatch insight can guide critic design when stable environment identifiers and sufficient revisits are unavailable.

Questions for authors

To quantify the boundary already acknowledged in Section 6 and Appendix B.4, can you compare shared, multihead, and FiLM critics as level revisit rates decrease, keeping total environment steps, parallelism, and the update protocol fixed?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer Xqjc (1/3): Applicability with sparse revisits and missing environment IDs (Weakness)

Thank you for showing positive feedback and identifying one important problem for us. This helps us improve our algorithm to more general application. Our practical target is training over recurring environments with value relevant context: procedural levels, terrain configurations, or logged simulator parameters. The identifier selects the critic's conditioning information; the actor remains shared and receives no identifier.

Since the benefit of conditioning baselines mainly comes from more revisits of each sampled environment level, it is indeed necessary for us to consider the case when such revisits are less frequent given larger space of environments. On the other hand, conditioning itself may even hurt the training since more levels without enough exposure may increase the complexity of value networks.

We now test substantially larger, randomly resampled training pools for games in Procgen: 1,000 and 2,000 levels, five and ten times the original pool size (200), at the same 25M step budget.

In these experiments, we figure out one simple solution which can still utilize the benefit of conditional baselines without hurting the training (or at least as good as the shared baseline). We let FiLM and multihead use the shared initialization variants designed for many levels with sparse revisits. These results extend the evidence to larger pools, with gains depending on the task and critic design. The next post gives the complete implementation and comparison.

Specifically, we train an additional shared critic on pooled rollouts solely to initialize newly encountered levels. Each new level copies a shared value reference, then learns its own conditional correction through multihead updates or FiLM modulation. The auxiliary critic keeps learning to initialize later levels, and existing levels continue training their own parameters. The third post explains the shared-initialization design used in these experiments.

Manuscript revision (blue): Section 6, “Conditioning with fewer revisits,” describes the shared initialization variants evaluated at 1,000 and 2,000 levels; Appendix B.4 gives their experimental protocol and initialization design.

AuthorsOpenReview ↗

Reply to Reviewer Xqjc (2/3): compare shared, multihead, and FiLM critics as level revisit rates decrease (Question)

We compare Shared, FiLM and Multihead on CoinRun, BigFish and Maze with 1,000 or 2,000 training levels and five training seeds. Each pool contains equal numbers of easy and hard levels, disjoint from the evaluation levels, with uniform resampling at episode boundaries. We hold fixed the actor architecture, 25M environment steps, 200 parallel environments, 256 step rollouts, PPO's 3 epochs × 8 minibatches, and the remaining hyperparameters.

For these larger pools, FiLM and Multihead are the shared initialization variants: an auxiliary shared critic is trained on pooled rollouts and initializes each new level, after which the conditional critic learns its own value. Shared remains the standard pooled critic baseline.

Table R3 reports mean ± sd. The 200 level rows are historical references from Table 1 under its evaluation protocol; the 1,000/2,000-level rows report sampled action returns under the resampling protocol. The controlled comparison of larger pools is between 1,000 and 2,000 levels.

GameLevelsSharedFiLMMultihead
CoinRun2005.76 ± 0.296.35 ± 0.286.80 ± 0.37
CoinRun1,0008.10 ± 0.298.20 ± 0.128.20 ± 0.10
CoinRun2,0008.30 ± 0.108.22 ± 0.418.28 ± 0.30
BigFish2001.87 ± 0.172.88 ± 0.572.99 ± 0.61
BigFish1,0002.13 ± 1.351.53 ± 0.381.60 ± 0.61
BigFish2,0002.35 ± 1.223.66 ± 0.702.25 ± 1.00
Maze2001.23 ± 0.131.33 ± 0.131.49 ± 0.18
Maze1,0006.09 ± 0.336.19 ± 0.357.09 ± 0.88
Maze2,0006.16 ± 0.336.37 ± 0.327.13 ± 0.52

Above results support that given our residual alike design, conditioning can remain useful when a fixed training budget is distributed over more levels. We also observe that increasing numbers of training environments does not necessary cause the increment or downgrade of performance in the held-out set, and we assume that this will depend on specific game, such as the tradeoff between fully learn one environment or explore more environment for better generality. Here we only evaluate the comparisons of shared baselines with our implemented conditional methods.

Manuscript revision (blue): Appendix B.4, “Fixed-budget comparison with reduced revisits,” specifies that the larger pool FiLM and Multihead runs use shared initialization and reports their comparison with Shared baseline in Table 2.

AuthorsOpenReview ↗

Reply to Reviewer Xqjc (3/3): details of the conditional baseline algorithm specified for less revisits

We use an additional shared critic alongside the conditioned critics to initialize newly encountered levels. It learns from pooled rollouts and serves solely to initialize new levels. Each new level starts from the current shared value estimate, then learns its own conditional value; the auxiliary critic continues learning to initialize later arrivals.

Let τz\tau_z be the arrival time of level zz and ht(s)h_t(s) the current shared features. Copy the auxiliary readout (wˉτz,bˉτz)(\bar w_{\tau_z},\bar b_{\tau_z}) and write

Vz,tinit(s)=wˉτz⊤ht(s)+bˉτz,Vz,t(s)=Vz,tinit(s)+δz,t(s),δz,τz(s)=0.V_{z,t}^{\mathrm{init}}(s)=\bar w_{\tau_z}^{\top}h_t(s)+\bar b_{\tau_z},\qquad V_{z,t}(s)=V_{z,t}^{\mathrm{init}}(s)+\delta_{z,t}(s),\quad \delta_{z,\tau_z}(s)=0.

For multihead, copy the shared readout into the new trainable head; subsequent parameter updates learn the residual relative to this initialization. The shared initialized FiLM variant retains a readout copy for the new level and initializes its modulation to the identity. FiLM then learns a conditional residual through feature modulation. The auxiliary critic's later updates do not overwrite an existing level's copied readout. Appendix B.4 gives the explicit constructions, including neutral initialization of the FiLM modulation.

This provides a practical starting point for sparse revisits: new levels inherit pooled experience and then learn their own value differences. Adaptive grouping can share predictions among contexts with similar values and separate them as differences become clear relative to estimation uncertainty. Without stable IDs, observable simulator parameters or context inferred before the current action can define the conditioning context or group; the actor remains shared.

Manuscript revision (blue): Section 6, “Conditioning with fewer revisits”.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief Message to All Reviewers

Dear all reviewers,

We thank all reviewers for their thoughtful feedback and constructive discussion. In response, we have clarified and extended the theoretical analysis, added experiments with larger training pools and fewer revisits, and strengthened the PopArt comparison with target scale diagnostics and normalization schedule controls. These additions help clarify both the contribution and the limitations of our work. All revisions are marked in blue. We appreciate the reviewers’ careful consideration of our responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.