acceptodds
Under review as a conference paper at ICLR 2027

Training Stability under Adaptive Curricula

Abstract

Unsupervised Environment Design (UED) approaches train reinforcement learning (RL) agents on diverse curricula that may evolve with their capabilities. While prior work has largely focused on how these curricula are constructed, we study whether their students remain trainable under the resulting distributions. Across multiple UED mechanisms, environment complexities, and extended training horizons, we identify recurring learner-side pathologies, including parameter growth, degraded value–outcome ordering, and loss of representational diversity, that accompany stagnation and performance degradation. These findings connect limitations in UED to optimization failures previously identified in conventional deep RL, while showing how adaptive curricula couple the learner's optimization state to its future training distribution. Controlled interventions show that mitigating selected pathologies improves learning. Stabilizing the learner improves long-horizon performance and zero-shot generalization without changing the curriculum rule or increasing environment interaction. Our results establish learner trainability as a complementary design principle for UED. Effective curricula require students that remain capable of learning from the distributions they produce.

Primary area:
reinforcement learning
Keywords:
deep rl, ued, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_fXJfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates learner-side training instability in Unsupervised Environment Design (UED), arguing that the effectiveness of adaptive curricula depends not only on task selection but also on the student’s ability to sustain learning. Through empirical analyses of PPO agents in Kinetix, the authors identify several optimization pathologies associated with training stagnation, including parameter growth, degraded value prediction, and reduced representational diversity. They evaluate existing stabilization techniques and propose combining weight clipping (WC) with categorical value prediction (HL-Gauss) to mitigate these issues. Experiments across multiple UED methods (PLR, SFL, and ACCEL), environment complexities, and training regimes demonstrate improved performance and generalization without modifying the curriculum mechanisms or increasing environment interaction budgets.

Critical strengths

The paper presents a meaningful empirical investigation of learner-side training stability in UED. I appreciate its focus on an important aspect of adaptive curriculum learning that has received relatively less attention, as well as the systematic experimental evidence supporting the practical benefits of learner stabilization. The main strengths are as follows:

  1. An important and complementary perspective on UED. Rather than focusing exclusively on how curricula are constructed, the paper highlights the importance of maintaining the student's ability to learn from evolving training distributions. This perspective is particularly relevant to adaptive curricula, where learner behavior and subsequent task selection are coupled.

  2. A systematic empirical analysis of training instability. The authors examine multiple indicators of learner degradation, including parameter growth, value prediction quality, and representational diversity, across different environment complexities and training stages. These analyses provide useful empirical insights beyond conventional performance-based evaluations.

  3. Consistent performance improvements supported by extensive experiments. The combination of weight clipping and HL-Gauss achieves substantial improvements across multiple UED methods. The evaluation further includes component ablations, extended training horizons, varying rollout parallelism, cross-complexity generalization, and sequential distribution shifts, providing reasonably strong evidence for the practical effectiveness and robustness of the approach.

Critical weaknesses
  1. The methodological novelty is somewhat limited, and the interaction between curriculum adaptation and learner-side stabilization could be explored more deeply. The proposed approach combines two established techniques, weight clipping and HL-Gauss, whose benefits for training stability have already been studied in deep RL.

  2. A deeper investigation of the interaction between UED methods and learner optimization would be valuable. Given the broader motivation of the paper, I wonder whether these two components could be studied in a more tightly coupled manner. In particular, it would be interesting to understand (i) how different UED methods contribute to specific forms of learner degradation, and whether targeted learner-side interventions could address these issues; and (ii) whether curriculum adaptation and learner stabilization could be jointly optimized. The reviewer views the current work as a valuable starting point and does not consider this a major weakness. Nevertheless, a discussion of these directions would help clarify the broader significance of the findings and motivate further research.

Questions for authors

The paper presents a relatively comprehensive empirical study, and the reviewer appreciates the extensive experiments and analyses provided. Nevertheless, a few aspects of the methodology and experimental setup remain unclear. The reviewer would appreciate clarification on the following points:

  1. In Section 3 and Section 4.2, the authors discuss Positive Value Loss (PVL) as the priority signal used by PLR and interpret its relationship with learnability and curriculum dynamics. However, Table 3 in the appendix specifies MaxMC as the scoring function for PLR. Could the authors clarify which scoring function was actually used in the experiments? If MaxMC was used, how should the PVL-based analyses in Figures 4 and 7 be interpreted in relation to the actual replay priorities?

  2. Appendix D.4 describes ReGen as zeroing weights below a threshold of 10−510^{-5}, whereas the cited work by Kumar et al. (2023) proposes regenerative regularization through an L2 penalty toward the initial parameters. These appear to be different interventions. Could the authors clarify the exact implementation used and whether the reported comparison corresponds to the method proposed in the cited work?

  3. The paper interprets declining effective rank and increasing representation similarity as indicators of learner degradation. However, these statistics may also be influenced by changes in the state distribution induced by the evolving policy and curriculum. In particular, Appendix E.7 indicates that representation similarity is measured on training batches, while the probe distribution used for effective rank is less clear. Could the authors clarify how these metrics are evaluated and whether similar trends can be observed on a fixed set of probe states? This would help distinguish changes in the learned representations from changes in the sampled state distribution.

  4. Section 5 describes the experiment in Figure 9 as extending PLR training on Small environments from 2B to 5B interactions. However, the performance scale around 2B steps appears noticeably different from the corresponding Small-environment results reported earlier. Could the authors clarify whether the evaluation configuration and training settings are identical across these experiments?

The reviewer would be open to reconsidering the current rating if the authors can satisfactorily address the above concerns during the rebuttal.

7 comments

We thank the reviewer for their detailed and encouraging assessment. We appreciate that they found the paper "Clear and easy to follow," reported gaining "a substantially deeper understanding of the problem or question," and had "no substantive concerns about the paper’s soundness." We are encouraged that they described our work as "a meaningful empirical investigation of learner-side training stability in UED" and "an important and complementary perspective on UED." We also appreciate their recognition of our "systematic empirical analysis of training instability" and "consistent performance improvements supported by extensive experiments," as well as their assessment that these experiments provide "reasonably strong evidence for the practical effectiveness and robustness of the approach." We address the reviewer's questions below and describe the corresponding clarifications and revisions to the manuscript.


[CW-1]. The methodological novelty is somewhat limited, and the interaction between curriculum adaptation ..

We appreciate this observation and agree that the two components of our stabilization approach, weight clipping [4] and HL-Gauss [5], are established techniques. In Section 4.1, we evaluate several interventions and use WC + HL-Gauss as a concrete example of how to mitigate learner-side degradation in UED. This combination provides a reference point for researchers evaluating other stabilization approaches. We have made this role explicit in Section 4.1, with the revised text highlighted in blue.

Our primary goal is not to propose a novel or optimal stabilization method, but to identify and systematically characterize learner-side pathologies in UED, an aspect that has received considerably less attention than curriculum construction. As the reviewer also acknowledges, our extensive evaluation across multiple curriculum mechanisms, environment complexities, training horizons, and generalization settings demonstrates the practical relevance of learner stability. Our central hypothesis is that addressing these learner-side pathologies can not only improve the student's trainability but also enhance the effectiveness of adaptive curricula without modifying their underlying generation mechanisms.


[CW-2]. A deeper investigation of the interaction between UED methods and learner optimization would be valuable.

We thank the reviewer for these suggestions and agree that understanding the interaction between curriculum adaptation and learner optimization is an important direction for future research. We have expanded the Discussion section of the revised manuscript to highlight these directions. Specifically, we discuss investigating whether different curriculum mechanisms induce distinct optimization pathologies that could benefit from targeted interventions and whether curriculum selection could incorporate learner-side stability or learning-progress signals to jointly improve curriculum adaptation and learner trainability. We believe these directions provide promising opportunities to extend our findings beyond the stabilization interventions studied here.


[Q1]. In Section 3 and Section 4.2, the authors discuss Positive Value Loss (PVL) as the priority signal ...

We thank the reviewer for identifying this important inconsistency. We confirm that MaxMC was the active scoring function used by PLR to prioritize replay in all original experiments, following the standard Kinetix benchmark configuration [1] to ensure comparability with published baselines. PVL and learnability were also logged throughout these experiments but served as diagnostic metrics rather than replay signals. Accordingly, the PVL-based analyses in Figures 4 and 7 should be interpreted as auxiliary diagnostics, not as analyses of the actual replay priorities. We acknowledge that this distinction was insufficiently clear.

We placed particular emphasis on PVL because prior work (e.g., Rutherford et al., 2024) [2] has highlighted limitations of positive advantage-based regret estimates, which may be affected by critic noise and become misaligned with actual learnability. Our results show that, particularly in more complex environments, baseline PVL can remain elevated despite low learnability and poor performance. Conversely, learner stabilization is associated with lower PVL alongside improvements in learnability and evaluation performance. Persistently high PVL in conjunction with low learnability and poor performance suggests that positive value residuals do not necessarily reflect genuine learning potential; our diagnostics do not isolate critic noise as the cause of this mismatch.

continued below

To resolve the discrepancy, we have clarified the distinction between active replay scores and diagnostic metrics in Sections 3 and 4.2 and Table 3. We have also included additional experiments using PVL-based replay in the new Appendix L, titled PVL-Based Replay: Optimization and UED Dynamics, allowing readers to compare the behavior of PLR under different replay scoring functions. Importantly, the replay mechanism and scoring function remain unchanged between the baseline and stabilized learners within each comparison.

In this new appendix, Figure 25 (Fig. 25 in the paper) shows that PVL-based replay produces performance comparable to MaxMC-based replay while exhibiting similar learner optimization dynamics across Small, Medium, and Large environments. The same optimization pathologies observed under MaxMC-based replay are also present in the PVL-based PLR baseline, whereas HL + WC consistently mitigates these effects. These results suggest that learner stabilization remains effective under both replay scoring functions.

We also provide an analysis showing that the UED metrics under PVL-based scoring exhibit trends similar to those observed under MaxMC. Notably, Figure 26 (Fig. 26 in the paper) shows that learnability exhibits a clearer separation between the PLR baseline and the HL + WC intervention, highlighting the benefits of stabilization in maintaining task learnability across different environment complexities.

Regarding the interpretation of PVL, a high value can initially indicate unexpectedly large positive value residuals and potential learning opportunities. However, its interpretation becomes less straightforward over extended training horizons and with increasing environment complexity. In particular, persistently high PVL alongside low learnability and poor performance suggests that positive value residuals may be inflated by critic noise rather than reflecting genuine learning potential.


[Q-2]. Appendix D.4 describes ReGen as zeroing weights below a threshold of ..

We apologize for the incorrect description in Appendix D.4. Our implementation follows the ReGen regularization approach proposed by Kumar et al. (2023) [3], using an L2 penalty that encourages network parameters to remain close to their initial values rather than penalizing deviations from zero. Specifically, the regularization loss is computed as the sum of squared deviations between the current network parameters and their corresponding initial values, with a regularization coefficient of 10−510^{-5}. Thus, the implementation used in our experiments is consistent with the ReGen formulation; the inconsistency was limited to the mathematical description in the manuscript. We have corrected Appendix D.4 to accurately reflect the implemented method.


[Q-3] . The paper interprets declining effective rank and increasing representation similarity as indicators of learner ..

We thank the reviewer for pointing out this ambiguity. Both representation similarity and effective rank are computed from the same latent representations extracted from the neural network's training batches. We have clarified this in Appendix E.10 by explicitly specifying the source of the feature matrix used to compute effective rank, ensuring consistency in the descriptions of both metrics.

We agree that changes in the state distribution induced by the evolving policy and curriculum may influence representation diagnostics. To address this concern, we conducted an additional experiment using a fixed probe set shared across the baseline and stabilized learners. Specifically, we generated 2,048 random Kinetix levels using a fixed random seed and saved their initial states. These states remain unchanged throughout training and are used to evaluate both learners at periodic checkpoints. We compute effective rank and cosine similarity from the corresponding network representations using forward passes without gradient updates. This protocol allows us to assess whether the observed representation degradation persists when the evaluation input distribution is held fixed.

As shown in Figure 23 (Fig. 23 in the paper), representation collapse and degradation persist across all three environment complexities (S, M, and L) under fixed-probe evaluation. This indicates that the observed degradation cannot be explained solely by changes in the evaluation state distribution.

continued below

We also evaluated the HL + WC intervention using the same fixed-probe protocol. As shown in Figure 24 (Fig. 24 in the paper), HL + WC mitigates representation collapse across all three environment complexities, even when evaluated on the same fixed set of states. This preservation of representational quality is accompanied by improved learning performance, providing further evidence of an association between representation stability and agent performance.

All experimental details and additional analyses are provided in the newly added Section K, Representation Collapse: Empirical Analysis and Ablations, of the revised manuscript. We emphasize that these additional analyses do not establish a complete causal mechanism. Rather, they demonstrate that representation degradation persists under a fixed evaluation distribution, indicating that changes in the sampled evaluation states alone cannot explain the observed degradation.


[Q-4]. Section 5 describes the experiment in Figure 9 as extending PLR training on Small ..

We thank the reviewer for bringing this discrepancy to our attention. Figure 9 reports results for Medium (M) environments rather than Small (S) environments, as incorrectly stated in the text. We selected Medium environments for the extended 5B-interaction experiment because they remain more challenging and exhibit less performance saturation than Small environments, providing greater scope to investigate whether additional training leads to continued improvement or learner degradation. The difference in performance scale therefore arises from comparing results across different environment complexities. We have corrected the figure caption and the corresponding discussion in Section 5 to explicitly indicate that the experiment uses Medium environments, ensuring consistency with the earlier results.


The reviewer would be open to reconsidering the current rating if the authors can satisfactorily address the above concerns during the rebuttal.

We thank the reviewer for their constructive feedback and willingness to reconsider their assessment. We appreciate the opportunity to clarify these methodological details and strengthen the presentation of our results. We have addressed the identified inconsistencies and incorporated additional experimental analyses and clarifications into the revised manuscript, with all changes highlighted in blue. We hope these revisions satisfactorily address the reviewer's concerns.


[1] Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks (Matthews et al., ICLR 2025).

[2] No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery (Rutherford et al., NeurIPS 2024).

[3] Maintaining Plasticity in Continual Learning via Regenerative Regularization (Kumar et al., CoLLAs 2025).

[4] Weight Clipping for Deep Continual and Reinforcement Learning (Elsayed et al., RLC 2024).

[5] Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (Farebrother et al., ICML 2024).

Reviewer_z88zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies learner-side optimization degradation in unsupervised environment design. In Kinetix with PPO, it tracks parameter scale, value–outcome ordering, and representation diversity, and tests established stabilization interventions. Weight clipping combined with HL-Gauss value prediction improves performance across PLR, SFL, ACCEL, and domain randomization. Further experiments examine longer training, rollout parallelism, cross-complexity generalization, and retention under sequential distribution shifts.

Critical strengths

The empirical study makes a useful case for evaluating the learner alongside the curriculum. The evidence extends beyond training return: solve rates use static human-designed holdouts, and sampled-return evaluation uses the DR distribution. Five seeds, IQM and bootstrap reporting, the component ablation, multiple curriculum mechanisms, and the five-billion-interaction experiment provide a substantial basis for the practical stabilization result.

The paper acknowledges that an unchanged adaptive rule can produce different realized curricula for different learners, that representation diagnostics do not establish a single causal mechanism, and that the intervention components are established methods. The contribution is a systematic diagnosis and actionable empirical finding in UED; it need not introduce a new optimizer to be useful.

Critical weaknesses

The account of PLR’s priority signal is inconsistent with the reported configuration. Section 3 states that PVL is used to prioritize replay and interprets high PVL as high replay priority; Section 4.2 similarly links lower PVL to changed priority signals. Table 3 instead specifies MaxMC as the PLR score function. Appendix E distinguishes these scores but does not resolve which was actually used for the figures. Since the priority–learnability mismatch is part of the central diagnosis, this is more consequential than a naming error. The authors should state the actual score per experiment and analyze that score’s relationship with learnability. The discrepancy does not, by itself, invalidate the held-out performance improvements.

The diagnostic and intervention specification leaves important aspects of the proposed explanation difficult to check. Appendix E.7 measures representation similarity on the current training batch, so changing occupancy and curriculum can change the diagnostic even without a corresponding change on a common probe distribution. Appendix E.8 allows either empirical returns or targets for value ordering without specifying the estimator used, while Appendix D does not instantiate HL-Gauss support endpoints and smoothing width or the rule converting the clipping factor into its numerical bound. These omissions do not erase the performance evidence, but they limit reproducibility and the strength of the claim that the measured changes specifically reflect preserved learner trainability.

Questions for authors

Was PLR run with PVL or MaxMC in each diagnostic and intervention experiment? If MaxMC was used, do the priority–learnability conclusions hold when the actual replay score is analyzed, and can the text and configuration tables be made consistent?

Which network features, state samples, and return/target estimator define the representation and value-ordering diagnostics? Can the authors provide the missing WC/HL-Gauss settings and show whether the diagnostic differences persist on one shared probe set? The latter would strengthen the interpretation; I do not require a claim of complete causal identification.

6 comments

We thank the reviewer for their constructive assessment of our work. We are pleased that the reviewer found the paper "Clear and easy to follow" and acknowledged learning "something interesting or useful" from our study. We appreciate their recognition that our work "makes a useful case for evaluating the learner alongside the curriculum" and that our extensive experimental evaluation provides "a substantial basis for the practical stabilization result." We also appreciate their acknowledgment of our careful interpretation of optimization and representation diagnostics, as well as their recognition that "The contribution is a systematic diagnosis and actionable empirical finding in UED; it need not introduce a new optimizer to be useful." Finally, we thank the reviewer for identifying the remaining soundness concerns as "minor or fixable."

We address these concerns in detail below and highlight the corresponding clarifications and revisions made to the manuscript.


[CW-1]. The account of PLR’s priority signal is inconsistent with the reported configuration. Section 3 states that PVL ..

We confirm that MaxMC was the active scoring function used by PLR to prioritize replay in all main experiments, following the standard Kinetix benchmark configuration to ensure comparability with published baselines. PVL and learnability were also systematically logged throughout these experiments but served as diagnostic metrics rather than replay signals.

We placed particular emphasis on PVL because prior work [1] has highlighted the limitations of positive advantage-based regret estimates, which can be strongly affected by critic noise and become misaligned with actual learnability. Our results show that, particularly in more complex environments, baseline PVL can remain artificially elevated despite low learnability and poor performance. Conversely, learner stabilization reduces this inflation, alongside improvements in learnability and evaluation performance.

To resolve the discrepancy, we have clarified the distinction between active replay scores and diagnostic metrics in Sections 3 and 4.2. We have also included additional experiments using PVL-based replay in the new Appendix L, "PVL-Based Replay: Optimization and UED Dynamics," allowing readers to compare the behavior of PLR under different replay scoring functions. Importantly, the replay mechanism and scoring function remain unchanged between the baseline and stabilized learners within each comparison.

In this new appendix, Figure 25 (Fig. 25 in the paper) shows that PVL-based replay produces performance comparable to MaxMC-based replay while exhibiting similar learner optimization dynamics across Small, Medium, and Large environments. The same optimization pathologies observed under MaxMC-based replay are also present under PVL-based replay, whereas HL + WC consistently mitigates these effects. These results suggest that the benefits of learner stabilization are not specific to the choice of replay scoring function.

We also provide an analysis of UED metrics under PVL-based replay, which exhibit trends similar to those observed under MaxMC-based replay. Notably, Figure 26 (Fig. 26 in the paper) shows that learnability exhibits a clearer separation between the PLR baseline and the HL + WC intervention, highlighting the benefits of stabilization in maintaining task learnability across different environment complexities.

Regarding the interpretation of PVL, a high value may initially indicate unexpectedly large positive value residuals and potential learning opportunities. However, its interpretation becomes less straightforward over extended training horizons and with increasing environment complexity. In particular, persistently high PVL alongside low learnability and poor performance suggests that positive value residuals may be inflated by critic noise rather than reflecting genuine learning potential.


[CW-2]. The diagnostic and intervention specification leaves important aspects of the proposed explanation ..

We agree that changes in the state distribution induced by the evolving policy and curriculum may influence representation diagnostics. To address this concern, we conducted an additional experiment using a fixed probe set shared across the baseline and stabilized learners. Specifically, we generated 2,048 random Kinetix levels using a fixed random seed and saved their initial states. These states remain unchanged throughout training and are used to evaluate both learners at periodic checkpoints. We compute effective rank and cosine similarity from the corresponding network representations using forward passes without gradient updates. This protocol allows us to assess whether the observed representation degradation persists when the evaluation input distribution is held fixed.

continued below

As shown in Figure 23 (Fig. 23 in the paper), representation collapse and degradation persist across all three environment complexities (S, M, and L) under fixed-probe evaluation. This indicates that the observed degradation cannot be explained solely by changes in the evaluation state distribution.

We also evaluated the HL + WC intervention using the same fixed-probe protocol. As shown in Figure 24 (Fig. 24 in the paper), HL + WC mitigates representation collapse across all three environment complexities, even when evaluated on the same fixed set of states. This preservation of representational quality is accompanied by improved learning performance, providing further evidence of an association between representation stability and agent performance.

All experimental details and additional analyses are provided in the newly added Section K, "Representation Collapse: Empirical Analysis and Ablations," of the revised manuscript. We emphasize that our interpretation does not rely on establishing a complete causal mechanism. Rather, these additional analyses distinguish representation degradation from changes in the sampled evaluation distribution and provide further evidence that the observed degradation cannot be explained solely by changes in the evaluation state distribution.


[CW-3]. Appendix E.8 allows either empirical returns or targets for value ordering without specifying the estimator used, while ...

We thank the reviewer for pointing out this ambiguity. We have clarified in Appendix E.8 that Spearman's rank correlation coefficient measures the rank consistency between the critic's value predictions and the corresponding GAE-based value targets. The latter are computed as $ G_t^{\mathrm{GAE}} = V(s_t) + \hat{A}_t^{\mathrm{GAE}}. $

Here, V(st)V(s_t) denotes the critic's value prediction used during rollout collection, and A^tGAE\hat{A}_t^{\mathrm{GAE}} is the corresponding generalized advantage estimate (GAE). We compute Spearman's rank correlation coefficient between V(st)V(s_t) and GtGAEG_t^{\mathrm{GAE}} across states within each rollout. Higher values indicate greater rank consistency between the critic's value predictions and their GAE-based targets. Since these targets incorporate the original value predictions, this metric reflects consistency with the learning targets rather than independent ranking accuracy with respect to empirical returns.


[CW-4]. Appendix D does not instantiate HL-Gauss support endpoints and smoothing width

We thank the reviewer for identifying this omission. Our implementation follows the HL-Gauss formulation introduced by Farebrother et al. (2024) [2]. We conducted preliminary experiments with different value support ranges, evaluating supports both narrower and wider than [−2.0,2.0][-2.0, 2.0], and selected the fixed support [vmin⁡,vmax⁡]=[−2.0,2.0][v_{\min}, v_{\max}] = [-2.0, 2.0]. We additionally evaluated different numbers of categorical bins and selected 51 based on preliminary experiments (Figure 19 in the paper). The Gaussian smoothing width was set to σ=0.06\sigma = 0.06, following the rule recommended by Farebrother et al. (2024) [2]. These settings were kept fixed throughout all subsequent experiments. We have updated Table 5 and Appendix D of the revised manuscript to explicitly report these implementation details and improve reproducibility.


[CW-5] or the rule converting the clipping factor into its numerical bound. These omissions do not erase ...

Our implementation follows the weight-clipping approach introduced by Elsayed et al. (2024) [3], where the clipping bound is determined by the clipping factor and the fan-in of each layer. Specifically, the bound is computed as bp=κ/Npb_p = \kappa / \sqrt{N_p}, where κ\kappa denotes the clipping factor and NpN_p is the fan-in of the corresponding layer. Parameters are then clipped to the interval [−bp,bp][-b_p, b_p] after each optimizer update. We have explicitly clarified this rule in the revised manuscript. Furthermore, to ensure reproducibility and facilitate future research, we will release our implementation and experimental code alongside the revised version of the paper.


[Q1]. Was PLR run with PVL or MaxMC in each diagnostic and intervention experiment? If MaxMC...

See our response to CW-1 above.


[Q2]. Which network features, state samples, and return/target estimator define the representation ...

See our responses to CW-2 through CW-5 above.


[1]. No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery (Rutherford et al., NeurIPS 2024).

[2]. Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (Farebrother et al., ICML 2024).

[3]. Weight Clipping for Deep Continual and Reinforcement Learning (Elsayed et al., RLC 2024).

Dear Reviewer,

As the discussion period is nearing its end, we wanted to kindly check whether our response has addressed your concerns. If any questions remain, we would be happy to provide further clarification. Otherwise, we would greatly appreciate it if you could consider revisiting your assessment and updating your score accordingly. Thanks for reviewing our work.

Reviewer_VzpGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper answers the question on whether the student in UED stays trainable under the curriculum it induces, rather than how to build better curricula. Authors show multiple learner-side pathologies that worsen with complexity and training length. Authors argue that learner trainability is a complementary design principle for UED and that stabilization changes the curriculum PLR realizes even thought the method is unchanged.

Critical strengths
  1. Useful insight on learner trainability.
  2. Gains seem to hold across four curriculum mechanisms and they are substantial.
Critical weaknesses
  1. Figure 1 indicates that PLR is more strained than DR, but the gain magnitude for adaptive versus fixed curricula is never compared.

  2. Some of the key relevant works are not cited: Wenjun et al. IJCAI 2023. “ Generalization through Diversity: Improving Unsupervised Environment Design”. IJCAI 2023. Teoh et al. NeurIPS 2024:“Improving Environment Novelty Quantification for Effective Unsupervised Environment Design” NeurIPS, 2024.

  3. Baseline tuning and selection protocols are not exhaustive.

  4. The scope of the experiments is a bit narrow, focussed primarily on Kinetix.

  5. Trainability measure is never operationalized for trainability to be a design principle.

  6. Significance tests are missing to indicate statistical significance.

Questions for authors

Please refer to weaknesses above.

5 comments

We thank the reviewer for their review. We appreciate that they found the paper “clear and easy to follow,” reported learning “something interesting or useful,” and had “no substantive concerns about the paper’s soundness.” We are encouraged by their recognition of the “useful insight on learner trainability” and their observation that the gains “seem to hold across four curriculum mechanisms and they are substantial.” We address their concerns and questions below.


[CW-1]. Figure 1 indicates that PLR is more strained than DR, but the gain magnitude ...

We thank the reviewer for bringing up this point. We compare two approaches to constructing the training distribution in our UED setting. DR provides a non-adaptive baseline that samples from a fixed level distribution, whereas PLR adapts level selection using student-dependent replay scores. PLR therefore introduces an additional source of non-stationarity beyond the evolving policy and value targets already present in PPO. Figure 1 shows that PLR achieves higher sampled return while exhibiting more aligned representations and weaker value–target ordering than DR. Our goal is to highlight that the performance benefits of adaptive level selection can coexist with less favorable learner diagnostics, making learner trainability an important complementary consideration when evaluating UED methods.

We agree that directly comparing the magnitude of stabilization gains strengthens this analysis. We therefore provide a plot comparing baseline and stabilized performance for DR and PLR at matched interaction budgets across Small, Medium, and Large environments. Table 1 further quantifies the improvements in solve rate at the final training step. We compute the absolute gain as Δ =Treatment-Baseline and the relative improvement as 100 x Δ /Baseline, using the reported mean values.

Table 1. Performance improvements from learner stabilization under adaptive (PLR) and non-adaptive (DR) training distributions across Kinetix environment complexities. Results are reported as mean ± standard deviation over five runs at the final training step. Δ denotes the absolute improvement in solve rate, and relative improvements are computed with respect to the baseline.

SizeMethodBaseline (Mean ± SD)Treatment (Mean ± SD)ΔImprovement (%)
SPLR0.536 ± 0.0070.728 ± 0.086+0.192+35.8%
MPLR0.228 ± 0.0260.369 ± 0.077+0.141+61.8%
LPLR0.075 ± 0.0130.128 ± 0.014+0.053+70.7%
SDR0.489 ± 0.0380.802 ± 0.104+0.313+64.0%
MDR0.249 ± 0.0270.315 ± 0.035+0.066+26.5%
LDR0.080 ± 0.0130.096 ± 0.030+0.016+20.0%

Results show that stabilization gains differ across curriculum mechanisms as environment complexity increases. Notably, the relative gains under adaptive PLR increase with complexity, from 35.8% in Small to 61.8% in Medium and 70.7% in Large, whereas the opposite trend is observed under non-adaptive DR, where improvements decrease from 64.0% to 26.5% and 20.0%, respectively. While DR exhibits larger absolute gains in Small (+0.313 versus +0.192), PLR benefits more strongly in Medium (+0.141 versus +0.066) and Large (+0.053 versus +0.016). These results suggest that the relative benefits of learner stabilization depend on the curriculum mechanism, with adaptive PLR exhibiting comparatively larger gains in more challenging environments. This reinforces our central argument that learner trainability should be considered alongside curriculum construction when evaluating UED methods.


[CW-2]. Some of the key relevant works are not cited: Wenjun et al. IJCAI 2023. ..

We thank the reviewer for these helpful suggestions. We have added both references to the revised manuscript. The updates are marked in blue.

[CW-3]. Baseline tuning and selection protocols are not exhaustive.

We acknowledge that our baseline tuning and selection protocols do not exhaustively cover all possible configurations. As discussed in our Limitations section, hyperparameter sensitivity and intervention interactions constrain causal attribution, and broader validation remains an important direction for future work. Our intention is not to claim optimality for the selected baselines or stabilization techniques, but to investigate recurring learner-side pathologies across multiple UED mechanisms and settings.

We would like to highlight that our baseline configurations follow the established experimental protocol of [1], which reports the UED baselines and their corresponding tuned hyperparameters. For the stabilization interventions, we start from commonly used hyperparameter values reported in previous work [2,3,4,5] and evaluate values above and below these defaults across multiple random seeds to assess their sensitivity and select effective configurations. Nevertheless, as discussed in [7], deep RL can be highly sensitive to hyperparameter choices. While our evaluation does not constitute an exhaustive search, the consistent improvements across multiple curriculum mechanisms, environment complexities, and training regimes support the robustness of our findings. We have explicitly clarified these details in the revised manuscript, with all additions and modifications highlighted in blue.


[CW-4]. The scope of the experiments is a bit narrow, focussed primarily on Kinetix.

We thank the reviewer for raising this concern, and agree that evaluation across additional environment families would strengthen the generality of our conclusions. However, we emphasize that Kinetix was deliberately chosen because several properties make it particularly well suited to studying learner trainability under adaptive curricula. Unlike many UED benchmarks that primarily vary layouts within a relatively homogeneous task family, Kinetix supports a broad range of physics-based control problems, including locomotion, manipulation, navigation, and physical reasoning, within a unified environment framework. Its Small, Medium, and Large configurations provide an interpretable axis of structural complexity, allowing us to examine how optimization pathologies vary with environment complexity under consistent task and learning formulations.

Additionally, Kinetix [1] provides an established experimental protocol, including documented PPO and UED configurations and held-out human-designed evaluation tasks. Importantly, its hardware-accelerated implementation makes it feasible to study training over billions of interactions, which is essential for investigating pathologies that emerge or become more pronounced over extended training horizons. The benchmark remains challenging, particularly at higher complexity levels, avoiding widespread performance saturation that could obscure the effects of learner stabilization. Although our experiments are concentrated in Kinetix, they extend beyond a single curriculum mechanism. We evaluate PLR, SFL, ACCEL, and domain randomization across three environment complexity levels, and further examine extended training horizons, rollout parallelism, cross-complexity generalization, and sequential distribution shifts. The consistent benefits of learner stabilization across these settings suggest that our findings are not specific to a particular UED mechanism or training regime. We nevertheless acknowledge that these results do not establish generality across fundamentally different environment families, and view such validation as an important direction for future work. We have stated this precisely in the revised version, with the corresponding changes highlighted in blue.

[CW-5]. Trainability measure is never operationalized for trainability to be a design principle.

In our work, we use learner trainability to refer to the student's ability to continue benefiting from additional training experience, rather than to a single optimization statistic. This perspective is motivated by prior work on neural network plasticity and trainability [8,9], but we examine its implications specifically in the context of UED. Unlike level learnability, which estimates the learning potential of individual environments, learner trainability concerns the student's ability to continue improving under the curriculum it experiences.

We assess learner trainability empirically through learning progress, measured by solve rate and return over extended training horizons, to determine whether additional environment interactions translate into sustained improvement. Our extended 5B-interaction experiment provides particularly clear evidence, showing that while the baseline stagnates and eventually exhibits declining returns, the stabilized learner continues to improve in return while maintaining its solve rate. We complement this behavioral evidence with optimization and representation diagnostics, including value–target rank correlation, effective rank, and representation similarity. These diagnostics serve as indicators of potential learner degradation rather than direct measures of trainability.

Our proposed design principle is therefore not that trainability can be captured by a universal scalar metric, but that the learner's continued ability to benefit from experience should be explicitly evaluated alongside curriculum quality. In practice, this motivates monitoring learning progress and optimization diagnostics over extended training horizons and considering learner-side stabilization as a complementary component of UED design. We have clarified this terminology in Section 3 of the revised manuscript.


[CW-6]. Significance tests are missing to indicate statistical significance.

Our evaluation already reports 95% bootstrap confidence intervals across random seeds for key performance metrics, including the aggregated results in Fig. 2. These intervals quantify uncertainty in the estimated performance metrics. However, we acknowledge that explicit statistical significance tests would provide further support for our conclusions. We have therefore conducted additional statistical significance tests for our main experiments (see the table below, which is also included in Appendix J, along with the corresponding discussion). The results provide further statistical support for our main claims, reinforcing the consistency of our findings. We have incorporated these analyses and the corresponding methodological details into Section J, "Statistical Significance of Stabilization Improvements," of the revised manuscript.

Table: Comparison between the PLR baseline and PLR + HL + WC across Kinetix environment complexities.

SizenBaselineHL + WCΔ (95% CI)Improvement (%)t(df)p-valuep-Holm
S50.5360.728+0.192 [+0.078, +0.306]+35.8% [+15, +57]4.67 (4)0.00960.0192
M50.2280.369+0.141 [+0.049, +0.234]+62.0% [+21, +103]4.24 (4)0.01320.0192
L50.0750.128+0.053 [+0.026, +0.080]+70.2% [+34, +106]5.44 (4)0.00560.0168

Note: Results are averaged over five runs per configuration. Confidence intervals are reported at the 95% level. Statistical significance is assessed using two-sided paired t-tests, with Holm-adjusted p-values reported to account for multiple comparisons. All improvements remain statistically significant after correction (p-Holm < 0.05).


[1]. Kinetix: Investigating the training of general agents through open-ended physics-based control tasks. (Matthews, ICLR, 2024)

[2]. Stable gradients for stable learning at scale in deep reinforcement learning (Castanyer, NeurIPS, 2025).

[3]. Weight clipping for deep continual and reinforcement learning (Elsayed, RLC, 2024).

[4]. Simplicial embeddings improve sample efficiency in actor–critic agents (Obando-Ceron, ICML, 2026).

[5]. In value-based deep reinforcement learning, a pruned network is a good network (Obando-Ceron, ICML, 2024).

[6]. Stop regressing: Training value functions via classification for scalable deep rl (Farebrother, ICML, 2024).

[7]. On the consistency of hyper-parameter selection in value-based deep reinforcement learning (Obando-Ceron, RLC, 2024).

[8]. Understanding Plasticity in Neural Networks (Lyle, ICMLR, 2023)

[9]. Learning Continually by Spectral Regularization (Lewandowski, ICLR, 2025)

Reviewer_x9fYOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates optimization instability in Unsupervised Environment Design (UED), focusing on whether reinforcement learning agents remain trainable under adaptive curricula. The authors identify several learner-side pathologies, including parameter growth, degraded value prediction, and reduced representational diversity. To address these issues, the paper evaluates existing stabilization techniques and combines weight clipping with HL-Gauss categorical value prediction. Experiments using PPO in the Kinetix environment demonstrate improved training stability and performance across PLR, SFL, ACCEL, and domain randomization. Additional evaluations investigate long-horizon training, rollout parallelism, zero-shot generalization, and performance retention under distribution shifts.

Critical strengths
  1. The paper highlights an important but relatively underexplored aspect of adaptive curriculum learning by shifting attention from curriculum construction to learner trainability. The analysis provides a useful perspective on how optimization instability may limit the effectiveness of otherwise informative curricula.
  2. The paper systematically investigates multiple optimization pathologies, including parameter growth, value prediction degradation, and representational collapse. The combination of diagnostic measurements and controlled stabilization interventions provides meaningful empirical support for the proposed perspective.
  3. The experimental evaluation is extensive, covering multiple curriculum mechanisms, environment complexities, training horizons, and generalization settings. The consistent improvements across these configurations strengthen the practical relevance of maintaining learner stability.
Critical weaknesses

The methodological novelty and causal interpretation could be further strengthened. The proposed stabilization approach primarily combines existing techniques, namely weight clipping and HL-Gauss, rather than introducing a new optimization mechanism. Although the experiments demonstrate consistent performance improvements, they do not fully establish whether the observed optimization pathologies arise specifically from adaptive curriculum mechanisms or reflect more general limitations of PPO training. Consequently, the distinct contribution of curriculum–learner coupling beyond known reinforcement learning optimization issues remains somewhat unclear.

Questions for authors

Could the authors further distinguish optimization degradation caused by adaptive curriculum mechanisms from instability already present in conventional PPO training, and clarify how the proposed findings extend beyond existing reinforcement learning stabilization results?

6 comments

We thank the reviewer for their thoughtful and constructive review. We appreciate that they found the paper “clear and easy to follow,” reported gaining “a substantially deeper understanding of the problem or question,” and had “no substantive concerns about the paper’s soundness.” We are encouraged that they recognized the value of “shifting attention from curriculum construction to learner trainability” and found that our analysis offers “a useful perspective on how optimization instability may limit the effectiveness of otherwise informative curricula.” We also appreciate their observation that the combination of diagnostic measurements and controlled interventions “provides meaningful empirical support for the proposed perspective,” and that the consistent gains across curriculum mechanisms, environment complexities, training horizons, and generalization settings “strengthen the practical relevance of maintaining learner stability.” We respond to their main concerns below.


[CW-1]. The methodological novelty and causal interpretation could be further strengthened...

We appreciate this observation and agree that the two components of our stabilization approach, weight clipping and HL-Gauss, are established techniques. In Section 4.1, we evaluate several interventions and use WC+HL as one working example of how to mitigate learner-side degradation in UED. This combination provides a concrete reference point for researchers evaluating other stabilization approaches. We have made this role explicit in Section 4.1 and mark the revised text in blue. Our primary goal is not to propose a novel or optimal stabilization method, but to identify and extensively characterize learner-side pathologies in UED, an aspect that has received considerably less attention than curriculum construction. As you also acknowledge, our extensive evaluation across multiple curriculum mechanisms, environment complexities, training horizons, and generalization settings demonstrates the practical relevance of learner stability. Our central hypothesis is that addressing these learner-side pathologies can not only improve the student's trainability but also enhance the effectiveness of adaptive curricula, without modifying their underlying generation mechanisms.

[CW-2 & Q1]. PPO instability and curriculum–learner coupling...

We agree that the identified pathologies have also been documented in conventional deep RL. Our contribution is to characterize how they manifest under adaptive curricula and demonstrate that intervening on the learner improves what existing UED mechanisms achieve. Importantly, under adaptive UED, learner degradation can affect not only how effectively experience is used, but also which environments are subsequently selected for training.

Under domain randomization (DR), the generator distribution is fixed, although policy updates and bootstrapped targets still make PPO training non-stationary. DR and PLR share the same underlying PPO learner, but PLR introduces an additional source of non-stationarity by adapting level selection through learner-generated replay scores. Figure 1 shows that PLR achieves higher sampled return than DR while exhibiting more aligned representations and weaker value–target ordering. Figure 3 further shows that greater environment complexity under PLR is associated with larger parameter growth and stronger deterioration in effective rank and value–target ordering, particularly in Large. These findings characterize how learner degradation varies across training regimes and show why performance alone provides an incomplete picture of learning under adaptive curricula. Although these comparisons do not isolate curriculum adaptation as the cause of individual optimization pathologies, they reveal a performance–optimization tension under adaptive replay that is not captured by return alone.

Figure 7 connects learner stabilization to curriculum dynamics. Under the same PLR rule, stabilization improves performance even though measured learnability follows different trajectories across complexities. It increases in Medium, remains broadly comparable late in Small, and decreases late in Large. The sampled environment composition also changes, with similar shape counts, comparable or higher joint counts, and moderate changes in thruster counts. These observations show that changing the learner affects both student-dependent curriculum statistics and aspects of the environments encountered during training. Thus, unlike non-adaptive level sampling, adaptive replay provides an additional feedback pathway through which changes in learner optimization can influence subsequent training experience. They demonstrate why curriculum statistics must be interpreted jointly with the learner’s optimization state and evaluation performance.

The controlled interventions establish the practical importance of this perspective. Stabilization yields higher performance across DR, PLR, SFL, and ACCEL at matched interaction budgets. Importantly, the gains differ substantially across curriculum mechanisms and environment complexities. In Medium, stabilization improves PLR solve rate by 61.8% (+0.141), compared with 26.5% (+0.066) for DR. In Large, the corresponding gains are 70.7% (+0.053) and 20.0% (+0.016) (See Sec. M for details). Conversely, DR benefits more strongly in Small. Notably, while baseline PLR performs slightly below DR in Medium and Large, this ordering reverses after stabilization. These results show that the measured relative effectiveness of adaptive replay depends on the learner's optimization configuration, particularly in more challenging environments. Over extended training, the stabilized learner continues improving as baseline performance deteriorates, and the benefits extend to held-out tasks through zero-shot generalization. Thus, the measured effectiveness and useful training horizon of an existing curriculum depend materially on the learner paired with it. Beyond demonstrating improvements from established RL stabilization techniques, our findings show that learner-side interventions affect the relative effectiveness of curriculum mechanisms and aspects of the training distribution realized under an unchanged adaptive selection rule. This is the additional design and evaluation insight our study brings to UED, supporting learner trainability as a complementary principle alongside curriculum construction. We have made this connection clearer in the revised manuscript.

Dear Reviewer,

As the discussion period is nearing its end, we wanted to kindly check whether our response has addressed your concerns. If any questions remain, we would be happy to provide further clarification. Otherwise, we would greatly appreciate it if you could consider revisiting your assessment and updating your score accordingly. Thanks for reviewing our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

We sincerely thank the Area Chair and all reviewers for their thoughtful evaluations and constructive feedback. We appreciate their recognition of the importance of learner trainability in Unsupervised Environment Design (UED), the systematic nature of our empirical investigation, and the substantial improvements demonstrated across curriculum mechanisms, environment complexities, and training regimes.


Strengths highlighted by the reviewers

  • x9fY (Rating: 3): "The analysis provides a useful perspective on how optimization instability may limit the effectiveness of otherwise informative curricula."
  • VzpG (Rating: 3): "Gains seem to hold across four curriculum mechanisms and they are substantial."
  • Z88z (Rating: 2): "The contribution is a systematic diagnosis and actionable empirical finding in UED; it need not introduce a new optimizer to be useful."
  • fXJf (Rating: 3): "The paper presents a meaningful empirical investigation of learner-side training stability in UED." Additional evidence and revisions

In response to the reviewers' feedback, we have substantially strengthened the manuscript through additional experiments and methodological clarifications. These include statistical significance tests with multiple-comparison correction (App. J), fixed-probe representation analyses (App. K), evaluations under alternative PLR replay scoring functions (App. L), and direct comparisons of stabilization gains under adaptive and non-adaptive training distributions (App. M). We have also clarified implementation details, corrected inconsistencies, and refined the interpretation and limitations of our findings. All these changes have been incorporated into the revised manuscript and are highlighted in blue.


Central scientific contribution

Our contribution extends beyond demonstrating improvements from existing RL stabilization techniques. While prior UED research has largely focused on constructing better curricula, we systematically investigate whether learners remain capable of benefiting from the distributions those curricula produce. We identify recurring optimization and representation pathologies associated with learning stagnation and demonstrate that mitigating them improves sustained performance and generalization without modifying the curriculum rules. Our findings reveal that learner stabilization also changes aspects of the training distribution realized under an unchanged adaptive curriculum rule, and that the relative effectiveness of curriculum mechanisms depends on the learner's optimization configuration. We believe that systematically identifying and characterizing these interactions is a meaningful scientific contribution, providing actionable insights into the limitations of existing UED methods and an empirical foundation for developing more effective approaches that jointly consider curriculum adaptation and learner trainability.


Discussion phase

We appreciate the reviewers' time and consideration of our rebuttal. While we did not receive further feedback from reviewer VzpG, reviewer z88z kindly revisited our response and indicated near the end of the discussion period that some concerns remained. We would have welcomed the opportunity to better understand these remaining concerns and provide any additional clarifications that might have been helpful.

We respectfully hope the Area Chair will consider the substantial additional experimental evidence and manuscript revisions alongside the original reviews.


We once again thank the Area Chair and reviewers for their time and constructive feedback. We hope our work will be evaluated as an empirical contribution that advances our understanding of the limitations of adaptive curricula and provides actionable insights for developing more reliable and effective UED systems.

Thanks!

  • Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.