- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates learner-side training instability in Unsupervised Environment Design (UED), arguing that the effectiveness of adaptive curricula depends not only on task selection but also on the student’s ability to sustain learning. Through empirical analyses of PPO agents in Kinetix, the authors identify several optimization pathologies associated with training stagnation, including parameter growth, degraded value prediction, and reduced representational diversity. They evaluate existing stabilization techniques and propose combining weight clipping (WC) with categorical value prediction (HL-Gauss) to mitigate these issues. Experiments across multiple UED methods (PLR, SFL, and ACCEL), environment complexities, and training regimes demonstrate improved performance and generalization without modifying the curriculum mechanisms or increasing environment interaction budgets.
Critical strengths
The paper presents a meaningful empirical investigation of learner-side training stability in UED. I appreciate its focus on an important aspect of adaptive curriculum learning that has received relatively less attention, as well as the systematic experimental evidence supporting the practical benefits of learner stabilization. The main strengths are as follows:
-
An important and complementary perspective on UED. Rather than focusing exclusively on how curricula are constructed, the paper highlights the importance of maintaining the student's ability to learn from evolving training distributions. This perspective is particularly relevant to adaptive curricula, where learner behavior and subsequent task selection are coupled.
-
A systematic empirical analysis of training instability. The authors examine multiple indicators of learner degradation, including parameter growth, value prediction quality, and representational diversity, across different environment complexities and training stages. These analyses provide useful empirical insights beyond conventional performance-based evaluations.
-
Consistent performance improvements supported by extensive experiments. The combination of weight clipping and HL-Gauss achieves substantial improvements across multiple UED methods. The evaluation further includes component ablations, extended training horizons, varying rollout parallelism, cross-complexity generalization, and sequential distribution shifts, providing reasonably strong evidence for the practical effectiveness and robustness of the approach.
Critical weaknesses
-
The methodological novelty is somewhat limited, and the interaction between curriculum adaptation and learner-side stabilization could be explored more deeply. The proposed approach combines two established techniques, weight clipping and HL-Gauss, whose benefits for training stability have already been studied in deep RL.
-
A deeper investigation of the interaction between UED methods and learner optimization would be valuable. Given the broader motivation of the paper, I wonder whether these two components could be studied in a more tightly coupled manner. In particular, it would be interesting to understand (i) how different UED methods contribute to specific forms of learner degradation, and whether targeted learner-side interventions could address these issues; and (ii) whether curriculum adaptation and learner stabilization could be jointly optimized. The reviewer views the current work as a valuable starting point and does not consider this a major weakness. Nevertheless, a discussion of these directions would help clarify the broader significance of the findings and motivate further research.
Questions for authors
The paper presents a relatively comprehensive empirical study, and the reviewer appreciates the extensive experiments and analyses provided. Nevertheless, a few aspects of the methodology and experimental setup remain unclear. The reviewer would appreciate clarification on the following points:
-
In Section 3 and Section 4.2, the authors discuss Positive Value Loss (PVL) as the priority signal used by PLR and interpret its relationship with learnability and curriculum dynamics. However, Table 3 in the appendix specifies MaxMC as the scoring function for PLR. Could the authors clarify which scoring function was actually used in the experiments? If MaxMC was used, how should the PVL-based analyses in Figures 4 and 7 be interpreted in relation to the actual replay priorities?
-
Appendix D.4 describes ReGen as zeroing weights below a threshold of , whereas the cited work by Kumar et al. (2023) proposes regenerative regularization through an L2 penalty toward the initial parameters. These appear to be different interventions. Could the authors clarify the exact implementation used and whether the reported comparison corresponds to the method proposed in the cited work?
-
The paper interprets declining effective rank and increasing representation similarity as indicators of learner degradation. However, these statistics may also be influenced by changes in the state distribution induced by the evolving policy and curriculum. In particular, Appendix E.7 indicates that representation similarity is measured on training batches, while the probe distribution used for effective rank is less clear. Could the authors clarify how these metrics are evaluated and whether similar trends can be observed on a fixed set of probe states? This would help distinguish changes in the learned representations from changes in the sampled state distribution.
-
Section 5 describes the experiment in Figure 9 as extending PLR training on Small environments from 2B to 5B interactions. However, the performance scale around 2B steps appears noticeably different from the corresponding Small-environment results reported earlier. Could the authors clarify whether the evaluation configuration and training settings are identical across these experiments?
The reviewer would be open to reconsidering the current rating if the authors can satisfactorily address the above concerns during the rebuttal.
7 comments
To resolve the discrepancy, we have clarified the distinction between active replay scores and diagnostic metrics in Sections 3 and 4.2 and Table 3. We have also included additional experiments using PVL-based replay in the new Appendix L, titled PVL-Based Replay: Optimization and UED Dynamics, allowing readers to compare the behavior of PLR under different replay scoring functions. Importantly, the replay mechanism and scoring function remain unchanged between the baseline and stabilized learners within each comparison.
In this new appendix, Figure 25 (Fig. 25 in the paper) shows that PVL-based replay produces performance comparable to MaxMC-based replay while exhibiting similar learner optimization dynamics across Small, Medium, and Large environments. The same optimization pathologies observed under MaxMC-based replay are also present in the PVL-based PLR baseline, whereas HL + WC consistently mitigates these effects. These results suggest that learner stabilization remains effective under both replay scoring functions.
We also provide an analysis showing that the UED metrics under PVL-based scoring exhibit trends similar to those observed under MaxMC. Notably, Figure 26 (Fig. 26 in the paper) shows that learnability exhibits a clearer separation between the PLR baseline and the HL + WC intervention, highlighting the benefits of stabilization in maintaining task learnability across different environment complexities.
Regarding the interpretation of PVL, a high value can initially indicate unexpectedly large positive value residuals and potential learning opportunities. However, its interpretation becomes less straightforward over extended training horizons and with increasing environment complexity. In particular, persistently high PVL alongside low learnability and poor performance suggests that positive value residuals may be inflated by critic noise rather than reflecting genuine learning potential.
[Q-2]. Appendix D.4 describes ReGen as zeroing weights below a threshold of ..
We apologize for the incorrect description in Appendix D.4. Our implementation follows the ReGen regularization approach proposed by Kumar et al. (2023) [3], using an L2 penalty that encourages network parameters to remain close to their initial values rather than penalizing deviations from zero. Specifically, the regularization loss is computed as the sum of squared deviations between the current network parameters and their corresponding initial values, with a regularization coefficient of . Thus, the implementation used in our experiments is consistent with the ReGen formulation; the inconsistency was limited to the mathematical description in the manuscript. We have corrected Appendix D.4 to accurately reflect the implemented method.
[Q-3] . The paper interprets declining effective rank and increasing representation similarity as indicators of learner ..
We thank the reviewer for pointing out this ambiguity. Both representation similarity and effective rank are computed from the same latent representations extracted from the neural network's training batches. We have clarified this in Appendix E.10 by explicitly specifying the source of the feature matrix used to compute effective rank, ensuring consistency in the descriptions of both metrics.
We agree that changes in the state distribution induced by the evolving policy and curriculum may influence representation diagnostics. To address this concern, we conducted an additional experiment using a fixed probe set shared across the baseline and stabilized learners. Specifically, we generated 2,048 random Kinetix levels using a fixed random seed and saved their initial states. These states remain unchanged throughout training and are used to evaluate both learners at periodic checkpoints. We compute effective rank and cosine similarity from the corresponding network representations using forward passes without gradient updates. This protocol allows us to assess whether the observed representation degradation persists when the evaluation input distribution is held fixed.
As shown in Figure 23 (Fig. 23 in the paper), representation collapse and degradation persist across all three environment complexities (S, M, and L) under fixed-probe evaluation. This indicates that the observed degradation cannot be explained solely by changes in the evaluation state distribution.
continued below
We also evaluated the HL + WC intervention using the same fixed-probe protocol. As shown in Figure 24 (Fig. 24 in the paper), HL + WC mitigates representation collapse across all three environment complexities, even when evaluated on the same fixed set of states. This preservation of representational quality is accompanied by improved learning performance, providing further evidence of an association between representation stability and agent performance.
All experimental details and additional analyses are provided in the newly added Section K, Representation Collapse: Empirical Analysis and Ablations, of the revised manuscript. We emphasize that these additional analyses do not establish a complete causal mechanism. Rather, they demonstrate that representation degradation persists under a fixed evaluation distribution, indicating that changes in the sampled evaluation states alone cannot explain the observed degradation.
[Q-4]. Section 5 describes the experiment in Figure 9 as extending PLR training on Small ..
We thank the reviewer for bringing this discrepancy to our attention. Figure 9 reports results for Medium (M) environments rather than Small (S) environments, as incorrectly stated in the text. We selected Medium environments for the extended 5B-interaction experiment because they remain more challenging and exhibit less performance saturation than Small environments, providing greater scope to investigate whether additional training leads to continued improvement or learner degradation. The difference in performance scale therefore arises from comparing results across different environment complexities. We have corrected the figure caption and the corresponding discussion in Section 5 to explicitly indicate that the experiment uses Medium environments, ensuring consistency with the earlier results.
The reviewer would be open to reconsidering the current rating if the authors can satisfactorily address the above concerns during the rebuttal.
We thank the reviewer for their constructive feedback and willingness to reconsider their assessment. We appreciate the opportunity to clarify these methodological details and strengthen the presentation of our results. We have addressed the identified inconsistencies and incorporated additional experimental analyses and clarifications into the revised manuscript, with all changes highlighted in blue. We hope these revisions satisfactorily address the reviewer's concerns.
[1] Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks (Matthews et al., ICLR 2025).
[2] No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery (Rutherford et al., NeurIPS 2024).
[3] Maintaining Plasticity in Continual Learning via Regenerative Regularization (Kumar et al., CoLLAs 2025).
[4] Weight Clipping for Deep Continual and Reinforcement Learning (Elsayed et al., RLC 2024).
[5] Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (Farebrother et al., ICML 2024).
We thank the reviewer for their detailed and encouraging assessment. We appreciate that they found the paper "Clear and easy to follow," reported gaining "a substantially deeper understanding of the problem or question," and had "no substantive concerns about the paper’s soundness." We are encouraged that they described our work as "a meaningful empirical investigation of learner-side training stability in UED" and "an important and complementary perspective on UED." We also appreciate their recognition of our "systematic empirical analysis of training instability" and "consistent performance improvements supported by extensive experiments," as well as their assessment that these experiments provide "reasonably strong evidence for the practical effectiveness and robustness of the approach." We address the reviewer's questions below and describe the corresponding clarifications and revisions to the manuscript.
[CW-1]. The methodological novelty is somewhat limited, and the interaction between curriculum adaptation ..
We appreciate this observation and agree that the two components of our stabilization approach, weight clipping [4] and HL-Gauss [5], are established techniques. In Section 4.1, we evaluate several interventions and use WC + HL-Gauss as a concrete example of how to mitigate learner-side degradation in UED. This combination provides a reference point for researchers evaluating other stabilization approaches. We have made this role explicit in Section 4.1, with the revised text highlighted in blue.
Our primary goal is not to propose a novel or optimal stabilization method, but to identify and systematically characterize learner-side pathologies in UED, an aspect that has received considerably less attention than curriculum construction. As the reviewer also acknowledges, our extensive evaluation across multiple curriculum mechanisms, environment complexities, training horizons, and generalization settings demonstrates the practical relevance of learner stability. Our central hypothesis is that addressing these learner-side pathologies can not only improve the student's trainability but also enhance the effectiveness of adaptive curricula without modifying their underlying generation mechanisms.
[CW-2]. A deeper investigation of the interaction between UED methods and learner optimization would be valuable.
We thank the reviewer for these suggestions and agree that understanding the interaction between curriculum adaptation and learner optimization is an important direction for future research. We have expanded the Discussion section of the revised manuscript to highlight these directions. Specifically, we discuss investigating whether different curriculum mechanisms induce distinct optimization pathologies that could benefit from targeted interventions and whether curriculum selection could incorporate learner-side stability or learning-progress signals to jointly improve curriculum adaptation and learner trainability. We believe these directions provide promising opportunities to extend our findings beyond the stabilization interventions studied here.
[Q1]. In Section 3 and Section 4.2, the authors discuss Positive Value Loss (PVL) as the priority signal ...
We thank the reviewer for identifying this important inconsistency. We confirm that MaxMC was the active scoring function used by PLR to prioritize replay in all original experiments, following the standard Kinetix benchmark configuration [1] to ensure comparability with published baselines. PVL and learnability were also logged throughout these experiments but served as diagnostic metrics rather than replay signals. Accordingly, the PVL-based analyses in Figures 4 and 7 should be interpreted as auxiliary diagnostics, not as analyses of the actual replay priorities. We acknowledge that this distinction was insufficiently clear.
We placed particular emphasis on PVL because prior work (e.g., Rutherford et al., 2024) [2] has highlighted limitations of positive advantage-based regret estimates, which may be affected by critic noise and become misaligned with actual learnability. Our results show that, particularly in more complex environments, baseline PVL can remain elevated despite low learnability and poor performance. Conversely, learner stabilization is associated with lower PVL alongside improvements in learnability and evaluation performance. Persistently high PVL in conjunction with low learnability and poor performance suggests that positive value residuals do not necessarily reflect genuine learning potential; our diagnostics do not isolate critic noise as the cause of this mismatch.
continued below