- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces FRONTIER-RSI, a method for dynamically adapting the set of coding environments on which RL training is performed. FRONTIER-RSI categorizes tasks into unsolved, frontier, and saturated and creates easier variants for the unsolved, harder variants for the saturated, and similarly hard variants for the frontier, in order for the next iteration of training to have a higher concentration of tasks at the frontier of the agent’s capabilities. They perform two experiments using different model sizes and initial set of tasks, and find that the model performance improves on evaluation benchmarks as training proceeds. Its performance exceeds that of baselines and of training without a curriculum.
Critical strengths
It tackles an interesting problem, and the proposed mechanisms for creating harder variants of saturated tasks and easier variants of unsolved tasks are reasonable.
Critical weaknesses
Contribution.
The novelty in the main text is not clearly presented. Using a curriculum that favors tasks where difficulty is at an intermediate level is not novel (as recognized by the authors). Moreover, past methods have already used success rates from observed rollouts to estimate difficulty, a connection I don’t think is adequately acknowledged in appendix G.1. The main contribution to me seems to be the exact mechanism by which coding tasks are altered to make variants that are appropriately hard for the next training iteration. I don’t think the paper is clearly framed to reflect this. I also don’t like the strong framing as an RSI method, since it relies on using a stronger model for modifying the problems used for training.
Experiments.
- I would like to see more comparisons to other methods centered around shaping the environments used for RL training, leaving all other conditions matched or as close as possible (same learner, budget, etc), since that is where the contribution of the method lies. In particular, SSR seems to be the only baseline that also uses the learner's current capabilities to guide task generation.
- The differences in cost between running the method and the baselines is not clear.
- I don’t think the ablations are sufficient. The contributions from the different components of the method to the observed gains are not well studied. Table 3 compares only against using a fixed distribution of tasks. Also, the ‘Ablation on long-horizon coding task ‘ seems to me more like another set of main results than an ablation.
- In the experiments, the adaptation of environments is only applied twice throughout training. To me this seems like too small of a number for validating a curriculum/RSI method. Do the improvements continue if run for longer?
- There is mention in the appendix of running several training seeds, however I did not find reporting of how these were aggregated into the final results or on their variability. Could you clarify this?
Presentation. I think the overall presentation and writing aren't good. This concerns both the organization and selection of material presented in the main text and the quality of the writing throughout the paper.
Questions for authors
None.
5 comments
Q5: Ablations
A5: Thanks for your comment. Our core modules are designed around a unified objective: effectively advancing agent capabilities through RL by providing richer and more informative training signals, rather than maximizing the isolated contribution of any individual component.
We initially omitted module-removal ablations because, under our training setup, matching the number of training examples across settings would require reusing the tasks associated with the removed module. Such repeated exposure could itself affect training performance, making it harder to separate the module’s direct contribution from the effects of task repetition.
Nevertheless, to address this concern, we can conduct fine-grained ablations that independently remove Prefix Scaffolding (PS), Difficulty Escalation (DE), while keeping the learner, RL training configuration, and optimization budget matched. The experiments are conducted during the second training round of Qwen3-8B on the TMax dataset, corresponding to the first RSI iteration. The results below show that removing key modules reduces the performance gains achieved through RSI. These ablation results demonstrate the contribution of these modules to the effectiveness of our framework.
| Method | Pass@1 | Pass@4 | Pass@8 |
|---|---|---|---|
| Ours w/o PS | 4.2 | 10.5 | 14.6 |
| Ours w/o DE | 4.4 | 10.7 | 16.9 |
| Ours | 4.5 | 11.9 | 18.0 |
Furthermore, to distinguish environment transformation from curriculum selection, we include a frontier-only sampling (FS) baseline that prioritizes tasks based on the current policy's success rate without modifying their executable environments. This comparison directly examines whether policy-conditioned environment transformations (i.e., prefix scaffolding, difficulty escalation) provide additional benefits beyond selecting intermediate-difficulty tasks. Starting from the same Round-1 checkpoint, we construct it by selecting original TMax tasks whose empirical success rates fall within the same frontier interval used by our method. Their instructions, initial environments, and verifiers remain unchanged. We train on these tasks using fresh rollouts while keeping the RL algorithm, dynamic filtering, and accepted-trajectory optimization budget identical to those of the full method. This comparison evaluates whether environment transformation provides additional gains over success-rate-based curriculum selection during the first RSI iteration. These results below demonstrate the substantial benefits of environment transformation. We have also previously observed that model performance declines after more than four epochs of training on the same environments, and environment transformation helps mitigate this degradation.
| Method | Pass@1 | Pass@4 | Pass@8 |
|---|---|---|---|
| Ours w. FO | 4.1 | 10.2 | 12.4 |
| Ours | 4.5 | 11.9 | 18.0 |
We will also analyze the task-state transitions induced by each operator, including Unsolved-to-Frontier and Saturated-to-Frontier conversion rates, to better explain how the proposed transformations maintain an effective training curriculum.
Finally, we agree that the experiments on Qwen3.6-35B-A3B with Cross-PR demonstrate the generalizability of our framework across model and task configurations, rather than constituting a component-level ablation. We will reorganize this section accordingly.
Q6: Iteration number
A6: Thank you for this suggestion. We extend the experiment to a fourth training round. The fourth round yields only a marginal improvement over Round 3, indicating diminishing marginal returns under the current setup. The earlier rounds capture the more readily attainable gains, while the current seed task pool and evolution operators provide fewer new learning opportunities as the policy improves. This suggests that sustaining further gains may require a broader and more diverse task pool or richer evolution operators. High-quality, diverse training data fuel post-training, and expanding the pool of verifiable tasks may provide new learning opportunities that sustain improvement over more iterations.
| Method | Pass@1 | Pass@4 | Pass@8 |
|---|---|---|---|
| +Round 3 | 5.6 | 13.8 | 22.5 |
| +Round 4 | 5.8 | 14.0 | 23.6 |
Q7: Training seeds
A7: Thank you for raising this point. The three-seed experiments in Appendix F.5 were conducted to assess the practical feasibility and optimization stability of our RL training recipe. For each recipe, we randomly select three training seeds and examine whether training remains stable and the reward exhibits sustained improvement across these runs. Consistent behavior across the three seeds provides empirical support that the recipe can reliably support RL optimization under the tested configuration. We will clarify this purpose and the criterion used to assess recipe viability in Appendix F.5.
Thank you for your responses and the additional experiments. I appreciate the new ablations, which help address some of my concerns, and the extension to a fourth training round, which is informative despite suggesting diminishing returns. However, I still find that several of my main concerns, particularly regarding the presentation, framing of the contribution, and the experimental evaluation and reporting of results, remain insufficiently addressed. I will therefore maintain my original recommendation.
We sincerely thank the reviewer for the constructive feedback and apologize for the delayed response, as the additional ablation experiments on post-training require three to four days per run.
A1: Thanks for your comment. The key to RSI is not whether all information originates from within the system itself, but rather: whether improvements made in one iteration are retained and leveraged to drive subsequent improvements, thereby forming a closed-loop process of RSI. The relevant distinction is between external assistance in an improvement loop and external control over the improvement process.
This distinction is consistent with prior literature. For example, Darwin Gödel Machine studies self-improving agents that use a foundation model to generate agent modifications and external coding benchmarks to evaluate them. Similarly, a recent survey [1] of 1,250 papers on RSI, characterize the field along two separate dimensions: the target of improvement and the degree of loop closure, rather than requiring the absence of external feedback or auxiliary models.
Therefore, the stronger model is an auxiliary resource for constructing candidate environments, not the sole driver of the RSI process.
[1] Chen M, Wang L, Qu B. Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops[J]. arXiv preprint arXiv:2607.07
A2: Thanks for your comment. We agree that the concrete environment-transformation mechanism should be the central contribution of the paper. Our presentation did not sufficiently distinguish this contribution from the established principles of adaptive curriculum learning. Specifically, our method constructs verified intermediate workspace states for difficult tasks while preserving the original instruction and verifier, creates recoverable workspace perturbations or explicit requirement extensions for saturated tasks, and combines these transformations with consistent task rewriting. Candidates undergo executable validation and calibration against the current learner before entering the next training round.
A3: Thanks for your comment. We would like to clarify that environment RSI for coding agents is a very recent research direction. We conducted a thorough literature review before our submission and included representative methods available at that time.
Only four paper is related. Specifically, we experimentally compare against SSR and RST. Among other closely related studies available before submission, Environment Evolution for Terminal Agents explores difficulty escalation for terminal tasks. However, we could not identify an official implementation enabling faithful reproduction under our training setup. SETA proposes learner-calibrated environment evolution, but its reported experiments primarily focus on environment construction followed by RL training, rather than demonstrating repeated policy updates interleaved with environment re-evolution.
We are still open to make any further comparisons if you suggest.
A4: Thanks. We would like to clarify the relative scale of training and teacher-inference costs. Appendix F has already provided the cost between different methods. Actually, the average strong model guidance cost is below US$1 per task across the compared task-synthesis methods. However, for a run with approximately 1k tasks, this amounts to less than US$1,000 in total expenditure, compared with approximately US$17–20k for training a single run on 4 H200 nodes and 384 cpu node. Strong teacher model therefore costs less than approximately 6% of the training expenditure and represents a relatively small overhead. Training remains the dominant cost component in this setting.