acceptodds
Under review as a conference paper at ICLR 2027

Frontier-RSI: Advancing the Agent Capability Frontier through Continual Training and Environment Evolution

Abstract

Recursive self-improvement (RSI) seeks to enable agents to continually expand their capabilities through iterative learning and evolution. However, existing approaches largely treat evolution as an open-loop process: as the agent improves, the evolving training environments are not systematically adapted to its changing capabilities, leaving no closed loop that continually pushes the agent’s capability frontier. We introduce Frontier-RSI, a closed-loop framework that couples agent continual learning, policy-relative task states, and environment evolution. After each learning round, Frontier-RSI measures each task relative to the updated agent and categorizes it as unsolved, frontier, or saturated. A curriculum operator then adapts the environment through two complementary knobs, difficulty and scaffolding, to continuously reshape the task distribution around the agent’s learnable frontier. Specifically, saturated tasks that have become too easy are evolved into harder escalation tasks, while unsolved tasks that remain beyond the agent’s reach are transformed into learnable prefix tasks by providing verified intermediate progress. Experiments across foundation models with different capability levels and coding tasks of varying difficulty demonstrate that Frontier-RSI consistently sustains improvement across successive rounds, effectively advancing the agent’s capability frontier. Code is open-sourced at https://anonymous.4open.science/r/Frontier-RSI-1C0D/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Recursive Self-improvement

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_zr8zOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces FRONTIER-RSI, a method for dynamically adapting the set of coding environments on which RL training is performed. FRONTIER-RSI categorizes tasks into unsolved, frontier, and saturated and creates easier variants for the unsolved, harder variants for the saturated, and similarly hard variants for the frontier, in order for the next iteration of training to have a higher concentration of tasks at the frontier of the agent’s capabilities. They perform two experiments using different model sizes and initial set of tasks, and find that the model performance improves on evaluation benchmarks as training proceeds. Its performance exceeds that of baselines and of training without a curriculum.

Critical strengths

It tackles an interesting problem, and the proposed mechanisms for creating harder variants of saturated tasks and easier variants of unsolved tasks are reasonable.

Critical weaknesses

Contribution.

The novelty in the main text is not clearly presented. Using a curriculum that favors tasks where difficulty is at an intermediate level is not novel (as recognized by the authors). Moreover, past methods have already used success rates from observed rollouts to estimate difficulty, a connection I don’t think is adequately acknowledged in appendix G.1. The main contribution to me seems to be the exact mechanism by which coding tasks are altered to make variants that are appropriately hard for the next training iteration. I don’t think the paper is clearly framed to reflect this. I also don’t like the strong framing as an RSI method, since it relies on using a stronger model for modifying the problems used for training.

Experiments.

  • I would like to see more comparisons to other methods centered around shaping the environments used for RL training, leaving all other conditions matched or as close as possible (same learner, budget, etc), since that is where the contribution of the method lies. In particular, SSR seems to be the only baseline that also uses the learner's current capabilities to guide task generation.
  • The differences in cost between running the method and the baselines is not clear.
  • I don’t think the ablations are sufficient. The contributions from the different components of the method to the observed gains are not well studied. Table 3 compares only against using a fixed distribution of tasks. Also, the ‘Ablation on long-horizon coding task ‘ seems to me more like another set of main results than an ablation.
  • In the experiments, the adaptation of environments is only applied twice throughout training. To me this seems like too small of a number for validating a curriculum/RSI method. Do the improvements continue if run for longer?
  • There is mention in the appendix of running several training seeds, however I did not find reporting of how these were aggregated into the final results or on their variability. Could you clarify this?

Presentation. I think the overall presentation and writing aren't good. This concerns both the organization and selection of material presented in the main text and the quality of the writing throughout the paper.

Questions for authors

None.

5 comments

We sincerely thank the reviewer for the constructive feedback and apologize for the delayed response, as the additional ablation experiments on post-training require three to four days per run.

Q1: RSI Concept

A1: Thanks for your comment. The key to RSI is not whether all information originates from within the system itself, but rather: whether improvements made in one iteration are retained and leveraged to drive subsequent improvements, thereby forming a closed-loop process of RSI. The relevant distinction is between external assistance in an improvement loop and external control over the improvement process.

This distinction is consistent with prior literature. For example, Darwin Gödel Machine studies self-improving agents that use a foundation model to generate agent modifications and external coding benchmarks to evaluate them. Similarly, a recent survey [1] of 1,250 papers on RSI, characterize the field along two separate dimensions: the target of improvement and the degree of loop closure, rather than requiring the absence of external feedback or auxiliary models.

Therefore, the stronger model is an auxiliary resource for constructing candidate environments, not the sole driver of the RSI process.

[1] Chen M, Wang L, Qu B. Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops[J]. arXiv preprint arXiv:2607.07

Q2: Novelty

A2: Thanks for your comment. We agree that the concrete environment-transformation mechanism should be the central contribution of the paper. Our presentation did not sufficiently distinguish this contribution from the established principles of adaptive curriculum learning. Specifically, our method constructs verified intermediate workspace states for difficult tasks while preserving the original instruction and verifier, creates recoverable workspace perturbations or explicit requirement extensions for saturated tasks, and combines these transformations with consistent task rewriting. Candidates undergo executable validation and calibration against the current learner before entering the next training round.

Q3: More comparisons

A3: Thanks for your comment. We would like to clarify that environment RSI for coding agents is a very recent research direction. We conducted a thorough literature review before our submission and included representative methods available at that time.

Only four paper is related. Specifically, we experimentally compare against SSR and RST. Among other closely related studies available before submission, Environment Evolution for Terminal Agents explores difficulty escalation for terminal tasks. However, we could not identify an official implementation enabling faithful reproduction under our training setup. SETA proposes learner-calibrated environment evolution, but its reported experiments primarily focus on environment construction followed by RL training, rather than demonstrating repeated policy updates interleaved with environment re-evolution.

We are still open to make any further comparisons if you suggest.

Q4: Cost

A4: Thanks. We would like to clarify the relative scale of training and teacher-inference costs. Appendix F has already provided the cost between different methods. Actually, the average strong model guidance cost is below US$1 per task across the compared task-synthesis methods. However, for a run with approximately 1k tasks, this amounts to less than US$1,000 in total expenditure, compared with approximately US$17–20k for training a single run on 4 H200 nodes and 384 cpu node. Strong teacher model therefore costs less than approximately 6% of the training expenditure and represents a relatively small overhead. Training remains the dominant cost component in this setting.

Q5: Ablations

A5: Thanks for your comment. Our core modules are designed around a unified objective: effectively advancing agent capabilities through RL by providing richer and more informative training signals, rather than maximizing the isolated contribution of any individual component.

We initially omitted module-removal ablations because, under our training setup, matching the number of training examples across settings would require reusing the tasks associated with the removed module. Such repeated exposure could itself affect training performance, making it harder to separate the module’s direct contribution from the effects of task repetition.

Nevertheless, to address this concern, we can conduct fine-grained ablations that independently remove Prefix Scaffolding (PS), Difficulty Escalation (DE), while keeping the learner, RL training configuration, and optimization budget matched. The experiments are conducted during the second training round of Qwen3-8B on the TMax dataset, corresponding to the first RSI iteration. The results below show that removing key modules reduces the performance gains achieved through RSI. These ablation results demonstrate the contribution of these modules to the effectiveness of our framework.

MethodPass@1Pass@4Pass@8
Ours w/o PS4.210.514.6
Ours w/o DE4.410.716.9
Ours4.511.918.0

Furthermore, to distinguish environment transformation from curriculum selection, we include a frontier-only sampling (FS) baseline that prioritizes tasks based on the current policy's success rate without modifying their executable environments. This comparison directly examines whether policy-conditioned environment transformations (i.e., prefix scaffolding, difficulty escalation) provide additional benefits beyond selecting intermediate-difficulty tasks. Starting from the same Round-1 checkpoint, we construct it by selecting original TMax tasks whose empirical success rates fall within the same frontier interval used by our method. Their instructions, initial environments, and verifiers remain unchanged. We train on these tasks using fresh rollouts while keeping the RL algorithm, dynamic filtering, and accepted-trajectory optimization budget identical to those of the full method. This comparison evaluates whether environment transformation provides additional gains over success-rate-based curriculum selection during the first RSI iteration. These results below demonstrate the substantial benefits of environment transformation. We have also previously observed that model performance declines after more than four epochs of training on the same environments, and environment transformation helps mitigate this degradation.

MethodPass@1Pass@4Pass@8
Ours w. FO4.110.212.4
Ours4.511.918.0

We will also analyze the task-state transitions induced by each operator, including Unsolved-to-Frontier and Saturated-to-Frontier conversion rates, to better explain how the proposed transformations maintain an effective training curriculum.

Finally, we agree that the experiments on Qwen3.6-35B-A3B with Cross-PR demonstrate the generalizability of our framework across model and task configurations, rather than constituting a component-level ablation. We will reorganize this section accordingly.

Q6: Iteration number

A6: Thank you for this suggestion. We extend the experiment to a fourth training round. The fourth round yields only a marginal improvement over Round 3, indicating diminishing marginal returns under the current setup. The earlier rounds capture the more readily attainable gains, while the current seed task pool and evolution operators provide fewer new learning opportunities as the policy improves. This suggests that sustaining further gains may require a broader and more diverse task pool or richer evolution operators. High-quality, diverse training data fuel post-training, and expanding the pool of verifiable tasks may provide new learning opportunities that sustain improvement over more iterations.

MethodPass@1Pass@4Pass@8
+Round 35.613.822.5
+Round 45.814.023.6

Q7: Training seeds

A7: Thank you for raising this point. The three-seed experiments in Appendix F.5 were conducted to assess the practical feasibility and optimization stability of our RL training recipe. For each recipe, we randomly select three training seeds and examine whether training remains stable and the reward exhibits sustained improvement across these runs. Consistent behavior across the three seeds provides empirical support that the recipe can reliably support RL optimization under the tested configuration. We will clarify this purpose and the criterion used to assess recipe viability in Appendix F.5.

Thank you for your responses and the additional experiments. I appreciate the new ablations, which help address some of my concerns, and the extension to a fourth training round, which is informative despite suggesting diminishing returns. However, I still find that several of my main concerns, particularly regarding the presentation, framing of the contribution, and the experimental evaluation and reporting of results, remain insufficiently addressed. I will therefore maintain my original recommendation.

Reviewer_iggfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FRONTIER-RSI is a closed-loop framework for recursive self-improvement that ties together agent continual learning, policy-relative task states, and environment evolution. The motivating point is that prior work evolves environments open-loop — making tasks harder or more numerous without re-shaping them around the agent's current ability — so nothing keeps steering the agent toward its learnable frontier. Here the loop is closed: after each training round the updated policy measures pass@k on every task and labels it Unsolved, Frontier, or Saturated. A curriculum operator reads how tasks flow between these states and turns two knobs, difficulty and scaffolding: saturated (too-easy) tasks are escalated with recoverable obstacles or stronger requirements, unsolved tasks are turned into prefix tasks by injecting verified intermediate progress, and frontier tasks are refreshed by rewriting the instruction, solution, and verifier. The method is evaluated on Qwen3-8B and Qwen3.6-35B-A3B over terminal and software-engineering coding tasks, using Terminal-Bench 2, TerminalWorld, SWE-Bench Verified, and SWE-Bench Pro. The authors report sustained improvement across rounds (mean 2.9 points / +16.3% over the strongest baseline after three rounds), an ablation showing performance stalls or regresses without evolution, and a training-dynamics analysis in which evolved tasks make policy entropy recover.

Critical strengths
  • The core idea is clean and well-motivated. Measuring where the learner currently sits and then evolving the environment toward that boundary is a simple move, but it isolates the failure mode of open-loop curricula, and the U/F/S task states give it a concrete, operational handle. The two operators (prefix for what is out of reach, escalation for what is saturated) follow directly from this framing rather than being added on top.

  • The ablation isolates the effect of environment evolution. Removing evolution (Table 3) does not merely slow progress: Round 3 falls below Round 1, with SWE-Bench Pro dropping from 5.0 to 2.0, which indicates that the evolution step drives the gains rather than continued RL alone. The entropy curve in Figure 3 (a dip to ~0.17, then recovery to ~0.24 after each re-evolution) provides a consistent mechanistic account, and the mechanism and headline numbers agree.

  • The held-out task-state analysis provides strong evidence of real capability gain. Because the held-out tasks in Figure 4 are never evolved, the shift from 48%→31% Unsolved and 20%→31% Saturated, with the Frontier band holding at 32–38%, is a direct readout that the policy itself moved rather than an artifact of a changing test distribution. This is a more convincing form of evidence than the aggregate benchmark deltas.

Critical weaknesses
  • The main results lack any measure of variance, and the deciding numbers are small enough that significance is unclear. The paper reports no seed count, no error bars, and no significance test, while the key cells are very low in absolute terms: Qwen3-8B Pass@1 on Terminal-Bench 2 moves 3.2 → 5.6, and SWE-Bench Pro Pass@1 moves across 2.0 / 4.0 / 6.0 / 8.0, where flipping a single task changes the reported score. The large relative percentages (+27%, +100%) are largely an artifact of these small denominators. For a high-variance metric such as pass@k on a 100-task subset, the 2.9-point mean gap over the strongest baseline cannot be established without a measure of spread.

  • The claim of continual improvement is in tension with the reported trajectory, and baseline comparability is not established. The validation reward rises +0.17, then +0.025, then +0.015, and the paper itself notes that a fixed seed pool "has a ceiling and cannot sustain unbounded improvement." After three rounds the curve has essentially flattened, so the "continually advancing the frontier" framing of the title and abstract would be better scoped to bounded in-pool advancement over a limited number of rounds. In addition, the baselines (Meta-Harness, HarnessBridge, SSR, RST) are all reproductions with little detail on fidelity or matched compute, leaving the "beats the strongest baseline" comparison not fully auditable.

  • The stated scope is broader than the experimental evidence supports. All results come from a single model family and a single domain, and the full pipeline — pass@k task states, prefix validation to a 0.99 verifier reward, and verifier-checked escalation — presumes an executable sandbox with a reliable automatic verifier, a condition that does not hold for most agent settings. The contribution is well supported as a method for verifiable coding environments, but the generality implied by the title and introduction is not demonstrated.

Questions for authors
  1. In Table 3, "w/o Evolution" drops from the shared 15.7 at Round 1 to 14.6 and then 13.1 — below where it started — while SSR and RST keep rising over the same rounds. Why does the no-evolution variant degrade so far below its starting point, and to what extent is environment evolution rescuing an unstable continued-RL setup rather than providing a general advantage? Relative to the methods that do not collapse (SSR 17.8, RST 17.1), how large is the margin?

  2. The gains are much larger on Qwen3-8B (+31.8% mean) than on Qwen3.6-35B-A3B (+8.4%), but the two settings also use different data (TMax vs. Cross-PR), so model scale and dataset are confounded. Does the method provide a substantial benefit on the stronger model, or is most of the gain concentrated in lifting the weaker one? Would a shared dataset across both models separate these factors?

  3. Several deciding cells are very small integers (Terminal-Bench 2 Pass@1 at 4.4/4.5/5.6; SWE-Bench Pro at 2.0/4.0/6.0/8.0), where a single task changes the number. How many seeds or rollout groups underlie each cell, and can standard deviations or a significance test be provided for the Ours-vs-best-baseline mean? Given the +0.015 final step, what evidence indicates that the frontier is still advancing at Round 3 rather than holding in place?

  4. The escalation operator builds harder variants from "open-source coding datasets." Which datasets are used, and have they been checked for overlap with SWE-Bench Verified/Pro, which are themselves drawn from open-source GitHub issues? What steps rule out leakage between evolved training content and the evaluation sets?

3 comments

We sincerely thank the reviewer for the constructive feedback and apologize for the delayed response, as the additional ablation experiments on post-training require three to four days per run.

Q1: Statistical robustness and uncertainty

A1: Thank you for raising this concern. We select Qwen3-8B to study iterative improvement in a setting with substantial learning headroom. To assess the stability of our training recipe, we conduct experiments with three randomly selected training seeds and examine whether RL training remains stable and rewards exhibit sustained improvement, as discussed in Appendix F.5. These checks provide empirical support for the recipe’s optimization stability under the tested configuration. We agree, however, that training stability alone does not quantify uncertainty in the reported benchmark differences.

Q2: Scope of RSI and Applicability to Coding Agents

A2: Thanks for your comment. We agree with the reviewer that our framing should more closely reflect the experimental evidence. We will revise the abstract, introduction, and conclusion to emphasize that our results demonstrate RSI over a limited number of training and environment-adaptation rounds, with diminishing gains in later rounds. The revised manuscript will make this finite experimental horizon explicit and appropriately qualify claims of continual improvement.

We will also clarify that our method primarily targets coding agents in executable environments with reliable automatic verifiers. These environments support automated task generation, execution, and validation, making it practical to construct and calibrate new training tasks.

Regarding baseline comparability, we will expand the implementation details to document how each baseline was reproduced or adapted, specify the shared training conditions and matched optimization budgets, and clarify the scope of the cost comparisons.

Q3: w/o Evolution drop.

A3: Thanks for your comment. Removing the environment-evolution loop leaves the training task pool unchanged, so subsequent rounds repeatedly train on the same tasks. The degradation over the two additional rounds is consistent with overtraining on a fixed distribution, where repeated exposure may provide progressively less useful learning signals. Our interpretation is that environment transformation helps mitigate this effect by introducing new, learnable challenges. SSR and RST also modify or generate training tasks and continue to improve in the reported experiments. We will clarify that this is a plausible explanation of the observed behavior; Table 3 alone does not establish the precise cause of the degradation.

Our method also outperforms the baselines that continue to improve. At Round 3, it achieves 20.7, compared with 17.8 for SSR and 17.1 for RST, corresponding to absolute improvements of 2.9 and 3.6 points on the reported aggregate metric (Table 1). These comparisons support the added value of our evolution strategy in the evaluated setting, beyond its advantage over continued training on the fixed task pool.

We have also added more ablations on environment ablations. Please refer to A5 for Reviewer zr8z.

Q4: Model scaling and dataset difficulty

A4: Thank you for raising this point. We select seed datasets according to each learner’s initial coding capability: shorter-horizon TMax tasks for Qwen3-8B and longer-horizon Cross-PR tasks for Qwen3.6-35B-A3B. Our aim was to maintain a useful mixture of unsolved, frontier, and saturated tasks, targeting approximately 30–40% frontier tasks to provide sufficient learning opportunities.

The stronger-model setting also shows meaningful numerical gains: its aggregate score increases from 57.9 to 62.8, an absolute improvement of 4.9 points, compared with 5.0 points for Qwen3-8B (15.7 to 20.7). It also exceeds SSR by 2.1 points (62.8 versus 60.7). Thus, the observed gains are not confined to the weaker model, and the difference between the relative percentages should be interpreted in light of the different starting scores.

Q5: Escalation operator

A5: Thank you for highlighting this concern. Our seed tasks come from TMax and our internally constructed Cross-PR dataset. For TMax, we will release the exact training subset used in our experiments. We have conducted LLM-assisted screening of its task descriptions against TerminalWorld and Terminal-Bench 2 and found no duplicate descriptions in this check.

Our Cross-PR dataset consists of long-horizon tasks constructed from curated GitHub repositories, with each task spanning multiple pull requests. These tasks are designed to emphasize demanding development workflows, and we keep the training tasks disjoint from the evaluated benchmark instances, including those in SWE-Bench Verified and SWE-Bench Pro.

Regarding comparison fairness, our method and the RSI training baselines start from the same checkpoint after the first training round. The subsequent comparisons therefore assess the additional gains achieved during RSI from a common initial policy and training history.

Dear reviewer, please respond to the author's comment to further facilitate the discussion as the author-reviewer discussion period ends at November 18 at 11:59pm AoE

Best, AC

Reviewer_tTGgOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary
  • presents a contribution to RSI where the focus is on a closed loop as opposed to an open loop framework.
  • presents a way to determine how to proceed with the curriculum through a specific curriculum operator that can tell us how well the updated model does on each task and reshapes the task pool so that most tasks are in the so called sweet spot to learn from. The curriculum operator seems to be a significant part of the paper. This method is coupled with an RL training loop for coding agents that, after every round, re-measures which tasks the agent finds too hard, too easy, or just right, and rewrites the task pool so training stays focused on what the agent can currently learn. Said simply, the idea is that an AI learns most from tasks it sometimes solves and sometimes fails. Tasks it always solves, or never solves, teach it nothing. After three rounds, the AI got better on standard coding benchmarks than with other self-improvement methods. Without adjusting the tasks, it actually got worse. The idea makes sense, and they test on real, public benchmarks against real competing methods.
Critical strengths

The paper tackles a real problem in multi-round RL for coding agents where tasks the policy always solves or always fails provide no learning signal. It addresses this with a sensible closed loop that re-labels tasks against the current policy after each round. The task transformations are well designed. Prefix tasks keep the original instruction and full verifier, so easier variants still require completing the real task, and every new variant is checked to be initially unsolved but solvable, then calibrated against the policy. The evaluation uses public benchmarks, including out-of-distribution terminal tasks, and compares against open-source baselines from harness-, model-, and environment-level self-improvement. The analysis of how task states and policy entropy change across rounds gives useful insight into the loop’s dynamics, and the stability and context-compaction engineering is carefully documented and code release is appreciated.

Critical weaknesses

Good idea and solid engineering, but the evidence doesn’t yet prove that the task-adjusting is what made the AI better. Fig1a does not add clarity and generally Fig.1 should be improved. Figure 1a for example is enough to just describe in words without an visual analogy. Also, it seems that you only used only about 6% of the dataset. Simply giving the AI new, unused tasks each round might work just as well. Without that comparison, we can’t tell whether “adjusting tasks” is what helped, or just “having new tasks.” It is also not entirely what is working. For example, was it the head starts, the harder tasks, or the rewording that helped? Another concern is that the gains are small and measured only once. Sometimes the difference is a single task out of 100. So the question is whether this is really “self”-improvement and at times the explainations are confusing.

Questions for authors

Only about 6% of TMax is used for training. How does Frontier-RSI compare with simply adding fresh, unused TMax tasks each round, or filtering them to the frontier band, at the same compute? This seems necessary to attribute the gains to environment evolution rather than to data refresh. What does each operator (prefix, escalation, rewrite) contribute on its own? How does frontier filtering without any task transformation perform? Figure 2 describes an operator that reads state flows (Δ_U→F, Δ_F→S) and adjusts continuous knobs (Δd, Δs), but the method section describes fixed per-task transformations. How exactly are flows mapped to knob adjustments? Prefix tasks rely on trajectories from a stronger external model. How much of the gain depends on this? Which models generate escalations and rewrites, and does the loop still work without access to a stronger model? How much do results vary across random seeds? Please report absolute differences with confidence interval.

4 comments

We sincerely thank the reviewer for the constructive feedback and apologize for the delayed response, as the additional ablation experiments on post-training require three to four days per run.

Q1: Environment evolution ablation.

A1: Thank you for this suggestion. We select pproximately 6% of TMax based on learner-relative task difficulty, aiming to maintain an appropriate mixture of unsolved, frontier, and saturated tasks while keeping iterative RL experiments computationally manageable. We chose Qwen3-8B to study RSI from a starting policy with substantial learning headroom.

We agree that this subset-selection rationale alone does not distinguish the benefits of environment transformation from those of introducing previously unused tasks. To examine the contributions of the individual evolution components, we have conducted additional ablation experiments.

We initially omitted module-removal ablations because, under our training setup, matching the number of training examples across settings would require reusing the tasks associated with the removed module. Such repeated exposure could itself affect training performance, making it harder to separate the module’s direct contribution from the effects of task repetition.

Nevertheless, to address this concern, we can conduct fine-grained ablations that independently remove Prefix Scaffolding (PS), Difficulty Escalation (DE), while keeping the learner, RL training configuration, and optimization budget matched. The experiments are conducted during the second training round of Qwen3-8B on the TMax dataset, corresponding to the first RSI iteration. The results below show that removing key modules reduces the performance gains achieved through RSI. These ablation results demonstrate the contribution of these modules to the effectiveness of our framework.

MethodPass@1Pass@4Pass@8
Ours w/o PS4.210.514.6
Ours w/o DE4.410.716.9
Ours4.511.918.0

Furthermore, to distinguish environment transformation from curriculum selection, we include a frontier-only sampling (FS) baseline that prioritizes tasks based on the current policy's success rate without modifying their executable environments. This comparison directly examines whether policy-conditioned environment transformations (i.e., prefix scaffolding, difficulty escalation) provide additional benefits beyond selecting intermediate-difficulty tasks. Starting from the same Round-1 checkpoint, we construct it by selecting original TMax tasks whose empirical success rates fall within the same frontier interval used by our method. Their instructions, initial environments, and verifiers remain unchanged. We train on these tasks using fresh rollouts while keeping the RL algorithm, dynamic filtering, and accepted-trajectory optimization budget identical to those of the full method. This comparison evaluates whether environment transformation provides additional gains over success-rate-based curriculum selection during the first RSI iteration. These results below demonstrate the substantial benefits of environment transformation. We have also previously observed that model performance declines after more than four epochs of training on the same environments, and environment transformation helps mitigate this degradation.

MethodPass@1Pass@4Pass@8
Ours w. FO4.110.212.4
Ours4.511.918.0

Q2: Marginal performance gain

A2: Thank you for raising this concern. We chose Qwen3-8B because its relatively limited initial coding performance provides substantial learning headroom, allowing us to investigate whether iterative training and environment adaptation can improve a less capable starting policy.

Our Pass@1, Pass@4, and Pass@8 results are computed from eight attempts per task. The reported gains are observed across multiple training rounds and sampling budgets, providing evidence of a consistent improvement trend. However, these metrics share the same evaluation samples and do not, by themselves, establish statistical significance or eliminate uncertainty in small performance differences. We will make the evaluation protocol and the number of training runs underlying each result explicit.

Q3: Knob adjustments

A3: Thank you for highlighting this ambiguity. Figure 2 should more clearly distinguish aggregate state-transition statistics from the implemented task-level feedback. Our implementation uses each task’s current state to select prefix construction, rewriting, or escalation, as specified in Eq. (6). It does not implement a numerical mapping from aggregate flows to continuous knob updates. The operational details are provided in Appendix B.1–B.2 and Figures 6–7: scaffolding is adjusted through the cutoff of a verified trajectory, while difficulty is adjusted through discrete changes to the environment or task requirements. Candidates are calibrated using 16 valid rollouts from the frozen learner. Overly difficult escalation candidates receive weaker perturbations or additional scaffolding, while overly easy candidates undergo further escalation. We will revise Figure 2 to make this mechanism explicit.

Q4: No stronger-model dependency

A4: Thanks for your comment. We have clarified clearly in Appendix B.1 that we use Fable 5 as the teacher model to provide verified successful trajectories. However, these trajectories are not directly supplied as target answers to the policy model. Instead, the updated policy model's own rollout outcomes determine the task states and condition the subsequent environment adaptations. Thus, the self-improving feedback loop lies in the dependency between policy improvement and future training conditions, rather than requiring every environment construction step to be performed by the policy model itself. Overall, our loop can operate without the teacher model. The teacher model primarily serves to automate and scale up task construction, improving the efficiency of what would otherwise require manual task design.

Q5: Statistical robustness

A5: Thanks for your comment. We have made substantial efforts to ensure the robustness. Specifically, we focus on maintaining stable reward improvements throughout the RL training while minimizing systematic errors during evaluation.

As explicitly described in Appendix F.5, we assess training stability across different random seeds and select runs that exhibit stable training behavior. Furthermore, we report multiple evaluation metrics, including Pass@1, 4, 8 in our experimental tables to provide a more comprehensive assessment of model performance. Together, these measures strengthen the reliability and consistency of our reported results.

Dear Reviewer,

Thank you again for your constructive feedback. We have carefully addressed your concerns in our rebuttal, including additional clarifications and experimental evidence.

As we have not yet had the opportunity to discuss your remaining concerns, we would sincerely appreciate it if you could take a moment to review our responses before the discussion period concludes. We would be happy to provide any further clarification if needed.

Thank you very much for your time and consideration!

Dear reviewer, please respond to the author's comment to further facilitate the discussion as the author-reviewer discussion period ends at November 18 at 11:59pm AoE

Best, AC

Reviewer_W7YROpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Targeting the critical pain point where agents lack effective learning signals during multi-round reinforcement learning due to tasks being either overly trivial or overly difficult, this manuscript investigates adaptive environment evolution in RSI and proposes the Frontier-RSI framework. By categorizing tasks into three policy-relative states based on the model's current problem-solving success rate, and applying intermediate state assistance, task rewriting, and difficulty escalation respectively, the manuscript integrates adaptive curriculum learning and executable software task generation into a closed feedback loop. Multi-round experiments conducted across two foundation model scales and multiple external software engineering and terminal benchmarks provide persuasive preliminary evidence for the effectiveness of this composite system. Overall, in my assessment, the manuscript tackles an important research problem, demonstrates solid system engineering, and provides rich information, presenting substantial research and practical value, although critical ablations and cost-alignment analyses are still needed to further solidify its core conclusions.

Critical strengths
  1. The manuscript identifies an important and acute research problem, keenly pointing out the feedback gap between existing environment evolution and agent reinforcement learning. In my view, the underlying motivation is exceptionally clear and sound.
  2. The design of the bidirectional task adaptation mechanism is intuitive and comprehensive. I find that this technical design holds strong practical significance for training complex software agents.
  3. The task construction pipeline incorporates rigorous executability and verifiability checks, which substantially enhances the engineering credibility of the generated tasks.
  4. The empirical evaluation covers two model scales across four external evaluation benchmarks, providing dynamic analyses on held-out task sets. The experiments convincingly demonstrate the vital role of dynamically reshaping the task distribution during multi-round training to prevent performance degradation.
  5. The system engineering implementation is relatively thorough, and the supplementary material provides comprehensive hyperparameters and implementation details.
Critical weaknesses
  1. The current ablations in the manuscript mainly compare no evolution, rewriting only, and the full Frontier-RSI framework, but multiple variables are conflated across these configurations. In my view, the existing experiments are not sufficiently clean, making it impossible to clearly disentangle whether the performance gains stem from the environment state editing in Prefix and Escalation, the state-driven routing mechanism, or simply the 4x reweighted sampling of medium-difficulty tasks.
  2. Frontier-RSI introduces substantial overhead during candidate task construction, including querying a stronger teacher model for successful trajectories, sandbox replay verification, and 16 calibration rollouts per candidate task alongside failure retries. The empirical evaluation lacks an alignment of the end-to-end computational budget.
  3. In Prefix Construction, unsolved tasks rely heavily on an external, stronger teacher model to supply reliable trajectories. Furthermore, I observe that the current experiments only showcase three training rounds and already exhibit diminishing returns on the larger model, which I consider insufficient to fully support long-term autonomous recursive self-improvement.
  4. In Equation (2) of the main text, qt(x)q_t(x) is termed Pass@k\text{Pass}@k, yet what is actually used in Equation (7) and in the empirical 16-sample calibration appears to me to be the empirical single-attempt success rate. Additionally, Equation (8) in the main text adopts the strict inequality 0.25<q<0.750.25 < q < 0.75, whereas Figures 6 and 7 in the appendix accept any candidate with 4 to 12 successes.
  5. Under the 35B model setting, Frontier-RSI achieves only marginal gains over the strongest baseline SSR, yet the manuscript does not report the mean, standard deviation, or confidence intervals across multiple training seeds. Moreover, certain evaluations rely solely on task subsets, and Appendix F.5 notes that multi-seed results hold only under the premise of training stability, which leaves me concerned about empirical robustness.
Questions for authors
  1. Could the authors provide additional ablation results to clarify the independent contribution of each core component?
  2. Can the authors report a detailed breakdown of the end-to-end computational overhead for each method across training rounds?
4 comments

We sincerely thank the reviewer for the constructive feedback and apologize for the delayed response, as the additional ablation experiments on post-training require three to four days per run.

Q1: Ablations.

A1: Thanks for your suggestion. We agree that this subset-selection rationale alone does not distinguish the benefits of environment transformation from those of introducing previously unused tasks. To examine the contributions of the individual evolution components, we have conducted additional ablation experiments.

We initially omitted module-removal ablations because, under our training setup, matching the number of training examples across settings would require reusing the tasks associated with the removed module. Such repeated exposure could itself affect training performance, making it harder to separate the module’s direct contribution from the effects of task repetition.

Nevertheless, to address this concern, we can conduct fine-grained ablations that independently remove Prefix Scaffolding (PS), Difficulty Escalation (DE), while keeping the learner, RL training configuration, and optimization budget matched. The experiments are conducted during the second training round of Qwen3-8B on the TMax dataset, corresponding to the first RSI iteration. The results below show that removing key modules reduces the performance gains achieved through RSI. These ablation results demonstrate the contribution of these modules to the effectiveness of our framework.

MethodPass@1Pass@4Pass@8
Ours w/o PS4.210.514.6
Ours w/o DE4.410.716.9
Ours4.511.918.0

Furthermore, to distinguish environment transformation from curriculum selection, we include a frontier-only sampling (FS) baseline that prioritizes tasks based on the current policy's success rate without modifying their executable environments. This comparison directly examines whether policy-conditioned environment transformations (i.e., prefix scaffolding, difficulty escalation) provide additional benefits beyond selecting intermediate-difficulty tasks. Starting from the same Round-1 checkpoint, we construct it by selecting original TMax tasks whose empirical success rates fall within the same frontier interval used by our method. Their instructions, initial environments, and verifiers remain unchanged. We train on these tasks using fresh rollouts while keeping the RL algorithm, dynamic filtering, and accepted-trajectory optimization budget identical to those of the full method. This comparison evaluates whether environment transformation provides additional gains over success-rate-based curriculum selection during the first RSI iteration. These results below demonstrate the substantial benefits of environment transformation. We have also previously observed that model performance declines after more than four epochs of training on the same environments, and environment transformation helps mitigate this degradation.

MethodPass@1Pass@4Pass@8
Ours w. FO4.110.212.4
Ours4.511.918.0

Q2: End-to-end computational overhead

A2: Thanks for your comment. We acknowledge that environment evolution introduces additional costs beyond RL optimization, including task generation, teacher inference, sandbox verification, candidate calibration, and retries. Our current experiments emphasize comparisons under a shared learner-side training configuration. We agree that a more detailed cost analysis is necessary to evaluate the efficiency of the complete pipeline.

We will provide a per-round breakdown of RL training cost, environment generation, calibration rollouts, verification overhead, and total execution time for the relevant baselines. We will also clarify which computational resources are matched and which constitute additional overhead, rather than conflating matched training budgets with matched end-to-end costs.

Q3: Stronger-model dependency.

A3: Thanks for this suggestion. The stronger teacher is primarily used to automate and scale up the construction of verified Prefix environments. The policy-conditioned feedback loop itself does not inherently require a stronger teacher model: task-state estimation is performed using the evolving learner, and these measurements determine subsequent environment adaptations. Verified intermediate states may also be supplied through alternative task-construction procedures.

Crucially, the teacher does not directly determine the learner's policy updates. The updated learner's own rollout outcomes drive the next round of curriculum adaptation, while policy improvement is achieved through RL. Thus, the self-improving mechanism lies in the recursive dependency between learner capabilities and future training conditions.

Regarding long-term improvement, our current experiments demonstrate repeated improvement over three training rounds, rather than indefinite or fully autonomous RSI. We acknowledge the diminishing marginal gains and will clarify this empirical scope. Nevertheless, the improvements on fixed held-out tasks and their task-state transitions provide evidence that the learner's capabilities advance across the evaluated rounds.

Q4: Success-rate notation.

A4: We thank the reviewer for identifying these inconsistencies. The task-state calibration in our method is based on the empirical per-attempt success rate over K independent rollouts, rather than the conventional Pass@K metric measuring whether at least one attempt succeeds. We will clarify the notation in Equations (2) and (7).

We will also reconcile the boundary conditions in Equation (8) with the implementation and Appendix Figures 6–7. For K=16, the strict interval (0.25, 0.75) corresponds to 5–11 successful attempts, whereas an inclusive interval [0.25, 0.75] corresponds to 4–12. We will ensure that the equations, implementation, and illustrations use consistent acceptance criteria.

Q5: Statistical robustness.

A5: We agree that explicit measures of variability are important, particularly for small benchmark subsets. As discussed in Appendix F.5, we have devoted substantial attention to RL training stability and evaluation consistency. We also report multiple evaluation budgets (Pass@1/4/8) across the relevant benchmarks.

Importantly, Frontier-RSI improves the 35B model's mean score from 57.9 after Round 1 to 62.8 after Round 3, an absolute gain of 4.9 percentage points, and exceeds the strongest baseline, SSR, by 2.1 percentage points. However, we acknowledge that these point estimates alone do not establish statistical significance.

We will clarify the number of independent training seeds and evaluation rollouts underlying the reported results, document the training-stability criteria, and provide absolute performance differences with appropriate confidence intervals wherever supported by the available experimental data. We will also clarify the evaluation subsets and their selection procedures.

Thank you for the detailed responses and the additional ablation experiments. The new module removal results and the frontier only sampling baseline address my first concern, so I temporarily maintain my rating. My support is conditional on your action. ICLR allows authors to revise their submission, yet the current PDF remains unchanged. Please update the manuscript itself with the new ablation results, the corrected equations, and the clarified statistical reporting, instead of describing these revisions only in the comments. Please also include the promised per round breakdown of end to end computational cost, meaning RL training time, environment generation, teacher inference, calibration rollouts, and verification overhead for each method and each round, directly as a table in the revised manuscript.

Thank you very much for your positive assessment of our work! Your thoughtful and expert feedback has provided valuable guidance for our revisions.

We are also especially grateful for your reminder that we could update the PDF. We have now revised both the main text and the appendix accordingly. Thank you again for your constructive feedback and support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.