acceptodds
Under review as a conference paper at ICLR 2027

Environment Evolution for Terminal Agents

Abstract

Scaling interactive and verifiable environments is critical for training terminal agents. As frontier models become more capable, environments synthesized from scratch become less challenging and thus provide limited learning signals. Recent co-evolution methods iteratively synthesize environments near the model's learnable frontier based on weaknesses exposed during rollouts. However, their dependence on on-policy rollouts limits generalization and the continuous provision of learning signals as the model becomes stronger. In this paper, we propose environment evolution, which incrementally increases environment difficulty off-policy and schedules the evolved environments generation by generation during training to provide continuous learning signals. We derive three evolution directions that influence environment difficulty from the multi-turn learning objective and then implement evolution along these directions through a loop-engineered multi-agent harness. Quantitative rollout experiments with Hy4 preview, Claude Opus 5, and GPT-5.6 Sol show that environment evolution consistently produces more difficult environments. We validate its effectiveness on Qwen3.6-27B and Qwen3.6-35B-A3B through simple long-horizon RL training, improving their performance by 14.4 and 18.0 percentage points on Terminal-Bench 2.1, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Terminal Agents, Environment Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oiEZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper constructs terminal-agent training environments by incrementally editing scenario novelty, skill rarity, or execution length, without requiring rollouts from the target learner during generation. A two-loop harness reviews plans and verifies executable environments, and an Evolution-Lineage scheduler advances the learner through the resulting generations. Experiments measure difficulty across three rollout models and report improved Terminal-Bench 2.1 performance after RL on two Qwen backbones.

Critical strengths
  1. The approach offers a practical way to construct harder, executable tasks without continually querying the target learner. Oracle, no-op, rubric-based, and development-time human checks address the important risk that a lower pass rate merely reflects broken tasks. The experiments also examine effort levels and individual evolution directions, making the generation procedure more concrete.

  2. Figure 5 shows sizable and sustained late-training gains over the implemented co-evolution and ensemble baselines on both backbones, from a shared post-RFT checkpoint. The cross-model difficulty measurements support the usefulness of the generated lineages beyond a single rollout model. Appendix D also usefully states the approximations behind the difficulty formulation, and Appendix B openly reports the lack of continued gains at 500 steps.

Critical weaknesses
  1. The scheduler used for each arm of the main comparison should be made explicit before attributing its advantage specifically to environment generation. Figure 4 compares scheduling choices only through training reward and partially solved groups in the first 50 updates; there is no corresponding full-horizon held-out benchmark comparison. The three generation methods share the synthesis model and environment count, which is helpful, but equal counts and update steps do not ensure equal token exposure or compute as trajectories become longer. The early scheduler experiment does not isolate its contribution to final benchmark performance; if the comparison arms use different schedulers, generation-specific attribution is additionally confounded. These omissions leave the value of the complete pipeline clearer than the distinct sources of its advantage. A matched final-performance scheduler control and per-method rollout/token accounting would materially strengthen that attribution.

  2. The experimental specification is not yet sufficient to reproduce and assess the main gains. The paper describes GRPO, RFT, staleness, and evaluation settings, but does not instantiate key training quantities such as the learning rate, effective update batch, and RFT data budget, or provide the exact plan-review rubrics, repair limits, and selected seed-environment identities. Appendix C reports aggregate API and GPU costs, rather than the per-method counts needed for the comparison. The five reported runs are not clearly identified as independent training runs or repeated evaluations, and Figure 5 emphasizes the best benchmark checkpoint from evaluations every ten steps without uncertainty. The persistent curve separation is encouraging, but a fixed selection rule and a clear account of run-level variability are needed to establish how robust the reported magnitude is.

Questions for authors
  1. Which scheduler is used for each comparison arm? How much of the final Terminal-Bench gain remains with the same evolved environment pool but random scheduling, using a matched rollout and token budget? If full-run controls are unavailable, can the authors delimit the conclusion to the combined generation-and-scheduling pipeline and provide the existing per-method resource counts?

  2. Can the authors supply the actual RFT/GRPO configuration, seed-pool identities, prompts/rubrics and repair limits, and clarify what the five runs repeat? Please also give results under a fixed checkpoint-selection rule, with per-run or paired uncertainty for the main comparisons. These are requests about the reported experiment, rather than suggestions for a new application domain.

1 comment
AuthorsOpenReview ↗

Response to Reviewer oiEZ [1/n]

Thank you for your questions. We address them point by point below.


Weakness 1 & Q1

In the final experiments, all three methods follow the same scheduling principle: sampling environments within the model’s learnable range to ensure consistency in effective training samples. Specifically, Environment Evolution uses the EL Scheduler. For the other two methods, which lack a generation-by-generation structure and therefore cannot use the EL Scheduler, we prioritize environments within the model’s learnable range. This follows the same motivation as the EL Scheduler.

The results are as follows:

Sampling strategyMethodQwen3.6-27B ↑Qwen3.6-35B-A3B ↑
Learnability-prioritized samplingEnvironment Ensemble60.0%52.8%
Learnability-prioritized samplingAgent–Environment Co-Evolution62.9%55.1%
Learnability-prioritized samplingEnvironment Evolution71.5%64.9%
Random samplingEnvironment Ensemble58.8%49.6%
Random samplingAgent–Environment Co-Evolution61.3%52.8%
Random samplingEnvironment Evolution68.9%63.5%

Under both sampling strategies, Environment Evolution outperforms the other two methods. Learnability-prioritized sampling improves performance for all three methods under the same rollout and training-step budgets.

We will separately describe the contribution of the overall pipeline and that of each component in the paper.


Weakness 2 & Q2

Thank you for these thorough and detailed questions.

  • RFT configuration: For each environment, we sample 16 trajectories and retain the successful ones. We then use embeddings to deduplicate the pool of successful trajectories.

  • GRPO configuration: Learning rate: 1e-6; maximum staleness: 5; batch size: 16; group size: 8.

  • Seed-pool identities and prompts/rubrics: We will release these materials upon acceptance of the paper.

  • Repair limits: Environments that fail the oracle or rubric checks are allowed up to 10 repair attempts. Those that still fail after 10 attempts are discarded.

  • Five evaluation runs: Following the official Terminal-Bench evaluation protocol, we evaluate each checkpoint five times and report the average. For each result reported in the paper and in our response to Weakness 1 and Q1, the same selected checkpoint is independently evaluated five times, since individual rollout results can be unstable on long-horizon tasks.

Reviewer_zDAUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the loss of RL learning signal when terminal-agent training environments become too easy. It proposes environment evolution: an LLM-based multi-agent harness incrementally edits existing environments along three evolution directions (scenario novelty, skill rarity, and execution length) without rollouts from the target policy. Each candidate must pass an Oracle verifier, an Invalid-test verifier, and a rubric-based verifier. During GRPO training, the EL Scheduler moves to the next environment, and then to the next generation, once the learner's pass rate exceeds 6/8. On Terminal-Bench 2.1, Qwen3.6-27B and Qwen3.6-35B-A3B reach peak accuracies of 71.5% and 64.9%, above the implemented co-evolution and ensemble baselines.

Critical strengths

Strengths

  1. Relevant problem, large gains. Public terminal environments are often too easy or broken for current models, which limits what RL can learn from them; the paper keeps only 127 of 47,678 collected environments. Both Qwen models improve over the RFT checkpoint, and at peak they exceed co-evolution by 8.6 and 9.8 points.
  2. Difficulty checked with three models. In Figure 3, pass rates fall across generations for Hy4 preview, Claude Opus 5 and GPT-5.6 Sol. Because the drop appears for models from three providers, it is unlikely to reflect the weaknesses of a single model.
  3. Scheduler evidence. GRPO gets no learning signal from a rollout group in which all attempts fail or all succeed. Over the first 50 RL steps, the EL Scheduler keeps more groups partially solved than random scheduling at the same rollout budget (Figure 4). No final benchmark comparison is reported.
  4. Environment validation. Each environment must pass its reference solution, fail an empty solution, and pass rubric checks. This excludes unsolvable tasks and tests that pass without any work. Human review during development was used to refine the rubrics.
Critical weaknesses

Weaknesses

  1. Theory does not support the claims. Appendix D states that the factorized requirement model is an approximation and that D_T is not evaluated numerically. The derivation motivates the three evolution directions but does not measure difficulty. Appendix D also states that failure-driven updates can change scenarios or execution length, which weakens the claim that environment evolution is more general than co-evolution.
  2. Source of the gains unclear. The three environment-scaling methods share the environment synthesis model, number of training environments, RFT checkpoint and 200 RL training steps. Direct RL trains on the 500 seed environments only, while environment evolution trains on 15-generation lineages of each. Baseline scheduling and compute per method are not reported. The five reported runs are evaluation runs; no repeated training runs are reported. Headline numbers are peak accuracies on Terminal-Bench 2.1, although the method also leads at step 200.
  3. Table 1 appears inconsistent with seed selection. For length, the 15-step mean pass-rate change is −4.8 pp, or 72 pp in total. If the same environments are used in every generation, the starting pass rate must be at least 72%. Seed environments were selected at a Claude Opus 5 pass rate of at most 4/8, and Figure 3 starts below 50%. The paper should state which environments Table 1 uses and their pass rates at generations 0 and 15.
  4. Task-quality evidence is incomplete. Oracle success does not establish that every requirement checked by the verification system is stated in the instruction. Rubric checks and human review are described, but I could not find a quantitative assessment of instruction–verifier mismatches. I also could not find some details needed to reproduce training and environment generation (prompts, learning rate, and repair budget).
Questions for authors
  1. How does environment evolution compare with a generic "make this harder" editor that uses the same synthesis model, verifiers and EL Scheduler?
  2. Which environments does Table 1 use, and what are their pass rates at generations 0 and 15?

I am willing to increase my score if authors clarify the weaknesses and questions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer zDAU [1/n]

Thank you for your questions. We address your concerns point by point below.


Weakness 1: Theoretical formulation

Thank you for your question. As described in lines 164–167, the reference distributions can be estimated qualitatively by a deep-research agent through broad web search, grounded in world knowledge and conditioned on the relevant context of gg and ξ\xi.

Although failure-driven updates can change scenarios or execution length, these modifications remain guided by the failures observed under the current rollout policy and seed environments, as discussed in lines 186–197. Environment Evolution explicitly operates on all three requirement-level factors.

Weakness 2: Sources of the performance gains

Thank you for your question. We have added experiments that separately examine the effects of scheduling strategy and environment synthesis.

Under learnability-prioritized sampling, all three methods prioritize environments within the model’s learnable range. Environment Evolution uses the EL Scheduler. For the other two methods, which lack a generation-by-generation structure, we prioritize learnable environments following the same scheduling principle. We additionally evaluate all three methods with random sampling.

Sampling strategyMethodQwen3.6-27B ↑Qwen3.6-35B-A3B ↑
Learnability-prioritized samplingEnvironment Ensemble60.0%52.8%
Learnability-prioritized samplingAgent–Environment Co-Evolution62.9%55.1%
Learnability-prioritized samplingEnvironment Evolution71.5%64.9%
Random samplingEnvironment Ensemble58.8%49.6%
Random samplingAgent–Environment Co-Evolution61.3%52.8%
Random samplingEnvironment Evolution68.9%63.5%

Under both sampling strategies, Environment Evolution outperforms the other two methods. Learnability-prioritized sampling improves performance for all three methods under the same rollout and training-step budgets.

The five runs refer to five independent evaluations of the same selected checkpoint, and we report their average.

Weakness 3 & Q2: Environment selection and statistics in Table 1

Thank you for identifying this inconsistency. The previous implementation incorrectly excluded zero-pass-rate entries from the denominator, thus overestimating the magnitude of the average pass-rate decrease.

Weakness 4: Task quality and reproducibility

Thank you for your question. As discussed in the paper, task quality is critical to successful RL training. We therefore use Oracle verification, rubric checks, and human review to strictly control task quality.

We provide the following quantitative results on the effect of quality control on downstream RL performance:

Quality-controlQwen3.6-27B ↑Qwen3.6-35B-A3B ↑
True71.5%64.9%
False57.3%47.4%

Low-quality environments can cause the model to learn undesirable patterns, such as excessively long reasoning and indecision resulting from misalignment between the instruction and the verifier.

Regarding the details needed to reproduce training and environment generation, we provide the training configuration and repair budget below:

ConfigurationValue
RFT sampling16 trajectories per environment
RFT trajectory selectionRetain successful trajectories and deduplicate them using embeddings
GRPO learning rate1e-6
Maximum staleness5
Batch size16
Group size8
Repair budgetUp to 10 attempts for environments that fail Oracle verification or rubric checks; discard those that still fail

We will include the prompts upon acceptance of the paper, as their release requires additional approval.

AuthorsOpenReview ↗

Response to Reviewer zDAU [2/n]

Q1: Comparison with a generic “make this harder” editor

Thank you for this insightful question. We initially tried directly using the prompt “make this harder.” However, with the same synthesis model, we found that this approach tended to degrade environment quality; for example, by introducing instruction–verifier mismatches, corrupting files, hiding accessible files, or adding data noise. These changes made the environments appear harder by lowering their pass rates.

However, the lower pass rates resulted from degraded environment quality rather than increased requirements on the model’s execution capabilities. Training on these environments did not improve downstream performance. Moreover, even with the EL Scheduler, gradient norms and policy entropy remained unstable during training.

Reviewer_sKYnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper employs environment evolution to provide continuous learning signals while training, which are then scheduled generation by generation for training. Rollout experiments on multiple LLMs indicate that environment evolution consistently provides more difficult environments.

Critical strengths
  1. Relevant problem
  2. Intuitive method for solving the problem
  3. Key important results
Critical weaknesses
  1. Difficulty alone does not suffice for better learning. This has been established in UED literature quite a bit. In fact, some of the UED works that have previously defined diversity and novelty as better learning signals than difficulty: Wenjun et al. IJCAI 2023. “ Generalization through Diversity: Improving Unsupervised Environment Design”. IJCAI 2023. Teoh et al. NeurIPS 2024:“Improving Environment Novelty Quantification for Effective Unsupervised Environment Design” NeurIPS, 2024.

  2. There is no validation/intuition of the core hypothesis that difficulty helps learn better.

  3. It seems like the environment is shaped by a frontier model's notion of harder. For instance, Claude Opus 5 filters the seeds, synthesizes the environments, acts as verifier and is the main difficulty estimator (Table 1).

  4. Some of the baselines are author's own implementations. There is no comparison with published methods or with UED methods. So, I am not convinced that the provided results are real improvements over SOTA methods.

  5. Confounds: do baselines get an equivalent curriculum or are sampled randomly?

  6. There are many frontier-model calls. Is this approach scalable with regards to costs?

Questions for authors

Please refer to the weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer sKYN [1/n]

Thank you for your questions. We address them point by point below.


Weakness 1: Difficulty, diversity, and novelty

Thank you for your question. Here, difficulty refers to whether an environment falls within the current model’s learnable range. For example, with eight rollouts, one to seven successful trajectories provide nonzero within-group advantages under binary rewards. This is relevant to GRPO, a widely used algorithm for LLM reinforcement learning [1]. arxiv.org

For LLM training, we consider both diversity, referring to coverage of the environment distribution, and difficulty, referring to whether environments fall within the current model’s learnable range. We follow the line of research on difficulty alignment in LLM reinforcement learning and agent training, as discussed further in our response to Weakness 2.

Regarding diversity and novelty, we view novelty as an aspect of diversity: it reflects whether an environment induces experiences that are sufficiently novel relative to the existing training distribution. This interpretation is consistent with the works you mentioned: DIPLR measures diversity through distances between state–action occupancy distributions, while CENIE measures novelty relative to previously encountered state–action experiences [2, 3]. ijcai.org

We agree that diversity and novelty help ensure broad coverage of the environment distribution, avoiding a biased training distribution. In our GRPO-based LLM setting, matching difficulty to the current model is also necessary for obtaining nonzero reward-based advantages. The learning signal used here differs from that in the RL algorithms studied in the UED papers you cited.


Weakness 2: Evidence supporting difficulty alignment

The importance of aligning difficulty with the model’s current capabilities is supported by prior work on LLM reinforcement learning and agent training:

  • DAPO [1] filters out rollout groups with accuracy equal to zero or one, retaining groups that provide nonzero reward-based policy-gradient signals. arxiv.org
  • GenEnv [4] generates tasks aligned with the agent’s current capabilities through a difficulty-aligned curriculum reward, targeting an intermediate success rate. arxiv.org
  • WebRL [5] adapts its curriculum to the agent’s capabilities and finds that replaying experiences of moderate difficulty performs better than selecting overly easy or difficult experiences. arxiv.org

Weakness 3: Difficulty assessment by frontier models

Thank you for raising this point. Environment difficulty is assessed through rollouts from multiple frontier models, including proprietary models such as Claude Opus 5 and GPT 5.6 Sol, as well as open-source models including HY4 and Qwen3.6-27B. Our goal is to train stronger frontier models, particularly to improve the capabilities of current open-source frontier language models. We therefore use highly capable models to assess environment difficulty through rollouts.

As discussed in the paper, we find that many existing environments provide limited learning signals when training frontier models because they are solved in all eight rollouts, yielding zero within-group reward advantages.


Weakness 4: Baseline comparisons

Thank you for your question. Our baseline implementations achieve stronger results than those reported in contemporaneous work. For example, our implementation of Agent–Environment Co-Evolution achieves peak accuracies of 62.9% and 55.1% on Terminal-Bench 2.1 with Qwen3.6-27B and Qwen3.6-35B-A3B, respectively. We therefore use strong baseline implementations in our comparisons.


Weakness 5: Curriculum controls

Thank you for your question. All methods follow the same scheduling principle under the same training settings: prioritizing environments within the model’s learnable range rather than sampling randomly.

AuthorsOpenReview ↗

Response to Reviewer SKYN [2/n]

Weakness 6: Computational cost and scalability

Yes, our approach requires frontier-model calls, but we consider the cost worthwhile. Companies such as Turing provide RL environments for frontier LLM labs, and reported commercial prices can reach approximately USD 2,000 per task [6, 7]. Our environment construction costs are substantially lower than the quoted costs of human-built environments. turing.com

Furthermore, the environments produced by our method have already been used to train one of the world’s top-three open-source foundation models. We will provide this information in the non-anonymous version of the paper.


References

[1] Yu et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale.

[2] Li et al. (2023). Generalization through Diversity: Improving Unsupervised Environment Design. IJCAI 2023.

[3] Teoh et al. (2024). Improving Environment Novelty Quantification for Effective Unsupervised Environment Design. NeurIPS 2024.

[4] Guo et al. (2025). GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators.

[5] Qi et al. (2025). WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning. ICLR 2025.

[6] Turing. RL Environments.

[7] Denain and Barber (2026). An FAQ on Reinforcement Learning Environments. Epoch AI.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.