acceptodds
Under review as a conference paper at ICLR 2027

TerminalEvo: Evolving Environments at the Capability Frontier of Terminal Agents

Abstract

Synthetic data has become a central paradigm for scaling supervision in language model training. Extending this paradigm to terminal agents requires synthesizing executable environments where models can acquire experience through interaction. However, the learning value of generated environments is highly dependent on their alignment with the capabilities of the target agent: tasks beyond a model's current competence may consume substantial interaction budgets while providing limited learning signal. We introduce **TerminalEvo**, a framework for evolving terminal environments toward the empirical capability frontier of a target agent. Through iterative agent rollouts, TerminalEvo collects complementary feedback signals, where success rates estimate task difficulty and action-observation trajectories reveal behavioral limitations, missing dependencies, and opportunities for environment refinement. An LLM-based editor uses these signals to propose semantic modifications to task structures, constraints, and dependencies, which are then validated through executable contracts and automated verification before being incorporated into future training iterations. We investigate whether frontier-aligned environments enable more efficient learning under fixed interaction budgets and whether behavioral feedback improves environment evolution beyond outcome-only signals.

Primary area:
datasets and benchmarks
Keywords:
Agent Data, Data synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_d6J3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Large Language Models (LLMs) have become increasingly the dominant paradigm for generally capable intelligent agents. While frontier LLMs have led to significant improvements in AI agentic capabilities, for various use cases, it is important to have smaller LLMs that can act as agents and be deployed locally without significant computational burdens. Training small models to have better agentic capabilities is therefore an important current research question.

However, small language models often have limited capacity, which makes training particularly sensitive to the difficulty of available experience. The paper argues that more data or more complex tasks do not necessarily lead to better learning, and often it is required to generate interaction experience that is actually learnable and useful for a model with limited capacity. The paper proposes a training pipeline targeting this problem: in particular, they propose using both the small model’s performance and behavior related feedback to generate custom new environments that are suitable for their learning, use a larger teacher model to generate useful experience in these custom environments, and then use supervised finetuning to improve the smaller models. The proposed training framework is then evaluated across a variety of benchmarks and compared against similar compute matched training frameworks, and shows favorable results.

Critical strengths
  1. The paper is well-written and easy to follow.

  2. The presented framework is well-principled and intuitive. I am confident that the framework works well once one has taken care of the implementation details carefully.

  3. The problem of training small language models to be strong agents is timely and important.

Critical weaknesses

(Missing important details about the experiments presented in the paper)

The paper mentions:

A stronger teacher agent then generates successful demonstrations on the resulting tasks, yielding a separate training corpus for each student.

I could not find any details about the teacher agent. Which model is it? What is the teacher’s own performance? Are there any other teacher models used in the compared baselines in Table 1? For example, SETA in Table 1 uses Kimi-K2.5-Thinking as the teacher model. Could the performance improvement be solely based on the teacher’s strength? A controlled study of the teacher will significantly strengthen the paper. I will be happy to raise my score if the authors could provide this ablation, especially if they can use the same models they are training (e.g., Qwen3.5-27B) as the teacher model, since it will show that the method can be scaled to the largest models as well, significantly strengthening the paper’s claim.

(Thin Improvement Margins)

The improvement margins are thin, and it is unclear to me if they are statistically significant. Moreover, the main motivation of the paper is to train small language models to be strong agents, but the performance gains are the smallest on the 4B model (the smallest model tested in the paper), contradicting the motivation for the proposed framework.

(Missing experiments)

  1. There is no ablation comparing behavior-guided editing against outcome-only gating. Without this ablation, it is unclear to me the relative significance of different parts of the framework.

  2. Separate training data is built for 4B, 9B and 27B students. Could the authors train one model (eg., 4B) on another model’s training data (e.g., 9B)? Without it, it is unclear to me that student specific training data generation is helpful.

  3. The paper is missing few key important baselines. For example, how does a well-tuned outcome reward only RL run on the modified/augmented dataset of environments perform? Since a teacher model is used to generate SFT data, why not run OPD or some other well-tuned distillation baselines and compare against the paper’s method?

(Extremely poorly written related works)

The paper’s relevant works are poorly written. It misses key citations of even the well-established methods that it uses for training its models. For example, the paper uses Direct Preference Optimization (DPO) on top of SFT to further improve the LLM agents’ performance. I could not find any citations to DPO [1] or its multi-turn extension [2] anywhere in the paper, even though these citations will be vital. Similarly, the idea of length normalized DPO can be traced back to SimPO [6] and Tulu 3 [7], but none of these are cited as well.

There is also a significant body of research studying curriculum learning for LLM agents that discuss the “frontier of capability” based curriculum. A few notable examples are PAPRIKA [3] and SEC [4], again not cited or discussed. Related works in environment generation such as SPADE [5], though concurrent, should at least be cited and discussed.

References

[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model, https://arxiv.org/abs/2305.18290

[2] From rr to Q∗Q^*: Your Language Model is Secretly a Q-Function, https://arxiv.org/abs/2404.12358

[3] Training a Generally Curious Agent, https://arxiv.org/abs/2502.17543

[4] Self-Evolving Curriculum for LLM Reasoning, https://arxiv.org/abs/2505.14970

[5] SPADE: Self-Play in Adaptive Synthetic Executable Environments, https://arxiv.org/abs/2608.19197

[6] SimPO: Simple Preference Optimization with a Reference-Free Reward, https://arxiv.org/abs/2405.14734

[7] Tulu 3: Pushing Frontiers in Open Language Model Post-Training, https://arxiv.org/abs/2411.15124

Questions for authors

Please address the weaknesses mentioned above.

Reviewer_GXcFOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents TerminalEvo, a framework for generating student-specific training data for compact terminal agents. It evaluates a fixed student on executable tasks and uses the success rate to decide whether to continue task evolution. An LLM analyzes the student’s interaction trajectories to propose additional requirements, which are implemented through coordinated updates to the reference solution, instruction, environment, and verifier. Validated tasks are then used to collect successful demonstrations from a stronger teacher for supervised fine-tuning, followed by optional DPO. The main contribution is combining outcome-based evolution control with trajectory-guided task modification. Experiments evaluate Qwen3.5-4B, 9B, and 27B on Long-Horizon Terminal-Bench, Terminal-Bench 2.1, and SWE-bench Verified against four training datasets under approximately matched supervised-token budgets. TerminalEvo achieves the best performance in seven of nine model–benchmark settings and ranks second in the remaining two, with subsequent DPO improving all nine scores.

Critical strengths
  1. TerminalEvo achieves the best performance in most of the evaluated model–benchmark settings.

  2. The research problem is clearly formulated.

  3. The evaluation covers multiple model sizes and diverse tasks, including terminal interaction and repository-level software repair.

Critical weaknesses
  1. Although TerminalEvo generally outperforms the competing datasets, its improvements over the strongest baselines are relatively small. The paper does not report the random seeds or the mean and standard deviation across independent training runs, making it difficult to assess the reliability and statistical significance of these improvements.

  2. The DPO evaluation only compares TerminalEvo SFT checkpoints with their DPO-refined counterparts. While this demonstrates an incremental benefit from DPO, it does not establish whether TerminalEvo retains its advantage when competing baselines also receive comparable preference optimization.

  3. Important details of the SFT comparisons are missing, including the numbers of retained training trajectories, trajectory-length statistics, and training epochs or optimization steps for each method. Although the paper states that loss-bearing token budgets are approximately matched, the actual budgets and matching deviations are not reported. These omissions limit reproducibility and make it difficult to verify the comparability of the training conditions.

Questions for authors
  1. Could you report the mean and standard deviation across multiple independent training runs with different random seeds, at least for TerminalEvo and the strongest baselines? Please specify the number of runs and the seeds used.

  2. Could you provide a table of SFT data and training statistics for each method and model size, including the number of unique tasks and retained trajectories, mean or median trajectory length, training epochs or optimization steps, and actual loss-bearing token counts? Please also explain how the budgets were matched and report the deviations from the target budget.

Reviewer_hcsjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present TerminalEvo, which evolves the training environments to adapt to the capability frontier of terminal agents. Thereby, ti provides more attainable trianing tasks/signals to the agents. For environment/task evolution, eight rollouts from a fixed student provide complementary feedback: interaction histories guide a transformation contract, while success counts determine whether the task remains eligible for evolution. After evolution, a stronger teacher generates demonstrations on terminal candidates, which are then used for SFT.

Critical strengths

This paper presents a practical methodological loop that synthesizes tasks/environments for training terminal agents, evaluated on three models of varying sizes. The results are promising when compared with other static datasets.

Critical weaknesses
  • The idea seems to be a standard practice in synthesizing tasks for training agents. To ensure the task/environment is bug-free and learnable, we would naturally want to rollout multiple times and revise the task/environment based on the pass rate and trajectories.
  • The data is separately evolved for each model. This implies, in contrast to the baselines, that the dataset is policy-dependent and therefore not generally beneficial.
  • Repeated student rollouts, editing, validation, and teacher collection add substantial compute that is never reported, and budget matching does not equalize total computation.
  • The way budget matching is done during comparison is not reported. For example, is the dataset like Tmax downsampled for budget matching? Is the downsampling biased towards a certain data distribution?
Questions for authors

Is the teacher model the same as the student model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.