- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how the supervision horizon affects the reliability and efficiency of terminal agent training. They argues that longer supervision helps agents learn error recovery, but its benefits eventually saturate and can lead to over-persistence. To address this, the paper proposes selective long-horizon refinement, which first trains on short trajectory prefixes and then extends supervision using trajectories whose newly exposed continuations have high likelihood under the warm-start model. Experiments show improved solving reliability compared with one-stage and full data training at the same horizon, while selected configurations also reduce training time.
Critical strengths
- The paper addresses an important question in terminal-agent training. Longer trajectories can expose the student to useful error-recovery behaviors, but they also increase training cost and may encourage unnecessary persistence. The paper systematically examines how supervision length affects performance, agent behavior, and computational cost, providing practical guidance for using long-horizon demonstrations effectively.
- The paper evaluates both whether an agent can solve a task at least once and whether it can solve that task consistently across repeated attempts. This distinction reveals trade-off that aggregate success alone can obscure, such as improved repeatability without broader task coverage. The evaluation therefore helps clarify whether a training strategy expands the range of tasks the agent can solve or makes its existing capabilities more reliable.
Critical weaknesses
- The choice to warm-start on trajectories truncated at 8K tokens and then refine at 12K or 16K tokens appears highly heuristic. Appropriate truncation points are likely to depend on several factors, including the capabilities of the teacher and student models, the characteristics of the training data, and task difficulty. The paper does not provide a principled criterion for choosing these horizons or sufficient evidence that the selected values remain effective across different settings. Without accounting for these factors, the method may require substantial manual tuning for each new teacher–student pair or dataset, leaving its scalability and generalizability insufficiently established.
- The paper motivates learning from later portions of trajectories as a way to acquire the sequences of actions that ultimately complete a task, beyond the initial steps alone. However, truncating trajectories at 12K or 16K tokens does not necessarily expose the student to these completion behaviors. For sufficiently long trajectories, these cutoffs may still fall within the early or middle portions, leaving the actions near the end of the original completed trajectories excluded from supervision. Extending the prefix therefore does not, by itself, establish that the model learns how to bring tasks to completion. The paper should clarify how often the selected horizons actually include task-completion behavior and how the proposed training procedure addresses its absence when they do not.
Questions for authors
- Could the authors report performance as a function of cumulative optimizer steps, separately for the 12K and 16K refinement horizons, covering the different retention ratios and the existing one-stage, two-stage-all, and random-selection baselines? Reporting cumulative GPU-hours alongside these curves, including warm-start, likelihood scoring, and refinement costs, would also clarify the computational efficiency of each configuration. Appendix B.1 states that every refinement setting is trained for 2 epochs. Consequently, retaining 75% of the candidates entails approximately three times as many refinement updates as retaining 25%. The comparisons across retention ratios therefore vary both the training data and the optimization budget, making it difficult to determine which configuration is most effective at a fixed compute budget.
- Prior works [1,2] have highlighted potential limitations of SFT (or Imitation Learning), while other studies suggest that it may struggle to develop richer reasoning behaviors. Since the proposed method selectively trains on trajectories with high likelihood under the warm-start model, could this further bias the policy toward familiar problem types and reasoning patterns, eventually causing distribution shift in inference stage? How well does the resulting model generalize to OOD tasks, and could the authors provide evidence that the proposed selection strategy does not compromise OOD generalization?’
- For Table 5 and Figure 5, the paper interprets the different preferred retention ratios at the 12K and 16K refinement horizons through the trade-off associated with Theorem 5.2. Could this qualitative explanation be developed into a practical criterion for choosing the retention ratio for a given experimental setting? Such guidance would make the method easier for practitioners to apply and reduce the need for an additional hyperparameter search in each new setting.
[1] Chu, Tianzhe, et al. "Sft memorizes, rl generalizes: A comparative study of foundation model post-training." arXiv preprint arXiv:2501.17161 (2025).
[2] Lauffer, Niklas, et al. "Imitation learning for multi-turn lm agents via on-policy expert corrections." arXiv preprint arXiv:2512.14895 (2025).
8 comments
Question 1 - Compute-matched Comparisons
(Q1): report performance as a function of cumulative optimizer steps
A:
Using a global batch size of 4 and approximately 39,200 eligible trajectories in the full refinement dataset, we use the 25% selection ratio as the common unit for compute-matched comparisons. Accordingly, the minimum comparison interval corresponds to: 0.25*39200/4 = 2450.
This allows us to compare different selection ratios at identical cumulative optimizer-step budgets, regardless of their dataset sizes or number of training epochs.
| Steps (approx.) | Top-25 epochs | Top-50 epochs | Top-75 epochs |
|---|---|---|---|
| 2,450 | 1 | 0.5 | 0.33 |
| 4,900 | 2 | 1 | 0.67 |
| 7,350 | 3 | 1.5 | 1 |
| 9,800 | 4 | 2 | 1.33 |
| 12,250 | 5 | 2.5 | 1.67 |
Compute-Matched Comparison of Selection Ratios for 12K Two-stage Training
| Steps (approx.) | Top-25 | Top-50 | Top-75 |
|---|---|---|---|
| 2,450 | 112 | 99 | 93 |
| 4,900 | 125 | 109 | 102 |
| 7,350 | 125 | 116 | 109 |
| 9,800 | 126 | 121 | 113 |
| 12,250 | 126 | 122 | 116 |
| 14,700 | 125 | 123 | 119 |
| 17,150 | 125 | 124 | 121 |
| 19,600 | 124 | 124 | 122 |
Compute-Matched Comparison of Selection Ratios for 16K Two-stage Training
| Steps (approx.) | Top-25 | Top-50 | Top-75 |
|---|---|---|---|
| 2,450 | 106 | 94 | 87 |
| 4,900 | 114 | 104 | 96 |
| 7,350 | 115 | 115 | 106 |
| 9,800 | 115 | 122 | 113 |
| 12,250 | 115 | 123 | 120 |
| 14,700 | 114 | 124 | 126 |
| 17,150 | 114 | 124 | 127 |
| 19,600 | 113 | 124 | 128 |
Our observations suggest a training-budget-dependent trade-off between selecting highly learnable trajectories and retaining more diverse training examples. During early training, Top-25 achieves stronger performance at both supervision horizons, as concentrating the limited training budget on highly learnable trajectories may reduce optimization difficulty and accelerate learning. At the shorter 12K horizon, Top-25 remains competitive even with additional training, suggesting that a smaller subset of high-likelihood trajectories may provide sufficient supervision given the relatively lower policy complexity. However, at the longer 16K horizon, Top-25 tends to saturate, potentially because its limited diversity fails to cover important later-stage behaviors. As training progresses, Top-50 and Top-75 gradually improve, with Top-75 eventually achieving the best performance at 16K. These observations suggest that the optimal selection ratio should depend on both the supervision horizon and the available training budget: under limited compute, selecting a smaller subset of highly learnable trajectories may be more effective, whereas with sufficient compute, retaining more diverse trajectories and training at longer supervision horizons may yield greater benefits.
Question 2 (Part 1) - Biased Behaviors
(Q2) (1/2): ...could this further bias the policy toward familiar problem types and reasoning patterns, eventually causing distribution shift in inference stage?
We analyze Complete-12K (one stage), 12K-Top50, 12K-Bottom50. Our qualitative analysis suggests that 12K-Top50 tends to reduce variability in task-solving behavior by reinforcing consistent execution patterns. In some cases, these patterns correspond to good practices, such as validating input preconditions (fibonacci-server), preserving numerical precision (heterogeneous-dates), and correcting improperly formatted outputs (simple-web-scraper). However, increased behavioral consistency is not always beneficial: it can also reinforce incorrect decisions, leading to repeated failures (crack-7z-hash.easy).
We observe that 12K-Top50 more consistently satisfies critical task requirements across repeated attempts than Complete-12K and 12K-Bottom50.
fibonacci-server
The task requires rejecting negative inputs while accepting n = 0. The 12K-Top50 model correctly implements this validation in 8/8 attempts, compared with 3/8 for Complete-12K and 5/8 for 12K-Bottom50.
heterogeneous-dates
The agent must preserve full numerical precision when writing values to a file, using two-decimal formatting (.2f) only for printed output. The 12K-Top50 model satisfies this requirement in 8/8 attempts, compared with 4/8 for Complete-12K and 5/8 for 12K-Bottom50.
simple-web-scraper
The agent must remove the "by " prefix from author fields (e.g., converting "by F. Scott Fitzgerald" to "F. Scott Fitzgerald"). The 12K-Top50 model correctly performs this operation in 6/8 attempts, compared with 3/8 for Complete-12K and 5/8 for 12K-Bottom50.
Table: Repeatability of Correct Task-Solving Behaviors
| Task | Required Behavior | Complete-12K | 12K-Bottom50 | 12K-Top50 |
|---|---|---|---|---|
| fibonacci-server | Rejects negative n and accepts n = 0 | 3/8 | 5/8 | 8/8 |
| heterogeneous-dates | Preserves full precision in files (.2f only for printing) | 4/8 | 5/8 | 8/8 |
| simple-web-scraper | Removes the "by " prefix from author fields | 3/8 | 5/8 | 6/8 |
However, greater behavioral consistency does not necessarily imply better performance. In some cases, high-likelihood selection may reinforce an incorrect interpretation or decision, causing the agent to repeat the same mistake.
crack-7z-hash.easy
This task provides a representative failure case in which 12K-Top50 experiences a substantial performance degradation.
The agent must determine the correct content to write into solution.txt. Two possible answers arise during execution:
honeybear: The correct final answer.1998: The cracked password, which is not the required final answer.
Out of 3 incorrect attempts, the 12K-Top50 agent explicitly recognized that the instruction could be interpreted in two ways. It considered both honeybear and 1998, but ultimately selected 1998. Thus, even recognizing the ambiguity did not prevent the agent from repeating an incorrect decision.
Table: Distribution of Final Answers Across Eight Attempts
Final content of solution.txt | Complete-12K | 12K-Top50 | 12K-Bottom50 |
|---|---|---|---|
honeybear (correct) | 6/8 | 3/8 | 6/8 |
1998 (cracked password) | 1/8 | 3/8 | 0/8 |
| File never written (timeout/context exhaustion) | 0/8 | 2/8 | 2/8 |
| Other incorrect content | 1/8 | 0/8 | 0/8 |
While Complete-12K and 12K-Bottom50 produce the correct answer in 6/8 attempts, 12K-Top50 succeeds in only 3/8 attempts. Instead, it repeatedly selects the incorrect answer 1998 or fails to write the output file.
This example illustrates a potential limitation of high-likelihood selection: the same mechanism that stabilizes correct execution patterns may also reinforce systematic errors, reducing the agent's ability to recover from an incorrect interpretation.
We will put these qualitative examples in the revised paper.
Weaknesses
(W1): Appropriate truncation points are likely to depend on several factors, including the capabilities of the teacher and student models, the characteristics of the training data, and task difficulty.
A:
We thank the reviewer for highlighting these important factors. To the best of our knowledge, our work is the first to systematically investigate long-horizon trajectory selection for terminal-agent training. We acknowledge that optimal truncation points may depend on teacher–student capabilities, training-data characteristics, and task difficulty.
While these factors have been explored in the broader literature on LLM data selection, extending such insights to agent training is nontrivial due to the complexity of multi-step interactions with dynamic environments. In particular, defining and quantifying task difficulty for terminal agents is itself a challenging problem. Unlike conventional reasoning tasks, where difficulty can often be approximated using solution accuracy or model success rates, terminal-agent tasks involve multiple interdependent decisions, terminal execution, environment feedback, error recovery, and potentially diverse solution paths.
Consequently, establishing a reliable evaluation rubric that captures how well an agent performs throughout a trajectory, beyond simply whether the final task succeeds, remains an open challenge. For instance, two agents may both complete the same task but differ substantially in execution efficiency, recovery ability, and robustness, making it difficult to characterize task difficulty or determine the appropriate supervision horizon.
A comprehensive investigation of how these factors jointly influence the optimal truncation point is beyond the scope of this initial study and represents an important research direction in its own right. We appreciate the reviewer's suggestions and will explicitly discuss these challenges and potential research directions in the Limitations and Future Work section of the revised manuscript.
(W2): should clarify how often the selected horizons actually include task-completion behavior and how the proposed training procedure addresses its absence when they do not.
A:
We thank the reviewer for this important suggestion. To clarify how frequently the selected supervision horizons capture task-completion behavior, we provide the distribution of assistant-token lengths in our training corpus.
Table: Distribution of Trajectory Lengths
As shown in the dataset statistics, the 8K cutoff covers the complete length of approximately 58% of trajectories, while the 12K and 16K cutoffs cover approximately 87% and 97%, respectively. Thus, longer supervision horizons expose the model to more complete trajectories, including later-stage reasoning, error recovery, and task-completion behaviors.