acceptodds
Under review as a conference paper at ICLR 2027

Selecting Long-Horizon Trajectories for Reliable and Efficient Terminal-Agent Training

Abstract

Terminal agents are commonly trained by imitating long teacher trajectories, yet how much of each trajectory to supervise remains unexplored. We study the supervision horizon, the number of trajectory tokens retained for training, and show that it is a key design axis for reliability and cost. Reliability improves with longer horizons but saturates: on Terminal-Bench, a 12K-token horizon solves more tasks than 16K ( vs. ) while requiring 30% less training time. The horizon also shapes agent behavior: short horizons cause premature termination, intermediate horizons yield productive error recovery, and long horizons induce over-persistence. We analyze this saturation through a bias–complexity bound, in which longer supervision reduces temporal supervision bias but increases finite-sample estimation error from more heterogeneous late-stage histories. Guided by this analysis, we propose selective long-horizon refinement, which first trains on short prefixes and then refines only on continuations that are most likely under the warm-start model. It consistently outperforms full long-horizon training. At 16K, it raises successful attempts from to and tasks solved in at least six of eight attempts from to ; with half of the long-horizon data, it still reaches while cutting training time by 23%. The gains transfer across benchmarks, from to on Terminal-Bench v2.0 and from to on OpenThoughts-TBLite. For long-horizon supervision, selecting the right trajectories matters more than training on all of them.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM-based Agent, Terminal Agent, Post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1pyPOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how the supervision horizon affects the reliability and efficiency of terminal agent training. They argues that longer supervision helps agents learn error recovery, but its benefits eventually saturate and can lead to over-persistence. To address this, the paper proposes selective long-horizon refinement, which first trains on short trajectory prefixes and then extends supervision using trajectories whose newly exposed continuations have high likelihood under the warm-start model. Experiments show improved solving reliability compared with one-stage and full data training at the same horizon, while selected configurations also reduce training time.

Critical strengths
  1. The paper addresses an important question in terminal-agent training. Longer trajectories can expose the student to useful error-recovery behaviors, but they also increase training cost and may encourage unnecessary persistence. The paper systematically examines how supervision length affects performance, agent behavior, and computational cost, providing practical guidance for using long-horizon demonstrations effectively.
  2. The paper evaluates both whether an agent can solve a task at least once and whether it can solve that task consistently across repeated attempts. This distinction reveals trade-off that aggregate success alone can obscure, such as improved repeatability without broader task coverage. The evaluation therefore helps clarify whether a training strategy expands the range of tasks the agent can solve or makes its existing capabilities more reliable.
Critical weaknesses
  1. The choice to warm-start on trajectories truncated at 8K tokens and then refine at 12K or 16K tokens appears highly heuristic. Appropriate truncation points are likely to depend on several factors, including the capabilities of the teacher and student models, the characteristics of the training data, and task difficulty. The paper does not provide a principled criterion for choosing these horizons or sufficient evidence that the selected values remain effective across different settings. Without accounting for these factors, the method may require substantial manual tuning for each new teacher–student pair or dataset, leaving its scalability and generalizability insufficiently established.
  2. The paper motivates learning from later portions of trajectories as a way to acquire the sequences of actions that ultimately complete a task, beyond the initial steps alone. However, truncating trajectories at 12K or 16K tokens does not necessarily expose the student to these completion behaviors. For sufficiently long trajectories, these cutoffs may still fall within the early or middle portions, leaving the actions near the end of the original completed trajectories excluded from supervision. Extending the prefix therefore does not, by itself, establish that the model learns how to bring tasks to completion. The paper should clarify how often the selected horizons actually include task-completion behavior and how the proposed training procedure addresses its absence when they do not.
Questions for authors
  1. Could the authors report performance as a function of cumulative optimizer steps, separately for the 12K and 16K refinement horizons, covering the different retention ratios and the existing one-stage, two-stage-all, and random-selection baselines? Reporting cumulative GPU-hours alongside these curves, including warm-start, likelihood scoring, and refinement costs, would also clarify the computational efficiency of each configuration. Appendix B.1 states that every refinement setting is trained for 2 epochs. Consequently, retaining 75% of the candidates entails approximately three times as many refinement updates as retaining 25%. The comparisons across retention ratios therefore vary both the training data and the optimization budget, making it difficult to determine which configuration is most effective at a fixed compute budget.
  2. Prior works [1,2] have highlighted potential limitations of SFT (or Imitation Learning), while other studies suggest that it may struggle to develop richer reasoning behaviors. Since the proposed method selectively trains on trajectories with high likelihood under the warm-start model, could this further bias the policy toward familiar problem types and reasoning patterns, eventually causing distribution shift in inference stage? How well does the resulting model generalize to OOD tasks, and could the authors provide evidence that the proposed selection strategy does not compromise OOD generalization?’
  3. For Table 5 and Figure 5, the paper interprets the different preferred retention ratios at the 12K and 16K refinement horizons through the trade-off associated with Theorem 5.2. Could this qualitative explanation be developed into a practical criterion for choosing the retention ratio for a given experimental setting? Such guidance would make the method easier for practitioners to apply and reduce the need for an additional hyperparameter search in each new setting.

[1] Chu, Tianzhe, et al. "Sft memorizes, rl generalizes: A comparative study of foundation model post-training." arXiv preprint arXiv:2501.17161 (2025).

[2] Lauffer, Niklas, et al. "Imitation learning for multi-turn lm agents via on-policy expert corrections." arXiv preprint arXiv:2512.14895 (2025).

8 comments
AuthorsOpenReview ↗

Weaknesses

(W1): Appropriate truncation points are likely to depend on several factors, including the capabilities of the teacher and student models, the characteristics of the training data, and task difficulty.

A:

We thank the reviewer for highlighting these important factors. To the best of our knowledge, our work is the first to systematically investigate long-horizon trajectory selection for terminal-agent training. We acknowledge that optimal truncation points may depend on teacher–student capabilities, training-data characteristics, and task difficulty.

While these factors have been explored in the broader literature on LLM data selection, extending such insights to agent training is nontrivial due to the complexity of multi-step interactions with dynamic environments. In particular, defining and quantifying task difficulty for terminal agents is itself a challenging problem. Unlike conventional reasoning tasks, where difficulty can often be approximated using solution accuracy or model success rates, terminal-agent tasks involve multiple interdependent decisions, terminal execution, environment feedback, error recovery, and potentially diverse solution paths.

Consequently, establishing a reliable evaluation rubric that captures how well an agent performs throughout a trajectory, beyond simply whether the final task succeeds, remains an open challenge. For instance, two agents may both complete the same task but differ substantially in execution efficiency, recovery ability, and robustness, making it difficult to characterize task difficulty or determine the appropriate supervision horizon.

A comprehensive investigation of how these factors jointly influence the optimal truncation point is beyond the scope of this initial study and represents an important research direction in its own right. We appreciate the reviewer's suggestions and will explicitly discuss these challenges and potential research directions in the Limitations and Future Work section of the revised manuscript.

(W2): should clarify how often the selected horizons actually include task-completion behavior and how the proposed training procedure addresses its absence when they do not.

A:

We thank the reviewer for this important suggestion. To clarify how frequently the selected supervision horizons capture task-completion behavior, we provide the distribution of assistant-token lengths in our training corpus.

Table: Distribution of Trajectory Lengths

Assistant TokensTrajectoriesShare
≤ 2K5220.9%
2K–4K7,72613.0%
4K–8K26,15244.0%
8K–12K17,56729.6%
12K–16K5,8589.9%
16K–20K1,2142.0%
> 20K3680.6%

As shown in the dataset statistics, the 8K cutoff covers the complete length of approximately 58% of trajectories, while the 12K and 16K cutoffs cover approximately 87% and 97%, respectively. Thus, longer supervision horizons expose the model to more complete trajectories, including later-stage reasoning, error recovery, and task-completion behaviors.

AuthorsOpenReview ↗

Question 1 - Compute-matched Comparisons

(Q1): report performance as a function of cumulative optimizer steps

A:

Using a global batch size of 4 and approximately 39,200 eligible trajectories in the full refinement dataset, we use the 25% selection ratio as the common unit for compute-matched comparisons. Accordingly, the minimum comparison interval corresponds to: 0.25*39200/4 = 2450.

This allows us to compare different selection ratios at identical cumulative optimizer-step budgets, regardless of their dataset sizes or number of training epochs.

Steps (approx.)Top-25 epochsTop-50 epochsTop-75 epochs
2,45010.50.33
4,900210.67
7,35031.51
9,800421.33
12,25052.51.67

Compute-Matched Comparison of Selection Ratios for 12K Two-stage Training

Steps (approx.)Top-25Top-50Top-75
2,4501129993
4,900125109102
7,350125116109
9,800126121113
12,250126122116
14,700125123119
17,150125124121
19,600124124122

Compute-Matched Comparison of Selection Ratios for 16K Two-stage Training

Steps (approx.)Top-25Top-50Top-75
2,4501069487
4,90011410496
7,350115115106
9,800115122113
12,250115123120
14,700114124126
17,150114124127
19,600113124128

Our observations suggest a training-budget-dependent trade-off between selecting highly learnable trajectories and retaining more diverse training examples. During early training, Top-25 achieves stronger performance at both supervision horizons, as concentrating the limited training budget on highly learnable trajectories may reduce optimization difficulty and accelerate learning. At the shorter 12K horizon, Top-25 remains competitive even with additional training, suggesting that a smaller subset of high-likelihood trajectories may provide sufficient supervision given the relatively lower policy complexity. However, at the longer 16K horizon, Top-25 tends to saturate, potentially because its limited diversity fails to cover important later-stage behaviors. As training progresses, Top-50 and Top-75 gradually improve, with Top-75 eventually achieving the best performance at 16K. These observations suggest that the optimal selection ratio should depend on both the supervision horizon and the available training budget: under limited compute, selecting a smaller subset of highly learnable trajectories may be more effective, whereas with sufficient compute, retaining more diverse trajectories and training at longer supervision horizons may yield greater benefits.

AuthorsOpenReview ↗

Question 2 (Part 1) - Biased Behaviors

(Q2) (1/2): ...could this further bias the policy toward familiar problem types and reasoning patterns, eventually causing distribution shift in inference stage?

We analyze Complete-12K (one stage), 12K-Top50, 12K-Bottom50. Our qualitative analysis suggests that 12K-Top50 tends to reduce variability in task-solving behavior by reinforcing consistent execution patterns. In some cases, these patterns correspond to good practices, such as validating input preconditions (fibonacci-server), preserving numerical precision (heterogeneous-dates), and correcting improperly formatted outputs (simple-web-scraper). However, increased behavioral consistency is not always beneficial: it can also reinforce incorrect decisions, leading to repeated failures (crack-7z-hash.easy).

We observe that 12K-Top50 more consistently satisfies critical task requirements across repeated attempts than Complete-12K and 12K-Bottom50.

fibonacci-server

The task requires rejecting negative inputs while accepting n = 0. The 12K-Top50 model correctly implements this validation in 8/8 attempts, compared with 3/8 for Complete-12K and 5/8 for 12K-Bottom50.

heterogeneous-dates

The agent must preserve full numerical precision when writing values to a file, using two-decimal formatting (.2f) only for printed output. The 12K-Top50 model satisfies this requirement in 8/8 attempts, compared with 4/8 for Complete-12K and 5/8 for 12K-Bottom50.

simple-web-scraper

The agent must remove the "by " prefix from author fields (e.g., converting "by F. Scott Fitzgerald" to "F. Scott Fitzgerald"). The 12K-Top50 model correctly performs this operation in 6/8 attempts, compared with 3/8 for Complete-12K and 5/8 for 12K-Bottom50.

Table: Repeatability of Correct Task-Solving Behaviors

TaskRequired BehaviorComplete-12K12K-Bottom5012K-Top50
fibonacci-serverRejects negative n and accepts n = 03/85/88/8
heterogeneous-datesPreserves full precision in files (.2f only for printing)4/85/88/8
simple-web-scraperRemoves the "by " prefix from author fields3/85/86/8

However, greater behavioral consistency does not necessarily imply better performance. In some cases, high-likelihood selection may reinforce an incorrect interpretation or decision, causing the agent to repeat the same mistake.

crack-7z-hash.easy

This task provides a representative failure case in which 12K-Top50 experiences a substantial performance degradation.

The agent must determine the correct content to write into solution.txt. Two possible answers arise during execution:

  • honeybear: The correct final answer.
  • 1998: The cracked password, which is not the required final answer.

Out of 3 incorrect attempts, the 12K-Top50 agent explicitly recognized that the instruction could be interpreted in two ways. It considered both honeybear and 1998, but ultimately selected 1998. Thus, even recognizing the ambiguity did not prevent the agent from repeating an incorrect decision.

Table: Distribution of Final Answers Across Eight Attempts

Final content of solution.txtComplete-12K12K-Top5012K-Bottom50
honeybear (correct)6/83/86/8
1998 (cracked password)1/83/80/8
File never written (timeout/context exhaustion)0/82/82/8
Other incorrect content1/80/80/8

While Complete-12K and 12K-Bottom50 produce the correct answer in 6/8 attempts, 12K-Top50 succeeds in only 3/8 attempts. Instead, it repeatedly selects the incorrect answer 1998 or fails to write the output file.

This example illustrates a potential limitation of high-likelihood selection: the same mechanism that stabilizes correct execution patterns may also reinforce systematic errors, reducing the agent's ability to recover from an incorrect interpretation.

We will put these qualitative examples in the revised paper.

Reviewer_7ygdOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how much of a teacher trajectory should be supervised when training a terminal agent. Using Qwen3-8B and a fixed terminal-trajectory corpus, it varies the supervision horizon from 4K to 16K tokens and examines task coverage, repeated success, and recovery behavior. It then proposes an 8K warm start followed by refinement on trajectories selected by continuation likelihood. The reported results show diminishing returns from longer supervision and a useful training-cost trade-off across three benchmarks. A bias-complexity bound further provides a qualitative interpretation.

Critical strengths
  1. The reliability analysis goes beyond an aggregate score and reveals a useful trade-off in supervision design. From 12K to 16K, total successes barely change, but the task-level breakdown reports 18 gained and 19 lost successes (Appendix C.2). Similarly, high-likelihood selection favors repeated success, whereas low-likelihood selection can cover more tasks. This makes the finding more informative than simply identifying a best cutoff: longer or easier-to-imitate supervision changes which capabilities become reliable, rather than uniformly improving the agent.

  2. The paper translates this observation into a simple, usable training recipe. At 16K, Top-50 refinement reports 122 successful attempts versus 110 for one-stage training, while reducing training time from 50.81 to 39.13 hours, including the warm start (Tables 4–5). Related gains appear on Terminal-Bench v2.0 and OpenThoughts-TBLite.

Critical weaknesses
  1. The contribution of likelihood selection is smaller and less consistent than the headline comparison suggests. Table 10 reports 123 successes for two-stage training with all eligible data at 16K, versus 110 for one-stage training and 126 for Top-75. At 50% retention, Random and Top obtain 121 and 122 successes. Thus, much of the improvement comes from the training schedule, and selective training’s clearest benefit may be retaining performance with less refinement data.

  2. Some experimental definitions need clarification before the evidence can be interpreted precisely. The reported “estimated run-to-run variation” is not defined as a standard deviation, standard error, or interval, and evaluation repeats do not establish training-seed robustness. The selection pool is also inconsistent: Sec. 6 requires trajectories to reach the target horizon, while Appendix B.1 describes candidates extending beyond 8K. Under the former rule, the 16K pool should be contained in the 12K pool, yet Table 4 lists more Top-50 samples at 16K (19,600 versus 19,568). These appear potentially fixable, but the implemented filtering rule determines what method was actually evaluated.

Questions for authors

Please check the weaknesses.

2 comments
AuthorsOpenReview ↗

Answers for Reviewer 7ygd

(Q1): The contribution of likelihood-based selection is limited.

A:

We acknowledge this limitation, as discussed in Section E.3, where we note that “the warm-start schedule itself provides a substantial benefit.” However, the primary advantage of likelihood-based selection lies not only in improving performance but also in enhancing training efficiency by achieving comparable or better reliability with less training data and compute.

As discussed in Section 7.2 (“How much long-horizon data should be retained?”), selection introduces a fundamental trade-off: retaining fewer trajectories reduces the learning burden and computational cost but may increase approximation error due to reduced data coverage. These competing effects can limit the additional performance gains from selection.

(Q2): “Estimated run-to-run variation” is not clearly defined.

A:

We clarify that “estimated run-to-run variation” refers to the standard deviation across evaluation runs. We refer the reviewer to our response to Question 1 (Paired Test) from Reviewer gfkY, which provides the results of all evaluation runs.

(Q3): Lack training-seed robustness

A:

We thank the reviewer for raising this concern. To assess training-seed robustness, we consider four configurations at the 12K supervision horizon: one-stage training, two-stage All, Top-25%, and Bottom-25%. For each configuration, we compare two training seeds, with eight evaluation runs per seed. The results are summarized below.

Table 1. One-stage (12K)

EvaluationSeed 1Seed 2
1113113
2107112
3110112
4111113
5113109
6114112
7111116
8109109
Mean ± SD111.00 ± 2.33112.00 ± 2.27

Table 2. Two-stage All

EvaluationSeed 1Seed 2
1118117
2122116
3118117
4120119
5117121
6123120
7119119
8123123
Mean ± SD120.00 ± 2.39119.00 ± 2.33

Table 3. Top-25%

EvaluationSeed 1Seed 2
1128124
2125124
3127123
4122123
5125128
6126126
7125122
8122122
Mean ± SD125.00 ± 2.14124.00 ± 2.07

Table 4. Bottom-25%

EvaluationSeed 1Seed 2
1112110
2104108
3111104
4108109
5109110
6105114
7106108
8109109
Mean ± SD108.00 ± 2.83109.00 ± 2.78

These results show that performance remains relatively stable across the two different training seeds.

(Q4): Table 4 reports more Top-50 samples at 16K than at 12K (19,600 vs. 19,568). Why?

A:

Although the 16K window contains the 12K window, some trajectories are ineligible at 12K but become eligible at 16K. Specifically, a trajectory must (1) exceed 8,192 tokens and (2) contain at least one supervised assistant token within the continuation window (8K–12K or 8K–16K).

Some trajectories contain only environment feedback between 8K and 12K, with assistant actions appearing after 12K. These trajectories are therefore excluded at 12K but included at 16K, explaining the slightly larger sample count (19,600 vs. 19,568).

We will clarify this in the revised manuscript.

AuthorsOpenReview ↗

Remaining Questions and Clarifications

As the rebuttal deadline approaches, we would like to thank the reviewer again for the constructive feedback. If you have any remaining questions or concerns, we would be happy to address them.

Reviewer_gfkYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how much of a teacher trajectory to retain when training terminal agents. With the same model and training data, extending supervision from 12K to 16K yields no further gain: successful attempts change from 111 to 110 (Table 2). The proposed method starts with short-horizon training, then continues training on trajectories whose later tokens have high likelihood under that checkpoint. Its best 16K setting reaches 126 successes, with results on two additional benchmarks (Tables 5, 11, and 12). A bias–complexity bound and trajectory analyses are used to interpret the observed saturation.

Critical strengths
  1. The reliability analysis reveals useful trade-offs: from 12K to 16K, coverage decreases while Rel≥4 increases (Table 2). Appendix C.2 shows how gains and losses across tasks cancel out.

  2. The paper includes two-stage All and matched-retention Random/Bottom controls. At 12K-25%, Top achieves 125 successes versus 116 for Random, supporting selection gains in some settings (Table 10).

Critical weaknesses
  1. The independent selection gain is limited: 16K Top-75 achieves 126 successes, versus 123 for two-stage All and 122 for Random-75 (Table 10). Cost and cross-benchmark comparisons lack these controls.

  2. Likelihood is not calibrated against effective recovery or correct stopping. Training-set likelihood and successful-run lengths do not establish estimation error or over-persistence; the bound does not predict rollout success either (Figures 2–5; Appendix F).

Questions for authors
  1. How is “± variation” computed in Tables 2, 5, and 10–12? Does each of the four evaluations contain eight independent attempts per task? Do results cover independent training and Random-subset seeds? Please provide raw results and paired tests.

  2. Does the cutoff count all serialized tokens or only assistant tokens? Must candidates satisfy Li≥H2 or merely exceed 8K? Why does 16K Top-50 contain more samples than 12K Top-50 in Table 4? Please clarify the implementation and supervised-token counts.

  3. Under matched total training tokens or GPU time, including scoring costs, does Top consistently outperform two-stage All, Random, and longer-trained one-stage models? Can these controls also be reported on the other two benchmarks?

  4. After controlling for task type, length, and repetition, does likelihood predict effective recovery and correct stopping? Can you measure how often agents continue after first satisfying task checks and subsequently turn a passing state into a failure?

10 comments
AuthorsOpenReview ↗

Weakness 1 (Part 1): cost and cross-benchmark comparisons lack these controls.

(W1) (1/2): Cost and cross-benchmark comparisons lack these controls

A:

We further provide results on Terminal-Bench v2.0 and OpenThoughts-TBLite to disentangle the contributions of the two-stage training schedule, the amount of retained training data, and likelihood-based trajectory selection.

For Two-stage All, we train for two epochs at each stage (e.g., 8K followed by 12K), so the total cost is approximately the sum of both one-stage training costs. We use this baseline to isolate the effect of the two-stage schedule, not to compare computational efficiency.

Table: Reliability Comparison on Terminal-Bench v2.0

SelectionρSucc. ↑Rel≥4 ↑Rel≥6 ↑Rel≥8 ↑Fragile ↓
12K Refinement
One-stage (full)—65±2.48±0.55±0.53±0.57±0.8
Two-stage All100%70±2.410±0.86±0.53±0.55±0.5
Random25%67±2.49±0.55±0.52±0.56±0.5
Top25%72±2.211±0.56±0.53±0.54±0.5
Bottom25%63±2.78±0.54±0.52±0.58±0.8
Random50%69±2.69±0.55±0.53±0.56±0.5
Top50%71±2.29±0.56±0.54±0.55±0.5
Bottom50%66±2.49±0.55±0.52±0.55±0.5
Random75%70±2.69±0.55±0.53±0.56±0.5
Top75%70±2.49±0.55±0.54±0.56±0.5
Bottom75%69±2.68±0.55±0.53±0.58±0.8
16K Refinement
One-stage (full)—64±2.79±0.55±0.52±0.55±0.5
Two-stage All100%72±2.410±0.57±0.83±0.54±0.5
Random25%65±2.68±0.55±0.83±0.56±0.5
Top25%66±2.78±0.55±0.53±0.56±0.5
Bottom25%63±2.98±0.55±0.52±0.55±0.5
Random50%71±2.69±0.56±0.83±0.55±0.5
Top50%71±2.49±0.56±0.53±0.55±0.5
Bottom50%70±2.69±0.57±0.53±0.55±0.5
Random75%71±2.610±0.57±0.83±0.55±0.5
Top75%73±2.19±0.58±0.54±0.55±0.5
Bottom75%66±2.79±0.56±0.52±0.58±0.8

Table: Reliability Comparison on OpenThoughts-TBLite

SelectionρSucc. ↑Rel≥4 ↑Rel≥6 ↑Rel≥8 ↑Fragile ↓
12K Refinement
One-stage (full)—139±2.517±0.911±0.56±0.515±0.8
Two-stage All100%150±2.520±0.812±0.56±0.512±0.8
Random25%143±2.619±0.810±0.55±0.513±0.8
Top25%153±2.223±0.813±0.56±0.59±0.8
Bottom25%135±2.917±0.88±0.54±0.517±0.8
Random50%147±2.619±0.811±0.56±0.512±0.8
Top50%151±2.419±0.813±0.58±0.511±0.8
Bottom50%141±2.619±0.811±0.54±0.511±0.8
Random75%148±2.719±0.811±0.57±0.513±0.8
Top75%149±2.519±0.811±0.58±0.513±0.8
Bottom75%147±2.817±0.811±0.86±0.517±0.9
16K Refinement
One-stage (full)—137±2.819±0.811±0.84±0.511±0.8
Two-stage All100%153±2.521±0.815±0.56±0.510±0.9
Random25%138±2.717±0.911±0.85±0.512±0.9
Top25%141±2.717±0.811±0.56±0.513±0.8
Bottom25%135±2.917±0.811±0.84±0.511±0.8
Random50%150±2.819±0.814±0.86±0.512±0.9
Top50%151±2.319±0.813±0.56±0.511±0.8
Bottom50%149±2.619±0.815±0.86±0.511±0.8
Random75%150±2.620±0.915±0.87±0.512±0.9
Top75%155±2.219±0.817±0.59±0.511±0.8
Bottom75%141±2.819±0.813±0.84±0.517±0.8

Horizon-dependent selection: Top-25 achieves the highest successful attempts at 12K, while Top-75 performs best at 16K on both benchmarks.

Improved reliability: High-likelihood selection generally improves successful attempts and repeatability compared with one-stage training.

Selection versus random sampling: At larger retention ratios, Random and Top achieve relatively similar performance, indicating that the additional benefit of likelihood-based selection may be modest.

AuthorsOpenReview ↗

Weakness 1 (Part 2): The independent selection gain is limited

(W1) (2/2): The independent selection gain is limited

We agree with the reviewer that the independent gain from likelihood-based selection is modest, as acknowledged in Appendix E.3, where we note that “the warm-start schedule itself provides a substantial benefit.” However, the primary contribution of selective refinement is not solely to maximize performance gains, but also to improve training efficiency by achieving comparable or better reliability with fewer training trajectories and lower computational cost.

As discussed in Section 7.2, the limited additional gain can be understood through a trade-off between continuation learnability and data coverage. Selecting high-likelihood trajectories reduces the learning burden by prioritizing continuations that are more compatible with the warm-start policy. However, aggressive selection also reduces the number and diversity of training examples, potentially increasing estimation error and selection-induced approximation bias. Consequently, the benefit of selection is not necessarily monotonic with the retention ratio. Our results suggest that appropriately balancing learnability and data coverage can improve the reliability–efficiency trade-off, even when the incremental performance gain over two-stage full training is modest.

We will make the contribution of selection clearer in the revised paper.

AuthorsOpenReview ↗

Weakness 2: Likelihood is not calibrated against effective recovery or correct stopping.

(W2): Likelihood is not calibrated against effective recovery or correct stopping.

A:

We thank the reviewer for this suggestion. Our work focuses on the learnability of long-horizon trajectories, using continuation likelihood as a proxy for compatibility with the student model. However, assessing effective recovery and correct stopping requires costly, task-specific evaluation and is difficult to scale. We acknowledge this limitation and will discuss calibrating likelihood against these behaviors as an important direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.