acceptodds
Under review as a conference paper at ICLR 2027

Learning to Solve and Execute: Trajectory Refinement for Harness-Aware Agentic Fine-Tuning

Abstract

Language-model agents solve difficult real world tasks under tool execution environments and harness constraints. Supervised fine-tuning, usually called SFT, typically uses demonstrations from stronger teachers, yet these may not cover the states encountered by students. Revising student trajectories offers a way to address this mismatch, but changing an action can invalidate subsequent observations. We introduce Repair SFT, which constructs offline supervision by asking a black-box teacher to correct causal errors in student trajectories while retaining valid behavior. Revised tool calls are executed in the environment to regenerate the affected continuation, keeping actions and observations consistent. Hybrid SFT selects between repaired trajectories and direct teacher demonstrations using training-task outcomes and agent roles. We evaluate both methods with Qwen2.5-14B and Qwen3.5-9B under a shared memory harness. On the Qwen3.5 experiment, Repair SFT reduces step-budget exhaustion from 44.1% to 14.3%, while Hybrid SFT raises MuSiQue F1 from 20.24% to 49.59%. On Qwen2.5, Hybrid SFT improves MuSiQue and Qasper F1 over Direct SFT and achieves the highest IIRC F1 among the three SFT variants. These findings support student-conditioned trajectory repair and selective supervision for agentic SFT.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Agentic Fine-Tuning, Data Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uPzbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to construct trajectories for supervised fine-tuning of language-model agents. Repair SFT uses a black-box teacher to revise a student's trajectory while retaining valid behavior. When a tool call changes, the authors execute it in the environment and regenerate the dependent continuation so that actions and observations remain consistent. Hybrid SFT chooses between repaired trajectories and direct teacher demonstrations based on training-task outcomes and role-specific rules.

The authors evaluate Qwen2.5-14B and Qwen3.5-9B in a shared memory harness on retrieval-based QA and interactive tasks, with transfer tests on IIRC and BFCL. Repair SFT generally increases recorded completion and reduces budget exhaustion. Hybrid SFT improves QA F1 over Direct SFT for both students. The results also show that completing more trajectories does not consistently improve task performance.

Critical strengths
  1. Repair SFT handles a real problem in trajectory editing: changing a tool action can invalidate the observations that follow it. Executing the changed call and regenerating its continuation makes the resulting training example more coherent. It also allows the authors to use black-box teachers without teacher probabilities or aligned tokenizers.

  2. I appreciate the paper's separation of execution behavior from task success. It reports results for two students across several task families and breaks QA F1 into answer coverage and conditional answer quality. Qwen2.5 Repair SFT, for example, has nearly complete MuSiQue answer coverage but low F1. The reporting of budget exits, infrastructure failures, and uncertainty makes the completion numbers easier to interpret.

Critical weaknesses
  1. The experiments leave several parts of the proposed supervision recipe entangled. Hybrid SFT changes outcome-based selection, role-specific source choices, memory-action balance, and target lengths together. Table 11 reports 3.879 million supervised tokens for Qwen3.5 Direct SFT and 2.259 million for Hybrid SFT, despite matched instance counts and optimizer steps. A random Direct/Repair mixture matched for supervised tokens and role composition would help show what the selection rules contribute.

The case for retaining student behavior also needs more direct evidence. Repair SFT is not compared with teacher regeneration under a comparable verification and retry budget, and the paper does not quantify how much of a student trajectory survives repair, how extensive the edits are, or what errors are identified. As presented, the results support the full construction recipes more clearly than the specific value of repairing student errors. I would also like a sharper distinction from AgentRefine (ICLR 2025), which trains on error-and-refinement trajectories but uses synthesized interactions rather than repairs of frozen student trajectories executed in real environments.

  1. I am less confident in comparisons drawn from the historical evaluation records. Appendix E says that reruns may replace per-item records, attempt counts cannot be checked for every item, and the available records do not establish common output-token limits or timeouts across historical runs. These details matter when answer coverage and budget exhaustion are central outcomes. Appendix G's sensitivity analysis is helpful: Hybrid SFT remains ahead even if failed Direct SFT QA records are assigned perfect F1. That bound does not address differences in retry policies or generation limits, however. The bootstrap intervals also condition on retained outcomes and omit retry and training variability. A controlled comparison with documented generation limits and a uniform retry policy would make the policy differences easier to assess.
Questions for authors
  1. Does Hybrid SFT retain its QA advantage over a random Direct/Repair mixture matched for supervised tokens, role composition, and Step Tree memory-action balance? This would help separate the selection rules from the benefit of having access to both sources.

  2. Can the historical logs confirm identical output-token limits, timeouts, and retry policies for the main Qwen3.5 comparisons? If first-attempt outcomes cannot be recovered, could you rerun a fixed QA subset under common settings and report F1, answer coverage, and budget exhaustion?

  3. How much of each student trajectory remains after repair? How do you identify the earliest causal error, and what verification criteria and retry limits govern acceptance? Repair statistics or a comparison with teacher regeneration under the same verification budget would help substantiate the benefit of retaining student behavior.

Reviewer_ajYeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates trajectory-based supervised fine-tuning for LLM agents under tool-execution and memory-harness constraints. The authors propose Repair SFT, which uses a stronger teacher model to identify and correct errors in student-generated trajectories, while re-executing modified tool calls to maintain consistency between actions and observations. They further introduce Hybrid SFT, which combines direct teacher demonstrations and repaired student trajectories through task-outcome-based selection and role-specific sampling strategies. The framework is evaluated on Qwen2.5-14B and Qwen3.5-9B across several QA and interactive agent benchmarks.

Critical strengths
  1. Distilling agentic capabilities into smaller models is an important research direction, particularly for agents operating under complex tool-execution and memory-management constraints. Constructing executable training trajectories that are better aligned with student behavior has practical value.

  2. The experiments on Qwen3.5-9B provide encouraging evidence for student-conditioned trajectory repair. Compared with Direct SFT, Repair SFT improves MuSiQue F1 from 20.24% to 48.22% and Qasper F1 from 9.48% to 21.63%, while reducing step-budget exhaustion from 44.11% to 14.28%. These results suggest that repairing student-generated trajectories can improve agent execution and QA performance.

Critical weaknesses
  1. Missing closely related work. The paper does not discuss SCoRe (Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs, ICML 2026), which appears highly relevant. SCoRe also constructs student-centered training trajectories by identifying and correcting the earliest error in student executions, followed by SFT. This is closely related to the core idea of Repair SFT. Although the present work additionally emphasizes environment replay and introduces Hybrid SFT, the methodological distinctions and contributions beyond SCoRe remain insufficiently clarified. A direct comparison or a more detailed discussion is necessary.

  2. Limited and inconsistent improvements from Hybrid SFT. Although Hybrid SFT generally outperforms Direct SFT on QA benchmarks, its improvements over Repair SFT are relatively modest in some settings. For example, on Qwen3.5-9B, MuSiQue F1 only increases from 48.22% to 49.59%, and the reported confidence interval does not establish a significant advantage. Moreover, Hybrid SFT substantially underperforms Direct SFT on ScienceWorld (60.77 vs. 79.18). On MuSiQue, Hybrid SFT also fails to outperform the original untuned models on either student. These results raise concerns about the consistency and general effectiveness of the proposed selection strategy.

  3. Insufficient ablation studies for Hybrid SFT. Hybrid SFT combines several design choices, including task-outcome-based trajectory selection, role-specific data sources, and balancing strategies for memory-management actions. However, the paper does not adequately isolate their individual contributions. In particular, it lacks a simple random-mixture baseline with matched supervised token budgets. It is therefore unclear whether the improvements come from the proposed selection strategy or simply from combining different sources of training data. Additionally, the motivation for assigning different selection rules to the four harness roles is not sufficiently justified through controlled experiments.

Questions for authors

See above.

Reviewer_NyniOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies offline trajectory supervision for agents operating under a shared memory harness. Repair SFT asks a black-box teacher to revise a frozen student attempt, then executes changed tool calls to regenerate consistent observations. Hybrid SFT selects between repaired trajectories and direct teacher demonstrations using training-task outcomes and role-specific rules. Both approaches are evaluated against Direct SFT with Qwen2.5-14B and Qwen3.5-9B on QA and interactive benchmarks. Repair SFT improves recorded completion and reduces budget exits; Hybrid SFT improves MuSiQue and Qasper F1 over Direct SFT.

Critical strengths
  1. Executable repair produces consistent training records. Resetting and replaying the environment after a revised tool call avoids pairing an action with observations it could not have produced. The approach works with a black-box teacher and can be inspected before SFT.

  2. The paper reports meaningful QA and execution outcomes. On Qwen3.5, Hybrid SFT raises planned MuSiQue F1 from 20.24% to 49.59% and Qasper F1 from 9.48% to 25.80% relative to Direct SFT. Repair SFT lowers the reported budget-exit rate from 44.1% to 14.3%. Reporting coverage and conditional F1 helps interpret these changes.

Critical weaknesses
  1. The comparisons do not isolate repair or outcome-based selection. Hybrid SFT changes source selection, role composition, and supervised target length together. For Qwen3.5, the reported target counts are 3.879 million tokens for Direct SFT, 2.450 million for Repair SFT, and 2.259 million for Hybrid SFT. There is no random-mixture control matched for roles and tokens, nor a control showing that retaining and correcting the student’s causal error is better than a teacher rewrite or a shorter direct demonstration. Consequently, the large QA gain cannot yet be attributed to the specific proposed mechanisms.

  2. Benefits vary sharply by task. On Qwen3.5, Repair SFT lowers ScienceWorld score from 79.18 to 32.87 and pooled ALFWorld success from 95.99% to 88.32% relative to Direct SFT. Hybrid SFT also trails Direct SFT on ScienceWorld and WebShop success, while the untrained model remains stronger than all trained variants on MuSiQue. These trade-offs matter for a method presented as broadly improving agentic fine-tuning.

Questions for authors
  1. Does outcome-based Hybrid selection beat a random mixture matched for role proportions, supervised tokens, and training steps across multiple training seeds? This would materially affect whether I credit the selection rule for the QA gains.

  2. How many repaired trajectories actually preserve the student prefix and correct an identifiable causal error? How does Repair SFT compare with a length-matched teacher rewrite that also re-executes tool calls?

  3. Can the authors report fresh, standardized evaluations with a fixed retry and timeout policy, together with total model calls or tokens per task? This would clarify the execution gains and the effects of historical run consolidation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.