- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a trajectory refinement framework that leverages real-world code agent interactions to construct supervision from both successful and failed trajectories, without requiring additional sandbox rollouts. This provides a practical approach to improving code agents using existing interaction data.
Critical strengths
The proposed method demonstrates consistent improvements across open-weight models and multiple SWE-bench benchmarks. The reported reductions in interaction rounds and token costs further demonstrate its potential practical value.
Critical weaknesses
-
Although Table 1 discusses several closely related trajectory-supervised methods, the experiments mainly focus on internal ablations. Without comparisons against representative existing methods under comparable settings, it is difficult to establish the effectiveness and advantages of the proposed approach.
-
The proposed masked reconstruction relies on subsequent execution results and future context. However, the paper does not compare this design with prefix-only action reconstruction. Therefore, it remains unclear whether incorporating future information provides additional benefits beyond standard action prediction.
Questions for authors
-
Why are related trajectory-supervised methods, not included in the experimental comparisons? Can the authors provide comparisons with representative methods under comparable settings?
-
Whether incorporating future information provides additional benefits beyond standard action prediction?
-
Are the total training tokens and computational budgets match or are comparable across the different SFT configurations in Table 4?
-
Can the authors provide the implementation code needed to reproduce the trajectory refinement and supervision construction processes?
2 comments
reviewer
Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the newly provided comparisons with existing methods and the ablation study on prefix-only reconstruction. These results help clarify the effectiveness of the proposed approach.
However, I believe that the current evidence is still insufficient to fully establish the novelty and advantages of the proposed method over existing trajectory-supervised approaches. In addition, the reproducibility of the trajectory refinement and supervision construction processes remains limited without publicly available implementation details.
Therefore, while I acknowledge the additional evidence and clarifications, I would like to maintain my original score.
We thank the reviewer for the constructive questions and provide supplementary comparisons and clarifications.
1. Comparisons with existing methods. We compare with SWE-Lego[1] and Orchard[2] on Ling-3.0-flash under OpenHands/THINK, using comparable SFT token budgets and shared training settings. Both improve over Baseline, while TRACE-SWE exceeds SWE-Lego by 4.40/4.92/7.10 percentage points and Orchard by 3.80/4.02/5.70 points. The mean score advantages are 5.48 and 4.51 points, respectively. These results demonstrate improvements over representative trajectory-supervision methods across all three benchmarks.
2. Single-step masked reconstruction versus prefix-only reconstruction. Using the same source trajectories, the prefix-only teacher reconstructs targets from pre-action history ht, while Step B.3 also retains posterior context (ot,pt): the associated tool results and subsequent context. The rationale is that next-action prediction from a prefix is underdetermined: several actions can plausibly pursue the same subgoal, and the prefix does not reveal their eventual utility. Tool results constrain the effects of the recorded action, while subsequent context shows how those effects contribute to later progress. Together, they reduce ambiguity in reconstructing that action and provide evidence for its explanation. This makes cloze-style reconstruction a retrospective learning task that explicitly relates an action's purpose, execution, and consequences. The teacher generates targets under each context condition, so the comparison evaluates the resulting supervision constructions. On Ling-3.0-flash under OpenHands/THINK, with scores averaged over three inference runs from the same checkpoint, single-step masked reconstruction exceeds prefix-only by 2.80/3.72/6.50 points, with a mean score advantage of 4.34 points. Across all generation attempts, over 70% of prefix-only outputs differed from the recorded source actions. Multiple continuations can be valid, so this mismatch reflects the difficulty of recovering a particular recorded decision from its prefix. The downstream gains support using observed consequences to construct more informative action-level supervision.
3. Training tokens and compute. Total training-token counts and computational budgets are close and comparable across SFT conditions. Within each model comparison, the starting checkpoint, training hyperparameters, three-epoch schedule, and evaluation protocol remain unchanged; the experimental variable is supervision construction.
4. Implementation code. We will release the implementation for trajectory refinement and supervision construction upon acceptance, together with the construction configurations.
We hope these clarifications and results address your concerns and respectfully ask you to reconsider your score.
[1] Tao C, Chen J, Jiang Y, et al. Swe-lego: Pushing the limits of supervised fine-tuning for software issue resolving[J]. arXiv preprint arXiv:2601.01426, 2026.
[2] Peng B, Yao W, Wu Q, et al. Orchard: An Open-Source Agentic Modeling Framework[J]. arXiv preprint arXiv:2605.15040, 2026.