acceptodds
Under review as a conference paper at ICLR 2027

TRACE-SWE: Trajectory Refinement with Action-oriented Execution Evidence for Code Agents

Abstract

Effective supervised fine-tuning (SFT) of code agents benefits from trajectories that connect intermediate actions with execution feedback. Real-user interaction histories from code agents such as Claude Code provide these records. However, they interleave useful actions with redundant exploration and failed attempts, limiting their value as direct training supervision. We introduce TRACE-SWE (Trajectory Refinement with Action-oriented Execution Evidence for Code Agents), a framework that refines real-user trajectories into action-oriented supervision using recorded execution evidence. For successful trajectories assessed as completing user requests, TRACE-SWE uses recorded outcome evidence to retain relevant actions and their supporting context while removing unnecessary steps. Inspired by cloze-style learning, TRACE-SWE then masks individual actions or action sequences and reconstructs them from preceding and subsequent context. For failed trajectories with unresolved user requests, TRACE-SWE uses recorded failure evidence to construct corrective supervision. TRACE-SWE improves performance on SWE-bench Verified, Pro, and Multilingual across four open-weight models. On LING-3.0-FLASH, it achieves an average gain of 6.70 percentage points across six SWE-bench configurations under THINK inference, with improvements also extending to Terminal-Bench. TRACE-SWE also reduces average interaction rounds and non-cached tokens per solved task, indicating improved efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM、Code Agent、SWE

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_N2UeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a trajectory refinement framework that leverages real-world code agent interactions to construct supervision from both successful and failed trajectories, without requiring additional sandbox rollouts. This provides a practical approach to improving code agents using existing interaction data.

Critical strengths

The proposed method demonstrates consistent improvements across open-weight models and multiple SWE-bench benchmarks. The reported reductions in interaction rounds and token costs further demonstrate its potential practical value.

Critical weaknesses
  1. Although Table 1 discusses several closely related trajectory-supervised methods, the experiments mainly focus on internal ablations. Without comparisons against representative existing methods under comparable settings, it is difficult to establish the effectiveness and advantages of the proposed approach.

  2. The proposed masked reconstruction relies on subsequent execution results and future context. However, the paper does not compare this design with prefix-only action reconstruction. Therefore, it remains unclear whether incorporating future information provides additional benefits beyond standard action prediction.

Questions for authors
  1. Why are related trajectory-supervised methods, not included in the experimental comparisons? Can the authors provide comparisons with representative methods under comparable settings?

  2. Whether incorporating future information provides additional benefits beyond standard action prediction?

  3. Are the total training tokens and computational budgets match or are comparable across the different SFT configurations in Table 4?

  4. Can the authors provide the implementation code needed to reproduce the trajectory refinement and supervision construction processes?

2 comments

We thank the reviewer for the constructive questions and provide supplementary comparisons and clarifications.

1. Comparisons with existing methods. We compare with SWE-Lego[1] and Orchard[2] on Ling-3.0-flash under OpenHands/THINK, using comparable SFT token budgets and shared training settings. Both improve over Baseline, while TRACE-SWE exceeds SWE-Lego by 4.40/4.92/7.10 percentage points and Orchard by 3.80/4.02/5.70 points. The mean score advantages are 5.48 and 4.51 points, respectively. These results demonstrate improvements over representative trajectory-supervision methods across all three benchmarks.

MethodSWE-bench VerifiedSWE-bench ProSWE-bench Multilingual
Baseline63.8038.0049.30
SWE-Lego65.8038.7051.30
Orchard66.4039.6052.70
TRACE-SWE70.2043.6258.40

2. Single-step masked reconstruction versus prefix-only reconstruction. Using the same source trajectories, the prefix-only teacher reconstructs targets from pre-action history hth_t, while Step B.3 also retains posterior context (ot,pt)(o_t,p_t): the associated tool results and subsequent context. The rationale is that next-action prediction from a prefix is underdetermined: several actions can plausibly pursue the same subgoal, and the prefix does not reveal their eventual utility. Tool results constrain the effects of the recorded action, while subsequent context shows how those effects contribute to later progress. Together, they reduce ambiguity in reconstructing that action and provide evidence for its explanation. This makes cloze-style reconstruction a retrospective learning task that explicitly relates an action's purpose, execution, and consequences. The teacher generates targets under each context condition, so the comparison evaluates the resulting supervision constructions. On Ling-3.0-flash under OpenHands/THINK, with scores averaged over three inference runs from the same checkpoint, single-step masked reconstruction exceeds prefix-only by 2.80/3.72/6.50 points, with a mean score advantage of 4.34 points. Across all generation attempts, over 70% of prefix-only outputs differed from the recorded source actions. Multiple continuations can be valid, so this mismatch reflects the difficulty of recovering a particular recorded decision from its prefix. The downstream gains support using observed consequences to construct more informative action-level supervision.

ConstructionSWE-bench VerifiedSWE-bench ProSWE-bench Multilingual
Baseline63.8038.0049.30
Prefix-only action reconstruction63.2038.0051.50
Single-step masked reconstruction66.0041.7958.00

3. Training tokens and compute. Total training-token counts and computational budgets are close and comparable across SFT conditions. Within each model comparison, the starting checkpoint, training hyperparameters, three-epoch schedule, and evaluation protocol remain unchanged; the experimental variable is supervision construction.

4. Implementation code. We will release the implementation for trajectory refinement and supervision construction upon acceptance, together with the construction configurations.

We hope these clarifications and results address your concerns and respectfully ask you to reconsider your score.

[1] Tao C, Chen J, Jiang Y, et al. Swe-lego: Pushing the limits of supervised fine-tuning for software issue resolving[J]. arXiv preprint arXiv:2601.01426, 2026.

[2] Peng B, Yao W, Wu Q, et al. Orchard: An Open-Source Agentic Modeling Framework[J]. arXiv preprint arXiv:2605.15040, 2026.

Reviewer_N2UeOpenReview ↗

reviewer

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the newly provided comparisons with existing methods and the ablation study on prefix-only reconstruction. These results help clarify the effectiveness of the proposed approach.

However, I believe that the current evidence is still insufficient to fully establish the novelty and advantages of the proposed method over existing trajectory-supervised approaches. In addition, the reproducibility of the trajectory refinement and supervision construction processes remains limited without publicly available implementation details.

Therefore, while I acknowledge the additional evidence and clarifications, I would like to maintain my original score.

Reviewer_1tenOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TRACE-SWE converts real-user Claude Code traces into SFT data through dependency-based pruning, masked-action reconstruction, and failure correction. It aims to extract useful supervision from noisy interactions without collecting new agent rollouts.

Critical strengths
  • The paper addresses a practical problem and uses a substantial dataset that processes 45,364 raw trajectories into 19,948 training records.

  • The evaluation covers four models from two families, multiple benchmarks, and multiple agent harnesses. Consistent performance gains and reduced interaction costs are encouraging.

Critical weaknesses
  • Reusing noisy interaction traces is well motivated, but the need for this particular pipeline is less clear. The contribution appears incremental, and the benefits of combining these data-processing and supervision choices need stronger justification. I can imagine some simpler data cleaning pipelines work better.
  • For step A, rule-based dependencies may miss cross-file relationships or connect edits to unrelated verification steps. Pruning could therefore remove information needed to understand later actions, including context carried in earlier reasoning. The paper does not sufficiently establish that semantic consistency is preserved. And, if the traces contain reasoning contents, such modifications will have very big disruptions to the original model thinking process.
  • For step B, reconstruction uses future observations that are unavailable during deployment. Its advantage over conventional trajectory SFT remains unclear. Comparisons using identical teacher-generated targets, with and without future observations, would help separate the value of hindsight context from that of rewritten explanations.
  • The source and collection process of the Claude Code traces and teacher model identity are not clearly reported.
  • Although four models are evaluated, component ablations are limited to Ling-3.0-flash. Repeating key comparisons on a Qwen model would strengthen the evidence that the design choices generalize across model families.
  • Failure corrections are not execution-verified. The human audit also reports neither sample sizes nor error rates, leaving the reliability of generated diagnoses and the completeness of retained context unclear.
  • Overall, the improvement is not satisfying given the dataset size.
Questions for authors
  • Could the authors clarify the motivation for each stage and explain why they should be combined?
  • Could dependency-based refinement (Steps B.1–B.2) remove necessary context or introduce inconsistencies, particularly when trajectories contain reasoning text? How do the authors assess whether pruning preserves the information needed to justify subsequent actions?
2 comments

1. Why combine the stages? The motivation for trajectory pruning stems from our analysis, which reveals that training on raw, unpruned trajectories causes the model to exhibit cumbersome and error-prone behavior with increased turn counts during task execution. Refinement concentrates learning on outcome-related actions; Step B.3 makes the rationale behind these actions explicit through single-step masked reconstruction. However, qualitative evaluation shows that while this improves immediate continuations, errors persist in subsequent decisions, leading to suboptimal downstream outcomes. This motivates Step B.4's multi-step masked reconstruction: intervening observations connect consecutive decisions, and subsequent context reveals their collective contribution to downstream progress. Combining both reconstruction views outperforms using either in isolation. Furthermore, Step A separates successful interactions from failed ones, and Step C explicitly targets failure diagnosis and correction as training objectives, addressing a distinct practical need: the model should learn not only successful behaviors but also how to respond when attempted actions go wrong, which more closely aligns with real-world user scenarios.

2. Simpler pruning and context preservation. We test simple data cleaning including format cleaning, deduplication, length filtering, and basic trajectory trimming, and obtained training results. Our method traces dependencies backward from outcome anchors, retaining supporting observations and edit history while maintaining source order and complete tool-call–result pairings, reducing information loss caused by retaining actions in isolation. For reasoning text, we compared removing all original reasoning versus preserving source-mapped reasoning (supplemented with human and LLM checks); their similar performance indicates that supervision gains do not strictly depend on the continuous retention of original reasoning text. Fresh execution observations also provide updated states for subsequent actions, mitigating their reliance on early reasoning. However, rule matching may still miss cross-file relations or associate irrelevant verifications, so we further employed human checks to assess context adequacy, reasoning-action consistency, and evidence attribution, alongside LLM evaluations for evidence support and contextual coherence; per-sample loss monitoring revealed no anomalous spikes.

ConditionVerifiedProMultilingual
Baseline63.8038.0049.30
Simple pruning64.0038.6050.20
DC-proj.67.2038.9552.33

3. Posterior context. A prefix can support several reasonable next actions, making the particular source action underdetermined. Recorded tool results and subsequent progress add constraints on which action fits the observed interaction. Cloze-style reconstruction uses this evidence to connect the action's rationale with its consequences, turning a recorded action into an explanatory learning target. Using the same source trajectories on Ling-3.0-flash, we construct targets with and without posterior context. Cloze achieves a 4.34-point mean advantage. Over 70% of prefix-only generations did not recover the source action, a source-action mismatch rate rather than a semantic error rate.

ConstructionVerifiedProMultilingual
Baseline63.8038.0049.30
Prefix-only action reconstruction63.2038.0051.50
Single-step masked reconstruction66.0041.7958.00

4. Data and models. Internal systems record employees' private-repository interactions with consent through tool agreements and internal authorization. We anonymize personal information and remove credentials/sensitive content. Sources include Claude Opus 4.6/4.7, GLM-5, and Kimi K2.7; Kimi K2.7 judges trajectories and GLM-5.2 constructs supervision.

5. Qwen ablations. On Qwen3.5-4B under OpenHands/THINK, the three views gain 3.71/4.22/3.27 points on average; TRACE-SWE gains 6.72, supporting cross-family applicability.

SFT conditionVerifiedProMultilingual
Baseline42.8026.7042.67
SA-only41.6026.5043.60
Raw-rollout42.0026.8043.20
DsingleD_{\mathrm{single}}44.8029.5049.00
DmultiD_{\mathrm{multi}}46.6229.6048.60
DcorrectionD_{\mathrm{correction}}47.7528.9045.33
TRACE-SWE48.6031.3352.40

6. Correction reliability. Since access to users' sandbox environments is unavailable, executing patches for verification is infeasible; thus, we independently employed Kimi K2.7 for verification. A human audit reviewed over 4,000 samples (proportionally distributed across supervision views), and as the data was pre-filtered by an LLM, no significant errors were detected.

7. Improvement magnitude. We found that model scale has a substantial impact on the observed gains. Our related experiments indicate that methods from other papers reporting significant improvements on smaller models often fail to yield similar gains on larger-scale models. As Ling-3.0-flash is a model with 124B parameters, therefore, we consider the current gains to be acceptable.

Reviewer_CKNVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRACE-SWE, which turns real-user Claude Code interaction histories into supervised fine-tuning data for code agents. Successful trajectories are pruned to the steps that verified edits depend on. The kept actions are then masked and reconstructed from both earlier and later context, cloze-style, with teacher-written rationales. Failed trajectories are converted into teacher-written diagnoses and repair plans. The authors fine-tune four open models on the resulting records and report gains on SWE-bench Verified, Pro, and Multilingual, as well as on Terminal-Bench, together with lower interaction and token cost per solved task.

Critical strengths
  1. A timely and practical problem. Real-user agent logs are abundant and reflect realistic usage, but they are noisy and hard to be replayed.

  2. The gains are consistent across two model families, four model sizes, two scaffolds, and several benchmarks.

  3. The pipeline is described in detail. The anchor and edge rules, the selection weights, and all teacher and SFT prompts are provided, so others can reimplement it on their own logs.

Critical weaknesses
  1. The training data is a black box. Where the logs come from? Which models produced them? Which repositories, languages, and task types they cover< How consent and privacy were handled? Would be better if the author could describe the collection process and categorize the collected repositories.

  2. Model details are not specified. Which LLMs are used as the teacher and the Step A judge? Which models produced the original Claude Code sessions?

  3. How were the selection weights in Table 7 and the threshold η\eta chosen? Was a held-out development set used, or were they tuned on the test benchmarks? How does the method compare with randomly selecting the masked actions?

Questions for authors

See weaknesses.

1 comment

We thank the reviewer for recognizing the practical motivation and the consistent gains across models and scaffolds. For your constructive questions, we now clarify the trajectory data and model provenance, explain the action-selection weights and projection-confidence threshold, and report a supplementary comparison with random action selection.

1. Data source, composition, consent, and privacy. We use real-user code agent trajectories automatically recorded by internal Claude Code systems during employees' work on internally developed repositories (private repositories), covering everyday tasks and code development. Employees were informed of their use for model training and research through the tool-use agreement and internal data-use notices, with consent obtained through the corresponding authorization process. Before supervision construction, we anonymize personal information and remove credentials and sensitive content. These measures protect employee privacy and restrict use to the authorized purposes. The task types covered by the raw trajectories include: implementation (72.07%), debugging (56.01%), code Q&A (49.93%), operations (30.85%), review/refactoring (20.79%), and testing (15.00%). These tags are non-exclusive. Languages include TypeScript/TSX (30%), Java (37%), Python (27%) and Go (6%). Sampling approximates the source task/language composition under quality filtering, separately from the equal sampling weights of the three supervision views. Training repositories and public evaluation tasks come from different sources.

2. Source, assessment, and teacher models. Original Claude Code sessions used multiple open- and closed-weight models, including Claude Opus 4.6, Claude Opus 4.7, GLM-5, and Kimi K2.7. Kimi K2.7 performs execution-grounded routing and projection-confidence assessment, while GLM-5.2 performs teacher-based supervision construction.

3. Weights, threshold, and random selection. The weights encode a ranking preference. Our analysis of the Ling model identified code modification, verification, and the observations supporting these actions as key weaknesses. We therefore assign higher role priors to implementation, verification, and evidence acquisition. Evidence bonuses adjust this initial ranking according to recorded outcomes and dependencies, while diversity bonuses encourage coverage of different operational roles and repository paths. The specific coefficients were selected empirically from the configurations we evaluated, and were further determined as the optimal weights through comparative experiments. The projection-confidence threshold was also determined through comparative experiments to ensure that both the optimization effect and data quality meet the required standards. We selected the construction weights and threshold using downstream performance on the reported SWE-bench Verified, Pro, and Multilingual evaluation sets, while keeping the SFT hyperparameters and supervision-view sampling weights fixed. The table compares Step B.3 with random action selection for single-step masked reconstruction on Ling-3.0-flash under OpenHands/THINK. Scores are percentages averaged over three inference runs from the same checkpoint. TRACE-SWE action selection exceeds random selection by 1.60, 2.22, and 4.00 percentage points, with a mean score advantage of 2.61 points. This supports selecting consequential actions for reconstruction. The selection criteria target the model's observed weaknesses, concentrating supervision on relevant actions.

Selection conditionSWE-bench VerifiedSWE-bench ProSWE-bench Multilingual
Baseline63.8038.0049.30
Random action selection64.4039.5054.00
TRACE-SWE action selection66.0041.7958.00

If the additional information and experiments adequately address your concerns, we kindly ask you to reconsider your score. Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.