acceptodds
Under review as a conference paper at ICLR 2027

TrajFix : Correcting Failed Trajectories to Scale Verifiable Software-Engineering Data

15 views

Abstract

Training an agent on real software-engineering (SWE) tasks requires trajectories that pass the tests. However, agents struggle to produce such a trajectory on hard tasks. Failed trajectories are normally thrown away, so most of the compute spent is wasted. Worse, the discarded trajectories come from the hard tasks and provide particularly valuable training signals. To make use of these failed trajectories, we present TrajFix, a multi-agent framework that repairs failed trajectories. It keeps the correct prefix and repairs only the first step that a diagnosis agent identifies as wrong, letting the agent continue the rollout until it submits a solution. Because the correction relies on oracle information that could leak the answer, a hallucination check identifies reasoning jumps and sends those corrections back for regeneration. Under a matched rollout budget per task, TrajFix improves the resolve rate by 44.0 points over resampling while reducing the cost of producing a trajectory usable for training by 47.6%. On large-scale GitHub tasks, we achieve a recovery rate of 53.60% and turn 19171 failed trajectories into training data. Compared with training on the original successful trajectories alone, adding recovered trajectories consistently improves SWE performance. For Qwen3-8B, SWE-Bench Verified improves by 5.25 points, SWE-Bench Multilingual by 6.16 points, and SWE-Bench Pro by 5.75 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, failed trajectory repair, Software Engineering tasks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uRikOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the production of training data for software-engineering agents. Failed trajectories on hard tasks are usually discarded, which wastes sampling cost and loses the learning signal from those tasks. The authors propose TrajFix, a multi-agent framework that repairs failed trajectories: using the task's reference solution, it locates the first wrong step, corrects only that step while keeping the earlier correct part, and lets the model finish the task on its own, so that trajectories passing the tests can be used as training data. The framework also includes a check against leaking the reference solution and supports multiple repair rounds. Experiments on failed trajectories from a DeepSeek model show that TrajFix solves more tasks than resampling, especially tasks the model could not solve before, at a lower cost. The paper further analyzes the repair ability of different models and the error types of failed trajectories, and shows that fine-tuning with the recovered trajectories improves performance on several SWE-Bench benchmarks.

Critical strengths
  1. The problem is practically important, and repaired trajectories are verified by real execution. Failed trajectories on hard tasks are usually discarded, yet these hard tasks are exactly what existing datasets of successful trajectories lack most. Rather than regenerating whole trajectories, TrajFix keeps the parts of a failed trajectory that are already correct and fixes only the wrong step, thereby recovering this data. A repaired trajectory is kept only if it passes the task's tests in a real environment, which gives a more reliable guarantee of correctness than data synthesis judged only by an LLM and makes the approach useful for producing training data at scale.
  2. The paper explicitly identifies and addresses the risk of leaking the reference solution, describes the pipeline concretely, and tests it on large-scale data.
Critical weaknesses
  1. The method still relies on repeated environment execution and continuation rollouts, and its time and compute cost are not measured. TrajFix does not remove sampling. Every repair round starts a fresh container, replays the whole prefix, lets the model continue from the corrected step until it submits, and then runs the tests in another fresh container; a failed round moves on to the next round. Diagnosis, correction, hallucination detection, replay, continuation and verification run serially, and a step rejected by the hallucination detector is regenerated repeatedly. The paper measures cost only in tokens and in dollars at API prices, amortized over successful trajectories, and reports neither wall-clock time nor container and CPU time, nor the execution overhead of prefix replay and test verification. The claim that TrajFix is cheaper than resampling therefore holds only in token-priced terms, whereas time and compute are also major costs when training data is produced at scale.
  2. The experiments rest almost entirely on the failures of a single DeepSeek policy, so it is unclear whether the conclusions transfer to other models. All repaired trajectories come from DeepSeek-V4-Flash, which also performs the continuation rollouts; hallucination detection uses DeepSeek-V4-Pro, and the downstream training data also comes from DeepSeek-V4-Flash trajectories. The only experiment that varies the model replaces the diagnosis and correction roles while keeping the repaired trajectories and the continuation policy fixed, so it shows the effect of who diagnoses, not of whose trajectories are repaired. The error-type distribution likewise comes only from DeepSeek-V4-Flash failures, labeled by that same model and by Kimi-K3, with only agreeing labels retained. This distribution is the basis of the method's design: the authors argue that most failures already reach the right location and only make a wrong edit, so the prefix is worth keeping and changing a single step suffices. With a weaker policy, another model family or another scaffold, failures may occur more often during localization or even earlier, and a correction limited to one step and one function may no longer be sufficient.
Questions for authors
  1. Which information and attempt budgets are available for each Table 1 baseline? Please distinguish matched-k from equal-maximum-attempt comparisons and state whether privileged-hint or prefix-replay controls already exist.
  2. How are Recovered, Unrepaired and Resampled constructed and matched by tasks, trajectories, tokens and training updates? How are 19,171 corresponding unrepaired trajectories obtained from 17,894 original failed tasks, and are duplicates shared across arms?
7 comments

We thank the reviewer for their insightful comments. We will incorporate the suggestions in the revision.

W1: The method still relies on repeated environment execution and continuation rollouts, and its time and compute cost are not measured. The paper measures cost only in tokens and dollars, not wall-clock time, container time, or the execution overhead of prefix replay and verification.

This is a fair point about the completeness of our cost accounting. First, we note that a corrected step rejected by the hallucination detector is regenerated until it passes, but each regeneration is an additional correction call plus an additional detection call, so that cost is already inside the API cost reported in Table 1. What Table 1 omits is the cost incurred during container lifetime, which we add below by measuring container time on the same tasks. Since all tasks are executed under the same workload, the time intervals between model calls are comparable across methods.

MetricResampling (matched-k)Resampling (early stop)Resampling (4 rounds)TrajFix
Trajectories produced8221,1421,600822
Container time, total (h)498.3613.1850.9226.1
Container time / traj (min)36.432.231.916.5
Model-interaction time / traj (min)30.026.225.312.1
Tool-call time / traj (min)6.406.016.614.40
# Model interactions / traj74.468.771.133.3
Per-interaction latency (s)24.523.221.621.8

TrajFix achieves lower container time than resampling under every configuration. TrajFix runs its error attribution and correction before the container starts, so its container time is the interval from the first replayed step to the final reward computation, whereas resampling's is the lifetime from the first model interaction to the final reward computation. Both TrajFix and resampling produce trajectories of roughly 70 steps, but because TrajFix replays on average 37.4 prefix steps, its in-container model-interaction time is sharply reduced; the prefix replay re-executes already-known actions at sub-second speed, and its total cost is already counted in the in-container tool-call time.

Each trajectory runs in a container verified in the live pod to be 2 vCPUs and 8 GB on AMD EPYC, priced at the AWS m5a.large on-demand rate of $0.086 per container-hour.

Method# Succ. traj# TrajContainer-hContainer cost ($)API cost ($)Total ($)/ succ. traj ($)
Resampling (matched-k rounds)178822498.342.9250.9293.81.65
Resampling (early stop)1901,142613.152.7295.0347.71.83
Resampling (4 rounds)4721,600850.973.2450.3523.51.11
TrajFix316822226.119.4233.5252.90.80

Including container costs does not materially change the overall conclusion. Under a matched rollout budget, relative to resampling, TrajFix reduces the cost of producing one usable successful trajectory by 51.5% (47.6% without the container cost). Machine costs account for only a small share of the total, and TrajFix has the lowest cost across all reported measures. Its only additional overhead compared with resampling comes from the out-of-container diagnosis, correction, and hallucination-detection calls, whose token costs are already reported in Table 1. We will add container costs to Table 1 and provide a full breakdown of execution overhead in the appendix.

We thank the reviewer for their insightful comments. We will incorporate the suggestions in the revision.

W2: The experiments rest almost entirely on the failures of a single DeepSeek policy, so it is unclear whether the conclusions transfer to other models. The error-type distribution that motivates the design also comes only from DeepSeek-V4-Flash failures.

This is the most important robustness check, so we ran a full model-family swap in which GLM-5.3-Flash produces the failed trajectories and also performs diagnosis, correction, and continuation rollout, while hallucination detection stays fixed (DeepSeek-V4-Pro) as before. To make the results directly comparable with those in the paper, we repeated the two main comparisons in Section 3.2 using GLM-5.3-Flash under the same protocol: the resolve-rate and cost comparison and the resolve-rate comparison across rounds and task subsets. We selected the test set based on four-round resampling, including 200 never-solved tasks (Pass@4 = 0) and 200 solved-before tasks (0 < Pass@4 < 1).

We report, for each setting: the task-level resolve rate (# Resolved / Resolved %, where a task counts as resolved if any of its trajectories passes F2P); the total number of produced trajectories (# Traj); the trajectory success rate (# Succ. traj / Succ. %, the fraction of all produced trajectories that pass F2P); input and output tokens per trajectory in thousands; and cost as total and per successful trajectory.

Resolve rate and cost (cf. Table 1).

Method# ResolvedResolved (%)# Succ. traj# TrajSucc. (%)In (k)Out (k)Total ($)/ succ. traj ($)
Resampling (matched-k rounds)10426.0012286614.084,836.246.1294.82.4160
Resampling (early stop)16842.001681,23013.654,434.543.9388.12.3101
Resampling (4 rounds)16842.003561,60022.254,442.744.3508.11.4273
TrajFix33684.0033686638.804,139.860.3325.20.9679

The ranking on GLM-5.3-Flash matches that on DeepSeek-V4-Flash-0731, showing that the mechanism and the conclusions transfer to another model. Applied to GLM-5.3-Flash's own failed trajectories, TrajFix still delivers a large resolve-rate gain, and its cost per successful trajectory is still the lowest. It produces more output tokens per trajectory because of the extra diagnosis, correction, and hallucination-detection stages, but fewer input tokens because prefix replay re-executes the valid prefix instead of regenerating it, so the overall cost per usable trajectory stays the lowest.

Resolve rate by round and subset (cf. Table 6).

MethodFull R1R2R3R4Never-solved R1R2R3R4Solved-before R1R2R3R4
Resampling (matched-k rounds)23.025.025.526.02.02.02.02.044.048.049.050.0
Resampling (early stop)23.032.537.042.02.03.04.06.044.062.070.078.0
Resampling (4 rounds)23.032.537.042.02.03.04.06.044.062.070.078.0
TrajFix42.064.577.084.024.041.059.070.060.088.095.098.0

Each cell is the cumulative resolve rate after k rounds: for resampling this is Pass@k, for TrajFix it is the fraction recovered within the first k diagnosis rounds. The two subsets are never-solved (Pass@4 = 0) and solved-before (0 < Pass@4 < 1), the same difficulty split used in the paper's per-subset analysis. TrajFix achieves a higher resolve rate than every resampling setting at every round in both subsets. This is consistent with our main finding that resampling is limited by the model's own solving ability, whereas TrajFix can recover even hard tasks by replaying valid prefixes and repairing individual steps. Since both the resolve-rate and cost comparison and the per-round, per-subset comparison yield consistent results with a different model family that repairs its own failed trajectories, our conclusions are not specific to a single DeepSeek policy. We will include GLM-5.3-Flash as a second policy throughout the paper.

We thank the reviewer for their insightful comments. We will incorporate the suggestions in the revision.

Q1: Which information and attempt budgets are available for each Table 1 baseline? Please distinguish matched-k from equal-maximum-attempt comparisons and state whether privileged-hint or prefix-replay controls already exist.

We will make the budget matrix explicit. All runs use the same model DeepSeek-V4-Flash-0731, and only TrajFix uses prefix replay.

SettingAttempt budgetPrivileged hintPrefix replay
Resampling (matched-k)k = round TrajFix recovered on, else 4noneno
Resampling (early-stop)up to 4, stop on first successnoneno
Resampling (4-round)fixed 4noneno
TrajFix4 diagnosis rounds × 1 continuationgolden patch (diagnosis only)yes

Matched-k (equal-rollout-budget) sets the resampling rounds k per task equal to the number of diagnosis rounds TrajFix used to recover that task (k=4 if unrecovered), so the two consume the same rollouts per task. 4-round (equal-maximum-attempt) runs a fixed four independent resampling rollouts per task, whose cumulative rate is Pass@4, independent of when TrajFix succeeds, and thus grants resampling no less budget than matched-k.

As the matrix shows, among the runs reported in the paper only TrajFix touches the golden patch (for diagnosis only) and uses prefix replay, while the resampling runs have neither. Since this difference between the two variables may raise the reviewer's concern, we add two control settings to better support the conclusion: golden-patch-guided resampling, where resampling may consult the golden patch but a rule forbids leaking any privileged information; and resampling before the error step, which keeps the prefix without any golden-patch attribution so as to isolate the effect of prefix replay. Together with the paper's TrajFix and 4-round resampling, the four settings share the same 400 tasks and the same rollout model. We report, for each setting: Overall = the fraction of the 400 tasks recovered (or passed); Solved-before = recovery rate on tasks the model had already solved at least once by resampling (0 < Pass@4 < 1); Never-solved = recovery rate on tasks the model never solved (Pass@4 = 0); and Gold overlap (median) = the median add-line-overlap with the gold patch, which measures how line-for-line the final patch copies the reference.

SettingGolden patchPrefix replayOverallSolved-beforeNever-solvedGold overlap (median)
Golden-patch-guided resamplingas referenceno96.0%99.0%93.0%0.976
TrajFix (ours)diagnosis onlyyes79.0%89.0%69.0%0.325
Resample before the attributed error stepnoneyes51.25%85.5%17.0%0.305
Resampling (4-round)noneno47.5%81.0%14.0%0.294

(1) Golden-patch-guided resampling scores highest yet cannot serve as training data. It reaches 96.0%, but its trajectories copy the reference line-for-line (gold overlap 0.976, versus 0.325 for TrajFix), and even under a strict no-leak rule more than half the steps still mention the gold patch in their reasoning. This data leaks privileged information that a normal rollout cannot obtain and is out of the normal rollout distribution, so it cannot be used for training; TrajFix instead uses the golden patch only to attribute the error and never copies it into the trajectory, so its recovered data stays clean, which shows the golden patch is used safely as a localization signal rather than a copied answer.

(2) A substantial portion of the failed prefix is in fact correct. Resampling before the error step (51.25%) already exceeds resampling from scratch (47.5%), showing that much of the failed prefix is correct and that replaying it alone yields gains, which validates the necessity of prefix replay.

Reviewer_EXfHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a multi-agent framework for recovering failed software engineering trajectories. It identifies and corrects erroneous steps while preserving valid prefixes, enabling failed trajectories to be reused as training data. Experiments demonstrate improvements in trajectory recovery efficiency, collection cost, and downstream SWE agent performance.

Critical strengths
  1. The paper highlights computational waste in existing trajectory collection pipelines and makes a compelling observation that failed trajectories may still contain valuable intermediate steps. This provides a meaningful motivation for improving SWE training data efficiency.
  2. The comparison against three resampling baselines is well designed, with both task resolution rates and computational costs reported. The downstream SWE improvements further demonstrate the practical value of the recovered trajectories.
Critical weaknesses
  1. It remains unclear whether the improvements come from learning meaningful decision-making skills or from privileged supervision provided by the golden patch. The current experiments do not clearly distinguish between these two possibilities. This raises concerns about the quality of the recovered training signals.
  2. The paper constructs a large-scale task pool, but the collection and filtering procedures are not sufficiently documented. Although the authors report repository-level non-overlap, more details are needed to independently assess potential data leakage, as both the training data and evaluation benchmarks originate from GitHub. Additionally, the Reproducibility Statement does not specify whether the task pool, trajectories, or collection scripts will be publicly released. These omissions limit reproducibility and independent verification.
Questions for authors
  1. How much would the trajectory recovery rate decrease if the diagnosis agent had no access to the golden patch?
  2. Could the authors provide more details on task collection, filtering, deduplication, and benchmark overlap verification?
  3. Could the authors evaluate sensitivity to inaccurate error localization? For example, the authors could introduce random perturbations to the identified error step and measure their effects on recovery rate and downstream SWE performance.
4 comments

We thank the reviewer for their insightful comments. We will incorporate the suggestions in the revision.

W1 / Q1: It remains unclear whether the improvements come from learning meaningful decision-making skills or from privileged supervision provided by the golden patch. How much would the trajectory recovery rate decrease if the diagnosis agent had no access to the golden patch?

This mirrors Reviewer 1's W2. The paper already reports results for TrajFix and No-privilege resampling (4 rounds, from scratch), while the reviewer's question concerns No-privilege TrajFix, which removes the golden patch from diagnosis. To further distinguish the contribution of the golden patch from that of the repair mechanism, we add two additional controls: Golden-patch-guided resampling (no-leak constraint) and Resample before the error step. The five settings below are identical in everything except the data-collection method (same 400 tasks, same rollout model DeepSeek-V4-Flash-0731). We report, for each setting: Overall = the fraction of the 400 tasks recovered (or passed); Solved-before = recovery rate on tasks the model had already solved at least once (0 < Pass@4 < 1); Never-solved = recovery rate on tasks the model never solved (Pass@4 = 0); and Gold overlap (median) = the median add-line-overlap with the gold patch, which measures how line-for-line the final patch copies the reference.

#SettingHow the patch is usedOverallSolved-beforeNever-solvedGold overlap (median)
1Golden-patch-guided resampling (no-leak constraint)reference solution; rules forbid leakage96.00%99.0%93.0%0.976
2TrajFix (ours)localization & attribution only; rule constraints + hallucination detection block leakage79.00%89.0%69.0%0.325
3No-privilege TrajFixnot used27.75%47.5%8.0%0.250
4No-privilege resampling (4 rounds, from scratch)not used47.50%81.0%14.0%0.294
5Resample before the attributed error step (no-privilege)step location borrowed only51.25%85.5%17.0%0.305

Without access to the golden patch during diagnosis, the overall recovery rate drops by about 50 percentage points. However, the key point is that the gain comes from the golden patch's improved error-step identification and attribution, rather than copying its content into subsequent trajectory generation. Golden-patch-guided resampling achieves a 96.0% recovery rate, but results in substantial copying of the reference solution (gold overlap of 0.976), with more than half of the steps mentioning the gold patch in their reasoning. Such trajectories are unsuitable for training. In contrast, TrajFix keeps the patch out of the generated trajectory and produces recovered data with much lower gold overlap.

The golden patch is also necessary for accurate error attribution. Without the patch, the model may attribute the failure to the wrong step, leading to misguided repair actions that hurt the subsequent rollout. No-privilege TrajFix achieves only 27.75% recovery, performing worse than Resample before the error step (51.25%), which continues from the prefix without attempting error attribution. The golden patch therefore provides an essential reference for accurate error attribution.

The comparison also supports the necessity of prefix replay. Resample before the error step (51.25%) outperforms No-privilege resampling (47.5%). This suggests that a substantial portion of a failed trajectory's prefix is already correct, and preserving it while continuing from the error point is more effective than starting over. Taken together, these results show that TrajFix recovers useful trajectories by correcting errors while preserving valid prefix steps, with the golden patch guiding error attribution but not leaking into the generated trajectory. The downstream SFT gains in Table 4 further confirm that the recovered trajectories provide useful training signals.

We thank the reviewer for their insightful comments. We will incorporate the suggestions in the revision.

W2 / Q2: The task pool's collection and filtering procedures are not sufficiently documented. More details are needed to assess potential data leakage, and the Reproducibility Statement does not specify whether the pool, trajectories, or scripts will be released.

We will expand Appendix G.1 with the full construction-and-filtering pipeline, summarized below.

StageOperation and thresholdsCount
Collect reposcrawl GitHub repos across 10 languages39,725 repos
Filter reposStars ≥1,000; Forks, Issues, PRs each ≥200; primary-language share ≥60%8,000 repos
Collect + rule-filter instancesextract Issue–PR pairs from kept repos; issue closed; non-empty description, fix patch, test patch; involves code change; changed lines ≤1,000; files ≤10317,283 instances
LLM quality filterDeepSeek-V3.2 scores each task 0–10; drop tasks below 5213,766 instances
Environment build + F2P checkbuild or reuse a Docker env; test fails with only the test patch applied, passes after adding the fix patch55,366 instances from 5,101 repos

The pipeline runs top to bottom. Note that the pool construction and filtering was carried out in prior work, which is why the quality filter uses an earlier model, DeepSeek-V3.2, as the LLM judge. We first crawl 39,725 GitHub repos across ten languages (Python, Go, JavaScript, TypeScript, Java, Rust, PHP, Ruby, C, C++) and keep 8,000 that pass popularity and language-purity thresholds; from these we extract Issue–PR pairs and apply rule-based filters (closed issue; non-empty description, fix patch, and test patch; involves a code change; ≤1,000 changed lines; ≤10 files), giving 317,283 instances; an LLM quality filter (DeepSeek-V3.2, score ≥5 of 10) then narrows this to 213,766; finally, environment construction with the Fail-to-Pass (F2P) execution check yields the 55,366-instance pool from 5,101 repositories used in the paper.

Contamination handling. We retain a task only if the test suite fails on the pre-fix repository and passes after applying the golden patch. This ensures that every retained task is verifiable, while any environment that cannot reproduce this F2P behavior is discarded. Each instance is associated with a unique PR. To prevent repository-level overlap with evaluation data, we compare the owner/repo identifiers and exclude any repository that appears in SWE-Bench Verified, Multilingual, or Pro. Since both our training data and the evaluation benchmarks are sourced from GitHub, repository-level exclusion provides a relevant safeguard against contamination. Moreover, the evaluation benchmarks were released independently of our collection process and fall outside our data collection window.

Release. We will consider releasing the task pool and the trajectory sets involved in the paper so that others can reproduce the experiments.

We thank the reviewer for their insightful comments. We will incorporate the suggestions in the revision.

Q3: Could the authors evaluate sensitivity to inaccurate error localization, for example by introducing random perturbations to the identified error step and measuring the effect on recovery rate?

We ran this experiment on the same 400 tasks, with DeepSeek-V4-Flash-0731 as the rollout model, the same setup as our other experiments. After diagnosis outputs the first wrong step s*, we deliberately perturb the step that is actually corrected to a position near s*, using shifts within 5 steps and beyond 20 steps in both directions. To separate the effect of the position from the effect of the attribution, we test three attribution modes at each perturbed position:

  • keep original attribution — move the edit point but reuse the original attribution;
  • re-diagnose (self-consistent) — re-run diagnosis on the perturbed prefix and use the new attribution;
  • strip attribution (position only) — give only the perturbed position with no attribution, so recovery depends on position alone.

The cells report the recovery rate under each mode and shift.

Attribution modebefore >20before ≤5after ≤5after >20
keep original attribution72.0%78.5%74.5%74.0%
re-diagnose (self-consistent)52.5%46.5%49.0%55.5%
strip attribution (position only)42.5%33.0%12.5%16.0%

Four findings emerge from this analysis. (1) When attribution is correct, random perturbation of the error-step position has a limited effect on recovery. The keep mode remains around 75% across all shifts, including those beyond 20 steps, indicating that a correct attribution enables recovery even under substantially mislocalized edit points; this accounts for TrajFix's robustness to imperfect localization. (2) Attribution quality dominates localization accuracy. At matched positions, the recovery gap between providing a correct attribution (keep mode) and providing none (strip mode) substantially exceeds the variation across positions within either mode, indicating that the correctness of the attribution, rather than the exact edit position, is the principal determinant of recovery. (3) Without attribution, correcting after the true error is more harmful than correcting before it. In the strip mode, shifting the edit point 5 steps after the error results in only 12.5% recovery, compared with 33.0% when shifting it 5 steps before the error. A late correction leaves the root cause unaddressed in the prefix, making it harder for the subsequent rollout to recover. This supports targeting the first wrong step rather than a later, more visible symptom. (4) Re-diagnosis partially compensates for a perturbed position. The re-diagnose mode lies between the keep and strip modes: by re-attributing on the perturbed prefix, it recovers a substantial portion of the loss that the strip mode incurs under positional perturbation, supporting the multi-point progressive design in which subsequent rounds can correct an imperfect initial localization.

On the effect on downstream SWE performance. The reviewer asks about the effect of perturbation on both recovery rate and downstream SWE performance; the results above cover recovery rate. For SWE performance, our position is that the recovered trajectories carry richer learning signal, and we manually checked a sample of the perturbation-recovered trajectories and found no anomaly, so a drop in recovery quality should carry through to downstream SWE performance in the same direction. However, data collection, model training, and model evaluation together form a long-running pipeline that we cannot complete within the rebuttal window, so we prioritized the recovery-rate comparison here and will add the corresponding downstream SWE-performance experiments in the final version.

Thank you again for your valuable suggestions, and we look forward to further discussions with you.

I appreciate the response I will choose weak acceptance.

Reviewer_HEXeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TrajFix, a multi-agent framework for synthesizing training data by repairing failed software-engineering trajectories. Instead of discarding failed trajectories, TrajFix identifies and corrects the first erroneous step, replays the valid prefix, and continues the rollout. The framework also introduces hallucination detection and iterative repair to improve data quality. Experiments demonstrate significant improvements in trajectory recovery efficiency and downstream SWE performance.

Critical strengths

The paper addresses an important problem of efficiently generating high-quality training data for SWE agents.

The proposed method is intuitive, practical, and clearly presented.

The experiments demonstrate significant improvements in trajectory recovery rates and downstream performance across multiple SWE benchmarks.

Critical weaknesses
  1. Limited methodological novelty. The proposed framework mainly combines existing techniques, including LLM-based error diagnosis, correction, prefix replay, and verification. The individual components are relatively straightforward.

  2. Potential information advantage in baseline comparisons. TrajFix uses the ground-truth golden patch during diagnosis, while the resampling baselines do not have access to this information. Although using privileged information for offline data synthesis is reasonable, it is unclear how much of the improvement comes from the proposed repair mechanism rather than access to the golden patch.

  3. Insufficient comparison with alternative data generation and training approaches. The paper does not compare against recent on-policy distillation (OPD) methods, such as [1]. Although [1] is very recent and OPD targets a somewhat different setting, comparisons under similar teacher-compute budgets would help better understand the effectiveness of TrajFix.

Questions for authors

Contribution of oracle information: Since the diagnosis agent has access to the golden patch, how much of the improvement comes from this privileged information rather than the proposed trajectory repair mechanism? Would it be possible to compare against golden-patch-guided resampling?

3 comments

We thank the reviewer for their insightful comments. We will incorporate the suggestions in the revision.

W2 & Q: Potential information advantage in baseline comparisons. TrajFix uses the ground-truth golden patch during diagnosis, while the resampling baselines do not have access to this information. (Q) Since the diagnosis agent has access to the golden patch, how much of the improvement comes from this privileged information rather than the proposed repair mechanism, and would it be possible to compare against golden-patch-guided resampling?

We agree that this is an important concern. Beyond the two controls the reviewer asked for, namely (i) no-privilege trajectory repair and (ii) golden-patch-guided resampling under a strict no-leakage constraint, we added two further controls to disentangle the contributions of different factors: (iii) no-privilege trajectory repair with a stronger model (Kimi-K3) as the diagnosis/correction teacher, and (iv) direct resampling before the error step. The six settings below are identical in everything except the data-collection method. We report, for each setting: Overall = the fraction of the 400 tasks recovered (or passed); Solved-before = recovery rate on tasks the model had already solved (0 < Pass@4 < 1); Never-solved = recovery rate on tasks the model never solved (Pass@4 = 0); and Gold overlap (median) = the median add-line-overlap with the gold patch, which measures how line-for-line the final patch copies the reference.

#SettingHow the patch is usedOverallSolved-beforeNever-solvedGold overlap (median)
1Golden-patch-guided resampling (no-leak constraint)reference solution; rules forbid leakage96.00%99.0%93.0%0.976
2TrajFix (ours)localization & attribution only; rule constraints + hallucination detection block leakage79.00%89.0%69.0%0.325
3No-privilege TrajFixnot used27.75%47.5%8.0%0.250
4No-privilege TrajFix (Kimi-K3, stronger)not used29.00%50.0%8.0%0.274
5No-privilege resampling (4 rounds, from scratch)not used47.50%81.0%14.0%0.294
6Resample before the error stepstep location borrowed only51.25%85.5%17.0%0.305

Despite achieving the highest recovery rate, golden-patch-guided resampling produces unsuitable training data. First, even though the prompt strictly forbids citing any privileged information, more than 50% of the steps still mention the gold patch in their reasoning. Second, the extremely high gold overlap indicates that the generated code largely reproduces the reference solution, resulting in limited diversity and making the trajectories unsuitable as training data. In contrast, TrajFix uses the golden patch only to guide error diagnosis without leaking its information into the generated trajectory. With hallucination detection in single-step repair, the corrected step's chain-of-thought can only use information from the problem statement or observations obtained in previous steps, all of which are available in a normal rollout. Moreover, its gold overlap (0.325) is only slightly higher than that of no-privilege resampling (0.294) and much lower than that of golden-patch-guided resampling (0.976). This shows the safety of introducing the golden patch through single-step trajectory repair.

Removing privileged information from error attribution reduces the recovery rate by about 50 percentage points. Importantly, the gain comes from the golden patch's improved error-step identification and attribution, rather than copying its content into subsequent trajectory generation. (1) The golden patch is necessary for accurate error attribution. This is supported by the fact that No-privilege TrajFix (27.75%) performs worse than Resample before the error step (51.25%). Without the golden patch, TrajFix may produce error attributions that deviate from the actual error, leading to misguided repair actions that hurt the subsequent rollout. (2) Resample before the error step also slightly outperforms No-privilege resampling. This suggests that a substantial portion of the prefix in a failed trajectory is already correct. Preserving these steps and continuing from the error point is therefore more effective than starting over, providing evidence for the necessity of prefix replay. (3) Even when a stronger model (Kimi-K3) performs attribution without privileged information, the recovery rate remains low and is even lower than that of No-privilege resampling. This suggests that the bottleneck is not the diagnosis model's capability, but the lack of a golden patch to guide error attribution. Taken together, these results show that the combination of golden-patch-guided attribution, minimal single-step repair, and prefix replay is an effective design for balancing high recovery and training-data usability.

We thank the reviewer for their insightful comments. We will incorporate the suggestions in the revision.

W1: Limited methodological novelty. The proposed framework mainly combines existing techniques, including LLM-based error diagnosis, correction, prefix replay, and verification. The individual components are relatively straightforward.

The reviewer is right that the individual agents, namely LLM-based error diagnosis, correction, prefix replay, and verification, are each existing techniques. Our contribution is not to propose a new component but to show how to connect these components into a loop that works well together, so that oracle-guided repair produces clean training data, which prior work does not achieve. Three design choices in the loop matter. First, the golden patch is used only to locate the first wrong step, and it is never copied into the trajectory. Second, the Admissibility Constraint and Hallucination Detection reject any corrected step that leaks the answer. Third, prefix replay lets the model continue without supervision, so the recovered trajectory stays on-policy. The ablation shows that these choices are what make the loop effective. Removing them raises the hallucination rate by 5.6 times, from 2.53% to 14.23%, which means the recovered data would leak oracle information without them. The value therefore comes from how the known components are composed and constrained, not from any one of them alone.

W3: Insufficient comparison with alternative data generation and training approaches, in particular recent on-policy distillation (OPD) methods. Comparisons under similar teacher-compute budgets would help.

We see OPD and TrajFix as operating at different stages of the training pipeline, so they are complementary rather than competing. TrajFix is a data-production method that repairs failed trajectories into verifiable successful ones for supervised fine-tuning. OPD is a policy-transfer method that optimizes a student against a teacher during training. The two can be composed in sequence. A natural setup is to first fine-tune a model on the data that TrajFix recovers, and then use that stronger SFT model as the starting point for OPD. In this way TrajFix improves the student before OPD begins, and OPD further refines the policy on top. A matched-teacher-compute comparison is a fair request, but a full OPD pipeline under matched budget is a sizeable experiment that we cannot complete within the rebuttal window, so we may add it in a future version if resources allow.

Thank you again for your valuable suggestions, and we look forward to further discussions with you.

Thank you for your response. I will maintain my original score as acceptance.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (1/3)

To All Reviewers and the Area Chair

We thank all reviewers for their constructive feedback. This comment gives an overview of the revision. Each reviewer's individual points are answered one by one in a separate thread.

Background. TrajFix turns failed software-engineering trajectories into verifiable training data instead of discarding them. For a failed trajectory, it locates the first wrong step and attributes the error, corrects that single step under two leakage guards (the Admissibility Constraint and Hallucination Detection), replays the already-correct prefix, and lets the model finish the task on its own.

Three main results.

  • High recovery at low cost. On the same failed-trajectory set, TrajFix recovers far more tasks than resampling, especially tasks the model cannot solve by resampling, at the lowest cost per successful trajectory.
  • The recovered data is clean. The golden patch is used for error attribution only and is never copied into the trajectory.
  • The recovered data improves downstream SWE performance. Fine-tuning a model on the recovered trajectories improves downstream SWE-Bench performance.

Revision. The reviews share five concerns. For each we briefly summarize the added experiments and the resulting conclusions. Unless noted, new experiments use the paper's 400-task set and rollout model DeepSeek-V4-Flash-0731.

1. How much of the gain comes from the golden patch rather than the repair mechanism, and is the recovered data clean? (HEXe W2/Q; EXfH W1/Q1; uRik Q1)

Added. Four new control experiments, all differing only in the data-collection method, with the add-line-overlap probe to measure similarity to the golden patch.

  • No-privilege TrajFix — the same repair loop with the golden patch removed from diagnosis.
  • Golden-patch-guided resampling — resampling may consult the golden patch, but a rule forbids leaking any privileged information.
  • No-privilege TrajFix (Kimi-K3) — no-privilege diagnosis done by a stronger model, to test whether a better diagnoser can replace the patch.
  • Resample before the error step — resample from the step just before the error.
SettingHow the patch is usedResolved (%)Gold overlap (median)
Golden-patch-guided resampling (no-leak)reference solution; rules forbid leakage96.0%0.976
TrajFix (ours)attribution only79.0%0.325
Resample before the error stepstep location only51.25%0.305
No-privilege resampling (4 rounds)not used47.5%0.294
No-privilege TrajFix (Kimi-K3)not used29.0%0.274
No-privilege TrajFixnot used27.75%0.250

Result.

  • The golden patch's gain is large, yet TrajFix uses it only for localization and attribution and never copies it into the trajectory, so its gold overlap (0.325) stays close to no-privilege resampling (0.294) and far below the copy level (1).
  • Golden-patch-guided resampling is unusable despite 96.0%. It copies the reference line-for-line (overlap 0.976), and even under the strict no-leak rule over half its steps still mention the gold patch, so it leaks privileged information and cannot be training data.
  • Patch-guided attribution is necessary. Removing the patch drops recovery by about 50 points, and a no-privilege attribution (27.75%) is even worse than no attribution at all (51.25% for resample-before-error), because a wrong diagnosis misleads the repair; a stronger diagnoser (Kimi-K3, 29.0%) does not close the gap either.
  • The failed prefix is largely correct. Resampling from before the error (51.25%) beats resampling from scratch (47.5%), showing much of the prefix is already right and that replaying it is what prefix replay exploits.

2. Beyond token cost, should compute cost also be accounted for? (uRik W1)

Added. Wall-clock time on the same tasks, priced at the AWS m5a.large rate actually used (2 vCPU / 8 GB pod, $0.086 per container-hour); for every setting we compute the compute cost on top of the token cost.

Method# Succ.TrajWall-clock cost ($)API cost ($)Total ($)/ succ. traj ($)
Resampling (matched-k)17842.9250.9293.81.65
Resampling (early stop)19052.7295.0347.71.83
Resampling (4 rounds)47273.2450.3523.51.11
TrajFix31626.8233.5260.30.82

Result. Under the same rollout budget, TrajFix lowers the cost of producing one usable successful trajectory by 50.1% relative to resampling (47.6% without the wall-clock cost). TrajFix and resampling both produce trajectories of roughly 70 steps, but because TrajFix replays on average 37.4 prefix steps, its in-container model-interaction time is sharply reduced; prefix replay re-executes already-known actions at sub-second speed, and that cost is already counted in the wall-clock time.

AuthorsOpenReview ↗

General Response (2/3)

We thank all reviewers for their constructive feedback. This comment gives an overview of the revision. Each reviewer's individual points are answered one by one in a separate thread.

3. The method is applied to a single DeepSeek model; do the conclusions transfer to other models? (uRik W2)

Added. A full model-family swap in which GLM-5.3-Flash produces the failed trajectories and performs diagnosis, correction, and continuation (hallucination detection fixed to DeepSeek-V4-Pro). We repeat the same resolve-rate-and-cost comparison and the per-round, per-subset success-rate comparison.

Method# ResolvedResolved (%)# Succ. traj# TrajSucc. (%)In (k)Out (k)Total ($)/ succ. traj ($)
Resampling (matched-k)10426.0012286614.084,836.246.1294.82.4160
Resampling (early stop)16842.001681,23013.654,434.543.9388.12.3101
Resampling (4 rounds)16842.003561,60022.254,442.744.3508.11.4273
TrajFix33684.0033686638.804,139.860.3325.20.9679

Result. The conclusions transfer. On GLM-5.3-Flash's own failed trajectories, TrajFix still gives the largest resolve-rate gain and the lowest cost per successful trajectory.

4. How much does perturbing the error localization affect recovery? (EXfH Q3; uRik Q2)

Added. A perturbation study. After diagnosis outputs the first wrong step s*, we move the actually-corrected step to a position near s* (within 5 steps and beyond 20 steps, in both directions), under three attribution modes: keep (reuse the original attribution), re-diagnose (re-attribute on the perturbed prefix), and strip (give only the position, no attribution), to separate the effect of position from that of attribution.

Attribution modebefore >20before ≤5after ≤5after >20
keep original attribution72.0%78.5%74.5%74.0%
re-diagnose (self-consistent)52.5%46.5%49.0%55.5%
strip attribution (position only)42.5%33.0%12.5%16.0%

Result.

  • A correct attribution makes localization error nearly harmless. Keep stays around 75% across all shifts, including beyond 20 steps, which is why TrajFix tolerates imperfect localization.
  • Attribution quality dominates position. At matched positions, the keep-minus-strip gap far exceeds the variation across positions within either mode, so attribution correctness, not the exact edit position, drives recovery.
  • Without attribution, correcting after the true error is worse than before it. In strip, a 5-step shift after the error gives only 12.5% versus 33.0% before it, because a late edit leaves the root cause unfixed in the prefix; this supports targeting the first wrong step.
  • Re-diagnosis partially compensates. Re-diagnose sits between keep and strip, recovering much of what strip loses under perturbation, which supports the multi-round progressive design that can correct an imperfect initial localization.
AuthorsOpenReview ↗

General Response (3/3)

We thank all reviewers for their constructive feedback. This comment gives an overview of the revision. Each reviewer's individual points are answered one by one in a separate thread.

5. How is the task pool collected and filtered, and how is contamination handled? (EXfH W2/Q2)

Added. The full construction-and-filtering funnel with per-stage thresholds, together with repository-level contamination handling against the evaluation benchmarks.

Result. The funnel runs top to bottom. We crawl 39,725 GitHub repos across ten languages (Python, Go, JavaScript, TypeScript, Java, Rust, PHP, Ruby, C, C++) and keep 8,000 that pass popularity and language-purity thresholds; from these we extract Issue–PR pairs and apply rule-based filters, giving 317,283 instances; an LLM quality filter (DeepSeek-V3.2, score ≥5 of 10) narrows this to 213,766; finally, environment construction with the Fail-to-Pass (F2P) execution check yields the 55,366-instance pool from 5,101 repositories used in the paper. The F2P check is the quality gate: a task is kept only if the test suite fails on the pre-fix repo and passes after the golden patch. To guard against contamination, each instance is unique per PR and we verify repository-level non-overlap against SWE-Bench Verified, Multilingual, and Pro by matching on owner/repo, so no repository in the pool appears in any evaluation benchmark.

StageOperation and thresholdsCount
Collect reposcrawl GitHub repos across 10 languages39,725 repos
Filter reposStars ≥1,000; Forks, Issues, PRs each ≥200; primary-language share ≥60%8,000 repos
Collect + rule-filter instancesIssue–PR pairs; issue closed; non-empty description, fix patch, test patch; code change; ≤1,000 changed lines; ≤10 files317,283 instances
LLM quality filterDeepSeek-V3.2 scores each task 0–10; drop tasks below 5213,766 instances
Environment build + F2P checkbuild or reuse a Docker env; test fails with only the test patch, passes after the fix patch55,366 instances

Revised PDF. The new results are now in the paper, with all additions and edits marked in blue. The changes are made in the following sections.

  1. Golden-patch gain and clean data. A sentence in the Method section, with the full leakage-control ablation added to Appendix G.
  2. Compute cost. Table 1 now reports the wall-clock cost alongside the token cost, with the container / out-of-container breakdown in Appendix C.
  3. Transfer to another model. Table 1 adds GLM-5.3-Flash as a second policy, with its per-round, per-subset and wall-clock results in Appendix C; the Abstract and Conclusion are updated accordingly.
  4. Localization perturbation. The perturbation study is added to Appendix H and referenced in the Method section.
  5. Task-pool construction and contamination. The full funnel with repository-level decontamination is added to Appendix I.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.