- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses the production of training data for software-engineering agents. Failed trajectories on hard tasks are usually discarded, which wastes sampling cost and loses the learning signal from those tasks. The authors propose TrajFix, a multi-agent framework that repairs failed trajectories: using the task's reference solution, it locates the first wrong step, corrects only that step while keeping the earlier correct part, and lets the model finish the task on its own, so that trajectories passing the tests can be used as training data. The framework also includes a check against leaking the reference solution and supports multiple repair rounds. Experiments on failed trajectories from a DeepSeek model show that TrajFix solves more tasks than resampling, especially tasks the model could not solve before, at a lower cost. The paper further analyzes the repair ability of different models and the error types of failed trajectories, and shows that fine-tuning with the recovered trajectories improves performance on several SWE-Bench benchmarks.
Critical strengths
- The problem is practically important, and repaired trajectories are verified by real execution. Failed trajectories on hard tasks are usually discarded, yet these hard tasks are exactly what existing datasets of successful trajectories lack most. Rather than regenerating whole trajectories, TrajFix keeps the parts of a failed trajectory that are already correct and fixes only the wrong step, thereby recovering this data. A repaired trajectory is kept only if it passes the task's tests in a real environment, which gives a more reliable guarantee of correctness than data synthesis judged only by an LLM and makes the approach useful for producing training data at scale.
- The paper explicitly identifies and addresses the risk of leaking the reference solution, describes the pipeline concretely, and tests it on large-scale data.
Critical weaknesses
- The method still relies on repeated environment execution and continuation rollouts, and its time and compute cost are not measured. TrajFix does not remove sampling. Every repair round starts a fresh container, replays the whole prefix, lets the model continue from the corrected step until it submits, and then runs the tests in another fresh container; a failed round moves on to the next round. Diagnosis, correction, hallucination detection, replay, continuation and verification run serially, and a step rejected by the hallucination detector is regenerated repeatedly. The paper measures cost only in tokens and in dollars at API prices, amortized over successful trajectories, and reports neither wall-clock time nor container and CPU time, nor the execution overhead of prefix replay and test verification. The claim that TrajFix is cheaper than resampling therefore holds only in token-priced terms, whereas time and compute are also major costs when training data is produced at scale.
- The experiments rest almost entirely on the failures of a single DeepSeek policy, so it is unclear whether the conclusions transfer to other models. All repaired trajectories come from DeepSeek-V4-Flash, which also performs the continuation rollouts; hallucination detection uses DeepSeek-V4-Pro, and the downstream training data also comes from DeepSeek-V4-Flash trajectories. The only experiment that varies the model replaces the diagnosis and correction roles while keeping the repaired trajectories and the continuation policy fixed, so it shows the effect of who diagnoses, not of whose trajectories are repaired. The error-type distribution likewise comes only from DeepSeek-V4-Flash failures, labeled by that same model and by Kimi-K3, with only agreeing labels retained. This distribution is the basis of the method's design: the authors argue that most failures already reach the right location and only make a wrong edit, so the prefix is worth keeping and changing a single step suffices. With a weaker policy, another model family or another scaffold, failures may occur more often during localization or even earlier, and a correction limited to one step and one function may no longer be sufficient.
Questions for authors
- Which information and attempt budgets are available for each Table 1 baseline? Please distinguish matched-k from equal-maximum-attempt comparisons and state whether privileged-hint or prefix-replay controls already exist.
- How are Recovered, Unrepaired and Resampled constructed and matched by tasks, trajectories, tokens and training updates? How are 19,171 corresponding unrepaired trajectories obtained from 17,894 original failed tasks, and are duplicates shared across arms?
7 comments
We thank the reviewer for their insightful comments. We will incorporate the suggestions in the revision.
W2: The experiments rest almost entirely on the failures of a single DeepSeek policy, so it is unclear whether the conclusions transfer to other models. The error-type distribution that motivates the design also comes only from DeepSeek-V4-Flash failures.
This is the most important robustness check, so we ran a full model-family swap in which GLM-5.3-Flash produces the failed trajectories and also performs diagnosis, correction, and continuation rollout, while hallucination detection stays fixed (DeepSeek-V4-Pro) as before. To make the results directly comparable with those in the paper, we repeated the two main comparisons in Section 3.2 using GLM-5.3-Flash under the same protocol: the resolve-rate and cost comparison and the resolve-rate comparison across rounds and task subsets. We selected the test set based on four-round resampling, including 200 never-solved tasks (Pass@4 = 0) and 200 solved-before tasks (0 < Pass@4 < 1).
We report, for each setting: the task-level resolve rate (# Resolved / Resolved %, where a task counts as resolved if any of its trajectories passes F2P); the total number of produced trajectories (# Traj); the trajectory success rate (# Succ. traj / Succ. %, the fraction of all produced trajectories that pass F2P); input and output tokens per trajectory in thousands; and cost as total and per successful trajectory.
Resolve rate and cost (cf. Table 1).
| Method | # Resolved | Resolved (%) | # Succ. traj | # Traj | Succ. (%) | In (k) | Out (k) | Total ($) | / succ. traj ($) |
|---|---|---|---|---|---|---|---|---|---|
| Resampling (matched-k rounds) | 104 | 26.00 | 122 | 866 | 14.08 | 4,836.2 | 46.1 | 294.8 | 2.4160 |
| Resampling (early stop) | 168 | 42.00 | 168 | 1,230 | 13.65 | 4,434.5 | 43.9 | 388.1 | 2.3101 |
| Resampling (4 rounds) | 168 | 42.00 | 356 | 1,600 | 22.25 | 4,442.7 | 44.3 | 508.1 | 1.4273 |
| TrajFix | 336 | 84.00 | 336 | 866 | 38.80 | 4,139.8 | 60.3 | 325.2 | 0.9679 |
The ranking on GLM-5.3-Flash matches that on DeepSeek-V4-Flash-0731, showing that the mechanism and the conclusions transfer to another model. Applied to GLM-5.3-Flash's own failed trajectories, TrajFix still delivers a large resolve-rate gain, and its cost per successful trajectory is still the lowest. It produces more output tokens per trajectory because of the extra diagnosis, correction, and hallucination-detection stages, but fewer input tokens because prefix replay re-executes the valid prefix instead of regenerating it, so the overall cost per usable trajectory stays the lowest.
Resolve rate by round and subset (cf. Table 6).
| Method | Full R1 | R2 | R3 | R4 | Never-solved R1 | R2 | R3 | R4 | Solved-before R1 | R2 | R3 | R4 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Resampling (matched-k rounds) | 23.0 | 25.0 | 25.5 | 26.0 | 2.0 | 2.0 | 2.0 | 2.0 | 44.0 | 48.0 | 49.0 | 50.0 |
| Resampling (early stop) | 23.0 | 32.5 | 37.0 | 42.0 | 2.0 | 3.0 | 4.0 | 6.0 | 44.0 | 62.0 | 70.0 | 78.0 |
| Resampling (4 rounds) | 23.0 | 32.5 | 37.0 | 42.0 | 2.0 | 3.0 | 4.0 | 6.0 | 44.0 | 62.0 | 70.0 | 78.0 |
| TrajFix | 42.0 | 64.5 | 77.0 | 84.0 | 24.0 | 41.0 | 59.0 | 70.0 | 60.0 | 88.0 | 95.0 | 98.0 |
Each cell is the cumulative resolve rate after k rounds: for resampling this is Pass@k, for TrajFix it is the fraction recovered within the first k diagnosis rounds. The two subsets are never-solved (Pass@4 = 0) and solved-before (0 < Pass@4 < 1), the same difficulty split used in the paper's per-subset analysis. TrajFix achieves a higher resolve rate than every resampling setting at every round in both subsets. This is consistent with our main finding that resampling is limited by the model's own solving ability, whereas TrajFix can recover even hard tasks by replaying valid prefixes and repairing individual steps. Since both the resolve-rate and cost comparison and the per-round, per-subset comparison yield consistent results with a different model family that repairs its own failed trajectories, our conclusions are not specific to a single DeepSeek policy. We will include GLM-5.3-Flash as a second policy throughout the paper.
We thank the reviewer for their insightful comments. We will incorporate the suggestions in the revision.
Q1: Which information and attempt budgets are available for each Table 1 baseline? Please distinguish matched-k from equal-maximum-attempt comparisons and state whether privileged-hint or prefix-replay controls already exist.
We will make the budget matrix explicit. All runs use the same model DeepSeek-V4-Flash-0731, and only TrajFix uses prefix replay.
| Setting | Attempt budget | Privileged hint | Prefix replay |
|---|---|---|---|
| Resampling (matched-k) | k = round TrajFix recovered on, else 4 | none | no |
| Resampling (early-stop) | up to 4, stop on first success | none | no |
| Resampling (4-round) | fixed 4 | none | no |
| TrajFix | 4 diagnosis rounds × 1 continuation | golden patch (diagnosis only) | yes |
Matched-k (equal-rollout-budget) sets the resampling rounds k per task equal to the number of diagnosis rounds TrajFix used to recover that task (k=4 if unrecovered), so the two consume the same rollouts per task. 4-round (equal-maximum-attempt) runs a fixed four independent resampling rollouts per task, whose cumulative rate is Pass@4, independent of when TrajFix succeeds, and thus grants resampling no less budget than matched-k.
As the matrix shows, among the runs reported in the paper only TrajFix touches the golden patch (for diagnosis only) and uses prefix replay, while the resampling runs have neither. Since this difference between the two variables may raise the reviewer's concern, we add two control settings to better support the conclusion: golden-patch-guided resampling, where resampling may consult the golden patch but a rule forbids leaking any privileged information; and resampling before the error step, which keeps the prefix without any golden-patch attribution so as to isolate the effect of prefix replay. Together with the paper's TrajFix and 4-round resampling, the four settings share the same 400 tasks and the same rollout model. We report, for each setting: Overall = the fraction of the 400 tasks recovered (or passed); Solved-before = recovery rate on tasks the model had already solved at least once by resampling (0 < Pass@4 < 1); Never-solved = recovery rate on tasks the model never solved (Pass@4 = 0); and Gold overlap (median) = the median add-line-overlap with the gold patch, which measures how line-for-line the final patch copies the reference.
| Setting | Golden patch | Prefix replay | Overall | Solved-before | Never-solved | Gold overlap (median) |
|---|---|---|---|---|---|---|
| Golden-patch-guided resampling | as reference | no | 96.0% | 99.0% | 93.0% | 0.976 |
| TrajFix (ours) | diagnosis only | yes | 79.0% | 89.0% | 69.0% | 0.325 |
| Resample before the attributed error step | none | yes | 51.25% | 85.5% | 17.0% | 0.305 |
| Resampling (4-round) | none | no | 47.5% | 81.0% | 14.0% | 0.294 |
(1) Golden-patch-guided resampling scores highest yet cannot serve as training data. It reaches 96.0%, but its trajectories copy the reference line-for-line (gold overlap 0.976, versus 0.325 for TrajFix), and even under a strict no-leak rule more than half the steps still mention the gold patch in their reasoning. This data leaks privileged information that a normal rollout cannot obtain and is out of the normal rollout distribution, so it cannot be used for training; TrajFix instead uses the golden patch only to attribute the error and never copies it into the trajectory, so its recovered data stays clean, which shows the golden patch is used safely as a localization signal rather than a copied answer.
(2) A substantial portion of the failed prefix is in fact correct. Resampling before the error step (51.25%) already exceeds resampling from scratch (47.5%), showing that much of the failed prefix is correct and that replaying it alone yields gains, which validates the necessity of prefix replay.
We thank the reviewer for their insightful comments. We will incorporate the suggestions in the revision.
This is a fair point about the completeness of our cost accounting. First, we note that a corrected step rejected by the hallucination detector is regenerated until it passes, but each regeneration is an additional correction call plus an additional detection call, so that cost is already inside the API cost reported in Table 1. What Table 1 omits is the cost incurred during container lifetime, which we add below by measuring container time on the same tasks. Since all tasks are executed under the same workload, the time intervals between model calls are comparable across methods.
TrajFix achieves lower container time than resampling under every configuration. TrajFix runs its error attribution and correction before the container starts, so its container time is the interval from the first replayed step to the final reward computation, whereas resampling's is the lifetime from the first model interaction to the final reward computation. Both TrajFix and resampling produce trajectories of roughly 70 steps, but because TrajFix replays on average 37.4 prefix steps, its in-container model-interaction time is sharply reduced; the prefix replay re-executes already-known actions at sub-second speed, and its total cost is already counted in the in-container tool-call time.
Each trajectory runs in a container verified in the live pod to be 2 vCPUs and 8 GB on AMD EPYC, priced at the AWS m5a.large on-demand rate of $0.086 per container-hour.
Including container costs does not materially change the overall conclusion. Under a matched rollout budget, relative to resampling, TrajFix reduces the cost of producing one usable successful trajectory by 51.5% (47.6% without the container cost). Machine costs account for only a small share of the total, and TrajFix has the lowest cost across all reported measures. Its only additional overhead compared with resampling comes from the out-of-container diagnosis, correction, and hallucination-detection calls, whose token costs are already reported in Table 1. We will add container costs to Table 1 and provide a full breakdown of execution overhead in the appendix.