- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces FAIL2TASK, a framework for converting failed coding-agent trajectories into new executable tasks when the original repository and execution environment are unavailable. The framework first extracts evidence-supported failure mechanisms from a source trajectory, selects or creates an environment intended to instantiate the relevant conditions, and generates a task with an instruction, verifier, and reference solution. It then validates the task through a fail-before/fix-after check and repeatedly runs the source agent configuration on the task; an LLM judge accepts the task only when the replay is unsuccessful and appears to exhibit a semantically corresponding failure pattern.
The evaluation uses 96 failures produced by GPT-5.4-mini, evenly divided between SWE-bench Verified and Terminal-Bench 2.0. Within a budget of 15 construction attempts per source, the pipeline obtains an accepted task for every case. The authors report 80.18% author-reviewed failure-pattern alignment, 58.33% three-run failure persistence, and 85.66% cross-run failure-pattern overlap. The paper further uses reconstructed tasks to optimize several agent harnesses with AutoSaddler and reports higher pass@3 on the corresponding source tasks than when optimization uses failure-matched tasks from SWE-smith, SWE-ReBench, Terminal-Lego, or SETA.
Critical strengths
The paper identifies an important and practically motivated setting: a failure trajectory may be shareable even when the original code and environment are unavailable. Reconstructing an executable surrogate rather than claiming to recover the original task is a useful and appropriately scoped formulation.FAIL2TASK is a coherent closed-loop system. It combines causal failure analysis, task construction, execution checks, replay with the same agent configuration, and semantic screening rather than relying only on textual similarity between task descriptions.The paper evaluates both repository-level software repair and broader terminal tasks. The ablations show large degradation when failure analysis or replay screening is removed, which supports the importance of the main components.The appendices provide substantial protocol detail, including the failure-pairing prompt, overlap calculation, baseline-matching procedure, data-membership rules, and mismatch analysis.The downstream harness-evolution experiment is valuable because it tests whether the reconstructed tasks are useful rather than merely similar according to a judge. The qualitative mismatch analysis is candid and helps clarify why transfer remains incomplete.
Critical weaknesses
1.Task validity is not independently established. The same construction process produces the instruction, private verifier, environment, and reference solution, and validity is demonstrated mainly by showing that the initial state fails while that generated reference solution passes. This does not rule out weak, incomplete, overfitted, or unintended verifiers. No independent expert audit, mutation analysis, alternative-solution study, or test-adequacy evaluation is reported. This is especially important because the paper's central artifact is an executable task, not only a trajectory with a matching failure label.
2.There is a substantial selection effect in the reconstruction results. A candidate can be refined for up to 15 attempts using replay-judge feedback until one stochastic run of the same agent fails in the desired way. Therefore, the 100% final acceptance rate is a pipeline search result rather than an out-of-sample success rate. The fresh-run analysis partially addresses this concern, but only 58.33% of tasks make the agent fail in all three runs. Results should be reported by benchmark and should include how alignment changes across independent fresh runs rather than centering the accepted or run-1 replay.
3.The primary 80.18% alignment metric is not sufficiently independent of the construction loop. The final annotation is model-assisted, the authors inspect the model's annotation and the accepted-round analysis.json, and review is not blinded. Although the authors explicitly disclose this and report inter-annotator agreement, the same conceptual decomposition used during construction can anchor the final assessment. Evaluation by independent annotators who are blind to acceptance status, method variant, and generated analysis would provide stronger evidence.
Questions for authors
1.What independent evidence establishes that the generated verifier fully and correctly represents the generated instruction? Could the authors provide a blinded human audit of a representative sample, verifier mutation scores, or results from independently produced alternative solutions?
2. For each benchmark separately, what are failure-pattern alignment and failure rate on each of the three fresh replays? How many tasks reproduce the aligned source mechanism in all three runs, rather than merely remaining unsolved?
3. Can the authors repeat the final alignment assessment with independent annotators who are blind to the method/ablation, acceptance decision, and accepted-round analysis.json? How does that score compare with 80.18%?
4. What are the fresh unevolved pass@3 scores on exactly the same 48 source tasks for every harness in Table 3? Please report absolute and paired improvements from round 0, not only post-evolution comparisons among training-task sources.
5. How stable are the AutoSaddler results across optimization seeds or repeated runs? Do paired bootstrap confidence intervals or randomization tests support the claimed advantage over the strongest baseline?
3 comments
Response to reviewer tw2q
Q3 The primary 80.18% alignment metric is not sufficiently independent of the construction loop. The final annotation is model-assisted, the authors inspect the model's annotation and the accepted-round analysis.json, and review is not blinded. Although the authors explicitly disclose this and report inter-annotator agreement, the same conceptual decomposition used during construction can anchor the final assessment. Evaluation by independent annotators who are blind to acceptance status, method variant, and generated analysis would provide stronger evidence. Can the authors repeat the final alignment assessment with independent annotators who are blind to the method/ablation, acceptance decision, and accepted-round analysis.json? How does that score compare with 80.18%?
A3 Table 2 reports 80.18% alignment under the original annotation procedure. Two authors reviewed model annotations with access to the construction analysis, and their agreement on those review decisions was κ = 0.74. For the blinded assessment, independent annotators received anonymized source and replay trajectories from the full method and its ablations in random order. We concealed the method variant and the screening decision, and withheld both the model annotations and the construction analysis. The annotators matched mechanisms by their causal role and cited evidence from both trajectories. We calculated Jaccard overlap from their pairings using Appendix A. On the same 48 trajectory pairs, the blinded mean alignment for the full method was 78.82%, compared with 80.56% under the original procedure on those pairs. The paired difference was −1.74 percentage points, with a 95% confidence interval of −6.94 to 3.13. Independent reviewers still found substantial alignment with the source failures when they saw only the trajectories, although the interval allows for some inflation in the original score.
Q4 What are the fresh unevolved pass@3 scores on exactly the same 48 source tasks for every harness in Table 3? Please report absolute and paired improvements from round 0, not only post-evolution comparisons among training-task sources.
A4 Round 0 is a fresh Pass@3 evaluation on the same 48 source tasks used to assess the evolved harness. Figure 2 includes the initial SWE-bench results. With mini-SWE-agent, source Pass@3 rose from 3 of 48 (6.3%) to 15 of 48 (31.3%), a gain of 25.0 percentage points. It solved 13 previously unsolved tasks and lost one previously solved task. SWE-agent went from 4 of 48 (8.3%) to 13 of 48 (27.1%), a gain of 18.8 points, with 10 new solves and one regression. On Terminal-Bench both initial harnesses solved zero tasks. After evolution, mini-SWE-agent solved 8 of 48 (16.7%) and Terminus 2 solved 10 of 48 (20.8%). Every Terminal-Bench success was a new solve. Counting both new solves and regressions, harness evolution produced a net gain on the original source tasks in all four settings.
Q5 How stable are the AutoSaddler results across optimization seeds or repeated runs? Do paired bootstrap confidence intervals or randomization tests support the claimed advantage over the strongest baseline?
A5 We ran AutoSaddler three times from the original harness on the same training tasks and evaluated each resulting harness on the same 48 source tasks. Across the four benchmark and harness settings, the standard deviation of FAIL2TASK source Pass@3 across optimizations ranged from 2.1 to 3.2 percentage points. For each comparison we averaged the three outcomes for each task and resampled source tasks in pairs across methods. On SWE-bench, the mean difference from SWE-smith was 9.7 points for mini-SWE-agent (95% CI −2.8 to 22.2) and 11.1 points for SWE-agent (95% CI 0.7 to 21.5). On Terminal-Bench, the corresponding differences were 9.7 points against SETA for mini-SWE-agent (95% CI −0.7 to 20.8) and 6.9 points against Terminal-Lego for Terminus 2 (95% CI −4.9 to 18.8). The average advantage over the strongest Table 3 baseline was positive in all four settings. Only SWE-agent on SWE-bench had a paired interval entirely above zero.
| Method comparison | Mean source Pass@3 difference (pp) | Paired 95% CI (pp) |
|---|---|---|
| FAIL2TASK vs. SWE-smith (SWE-bench, mini-SWE-agent) | +9.7 | −2.8 to 22.2 |
| FAIL2TASK vs. SWE-smith (SWE-bench, SWE-agent) | +11.1 | 0.7 to 21.5 |
| FAIL2TASK vs. SETA (Terminal-Bench, mini-SWE-agent) | +9.7 | −0.7 to 20.8 |
| FAIL2TASK vs. Terminal-Lego (Terminal-Bench, Terminus 2) | +6.9 | −4.9 to 18.8 |
Post-rebuttal assessment
Thank you for your response. After considering the rebuttal, I will maintain my current rating.
Response to reviewer tw2q
Thank you for your detailed review of how we validate the tasks and evaluate harness improvements. We have grouped related comments into five questions and placed our response after each one.
Q1 Task validity is not independently established. The same construction process produces the instruction, private verifier, environment, and reference solution, and validity is demonstrated mainly by showing that the initial state fails while that generated reference solution passes. This does not rule out weak, incomplete, overfitted, or unintended verifiers. No independent expert audit, mutation analysis, alternative-solution study, or test-adequacy evaluation is reported. This is especially important because the paper's central artifact is an executable task, not only a trajectory with a matching failure label. What independent evidence establishes that the generated verifier fully and correctly represents the generated instruction? Could the authors provide a blinded human audit of a representative sample, verifier mutation scores, or results from independently produced alternative solutions?
A1 For tasks built in existing repositories, the reference patches draw on human implementations already validated in those repositories. We require the generated tests to pass on the corresponding reference patch, so the human implementation and the repository’s own tests provide an external standard for the behavior our verifier accepts. All 96 tasks follow the Harbor task format. We applied the verifier criteria in the Terminal-Bench 4.0 task implementation rubric to each task. We required the tests to cover every stated requirement, to check only behavior specified in the instruction, and to accept alternative implementations that satisfy those requirements. We did not conduct a separate human audit of the verifiers.
Q2 There is a substantial selection effect in the reconstruction results. A candidate can be refined for up to 15 attempts using replay-judge feedback until one stochastic run of the same agent fails in the desired way. Therefore, the 100% final acceptance rate is a pipeline search result rather than an out-of-sample success rate. The fresh-run analysis partially addresses this concern, but only 58.33% of tasks make the agent fail in all three runs. Results should be reported by benchmark and should include how alignment changes across independent fresh runs rather than centering the accepted or run-1 replay. For each benchmark separately, what are failure-pattern alignment and failure rate on each of the three fresh replays? How many tasks reproduce the aligned source mechanism in all three runs, rather than merely remaining unsolved?
A2 Table 1 reports that all 96 tasks passed the construction checks within 15 attempts per source. After each rejected attempt, feedback guides revision of the task. We froze the accepted tasks and excluded the screening run from the three fresh replays. On the 48 SWE-bench tasks, mean alignment with the source was 81.25%, 80.6% and 79.9% in runs one through three. The corresponding verifier failure rates were 85.4%, 83.3% and 85.4%. On the 48 Terminal-Bench tasks, alignment was 79.11%, 78.6% and 80.0%, with failure rates of 77.1%, 79.2% and 81.3%. We also counted cases in which all three replays exhibited the same mechanism documented in the source trajectory. This occurred on 38 of 48 SWE-bench tasks and 36 of 48 Terminal-Bench tasks. On both benchmarks, most tasks reproduced a source mechanism in all three fresh runs, showing that the mechanism recurred beyond the replay used to accept the task.