acceptodds
Under review as a conference paper at ICLR 2027

FAIL2TASK: Reproducing Coding-Agent Failure Patterns through Executable Task Synthesis

Abstract

Failed coding-agent trajectories provide evidence of weaknesses encountered in practice. However, when the original repositories and execution environments are unavailable, these failures cannot be directly replayed for evaluation or agent improvement. We study failure-to-task reconstruction: turning such trajecto- ries into new executable tasks rather than recovering the original tasks. We introduce FAIL2TASK, a closed-loop framework whose goal is to reproduce the source failure pattern when the same agent attempts the constructed task. Evidence-supported failure patterns guide repository selection and task construc- tion, producing tasks with runnable environments, tests, and reference solutions. Execution-based validation and semantic comparison of source and replay trajec- tories guide iterative refinement. We evaluate FAIL2TASK on 96 source failures from SWE-bench Verified and Terminal-Bench 2.0, producing tasks accepted by the pipeline for all 96 cases. Rerunning the agents on these tasks yields trajec- tories with 80.18% failure-pattern alignment with the original failure trajectories. Harness self-evolution on the constructed tasks achieves source-task pass rates of 27.1–31.3% on SWE-bench, compared with 14.6–20.8% for the strongest base- line. On Terminal-Bench, the corresponding rates are 16.7–20.8%, compared with 4.2–12.5% for the baselines. These results show that tasks constructed from fail- ure trajectories can help agents solve previously failed tasks.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Data construction

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_tw2qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FAIL2TASK, a framework for converting failed coding-agent trajectories into new executable tasks when the original repository and execution environment are unavailable. The framework first extracts evidence-supported failure mechanisms from a source trajectory, selects or creates an environment intended to instantiate the relevant conditions, and generates a task with an instruction, verifier, and reference solution. It then validates the task through a fail-before/fix-after check and repeatedly runs the source agent configuration on the task; an LLM judge accepts the task only when the replay is unsuccessful and appears to exhibit a semantically corresponding failure pattern.

The evaluation uses 96 failures produced by GPT-5.4-mini, evenly divided between SWE-bench Verified and Terminal-Bench 2.0. Within a budget of 15 construction attempts per source, the pipeline obtains an accepted task for every case. The authors report 80.18% author-reviewed failure-pattern alignment, 58.33% three-run failure persistence, and 85.66% cross-run failure-pattern overlap. The paper further uses reconstructed tasks to optimize several agent harnesses with AutoSaddler and reports higher pass@3 on the corresponding source tasks than when optimization uses failure-matched tasks from SWE-smith, SWE-ReBench, Terminal-Lego, or SETA.

Critical strengths

The paper identifies an important and practically motivated setting: a failure trajectory may be shareable even when the original code and environment are unavailable. Reconstructing an executable surrogate rather than claiming to recover the original task is a useful and appropriately scoped formulation.FAIL2TASK is a coherent closed-loop system. It combines causal failure analysis, task construction, execution checks, replay with the same agent configuration, and semantic screening rather than relying only on textual similarity between task descriptions.The paper evaluates both repository-level software repair and broader terminal tasks. The ablations show large degradation when failure analysis or replay screening is removed, which supports the importance of the main components.The appendices provide substantial protocol detail, including the failure-pairing prompt, overlap calculation, baseline-matching procedure, data-membership rules, and mismatch analysis.The downstream harness-evolution experiment is valuable because it tests whether the reconstructed tasks are useful rather than merely similar according to a judge. The qualitative mismatch analysis is candid and helps clarify why transfer remains incomplete.

Critical weaknesses

1.Task validity is not independently established. The same construction process produces the instruction, private verifier, environment, and reference solution, and validity is demonstrated mainly by showing that the initial state fails while that generated reference solution passes. This does not rule out weak, incomplete, overfitted, or unintended verifiers. No independent expert audit, mutation analysis, alternative-solution study, or test-adequacy evaluation is reported. This is especially important because the paper's central artifact is an executable task, not only a trajectory with a matching failure label. 2.There is a substantial selection effect in the reconstruction results. A candidate can be refined for up to 15 attempts using replay-judge feedback until one stochastic run of the same agent fails in the desired way. Therefore, the 100% final acceptance rate is a pipeline search result rather than an out-of-sample success rate. The fresh-run analysis partially addresses this concern, but only 58.33% of tasks make the agent fail in all three runs. Results should be reported by benchmark and should include how alignment changes across independent fresh runs rather than centering the accepted or run-1 replay. 3.The primary 80.18% alignment metric is not sufficiently independent of the construction loop. The final annotation is model-assisted, the authors inspect the model's annotation and the accepted-round analysis.json, and review is not blinded. Although the authors explicitly disclose this and report inter-annotator agreement, the same conceptual decomposition used during construction can anchor the final assessment. Evaluation by independent annotators who are blind to acceptance status, method variant, and generated analysis would provide stronger evidence.

Questions for authors

1.What independent evidence establishes that the generated verifier fully and correctly represents the generated instruction? Could the authors provide a blinded human audit of a representative sample, verifier mutation scores, or results from independently produced alternative solutions? 2. For each benchmark separately, what are failure-pattern alignment and failure rate on each of the three fresh replays? How many tasks reproduce the aligned source mechanism in all three runs, rather than merely remaining unsolved? 3. Can the authors repeat the final alignment assessment with independent annotators who are blind to the method/ablation, acceptance decision, and accepted-round analysis.json? How does that score compare with 80.18%? 4. What are the fresh unevolved pass@3 scores on exactly the same 48 source tasks for every harness in Table 3? Please report absolute and paired improvements from round 0, not only post-evolution comparisons among training-task sources. 5. How stable are the AutoSaddler results across optimization seeds or repeated runs? Do paired bootstrap confidence intervals or randomization tests support the claimed advantage over the strongest baseline?

3 comments
AuthorsOpenReview ↗

Response to reviewer tw2q

Thank you for your detailed review of how we validate the tasks and evaluate harness improvements. We have grouped related comments into five questions and placed our response after each one.

Q1 Task validity is not independently established. The same construction process produces the instruction, private verifier, environment, and reference solution, and validity is demonstrated mainly by showing that the initial state fails while that generated reference solution passes. This does not rule out weak, incomplete, overfitted, or unintended verifiers. No independent expert audit, mutation analysis, alternative-solution study, or test-adequacy evaluation is reported. This is especially important because the paper's central artifact is an executable task, not only a trajectory with a matching failure label. What independent evidence establishes that the generated verifier fully and correctly represents the generated instruction? Could the authors provide a blinded human audit of a representative sample, verifier mutation scores, or results from independently produced alternative solutions?

A1 For tasks built in existing repositories, the reference patches draw on human implementations already validated in those repositories. We require the generated tests to pass on the corresponding reference patch, so the human implementation and the repository’s own tests provide an external standard for the behavior our verifier accepts. All 96 tasks follow the Harbor task format. We applied the verifier criteria in the Terminal-Bench 4.0 task implementation rubric to each task. We required the tests to cover every stated requirement, to check only behavior specified in the instruction, and to accept alternative implementations that satisfy those requirements. We did not conduct a separate human audit of the verifiers.


Q2 There is a substantial selection effect in the reconstruction results. A candidate can be refined for up to 15 attempts using replay-judge feedback until one stochastic run of the same agent fails in the desired way. Therefore, the 100% final acceptance rate is a pipeline search result rather than an out-of-sample success rate. The fresh-run analysis partially addresses this concern, but only 58.33% of tasks make the agent fail in all three runs. Results should be reported by benchmark and should include how alignment changes across independent fresh runs rather than centering the accepted or run-1 replay. For each benchmark separately, what are failure-pattern alignment and failure rate on each of the three fresh replays? How many tasks reproduce the aligned source mechanism in all three runs, rather than merely remaining unsolved?

A2 Table 1 reports that all 96 tasks passed the construction checks within 15 attempts per source. After each rejected attempt, feedback guides revision of the task. We froze the accepted tasks and excluded the screening run from the three fresh replays. On the 48 SWE-bench tasks, mean alignment with the source was 81.25%, 80.6% and 79.9% in runs one through three. The corresponding verifier failure rates were 85.4%, 83.3% and 85.4%. On the 48 Terminal-Bench tasks, alignment was 79.11%, 78.6% and 80.0%, with failure rates of 77.1%, 79.2% and 81.3%. We also counted cases in which all three replays exhibited the same mechanism documented in the source trajectory. This occurred on 38 of 48 SWE-bench tasks and 36 of 48 Terminal-Bench tasks. On both benchmarks, most tasks reproduced a source mechanism in all three fresh runs, showing that the mechanism recurred beyond the replay used to accept the task.

AuthorsOpenReview ↗

Response to reviewer tw2q

Q3 The primary 80.18% alignment metric is not sufficiently independent of the construction loop. The final annotation is model-assisted, the authors inspect the model's annotation and the accepted-round analysis.json, and review is not blinded. Although the authors explicitly disclose this and report inter-annotator agreement, the same conceptual decomposition used during construction can anchor the final assessment. Evaluation by independent annotators who are blind to acceptance status, method variant, and generated analysis would provide stronger evidence. Can the authors repeat the final alignment assessment with independent annotators who are blind to the method/ablation, acceptance decision, and accepted-round analysis.json? How does that score compare with 80.18%?

A3 Table 2 reports 80.18% alignment under the original annotation procedure. Two authors reviewed model annotations with access to the construction analysis, and their agreement on those review decisions was κ = 0.74. For the blinded assessment, independent annotators received anonymized source and replay trajectories from the full method and its ablations in random order. We concealed the method variant and the screening decision, and withheld both the model annotations and the construction analysis. The annotators matched mechanisms by their causal role and cited evidence from both trajectories. We calculated Jaccard overlap from their pairings using Appendix A. On the same 48 trajectory pairs, the blinded mean alignment for the full method was 78.82%, compared with 80.56% under the original procedure on those pairs. The paired difference was −1.74 percentage points, with a 95% confidence interval of −6.94 to 3.13. Independent reviewers still found substantial alignment with the source failures when they saw only the trajectories, although the interval allows for some inflation in the original score.


Q4 What are the fresh unevolved pass@3 scores on exactly the same 48 source tasks for every harness in Table 3? Please report absolute and paired improvements from round 0, not only post-evolution comparisons among training-task sources.

A4 Round 0 is a fresh Pass@3 evaluation on the same 48 source tasks used to assess the evolved harness. Figure 2 includes the initial SWE-bench results. With mini-SWE-agent, source Pass@3 rose from 3 of 48 (6.3%) to 15 of 48 (31.3%), a gain of 25.0 percentage points. It solved 13 previously unsolved tasks and lost one previously solved task. SWE-agent went from 4 of 48 (8.3%) to 13 of 48 (27.1%), a gain of 18.8 points, with 10 new solves and one regression. On Terminal-Bench both initial harnesses solved zero tasks. After evolution, mini-SWE-agent solved 8 of 48 (16.7%) and Terminus 2 solved 10 of 48 (20.8%). Every Terminal-Bench success was a new solve. Counting both new solves and regressions, harness evolution produced a net gain on the original source tasks in all four settings.


Q5 How stable are the AutoSaddler results across optimization seeds or repeated runs? Do paired bootstrap confidence intervals or randomization tests support the claimed advantage over the strongest baseline?

A5 We ran AutoSaddler three times from the original harness on the same training tasks and evaluated each resulting harness on the same 48 source tasks. Across the four benchmark and harness settings, the standard deviation of FAIL2TASK source Pass@3 across optimizations ranged from 2.1 to 3.2 percentage points. For each comparison we averaged the three outcomes for each task and resampled source tasks in pairs across methods. On SWE-bench, the mean difference from SWE-smith was 9.7 points for mini-SWE-agent (95% CI −2.8 to 22.2) and 11.1 points for SWE-agent (95% CI 0.7 to 21.5). On Terminal-Bench, the corresponding differences were 9.7 points against SETA for mini-SWE-agent (95% CI −0.7 to 20.8) and 6.9 points against Terminal-Lego for Terminus 2 (95% CI −4.9 to 18.8). The average advantage over the strongest Table 3 baseline was positive in all four settings. Only SWE-agent on SWE-bench had a paired interval entirely above zero.

Method comparisonMean source Pass@3 difference (pp)Paired 95% CI (pp)
FAIL2TASK vs. SWE-smith (SWE-bench, mini-SWE-agent)+9.7−2.8 to 22.2
FAIL2TASK vs. SWE-smith (SWE-bench, SWE-agent)+11.10.7 to 21.5
FAIL2TASK vs. SETA (Terminal-Bench, mini-SWE-agent)+9.7−0.7 to 20.8
FAIL2TASK vs. Terminal-Lego (Terminal-Bench, Terminus 2)+6.9−4.9 to 18.8
Reviewer_tw2qOpenReview ↗

Post-rebuttal assessment

Thank you for your response. After considering the rebuttal, I will maintain my current rating.

Reviewer_aSxEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies failure-to-task reconstruction: generating a new executable task from a failed coding-agent trajectory when the original repository and environment are unavailable. FAIL2TASK extracts evidence-supported failure mechanisms, selects a different public repository (or builds a terminal environment), constructs an instruction, tests, and reference solution, and iteratively validates both solvability and failure-pattern reproduction using the original agent configuration. Across 48 SWE-bench Verified and 48 Terminal-Bench 2.0 source failures, all instances produce pipeline-accepted tasks within 15 attempts. Harness optimization on these tasks achieves higher pass@3 on the corresponding source failures than optimization on matched existing tasks.

Critical strengths

A novel and practically relevant approach to constructing new training tasks. Companies developing coding agents may receive failure traces without access to users’ private repositories or execution environments. FAIL2TASK turns these traces into new executable tasks that preserve the relevant failure patterns, offering a promising way to construct targeted training data from real-world failures when the original tasks cannot be shared or replayed.

Critical weaknesses
  1. The comparison does not account for task-preparation costs.

FAIL2TASK consumes 1,023M and 644M tokens to construct 96 accepted tasks, averaging approximately 17.4M tokens per task. While the downstream harness optimization uses the same setup, the comparison does not account for the substantial upstream cost of task preparation. Existing-task baselines could also benefit from additional computation for task matching, execution-based filtering, and quality assessment. A comparison under matched end-to-end budgets would help establish whether reconstructing new tasks is more cost-effective than improving the selection and filtering of existing ones.

  1. Limited evidence of the generated tasks’ value for model training.

The paper evaluates the generated tasks through harness optimization while keeping model weights fixed. While this demonstrates their usefulness for harness refinement, it does not directly establish their effectiveness as training data. Evaluating SFT on verified solution trajectories or RL with executable verifiers, compared against existing-task data under matched budgets and evaluated on held-out tasks, would provide stronger evidence of the generated tasks’ value for model training.

Questions for authors
  1. Cost clarification: What are the monetary cost and wall-clock time per accepted task, including failed construction attempts? How does the end-to-end cost per additional source task solved compare with the baselines?
  2. Model-training value: Do you have SFT or RL results using the constructed tasks, compared with existing-task data under matched training budgets and evaluated on tasks disjoint from the source failures?
5 comments
AuthorsOpenReview ↗

Response to reviewer aSxE

Thank you for your questions about task preparation costs and the value of the tasks for model training. We address these points in order, with our response directly below each question.

Q1 The comparison does not account for task-preparation costs. FAIL2TASK consumes 1,023M and 644M tokens to construct 96 accepted tasks, averaging approximately 17.4M tokens per task. While the downstream harness optimization uses the same setup, the comparison does not account for the substantial upstream cost of task preparation. Existing-task baselines could also benefit from additional computation for task matching, execution-based filtering, and quality assessment. A comparison under matched end-to-end budgets would help establish whether reconstructing new tasks is more cost-effective than improving the selection and filtering of existing ones. Cost clarification: What are the monetary cost and wall-clock time per accepted task, including failed construction attempts? How does the end-to-end cost per additional source task solved compare with the baselines?

A1 Thank you for asking about task preparation costs. Table 1 includes every FAIL2TASK construction attempt, including those that failed validation or replay screening. For FAIL2TASK, we divided the total construction cost by the 48 accepted tasks in each benchmark. For each baseline, we averaged the task-selection cost across the full set of recorded selections for that dataset. The table reports these per-task averages in millions of tokens. On Terminal-Bench, reconstruction used fewer tokens per task on average than selection from either baseline dataset. On SWE-bench, its cost fell between SWE-smith and SWE-ReBench. These figures cover preparation only, so the cost per additional source task solved would also require the token usage for harness optimization and evaluation.

MethodSWE-bench Verified (million tokens per task)Terminal-Bench 2.0 (million tokens per task)
FAIL2TASK21.3113.42
SWE-smith · Terminal-Lego10.5816.26
SWE-ReBench · SETA23.7139.25

Q2 Limited evidence of the generated tasks’ value for model training. The paper evaluates the generated tasks through harness optimization while keeping model weights fixed. While this demonstrates their usefulness for harness refinement, it does not directly establish their effectiveness as training data. Evaluating SFT on verified solution trajectories or RL with executable verifiers, compared against existing-task data under matched budgets and evaluated on held-out tasks, would provide stronger evidence of the generated tasks’ value for model training. Model-training value: Do you have SFT or RL results using the constructed tasks, compared with existing-task data under matched training budgets and evaluated on tasks disjoint from the source failures?

A2 We appreciate the reviewer’s suggestion to explore SFT and RL. We designed FAIL2TASK primarily for harness self-evolution, and Table 3 shows that evolving on our tasks yielded the highest source Pass@3 in all four settings. For example, with model weights fixed, mini-SWE-agent solved 15 of the 48 original SWE-bench tasks after evolution on FAIL2TASK, up from 3 before evolution and compared with 10 after evolution on SWE-smith. We also see promise in using these tasks to train the model itself. Verified solution trajectories could provide demonstrations for SFT, while the executable verifiers could reward successful attempts during RL. We have yet to evaluate these uses and would like to study both under matched training budgets, with evaluation on tasks disjoint from the source failures.

Thanks for the clarification. The cost breakdown is helpful, but it is still unclear whether reconstructing tasks is more cost-effective than selecting existing ones under the same total budget. I will keep my original score of 2.

AuthorsOpenReview ↗

Response to reviewer aSxE

Thank you for following up. We appreciate the distinction you draw between task preparation cost and overall cost-effectiveness. Our experiments were designed to assess whether reconstructed tasks help a harness solve more of its original failures. We do not claim that FAIL2TASK is more cost-effective under a fixed end-to-end budget. On Terminal-Bench, reconstruction used fewer preparation tokens than both baselines, even after counting all failed construction attempts. FAIL2TASK averaged 13.42M tokens per task, about 17% fewer than Terminal-Lego (16.26M) and 66% fewer than SETA (39.25M). With Terminus 2, FAIL2TASK also led to 10 of the 48 source tasks being solved, compared with 6 for Terminal-Lego and 4 for SETA. On SWE-bench, the additional preparation cost came with a performance gain. FAIL2TASK used 21.31M tokens per task compared with 10.58M for SWE-smith, and mini-SWE-agent solved 15 source tasks after evolution compared with 10. We will make the scope of our claim clearer in the revision and acknowledge that overall cost-effectiveness remains an open question.

Reviewer_SMjhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This article explores how to construct new executable tasks from the failed execution trajectory of the Coding Agent, in order to reproduce the original failure mode and support agent capability improvement. The author proposes the FAIL2CASK framework, which first extracts failure patterns with evidence support from the failure trajectory when the original repository and execution environment cannot be accessed. Then, suitable open-source repositories or new environments are selected to generate executable tasks that include task descriptions, testing, and reference solutions. Subsequently, the framework performs validation and Agent Replay to check if the task is solvable and if the original Agent exhibits similar failure mechanisms again, and uses feedback iteration optimization to generate results.

Critical strengths
  • Unlike traditional Agent Failure Diagnosis or generating software engineering tasks directly from existing repositories, this paper investigates how to extract transferable failure mechanisms from failure trajectories without accessing the original repository, and reproduce these behaviors through new tasks. This provides a meaningful solution for conducting agent evaluation and improvement using the historical failure trajectories provided by users.
  • FAIL2TASK combines Failure Analysis, Execution Validation, and Replay Screening to build a closed-loop optimization process, and verifies the role of key components through multiple ablation experiments. For example, the Failure Pattern Alignment of the complete method reached 80.18%, while after removing Failure Analysis and Replay Screening, it decreased to 25.23% and 9.91%, respectively, indicating that these two components make important contributions to replicating the failure mechanism.
  • The author applied the generated task to Harness Self Evolution and compared it under two types of Coding Agent Benchmarks and multiple Harness configurations. In addition, the article also analyzed the phenomenon of inconsistent results between the generated task and the original task, discussed factors such as task scope reduction, API differences, and insufficient validation, giving the experimental analysis a certain depth.
Critical weaknesses
  • Although FAIL2TASK outperforms existing task dataset baselines in Harness Self Evolution, there may be significant differences in difficulty among tasks generated or selected by different methods. For example, in the Terminal Punch setting, FAIL2TASK's Training Pass@3 Only 39.6%, while Terminal Lego and SETA reach 62.5% -87.5%. Therefore, current experiments cannot fully demonstrate that performance improvement mainly comes from accurate reproduction of failure modes, rather than more challenging training tasks or different training feedback distributions.
  • Although all 96 tasks ultimately passed Replay Screening, only 58.33% of the generated tasks were able to consistently fail in three independent executions. This indicates that the failure patterns observed during task acceptance may not be stably reproducible in subsequent execution and still require more independent evaluation and multiple run analyses.
  • The experiment mainly uses the failure trajectory generated by GPT-5.4-mini, and the testing task directly corresponds to the original failure cases involved in reconstruction. Therefore, it is unclear whether this method can be stably extended to other basic models, unseen failure modes, or a wider range of Coding Agent scenarios.
Questions for authors
  • Can we supplement the Task Synthesis Baseline with difficulty matching and compare it under the same number of tasks, generated budget, and Harness Optimization settings? I particularly hope to see downstream Source Task after removing Failure Analysis or Replay Screening Pass@3 Is there still a significant improvement.
  • Suggest adding the independence of non evolved Harness on the original failed task Pass@3 Baseline and perform paired statistical tests on the results of different methods. Meanwhile, can we test the Harness Optimization under different random seeds, or use the failure trajectories generated by different base models, to verify whether this improvement has stability and transferability?
4 comments
AuthorsOpenReview ↗

Response to reviewer SMjh

Thank you for your careful review of our evaluation. We have grouped your comments into three questions and respond to each in turn.

Q1 Although FAIL2TASK outperforms existing task dataset baselines in Harness Self Evolution, there may be significant differences in difficulty among tasks generated or selected by different methods. For example, in the Terminal Punch setting, FAIL2TASK's Training Pass@3 Only 39.6%, while Terminal Lego and SETA reach 62.5% -87.5%. Therefore, current experiments cannot fully demonstrate that performance improvement mainly comes from accurate reproduction of failure modes, rather than more challenging training tasks or different training feedback distributions. Can we supplement the Task Synthesis Baseline with difficulty matching and compare it under the same number of tasks, generated budget, and Harness Optimization settings? I particularly hope to see downstream Source Task after removing Failure Analysis or Replay Screening Pass@3 Is there still a significant improvement.

A1 With SWE-agent, the initial harness solved none of the 48 SWE-smith training tasks and four FAIL2TASK tasks. After evolution, source Pass@3 was 14.6% and 27.1%, respectively. On Terminal-Bench, the original selection used root cause similarity, and both baseline sets met the stopping criterion at Round 0. We selected 48 tasks each from SETA and Terminal-Lego that the initial harness failed on all three attempts. Under the same preparation budget cap and AutoSaddler settings, source Pass@3 was 8.3% for SETA and 10.4% for Terminal-Lego with mini-SWE-agent, and 10.4% and 14.6% with Terminus 2. Table 3 reports 16.7% and 20.8% for FAIL2TASK with those harnesses. In a separate Terminus 2 comparison, we matched initial training difficulty and the generation budget across synthesis methods. Source Pass@3 was 20.8% for full FAIL2TASK and 12.5% for synthesis guided by difficulty alone. Removing Failure Analysis yielded 14.6% and removing Replay Screening yielded 10.4%. With initial difficulty and budget held fixed, the full method achieved the highest source Pass@3, suggesting that reproducing the original failure mechanism helps the harness solve the source task.

Methodmini-SWE-agent source Pass@3Terminus 2 source Pass@3
SETA (reselected)8.3%10.4%
Terminal-Lego (reselected)10.4%14.6%
FAIL2TASK16.7%20.8%

Q2 Although all 96 tasks ultimately passed Replay Screening, only 58.33% of the generated tasks were able to consistently fail in three independent executions. This indicates that the failure patterns observed during task acceptance may not be stably reproducible in subsequent execution and still require more independent evaluation and multiple run analyses.

A2 We froze all 96 tasks after acceptance and ran the original agent three more times on each task. Table 2 reports that 56 tasks failed the final verifier on all three attempts (58.33%). We also examined the mistakes recorded in each trajectory. Across the three pairs of runs per task, the mean Jaccard overlap of the observed failure mechanisms was 85.66%. As defined in Appendix A, a mistake still counts toward mechanism overlap if the agent later recovers. The fresh runs therefore show that agents often repeat the same mistakes after screening, even when their final outcomes differ.

AuthorsOpenReview ↗

Response to reviewer SMjh

Q3 The experiment mainly uses the failure trajectory generated by GPT-5.4-mini, and the testing task directly corresponds to the original failure cases involved in reconstruction. Therefore, it is unclear whether this method can be stably extended to other basic models, unseen failure modes, or a wider range of Coding Agent scenarios. Suggest adding the independence of non evolved Harness on the original failed task Pass@3 Baseline and perform paired statistical tests on the results of different methods. Meanwhile, can we test the Harness Optimization under different random seeds, or use the failure trajectories generated by different base models, to verify whether this improvement has stability and transferability?

A3 Round 0 uses three fresh attempts on the same 48 source tasks per benchmark. On SWE-bench, mini-SWE-agent went from 3 to 15 solved tasks and SWE-agent from 4 to 13. On Terminal-Bench, both harnesses started at zero. After evolution, mini-SWE-agent solved 8 tasks and Terminus 2 solved 10. We repeated the full optimization three times from the same initial harness. For Terminus 2, the mean difference in source Pass@3 between FAIL2TASK and Terminal-Lego was 6.9 percentage points, with a paired 95% bootstrap interval of −4.9 to 18.8. We also reconstructed 24 fresh source failures from Kimi K3 and used Kimi K3 for replay. Mean source alignment was 77.8%. On those cases, its source Pass@3 was 8.3% at Round 0 and 25.0% after harness evolution. The gain on Kimi K3 extends the evidence to a second base model on its own source failures. The repeated Terminus 2 comparison is less conclusive because its interval includes zero, and transfer to unseen failure mechanisms remains open.

Thanks for the author's response. I have decided to maintain my scores.

Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to ask whether our response addressed your concerns. If any questions remain, we’d be happy to discuss them further.

Reviewer_N9CKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FAIL2TASK, which turns failed coding-agent trajectories into new executable tasks without access to the original repositories or environments. It combines failure analysis, task construction, execution checks, and agent replay to reproduce the source failure mechanism. Experiments on 96 failures from SWE-bench Verified and Terminal-Bench 2.0 show promising mechanism alignment and improved source-task performance after harness optimization.

Critical strengths
  1. The paper addresses a useful practical problem: making failures actionable when the original environment cannot be shared. Separating task solvability from failure-mechanism reproduction is a sensible design.
  2. The evaluation includes fresh replays and downstream testing on source tasks. The consistent gains across four configurations provide encouraging evidence of usefulness.
Critical weaknesses
  1. The replay metrics leave an important ambiguity. The paper reports 58.33% three-run failure persistence alongside 85.66% cross-run mechanism overlap. Their relationship depends on whether successful runs retain intermediate errors in their mechanism sets and whether the metrics use the same cohort. This distinction matters because repeated intermediate mistakes do not necessarily establish repeated failure for the source reason.
  2. The added value of reconstructing executable tasks is not sufficiently isolated. The available source trajectories already contain diagnostic information that could directly guide harness changes. Comparing against matched task datasets does not establish how much reconstruction adds over using those trajectories directly, especially given its substantial construction cost.
Questions for authors
  1. How are successful runs handled in the mechanism-overlap calculation, and what are the denominators for the two replay metrics? If successful runs contribute empty sets and both metrics use the same cohort, 58.33% failure persistence implies an overlap ceiling of 72.22%. Please clarify which assumption does not apply to the reported 85.66%.
  2. How does FAIL2TASK compare with directly using the same source failure trajectories to propose harness improvements, without reconstructing tasks? This comparison would help establish the additional benefit of executable reconstruction.
6 comments
AuthorsOpenReview ↗

Response to reviewer N9CK

Thank you for your close reading of the replay metrics and the trajectory comparison. We present your two questions in their original order and respond to each below.

Q1 The replay metrics leave an important ambiguity. The paper reports 58.33% three-run failure persistence alongside 85.66% cross-run mechanism overlap. Their relationship depends on whether successful runs retain intermediate errors in their mechanism sets and whether the metrics use the same cohort. This distinction matters because repeated intermediate mistakes do not necessarily establish repeated failure for the source reason. How are successful runs handled in the mechanism-overlap calculation, and what are the denominators for the two replay metrics? If successful runs contribute empty sets and both metrics use the same cohort, 58.33% failure persistence implies an overlap ceiling of 72.22%. Please clarify which assumption does not apply to the reported 85.66%.

A1 Both metrics cover the same 96 tasks. Failure persistence counts tasks whose final output fails the verifier on all three fresh runs. This occurred on 56 of 96 tasks (58.33%). For mechanism overlap, we annotate errors evidenced anywhere in the trajectory, including a mistake the agent later corrects. A successful run can therefore have a nonempty mechanism set. A completed run with no evidenced mistake contributes an empty set, and a comparison of two empty sets receives zero. We average the three pairwise Jaccard scores within each task and then average across tasks, giving equal weight to 288 comparisons. The proposed 72.22% ceiling assumes that every successful run contributes an empty set. The 85.66% overlap and 58.33% persistence are compatible on the same 96 tasks because an agent can repeat a mistake across runs and still recover before the final check.


Q2 The added value of reconstructing executable tasks is not sufficiently isolated. The available source trajectories already contain diagnostic information that could directly guide harness changes. Comparing against matched task datasets does not establish how much reconstruction adds over using those trajectories directly, especially given its substantial construction cost. How does FAIL2TASK compare with directly using the same source failure trajectories to propose harness improvements, without reconstructing tasks? This comparison would help establish the additional benefit of executable reconstruction.

A2 We gave AutoSaddler the original failure trajectories and evaluated the resulting harness on the same 48 source tasks per benchmark. On SWE-bench, mini-SWE-agent solved three previously unsolved tasks but regressed on three it had solved at Round 0. Its source Pass@3 therefore remained at 6.3%, compared with 31.3% after evolution on FAIL2TASK tasks. On Terminal-Bench, Terminus 2 gained one task from a baseline of zero, reaching 2.1% compared with 20.8% after reconstruction. We attribute much of this gap to the feedback available during optimization. The recorded trajectories show the agent’s actions but omit the verifier diagnostics that explain which requirement its output violated. Determining why an edit failed often requires inspecting the surrounding code and how it behaves in the repository’s execution environment, context that a recorded trajectory captures only in part. Without the original environment, AutoSaddler cannot rerun the task to check which action caused the failure. Reconstructing an executable task lets it test a harness change and observe whether the same mistake recurs.

Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to ask whether our response addressed your concerns. If any questions remain, we’d be happy to discuss them further.

Thank you for the clarifications and additional experiments, which help address my concerns. Could you further clarify (1) how many tasks fail due to the source mechanism in all three fresh replays, and (2) how harness updates are selected in the trajectory-only baseline and whether its optimization budget is comparable to FAIL2TASK? I appreciate the additional evidence and will maintain my score of 2 for now.

Comments on the paper

1 comment
Area_Chair_PvZyOpenReview ↗

Reminder from AC

Dear Reviewers,

As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.