acceptodds
Under review as a conference paper at ICLR 2027

Fails Before Repair Is Not Enough: Auditing the Validity of Executable Repair Benchmarks.

Abstract

Executable repair benchmarks are trusted because they run. A task counts as valid when the buggy artifact fails its tests and the reference fix passes them, and this two-sided check is close to the only validity evidence most benchmarks report. We show it misses three distinct failures, using a suite of 81 Verilog repair tasks and three instruments that apply to any executable repair benchmark. First, answer leakage: in 89.5% of our tasks the fix was written into the task inputs, and no task withheld it. Removing it costs a 1.5B-parameter model 16.9 points of pass@1 (95% CI [9.5, 24.7]) and four API models between -0.6 and 3.3 points, and with the answer withheld the share of tasks that separate two frontier models rises from 14.8% to 32.1% (paired +17.3 points, [7.4, 28.4]). Second, oracle weakness: our testbenches detect only 61.6% of injected single-site mutants, and four detect none. Applying the same mutation instrument to two benchmarks we did not build locates the cause in oracle design. RTLLM, whose testbenches are directed like ours, scores 52.9%; VerilogEval, whose testbenches compare the candidate against a reference cycle by cycle, scores 91.8%. Third, resolution: even with the answer withheld, 36 of 81 tasks are solved by all four API models on every sample. We also report that mining frontier-model failures into new tasks produced no novel tasks, with or without the answer available, and why. We release the suite, the harness and all three instruments.

Primary area:
datasets and benchmarks
Keywords:
benchmark validity, evaluation of LLM agents, automated program repair, RTL / Verilog, benchmark leakage, mutation testing, test oracles, discriminative power

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_neg3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines the validity of executable repair benchmarks, arguing that the two-sided check—failure of the buggy implementation and success of the reference repair—is insufficient to establish reliable evaluation. Using Fail2Bench, a suite of 81 Verilog repair tasks, the authors audit answer leakage in task inputs, testbench fault-detection capability, and the ability of tasks to distinguish model performance. They also apply mutation testing to RTLLM and VerilogEval. Additional findings concern defects in task construction and the evaluation harness, as well as unsuccessful attempts to derive novel tasks from stronger models’ failed repairs.

Critical strengths
  1. The problem is practically relevant and supported by concrete examples. The paper demonstrates why repair benchmarks require more than executable tests and successful validation of buggy and reference implementations. Examples involving explicit solutions in task inputs, testbenches that miss additional faults, and a patch-application defect that substantially changes measured performance provide actionable lessons for benchmark developers. The finding that the evaluation harness itself can materially alter results is particularly valuable.
  2. External audits and transparent reporting strengthen the contribution. Applying the mutation audit to RTLLM and VerilogEval extends the evidence beyond the authors’ own suite. The paper also explicitly reports differences in experimental budgets, instability in per-task outcomes across repeated runs, and findings that were withdrawn after further investigation. This transparency helps readers assess the scope and limitations of the evidence.
Critical weaknesses
  1. The cross-benchmark comparison does not sufficiently establish the proposed causal explanation for detection-rate differences. The abstract and Figure 1 attribute the difference between directed and differential testing primarily to oracle design. However, the three benchmarks differ in their designs, input stimuli, observation schedules, and mutant populations. Applying the same mutation operators does not control these differences. The results establish differences in mutation detection across the three testing setups, but do not isolate oracle type as their main cause.

    The treatment of equivalent mutants also needs clarification. The paper excludes non-compiling mutants but does not report equivalence checks on surviving mutants. VerilogEval’s ability to detect clock-edge mutations in its own designs does not establish that mutations of the same class in the other benchmarks are all non-equivalent. A sampled inspection of surviving mutants and an associated sensitivity analysis would help. Without a controlled comparison, the attribution in the abstract and Figure 1 should be narrowed. These concerns do not invalidate the descriptive detection rates, but they limit their interpretation.

  2. The connection between the reported increase in discriminating tasks and stable model discrimination remains insufficiently established. Section 7 labels a task discriminating whenever at least two models have different empirical pass rates. With only two or three samples per task, models with identical underlying success probabilities can nevertheless exhibit different empirical pass rates. Consequently, this metric reflects sampling variability and the level of task success probabilities, in addition to genuine differences between models.

    Matching the model set and sample counts in some comparisons is helpful, and the authors appropriately acknowledge the metric’s dependence on sampling budget. However, this does not establish how much of the increase from 14.8% to 32.1% reflects reproducible model differences rather than increased disagreement under finite sampling. The disagreement on 12 tasks between the two Sonnet runs makes this an important question.

    Table 2 is valid as a descriptive analysis of the observed runs. Interpreting the selected tasks as reliably carrying signal, however, requires evidence of stability across repeated runs. Separately, overlapping marginal confidence intervals do not by themselves establish that models cannot be ranked. Comparisons on the same tasks should report confidence intervals for paired performance differences.

Questions for authors
  1. Interpretation of the mutation audit: Do you have comparisons of directed and differential testing on the same designs and mutant populations? If not, would you narrow the causal language to an association and provide a sampled inspection of surviving mutants?
  2. Discrimination: What baseline level does the current discrimination metric attain when models have equal underlying success probabilities but outcomes vary because of finite sampling? Could you use independent repeated runs to assess the stability of the discriminating task set and whether the observed improvement exceeds this baseline?
  3. Experimental conditions and model comparisons: Could you report confidence intervals for paired pass@1 differences between the principal models, and clarify the sensitivity of hint effects to time limits, timeout scoring, and the recovery of Opus results? The additional Sonnet run without abandonments already addresses part of this concern for that model, but the implications for the other comparisons remain unclear.
  4. Reporting consistency: Table 1 lists three samples for Sonnet, whereas Appendix E lists five for its with-hints arm. Opus also has different sample counts across conditions, which the single sample-count column in Table 1 does not represent. In addition, Table 5 states that it covers 81 tasks, but the counts in both taxonomy columns sum to 52. Please clarify the samples and task populations underlying these results and reconcile the reporting.
Reviewer_R2HeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the hardware description language (Verilog) benchmarks validity and argues that the two-sided invariant (fail-to-pass) check missed three problems, i. e., (i) Leakage, (ii) Oracle Weakness, and (iii) Low resolution. They propose instrumentation for all three problems and apply it to their own 81-task Verilog repair suite. To validate the oracle, they also extend the mutation instrumentation to over RTLLM and VerlogEval benchmarks. They report defects invisible to the invariant: 77/86 directories leaked the fix; their oracles kill only 61.6% of viable mutants; 36/81 tasks are saturated.

Critical strengths

(1) The paper provides a cheap, reproducible instrumentation which doesn’t require any model calls for audits.

(2) The mutation instrumentation is validated across benchmarks measuring the oracle strength.

(3) The failure-mining experiment is interesting even though the authors could not produce new tasks from strong models.

Critical weaknesses

(1) Only the mutation instrumentation is applied across benchmarks. Although the paper title claims to be “auditing the validity of executable repair benchmarks”, both RTLLM and VerilogEval are generation benchmarks.

(2) The Fail2Bench benchmark suite is smaller (81 tasks) compared to HWE-bench. Moreover, 36 out of 81 tasks are saturated, and the authors could not rank the three top API based models based on the discriminating tasks.

(3) The tasks, harness, and authoring tools are co-written using Claude, and then the leakage and resolution findings are evaluated with Claude models. Although the authors have mentioned it in the limitations, it is not measured how much this circularity can affect the evaluation.

(4) The authors didn’t state the bug-class selection criteria or how the bugs were assigned for the 81 repair tasks.

Questions for authors

(1) Could the author run their instrumentation (leakage and discrimination) audits on HWE-Bench or even a subset of samples?

(2) Since Claude was used in both construction and evaluation of the benchmark, can the authors quantify how this overlap affects the leakage and low-resolution results?

(3) What criteria were used to select bug classes and assign bugs in 81 repair tasks?

(4) Could the authors provide per-bug-class breakdowns on the saturated tasks?

Reviewer_6ZwBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the validity of executable program repair benchmarks, arguing that the conventional fails-before-repair and passes-after-repair validation criteria are insufficient. The authors construct Fail2Bench, a benchmark containing 81 Verilog repair tasks, and introduce three auditing instruments to examine answer leakage, oracle strength, and benchmark discriminative power. Their findings reveal substantial answer leakage, inadequate test coverage as measured by mutation testing, and limited discriminative power among strong models.

Critical strengths

The paper addresses the reliability of executable repair benchmarks, which is important for accurately evaluating coding agents.

Critical weaknesses
  1. Limited methodological novelty and insufficient discussion of closely related work. The proposed auditing methods largely build on established techniques, including mutation testing, leakage detection, and model performance analysis. In particular, recent studies such as SWE-ABS (ICML 2026), UTBoost (ACL 2025) have already investigated the inadequacy of benchmark test suites and proposed techniques for detecting or addressing these issues. The paper does not sufficiently distinguish its methodological contributions from these studies.

  2. Limited external validity of the empirical findings. Most findings are based on a relatively small, self-constructed benchmark of 81 Verilog repair tasks, rather than real-world repair tasks. Although the authors additionally evaluate oracle strength on RTLLM and VerilogEval, these are code-generation benchmarks rather than repair benchmarks. Evaluating the proposed auditing methods on established repair benchmarks, such as HWE-Bench or SWE-bench, would substantially strengthen the conclusions.

  3. Insufficient evidence connecting oracle mutation scores to actual repair evaluation reliability. The mutation-based audit measures how many injected mutants are detected, but does not directly establish whether weak oracles incorrectly accept agent-generated repairs. Prior work has examined this issue by validating supposedly successful patches against strengthened test suites. Similar experiments would make the practical implications of the reported oracle weaknesses more convincing.

  4. The answer leakage findings may primarily reflect problems in the authors' own benchmark construction process. The paper reports that most tasks contain explicit repair hints introduced by its own task-authoring helper. While this finding is useful as a cautionary example, it is unclear how representative such leakage is of existing independently developed repair benchmarks. Without auditing external benchmarks for leakage, the broader implications remain uncertain.

  5. Limited insights from the discriminative power analysis. The observation that strong models solve most tasks and that saturated tasks reduce benchmark discrimination is relatively intuitive. The paper does not provide a sufficiently novel methodology or deeper analysis beyond identifying tasks with identical success rates across the evaluated models. The findings are also dependent on the particular models and sampling budgets.

Reference:

  1. SWE-ABS (ICML 2026): SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark

  2. UTBoost (ACL 2025): UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench.

Questions for authors
  1. How does the proposed mutation-based oracle audit differ from existing studies such as SWE-ABS, UTBoost? What are the main methodological contributions beyond applying established techniques?

  2. Can the authors apply all three auditing instruments to an existing real-world repair benchmark, such as HWE-Bench or SWE-bench, to demonstrate their generalizability?

  3. Do the observed oracle weaknesses lead to incorrect agent-generated patches being accepted? Can the authors quantify the relationship between mutation scores and false-positive repair success rates?

  4. How prevalent is answer leakage in existing repair benchmarks? Is there evidence that the leakage observed in Fail2Bench reflects a broader problem rather than an artifact of the authors' task-generation process?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.