- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines the validity of executable repair benchmarks, arguing that the two-sided check—failure of the buggy implementation and success of the reference repair—is insufficient to establish reliable evaluation. Using Fail2Bench, a suite of 81 Verilog repair tasks, the authors audit answer leakage in task inputs, testbench fault-detection capability, and the ability of tasks to distinguish model performance. They also apply mutation testing to RTLLM and VerilogEval. Additional findings concern defects in task construction and the evaluation harness, as well as unsuccessful attempts to derive novel tasks from stronger models’ failed repairs.
Critical strengths
- The problem is practically relevant and supported by concrete examples. The paper demonstrates why repair benchmarks require more than executable tests and successful validation of buggy and reference implementations. Examples involving explicit solutions in task inputs, testbenches that miss additional faults, and a patch-application defect that substantially changes measured performance provide actionable lessons for benchmark developers. The finding that the evaluation harness itself can materially alter results is particularly valuable.
- External audits and transparent reporting strengthen the contribution. Applying the mutation audit to RTLLM and VerilogEval extends the evidence beyond the authors’ own suite. The paper also explicitly reports differences in experimental budgets, instability in per-task outcomes across repeated runs, and findings that were withdrawn after further investigation. This transparency helps readers assess the scope and limitations of the evidence.
Critical weaknesses
-
The cross-benchmark comparison does not sufficiently establish the proposed causal explanation for detection-rate differences. The abstract and Figure 1 attribute the difference between directed and differential testing primarily to oracle design. However, the three benchmarks differ in their designs, input stimuli, observation schedules, and mutant populations. Applying the same mutation operators does not control these differences. The results establish differences in mutation detection across the three testing setups, but do not isolate oracle type as their main cause.
The treatment of equivalent mutants also needs clarification. The paper excludes non-compiling mutants but does not report equivalence checks on surviving mutants. VerilogEval’s ability to detect clock-edge mutations in its own designs does not establish that mutations of the same class in the other benchmarks are all non-equivalent. A sampled inspection of surviving mutants and an associated sensitivity analysis would help. Without a controlled comparison, the attribution in the abstract and Figure 1 should be narrowed. These concerns do not invalidate the descriptive detection rates, but they limit their interpretation.
-
The connection between the reported increase in discriminating tasks and stable model discrimination remains insufficiently established. Section 7 labels a task discriminating whenever at least two models have different empirical pass rates. With only two or three samples per task, models with identical underlying success probabilities can nevertheless exhibit different empirical pass rates. Consequently, this metric reflects sampling variability and the level of task success probabilities, in addition to genuine differences between models.
Matching the model set and sample counts in some comparisons is helpful, and the authors appropriately acknowledge the metric’s dependence on sampling budget. However, this does not establish how much of the increase from 14.8% to 32.1% reflects reproducible model differences rather than increased disagreement under finite sampling. The disagreement on 12 tasks between the two Sonnet runs makes this an important question.
Table 2 is valid as a descriptive analysis of the observed runs. Interpreting the selected tasks as reliably carrying signal, however, requires evidence of stability across repeated runs. Separately, overlapping marginal confidence intervals do not by themselves establish that models cannot be ranked. Comparisons on the same tasks should report confidence intervals for paired performance differences.
Questions for authors
- Interpretation of the mutation audit: Do you have comparisons of directed and differential testing on the same designs and mutant populations? If not, would you narrow the causal language to an association and provide a sampled inspection of surviving mutants?
- Discrimination: What baseline level does the current discrimination metric attain when models have equal underlying success probabilities but outcomes vary because of finite sampling? Could you use independent repeated runs to assess the stability of the discriminating task set and whether the observed improvement exceeds this baseline?
- Experimental conditions and model comparisons: Could you report confidence intervals for paired pass@1 differences between the principal models, and clarify the sensitivity of hint effects to time limits, timeout scoring, and the recovery of Opus results? The additional Sonnet run without abandonments already addresses part of this concern for that model, but the implications for the other comparisons remain unclear.
- Reporting consistency: Table 1 lists three samples for Sonnet, whereas Appendix E lists five for its with-hints arm. Opus also has different sample counts across conditions, which the single sample-count column in Table 1 does not represent. In addition, Table 5 states that it covers 81 tasks, but the counts in both taxonomy columns sum to 52. Please clarify the samples and task populations underlying these results and reconcile the reporting.