- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper questions fail-to-pass (F→P) as the sole quality criterion for bug reproduction tests (BRTs): a test can fail on the buggy code and pass on the golden fix while still accepting plausible but incorrect patches. It introduces Mutation Patch Evaluation (MPE), which generates incorrect variants of the golden fix under five operators and measures how many a generated test accepts but the golden test rejects, splitting tests into Rigorous, Lax and Misaligned. Two empirical findings follow. When BRTs from four published generators are injected into a repair agent, the gain comes from the Rigorous half (+8.5 Resolved) and not the Lax half (0.0). When one agent co-generates fix and test, their errors are coupled: 80 of 166 unresolved instances pair an F→P-passing test with a wrong fix. The paper then proposes CoHarden: a test-only first round (lax-init), followed by up to five hardening rounds in which mutants of the current fix are run against the current and previous test, and the survivors are fed back so the agent tightens the test and updates the fix together. On the 433-instance SWE-bench Verified ∩ SWT-bench Verified set with GPT-5-mini, CoHarden reports 69.4% Resolved against 61.4% for vanilla co-generation and 61.5% for InfCode, with smaller gains on GPT-5.4, Claude Opus 4.5 and SWE-bench Lite.
Critical strengths
-
The empirical study identifies a dimension of test quality that F→P hides and ties it to downstream repair utility. On 1,104 F→P (instance, BRT) pairs from four independent generators, tests in the low-laxity half raise Resolved by 8.5 points while the high-laxity half adds nothing (Table 1). The gap stays positive across laxity cutoffs from 0.05 to 0.60 (Table 12) and under two other mutant generators (4.45–5.98 points, Table 11), even though 20–30% of individual labels flip. This matters independently of CoHarden: it changes how BRT generators should be evaluated.
-
The method follows directly from the diagnosis and is well reported. The Temporal Matrix replaces the golden test with the previous-round test, so the signal is available inside the loop, and the cost is in the range of the baselines ($0.84 against $0.56–0.88 per instance). The paper also reports what goes against it: F→P falls from 83.3% to 78.9% during hardening, Misaligned tests rise from 18.5% to 24.7%, a quarter of failures are attributed to later-round test–fix coupling, and the main results are stated to be single runs. This makes it easy to see what is and is not done.
Critical weaknesses
-
All evidence comes from SWE-bench Verified and Lite, where the mechanism under test is specifically exposed to contamination.
- The two evaluation sets are drawn from the same 12 Python repositories and period, so Lite is not an independent benchmark and "transfer to a different benchmark" overstates what Table 4 shows.
- OpenAI announced it would stop reporting SWE-bench Verified: an audit of 138 problems found material test-design or description issues in 59.4%, and frontier models were shown to reproduce the human-written fix. Claude Opus 4.5, one of the three backbones here, is named as recalling an exact functional change with its inline comment; GPT-5.2, from the same family as the other two backbones, reproduced gold patches from minimal hints.
- A shared backbone cancels a general inflation in relative comparisons, but CoHarden's gain is mediated by test quality. Lax-init has to produce a test aligned with the golden test, and hardening has to push it toward golden behaviour. If the backbone has seen the golden test and fix, alignment and rigor may partly be recall, and asking the model to write and tighten the test first may help more on memorized instances than on unseen ones. The MPE labels have the same exposure, since the mutants are generated with the golden fix and golden test in context.
This is consequential because it bears on whether the central gain would hold on unseen issues. A post-cutoff or contamination-resistant set with fail-to-pass test patches (for example SWE-bench-Live, SWE-rebench or SWE-bench Pro) would settle it; at $0.84 per instance, even 100–200 instances comparing CoHarden with vanilla co-generation and one baseline is inexpensive, and a non-Python subset would also widen the scope.
-
The reported numbers are not consistent with each other, and the control that would attribute the gain to mutation feedback is missing.
- Two headline results. The abstract, Table 2 and Section 4.1 give 69.4% Resolved (+9.6 and +7.9 over the best baselines); the conclusion gives 67.9% (+8.1 and +6.4). Section 4.1 cites Table 15 for "63.8% versus 69.4%", but Table 15 and Section E.5 give 67.9%. The failure analysis covers 139 unresolved cases, i.e. 294/433 = 67.9%, although it is described as the full-budget 433-instance run. The two sets of numbers are each internally consistent, so they appear to come from two different runs.
- Denominators. 69.4%, 78.9%, and the R2/R3 values in Figure 4 (67.8%, 68.3%) do not correspond to integer counts out of 433 (e.g. 300/433 = 69.3%, 301/433 = 69.5%) but all do for 432, whereas 67.9% = 294/433 does. Several baseline entries show the same pattern (55.3, 46.5 and 61.5 of 433 in Table 2; 52.0, 51.0, 54.2 and 63.7 of 276 in Table 4), which suggests differing instance counts or numbers taken from other sources; the paper does not say.
- Table 1 versus Table 7. Reconstructing counts from the four Misaligned rows of Table 7 gives 211 → 185 of 474 (44.5% → 39.0%, −5.5), whereas Table 1 reports 44.5% → 40.9% (−3.6, i.e. 194). The FIX-ONLY count matches exactly and the gap of nine instances exceeds the ±4 attributable to rounding. The Rigorous and Lax rows reconcile within rounding.
- Table 13. With three seeds on a fixed 60-instance subset, a mean must be a multiple of 1/180 (0.56 points); 68.1, 64.0, 63.6 and 63.8 are not. The smallest non-zero standard deviation for three values on a 1/60 grid is 0.79 (population) or 0.96 (sample), so 0.7 is not attainable either.
- Table 5 versus Section 4.3. The text states that lax-init improves F→P by 8.7 points "both with and without hardening"; the table gives 78.9 − 70.2 = 8.7 with hardening but 82.6 − 72.7 = 9.9 without.
- Attribution. In Table 5, hardening without lax-init adds 1.0 point and lax-init without hardening −0.4; only the combination adds 8.0, as Section 4.3 acknowledges. The round-matched control (Ralph-Loop, 63.8%) starts from the vanilla prompt, so it differs from CoHarden in both initialization and feedback, as Section E.5 notes. The missing cell is lax-init followed by five generic "review and improve" rounds.
The two apparent runs of the same configuration already differ by 1.5 points, which is comparable to the cross-backbone (+2.7, +4.0) and Lite (+3.4, about nine instances) margins, all of which are single-run. Without the missing control, the 8-point gain cannot be assigned to mutation feedback rather than to "test first, then more rounds".
Questions for authors
- Do the gains of CoHarden over vanilla co-generation hold on a benchmark built from issues after the backbones' training cutoff (e.g. DeepSWE or SWE-bench Pro), where the golden test and fix cannot have been seen?
- Which run is the reference result, 69.4% or 67.9%, and which run do the failure analysis, Figure 4 and Table 16 describe? What are the denominators behind 69.4% / 78.9%, and how were the means and standard deviations in Table 13 computed?
- What is the Resolved rate for lax-init followed by five generic refinement rounds without mutation feedback?
- Were all baselines in Tables 2–4 rerun on the same 433 and 276 instances with the same GPT-5-mini snapshot, or are some entries taken from prior reports?
Response to Reviewer RoHj
Thank you for the thoughtful review and for recognizing MPE as a practically important dimension of BRT quality.
Regarding the concerns and questions you raised:
W1/Q1: Benchmark Contamination and OOD Generalization
COHARDEN requires an executable reproduction-test evaluation setup to measure F-to-P and MPE consistently. SWE-bench Pro and the other suggested post-cutoff benchmarks do not currently provide the corresponding BRT evaluation framework, whereas SWT-Bench supports this evaluation on the Verified and Lite subsets. We therefore use these two subsets and revised the manuscript to describe Lite as a second evaluation subset rather than a different-domain transfer.
We also conducted an additional paired evaluation on 98 shared SWE-bench-Live instances using GPT-5.4 and the official evaluator:
The golden test and fix are used only as evaluation-time oracles in the offline MPE analysis; they are never included in COHARDEN's inference context. During inference, mutation feedback is constructed solely from the current fix and consecutive generated tests.
W2/Q2: Result Reconciliation and Denominators
Thank you for carefully checking the reported results and pointing out these inconsistencies. These inconsistencies are confined to supplementary appendix analyses. The main-text experiments and the evidence supporting our central claim that COHARDEN improves both BRT quality and downstream repair performance through iterative hardening remain unchanged. We made the following corrections:
W2/Q3: Generic Refinement Control
We ran the missing matched control on a fixed 100-instance subset with GPT-5.4. It uses the same instances, lax-init, and five-round budget as COHARDEN, while replacing mutant-derived feedback with generic review-and-improve feedback:
The point estimates favor mutation-guided hardening on both repair and test outcomes under matched initialization and round budgets. The phase ablation further shows that the full gain requires lax-init and hardening together.
Q4: Baseline Provenance
Yes. All baselines in Tables 2--4 were rerun on the same instances and backbone stated in each table. ``Official implementation'' denotes the codebase used for rerunning, not results copied from prior reports.