acceptodds
Under review as a conference paper at ICLR 2027

Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes

Abstract

Bug reproduction tests (BRTs) turn bug reports into executable signals for automated program repair, but the usual fail-to-pass (\ftp) criterion does not establish whether a test supports correct repair. A test may fail on buggy code and pass on the golden fix while also accepting plausible-but-incorrect patches. We introduce Mutation Patch Evaluation () to distinguish rigorous tests from such lax tests, and show empirically that downstream repair gains concentrate in the rigorous tier. We also identify error coupling in co-generation, where a generated test accepts an incorrect generated patch. Based on these findings, we propose , a co-generation framework for iterative test-and-fix refinement. An initial test-only round targets the observable bug symptom before committing to a fix. Subsequent hardening rounds use mutated variants of the current fix to strengthen the test and refine the patch. On SWE-bench Verified, reaches Resolved, improving on the strongest fix-only and co-generation baselines by and percentage points, respectively. Further experiments show gains across LLM backbones and benchmarks. Code is available at https://anonymous.4open.science/r/COHARDEN-9753.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Automated Bug Repair, Bug Reproduction Test Generation, LLM4SE

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RoHjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper questions fail-to-pass (F→P) as the sole quality criterion for bug reproduction tests (BRTs): a test can fail on the buggy code and pass on the golden fix while still accepting plausible but incorrect patches. It introduces Mutation Patch Evaluation (MPE), which generates incorrect variants of the golden fix under five operators and measures how many a generated test accepts but the golden test rejects, splitting tests into Rigorous, Lax and Misaligned. Two empirical findings follow. When BRTs from four published generators are injected into a repair agent, the gain comes from the Rigorous half (+8.5 Resolved) and not the Lax half (0.0). When one agent co-generates fix and test, their errors are coupled: 80 of 166 unresolved instances pair an F→P-passing test with a wrong fix. The paper then proposes CoHarden: a test-only first round (lax-init), followed by up to five hardening rounds in which mutants of the current fix are run against the current and previous test, and the survivors are fed back so the agent tightens the test and updates the fix together. On the 433-instance SWE-bench Verified ∩ SWT-bench Verified set with GPT-5-mini, CoHarden reports 69.4% Resolved against 61.4% for vanilla co-generation and 61.5% for InfCode, with smaller gains on GPT-5.4, Claude Opus 4.5 and SWE-bench Lite.

Critical strengths
  1. The empirical study identifies a dimension of test quality that F→P hides and ties it to downstream repair utility. On 1,104 F→P (instance, BRT) pairs from four independent generators, tests in the low-laxity half raise Resolved by 8.5 points while the high-laxity half adds nothing (Table 1). The gap stays positive across laxity cutoffs from 0.05 to 0.60 (Table 12) and under two other mutant generators (4.45–5.98 points, Table 11), even though 20–30% of individual labels flip. This matters independently of CoHarden: it changes how BRT generators should be evaluated.

  2. The method follows directly from the diagnosis and is well reported. The Temporal Matrix replaces the golden test with the previous-round test, so the signal is available inside the loop, and the cost is in the range of the baselines ($0.84 against $0.56–0.88 per instance). The paper also reports what goes against it: F→P falls from 83.3% to 78.9% during hardening, Misaligned tests rise from 18.5% to 24.7%, a quarter of failures are attributed to later-round test–fix coupling, and the main results are stated to be single runs. This makes it easy to see what is and is not done.

Critical weaknesses
  1. All evidence comes from SWE-bench Verified and Lite, where the mechanism under test is specifically exposed to contamination.

    • The two evaluation sets are drawn from the same 12 Python repositories and period, so Lite is not an independent benchmark and "transfer to a different benchmark" overstates what Table 4 shows.
    • OpenAI announced it would stop reporting SWE-bench Verified: an audit of 138 problems found material test-design or description issues in 59.4%, and frontier models were shown to reproduce the human-written fix. Claude Opus 4.5, one of the three backbones here, is named as recalling an exact functional change with its inline comment; GPT-5.2, from the same family as the other two backbones, reproduced gold patches from minimal hints.
    • A shared backbone cancels a general inflation in relative comparisons, but CoHarden's gain is mediated by test quality. Lax-init has to produce a test aligned with the golden test, and hardening has to push it toward golden behaviour. If the backbone has seen the golden test and fix, alignment and rigor may partly be recall, and asking the model to write and tighten the test first may help more on memorized instances than on unseen ones. The MPE labels have the same exposure, since the mutants are generated with the golden fix and golden test in context.

    This is consequential because it bears on whether the central gain would hold on unseen issues. A post-cutoff or contamination-resistant set with fail-to-pass test patches (for example SWE-bench-Live, SWE-rebench or SWE-bench Pro) would settle it; at $0.84 per instance, even 100–200 instances comparing CoHarden with vanilla co-generation and one baseline is inexpensive, and a non-Python subset would also widen the scope.

  2. The reported numbers are not consistent with each other, and the control that would attribute the gain to mutation feedback is missing.

    • Two headline results. The abstract, Table 2 and Section 4.1 give 69.4% Resolved (+9.6 and +7.9 over the best baselines); the conclusion gives 67.9% (+8.1 and +6.4). Section 4.1 cites Table 15 for "63.8% versus 69.4%", but Table 15 and Section E.5 give 67.9%. The failure analysis covers 139 unresolved cases, i.e. 294/433 = 67.9%, although it is described as the full-budget 433-instance run. The two sets of numbers are each internally consistent, so they appear to come from two different runs.
    • Denominators. 69.4%, 78.9%, and the R2/R3 values in Figure 4 (67.8%, 68.3%) do not correspond to integer counts out of 433 (e.g. 300/433 = 69.3%, 301/433 = 69.5%) but all do for 432, whereas 67.9% = 294/433 does. Several baseline entries show the same pattern (55.3, 46.5 and 61.5 of 433 in Table 2; 52.0, 51.0, 54.2 and 63.7 of 276 in Table 4), which suggests differing instance counts or numbers taken from other sources; the paper does not say.
    • Table 1 versus Table 7. Reconstructing counts from the four Misaligned rows of Table 7 gives 211 → 185 of 474 (44.5% → 39.0%, −5.5), whereas Table 1 reports 44.5% → 40.9% (−3.6, i.e. 194). The FIX-ONLY count matches exactly and the gap of nine instances exceeds the ±4 attributable to rounding. The Rigorous and Lax rows reconcile within rounding.
    • Table 13. With three seeds on a fixed 60-instance subset, a mean must be a multiple of 1/180 (0.56 points); 68.1, 64.0, 63.6 and 63.8 are not. The smallest non-zero standard deviation for three values on a 1/60 grid is 0.79 (population) or 0.96 (sample), so 0.7 is not attainable either.
    • Table 5 versus Section 4.3. The text states that lax-init improves F→P by 8.7 points "both with and without hardening"; the table gives 78.9 − 70.2 = 8.7 with hardening but 82.6 − 72.7 = 9.9 without.
    • Attribution. In Table 5, hardening without lax-init adds 1.0 point and lax-init without hardening −0.4; only the combination adds 8.0, as Section 4.3 acknowledges. The round-matched control (Ralph-Loop, 63.8%) starts from the vanilla prompt, so it differs from CoHarden in both initialization and feedback, as Section E.5 notes. The missing cell is lax-init followed by five generic "review and improve" rounds.

    The two apparent runs of the same configuration already differ by 1.5 points, which is comparable to the cross-backbone (+2.7, +4.0) and Lite (+3.4, about nine instances) margins, all of which are single-run. Without the missing control, the 8-point gain cannot be assigned to mutation feedback rather than to "test first, then more rounds".

Questions for authors
  1. Do the gains of CoHarden over vanilla co-generation hold on a benchmark built from issues after the backbones' training cutoff (e.g. DeepSWE or SWE-bench Pro), where the golden test and fix cannot have been seen?
  2. Which run is the reference result, 69.4% or 67.9%, and which run do the failure analysis, Figure 4 and Table 16 describe? What are the denominators behind 69.4% / 78.9%, and how were the means and standard deviations in Table 13 computed?
  3. What is the Resolved rate for lax-init followed by five generic refinement rounds without mutation feedback?
  4. Were all baselines in Tables 2–4 rerun on the same 433 and 276 instances with the same GPT-5-mini snapshot, or are some entries taken from prior reports?
1 comment
AuthorsOpenReview ↗

Response to Reviewer RoHj

Thank you for the thoughtful review and for recognizing MPE as a practically important dimension of BRT quality.

Regarding the concerns and questions you raised:

W1/Q1: Benchmark Contamination and OOD Generalization

COHARDEN requires an executable reproduction-test evaluation setup to measure F-to-P and MPE consistently. SWE-bench Pro and the other suggested post-cutoff benchmarks do not currently provide the corresponding BRT evaluation framework, whereas SWT-Bench supports this evaluation on the Verified and Lite subsets. We therefore use these two subsets and revised the manuscript to describe Lite as a second evaluation subset rather than a different-domain transfer.

We also conducted an additional paired evaluation on 98 shared SWE-bench-Live instances using GPT-5.4 and the official evaluator:

VanillaCOHARDENBoth solvedCOHARDEN onlyVanilla onlyNeitherDifference (95% paired bootstrap CI)
43/9847/98398447+4.1 points [-3.1, 11.2]

The golden test and fix are used only as evaluation-time oracles in the offline MPE analysis; they are never included in COHARDEN's inference context. During inference, mutation feedback is constructed solely from the current fix and consecutive generated tests.

W2/Q2: Result Reconciliation and Denominators

Thank you for carefully checking the reported results and pointing out these inconsistencies. These inconsistencies are confined to supplementary appendix analyses. The main-text experiments and the evidence supporting our central claim that COHARDEN improves both BRT quality and downstream repair performance through iterative hardening remain unchanged. We made the following corrections:

  • Final results. All rates use the full 433-instance denominator, with the timed-out instance counted as unsuccessful. The final evaluation resolves 300 instances (69.3%), and 341 instances have F-to-P tests (78.8%).
  • Appendix analyses. We updated Figure 4, the appendix text and captions, the per-agent table, and the ablation arithmetic to use the final results consistently.
  • Table 13. The previous analysis script excluded timed-out evaluations from the denominator. We reaggregated each seed over the fixed 60-instance subset and counted timed-out evaluations as unsuccessful. The corrected results are:
VariantResolved (%)Test F-to-P (%)
COHARDEN66.7 +/- 1.772.2 +/- 1.0
w/o hardening62.2 +/- 3.579.4 +/- 1.0
w/o lax-init62.2 +/- 1.966.7 +/- 6.0
w/o both / vanilla COGEN61.7 +/- 1.768.3 +/- 1.7

W2/Q3: Generic Refinement Control

We ran the missing matched control on a fixed 100-instance subset with GPT-5.4. It uses the same instances, lax-init, and five-round budget as COHARDEN, while replacing mutant-derived feedback with generic review-and-improve feedback:

MetricGeneric feedbackCOHARDENDifference (95% paired bootstrap CI)
Resolved71.0%76.0%+5.0 points [-3.0, 13.0]
Test F-to-P64.0%69.0%+5.0 points [-4.0, 14.0]

The point estimates favor mutation-guided hardening on both repair and test outcomes under matched initialization and round budgets. The phase ablation further shows that the full gain requires lax-init and hardening together.

Q4: Baseline Provenance

Yes. All baselines in Tables 2--4 were rerun on the same instances and backbone stated in each table. ``Official implementation'' denotes the codebase used for rerunning, not results copied from prior reports.

Reviewer_Bx9pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the quality of bug reproduction tests (BRTs) for automated program repair. The authors argue that the conventional Fail-to-Pass (F→P) criterion, which requires a test to fail on the buggy program and pass on the reference fix, does not establish whether the test also accepts plausible but incorrect patches.

To address this limitation, the paper introduces Mutation Patch Evaluation (MPE), which uses semantic patch mutations to classify generated tests as Rigorous, Lax, or Misaligned, and investigates the relationship between these categories and downstream repair utility. Based on these findings, the authors propose COHARDEN. The framework first generates a reproduction test without modifying source code, then jointly refines the test and patch using mutation feedback. A Temporal Matrix compares consecutive tests on sampled patch variants, and the procedure returns the previous-round result when it detects a relaxation of test constraints.

The main evaluation uses the 433-instance intersection of SWE-bench Verified and SWT-bench Verified. Table 2 reports a Resolved rate of 69.4%. The paper also includes evaluations with additional model backbones and the Lite subset, phase ablations, mutant-pool sensitivity analyses, and a failure analysis. The overall results suggest a promising direction, although the reporting of the main results and several diagnostic metrics requires clarification.

Critical strengths
  1. The research question is well motivated and directly relevant to the reliability of practical program repair. A test satisfying F→P may accept both a correct patch and an incorrect one, making it insufficient as a correctness signal. Figure 1 illustrates this problem effectively. The test injection study further attempts to connect test quality with actual repair utility rather than measuring test generation success alone. This gives the analysis practical significance.

  2. The method directly addresses the identified failure modes and is supported by a broad set of empirical analyses. Test-first initialization, targeted patch mutations, and comparisons between consecutive tests form a concrete, implementable repair procedure. Reported improvements extend across multiple repair backbones and two benchmark subsets, with additional phase ablations and sensitivity analyses for mutation generators and classification thresholds. The authors also explicitly acknowledge limitations involving finite mutant pools, unstable labels, and later-round test–patch drift, which helps characterize the method’s scope.

Critical weaknesses
  1. Inconsistencies in the core results and evaluation populations undermine confidence in both the performance claims and the mechanism analysis. The following issues cannot be explained by rounding alone and require the corresponding run identifiers, denominators, and raw counts:

    • Table 2 reports a Resolved rate of 69.4% for COHARDEN on 433 instances. Under the stated definition of Resolved and the single-run evaluation protocol, this percentage should correspond to an integer number of resolved instances. However, 300/433 rounds to 69.3%, and 301/433 rounds to 69.5%; neither produces 69.4%. If a different denominator, an average across runs, or another aggregation scheme was used, this should be stated explicitly.
    • Section 4.1 directly compares COHARDEN’s 69.4% with Ralph-Loop’s 63.8%. However, Table 15 explicitly describes a separate round-matched experiment in which COHARDEN achieves 67.9%. Different runs can legitimately produce different results, but the comparison should use results from within the same control experiment rather than combining results across runs.
    • Under the taxonomy in Section 2.1, Rigorous and Lax tests together constitute the F→P population. The final row for COHARDEN in Table 8 therefore implies an F→P rate of 66.4% + 8.9% = 75.3%, which differs from the 78.9% in Table 2 and the final value in Figure 4. If these results use different runs, output-handling procedures, or evaluation populations, these distinctions need to be documented. The current explanation is insufficient to reconstruct their relationship.

    These issues affect the main performance results and the diagnostics used to interpret them. They require a consistent set of per-instance records, rather than only corrections to the abstract or conclusion.

  2. The evidence does not sufficiently establish that the constraint changes measured by MPE and the Temporal Matrix lead to useful repair improvements, or isolate the contribution of this mechanism. In offline MPE, a Rigorous label is primarily determined by F→P and a low β/|C|, without simultaneously constraining γ. Yet Section 2.1 interprets γ as counting patches that the generated test rejects but the reference test accepts. A test could therefore reject many patches accepted by the reference criterion and still be labeled Rigorous because of a low β. This limits the interpretation of the label as indicating consistency with correct repairs.

    In the online procedure, the Temporal Matrix substitutes the previous generated test for the reference test. Consequently, β = 0 only indicates that no relaxation relative to the previous test was observed on the currently sampled patches. It does not distinguish meaningful strengthening from no change or the further strengthening of an incorrect constraint. With at most 12 mutants per round, this judgment is also sensitive to sampling. The authors acknowledge shared drift, but do not directly ablate the contribution of the Temporal Matrix rollback rule.

    Furthermore, the Ralph-Loop control changes initialization, feedback content, and inference expenditure simultaneously, as Section E.5 explicitly acknowledges. Table 5 removes the entire hardening phase, so it does not distinguish the effects of additional iterations, detailed debugging instructions, mutation feedback, and the rollback rule. The current evidence more strongly supports the effectiveness of the complete procedure than the claim that the proposed feedback mechanism is the key source of improvement.

Questions for authors
  1. Clarification of result provenance: Please provide the run identifiers, instance counts, and integer success counts underlying Table 2, Figure 4, Table 8, and Table 15. How is the reported 69.4% obtained from the stated single-run evaluation on 433 instances? Please also explain why Section 4.1 compares 69.4% with the 63.8% in Table 15, and reconcile the 75.3% F→P implied by Table 8 with the 78.9% in Table 2.

  2. Clarification of the metric: Why does the Rigorous classification in MPE not also account for γ or the rejection rate among patches that pass the reference test? Are any tests with low β but high γ currently classified as Rigorous? Does the relationship between test quality and repair utility remain after accounting for such overconstraint?

  3. Clarification of algorithm behavior: Does the implementation terminate and roll back upon observing β > 0, as stated in Algorithm 1, or terminate when no Laxity cell remains, as described in Figure 3? How does it handle β = γ = 0 when some patches still pass both consecutive tests? How many instances terminate through rollback, and how many reach the round limit?

  4. Additional validation that would affect my assessment: Can the authors compare generic self-refinement, refinement with mutation feedback, and refinement with both mutation feedback and the Temporal Matrix rollback rule, while holding lax-init, debugging instructions, and inference budgets approximately constant? Per-instance paired outcomes and uncertainty estimates for the differences would help assess the mechanism’s contribution and the stability of the gains. The test injection analysis should also account for dependence when multiple BRTs correspond to the same issue.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Bx9p

Thank you for the careful review and for recognizing the practical importance of evaluating BRT quality beyond F-to-P. We hope the responses below address your concerns about the paper's soundness.

Regarding the concerns and questions you raised:

W1/Q1: Result Provenance

Thank you for carefully checking the reported results and pointing out these inconsistencies. These inconsistencies are confined to supplementary appendix analyses. The main-text experiments and the evidence supporting our central claim that COHARDEN improves both BRT quality and downstream repair performance through iterative hardening remain unchanged. We made the following corrections:

  • Earlier appendix results. We corrected numerical inconsistencies caused by earlier appendix results that had not been updated and standardized the headline results to the final 433-instance run, counting the timed-out instance as unsuccessful.
  • Table 8. Its analysis script omitted instances without a snapshot at a given round, resulting in round-specific effective denominators. We clarified this output-selection rule in the revision.

W2/Q2: Metric Definition

Yes, a low-beta, high-gamma test is classified as Rigorous because the Rigorous/Lax distinction measures false acceptance. We examined overconstraint by stratifying all 1,104 F-to-P pairs using median splits on beta/total and gamma/total:

beta/totalgamma/totalF-to-P pairsPaired Resolved gain
LowLow286+9.4 points
LowHigh266+7.5 points
HighLow266-1.5 points
HighHigh286+1.4 points

The repair gain remains concentrated in the low-beta groups at both gamma levels. We clarified that Rigorous denotes low observed false acceptance, while gamma separately captures rejection of reference-test-passing mutants.

W1/Q3: Algorithm Behavior

The stopping rule is based on surviving mutants: the loop converges when at least one admissible mutant is evaluated and none survives the current test; otherwise, the survivors guide the next hardening round. Appendix D.2 specifies the regression guard. We compute the temporal Laxity rate as beta/(beta+delta); when it exceeds epsilon=0.2, the test is rolled back to the previous accepted version while the current fix is retained, and hardening continues. Thus, when beta and gamma are both zero but alpha-cell survivors remain, those survivors are included in the next-round feedback.

In the final evaluation, 59 instances triggered at least one rollback and 61 reached the round limit. Zero-admissible-mutant rounds occurred in two instances (five rounds in total); the regression check was skipped for those rounds.

W2/Q4: Additional Validation

We ran a matched feedback control on a fixed 100-instance subset with GPT-5.4. It uses the same instances, lax-init, and five-round budget as COHARDEN, while replacing mutant-derived feedback with generic review-and-improve feedback:

MetricGeneric feedbackCOHARDENDifference (95% paired bootstrap CI)
Resolved71.0%76.0%+5.0 points [-3.0, 13.0]
Test F-to-P64.0%69.0%+5.0 points [-4.0, 14.0]

The point estimates favor mutation-guided hardening on both repair and test outcomes under matched initialization and round budgets. Table 5 further shows that neither lax-init nor hardening alone recovers the full gain; the improvement emerges from combining the two stages.

For the main comparison, we computed paired outcomes over the same 433 instances:

ComparisonBoth solvedCOHARDEN onlyBaseline onlyNeitherDifference95% paired CI
COHARDEN vs. OpenHands + Cogen2425824109+7.9 points[3.9, 12.0]

For the injection study, we resampled by underlying issue while keeping the shared no-test baseline paired within each replicate. The 1,104 pairs cover 404 underlying issues:

ComparisonF-to-P pairsResolved difference95% issue-clustered CI
Rigorous vs. no-test552+8.5 points[2.9, 14.3]
Lax vs. no-test5520.0 points[-4.1, 4.1]
Reviewer_nBtzOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a bug reproduction test (BRT) satisfying fail-to-pass (F→P) can still admit incorrect patches. It introduces Mutation Patch Evaluation (MPE), which compares generated and reference tests on mutated patches and categorizes tests as Rigorous, Lax, or Misaligned. In the test-injection study, repair success rises from 67.3% to 75.8% for the Rigorous group, while remaining at 69.5% for the Lax group (Table 1). The paper also examines correlated errors in jointly generated tests and patches. Its proposed CoHarden pipeline first generates a symptom-level test, then jointly refines the test and patch using mutations of the current patch and a Temporal Matrix comparing consecutive tests. The principal evaluation uses the 433-instance intersection of SWE-bench Verified and SWT-bench Verified. Table 2 reports 69.4% Resolved, 78.9% test F→P, and approximately $0.84 per instance. Additional evaluations cover stronger backbones, the Lite benchmark, phase ablations, mutant-pool sensitivity, and limited repeated runs.

Critical strengths

S1. An important evaluation question supported by downstream evidence. The distinction between reproducing a bug and constraining its repair is consequential. Figure 1 concretely shows why an F→P test can accept a symptom-avoiding patch. Table 1 goes beyond test-generation success by measuring the effect of injecting tests into a repair agent. Tables 11–12 further show a positive Rigorous-versus-Lax utility gap under alternative generators and thresholds. The resulting evidence is useful for BRT evaluation and repair-agent design, even though it does not establish a universal causal relationship.

S2. A method motivated by the diagnosis, with substantial supplementary analysis. Test-first initialization and subsequent joint hardening correspond directly to initial test–patch coupling and weak test constraints. The complete method's 69.4% Resolved exceeds the variants without lax-init (62.4%) and without hardening (61.0%), supporting complementary roles; this does not imply that lax-init independently improves repair. Tables 3–4 broaden the evidence across models and Lite instances. Shared-pool evaluation, sensitivity analyses, cost profiles, and failure analysis are valuable additions. The explicit acknowledgement that a finite mutant pool cannot certify correctness is also appropriate.

Critical weaknesses

W1. The stopping rule and the provenance of central results are not consistently specified. Figure 3 says the loop terminates when no Laxity cell remains, whereas Section 3.2 describes rollback on a nonempty Laxity cell and Algorithm 1 in Appendix D.2 explicitly returns the previous pair when k > 1 and β > 0. The appendix supplies a concrete rule. The figure may refer to the returned pair rather than the triggering condition, but this distinction is not explained and should be reconciled with the implementation.

Several reported results also require a run-level reconciliation. The abstract and Table 2 use 69.4%, while the conclusion uses 67.9%. Appendix E.5 explicitly identifies Table 15's 67.9% as a separate round-matched experiment; separate runs are legitimate. However, Section 4.1 compares the primary run's 69.4% directly with that experiment's Ralph-Loop result of 63.8%, although the within-table comparison is 67.9% versus 63.8%. Table 8 describes all 433 instances and reports 24.7% final Misaligned tests, implying 75.3% F→P under Section 2.1's taxonomy, rather than Table 2's 78.9%. Appendix E.6's 139 unresolved instances likewise imply approximately 67.9% Resolved. These discrepancies do not establish that the experiments are invalid, but the manuscript needs an explicit mapping from each table, figure, and failure analysis to its generating run.

W2. The controls do not fully isolate the feedback mechanism, and MPE remains dependent on its measurement instrument. The Ralph-Loop experiment is useful, so it would be inaccurate to say that extra refinement is entirely uncontrolled. Nevertheless, initialization differs, and equal rounds do not equalize tokens, expenditure, or time. The results support the complete procedure more directly than the specific contribution of mutation feedback or the Temporal Matrix. A stronger control would retain lax-init, the debugging instructions, and comparable resources while changing only feedback or rollback. Comparing survivor-count feedback with the full matrix would also clarify its additional value.

MPE labels are based on a median split of β/total and do not incorporate γ, which can indicate overconstraint relative to the reference test. Passing that reference test is itself not a semantic correctness guarantee. Moreover, Appendix B.1's original generator sees the reference patch, reference test, and evaluated test, making that instrument test-conditioned. The shared, method-agnostic pool in Appendix E.1 helps, but its generation context and its distinction from the original test-conditioned pools need clearer documentation. Changing generators flips 19.7–29.7% of labels, and the structural diversity audit does not establish that mutants represent realistic incorrect repairs. The authors acknowledge these limitations; the claim that gains concentrate in Rigorous tests should accordingly remain an empirical association under an operational classification. The 1,578 test–instance pairs may also share underlying issues and a common no-test baseline, requiring instance-clustered uncertainty estimates.

Questions for authors
  1. Stopping rule and rollback (Figure 3, p. 5; Section 3.2, p. 6; Algorithm 1 and Appendix D.2, p. 20): Algorithm 1 explicitly returns the previous pair when k > 1 and beta > 0, whereas Figure 3 says the loop terminates when no Laxity cell remains. Is the figure describing a property of the returned pair after rollback, or a different stopping condition? Please reconcile the wording with the implemented rule and report how often the regression and zero-admissible-mutant cases occur. Appendix D.2 already explains how zero-admissible-mutant rounds are handled; the request here is for their observed frequency.

  2. Run-level reconciliation (Table 2, p. 7; Table 8, p. 21; Appendix E.5 and Table 15, pp. 23–24; Table 18, p. 25): Appendix E.5 already identifies a separate round-matched experiment reporting 67.9% Resolved and 75.3% F-to-P. Please explain why Section 4.1 instead compares Ralph-Loop's 63.8% with 69.4%, and identify which runs supply Table 8 and the 139 unresolved cases in Table 18. Table 8's final Misaligned share implies 75.3% F-to-P, and 139 unresolved out of 433 implies approximately 67.9% Resolved. Please provide run identifiers, integer outcome counts, and output-selection rules so these results can be reconciled with Table 2's 69.4%/78.9%.

  3. Attribution of the improvement (Table 5, p. 7; Appendix E.5, pp. 23–24): The phase ablation and Ralph-Loop control are useful, and E.5 explicitly acknowledges that the latter compares complete procedures with different initialization and unmatched expenditure. Is there existing evidence that isolates mutation feedback while retaining the same lax-init and debugging instructions? A comparison with matched initialization and reported resource use would materially strengthen attribution. Otherwise, please keep the claim at the level of the complete procedure rather than an isolated benefit from the Temporal Matrix.

  4. Evaluation-pool construction and overconstraint (Section 2.1, p. 3; Appendix B.1, p. 15; Appendix E.1, p. 21): E.1 explicitly describes a shared, method-agnostic pool reused across methods and rounds, whereas B.1 supplies the evaluated test to the mutation generator. Please specify how the E.1 pool was constructed and whether any evaluated test was included in its generation context. Since the Rigorous/Lax split uses beta/total rather than gamma, can a low-beta test that rejects many reference-test-passing mutants still be classified as Rigorous? Please explain whether this occurs and how it affects the interpretation of the labels.

  5. Uncertainty in the principal comparisons (Appendix B.2, p. 15; Table 3, p. 7; Appendix E.3, pp. 22–23): The paper correctly states that three seeds on 60 instances are not uncertainty estimates for the full benchmark. Can the authors provide paired outcome counts and uncertainty estimates for the main-method differences, including the 2.7/4.0-point gains on stronger backbones? For the injection study, please account for repeated underlying issues and the shared no-test baseline, for example with instance-clustered resampling.

1 comment
AuthorsOpenReview ↗

Response to Reviewer nBtz

Thank you for the thoughtful review and for recognizing the value of using BRT quality beyond F-to-P to guide iterative test-fix refinement.

W1/Q1: Stopping Rule and Rollback

The loop converges after evaluating a nonempty admissible pool when no mutant survives; otherwise, survivors guide the next round. Appendix D.2's regression guard rolls the test back to its previous accepted version when beta/(beta+delta) exceeds epsilon=0.2, retains the current fix, and continues hardening. We corrected Figure 3, Section 3.2, and Appendix D.2. Of 433 instances, 59 triggered rollback and 61 reached the round limit; two had zero-admissible-mutant rounds (five rounds total).

W1/Q2: Run-Level Reconciliation

Thank you for carefully checking the reported results and pointing out these inconsistencies. These inconsistencies are confined to supplementary appendix analyses. The main-text experiments and the evidence supporting our central claim that COHARDEN improves both BRT quality and downstream repair performance through iterative hardening remain unchanged. We made the following corrections:

  • Earlier appendix results. We corrected numerical inconsistencies caused by earlier appendix results that had not been updated and standardized the headline results to the final 433-instance run, counting the timed-out instance as unsuccessful.
  • Table 8. Its analysis script omitted instances without a snapshot at a given round, resulting in round-specific effective denominators. We clarified this output-selection rule in the revision.

W2/Q3: Attribution of the Improvement

We ran a matched 100-instance GPT-5.4 control with the same lax-init and five-round budget, replacing mutant-derived feedback with generic review-and-improve feedback:

MetricGeneric feedbackCOHARDENDifference (95% paired bootstrap CI)
Resolved71.0%76.0%+5.0 points [-3.0, 13.0]
Test F-to-P64.0%69.0%+5.0 points [-4.0, 14.0]

Both point estimates favor mutation-guided hardening under matched initialization and round budgets. Table 5 also shows that the full gain requires lax-init and hardening together.

W2/Q4: Evaluation-Pool Construction and Overconstraint

Appendix E.1 generates one method-agnostic mutant pool per instance without any evaluated test in context. COHARDEN and Agent-CoEvo share that pool across rounds, enabling the common trajectory comparison in Table 8.

Yes, low-beta, high-gamma tests are classified as Rigorous because Rigorous/Lax is based on false acceptance. We therefore stratified all 1,104 F-to-P pairs by median beta/total and gamma/total:

beta/totalgamma/totalF-to-P pairsPaired Resolved gain
LowLow286+9.4 points
LowHigh266+7.5 points
HighLow266-1.5 points
HighHigh286+1.4 points

High gamma occurs among Rigorous tests, but the repair gain remains concentrated in the low-beta groups. Thus, Rigorous means low observed false acceptance; gamma captures rejection of reference-test-passing mutants.

W2/Q5: Uncertainty in the Principal Comparisons

We aligned instance IDs and used all jointly evaluated instances for the stronger-backbone comparisons:

BackboneNBoth solvedCOHARDEN onlyAgent-CoEvo onlyNeitherCOHARDENAgent-CoEvoDifference (95% paired CI)
GPT-5.430020032214777.3%73.7%+3.7 points [-1.0, 8.3]
Claude Opus 4.527620419124180.8%78.3%+2.5 points [-1.4, 6.5]

These use instance-level paired bootstrap intervals. For the injection study, we resampled by issue while keeping the shared no-test baseline paired; 1,104 pairs cover 404 issues:

ComparisonF-to-P pairsResolved difference95% issue-clustered CI
Rigorous vs. no-test552+8.5 points[2.9, 14.3]
Lax vs. no-test5520.0 points[-4.1, 4.1]

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers and Area Chair

We thank the reviewers for their constructive feedback and for recognizing the following contributions:

  • MPE reveals a practically important dimension of BRT quality that F-to-P alone does not capture (Bx9p, RoHj, nBtz).
  • The test-injection study connects BRT quality to downstream repair utility: Rigorous tests improve Resolved by 8.5 points, while Lax tests provide no gain (Bx9p, RoHj, nBtz).
  • COHARDEN provides a concrete inference-time procedure that uses BRT quality to iteratively refine tests and fixes. The reviewers also recognized its broad analysis across backbones, benchmark subsets, ablations, mutation generators, costs, and failure modes (Bx9p, RoHj, nBtz).

During the rebuttal, we added the following experiments and analyses. These additions and clarifications are incorporated into the revised manuscript, with revisions highlighted in blue.

  1. Matched mutation-feedback control (W2/Q4 of Bx9p; W2/Q3 of RoHj and nBtz).

    We evaluated generic refinement and COHARDEN on the same 100-instance GPT-5.4 subset with the same lax-init and five-round budget. COHARDEN reaches 76% Resolved and 69% Test F-to-P, compared with 71% and 64% under generic feedback. Both differences are +5 points, with paired bootstrap intervals of [-3, 13] and [-4, 14], respectively. This control more directly separates mutation-guided hardening from additional generic refinement rounds.

  2. Generalization to recent issues (W1/Q1 of RoHj).

    We evaluated COHARDEN and Vanilla cogeneration on 100 paired SWE-bench-Live instances from 53 repositories, dated June to August 2025, using the same GPT-5.4 backbone, inference budget, and official evaluator. COHARDEN resolves 45 instances and Vanilla resolves 44; the paired outcomes are 39 both, six COHARDEN-only, five Vanilla-only, and 50 neither. The difference is +1 point with a 95% paired bootstrap interval of [-5, 8]. The evaluator receives only source patches, excluding generated reproduction tests. Golden tests and fixes are used only as offline evaluation oracles and never appear in COHARDEN's inference context.

  3. MPE validity and overconstraint analysis (W2/Q2 of Bx9p; W2/Q4 of nBtz).

    Appendix E.1 uses one method-agnostic mutant pool per instance, generated without any evaluated test in context and shared across methods and rounds. We further stratified all 1,104 F-to-P pairs by both beta and gamma. Resolved gains are +9.4 points for low-beta/low-gamma tests and +7.5 points for low-beta/high-gamma tests, compared with -1.5 and +1.4 points in the high-beta groups. The repair gain remains concentrated in low-beta tests even when gamma is high. We clarified that Rigorous is an operational low-false-acceptance label, while gamma separately measures rejection of reference-test-passing mutants.

  4. Uncertainty estimates (W2/Q4 of Bx9p; W2/Q5 of nBtz).

    On the main 433-instance comparison, COHARDEN improves Resolved over OpenHands+Cogen by 7.9 points, with a 95% paired interval of [3.9, 12.0]. On matched stronger-backbone subsets, its gains over Agent-CoEvo are +3.7 points [-1.0, 8.3] with GPT-5.4 and +2.5 points [-1.4, 6.5] with Claude Opus 4.5. For the injection study, issue-clustered resampling gives +8.5 points [2.9, 14.3] for Rigorous tests and 0.0 points [-4.1, 4.1] for Lax tests.

  5. Result provenance and algorithm clarification (W1 of Bx9p and nBtz; W2 of RoHj).

    We reconciled the supplementary analyses with the canonical 433-instance evaluation. The final results count the timed-out instance as unsuccessful: 300/433 instances are Resolved (69.3%), and 341/433 have F-to-P tests (78.8%). We corrected the appendix tables, figures, failure analysis, ablation arithmetic, and Table 13 aggregation, and documented Table 8's round-specific effective denominators.

    We also unified the stopping and rollback descriptions across Section 3.2, Figure 3, and Appendix D.2. In the final evaluation, 59 instances trigger rollback, 61 reach the round limit, and two encounter zero-admissible-mutant rounds.

Together, these additions strengthen the evidence that MPE captures a repair-relevant dimension of BRT quality and that COHARDEN can use this signal during inference to improve the complete test-fix refinement procedure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.