acceptodds
Under review as a conference paper at ICLR 2027

SWEPATCHTEST: Evaluating Software Tests Beyond Reproduction

11 views

Abstract

Coding agents use generated tests to decide whether a proposed patch satisfies an issue’s requirements. A suite that fails on the buggy program and passes after applying the reference patch can still accept an incorrect patch or reject a correct patch. We introduce SWEPATCHTEST1, a test-generation benchmark with 619 tasks from SWE-bench Pro, SWE-bench Multilingual, and DeepSWE and 82,579 candidate patches collected from coding-agent trajectories. By replaying each suite on the buggy and patched codebases, SWEPATCHTEST jointly measures issue reproduction, acceptance of correct patches, and rejection of incorrect patches. Joint Validation Rate (JVR), which requires all three criteria on the same task, exposes failures hidden by single-reference evaluation: the best-performing configuration achieves only 9.21% JVR despite reproducing the reference transition on 59.29% of tasks. We further use validation scores to select and scale demonstrations for supervised fine-tuning (SFT). The resulting training recipe approximately triples JVR and improves downstream repair across three SWE-bench benchmarks. Patch discrimination thus provides both a diagnostic measure of test quality and an execution-based signal for curating agent training data

Primary area:
datasets and benchmarks
Keywords:
test generation, software engineering, coding agents, language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2FQqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper makes several contributions. First, the authors show that model-generated tests are often unreliable at distinguishing correct and incorrect patches, as demonstrated by low Joint Validation Rate (JVR), a metric introduced by the authors. Second, they propose a curated benchmark to evaluate test-generation models. Finally, they use TestGen trajectories to fine-tune a model and show that this can improve both JVR and downstream repair success.

Critical strengths
  • I think the authors have identified an important issue with test overfitting. The motivation and definition of JVR seem solid to me.

  • The motivation and process behind the benchmark construction seem reasonable to me. However, the authors did not introduce entirely new issue instances. Instead, they combined instances from existing benchmarks and constructed candidate-patch pools for evaluation.

  • The authors have fine-tuned a model using test-generation trajectories. I am not aware of any prior work that follows exactly this fine-tuning strategy using test-generation trajectories. The authors also report improvements from the proposed technique, which is interesting.

Critical weaknesses
  • The novelty of this work is relatively low. First of all, test overfitting is not a new problem. It is a well-known issue in the test-generation community, and the authors have cited a few relevant works. The framing of JVR is relatively new. However, I do not find it fundamentally novel, as prior work has already explored evaluating tests against different patches and identifying test overfitting [1-3]. The authors have introduced several new metrics, which are intuitive but do not seem particularly novel to me. While the benchmark construction is useful, I am not convinced that the overall methodological contribution is sufficiently novel.

  • Although the authors have done a good job with the metrics and benchmark, I am not fully convinced by the empirical results. First of all, JVR remains very low even after fine-tuning (3.88%). While this supports the authors' motivation that reliable test generation is challenging, it also raises questions about the practical effectiveness of the proposed fine-tuning approach. The authors report improvements over the baseline, but the absolute JVR remains low. Therefore, I am not convinced that the authors have achieved sufficiently meaningful improvements in test-generation reliability.

  • The authors have also reported downstream repair success using AVG@4 and Pass@4. I am not sure why the authors chose a four-attempt evaluation rather than also reporting Pass@1. Although the results show improvements over the SWE SFT baseline, the additional improvements from validation-guided selection over GRR-based selection are relatively small, particularly on SWE-bench Pro. This raises questions about how much of the improvement comes from the proposed selection strategy rather than simply adding more TestGen training data. Overall, I find the evidence for the effectiveness of the proposed fine-tuning strategy somewhat limited.

  • The weakest aspects of the paper are its writing, lack of methodological clarity, and missing baselines. I have some serious concerns about how the different experiments were actually performed.

  1. TestGen implementation: The paper does not clearly explain how the TestGen agents were designed or configured. I could not find sufficient descriptions of the prompts and implementation details needed to reproduce the results. That said, there are several existing reproduction-test generation techniques, but the authors did not provide a direct comparison with these approaches under comparable settings. Although the paper evaluates multiple coding-agent configurations, I believe comparisons with established reproduction-test generation approaches are important. This weakens the generality of the conclusions drawn from low JVR. Some existing approaches might achieve better JVR, and it is unclear how the proposed fine-tuned model would perform against them.

  2. Integration of TestGen into the repair pipeline: It is not sufficiently clear to me how TestGen capabilities are actually used in the downstream repair pipeline. The authors explain that TestGen trajectories are incorporated during fine-tuning, but do the repair agents also generate and execute tests during inference? If so, how is this done? I believe an important baseline is missing here. For example, generating reproduction tests through naïve prompting and providing them to the repair agent might produce similar improvements without additional fine-tuning. If the generated tests are not explicitly used during repair, a comparison against additional repair-only SFT data with a matched training budget would be useful. Without these comparisons, it is difficult to isolate the actual contribution of fine-tuning with TestGen trajectories.

  3. Training-data selection: Finally, it is not sufficiently clear how the training data were selected. The authors mention using their proposed metrics for the selection process, but I could not find a sufficiently detailed description of the exact selection procedure, including how the metrics were combined and what criteria were used to retain trajectories. Since validation-guided data selection is one of the main contributions, these details are important for reproducibility.

Although the paper starts with a good motivation and introduces a potentially useful benchmark, I am not convinced that it demonstrates sufficiently meaningful methodological or empirical contributions. In particular, the limited novelty, missing baseline comparisons, and insufficient methodological details weaken the overall contribution. These concerns prevent me from recommending acceptance of this paper.

[1] Edward K. Smith, Earl T. Barr, Claire Le Goues, and Yuriy Brun. Is the cure worse than the disease? overfitting in automated program repair. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE′15, pp. 532–543. ACM, August 2015. doi: 10.1145/2786805.2786825. URL http://dx.doi.org/10.1145/2786805.2786825.

[2] Ahmed, T., Ganhotra, J., Shinnar, A., & Hirzel, M. (2026, July). Investigating Test Overfitting on SWE-bench. In Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering (pp. 1292-1296).

[3] Wang, Y., Pradel, M., & Liu, Z. (2026, April). Are “solved issues” in SWE-bench really solved correctly? An empirical study. In Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering (pp. 169-181).

Questions for authors
  • Why did you not compare against established reproduction-test generation approaches, such as those represented on the SWT-bench leaderboard? How would these approaches perform under the proposed JVR evaluation?

  • How are TestGen capabilities actually incorporated into the downstream repair pipeline? Are tests generated and executed during repair, or are TestGen trajectories used only during fine-tuning? What baselines isolate the benefit of this supervision?

  • What is the rationale behind reporting AVG@4 and Pass@4 without also reporting Pass@1 for the repair task?

5 comments

We thank the reviewer for examining what the benchmark adds to earlier work and for identifying details needed to reproduce the experiments. We group related comments with their corresponding questions and respond to each below.

Q1 The novelty of this work is relatively low. First of all, test overfitting is not a new problem. It is a well-known issue in the test-generation community, and the authors have cited a few relevant works. The framing of JVR is relatively new. However, I do not find it fundamentally novel, as prior work has already explored evaluating tests against different patches and identifying test overfitting [1-3]. The authors have introduced several new metrics, which are intuitive but do not seem particularly novel to me. While the benchmark construction is useful, I am not convinced that the overall methodological contribution is sufficiently novel. Although the paper starts with a good motivation and introduces a potentially useful benchmark, I am not convinced that it demonstrates sufficiently meaningful methodological or empirical contributions. In particular, the limited novelty, missing baseline comparisons, and insufficient methodological details weaken the overall contribution. These concerns prevent me from recommending acceptance of this paper.

A1 We agree that test overfitting and evaluation across patches have been studied in prior work. Earlier studies established that a patch can pass available tests while leaving the bug unresolved. SWEPatchTest asks whether a generated test can reliably judge several repairs for the same repository issue. SWT-Bench checks reproduction against the reference fix, and SWE-Mutation evaluates detection of faulty changes. Our benchmark also includes alternative repairs that pass the official tests, allowing us to measure whether a generated test rejects a working alternative. This adds an empirically important dimension: in Table 2, GPT-5.6-sol detects 88.57% of officially failing patches on average but accepts only 50.20% of officially passing ones. Measuring both outcomes gives a fuller account of a test's usefulness for judging repairs. We will clarify this contribution and its relationship to prior work in the revised paper.


Q2 Although the authors have done a good job with the metrics and benchmark, I am not fully convinced by the empirical results. First of all, JVR remains very low even after fine-tuning (3.88%). While this supports the authors' motivation that reliable test generation is challenging, it also raises questions about the practical effectiveness of the proposed fine-tuning approach. The authors report improvements over the baseline, but the absolute JVR remains low. Therefore, I am not convinced that the authors have achieved sufficiently meaningful improvements in test-generation reliability.

A2 We agree that the absolute reliability of generated tests is important when assessing the training results. Table 3 shows the clearest improvement in detecting faulty patches: at the same training budget, validation-guided selection increases the share of tasks on which every faulty patch is detected from 13.73% to 25.69%, compared with selection using GRR. Under the same comparison, fully validated tasks increase from 14 to 20 out of 619. With the larger training set, JVR reaches 3.88%, compared with 1.29% for training on repair data alone; Appendix E.2 reports a paired 95% confidence interval of 1.29 to 4.04 percentage points for this gain. These results show measurable improvements in test generation while identifying complete validation of every candidate repair as a continuing challenge. We will make the distinction between these gains and absolute test reliability clearer in the revised paper.

Q3 The weakest aspects of the paper are its writing, lack of methodological clarity, and missing baselines. I have some serious concerns about how the different experiments were actually performed. TestGen implementation: The paper does not clearly explain how the TestGen agents were designed or configured. I could not find sufficient descriptions of the prompts and implementation details needed to reproduce the results.

A3 We agree that the TestGen setup should be described in enough detail to reproduce the experiments. Each agent receives the issue description and a repository checkout containing the bug. Appendix C.3 lists the model and agent software: GPT-5.6-sol runs in Codex, and Claude-Fable-5.1 and Claude-Opus-5 use Claude Code. Kimi-K3 uses Kimi Code, Qwen3.8-max uses Qwen Code, and GLM-5.3 uses ZCode. All DeepSeek-V4 configurations use OpenHands, as do the direct TestGen and downstream repair evaluations of the four SFT configurations. The prompt asks the agent to write an executable test and confirm that it fails on the original code because of the behavior described in the issue. An agent may try a temporary fix to check its test, then must revert the fix and reconfirm the failure before submission.

As described in Appendix C.1, we collect /tmp/run_verify_task.sh and its supporting files directly from the tmp directory for that task. Direct extraction preserves the test package without reconstructing it from a Git diff, modifying existing repository files, or conflicting with the official test patch. We run the same package unchanged on clean copies of the buggy code, the reference fix, and every candidate patch. The environment is reset after each execution to prevent fixtures, caches, or other side effects from affecting later runs. Execution records contain the command, exit code, timeout status, failing assertions, and harness output. They distinguish a pass or stable behavioral failure from a timeout, infrastructure failure, or missing artifact. We replay inconsistent executions; a patch whose execution remains unstable counts as neither accepted nor rejected and receives no credit. Metrics are averaged equally across tasks, with unresolved or incomplete runs retained in the denominator at zero. We will include the full prompt, a table of models, agent frameworks and inference settings, and these execution and scoring details in the revised paper.


Q4 That said, there are several existing reproduction-test generation techniques, but the authors did not provide a direct comparison with these approaches under comparable settings. Although the paper evaluates multiple coding-agent configurations, I believe comparisons with established reproduction-test generation approaches are important. This weakens the generality of the conclusions drawn from low JVR. Some existing approaches might achieve better JVR, and it is unclear how the proposed fine-tuned model would perform against them. Why did you not compare against established reproduction-test generation approaches, such as those represented on the SWT-bench leaderboard? How would these approaches perform under the proposed JVR evaluation?

A4 We agree that the connection to established reproduction test methods deserves a clearer explanation. SWT-Bench checks whether a generated test fails on the buggy code and passes with the reference fix. This corresponds to GRR in SWEPatchTest, which Table 2 reports for every configuration. SWEPatchTest extends the check to alternative repairs that pass the official tests and repairs that fail them, evaluating whether a test recognizes both working alternatives and incomplete fixes. This requires coverage of the issue's behavior and relevant boundaries while accommodating different correct implementations. GPT-5.6-sol achieves 59.29% GRR but 9.21% JVR, and the Caddy case illustrates this gap. Its generated test fails on the buggy code and passes with the reference fix, yet accepts officially incorrect patch #65 because it omits address order. It also rejects officially correct patch #13 by requiring an error for an empty default_bind, which neither the issue nor the official oracle requires. The case demonstrates how a successful reproduction test can still misjudge alternative repairs. Executable test packages from existing SWT-Bench methods can be evaluated with the same harness and candidate pools while retaining their original generation procedure. We will clarify this relationship and expand the accompanying case analysis in the revised paper.

Q5 Integration of TestGen into the repair pipeline: It is not sufficiently clear to me how TestGen capabilities are actually used in the downstream repair pipeline. The authors explain that TestGen trajectories are incorporated during fine-tuning, but do the repair agents also generate and execute tests during inference? If so, how is this done? I believe an important baseline is missing here. For example, generating reproduction tests through naïve prompting and providing them to the repair agent might produce similar improvements without additional fine-tuning. If the generated tests are not explicitly used during repair, a comparison against additional repair-only SFT data with a matched training budget would be useful. Without these comparisons, it is difficult to isolate the actual contribution of fine-tuning with TestGen trajectories. How are TestGen capabilities actually incorporated into the downstream repair pipeline? Are tests generated and executed during repair, or are TestGen trajectories used only during fine-tuning? What baselines isolate the benefit of this supervision?

A5 We agree that the role of TestGen during training and repair inference should be explicit. We use complete DeepSeek-V4-Flash sessions that write and check tests as supervision during fine tuning. At evaluation, every configuration uses the same OpenHands repair setup, where the trained agent solves the issue without receiving a suite from a separate test generator. Table 3 compares the two TestGen selection rules using the same source pool and training budget. On SWE-bench Multilingual, AVG@4 is 55.08% for SWE SFT, 62.05% for TestGen selected using GRR, and 63.02% for Validation-guided TestGen; their SWEPatchTest JVR scores are 1.29%, 2.26%, and 3.23%, respectively. We added a comparison using the same base model and training settings, expanding repair data from 2B to 4B tokens to match the 4B budget of the two TestGen configurations, each with 2B repair and 2B TestGen tokens. For the following scores, benchmarks are ordered Verified, Multilingual, and Pro, and each pair gives AVG@4 and Pass@4 in percent. Additional repair training scores (70.03, 79.20), (61.87, 74.67), and (48.59, 59.91), while Validation-guided TestGen scores (70.24, 80.60), (63.02, 76.33), and (49.18, 60.68). Its gains are therefore 0.21, 1.15, and 0.59 percentage points in AVG@4 and 1.40, 1.66, and 0.77 points in Pass@4. TestGen selected using GRR scores (69.47, 80.00), (62.05, 76.25), and (48.97, 60.47), including a Verified AVG@4 score 0.56 points below additional repair training. These results show that the benefit depends on how the TestGen data are selected.

Analysis of SWE-bench Pro repair trajectories also shows a change in test use during inference. Compared with SWE SFT, Validation-guided TestGen increases average unit test script executions from 2.7 to 3.4 per trajectory. Reproduction scripts appear in 81.2% of trajectories, up from 56.2%, and average writing operations for those scripts increase from 7.55 to 11.24. Average test file viewing or search operations rise from 3.61 to 4.20, while trajectories with no test interaction fall from 32.1% to 9.0%. This indicates that TestGen supervision encourages the repair agent to use tests when understanding issues and validating fixes. We will clarify this integration and include the additional repair baseline and trajectory analysis in the revised paper.

Reviewer_HiPhOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a new agentic benchmark, SWEPatchTest, for evaluating the quality of generated tests by coding agents. Given an issue and a codebase, a TestGen agent has to write a test suite that effectively addresses the issue. Specifically, the test suite should fail on the original buggy codebase but pass when the reference gold patch is applied. More over, for each task, the benchmark provides both a list of alternative correct patches and incorrect patches that is used to validate the test suite. SWEPatchTest covers 619 tasks and 82.5k candidate patches. The authors define three properties: whether generated tests reproduce the original bug (GRR), accept alternative correct solutions (CAR), and reject incorrect solutions (WDR). The authors introduce Joint Validation Rate (JVR), which requires all three properties to hold on the same task, and show that conventional single-reference evaluation overestimates test quality, e.g., the best configuration achieves 59.29% GRR but only 9.21% JVR. Finally, the authors use these richer patch-discrimination signals to select test-generation trajectories for supervised fine-tuning and report improvements in generated-test quality and downstream software repair tasks.

Critical strengths

This papers highlights an important shortcoming of relying heavily on coding agent's generated tests to discriminate among candidates patches. For instance, a coding agent can generate tests that look quite good if evaluated against individual wrong patches but fail badly if the test must provide a completely correct validation decision over the task.

This is best shown by the Caddy example in the paper (Figure 3) where a generated test:

  • passes the reference transition,
  • accepts an officially incorrect candidate because it normalizes away an ordering distinction,
  • rejects an officially correct candidate because it imposes behavior not required by the issue/oracle.
Critical weaknesses

A weakness of SWEPatchTest is the correct/incorrect labels are only as good as the original oracle. A candidate is called correct if it passes the official evaluation and incorrect if it fails it, but as the authors alluded to, tests are imperfect proxies for specification (even if written by humans). While the author relied on LLM audit (and manual review when needed) of the official tests, there is still a risk of mislabeling due to the inherent limitations of the oracle.

Questions for authors
  • Unclear to me, if the TestGen agent tries to fix the issue while coming up with the test suite? Something is might be easier to come up with a fix and derive the tests from it.
  • Are the incremental downstream gains of validation-guided selection statistically reliable?
  • How sensitive is JVR to number/diversity of candidate patches per task?
4 comments
AuthorsOpenReview ↗

Response to Reviewer HiPh

We thank the reviewer for examining how the official oracle and the choice of candidate patches affect JVR. We address the labeling concern first and then respond to the three questions in order.

Q1 A weakness of SWEPatchTest is the correct/incorrect labels are only as good as the original oracle. A candidate is called correct if it passes the official evaluation and incorrect if it fails it, but as the authors alluded to, tests are imperfect proxies for specification (even if written by humans). While the author relied on LLM audit (and manual review when needed) of the official tests, there is still a risk of mislabeling due to the inherent limitations of the oracle.

A1 We agree that the quality of the official tests affects the interpretation of patch labels. Our labels follow official test outcomes. We excluded tasks whose tests missed core behavior or depended too closely on one implementation, then audited disagreements using tests generated by GPT-5.6-sol and Claude-Opus-5. An automated judge flagged 39 of 619 tasks (6.3%). Two authors reviewed all 39 cases and 40 unflagged controls without seeing the judge's decisions; in about 30 flagged tasks, they found that the generated test better reflected behavior required or reasonably implied by the issue. The other 580 tasks were not flagged by the screen. In Powertools, for example, the issue requires sourceIp to preserve the original string, including its port. The generated test expects 10.1.15.242:1234 and passes on the gold patch, but 13 of the 16 officially passing candidates produce IPv4Network('10.1.15.242/32') and lose the port. The generated test correctly rejects these candidates, yielding a CAR of 3/16 under the official labels. The Redis example concerns a different situation: the official tests check that successful calls appear in the monitor, while the generated test also expects failed calls to appear, a behavior the issue leaves open. That plausible interpretation lowers CAR when it rejects a patch accepted by the official tests. These cases distinguish omissions in official coverage from ambiguity in the specification. We retain the upstream labels for reproducibility and comparability, and will add the full audit and clarify these distinctions in the revised paper.


Q2 Unclear to me, if the TestGen agent tries to fix the issue while coming up with the test suite? Something is might be easier to come up with a fix and derive the tests from it.

A2 Yes, the TestGen agent may try a temporary fix to check that its test passes when the bug is resolved. Before submission, it must revert the fix and rerun the test to confirm that it still fails on the original code for the reason described in the issue. We collect only the test script and its supporting files, then execute the same test on clean copies of the repository with each patch applied, as described in Appendix C.1. We will make this procedure for temporary fixes and subsequent evaluation clearer in the revised paper.

Q3 Are the incremental downstream gains of validation-guided selection statistically reliable? A3 We agree that the training comparisons should be supported by paired statistical analysis. We computed 95% confidence intervals from 10,000 bootstrap replicates, resampling tasks within each benchmark and applying the same task indices to both configurations. AVG@4 averages the four round resolve rates, and Pass@4 records whether each task succeeds in any round. We use paired permutation tests for AVG@4 and exact McNemar tests with a two sided alternative for Pass@4, applying Holm correction separately to the nine comparisons for each metric. The strongest overall result is the AVG@4 improvement from Scaled TestGen Supervision over SWE SFT, which remains significant after Holm correction on all three benchmarks. For its Pass@4 gains, the Verified interval is entirely above zero and the Multilingual gain remains significant after correction. The direct filtering comparison has positive point estimates with intervals spanning zero. The tables report gains in percentage points relative to the named baseline; TestGen Supervision denotes selection using GRR.

ComparisonBenchmarkAVG@4 gain, pp [95% CI]Holm p (permutation)
Scaled TestGen Supervision vs. SWE SFTVerified+4.31 [2.67, 5.98]<0.001
Scaled TestGen Supervision vs. SWE SFTMultilingual+8.22 [5.48, 11.01]<0.001
Scaled TestGen Supervision vs. SWE SFTPro+4.14 [2.55, 5.78]<0.001
TestGen Supervision vs. SWE SFTVerified+1.94 [0.40, 3.65]0.071
TestGen Supervision vs. SWE SFTMultilingual+6.97 [3.08, 8.55]<0.001
TestGen Supervision vs. SWE SFTPro+2.41 [0.63, 3.83]0.062
Validation-guided TestGen vs. TestGen SupervisionVerified+0.77 [-0.96, 2.30]0.802
Validation-guided TestGen vs. TestGen SupervisionMultilingual+0.97 [-0.28, 4.60]0.259
Validation-guided TestGen vs. TestGen SupervisionPro+0.21 [-1.00, 1.77]0.802
ComparisonBenchmarkPass@4 gain, pp [95% CI]Holm p (McNemar)
Scaled TestGen Supervision vs. SWE SFTVerified+2.80 [0.60, 5.20]0.231
Scaled TestGen Supervision vs. SWE SFTMultilingual+6.00 [2.33, 9.67]0.026
Scaled TestGen Supervision vs. SWE SFTPro+1.65 [-0.55, 3.97]1.000
Validation-guided TestGen vs. TestGen SupervisionVerified+0.60 [-1.40, 2.80]1.000
Validation-guided TestGen vs. TestGen SupervisionMultilingual+0.08 [-2.67, 3.33]1.000
Validation-guided TestGen vs. TestGen SupervisionPro+0.21 [-1.78, 2.05]1.000

We also compared additional repair training with Validation-guided TestGen using the same base model, training settings, and 4B token budget: 4B repair tokens versus 2B repair plus 2B TestGen tokens. For Verified, Multilingual, and Pro, the additional repair baseline achieves (AVG@4, Pass@4) scores of (70.03, 79.20), (61.87, 74.67), and (48.59, 59.91)%; Validation-guided TestGen achieves (70.24, 80.60), (63.02, 76.33), and (49.18, 60.68)%. The corresponding gains are 0.21, 1.15, and 0.59 points in AVG@4 and 1.40, 1.66, and 0.77 points in Pass@4. A separate selection ablation downsamples the main TestGen pool and uses the same base model with 1B repair tokens and 0.33B TestGen tokens per configuration. Both require GRR = 1; Validation-guided TestGen additionally requires WDR and CAR each to be at least 0.8. This stricter filter increases SWEPatchTest (GRR, WDR, CAR, JVR) from (13.09, 39.29, 25.88, 0.32)% to (15.35, 42.29, 27.13, 0.65)%. In the completed evaluation with one attempt per task, Pass@1 rises from 59.00% to 64.59% on Verified, from 56.90% to 57.58% on Multilingual, and from 30.68% to 36.76% on Pro, gains of 5.59, 0.68, and 6.08 points at the same TestGen volume. Paired exact McNemar tests against training on repair data alone give gains of 3.81 points on Verified and 6.00 on Multilingual with unadjusted p < 0.05, and 8.38 points on Pro with a 95% CI of [5.49, 11.26] and Holm p < 0.001 after correction over six comparisons. Only the filtered configuration achieves a significant gain over repair training alone after this correction, on Pro. These results establish robust overall repair gains from scaled TestGen supervision and show positive trends from validation selection at both tested scales. We will include the paired statistical analysis, the additional repair baseline, and the selection ablation in the revised paper.

Q4 How sensitive is JVR to number/diversity of candidate patches per task?

A4 We agree that the sensitivity of JVR to the candidate pool should be quantified. We evaluated the eight models in Table 2 and the four SFT configurations using random subsets of 96, 64, 32, 16, 8, or 4 patches per task. Each size was sampled 100 times, with all models evaluated on the same subset within each draw, allowing us to examine both pool size and changes in composition at a fixed size. Because JVR requires every candidate verdict to be correct, reducing the pool relaxes this condition and raises the score. The tables report JVR (%) for the eight models and the four SFT configurations. Sampled results show mean ± standard deviation over 100 draws, with rank in parentheses; the full pool results are fixed. Results at 16 and 64 candidates follow the same trend.

ModelFull pool963284
GPT-5.6-sol9.21 (1)10.61 ± 0.37 (1)16.53 ± 0.73 (1)27.44 ± 0.96 (1)34.54 ± 1.10 (3)
Claude-Fable-5.17.92 (2)9.17 ± 0.33 (2)14.46 ± 0.73 (2)26.49 ± 1.06 (3)35.28 ± 1.26 (2)
Claude-Opus-57.11 (3)8.21 ± 0.38 (3)14.23 ± 0.75 (3)27.07 ± 1.18 (2)36.53 ± 1.12 (1)
Kimi-K35.98 (4)6.81 ± 0.30 (4)9.83 ± 0.55 (5)15.94 ± 0.72 (5)19.78 ± 0.79 (7)
Qwen3.8-max5.17 (5)6.06 ± 0.29 (5)10.00 ± 0.61 (4)19.42 ± 0.92 (4)26.38 ± 1.02 (4)
DeepSeek-V4-Pro4.52 (6)5.36 ± 0.28 (6)8.45 ± 0.54 (6)15.46 ± 0.78 (6)20.23 ± 0.90 (6)
DeepSeek-V4-Flash3.88 (7)4.25 ± 0.21 (7)6.39 ± 0.52 (8)11.94 ± 0.67 (8)16.01 ± 0.81 (8)
GLM-5.33.72 (8)4.13 ± 0.24 (8)6.89 ± 0.50 (7)14.70 ± 0.81 (7)20.47 ± 0.99 (5)
SFT configurationFull pool963284
SWE SFT1.29 (4)1.43 ± 0.13 (4)2.18 ± 0.28 (4)3.51 ± 0.32 (4)4.37 ± 0.38 (4)
TestGen Supervision2.26 (3)2.89 ± 0.25 (3)4.65 ± 0.44 (3)10.15 ± 0.69 (3)14.01 ± 0.70 (3)
Validation-guided TestGen3.23 (2)3.97 ± 0.26 (2)6.60 ± 0.48 (2)13.41 ± 0.79 (2)18.28 ± 0.95 (2)
Scaled TestGen Supervision3.88 (1)4.76 ± 0.29 (1)7.81 ± 0.52 (1)14.34 ± 0.73 (1)18.83 ± 0.76 (1)

Spearman correlations between the mean rankings and the full pool ranking are 1.000, 0.976, 0.952, 0.929, 0.929, and 0.667 for 96, 64, 32, 16, 8, and 4 candidates, respectively. On the full pool, GPT-5.6-sol, Claude-Fable-5.1, and Claude-Opus-5 form the leading tier, followed by Kimi-K3 and Qwen3.8-max, then DeepSeek-V4-Pro, DeepSeek-V4-Flash, and GLM-5.3. These broad tiers remain distinguishable with large and medium pools, with ranking changes mainly within tiers or near their boundaries. Candidate composition matters more for close comparisons: at eight candidates, GPT-5.6-sol leads Claude-Opus-5 in 54 draws, ties in five, and trails in 41, despite mean JVRs of 27.44 ± 0.96% and 27.07 ± 1.18%. At four candidates, Claude-Opus-5 ranks first alone in 81 draws, compared with four for GPT-5.6-sol. GLM-5.3 moves from eighth on the full pool to fifth at four candidates, consistent with its generally narrower test coverage, since smaller samples are more likely to omit patches that expose missed behavior. The four SFT configurations retain their mean ranking at every tested size. Thus, pool size and composition affect absolute JVR and the ordering of models with close scores, especially for very small pools, while the ordering of the training configurations remains stable. We retain the full pool for the main leaderboard and will include this sensitivity analysis, including the complete results at 16 and 64 candidates, in the revised paper.

Thank you for the response. After "reading" other reviewers responses, I would advise avoiding verbose text with dense information especially mixed with numbers in the future.

Additional analysis does improve the paper quite a bit. It is interesting to see that official labels can be wrong, sometimes substantially so. Retain the upstream labels for reproducibility and comparability is a valid approach as long as it is clearly mentioned in the paper (as you committed to).

I still find the empirical finding compelling: tests that reproduce a bug and pass on the reference fix can still be unreliable for evaluating alternative repairs. I will maintain my positive score.

Reviewer_SZ1XOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SWEPatchTest, a benchmark for evaluating whether generated tests can reproduce issues, accept correct patches, and reject incorrect patches. It evaluates test suites against diverse candidate patches and shows that successful reproduction on a reference patch does not necessarily imply reliable patch discrimination. The paper also explores using validation scores to select test-generation trajectories for supervised fine-tuning.

Critical strengths
  1. The paper addresses an important limitation of reference-based test evaluation. Measuring both incorrect-patch detection and correct-patch acceptance provides a more comprehensive assessment of generated tests.
  2. The benchmark and candidate-patch collection are potentially useful resources for studying coding-agent reliability.
  3. The experiments expose substantial weaknesses in current test-generation agents, motivating further work on tests that generalize across different repair implementations.
Critical weaknesses
  1. The significance of the benchmark contribution needs stronger justification. While evaluating generated tests beyond reference-patch reproduction is well motivated, the paper does not yet convincingly establish the broader scientific insights enabled by this benchmark beyond identifying limitations of the evaluated systems. I would like to see stronger evidence of its value relative to existing test-generation evaluations, particularly through systematic failure analysis and robustness studies. In its current form, the work appears more developed as an evaluation and data-curation pipeline than as a sufficiently validated research contribution, especially the benchmark is filtered from the existing benchmarks.
  2. The benchmark needs a more systematic analysis of disagreements with the official oracle. The paper describes the construction pipeline and provides a qualitative case study, but these do not establish how frequently disagreements arise from weaknesses in generated tests versus limitations of the official tests. For example, a generated suite may miss required behavior covered by the official tests, impose an unjustified restriction, or reveal a genuine defect that the official tests overlook. An independently audited sample separating these cases would substantially strengthen the interpretation of CAR, WDR, and JVR as measures of test quality.
  3. The training results do not establish an advantage over additional repair supervision. At the same TestGen data budget, validation-guided selection improves downstream AVG@4 over GRR-based selection by 0.77, 0.97, and 0.21 percentage points on SWE-bench Verified, Multilingual, and Pro, respectively. JVR increases from 2.26% to 3.23%, corresponding to a net increase from 14 to 20 fully validated tasks out of 619. These gains are positive but modest, and their practical significance remains unclear given the additional candidate-pool construction and execution costs. Moreover, the experiments already combine TestGen trajectories with repair trajectories, but do not compare against adding an equivalent budget of further repair trajectories. Consequently, they do not establish the comparative advantage of allocating that budget to TestGen supervision. Since the benchmark is the central contribution, I would prefer stronger validation of the benchmark itself over broader training claims that remain insufficiently isolated.
Questions for authors

Please see the weakness. If the authors provide a systematic comparison with the official tests that distinguishes errors in generated tests from gaps in the official oracle, I would consider raising my rating.

3 comments
AuthorsOpenReview ↗

Response to Reviewer SZ1X

We thank the reviewer for asking what SWEPatchTest reveals about generated tests and how its training results should be interpreted. We address the benchmark evidence first, followed by the oracle audit and the comparison with repair supervision.

Q1 The significance of the benchmark contribution needs stronger justification. While evaluating generated tests beyond reference-patch reproduction is well motivated, the paper does not yet convincingly establish the broader scientific insights enabled by this benchmark beyond identifying limitations of the evaluated systems. I would like to see stronger evidence of its value relative to existing test-generation evaluations, particularly through systematic failure analysis and robustness studies. In its current form, the work appears more developed as an evaluation and data-curation pipeline than as a sufficiently validated research contribution, especially the benchmark is filtered from the existing benchmarks.

A1 We agree that the benchmark's value should be established through the insights it enables. A test that fails before the reference fix and passes afterward may still accept an incomplete repair or reject another working fix. SWEPatchTest evaluates both possibilities on the same issue. Writing such a test can be harder than producing a repair because it must capture the required behavior and relevant boundaries while accommodating different correct implementations.

(1) Evidence beyond reference-patch reproduction. Table 2 shows the gap between reproducing a bug and correctly evaluating every candidate patch:

ModelTasks with bug reproductionTasks with the expected verdict on every candidate patch
GPT-5.6-sol59.29%9.21%

SWT-Bench primarily measures the transition from failure on the buggy code to success on the reference fix, corresponding to GRR. Mutation evaluations depend on how mutants are constructed and how difficult they are to detect; simple rule changes may be caught by generic assertions that cover little of the issue's behavior. Evaluating a fixed pool of officially accepted alternatives and failed repairs connects bug reproduction to the ability to accept valid repairs and reject faulty ones.

(2) Systematic failure analysis. Our audit provides evidence of what the reference check misses:

Audit statisticCount
Models audited2
Combinations of model and task flagged by the judge140
Cases with missing required behavior48
Cases with requirements beyond those stated in the issue41

The audit further distinguishes weaknesses in generated tests from omissions or ambiguities in the official oracle.

(3) Candidate-pool sensitivity. Candidate selection affects the average model ranking:

Patches per issueAverage model ranking relative to the full benchmark
96Matched
4Changed

We will clarify these empirical contributions and incorporate the audit and candidate-pool sensitivity analysis into the revised paper.

Q2 The benchmark needs a more systematic analysis of disagreements with the official oracle. The paper describes the construction pipeline and provides a qualitative case study, but these do not establish how frequently disagreements arise from weaknesses in generated tests versus limitations of the official tests. For example, a generated suite may miss required behavior covered by the official tests, impose an unjustified restriction, or reveal a genuine defect that the official tests overlook. An independently audited sample separating these cases would substantially strengthen the interpretation of CAR, WDR, and JVR as measures of test quality. Please see the weakness. If the authors provide a systematic comparison with the official tests that distinguishes errors in generated tests from gaps in the official oracle, I would consider raising my rating.

A2 We agree that systematically examining disagreements is essential for interpreting the benchmark.

(1) Screening and manual audit. We screened all tasks using tests generated by GPT-5.6-sol and Claude-Opus-5. For each candidate patch with conflicting verdicts, an LLM judge assessed the disagreement using the issue, official test patch, generated test, candidate patch, and execution results.

SourceTasks screenedTasks flaggedFlag rate
SWE-bench Pro3423510.2%
SWE-bench Multilingual16742.4%
DeepSWE11000.0%
Total619396.3%

Two authors jointly reviewed all flagged tasks and unflagged controls without seeing the judge's decisions.

Audit statisticTasks
Not flagged by the judge580
Flagged tasks manually reviewed39
Unflagged controls manually reviewed40
Flagged tasks where the generated test better captured behavior required or reasonably implied by the issueAbout 30

Some cases remained ambiguous, including a ProtonMail issue that specified an email address count but left the wording for zero addresses open.

(2) Attributing disagreements. The following cases distinguish official coverage gaps, inconsistencies in the task specification, and errors in generated tests:

CaseRequirement and observed disagreementInterpretation
PowertoolsThe requirements explicitly preserve the original sourceIp string, including its port. The generated test expected 10.1.15.242:1234 and passed on the gold patch, but rejected 13 of 16 officially passing candidates that converted it to IPv4Network('10.1.15.242/32') and lost the port. These justified rejections reduced CAR to 3/16 under the official labels.Official coverage gap
“New interfaces”The issue specified a log_enabled parameter absent from the gold patch. The generated test followed a stated signature that the reference implementation did not provide.Inconsistency between the stated specification and reference implementation
CaddyThe generated test failed before the reference fix and passed afterward, but rejected officially correct patch #13 by requiring an error for an empty default_bind that neither the issue nor the official oracle required. It also missed address order and accepted officially incorrect patch #65.Generated-test errors: an unjustified restriction and missed address order

(3) Interpreting the metrics. The audit complements our construction process, which filtered official tests that depended too closely on the gold implementation or missed core behavior. The 6.3% figure is the rate of cases flagged for review, not an overall estimate of official label errors. CAR, WDR, and JVR quantify acceptance and rejection using a shared official oracle and candidate pool, and retaining the upstream labels preserves reproducibility and comparability.

We will add this audit and case analysis, together with the interpretation of the metrics, to the revised paper.

Q3 The training results do not establish an advantage over additional repair supervision. At the same TestGen data budget, validation-guided selection improves downstream AVG@4 over GRR-based selection by 0.77, 0.97, and 0.21 percentage points on SWE-bench Verified, Multilingual, and Pro, respectively. JVR increases from 2.26% to 3.23%, corresponding to a net increase from 14 to 20 fully validated tasks out of 619. These gains are positive but modest, and their practical significance remains unclear given the additional candidate-pool construction and execution costs. Moreover, the experiments already combine TestGen trajectories with repair trajectories, but do not compare against adding an equivalent budget of further repair trajectories. Consequently, they do not establish the comparative advantage of allocating that budget to TestGen supervision. Since the benchmark is the central contribution, I would prefer stronger validation of the benchmark itself over broader training claims that remain insufficiently isolated.

A3 We agree that matched training budgets and evaluation costs are important for interpreting the downstream results. These experiments examine whether TestGen supervision improves repair and whether validation selects more useful training trajectories.

(1) Matched-budget comparison. We added an extra repair baseline using the same base model and training settings, expanding repair data from 2B to 4B tokens. Each unscaled TestGen configuration uses 2B repair + 2B TestGen tokens. Entries are AVG@4 / Pass@4 (%); the last row reports percentage-point gains.

Training configurationVerifiedMultilingualPro
SWE SFT: 2B repair67.53 / 79.2055.08 / 72.0046.56 / 60.05
Extra repair: 4B repair70.03 / 79.2061.87 / 74.6748.59 / 59.91
TestGen: GRR selection69.47 / 80.0062.05 / 76.2548.97 / 60.47
TestGen: validation-guided70.24 / 80.6063.02 / 76.3349.18 / 60.68
Validation-guided − extra repair+0.21 / +1.40+1.15 / +1.66+0.59 / +0.77

GRR selection falls 0.56 points below extra repair on Verified AVG@4, indicating that the benefit depends on data selection. Corresponding SWEPatchTest scores (%) are:

ConfigurationGRRWDRCARJVR
SWE SFT8.8920.7525.281.29
TestGen: GRR selection39.2659.0043.982.26
TestGen: validation-guided40.3964.3750.273.23

(2) Downsampled ablation. We downsampled the main TestGen pool, using the same base model, 1B repair tokens, and 0.33B TestGen tokens per configuration. Both selections require GRR = 1; the validation filter additionally requires WDR and CAR each ≥ 0.8.

SelectionSWEPatchTest GRR (%)WDR (%)CAR (%)JVR (%)
GRR13.0939.2925.880.32
Validation filter15.3542.2927.130.65

The completed downstream evaluation uses one attempt per task. Gains are in percentage points.

BenchmarkGRR Pass@1 (%)Validation Pass@1 (%)Validation − GRRValidation − repair alone
Verified59.0064.59+5.59+3.81
Multilingual56.9057.58+0.68+6.00
Pro30.6836.76+6.08+8.38

Paired exact McNemar tests against repair-only training, with Holm correction over six comparisons, show a significant Pro gain for the validation filter (95% CI [5.49, 11.26], Holm p < 0.001). Its Verified and Multilingual gains have unadjusted p < 0.05 but do not remain significant after correction. GRR selection shows no significant gain over repair-only training on any benchmark. These comparisons support the usefulness of TestGen data selected through validation.

(3) Construction and execution costs. SWEPatchTest reuses repairs already generated and evaluated for the underlying SWE-bench tasks, requiring no additional repair rollouts.

Cost statisticValue
Patches / tasks82,579 / 619
Average patches per taskAbout 133
Verification time: mean / median / 90th percentile5.2 / 1.9 / 11.9 seconds
Total execution time per task for one generated test packageApproximately 690 seconds (11.5 minutes)

This is a small overhead compared with the hours required for an agent rollout. We will include these comparisons, statistical results, and execution costs in the revised paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Esteemed Area Chair and Reviewers,

We sincerely thank all reviewers for their thoughtful questions, constructive suggestions, and positive comments on SWEPatchTest. We appreciate their recognition of:

  • An important problem: tests that reproduce a bug on the reference fix can still misjudge other repairs (All reviewers).
  • A well-motivated metric and a useful resource: JVR jointly measures reproduction, acceptance of correct patches, and rejection of incorrect ones over 82,579 candidate patches (All reviewers).
  • A new training strategy: fine-tuning on test-generation trajectories, with reported repair gains (Reviewer 2FQq).

Following the discussion, Reviewer HiPh found that the additional analysis "does improve the paper quite a bit", found the core finding compelling, and maintained the positive rating. Reviewer 2FQq thanked us for addressing the concerns.

Novelty. Existing evaluations check a generated test against the reference fix or against synthetic mutants; SWEPatchTest asks whether the test can judge every repair a coding agent proposes for the same issue:

  • Joint validation over real alternative repairs: each test must accept officially correct alternatives and reject officially incorrect ones on the same task. GPT-5.6-sol reaches 59.29% GRR but only 9.21% JVR, and detects 88.57% of failing patches while accepting only 50.20% of passing ones.
  • Validation as a training signal: JVR-style scores select test-generation trajectories for fine-tuning, a strategy Reviewer 2FQq was not aware of in prior work.

In response to their questions and suggestions, we provided the following key clarifications and additions:

QuestionReviewerOur Response
How do disagreements with the official oracle arise?SZ1X, HiPhAll 619 tasks screened with tests from two models; two authors reviewed every flagged task and 40 unflagged controls without seeing the judge's decisions. The audit separates official coverage gaps (Powertools), specification–reference inconsistencies, and generated-test errors (Caddy). Reviewer HiPh found the cases interesting and the use of upstream labels a valid approach.
Does TestGen supervision beat the same budget of repair data?SZ1X, 2FQqAt a matched 4B-token budget, validation-guided TestGen beats extra repair training on all three benchmarks in both AVG@4 (+0.21, +1.15, +0.59) and Pass@4 (+1.40, +1.66, +0.77).
Are the downstream gains statistically reliable?HiPhWith paired tests and Holm correction, scaled TestGen supervision improves AVG@4 over SWE SFT on all three benchmarks (+4.31, +8.22, +4.14; Holm p < 0.001). The validation filter gives +8.38 Pass@1 on SWE-bench Pro over repair-only training (Holm p < 0.001).
Why AVG@4 and Pass@4, and what about Pass@1?2FQqAVG@4 averages four single-attempt rounds, a lower-variance estimate of Pass@1. Measured directly, validation selection raises Pass@1 by +5.59, +0.68, and +6.08 points over GRR selection at the same TestGen volume.
How is TestGen used in repair, and how is data selected?2FQqTestGen trajectories are used in fine-tuning; at inference the trained agent writes and runs its own tests in the same OpenHands setup. Reproduction scripts appear in 81.2% of trajectories (from 56.2%), and trajectories with no test interaction fall from 32.1% to 9.0%. Selection rules (GRR = 1, CAR and WDR thresholds), prompts, agent frameworks, artifact extraction, and scoring are fully specified.
How sensitive is JVR to the candidate pool?HiPh, SZ1XOver 100 random draws per size, the model ranking matches the full pool (Spearman 1.000 at 96 patches, 0.976 at 64, 0.952 at 32), the three model tiers stay distinct, and the four SFT configurations keep their order at every pool size.
Is the benchmark practical, and does training improve test quality?SZ1X, 2FQqThe benchmark reuses existing repairs and needs no new repair rollouts; one test package is evaluated in about 11.5 minutes, against hours for an agent rollout. Training triples JVR (1.29% → 3.88%, paired 95% CI [1.29, 4.04]) and, at the same budget, validation-guided selection nearly doubles tasks with every faulty patch detected over GRR selection (13.73% → 25.69%).

We thank the reviewers and the AC again for helping strengthen the paper. We hope these clarifications and additional results support favorable consideration of SWEPatchTest as an execution-based benchmark that measures whether generated tests can judge the many repairs coding agents propose, and a signal for training better test-writing agents.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.