- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper makes several contributions. First, the authors show that model-generated tests are often unreliable at distinguishing correct and incorrect patches, as demonstrated by low Joint Validation Rate (JVR), a metric introduced by the authors. Second, they propose a curated benchmark to evaluate test-generation models. Finally, they use TestGen trajectories to fine-tune a model and show that this can improve both JVR and downstream repair success.
Critical strengths
-
I think the authors have identified an important issue with test overfitting. The motivation and definition of JVR seem solid to me.
-
The motivation and process behind the benchmark construction seem reasonable to me. However, the authors did not introduce entirely new issue instances. Instead, they combined instances from existing benchmarks and constructed candidate-patch pools for evaluation.
-
The authors have fine-tuned a model using test-generation trajectories. I am not aware of any prior work that follows exactly this fine-tuning strategy using test-generation trajectories. The authors also report improvements from the proposed technique, which is interesting.
Critical weaknesses
-
The novelty of this work is relatively low. First of all, test overfitting is not a new problem. It is a well-known issue in the test-generation community, and the authors have cited a few relevant works. The framing of JVR is relatively new. However, I do not find it fundamentally novel, as prior work has already explored evaluating tests against different patches and identifying test overfitting [1-3]. The authors have introduced several new metrics, which are intuitive but do not seem particularly novel to me. While the benchmark construction is useful, I am not convinced that the overall methodological contribution is sufficiently novel.
-
Although the authors have done a good job with the metrics and benchmark, I am not fully convinced by the empirical results. First of all, JVR remains very low even after fine-tuning (3.88%). While this supports the authors' motivation that reliable test generation is challenging, it also raises questions about the practical effectiveness of the proposed fine-tuning approach. The authors report improvements over the baseline, but the absolute JVR remains low. Therefore, I am not convinced that the authors have achieved sufficiently meaningful improvements in test-generation reliability.
-
The authors have also reported downstream repair success using AVG@4 and Pass@4. I am not sure why the authors chose a four-attempt evaluation rather than also reporting Pass@1. Although the results show improvements over the SWE SFT baseline, the additional improvements from validation-guided selection over GRR-based selection are relatively small, particularly on SWE-bench Pro. This raises questions about how much of the improvement comes from the proposed selection strategy rather than simply adding more TestGen training data. Overall, I find the evidence for the effectiveness of the proposed fine-tuning strategy somewhat limited.
-
The weakest aspects of the paper are its writing, lack of methodological clarity, and missing baselines. I have some serious concerns about how the different experiments were actually performed.
-
TestGen implementation: The paper does not clearly explain how the TestGen agents were designed or configured. I could not find sufficient descriptions of the prompts and implementation details needed to reproduce the results. That said, there are several existing reproduction-test generation techniques, but the authors did not provide a direct comparison with these approaches under comparable settings. Although the paper evaluates multiple coding-agent configurations, I believe comparisons with established reproduction-test generation approaches are important. This weakens the generality of the conclusions drawn from low JVR. Some existing approaches might achieve better JVR, and it is unclear how the proposed fine-tuned model would perform against them.
-
Integration of TestGen into the repair pipeline: It is not sufficiently clear to me how TestGen capabilities are actually used in the downstream repair pipeline. The authors explain that TestGen trajectories are incorporated during fine-tuning, but do the repair agents also generate and execute tests during inference? If so, how is this done? I believe an important baseline is missing here. For example, generating reproduction tests through naïve prompting and providing them to the repair agent might produce similar improvements without additional fine-tuning. If the generated tests are not explicitly used during repair, a comparison against additional repair-only SFT data with a matched training budget would be useful. Without these comparisons, it is difficult to isolate the actual contribution of fine-tuning with TestGen trajectories.
-
Training-data selection: Finally, it is not sufficiently clear how the training data were selected. The authors mention using their proposed metrics for the selection process, but I could not find a sufficiently detailed description of the exact selection procedure, including how the metrics were combined and what criteria were used to retain trajectories. Since validation-guided data selection is one of the main contributions, these details are important for reproducibility.
Although the paper starts with a good motivation and introduces a potentially useful benchmark, I am not convinced that it demonstrates sufficiently meaningful methodological or empirical contributions. In particular, the limited novelty, missing baseline comparisons, and insufficient methodological details weaken the overall contribution. These concerns prevent me from recommending acceptance of this paper.
[1] Edward K. Smith, Earl T. Barr, Claire Le Goues, and Yuriy Brun. Is the cure worse than the disease? overfitting in automated program repair. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE′15, pp. 532–543. ACM, August 2015. doi: 10.1145/2786805.2786825. URL http://dx.doi.org/10.1145/2786805.2786825.
[2] Ahmed, T., Ganhotra, J., Shinnar, A., & Hirzel, M. (2026, July). Investigating Test Overfitting on SWE-bench. In Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering (pp. 1292-1296).
[3] Wang, Y., Pradel, M., & Liu, Z. (2026, April). Are “solved issues” in SWE-bench really solved correctly? An empirical study. In Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering (pp. 169-181).
Questions for authors
-
Why did you not compare against established reproduction-test generation approaches, such as those represented on the SWT-bench leaderboard? How would these approaches perform under the proposed JVR evaluation?
-
How are TestGen capabilities actually incorporated into the downstream repair pipeline? Are tests generated and executed during repair, or are TestGen trajectories used only during fine-tuning? What baselines isolate the benefit of this supervision?
-
What is the rationale behind reporting AVG@4 and Pass@4 without also reporting Pass@1 for the repair task?
5 comments
Q3 The weakest aspects of the paper are its writing, lack of methodological clarity, and missing baselines. I have some serious concerns about how the different experiments were actually performed. TestGen implementation: The paper does not clearly explain how the TestGen agents were designed or configured. I could not find sufficient descriptions of the prompts and implementation details needed to reproduce the results.
A3 We agree that the TestGen setup should be described in enough detail to reproduce the experiments. Each agent receives the issue description and a repository checkout containing the bug. Appendix C.3 lists the model and agent software: GPT-5.6-sol runs in Codex, and Claude-Fable-5.1 and Claude-Opus-5 use Claude Code. Kimi-K3 uses Kimi Code, Qwen3.8-max uses Qwen Code, and GLM-5.3 uses ZCode. All DeepSeek-V4 configurations use OpenHands, as do the direct TestGen and downstream repair evaluations of the four SFT configurations. The prompt asks the agent to write an executable test and confirm that it fails on the original code because of the behavior described in the issue. An agent may try a temporary fix to check its test, then must revert the fix and reconfirm the failure before submission.
As described in Appendix C.1, we collect /tmp/run_verify_task.sh and its supporting files directly from the tmp directory for that task. Direct extraction preserves the test package without reconstructing it from a Git diff, modifying existing repository files, or conflicting with the official test patch. We run the same package unchanged on clean copies of the buggy code, the reference fix, and every candidate patch. The environment is reset after each execution to prevent fixtures, caches, or other side effects from affecting later runs. Execution records contain the command, exit code, timeout status, failing assertions, and harness output. They distinguish a pass or stable behavioral failure from a timeout, infrastructure failure, or missing artifact. We replay inconsistent executions; a patch whose execution remains unstable counts as neither accepted nor rejected and receives no credit. Metrics are averaged equally across tasks, with unresolved or incomplete runs retained in the denominator at zero. We will include the full prompt, a table of models, agent frameworks and inference settings, and these execution and scoring details in the revised paper.
Q4 That said, there are several existing reproduction-test generation techniques, but the authors did not provide a direct comparison with these approaches under comparable settings. Although the paper evaluates multiple coding-agent configurations, I believe comparisons with established reproduction-test generation approaches are important. This weakens the generality of the conclusions drawn from low JVR. Some existing approaches might achieve better JVR, and it is unclear how the proposed fine-tuned model would perform against them. Why did you not compare against established reproduction-test generation approaches, such as those represented on the SWT-bench leaderboard? How would these approaches perform under the proposed JVR evaluation?
A4 We agree that the connection to established reproduction test methods deserves a clearer explanation. SWT-Bench checks whether a generated test fails on the buggy code and passes with the reference fix. This corresponds to GRR in SWEPatchTest, which Table 2 reports for every configuration. SWEPatchTest extends the check to alternative repairs that pass the official tests and repairs that fail them, evaluating whether a test recognizes both working alternatives and incomplete fixes. This requires coverage of the issue's behavior and relevant boundaries while accommodating different correct implementations. GPT-5.6-sol achieves 59.29% GRR but 9.21% JVR, and the Caddy case illustrates this gap. Its generated test fails on the buggy code and passes with the reference fix, yet accepts officially incorrect patch #65 because it omits address order. It also rejects officially correct patch #13 by requiring an error for an empty default_bind, which neither the issue nor the official oracle requires. The case demonstrates how a successful reproduction test can still misjudge alternative repairs. Executable test packages from existing SWT-Bench methods can be evaluated with the same harness and candidate pools while retaining their original generation procedure. We will clarify this relationship and expand the accompanying case analysis in the revised paper.
Q5 Integration of TestGen into the repair pipeline: It is not sufficiently clear to me how TestGen capabilities are actually used in the downstream repair pipeline. The authors explain that TestGen trajectories are incorporated during fine-tuning, but do the repair agents also generate and execute tests during inference? If so, how is this done? I believe an important baseline is missing here. For example, generating reproduction tests through naïve prompting and providing them to the repair agent might produce similar improvements without additional fine-tuning. If the generated tests are not explicitly used during repair, a comparison against additional repair-only SFT data with a matched training budget would be useful. Without these comparisons, it is difficult to isolate the actual contribution of fine-tuning with TestGen trajectories. How are TestGen capabilities actually incorporated into the downstream repair pipeline? Are tests generated and executed during repair, or are TestGen trajectories used only during fine-tuning? What baselines isolate the benefit of this supervision?
A5 We agree that the role of TestGen during training and repair inference should be explicit. We use complete DeepSeek-V4-Flash sessions that write and check tests as supervision during fine tuning. At evaluation, every configuration uses the same OpenHands repair setup, where the trained agent solves the issue without receiving a suite from a separate test generator. Table 3 compares the two TestGen selection rules using the same source pool and training budget. On SWE-bench Multilingual, AVG@4 is 55.08% for SWE SFT, 62.05% for TestGen selected using GRR, and 63.02% for Validation-guided TestGen; their SWEPatchTest JVR scores are 1.29%, 2.26%, and 3.23%, respectively. We added a comparison using the same base model and training settings, expanding repair data from 2B to 4B tokens to match the 4B budget of the two TestGen configurations, each with 2B repair and 2B TestGen tokens. For the following scores, benchmarks are ordered Verified, Multilingual, and Pro, and each pair gives AVG@4 and Pass@4 in percent. Additional repair training scores (70.03, 79.20), (61.87, 74.67), and (48.59, 59.91), while Validation-guided TestGen scores (70.24, 80.60), (63.02, 76.33), and (49.18, 60.68). Its gains are therefore 0.21, 1.15, and 0.59 percentage points in AVG@4 and 1.40, 1.66, and 0.77 points in Pass@4. TestGen selected using GRR scores (69.47, 80.00), (62.05, 76.25), and (48.97, 60.47), including a Verified AVG@4 score 0.56 points below additional repair training. These results show that the benefit depends on how the TestGen data are selected.
Analysis of SWE-bench Pro repair trajectories also shows a change in test use during inference. Compared with SWE SFT, Validation-guided TestGen increases average unit test script executions from 2.7 to 3.4 per trajectory. Reproduction scripts appear in 81.2% of trajectories, up from 56.2%, and average writing operations for those scripts increase from 7.55 to 11.24. Average test file viewing or search operations rise from 3.61 to 4.20, while trajectories with no test interaction fall from 32.1% to 9.0%. This indicates that TestGen supervision encourages the repair agent to use tests when understanding issues and validating fixes. We will clarify this integration and include the additional repair baseline and trajectory analysis in the revised paper.
We thank the reviewer for examining what the benchmark adds to earlier work and for identifying details needed to reproduce the experiments. We group related comments with their corresponding questions and respond to each below.
Q1 The novelty of this work is relatively low. First of all, test overfitting is not a new problem. It is a well-known issue in the test-generation community, and the authors have cited a few relevant works. The framing of JVR is relatively new. However, I do not find it fundamentally novel, as prior work has already explored evaluating tests against different patches and identifying test overfitting [1-3]. The authors have introduced several new metrics, which are intuitive but do not seem particularly novel to me. While the benchmark construction is useful, I am not convinced that the overall methodological contribution is sufficiently novel. Although the paper starts with a good motivation and introduces a potentially useful benchmark, I am not convinced that it demonstrates sufficiently meaningful methodological or empirical contributions. In particular, the limited novelty, missing baseline comparisons, and insufficient methodological details weaken the overall contribution. These concerns prevent me from recommending acceptance of this paper.
A1 We agree that test overfitting and evaluation across patches have been studied in prior work. Earlier studies established that a patch can pass available tests while leaving the bug unresolved. SWEPatchTest asks whether a generated test can reliably judge several repairs for the same repository issue. SWT-Bench checks reproduction against the reference fix, and SWE-Mutation evaluates detection of faulty changes. Our benchmark also includes alternative repairs that pass the official tests, allowing us to measure whether a generated test rejects a working alternative. This adds an empirically important dimension: in Table 2, GPT-5.6-sol detects 88.57% of officially failing patches on average but accepts only 50.20% of officially passing ones. Measuring both outcomes gives a fuller account of a test's usefulness for judging repairs. We will clarify this contribution and its relationship to prior work in the revised paper.
Q2 Although the authors have done a good job with the metrics and benchmark, I am not fully convinced by the empirical results. First of all, JVR remains very low even after fine-tuning (3.88%). While this supports the authors' motivation that reliable test generation is challenging, it also raises questions about the practical effectiveness of the proposed fine-tuning approach. The authors report improvements over the baseline, but the absolute JVR remains low. Therefore, I am not convinced that the authors have achieved sufficiently meaningful improvements in test-generation reliability.
A2 We agree that the absolute reliability of generated tests is important when assessing the training results. Table 3 shows the clearest improvement in detecting faulty patches: at the same training budget, validation-guided selection increases the share of tasks on which every faulty patch is detected from 13.73% to 25.69%, compared with selection using GRR. Under the same comparison, fully validated tasks increase from 14 to 20 out of 619. With the larger training set, JVR reaches 3.88%, compared with 1.29% for training on repair data alone; Appendix E.2 reports a paired 95% confidence interval of 1.29 to 4.04 percentage points for this gain. These results show measurable improvements in test generation while identifying complete validation of every candidate repair as a continuing challenge. We will make the distinction between these gains and absolute test reliability clearer in the revised paper.