- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a new benchmark, Multi-SWT-Bench, for reproduction test genmeration. It is similar to SWT-Bench, but multi-lingual. It is derived from Multi-SWE-Bench with some mild filtering, leaving 1,963 instances, plus a subset of 340 instances called Multi-SWT-Bench-Lite. The benchmark measures the usual metrics for reproduction tests: F->P rate and patch coverage. The paper also reports Lite results with 4 harnesses (MOpenHands, MSWE-Agent, Claude Code, and Codex) and with strong frontier and DeepSeek models. The overall F->P rate with the strongest models is 65%. Python numbers are highest and C++ numbers are lowest.
Critical strengths
S1. Having a multi-lingual benchmark for reproduction test generation is useful.
S1. The benchmark construction makes sense (but note my comments on the Lite subset below).
S2. I appreciate the choice of up-to-date models and harnesses used to report results on.
Critical weaknesses
W1. One gap is that the paper doesn't disclose the composition of the Lite subset, yet reports results only on that. With only 340 instances across 8 languages, there are at most 42.5 instances per language, which is barely enough for meaningful results.
W2. Overall, while this paper is solid work, it is not very innovative. It's mostly just following SWT-Bench and Multi-SWE-Bench.
Questions for authors
Q1. What is the composition of the Lite subset?
Q2. Did you include a "hints" field such as the one in Multi-SWE-Bench describing newly defined variables?
2 comments
Additional Response to Reviewer oDJ2: Full-Set Evaluation
Q2: Subset representativeness - Full-set evaluation. We have completed the full-benchmark evaluation announced in our previous response. We evaluated Codex + GPT-5.6 Luna and Claude Code + Claude Opus 5 on all 1,963 instances of MULTI-SWT-BENCH. The table below compares the success rates of the two configurations across programming languages on both the full benchmark and the Lite subset. For each language, the value on the left reports the full-benchmark result, while the value on the right reports the corresponding Lite result.
| Language | Codex + GPT-5.6 Luna | Claude Code + Claude Opus 5 |
|---|---|---|
| All | 43.71 / 43.53 | 66.63 / 65.29 |
| Python | 59.92 / 62.50 | 83.97 / 82.50 |
| Java | 23.42 / 40.00 | 29.73 / 50.00 |
| TypeScript | 50.00 / 55.56 | 73.21 / 66.67 |
| JavaScript | 46.63 / 42.22 | 69.38 / 66.67 |
| Go | 27.53 / 31.11 | 53.88 / 62.22 |
| Rust | 35.00 / 42.22 | 60.00 / 71.11 |
| C | 39.37 / 42.50 | 66.93 / 70.00 |
| C++ | 48.41 / 32.50 | 63.49 / 52.50 |
The overall success rates do not change substantially between the Lite and full benchmarks for either configuration. For Codex + GPT-5.6 Luna, the success rate is 43.53% on Lite and 43.71% on the full benchmark; for Claude Code + Claude Opus 5, it is 65.29% and 66.63%, respectively. Importantly, Python remains the highest-performing language for both configurations on the full benchmark, with success rates of 59.92% for Codex + GPT-5.6 Luna and 83.97% for Claude Code + Claude Opus 5. These results support our main conclusion that evaluations limited to Python can provide an overly optimistic view of reproduction test generation performance and that multilingual evaluation remains necessary.
Official Response to Reviewer oDJ2
We thank the reviewer for the positive assessment of our benchmark construction and evaluation setup, as well as the helpful comments on the Lite subset and novelty. We address these points below.
W1 and Q1: Subset composition. MULTI-SWT-BENCH-LITE was constructed primarily from Multi-SWE-Bench Flash, an official 300-instance subset selected by the original authors of Multi-SWE-Bench for rapid multilingual evaluation. Because Flash does not include Python, we additionally randomly sampled 40 Python instances from MULTI-SWT-BENCH, resulting in 340 instances across eight languages.
We acknowledge that the Lite subset is relatively small. Compared with the full benchmark, it also tends to contain smaller repositories and lighter tasks (e.g., average codebase size: 236.8K vs. 448.0K LoC; average test-patch size: 62.5 vs. 96.7 changed lines). We include a detailed Full-vs.-Lite comparison in Appendix A.
To complement the Lite evaluation, we are also evaluating Codex + GPT-5.6 Luna on all 1,963 instances in the full benchmark, which includes larger repositories and a broader range of task characteristics. We will report these results by November 17.
W2: Novelty. We agree that our benchmark builds on SWT-Bench and Multi-SWE-Bench. However, our main contribution is the empirical insight enabled by broadening reproduction test generation beyond Python-centric settings. In particular, our failure analysis reveals limitations of state-of-the-art approaches that could not be captured by SWT-Bench alone, including ecosystem-specific challenges (e.g., repository-specific testing conventions; Section 4.2.1) and cross-ecosystem challenges (e.g., implicit setup requirements and failures to preserve target behavior during iterative revisions; Section 4.2.2).
Q2: Hints field. No. We did not provide the hints field from Multi-SWE-Bench to the agents. MULTI-SWT-BENCH evaluates whether agents can generate reproduction tests from the issue description and the pre-fix repository. The hints field in Multi-SWE-Bench is derived from variables newly defined in the golden test.patch and fix.patch, so providing it would expose information from these golden patches before test generation. In particular, hints derived from the test.patch may reveal information directly useful for constructing the reproduction test.