acceptodds
Under review as a conference paper at ICLR 2027

Multi-SWT-Bench: A Multilingual Benchmark for Reproduction Test Generation

Abstract

Reproduction test generation translates a natural-language issue description into executable tests that fail on the original code and pass after the issue is resolved, providing executable evidence for verifying candidate patches. However, existing benchmarks are constructed for individual programming languages, preventing a unified evaluation across diverse programming ecosystems. To address this limitation, we introduce Multi-SWT-Bench, a multilingual benchmark for reproduction test generation consisting of 1,963 instances across eight programming languages: Python, Java, TypeScript, JavaScript, Go, Rust, C, and C++. Based on this benchmark, we evaluate state-of-the-art LLMs using four representative methods (MSWE-agent, MOpenHands, Codex, and Claude Code) and analyze performance variations across language-specific subsets and factors associated with successful reproduction test generation. Our evaluation reveals a systematic language gap. Across every evaluated method and LLM, the Success Rate on Python exceeds the Success Rate across all languages, while the Success Rate on C++ is lower. This gap suggests that evaluations limited to Python can provide an overly optimistic view of reproduction test generation performance.

Primary area:
datasets and benchmarks
Keywords:
Reproduction tests, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oDJ2OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a new benchmark, Multi-SWT-Bench, for reproduction test genmeration. It is similar to SWT-Bench, but multi-lingual. It is derived from Multi-SWE-Bench with some mild filtering, leaving 1,963 instances, plus a subset of 340 instances called Multi-SWT-Bench-Lite. The benchmark measures the usual metrics for reproduction tests: F->P rate and patch coverage. The paper also reports Lite results with 4 harnesses (MOpenHands, MSWE-Agent, Claude Code, and Codex) and with strong frontier and DeepSeek models. The overall F->P rate with the strongest models is 65%. Python numbers are highest and C++ numbers are lowest.

Critical strengths

S1. Having a multi-lingual benchmark for reproduction test generation is useful.

S1. The benchmark construction makes sense (but note my comments on the Lite subset below).

S2. I appreciate the choice of up-to-date models and harnesses used to report results on.

Critical weaknesses

W1. One gap is that the paper doesn't disclose the composition of the Lite subset, yet reports results only on that. With only 340 instances across 8 languages, there are at most 42.5 instances per language, which is barely enough for meaningful results.

W2. Overall, while this paper is solid work, it is not very innovative. It's mostly just following SWT-Bench and Multi-SWE-Bench.

Questions for authors

Q1. What is the composition of the Lite subset?

Q2. Did you include a "hints" field such as the one in Multi-SWE-Bench describing newly defined variables?

2 comments
AuthorsOpenReview ↗

Official Response to Reviewer oDJ2

We thank the reviewer for the positive assessment of our benchmark construction and evaluation setup, as well as the helpful comments on the Lite subset and novelty. We address these points below.

W1 and Q1: Subset composition. MULTI-SWT-BENCH-LITE was constructed primarily from Multi-SWE-Bench Flash, an official 300-instance subset selected by the original authors of Multi-SWE-Bench for rapid multilingual evaluation. Because Flash does not include Python, we additionally randomly sampled 40 Python instances from MULTI-SWT-BENCH, resulting in 340 instances across eight languages.

We acknowledge that the Lite subset is relatively small. Compared with the full benchmark, it also tends to contain smaller repositories and lighter tasks (e.g., average codebase size: 236.8K vs. 448.0K LoC; average test-patch size: 62.5 vs. 96.7 changed lines). We include a detailed Full-vs.-Lite comparison in Appendix A.

To complement the Lite evaluation, we are also evaluating Codex + GPT-5.6 Luna on all 1,963 instances in the full benchmark, which includes larger repositories and a broader range of task characteristics. We will report these results by November 17.

W2: Novelty. We agree that our benchmark builds on SWT-Bench and Multi-SWE-Bench. However, our main contribution is the empirical insight enabled by broadening reproduction test generation beyond Python-centric settings. In particular, our failure analysis reveals limitations of state-of-the-art approaches that could not be captured by SWT-Bench alone, including ecosystem-specific challenges (e.g., repository-specific testing conventions; Section 4.2.1) and cross-ecosystem challenges (e.g., implicit setup requirements and failures to preserve target behavior during iterative revisions; Section 4.2.2).

Q2: Hints field. No. We did not provide the hints field from Multi-SWE-Bench to the agents. MULTI-SWT-BENCH evaluates whether agents can generate reproduction tests from the issue description and the pre-fix repository. The hints field in Multi-SWE-Bench is derived from variables newly defined in the golden test.patch and fix.patch, so providing it would expose information from these golden patches before test generation. In particular, hints derived from the test.patch may reveal information directly useful for constructing the reproduction test.

AuthorsOpenReview ↗

Additional Response to Reviewer oDJ2: Full-Set Evaluation

Q2: Subset representativeness - Full-set evaluation. We have completed the full-benchmark evaluation announced in our previous response. We evaluated Codex + GPT-5.6 Luna and Claude Code + Claude Opus 5 on all 1,963 instances of MULTI-SWT-BENCH. The table below compares the success rates of the two configurations across programming languages on both the full benchmark and the Lite subset. For each language, the value on the left reports the full-benchmark result, while the value on the right reports the corresponding Lite result.

LanguageCodex + GPT-5.6 LunaClaude Code + Claude Opus 5
All43.71 / 43.5366.63 / 65.29
Python59.92 / 62.5083.97 / 82.50
Java23.42 / 40.0029.73 / 50.00
TypeScript50.00 / 55.5673.21 / 66.67
JavaScript46.63 / 42.2269.38 / 66.67
Go27.53 / 31.1153.88 / 62.22
Rust35.00 / 42.2260.00 / 71.11
C39.37 / 42.5066.93 / 70.00
C++48.41 / 32.5063.49 / 52.50

The overall success rates do not change substantially between the Lite and full benchmarks for either configuration. For Codex + GPT-5.6 Luna, the success rate is 43.53% on Lite and 43.71% on the full benchmark; for Claude Code + Claude Opus 5, it is 65.29% and 66.63%, respectively. Importantly, Python remains the highest-performing language for both configurations on the full benchmark, with success rates of 59.92% for Codex + GPT-5.6 Luna and 83.97% for Claude Code + Claude Opus 5. These results support our main conclusion that evaluations limited to Python can provide an overly optimistic view of reproduction test generation performance and that multilingual evaluation remains necessary.

Reviewer_LUiZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Multi-SWT-Bench by converting the issue-resolution tasks in Multi-SWE-Bench into reproduction test generation tasks. The benchmark contains 1,963 instances from 51 repositories across eight programming languages. The experiments evaluate four agents with 12 model–agent configurations on a Lite subset of 340 instances, analyzing performance differences across languages, failure causes, and inference costs.

Critical strengths

The research problem is meaningful. Different programming languages have different characteristics, and constructing a multilingual benchmark is valuable for understanding models’ performance in practical settings.

Critical weaknesses

When evaluating a benchmark across different harnesses, it is important to ensure consistency in the prompts. However, Appendix A shows that the prompts used across harnesses are neither identical nor sufficiently similar in meaning, which undermines the reliability of the subsequent experimental comparisons.

As a multilingual evaluation benchmark, the difficulty of tasks across languages should be as comparable as possible to better assess differences between languages. However, as the authors acknowledge in lines 275–283, the lower success rate on C++ may be related to its larger average codebase size. A well-designed benchmark should minimize potential biases in its conclusions arising from differences in the tasks selected for each language.

Questions for authors

Could the authors use identical or semantically equivalent task instructions across harnesses and verify whether the main experimental conclusions remain consistent?

Could the authors control for differences in task difficulty and codebase size across languages to determine whether the observed performance gaps, particularly the lower success rate on C++, still persist?

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer LUiZ

We thank the reviewer for raising important concerns regarding prompt consistency and potential confounding across languages. We conducted additional experiments and analyses to directly examine both issues, as described below.

W1 and Q1: Prompt consistency. In our original manuscript, we did not standardize the task instructions across harnesses since task instructions are treated as part of the harness configuration [1]. This design is also consistent with prior similar studies [2-4].

To examine whether task-instruction differences affect method comparisons, we selected MOpenHands, MSWE-agent, and Codex CLI because all three support the same underlying LLM (i.e., GPT-5.6 Luna), allowing us to isolate the effect of task instructions while holding the model constant. We then used an identical task instruction across the three harnesses, retaining only the harness-specific instructions required to operate each agent, and re-evaluated all 340 MULTI-SWT-BENCH-LITE instances.

The Success Rate changed from 47.35% to 45.88% for MOpenHands (−1.47 percentage points), from 40.00% to 42.94% for MSWE-agent (+2.94 points), and from 43.53% to 45.88% for Codex CLI (+2.35 points). Paired two-sided exact McNemar tests found no statistically significant differences for any of the three methods (MOpenHands: p=0.576; MSWE-agent: p=0.275; Codex CLI: p=0.341). Overall, performance remained similar after standardizing the task instructions across the three harnesses.

[1] Microsoft, “Agent Harness,” Microsoft Learn, https://learn.microsoft.com/agent-framework/agents/harness

[2] Mündler et al. SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents. NeurIPS 2024.

[3] Zan et al. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. NeurIPS 2025.

[4] Liu et al. Process-Centric Analysis of Agentic Software Systems. OOPSLA 2026.

W2 and Q2: Confounding. We conducted an additional instance-level logistic regression analysis using all 4,080 evaluations in MULTI-SWT-BENCH-LITE (340 instances ×\times 12 method—LLM configurations). The model included programming language, method—LLM configuration, repository size (codebase LoC), golden fix patch length, issue description length, the number of files modified by the golden fix, and the human-assigned task difficulty provided by Multi-SWE-Bench (used as a proxy for task complexity) as explanatory variables. We log-transformed and standardized the four continuous variables and estimated cluster-robust standard errors at the repository level.

Our analysis shows that programming language remained significantly associated with success after controlling for these variables (omnibus test, p=0.048). In particular, the adjusted marginal predicted success rates were 54.79% for Python and 34.87% for C++, leaving a 19.92 percentage-point gap. The cross-language differences observed in the original manuscript cannot be fully explained by the measured task and repository characteristics. We added this analysis to Appendix E and summarized the adjusted results in Section 4.1.1.

Thank you for your response.

Regarding W1 and Q1, after standardizing the prompts, the success-rate gap between MOpenHands and MSWE-agent decreased from 7.35 to 2.94 percentage points, while the gap between MOpenHands and Codex CLI decreased from 3.82 percentage points to zero. Therefore, the paper’s previous claim that MOpenHands achieves the highest overall success rate when using the same underlying model may not be reliable, and I remain concerned about the related conclusions drawn from this finding.

Regarding W2 and Q2, I found that all experiments were conducted on the Lite subset, where the average C++ codebase size is 446.8K LoC. However, the paper explains the lower success rate on C++ by stating that its average codebase size is 994.3K LoC and the average number of changed lines in its test patches is 294.3, both the highest among the eight languages. These statistics come from the full dataset. In the Lite subset, the average Python codebase size is 669.3K LoC. As a benchmark paper, the paper should be more cautious when interpreting the observed results. Therefore, I consider my concerns only partially addressed, and I maintain my current score.

AuthorsOpenReview ↗

Follow-up Response to Reviewer LUiZ

Thank you for the follow-up comments and for clarifying your remaining concerns.

W1 and Q1: Prompt consistency. We agree that standardizing the task instructions reduces the relative performance gaps between the methods, indicating that relative performance is sensitive to task-instruction design. However, using the same instruction across harnesses does not necessarily constitute a fairer comparison. Task instructions are part of the harness configuration [1] and are designed to work together with each harness’s interface and interaction protocol. An instruction that is effective for one harness may therefore not be equally effective for another.

For this reason, our evaluation compares each harness under its original configuration. Importantly, this evaluation setting is consistent with prior studies comparing existing harnesses [2-4], where each harness is evaluated with its own configuration rather than forcing a single shared instruction. Under this evaluation setting, the claim that MOpenHands achieves the highest success rate remains valid. We view the standardized-prompt experiment as a sensitivity analysis showing that the magnitude of the performance gap depends on task-instruction design.

W2 and Q2: Confounding. We agree that our previous explanation for the lower success rate on C++ relied on descriptive statistics from the full benchmark, although the reported results were obtained on the Lite subset. Because these statistics do not directly characterize the evaluated subset, we have removed this explanation from our manuscript.

This correction does not affect our additional analysis with a regression model, which was conducted entirely on the Lite subset. The analysis directly examines whether programming language remains associated with performance after accounting for repository and task characteristics, addressing the reviewer’s original concern.

Reviewer_kZx3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MULTI-SWT-BENCH adapts issue/patch instances from Multi-SWE-Bench into tasks that ask coding agents to generate issue-reproducing tests rather than fixes. The benchmark retains 1,963 instances across eight programming languages and 51 repositories by screening developer tests for failing-before/passing-after (F→P) behavior against the reference repair. Experiments on a 340-instance Lite subset compare several agent/model configurations using F→P success, patch well-formedness, and coverage-oriented measurements. The paper reports considerable performance variation across languages, with Python generally performing better than several compiled-language subsets.

Critical strengths
  1. A handy benchmark target across language ecosystems. Testing whether an agent can produce a regression test for a reported issue is valuable in its own right, and multilingual coverage exposes toolchain and test-framework constraints that are hidden by Python-only evaluations.

  2. Executable screening and more informative outcomes than test failure alone. Requiring tests to fail on the pre-fix version and pass on the reference fixed version excludes many trivially invalid examples. Distinguishing F→P from tests that simply fail or are malformed is an appropriate starting point.

Critical weaknesses
  1. F→P does not, by itself, validate issue-reproduction semantics. A generated test can distinguish the pre-fix and post-fix repository versions due to unrelated implementation details, environmental effects, or overfitting to the gold patch without capturing the behavior specified in the issue. The paper does not provide sufficiently independent semantic validation or testing against alternative correct patches. This directly affects what the reported success rate means.
  2. The empirical language comparisons rely on a small, potentially confounded subset. All model experiments use 340 Lite instances, despite the advertised full benchmark containing 1,963. The Lite selection protocol and representativeness are not sufficiently clear. Large differences in repository size, test frameworks, and patch complexity across languages make it difficult to attribute observed performance gaps to programming language rather than task distribution.
Questions for authors

Semantic validity: On a blinded sample of successful generated tests, what proportion genuinely verifies the issue's intended behavior? Do tests continue to pass with alternative correct fixes, and do they reject unrelated or incomplete fixes?

Subset representativeness: Exactly how were the 340 Lite instances sampled? Can the authors compare their repository, issue-type, and difficulty distributions with the full 1,963 instances, and provide at least one full-set evaluation configuration?

Confounding and uncertainty: Do the reported cross-language differences persist after controlling for repository size, test framework, patch size, and other available task-complexity proxies, with uncertainty clustered by repository?

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer kZx3 [1/2]

We thank the reviewer for the constructive and detailed feedback, particularly regarding semantic validity, subset representativeness, and potential confounding in our cross-language analysis. We conducted several additional analyses to directly address these concerns, as detailed below.

W1 and Q1: Semantic validity. We agree that F→P alone does not fully establish whether a generated test captures the intended behavior of the issue. To directly examine this concern, we conducted two additional validations: (1) a blinded semantic assessment of successful generated tests and (2) validation using repair patches different from the golden patch.

First, we randomly sampled 100 successful generated tests (i.e., F→P) and had two human annotators independently assess them from two perspectives, following the criteria used in SWE-Bench Pro [1,2]: (1) overly strict, i.e., requiring implementation details not specified in the issue, and (2) low coverage, i.e., failing to sufficiently check the requested behavior. After resolving disagreements between the two annotators, the final labels showed that 78/100 tests (78%) were neither overly strict nor low coverage, with high inter-annotator agreement (Cohen's κ=0.883). At the same time, 22% of the successful tests violated at least one of these semantic criteria, indicating that F→P alone does not fully capture the semantic quality of generated tests. Motivated by this finding, we introduce Semantic Quality, defined as the proportion of successful tests that are neither overly strict nor low coverage. To enable this assessment at scale, we used Codex + GPT-6 Astra to automatically evaluate all successful generated tests using the same criteria. Its judgments showed substantial agreement with the human labels (Cohen's κ=0.831), supporting the use of this automated assessment for large-scale evaluation. Semantic Quality reveals differences that Success Rate alone does not capture; for example, Claude Code + Claude Opus 5 achieved the highest Success Rate (65.29%) but the lowest Semantic Quality (77.48%) among the 12 configurations. We introduce Semantic Quality in Section 3.2, report the results for all 12 configurations in Table 4, and provide details of the human and automated assessments in Appendix D.

Second, to further examine whether successful generated test sets (i.e., F→P) depend on the specific implementation adopted in the golden patch, we conducted an additional validation using alternative repair patches that resolve the same issue. We collected publicly available successful repair patches from Multi-SWE-Bench [3] across all 36 configurations formed by three agents (MOpenHands, MSWE-agent, and MAgentless) and 12 LLMs (e.g., GPT-4o and Claude 3.7 Sonnet). After matching the repository and base commit to our benchmark instances and removing duplicate patches within each issue, we obtained 1,143 distinct repair patches for 155 issues. For each issue, we evaluated every successful generated test set against every available alternative repair patch for that issue by applying the patch and running the test set. Since an issue can have multiple successful generated test sets and multiple alternative repair patches, this resulted in 7,518 evaluations. The generated test set passed after applying the alternative repair in 7,209 of these evaluations (95.89%). This result indicates that successful generated test sets generally do not depend on the specific implementation of the golden patch and remain valid when the same issue is resolved by a different correct repair patch. We provide details of this validation in Appendix C.

[1] OpenAI. Separating Signal from Noise in Coding Evaluations. 2026. https://openai.com/index/separating-signal-from-noise-coding-evaluations/

[2] Deng et al. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? ICML 2026.

[3] Zan et al. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. NeurIPS 2025.

AuthorsOpenReview ↗

Official Response to Reviewer kZx3 [2/2]

Q2: Subset representativeness. MULTI-SWT-BENCH-LITE was constructed primarily from Multi-SWE-Bench Flash, an official 300-instance subset selected by the original authors of Multi-SWE-Bench for rapid multilingual evaluation. Because Flash does not include Python, we additionally randomly sampled 40 Python instances from MULTI-SWT-BENCH, resulting in 340 instances across eight languages. We have revised the manuscript to clarify this construction process.

We further compared MULTI-SWT-BENCH-LITE with the full benchmark in terms of repository and task characteristics. The Lite subset tends to contain smaller repositories and lighter tasks than the full benchmark (e.g., average codebase size: 236.8K vs. 448.0K LoC; average test-patch size: 62.5 vs. 96.7 changed lines). This is consistent with the purpose of Multi-SWE-Bench Flash as a rapid and cost-efficient evaluation subset. We included a detailed per-language comparison in Appendix A.

To further examine performance on the full benchmark, which includes larger repositories and more diverse tasks than the Lite subset, we are also evaluating Codex + GPT-5.6 Luna on all 1,963 instances. We will report the full-set results by November 17.

W2 and Q3: Confounding and uncertainty. We conducted an additional instance-level logistic regression analysis using all 4,080 evaluations in MULTI-SWT-BENCH-LITE (340 instances ×\times 12 method—LLM configurations). The model included programming language, method—LLM configuration, repository size (codebase LoC), golden fix patch length, issue description length, the number of files modified by the golden fix, and the human-assigned task difficulty provided by Multi-SWE-Bench (used as a proxy for task complexity) as explanatory variables. We log-transformed and standardized the four continuous variables and estimated cluster-robust standard errors at the repository level. We also considered test framework, but it is largely tied to programming language in MULTI-SWT-BENCH-LITE, making its independent effect difficult to estimate.

Our analysis shows that programming language remained significantly associated with success after controlling for these variables (omnibus test, p=0.048). This indicates that the observed cross-language differences cannot be fully explained by the measured confounders and supports the importance of multilingual evaluation beyond Python-centric settings. We added this analysis to Appendix E and summarized the adjusted results in Section 4.1.1.

AuthorsOpenReview ↗

Additional Response to Reviewer kZx3: Full-Set Evaluation

Q2: Subset representativeness - Full-set evaluation. We have completed the full-benchmark evaluation announced in our previous response. We evaluated Codex + GPT-5.6 Luna and Claude Code + Claude Opus 5 on all 1,963 instances of MULTI-SWT-BENCH. The table below compares the success rates of the two configurations across programming languages on both the full benchmark and the Lite subset. For each language, the value on the left reports the full-benchmark result, while the value on the right reports the corresponding Lite result.

LanguageCodex + GPT-5.6 LunaClaude Code + Claude Opus 5
All43.71 / 43.5366.63 / 65.29
Python59.92 / 62.5083.97 / 82.50
Java23.42 / 40.0029.73 / 50.00
TypeScript50.00 / 55.5673.21 / 66.67
JavaScript46.63 / 42.2269.38 / 66.67
Go27.53 / 31.1153.88 / 62.22
Rust35.00 / 42.2260.00 / 71.11
C39.37 / 42.5066.93 / 70.00
C++48.41 / 32.5063.49 / 52.50

The overall success rates do not change substantially between the Lite and full benchmarks for either configuration. For Codex + GPT-5.6 Luna, the success rate is 43.53% on Lite and 43.71% on the full benchmark; for Claude Code + Claude Opus 5, it is 65.29% and 66.63%, respectively. Importantly, Python remains the highest-performing language for both configurations on the full benchmark, with success rates of 59.92% for Codex + GPT-5.6 Luna and 83.97% for Claude Code + Claude Opus 5. These results support our main conclusion that evaluations limited to Python can provide an overly optimistic view of reproduction test generation performance and that multilingual evaluation remains necessary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.