- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces WideSWE, a benchmark of 120 real tasks in which one feature or bug fix must be implemented across two or more GitHub repositories of the same ecosystem (e.g., Sentry's Go, Python and Ruby SDKs). The tasks are mined from linked pull requests in 41 ecosystems, and prompts are written from the original issues and PRs. Hidden tests are taken from the reference patches and then manually relaxed, so that correct but different implementations are not rejected. A task counts as solved only if every target repository passes all its fail-to-pass and pass-to-pass tests. Seven agent configurations are evaluated; the best (Codex CLI with GPT-5.6-sol) solves 42.5% of tasks, and the others solve 10.8%–37.5%. The paper groups failures into missing scope, unfinished work and incorrect edits, and compares solving all repositories in one run ("joint") with one run per repository ("independent").
Critical strengths
A clear and important gap: real software changes often span several repositories, and existing benchmarks score one repository at a time.
Real tasks from 41 ecosystems, balanced between 60 bug fixes and 60 features.
Careful construction: manual review, executable checks, and at least one fail-to-pass test per target repository.
Sensible test relaxation (e.g., removing exact error-message checks), with a check that the revised tests still reject the original code and accept the reference.
Useful failure analysis: the best agent fully solves at least one repository in 83% of tasks but the whole task in only 42.5%.
The joint versus independent comparison is informative: separate runs recover forgotten repositories, while a joint run helps by letting the agent compare related code.
Critical weaknesses
Each configuration is run once, with no variance or confidence intervals. Many differences are only a few tasks (e.g., 42.5% vs. 37.5% is 6 of 120 tasks).
The joint versus independent result (36 vs. 32 of 89 tasks, with 20 reversals) and the bug-fix/feature split (only 29 bug-fix tasks) may be within run-to-run noise.
No contamination check. Tasks come from public PRs merged from June 2025 onward (constituent PRs may be older), but the evaluated models are much newer and may have seen these fixes during training.
The test relaxation is manual and subjective. The paper does not report how many tests were changed, or whether the changes were made before or after looking at agent outputs.
Relaxed tests may also accept some incorrect solutions; test strength after revision is not measured.
Most tasks involve only two repositories (107 of 120), so the conclusions about three-repository tasks rest on 13 tasks. Six of the seven configurations use Claude Code, and the scaffold changes results a lot (GPT-5.6-sol: 42.5% with Codex CLI vs. 32.5% with Claude Code). Model comparisons are therefore mixed with how well each model fits this scaffold.
In the independent setting, each repository gets a full run (3.1 times more API requests), so the two settings do not have the same total budget.
Questions for authors
Can you report results over several runs (or at least for the top configurations), with confidence intervals, so that the model ranking and the joint vs. independent comparison can be judged against noise?
How do you rule out training-data contamination, given that the reference PRs are public? For example, do results differ for tasks merged after each model's training cutoff?
How many hidden tests were relaxed or removed, and were these revisions finalized before any agent outputs were inspected?
How do models other than GPT-5.6-sol perform with a second scaffold (e.g., Codex CLI or a model-neutral harness)?
With equal total budget (e.g., splitting one run's budget across repositories), does independent execution still recover the omitted repositories?
2 comments
Response to Reviewer Sj6M (2/2)
W6 & Q4: How do models other than GPT-5.6-sol perform with a second scaffold (e.g., Codex CLI or a model-neutral harness)?
We thank the reviewers for this helpful suggestion. We additionally evaluated Qwen 3.8 Max with Codex CLI. The results are summarized below.
| Model | Scaffold | Task success (%) | Mean API calls/task |
|---|---|---|---|
| Qwen 3.8 Max | Codex CLI | 34.2 | 102.4 |
| Qwen 3.8 Max | Claude Code | 37.5 | 177.1 |
| GPT-5.6-sol | Codex CLI | 42.5 | 86.7 |
| GPT-5.6-sol | Claude Code | 32.5 | 134.8 |
Qwen performs better with Claude Code, while GPT-5.6-sol performs better with Codex CLI. This suggests that the relative advantages of different scaffolds vary across models. For these two models evaluated with both scaffolds, the main table reports the better-performing configuration for each. Thus, the main-table results do not underestimate their performance due to scaffold mismatch. Moreover, we report the performance of model–scaffold configurations, rather than a ranking of model capabilities independent of the scaffold.
W7 & Q5: With equal total budget (e.g., splitting one run's budget across repositories), does independent execution still recover the omitted repositories?
We thank the reviewers for highlighting the importance of a fair budget comparison. The original experiment was designed to simulate realistic usage: joint execution used one complete run, while independent execution used a separate complete run for each target repository.
To further examine performance under the same total budget, we conducted an additional experiment with matched API call budgets. For each task, we used the number of API calls made during joint execution as the total budget and divided it equally among the target repositories. Each independent run was terminated upon reaching its allocated budget, and its current patch was evaluated. Independent execution achieved a task success rate of 0.00% (0/89) and a repository success rate of 2.09% (4/191). It successfully recovered 0 of the 20 repositories omitted during joint execution.
Further inspection of the traces showed that, for 18 of the 20 omitted repositories (90%), the agent had explicitly identified the required changes before reaching the API limit but had not completed a correct implementation.
Response to Reviewer Sj6M (1/2)
We appreciate your careful evaluation and insightful comments. We provide additional experimental results and clarifications to address your concerns below.
W1 & W2 & Q1: Can you report results over several runs (or at least for the top configurations), with confidence intervals, so that the model ranking and the joint vs. independent comparison can be judged against noise?
We thank the reviewers for this valuable suggestion. Given the cost of repeated evaluations, we report results from three joint-execution runs for each of the two best-performing configurations in the paper (GPT-5.6-sol with Codex CLI and Qwen 3.8 Max with Claude Code), along with three independent-execution runs of GPT-5.6-sol with Codex CLI.
Standard deviations are expressed in percentage points. Confidence intervals are calculated from the success rates across the three runs using the t-distribution.
Across all three runs, the observed ranking of the two model configurations remained unchanged, with GPT with Codex achieving a higher success rate than Qwen with Claude Code. In the comparison between joint and independent execution, joint execution achieved a higher overall success rate in every run. Independent execution achieved a higher success rate on Bugfix tasks, while joint execution achieved a higher success rate on Feature tasks, consistently across all three runs.
W3 & Q2: How do you rule out training-data contamination, given that the reference PRs are public? For example, do results differ for tasks merged after each model's training cutoff?
We thank the reviewers for raising this concern. We evaluated the latest models available at the time of our experiments, and it is difficult to fully verify whether public PRs were included in their training data. WideSWE requires coordinated changes across multiple repositories. Even if a model has encountered an individual PR, it must still identify the full scope of changes and maintain consistent behavior across repositories.
To provide empirical evidence, we conducted a temporal analysis of Gemini 3.8 Flash and Claude Opus 5, for which official knowledge cutoff months are verifiable and tasks are available both before and after these cutoffs. For these two models, we did not observe higher success rates on older tasks.
W4 & W5 & Q3: How many hidden tests were relaxed or removed, and were these revisions finalized before any agent outputs were inspected?
We thank the reviewers for this important question. We relaxed implementation-specific constraints in 491 hidden tests and removed 26 hidden tests covering functionality not required by the tasks. To assess the revised test suites’ ability to distinguish correct from incorrect implementations, we conducted targeted mutation analysis of the behaviors affected by the revisions, evaluating 1,143 faulty variants and 875 correct alternative implementations. In this analysis, the revised test suites rejected all faulty variants and accepted all correct alternative implementations. Furthermore, all these revisions were finalized before inspecting any outputs from the evaluated agents.