acceptodds
Desk rejected at ICLR 2027

WideSWE: Can Coding Agents Coordinate Changes Across Repositories?

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Coding-agent evaluation has progressed from resolving individual issues to carrying out long-horizon development, yet task completion is still largely assessed within a single codebase. In software ecosystems, many features and bug fixes require coordinated changes across multiple repositories. We introduce WideSWE to evaluate coding agents on such cross-repository tasks. Mining and reviewing changes across 103 software ecosystems yields 120 real-world tasks, balanced between 60 bug fixes and 60 features. We derive prompts from related issues and pull requests. We systematically review and adapt hidden tests to support diverse correct implementations while preserving required behavior and regression checks. Across seven agent configurations, full task success ranges from 10.83% to 42.50%, with the configuration pairing Codex CLI with GPT-5.6-sol achieving the highest rate. Trajectories show agents failing to identify necessary changes, recognizing changes but leaving them unfinished, or modifying the required repositories without fully satisfying the request. To examine whether working on one repository at a time can alleviate these difficulties, we compare it with joint execution under identical prompts. Independent execution mainly recovers omitted work and is less effective at correcting previously attempted but unsuccessful implementations. Joint execution can use information from related repositories to guide implementation and verification. Code is available at https://anonymous.4open.science/r/WIDESWE-08BB.

Primary area:
datasets and benchmarks
Keywords:
Coding Agents, Cross-Repository Software Engineering, Repository-Level Code Generation, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Sj6MOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WideSWE, a benchmark of 120 real tasks in which one feature or bug fix must be implemented across two or more GitHub repositories of the same ecosystem (e.g., Sentry's Go, Python and Ruby SDKs). The tasks are mined from linked pull requests in 41 ecosystems, and prompts are written from the original issues and PRs. Hidden tests are taken from the reference patches and then manually relaxed, so that correct but different implementations are not rejected. A task counts as solved only if every target repository passes all its fail-to-pass and pass-to-pass tests. Seven agent configurations are evaluated; the best (Codex CLI with GPT-5.6-sol) solves 42.5% of tasks, and the others solve 10.8%–37.5%. The paper groups failures into missing scope, unfinished work and incorrect edits, and compares solving all repositories in one run ("joint") with one run per repository ("independent").

Critical strengths

A clear and important gap: real software changes often span several repositories, and existing benchmarks score one repository at a time.

Real tasks from 41 ecosystems, balanced between 60 bug fixes and 60 features.

Careful construction: manual review, executable checks, and at least one fail-to-pass test per target repository.

Sensible test relaxation (e.g., removing exact error-message checks), with a check that the revised tests still reject the original code and accept the reference.

Useful failure analysis: the best agent fully solves at least one repository in 83% of tasks but the whole task in only 42.5%.

The joint versus independent comparison is informative: separate runs recover forgotten repositories, while a joint run helps by letting the agent compare related code.

Critical weaknesses

Each configuration is run once, with no variance or confidence intervals. Many differences are only a few tasks (e.g., 42.5% vs. 37.5% is 6 of 120 tasks).

The joint versus independent result (36 vs. 32 of 89 tasks, with 20 reversals) and the bug-fix/feature split (only 29 bug-fix tasks) may be within run-to-run noise.

No contamination check. Tasks come from public PRs merged from June 2025 onward (constituent PRs may be older), but the evaluated models are much newer and may have seen these fixes during training.

The test relaxation is manual and subjective. The paper does not report how many tests were changed, or whether the changes were made before or after looking at agent outputs.

Relaxed tests may also accept some incorrect solutions; test strength after revision is not measured.

Most tasks involve only two repositories (107 of 120), so the conclusions about three-repository tasks rest on 13 tasks. Six of the seven configurations use Claude Code, and the scaffold changes results a lot (GPT-5.6-sol: 42.5% with Codex CLI vs. 32.5% with Claude Code). Model comparisons are therefore mixed with how well each model fits this scaffold.

In the independent setting, each repository gets a full run (3.1 times more API requests), so the two settings do not have the same total budget.

Questions for authors

Can you report results over several runs (or at least for the top configurations), with confidence intervals, so that the model ranking and the joint vs. independent comparison can be judged against noise?

How do you rule out training-data contamination, given that the reference PRs are public? For example, do results differ for tasks merged after each model's training cutoff?

How many hidden tests were relaxed or removed, and were these revisions finalized before any agent outputs were inspected?

How do models other than GPT-5.6-sol perform with a second scaffold (e.g., Codex CLI or a model-neutral harness)?

With equal total budget (e.g., splitting one run's budget across repositories), does independent execution still recover the omitted repositories?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Sj6M (1/2)

We appreciate your careful evaluation and insightful comments. We provide additional experimental results and clarifications to address your concerns below.

W1 & W2 & Q1: Can you report results over several runs (or at least for the top configurations), with confidence intervals, so that the model ranking and the joint vs. independent comparison can be judged against noise?

We thank the reviewers for this valuable suggestion. Given the cost of repeated evaluations, we report results from three joint-execution runs for each of the two best-performing configurations in the paper (GPT-5.6-sol with Codex CLI and Qwen 3.8 Max with Claude Code), along with three independent-execution runs of GPT-5.6-sol with Codex CLI.

Configuration and Task SetSuccess Rate: Mean ± SD95% Confidence Interval
GPT with Codex40.56% ± 1.73[36.25%, 44.86%]
Qwen with Claude Code35.83% ± 1.44[32.25%, 39.42%]
GPT joint39.33% ± 1.12[36.53%, 42.12%]
GPT independent31.46% ± 3.89[21.79%, 41.13%]
GPT joint, Bugfix33.33% ± 1.99[28.39%, 38.28%]
GPT independent, Bugfix48.28% ± 5.97[33.44%, 63.11%]
GPT joint, Feature42.22% ± 1.92[37.44%, 47.00%]
GPT independent, Feature23.33% ± 7.64[4.36%, 42.31%]

Standard deviations are expressed in percentage points. Confidence intervals are calculated from the success rates across the three runs using the t-distribution.

Across all three runs, the observed ranking of the two model configurations remained unchanged, with GPT with Codex achieving a higher success rate than Qwen with Claude Code. In the comparison between joint and independent execution, joint execution achieved a higher overall success rate in every run. Independent execution achieved a higher success rate on Bugfix tasks, while joint execution achieved a higher success rate on Feature tasks, consistently across all three runs.

W3 & Q2: How do you rule out training-data contamination, given that the reference PRs are public? For example, do results differ for tasks merged after each model's training cutoff?

We thank the reviewers for raising this concern. We evaluated the latest models available at the time of our experiments, and it is difficult to fully verify whether public PRs were included in their training data. WideSWE requires coordinated changes across multiple repositories. Even if a model has encountered an individual PR, it must still identify the full scope of changes and maintain consistent behavior across repositories.

To provide empirical evidence, we conducted a temporal analysis of Gemini 3.8 Flash and Claude Opus 5, for which official knowledge cutoff months are verifiable and tasks are available both before and after these cutoffs. For these two models, we did not observe higher success rates on older tasks.

ModelOfficial knowledge cutoffTasks before cutoffSuccess rate before cutoffTasks after cutoffSuccess rate after cutoff
Gemini 3.8 FlashMarch 2026758.00%4515.60%
Claude Opus 5May 202611234.80%837.50%

W4 & W5 & Q3: How many hidden tests were relaxed or removed, and were these revisions finalized before any agent outputs were inspected?

We thank the reviewers for this important question. We relaxed implementation-specific constraints in 491 hidden tests and removed 26 hidden tests covering functionality not required by the tasks. To assess the revised test suites’ ability to distinguish correct from incorrect implementations, we conducted targeted mutation analysis of the behaviors affected by the revisions, evaluating 1,143 faulty variants and 875 correct alternative implementations. In this analysis, the revised test suites rejected all faulty variants and accepted all correct alternative implementations. Furthermore, all these revisions were finalized before inspecting any outputs from the evaluated agents.

AuthorsOpenReview ↗

Response to Reviewer Sj6M (2/2)

W6 & Q4: How do models other than GPT-5.6-sol perform with a second scaffold (e.g., Codex CLI or a model-neutral harness)?

We thank the reviewers for this helpful suggestion. We additionally evaluated Qwen 3.8 Max with Codex CLI. The results are summarized below.

ModelScaffoldTask success (%)Mean API calls/task
Qwen 3.8 MaxCodex CLI34.2102.4
Qwen 3.8 MaxClaude Code37.5177.1
GPT-5.6-solCodex CLI42.586.7
GPT-5.6-solClaude Code32.5134.8

Qwen performs better with Claude Code, while GPT-5.6-sol performs better with Codex CLI. This suggests that the relative advantages of different scaffolds vary across models. For these two models evaluated with both scaffolds, the main table reports the better-performing configuration for each. Thus, the main-table results do not underestimate their performance due to scaffold mismatch. Moreover, we report the performance of model–scaffold configurations, rather than a ranking of model capabilities independent of the scaffold.

W7 & Q5: With equal total budget (e.g., splitting one run's budget across repositories), does independent execution still recover the omitted repositories?

We thank the reviewers for highlighting the importance of a fair budget comparison. The original experiment was designed to simulate realistic usage: joint execution used one complete run, while independent execution used a separate complete run for each target repository.

To further examine performance under the same total budget, we conducted an additional experiment with matched API call budgets. For each task, we used the number of API calls made during joint execution as the total budget and divided it equally among the target repositories. Each independent run was terminated upon reaching its allocated budget, and its current patch was evaluated. Independent execution achieved a task success rate of 0.00% (0/89) and a repository success rate of 2.09% (4/191). It successfully recovered 0 of the 20 repositories omitted during joint execution.

Further inspection of the traces showed that, for 18 of the 20 omitted repositories (90%), the agent had explicitly identified the required changes before reaching the API limit but had not completed a correct implementation.

Reviewer_uVGkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Built from linked PRs in 41 software ecosystems, WideSWE contains 120 real tasks that each require coordinated edits across two or three repositories, with the inherited hidden tests revised to match the stated requirements. Task success across 7 agent configurations ranges from 10.83% to 42.50%; the authors categorize failures from trajectories and compare joint execution with running each repository separately.

Critical strengths
  1. Tasks are mined systematically from genuinely linked PRs across 103 ecosystems, and every target repository must have its own F2P tests, so the setting closely mirrors real cross-repository development.
  2. The test-revision rules are clearly stated, and each revision is independently checked by two authors. Appendix A.8 walks through concrete examples, and agreement for the failure annotation is reported openly.
Critical weaknesses
  1. The paper frames WideSWE as a test of cross-repository coordination, yet in Table 3 independent runs that cannot see the other repositories match joint execution at the repository level (60.73% vs. 61.78%, with identical 64.40% F2P). Can the authors show that task difficulty stems from coordination, rather than from compounding several single-repository tasks?
  2. Appendix A.8 reports that tests were revised in 88 of the 120 tasks, 87 of them by relaxing implementation constraints, yet the revised suites are only checked to fail on the original code and pass on the reference solution. Is there any check that the relaxed tests still reject incorrect implementations that miss the requirement?
Questions for authors
  1. At evaluation time, are a downstream target's hidden tests run against the agent-modified upstream repository, or against the original or reference version? In the latter case, would an interface mismatch between the two repositories be caught at all?
  2. Does the prompt name the target repositories that need changes? The Laravel AI/MCP run fails after placing the fix in a context repository; what fraction of Scope failures stem from the prompt leaving the scope unspecified?
4 comments
AuthorsOpenReview ↗

Response to Reviewer uVGk (1/2)

We sincerely thank you for your valuable feedback and constructive suggestions. We address your questions and concerns below.

W1: Given the nearly identical repository-level performance of joint and independent execution, how can the authors show that task difficulty stems from coordination, rather than from compounding several single-repository tasks?

We thank the reviewer for this important question. Although independent and joint execution achieve similar aggregate repository-level scores, a breakdown by task type reveals changes in opposite directions, and fine-grained failure analysis also indicates that cross-repository coordination contributes to the practical difficulty of some tasks.

First, the breakdown by task type shows the following changes:

Task TypeRepository Success Rate: Joint → IndependentF2P: Joint → Independent
Bugfix53.45% → 65.52%55.17% → 72.41%
Feature65.41% → 58.65%68.42% → 60.90%

Joint execution performs better on feature tasks, whereas independent execution performs better on bugfix tasks. These opposing changes offset one another in the aggregate, so the similar overall scores mask differences in performance across task types.

Second, fine-grained failure analysis identifies specific coordination difficulties. Beyond implementation difficulties within individual repositories, we also observe failures in identifying the full scope, completing recognized work, implementing necessary adaptations, and maintaining cross-repository contract consistency. Among unresolved GPT–Codex runs, 37.68% are classified as Scope failures, and 2.90% involve recognized but undelivered work. Some Post-edit failures also leave necessary cross-repository adaptations or propagation steps incomplete despite modifying the relevant repositories. Further analysis shows that 41.46% of Post-edit failures involve inconsistent contracts across repositories. These observations show that identifying the full scope, following through on necessary work in each repository, and ensuring compatibility among changes constitute part of the practical difficulty.

W2: Is there any check that the relaxed tests still reject incorrect implementations that miss the requirement?

We thank the reviewer for raising this concern. In addition to validating the original code and reference implementations, we conducted targeted mutation analysis for the 491 tests whose implementation constraints were relaxed. Each test was associated with at least three correct alternative implementations and three faulty variants, with variants reused across tests where applicable. In total, we evaluated 1,143 faulty variants and 875 correct alternative implementations. The complete revised hidden test suite rejected all faulty variants and accepted all correct alternatives. For the other 26 tests removed because they checked functionality outside the task requirements, we followed the procedure in Appendix A.8: one author proposed each revision, and two other authors independently reviewed it. Revisions were accepted only with both reviewers’ approval. We have added these details to Appendix A.8.

AuthorsOpenReview ↗

Response to Reviewer uVGk (2/2)

Q1: At evaluation time, are a downstream target's hidden tests run against the agent-modified upstream repository, or against the original or reference version?

The former: at evaluation time, downstream hidden tests run directly against the agent-modified upstream code, enabling them to detect interface mismatches.

Q2: Does the prompt name the target repositories that need changes? The Laravel AI/MCP run fails after placing the fix in a context repository; what fraction of Scope failures stem from the prompt leaving the scope unspecified?

We thank the reviewer for highlighting this issue. The prompts do not explicitly list all repositories that require changes. To stay close to real-world development, we preserve the semantics of the requirements in the original PRs without manually adding the modification scope. Developers sometimes mention related repositories, but these mentions do not necessarily indicate that those repositories require changes or cover the full scope of necessary modifications. Agents must therefore determine which repositories need changes, and identifying the affected scope is itself part of the evaluation.

For the Laravel AI/MCP task, the prompt did not mention the AI, MCP, or Framework repositories. The agent was allowed to modify Framework, and the effects of those changes were included in the evaluation, enabling two F2P tests in the AI repository to pass. However, streaming output in the MCP repository still failed to meet the requirements, so the task was not fully resolved.

We conducted an additional experiment on the tasks that originally exhibited Scope failures under each configuration. We appended an explicit list of repositories requiring modification to the prompt, while keeping all other settings unchanged:

This task requires modifications to {repo_1} and {repo_2}.

The results with this additional scope information are as follows:

ModelFrameworkOriginal Scope FailuresFinal SuccessesFinal FailuresRemaining Scope Failures
GPT-5.6-solCodex CLI2611150 (0.00%)
GPT-5.6-solClaude Code258170 (0.00%)
DeepSeek V4 ProClaude Code277201 (3.70%)
Claude Opus 5Claude Code208120 (0.00%)
Qwen 3.8 MaxClaude Code166100 (0.00%)
Gemini 3.8 FlashClaude Code403374 (10.00%)
GLM 5.3Claude Code295241 (3.45%)
Total—183481356 (3.28%)

Percentages use the number of tasks that originally exhibited Scope failures under each configuration as the denominator. Runs with remaining Scope issues are a subset of the final failures; the other failures may fall under Delivery or Post-edit.

After the target repository list was added, 96.72% (177/183) of the runs that originally exhibited Scope failures no longer showed Scope issues. GPT-5.6-sol, Claude Opus 5, and Qwen 3.8 Max exhibited no further Scope failures on these tasks. However, some runs of DeepSeek V4 Pro, Gemini 3.8 Flash, and GLM 5.3 still failed to identify required changes after inspecting the relevant repositories, or incorrectly concluded that certain repositories required no changes. This shows that explicitly specifying which repositories require modification does not guarantee that the agent will identify the necessary changes within them. Autonomously identifying the scope of required changes is also an important capability in real-world software maintenance, particularly for Bugfix tasks, where agents must locate the affected repositories and code based on the observed symptoms.

Thank you for the author's response; my concerns have been addressed, and I will maintain my score.

Thank you for your positive feedback. We sincerely appreciate your time and support!

Reviewer_3F2cOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a benchmark where the task is to fix a SWE issue in one or more related code repositories. Six frontier models are evaluated on the benchmark and their failure modes are analyzed.

Critical strengths

The benchmark is timely, given the increasing evidence for the importance of agents swarms collaboratively developing a codebase.

The benchmark is constructed well, using real-world data which covers diverse ecosystems. The tasks were manually reviewed.

Critical weaknesses

The pass rate of current frontier models is relatively high and the benchmark could quickly become outdated.

The paper would benefit from a deeper analysis of model performance and failure modes. E.g., token cost is mentioned in text, but it is not plotted and the performance/cost tradeoff is not investigated. Similarly the presented failure mode taxonomy can be more fine-grained, compared to reference works such as the Multi-Agent System Taxonomy https://openreview.net/forum?id=wM521FqPvI.

Questions for authors

Can the agents take shortcuts by finding patches in git history?

When do some task prompts require changes for the individual execution setting?

Sec 4.1 should describe the procedure for classifying failures.

While the authors may style the paper as they prefer, many readers prefer visually uniform fonts across all the text in the paper, including the figures. Figures 3 & 5 in particular clearly use Comic Sans.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3F2c (1/2)

We sincerely thank the reviewer for the constructive feedback. Below, we address each of your comments point-by-point.

W1: The pass rate of current frontier models is relatively high and the benchmark could quickly become outdated.

We thank the reviewer for raising this concern. Current results remain far from saturation: the best configuration in the paper, GPT-5.6-sol (high) with Codex CLI, achieves 42.5% success. We additionally evaluated newer models, GPT-6 Astra (high) and GPT-6.1 Sol (high), using Codex CLI, obtaining success rates of 45.0% and 44.2%, respectively. These improvements of only 2.5 and 1.7 percentage points leave more than half of the tasks unsolved.

W2: The paper would benefit from a deeper analysis of model performance and failure modes.

Thank you for the suggestion. We use API calls as a cost proxy. Table 1 reports success rates and average API calls per task, as discussed in Section 4.1. For GPT-5.6-sol:

Execution FrameworkTask Success RateAverage API Calls per Task
Codex CLI42.5%86.7
Claude Code32.5%134.8

In this experiment, Codex CLI achieves both a higher task success rate and fewer API calls per task.

Regarding failure modes, we thank the reviewer for recommending MAST[1]. Inspired by its approach to distinguishing failure mechanisms through execution traces, we further refine our original three categories—Scope, Delivery, and Post-edit—into the following subcategories, using evidence from execution traces, final patches, and test results. We provide the detailed failure taxonomy and analysis in Appendix B.4.

CategoryIDSubcategoryClassification Criteria
ScopeS1Failure to inspect a required repositoryThe trace shows neither task-relevant inspection of the omitted repository nor recognition of the required changes in that repository.
ScopeS2Failure to recognize required changes after inspectionThe relevant repository was inspected, but the required changes within it were not identified.
ScopeS3Incorrect exclusion of required changesThe agent explicitly concluded that a repository required no changes, contrary to the task requirements.
DeliveryD1No implementation deliveredThe required work was identified, but no substantive implementation was delivered in any target repository.
DeliveryD2Partial implementation deliveredImplementations were delivered in some target repositories, while other identified required changes remained undelivered.
Post-editP1Regression in existing behaviorThe changes broke previously correct behavior, as evidenced by pass-to-pass (P2P) tests that passed before the changes and failed afterward because of those changes.
Post-editP2Incomplete implementation of required behaviorChanges were made across the relevant repositories, but some required behavior lacked an implementation, such as a missing entry point, adaptation branch, or propagation step.
Post-editP3Inconsistent contracts across repositoriesThe relevant implementations were present, but their interfaces, data types, default values, or handling of boundary cases were incompatible.
Post-editP4Consistent but incorrect behavior across repositoriesThe relevant implementations across repositories were mutually consistent but shared a behavior or representation that violated the task requirements.
ModelExecution FrameworkS1S2S3D1D2P1P2P3P4
GPT-5.6-solCodex CLI34.6257.697.690.00100.0043.9034.1541.4617.07
GPT-5.6-solClaude Code56.0040.004.000.00100.0052.7345.4530.9114.55
DeepSeek V4 ProClaude Code29.6348.1522.220.00100.0042.5946.3037.0412.96
Claude Opus 5Claude Code45.0040.0015.0020.0080.0028.3037.7432.0828.30
Qwen 3.8 MaxClaude Code56.2512.5031.250.00100.0021.8232.7336.3630.91
Gemini 3.8 FlashClaude Code62.5035.002.5082.1417.869.0954.5527.2718.18
GLM 5.3Claude Code51.7213.7934.4833.3366.6743.1034.4846.5517.24

Percentages are computed within each parent category. Scope failures follow the priority S3 > S2 > S1. P1 may co-occur with P2–P4, while P2–P4 are mutually exclusive.

Among Post-edit failures, incomplete implementation of required behavior (P2) and inconsistent contracts across repositories (P3) are common. Some runs also produce implementations that are consistent across repositories but jointly deviate from the task requirements (P4).

AuthorsOpenReview ↗

Response to Reviewer 3F2c (2/2)

Q1: Can the agents take shortcuts by finding patches in git history?

This shortcut is not available in our setting. Agents receive repositories with only a single synthetic root commit, so upstream Git history, tags, gold commits, and reference patches are unavailable. They can only use Git to inspect the current snapshot and their own changes.

Q2: When do some task prompts require changes for the individual execution setting?

In some bug-fix tasks, repository A requires a bug fix, while repository B only needs to adapt to that fix. In the joint setting, the shared request is to fix the bug, and the agent must modify A and adjust B accordingly. This is precisely the cross-repository coordination capability that WideSWE evaluates. However, B itself does not contain the bug. Running B independently therefore requires explicitly describing A’s changes in the prompt, directly providing information that the agent would otherwise need to discover in the joint setting. The information supplied in the two settings would thus differ. To ensure a fair comparison, the independent setting includes only the 89 tasks whose prompts require no rewriting.

Q3: Sec 4.1 should describe the procedure for classifying failures.

Thank you for the suggestion. Due to space constraints, Section 4.1 summarizes the classification criteria and refers readers to Appendix B.3, which describes the two-stage classification procedure based on final code diffs and execution trajectories, along with independent annotation and disagreement resolution.

Q4: Inconsistent fonts between figures and the main text, particularly in Figures 3 and 5.

Thank you for the suggestion. We have standardized the fonts across all figures in the revised manuscript to use a consistent sans-serif typeface.

[1] Melissa Z. Pan et al. “Why do multiagent systems fail?” ICLR 2025 Workshop on Building Trust in Language Models and Applications, 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and AC

We sincerely thank all reviewers (3F2c, uVGk, and Sj6M) for their constructive feedback. We appreciate their recognition of WideSWE’s practical relevance, diverse real-world tasks, and careful benchmark construction. We have addressed the concerns through additional experiments and clarifications, and we thank Reviewer uVGk for confirming that their concerns have been addressed. Below, we summarize the main responses, with revisions highlighted in blue in the revised manuscript.

1. Benchmark Validity and Transparency

  • Test Reliability: We clarified that 491 hidden tests were relaxed and 26 removed, all before inspecting agent outputs. Mutation analysis showed that the revised suites rejected 1,143 faulty variants and accepted 875 correct alternatives. (uVGk W2; Sj6M W4–W5/Q3.)

  • Evaluation Integrity: We confirmed that agents cannot access upstream Git history or reference patches, and downstream tests run against agent-modified upstream code to detect interface mismatches. (3F2c Q1; uVGk Q1.)

  • Contamination Analysis: Temporal analyses for two models found no higher success on pre-cutoff tasks, while we acknowledge that training-data contamination cannot be conclusively excluded. (Sj6M W3/Q2.)

2. Coordination and Execution Settings

  • Coordination Evidence: We provided task-type breakdowns and finer-grained failure analysis; 41.46% of GPT–Codex Post-edit failures involve inconsistent cross-repository contracts. (uVGk W1.)

  • Explicit Scope: Providing target repository lists eliminated Scope issues in 96.72% of 183 reruns, but only 48 fully succeeded, demonstrating challenges beyond scope identification. (uVGk Q2.)

  • Comparison Fairness: We clarified the restriction to 89 tasks with identical prompts and added a matched API-call-budget experiment. Under the tested equal allocation, independent execution recovered none of the 20 omitted repositories. (3F2c Q2; Sj6M W7/Q5.)

3. Reliability and Benchmark Headroom

  • Repeated Runs: Three-run evaluations with standard deviations and confidence intervals showed consistent ranking and comparison directions across runs. (Sj6M W1–W2/Q1.)

  • Scaffold Effects: Additional Qwen evaluations demonstrated model-dependent scaffold effects; our comparisons concern model–scaffold configurations. (Sj6M W6/Q4.)

  • Newer Models: GPT-6 Astra and GPT-6.1 Sol achieved 45.0% and 44.2%, respectively, leaving more than half of the tasks unsolved. (3F2c W1.)

4. Analysis and Presentation

  • Failure Analysis: We expanded the taxonomy into nine subcategories, reported configuration-specific statistics, and clarified the annotation procedure. (3F2c W2/Q3.)

  • Presentation Consistency: We standardized the fonts across all figures in the revised manuscript to ensure visual consistency. (3F2c Q4.)

We thank the reviewers and the Area Chair again for helping us strengthen the evaluation and presentation of WideSWE.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.