acceptodds
Under review as a conference paper at ICLR 2027

CoDev-Bench: Can Coding Agents Handle Naturally Concurrent Development?

Abstract

Current coding benchmarks usually assume a single developer working on a standalone issue. However, real-world software development is inherently concurrent and collaborative, with multiple developers authoring pull requests (PRs) within the same codebase. To bridge this gap, we introduce CoDev-Bench, a novel benchmark designed to evaluate coding agents on concurrent multi-task resolution. CoDev-Bench involves 206 concurrent PR-pair tasks drawn from 137 production-scale open-source repositories across six languages. Each task consists of two independently authored PRs that branch from a shared base commit and are subsequently integrated into the main branch. The benchmark reflects the complexity of real concurrent development, with an average of 17.1 non-test files and 507 lines of code modified per pair, and 45.1% of pairs exhibiting file-level overlap. As reference baselines, we evaluate a single agent that handles both PRs, as well as two multi-agent designs: assigning each PR to a separate agent and merging their patches afterwards, and letting a main agent delegate sub-tasks to sub-agents while it works. Even the best-performing configuration resolves only about one-fifth of tasks on average over three trials, compared with over 80% for state-of-the-art systems on SWE-Bench Verified, showing that naturally concurrent PR-composition tasks remain a largely unsaturated challenge for current coding agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
coding agent, llm, cooperation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RTtoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

CoDev-Bench evaluates coding agents on pairs of pull requests (PRs) independently developed from a shared repository base. The benchmark contains 206 PR pairs from 137 open-source repositories across six languages. Given the base repository and two task descriptions, an agent must produce a unified patch that passes tests associated with both PRs. The authors evaluate four models using single-agent, split-and-merge, and runtime subagent approaches. The best configuration achieves a 21.8% resolution rate, with no consistent improvement from multi-agent approaches. Further analyses identify insufficient verification and implicit cross-PR dependencies as potential causes of failure. The paper presents naturally concurrent PR pairs as a challenging evaluation setting for coding agents.

Critical strengths

S1. Relevant and underexplored evaluation setting. The benchmark uses naturally concurrent PRs rather than synthetic tasks, capturing realistic software development requirements that are underrepresented in existing coding benchmarks.

S2. Comprehensive empirical analysis. The evaluation covers four models, multiple agent architectures, and three trials. The failure analysis provides useful insights into verification gaps, cross-PR dependencies, and limitations of existing multi-agent approaches.

Critical weaknesses

W1. Unclear construct validity of concurrent development. The primary task requires an agent to jointly implement two known requirements, rather than independently developing changes and subsequently integrating them. Thus, it mainly evaluates joint multi-task coding, without fully capturing concurrency-specific challenges such as stale contexts, independently made implementation decisions, and integration conflicts. The existing split-and-merge baselines only partially address this gap.

W2. Insufficient isolation of concurrency-specific difficulty. The low resolution rate may largely reflect increased task complexity rather than concurrency itself. Moreover, the reported cross-PR dependencies may partly arise from tests tied to specific implementation details (e.g., the nginx case in Appendix E). These issues weaken the evidence that the observed failures specifically result from concurrent development.

Questions for authors

Q1. What capabilities does CoDev-Bench measure beyond jointly solving two coding tasks? How does the formulation capture the unique challenges of concurrent development?

Q2. Can the authors compare independent per-PR success rates with joint-resolution rates under comparable budgets to quantify the additional difficulty?

Q3. Among the 83 pairs exhibiting cross-PR test dependencies, how many involve genuine functional dependencies rather than implementation-specific test expectations?

6 comments
AuthorsOpenReview ↗

Response by Authors (1/2)

Thank you for the careful questions. Below we respond point by point: we clarify what capability the benchmark measures, compare independent with joint completion, and trace where cross-PR dependencies come from.

Weaknesses

W1

Unclear construct validity of concurrent development.

Concurrency in CoDev-Bench lives in the task structure rather than in the solving procedure. To test this, we add two workflows that realize the challenges you list, and compare all four workflows on the same 206 pairs (Copilot CLI, Opus 4.8).

In the sequential workflow, agent A’s PR is committed first; agent B reads it through the git history and builds PR B on top, handling the interaction itself. In the independent workflow, two sessions implement one PR each without seeing each other, and a third session merges them with the paper’s merger instructions. The coupled / non-coupled split below uses the test from W2: one side’s gold patch alone fails its own hidden tests. The table lists each workflow’s resolved rate, overall and by coupling:

WorkflowAll 206 pairsCoupled 83 pairsNon-coupled 123 pairsGap (pp)
Single Agent, joint completion (paper, first run)20.4%12.0%26.0%14.0
Sub-Agent (paper, first run)21.4%13.3%26.8%13.6
Sequential: developing on top of a teammate’s committed changes19.4%10.8%25.2%14.4
Two independent sessions + a third agent merges20.4%12.0%26.0%14.0

All four workflows land within 2 points of each other, and in every workflow coupled pairs resolve at less than half the rate of the rest. The paper’s 28 configurations show the same pattern: 9.5% versus 22.4%.

The three challenges you list map onto this comparison:

ChallengeCorresponding settingResult
Stale contextsTwo independent sessions invisible to each other (fully stale contexts), versus sequential development (the second agent sees its teammate’s commits)12.0% and 10.8% on coupled tasks; seeing the teammate’s commits leaves the rate unchanged
Independently made implementation decisionsTwo independent sessions each implement one side and then merge, versus a single agent completing both jointlyBoth 12.0% on coupled tasks
Integration conflictsWhat the merger’s own git apply runs report (Opus 4.8 × Mini-SWE-Agent, first run)Both applications raise no error on 94% of coupled and 96% of non-coupled pairs; after clean application, 11.5% versus 30.5% resolve

The difficulty sits in the task itself: the dependency hides outside both sides’ requirements and code, survives a clean merge, and the agent must recognize it on its own in every workflow.

W2

Insufficient isolation of concurrency-specific difficulty.

You raise two possibilities: the low Resolved Rate may come from task complexity rather than concurrency, and the reported cross-PR dependencies may come from tests tied to implementation details. We check both, with a size-controlled comparison and a case-by-case audit of the coupling. A pair counts as coupled when one side’s gold patch alone fails that side’s own hidden tests. With task size controlled, coupled pairs still resolve at a lower rate. The 83 coupled pairs are similar in size to the rest: 16.4 non-test code files on average, versus 17.8. Splitting all 28 configurations by code-file count and by overlap type:

GroupCoupled groupRemaining tasksDifference (pp)
1–7 code files13.2% (14 pairs)32.0% (39 pairs)18.8
8–11 code files11.1% (22 pairs)21.0% (28 pairs)9.9
12–18 code files1.1% (19 pairs)20.9% (26 pairs)19.8
≥19 code files11.9% (28 pairs)12.5% (30 pairs)0.5
Source-file overlap14.9% (22 pairs)31.4% (33 pairs)16.5
Non-source-file overlap only6.0% (20 pairs)10.8% (18 pairs)4.8
No file overlap8.2% (41 pairs)21.2% (72 pairs)12.9

The coupled group resolves lower in every row of the table, and the size-weighted gap is 11.7 percentage points. Only among the largest tasks (≥19 code files) do the two groups come close; in every other row the gap is at least 4.8 points. The nginx case does carry a dependency. With either side applied alone, what is missing is exactly the other PR’s change: the Host header that PR B adds, and the [] versus [""] value from PR A. The location the test expects is not arbitrary either: the shared header list is where the repository already handles request headers, and PR B’s own requirement that requestHeaders.set override the default points to the same place. Descheduler shows the same at build time: the generated conversion code refers to the fields that both PRs add, so with either side alone the build stops. Q3 below breaks down where all 83 dependencies come from.

AuthorsOpenReview ↗

Response by Authors (2/2)

Questions and Comments

Q1

What capabilities does CoDev-Bench measure beyond jointly solving two coding tasks? How does the formulation capture the unique challenges of concurrent development?

Beyond solving two tasks, CoDev-Bench measures whether an agent recognizes where two independently developed changes meet in the repository and keeps them consistent there. The task provides only the base repository and the two PRs’ requirements, so these meeting points are left for the agent to find. Two results show that the difficulty is not just doing two tasks at once. When each PR can be evaluated on its own (92 pairs), one agent implementing both PRs resolves only 1.1–3.3 points fewer pairs than two agents implementing them separately (Q2). When the two PRs are coupled, the Resolved Rate falls from 22.4% to 9.5% over the paper's 28 configurations, and from 25.2–26.8% to 10.8–13.3% in the four workflows of W1.

Q2

Can the authors compare independent per-PR success rates with joint-resolution rates under comparable budgets to quantify the additional difficulty?

Thank you for suggesting this comparison. Per-side success is defined only where each side can be scored alone, which holds for 92 of the 206 pairs. These 92 exclude all 83 coupled pairs, so the comparison below leaves out the extra difficulty of cross-PR dependencies. On these pairs we compare implementing the two PRs separately (two workers, 250 steps / 5 USD each) with implementing them jointly (one agent, 350 steps / 10 USD ) for the two strongest models under Mini-SWE-Agent, at the same 10 USD total cost cap. Joint completion is 3.3 (Opus) and 1.1 (Grok) points lower.

ModelA alone resolvedB alone resolvedImplemented separately, both passImplemented jointly, both passDifference (pp)
Opus 4.848.9%46.7%27.2%23.9%+3.3
Grok 4.545.7%48.9%26.1%25.0%+1.1

The remaining 114 pairs (the 83 coupled pairs and 31 whose tests span both sides) resolve jointly at rates 7.2–9.2 points lower:

Model92 independently evaluable pairsRemaining 114 pairsRemaining − evaluable (pp)
Opus 4.823.9%16.7%−7.2
Grok 4.525.0%15.8%−9.2

Q3

Among the 83 pairs exhibiting cross-PR test dependencies, how many involve genuine functional dependencies rather than implementation-specific test expectations?

Thank you for suggesting a finer distinction among the sources of cross-PR dependencies. The two PRs of a pair are developed independently from the same base commit, so one PR’s functionality does not build on the other’s; a dependency between them appears only where the two changes meet in the repository. To locate these meeting points, we read both sides’ gold patches and hidden tests for each of the 83 pairs.

Source of the dependencyPairsShareExample
Changes to shared code or interfaces1821.7%Playwright: both PRs edit toHaveScreenshot in toMatchSnapshot.ts
Synchronization of generated code and companion files44.8%Descheduler: zz_generated.conversion.go references the fields added by both PRs
Shared tests depending on both sides’ changes4959.0%Locust: test_runners.py uses PR A’s RPCError and PR B’s test_start event
Joint expectations over complete outputs1214.5%nginx-ingress: tests compare the complete configuration generated by virtualserver.go (see W2)

In all four cases the two PRs meet in a part of the repository they share, and that part has to be consistent with both changes at once. The hidden tests pass the fairness audit of Appendix C, which discards pairs whose assertions mostly check implementation details. These 83 pairs resolve at 9.5% across the paper’s 28 configurations, against 22.4% for the other 123.


Thank you again for your thoughtful feedback. Please feel free to let us know if anything remains unclear or if you have further questions.

Thanks for your response!

Q1: For the 49 shared-test-dependent pairs, how many reflect genuine functional interactions between the PRs, rather than tests that simply reference both independently implemented features? Does the performance gap persist after excluding test-induced dependencies?

Q2: Does "implemented separately, both pass" mean that the two independently generated patches also pass the full combined test suite after merging? If not, could the authors report this comparison under the same evaluation criterion?

Reviewer_KKu4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work aims to evaluate agents under concurrent development scenarios. The authors designed a benchmark with each task including a pair of PRs. Evaluation is performed with multiple development mode covering both single agent and main+subagents. The experimental section present analysis from different perspectives and dimensions.

Critical strengths
  1. Using PR pairs is a reasonable idea to evaluate concurrent development.
  2. The experiments involve multiple strategies for merge patches in concurent development and performed analysis from different perspectives/dimensions.
Critical weaknesses
  1. The biggest challenge in concurrent development is potential conflict among different PRs. But this work does not provide a particular and comprehensive analysis of conflicts among PRs and how they would affect the task completion and how models/agents perform in handling conflicts. Moreover, the average number of overlapped files is only 1.9.
  2. The benchmark assumes one merge of two PRs. But in practical development, there will be concurrent feature branches that involve multiple PRs and merges. The realism of the benchmark is limited. There is still a large gap between this benchmark and "full complexity of concurrent development".
  3. The evaluation does not will simulate the real workflow of concurrent development, i.e. one will write code and constantly pull latest changes by others. In cases of using multiple agent, developers also tend to use two independent windowed sessions rather than using subagents. The two agents percept each other through the repostory history and handle code merging themselves rather than let a main agent to do the integration. Of course, the subagent paradigm is indeed one of the options.
  4. It would be better if more PRs could be considered.
Questions for authors

For figure 6b, what are the reasons that some task combinations are harder for agents?

5 comments
AuthorsOpenReview ↗

Response by Authors (1/2)

Thank you for the concrete suggestions on conflict analysis and development workflows. Below we respond point by point, with grouped statistics and behavioral analyses of cross-PR conflicts, and new experiments on three-PR tasks and alternative workflows.

Weaknesses

W1

The biggest challenge in concurrent development is potential conflict among different PRs. But this work does not provide a particular and comprehensive analysis of conflicts among PRs and how they would affect the task completion and how models/agents perform in handling conflicts. Moreover, the average number of overlapped files is only 1.9.

File overlap is the most visible signal of conflict, and the 1.9 average reflects the two-PR setting. To cover heavier overlap, we newly mine three-PR concurrent tasks: in the 62 tasks we build (28 repositories), tasks with overlap average 6.58 overlapped files.

Following this line, we analyze what overlap does and does not capture. Each pair is checked at seven levels, from textual to semantic (a pair can match multiple):

LevelCriterionOf 206 pairsSingle Agent resolvedLLM Merge resolved
Textualgit merge-tree reports a conflict1 (0.5%)0/1 (0%)0/1 (0%)
FileThe two PRs modify the same file93 (45.1%)19/93 (20%)20/93 (22%)
Source fileThe overlap touches source code55 (26.7%)15/55 (27%)17/55 (31%)
LineNearest edits ≤10 lines apart18 (8.7%)5/18 (28%)6/18 (33%)
SymbolSame named declaration modified34 (16.5%)10/34 (29%)10/34 (29%)
ReferenceOne side imports a file the other modifies105 (51.0%)23/105 (22%)25/105 (24%)
SemanticOne side’s gold patch alone fails its own hidden tests83 (40.3%)11/83 (13%)10/83 (12%)
(All)20641/206 (20%)47/206 (23%)

The table gives two clear results. First, the 93 overlapping pairs resolve at the benchmark average: 20% / 22% (Single Agent / LLM Merge) versus 20% / 23% overall. Second, the 83 pairs at the semantic level, which we call coupled, resolve at half the rate of the rest: 13% / 12% versus 24% / 30%, and 41 of the 83 share no file, where overlap gives no warning. Git-level conflicts are rare because the gold patches already include the maintainers’ conflict resolutions.

How agents handle these conflicts: they handle what they can see and miss what they cannot.

We read all trial-1 trajectories (Opus 4.8 + Mini-SWE-Agent). For the single agent: when the two PRs share files, it often co-edits both sides in one command. When the two PRs are coupled but share no file, it never co-edits. For the merger agent in split-and-merge mode: it edits source almost only when patch application fails or conflict markers appear. When the two patches apply cleanly but the pair is coupled, it merges them as if they were independent. Coupled pairs that share no file end with the lowest resolve rate of any group.

W2

The benchmark assumes one merge of two PRs. But in practical development, there will be concurrent feature branches that involve multiple PRs and merges. The realism of the benchmark is limited. There is still a large gap between this benchmark and "full complexity of concurrent development".

Thanks for the valuable insight: real development can indeed involve more concurrent PRs, so we measure how often larger groups occur. Under the paper’s mining and candidate filters (Table 4), the supply falls fast: 19,716 two-PR groups, 332 three-PR groups, 68 four-PR groups. Within this mined and filtered pool, two- and three-PR groups cover over 99% of the concurrent groups. Groups of four or more PRs are too sparse to support a representative analysis. As a new addition for this rebuttal, we build 62 of the 332 three-PR windows into runnable tasks (28 repositories) and evaluate them end to end.

The table lists the three-PR results and pass rates. The setup keeps the two-PR protocol: Opus 4.8 + Mini-SWE-Agent, Single Agent and merger budgets raised by 50%, Git Merge and LLM Merge sharing the same worker patches.

MethodImplementation and integrationResolved
Single Agentone agent completes all three requirements14/62 (22.6%)
Git Mergethree workers implement separately; Git integrates11/62 (17.7%)
LLM Mergethree workers implement separately; a merger agent integrates11/62 (17.7%)

All three methods stay below a quarter of tasks resolved, the same band as the two-PR setting. The challenge carries over. Foundry shows the familiar failure shape. Its three PRs add compiler selection, a compilation progress display, and structured JSON output for inspect. The merger confirms all three are merged in. But the last two both write to stdout: the progress animation and a Compiling... line end up inside the JSON, and the test that expects clean output flips from pass to fail.

AuthorsOpenReview ↗

Response by Authors (2/2)

W3

The evaluation does not will simulate the real workflow of concurrent development, i.e. one will write code and constantly pull latest changes by others. In cases of using multiple agent, developers also tend to use two independent windowed sessions rather than using subagents. The two agents percept each other through the repostory history and handle code merging themselves rather than let a main agent to do the integration. Of course, the subagent paradigm is indeed one of the options.

You describe two real workflows: developers write code and pull the latest changes made by others, or open two independent windowed sessions instead of sub-agents. These are the natural settings, so we build one experiment for each with Copilot CLI (Opus 4.8) on the same 206 pairs.

The first workflow is sequential development on top of a teammate’s work. Agent A completes PR A, and its changes are committed into the repository. Agent B then starts inside that repository: it reads the teammate’s changes through git log and git show, implements PR B on top of them, and handles the interaction between the two itself.

The second workflow is two independent sessions with the merge kept outside any main agent. Two CLI sessions each implement one PR from the same base, with no visibility into each other. A third CLI session then merges the two patches; its instructions are identical to the paper’s merger, and only the executor changes from Mini-SWE-Agent to the CLI.

The table lists the resolved pairs of the two new workflows next to the paper’s two settings:

WorkflowResolved
Single Agent (paper setting, first run)42/206 (20.4%)
Sub-Agent (paper setting, first run)44/206 (21.4%)
Sequential, developing on top of a teammate’s commits (Experiment 1)40/206 (19.4%)
Two independent sessions, a third agent merges (Experiment 2)42/206 (20.4%)
Union of all workflows’ resolved pairs53/206 (25.7%)

All four workflows land within 2 points of each other. On the 83 coupled pairs they stay between 10.8% and 13.3%, versus 25.2% to 26.8% elsewhere. The tasks stay just as hard under the more realistic workflows, and the gap between coupled and other pairs shows up in every one of them.

W4

It would be better if more PRs could be considered.

Thanks for your suggestion. Indeed, We extend our benchmark to three-PRs. The three-PR extension in our W2 reply addresses this directly: Single Agent resolves 14/62 (22.6%), Git Merge 11/62 (17.7%), and LLM Merge 11/62 (17.7%), the same band as the two-PR setting. The supply statistics, setup, and the Foundry case are in W2.

Questions and Comments

Q1

For figure 6b, what are the reasons that some task combinations are harder for agents?

Two things shape the differences in Figure 6b. Coupled pairs are harder across the benchmark, and some pairs carry one hard side; with only 5 to 15 pairs in seven of the nine groups, a few pairs decide a group’s rate. We mark each pair as coupled (the semantic level in W1) and as ever resolved by any model, configuration, or trial:

CombinationNCoupled pairsCoupled ever resolvedNon-coupled ever resolvedMini single-agent Resolved Rate
bugfix+perf530/32/223.3%
bugfix+feature42145/1417/2820.0%
feature+refactor1553/56/1018.3%
refactor+refactor621/22/418.1%
feature+feature713110/3118/4017.0%
bugfix+refactor751/51/215.5%
bugfix+bugfix1340/45/99.6%
feature+perf710/13/68.3%
feature+removal951/51/40.0%
Whole benchmark (206 pairs)2068325/83 (30%)64/123 (52%)16.8%

Coupled pairs are ever resolved less often than non-coupled ones: 30% versus 52% (bottom row). feature+removal sits at 0% for the Mini single agent: 5 of its 9 pairs are coupled, and only 2 pairs are ever resolved by anyone. bugfix+perf tops the list although 3 of its 5 pairs are coupled, because its 2 non-coupled pairs both succeed.

Other failures come from one hard side. In docker/cli, the hard side reworks the plugin manager’s public API, un-exporting its Candidate interface and reclassifying plugin-not-found errors; agents resolve the registry side alone, and fail this side at the same point alone and in the joint run.


We sincerely appreciate your suggestions and would be glad to continue the discussion if any points require further clarification.

Dear Reviewer KKu4,

We appreciate that you consider our work as a reasonable way to evaluate concurrent development, with experiments that cover multiple merge strategies and analyses from different perspectives.

We would love to know whether our response has addressed your concerns regarding the introduction of more PRs, the increase in the number of files involved in overlapping conflicts, the analysis of cross-PR conflicts and concurrent development workflows. We would also be happy to continue the discussion and answer any further questions you may have.

Best Regards,

Authors

Reviewer_ucoYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a benchmark where models are evaluated on implementing two concurrent SWE tasks; the two tasks ask for implicitly coupled functionality. Four frontier models are evaluated on the benchmark and sources of their failures are analyzed. The agents are tested in a variety of configuration.

It took me time to convince myself that the presented tasks are genuinely harder than SWE-bench-like tasks of equal size. What convinced me is realizing that the agent-generated edits do not have human-written golden unit tests: the agent must merge two edits without falling back on trustworthy feedback from the test suite, which it never needs to do on SWE-bench tasks.

Critical strengths
  1. The benchmark is timely, given the increasing evidence for the importance of agents swarms collaboratively developing a codebase.
  2. The task is relatively challenging for frontier models, revealing a capability gap. Highest pass rate is 21.8%, and 57% of the tasks are not solved by any agent configuration.
  3. The benchmark is built well. The authors use naturally occuring data which is both diverse (in six programming languages) and difficult (from production-scale repositories). The tasks were human-audited.
Critical weaknesses
  1. The intuition that implicit semantic coupling is the reason for the benchmark's difficult is not supported by an analysis.
  2. Frontier models are not far from solving most tasks on which they fail (median of 1 to 2 failed tests).
  3. Some frontier models are not benchmarked in their native harness (eg Sol), which may make them underperform given the models are likely trained for their native harness.
Questions for authors
  • Sec 3.2: how was the problem statement created? Is it generated, or human-written?
  • Sec 4.1: please briefly describe the LLM merge strategy and clarify that the merge agent is prevented from running tests (as per the appendix description). This is unexpected given that the paper explains at least part of the difficulty comes from insufficient tests.
  • Please correct the cites in the text which attribute the authors as "Team" (e.g. L38) - the team should be named.
  • Please correct links on p2 and p3: the header, footer, and left margin are hyperlinks.
3 comments
AuthorsOpenReview ↗

Response by Authors (1/2)

Thank you for the positive assessment of the benchmark’s relevance, data quality, and difficulty. Below we respond point by point, with new analyses of cross-PR dependencies and agent behavior, and new experiments on a newer model, native harnesses, and the merger’s test access.

Weaknesses

W1

The intuition that implicit semantic coupling is the reason for the benchmark's difficult is not supported by an analysis.

To test whether implicit coupling drives the difficulty, we split the 206 pairs with a per-side check (Appendix D.1): apply one PR’s gold patch alone and run that side’s hidden tests; when they fail, the two PRs are coupled. This gives 83 coupled pairs and 123 remaining.

We then compare Resolved Rates, averaged over all 28 configurations:

ModelCoupled (83 pairs)Remaining (123 pairs)
Opus 4.811.7%27.7%
Grok 4.510.7%25.6%
GPT-5.6-Sol8.3%19.6%
Kimi K2.67.2%16.7%
All 28 configurations9.5%22.4%

Coupled pairs resolve at 9.5% versus 22.4%, and every model shows the gap. Coupled pairs touch similar amounts of code (16.4 non-test code files on average, versus 17.8 for the rest); to check size and overlap effects, we compare within each group:

GroupCoupled groupRemaining tasksDifference (pp)
1–7 code files13.2% (14 pairs)32.0% (39 pairs)18.8
8–11 code files11.1% (22 pairs)21.0% (28 pairs)9.9
12–18 code files1.1% (19 pairs)20.9% (26 pairs)19.8
≥19 code files11.9% (28 pairs)12.5% (30 pairs)0.5
Source-file overlap14.9% (22 pairs)31.4% (33 pairs)16.5
Non-source-file overlap only6.0% (20 pairs)10.8% (18 pairs)4.8
No file overlap8.2% (41 pairs)21.2% (72 pairs)12.9

The gap holds in every group, including the 113 pairs with zero file overlap.

To see how agents handle the dependency, we record whether each run points out the connection between the two sides (Opus 4.8 × Mini-SWE-Agent, 206 single-agent and 206 merger runs):

Pairs split byAgentWithWithoutRatio
file overlap (gold patches)single agent22.6% (21/93)8.8% (10/113)2.6×
file overlap (worker patches)merger63.6% (35/55)1.3% (2/151)49×
couplingsingle agent18.1% (15/83)13.0% (16/123)1.4×
couplingmerger19.3% (16/83)17.1% (21/123)1.1×

Agents notice the visible cue and miss the dependency itself.

W2

Frontier models are not far from solving most tasks on which they fail (median of 1 to 2 failed tests).

You note that failed runs typically miss only 1–2 tests, and suggest the models are not far from solving these tasks. We check this with a stronger model and by inspecting what those failing tests contain; both checks point the same way: a small count of failing tests marks a hard missing piece, rather than finishing work on a nearly-solved task.

We evaluate Claude Opus 5.5 under the paper’s best configuration (Copilot CLI Sub-Agent, reasoning effort high, $30 cost cap per pair, all 206 pairs):

ModelResolved RateAverage cost per pair
Claude Opus 4.8 (paper)21.8%$5.8
Claude Opus 5.5 (new)37.9% (78/206)$3.6

Opus 5.5 lifts the Resolved Rate to 37.9%, and the pattern stays: across its 85 wrong_answer runs, the median failing-test count is still 1. Being one test away also predicts little: of the 59 pairs Opus 4.8 missed by exactly one test in this configuration (first run), Opus 5.5 resolves 15 (25%).

The nginx/kubernetes-ingress case (Appendix E) shows what one missing test hides. PR A’s tests compare the complete generated configuration, which includes the default Host header added by PR B. Some agents emit that header through a separate template field instead of the shared header list the tests inspect, so the diff shows a single missing Host entry; the fix requires discovering that PR B’s implementation choice affects PR A’s validation, and coordinating the two sides’ edits to the same configuration.

AuthorsOpenReview ↗

Response by Authors (2/2)

W3

The benchmark is built well. The authors use naturally occuring data which is both diverse (in six programming languages) and difficult (from production-scale repositories). The tasks were human-audited.

We agree that a model may do better in its native harness, so we run a full evaluation of GPT-5.6-Sol in its native Codex CLI (v0.166.0) on all 206 pairs: one agent completes both PRs, reasoning effort medium as in the paper’s Sol configuration, with a $30 per-pair cost cap as in the paper’s two CLI baselines. All 206 pairs complete patch generation:

Harness (all GPT-5.6-Sol, Single Agent)Resolved RateAverage cost per pair
Mini-SWE-Agent (paper)15.7% ± 1.0%$1.1
Claude Code CLI (paper)14.6% ± 0.5%$7.2
Copilot CLI (paper)14.6% ± 1.3%$2.0
Codex CLI (new, native harness)14.08% (29/206)$1.62

The native harness lands in the same band: 29/206 pairs (14.08%), inside the 27–32-pair range of the paper’s two CLI baselines for this model (Claude Code resolves 30, 29, and 31 pairs across runs; Copilot 32, 31, and 27). Every task finishes within the cost cap, at $1.62 per pair on average, cheaper than both CLI baselines.

Questions and Comments

Q1

Sec 3.2: how was the problem statement created? Is it generated, or human-written?

Problem statements are generated by GPT-5.5 and then human-verified. Three passes produce each statement: an explorer distills the functional intent from the gold diff, a writer drafts the description from the public PR context, and a judge checks granularity and requests rewrites; every statement is reviewed and corrected by human experts before entering the dataset.

Q2

Sec 4.1: please briefly describe the LLM merge strategy and clarify that the merge agent is prevented from running tests (as per the appendix description). This is unexpected given that the paper explains at least part of the difficulty comes from insufficient tests.

In LLM Merge, a third agent receives the two workers’ patches, applies them to the same baseline repository, resolves conflicts and incompatibilities between the two sides, and produces the unified patch; its instructions limit it to integration and forbid installing dependencies, compiling, and running tests (Appendix F). To check what the test restriction costs, we re-run the merger stage on all 206 pairs with the same worker patches (Opus 4.8 × Mini-SWE-Agent, first run), now allowing the repository’s existing tests and compilation; everything else is unchanged:

Merger settingResolved pairsAverage stepsTotal cost (206 pairs)
Original: running tests forbidden47/2066.6$18.14
Ablation: existing tests allowed47/20615.2$50.52

Both settings resolve exactly the same 47 pairs: the merger runs targeted tests on 142/206 pairs, yet the final patches stay byte-identical on 205/206, with steps and cost at 2.3× and 2.8×. What is missing is not permission to test but the agent’s own ability to verify: the tests it chooses to run do not reveal how the two sides’ changes interact, so it changes nothing.

Q3&Q4

Please correct the cites in the text which attribute the authors as "Team" (e.g. L38) - the team should be named.

Please correct links on p2 and p3: the header, footer, and left margin are hyperlinks.

Thank you for flagging these formatting issues. We correct both: the L38 citations now read “FAIR CodeGen team et al., 2025” and “Kimi Team et al., 2026”, and the hyperlink boxes over the header, footer, and left margin on pages 2–3 are removed.


Thank you once again for your helpful suggestions. We are happy to continue the discussion and address any further questions you may have.

Area_Chair_XCXLOpenReview ↗

Reminder from AC

Dear Reviewer ucoY,

As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Paper Revision

We thank all reviewers for their constructive feedback, and we have responded to each reviewer individually. We have also uploaded a Paper Revision (changes marked in blue) including additional analyses and evaluations:

Sections 1–2, pages 2–3: We correct the two team-author citations and remove the hyperlink boxes over the header, footer, and left margin.

Section 3.2, Appendix F: We describe how the problem statements are generated and reviewed, and the instructions of the LLM Merger.

Section 4.2.3, Appendix D.1.2, Tables 7–8: We add a comparison of implementing the two PRs separately and jointly under the same total budget, on the 92 pairs where each side can be scored alone, under two criteria: each worker scored on its own side, and the two patches merged and scored on the full test suite.

Section 4.3, Appendix D.2, Table 9: We add seven levels of overlap between the two PRs, from textual to semantic, with the Resolved Rate at each level.

Section 4.3.1, Appendix D.3, Tables 10–13: We add an analysis of the 83 coupled pairs: where the two PRs meet in the repository, their Resolved Rate against the other 123 pairs by model and within groups of task size and file overlap, and the task combinations of Figure 6b split by coupling.

Section 4.4.1, Appendix D.4.2, Tables 15–16: We add a Merger test-access ablation and two further workflows on Copilot CLI: sequential development on top of a teammate’s commit, and two independent sessions merged by a third.

Appendix D.5, Tables 17–18: We add evaluations beyond the main matrix: Claude Opus 5.5, GPT-5.6-Sol in its native Codex CLI, and 62 three-PR tasks.

Appendix E, Table 19: We add a trajectory analysis of whether Single Agent and Merger runs point out where the two sides meet, split by file overlap and by coupling, and extend the nginx case study.

We hope our revisions have adequately addressed your concerns. If you have any further questions or would like us to clarify any points, please feel free to let us know. We would be happy to provide further explanations and continue the discussion.

AuthorsOpenReview ↗

Summary of Rebuttal

Dear Reviewers and Area Chair,

Thank you for your constructive feedback and recognition of the importance of evaluating agents on multi-PR development. During the rebuttal, we added experiments and analyses to address the main concerns about cross-PR interactions, task difficulty, scalability, and realistic development workflows. We summarize the key additions below.

  1. Cross-PR interactions and functional dependencies (Reviewers KKu4 and RTto). We manually examined all 83 test-dependent PR pairs, distinguishing 59 pairs with functional connections from 24 pairs whose dependence arises from test construction. The 59 functionally connected pairs have a lower average success rate than the 123 test-independent pairs: 11.9% vs. 22.4%, with the same direction across all 28 evaluated configurations. We also added analyses of conflicts at multiple levels and agent trajectories to explain how agents identify and handle interactions between changes. Coupled pairs consistently have lower success rates within groups of similar task size and file overlap, highlighting CoDev-Bench’s value in exposing the challenges of handling cross-PR interactions.

  2. Extending beyond two PRs (Reviewer KKu4). We added 62 three-PR tasks from 28 repositories. Single and Split+LLM solve 22.6% and 17.7%, respectively. These results support our use of two PRs as the smallest nontrivial setting for studying multi-requirement implementation and integration: the challenges it exposes persist when extending to more PRs.

  3. More realistic development workflows (Reviewers KKu4 and RTto). We evaluated sequential development and independent agent sessions followed by integration on the full benchmark, alongside the original Single and subagent workflows. Success rates across these four workflows remain between 19.4% and 21.4%, showing that the tasks remain challenging under several practical ways of organizing development.

  4. Joint development versus solving each side independently (Reviewer RTto). On the 92 pairs that support separate evaluation, we compared joint development with independently implementing both sides and then merging them, using the same final combined test suite. For the two stronger models, independent implementation followed by merging improves success from 23.9% to 29.3% for Opus and from 25.0% to 28.3% for Grok. These results help identify effective workflows for multi-PR development.

  5. Stronger models and the interpretation of near-success (Reviewer ucoY). We evaluated the newer Opus 5.5, which improves success from 21.8% to 37.9%, while still leaving most tasks unsolved. Among its wrong-answer runs, the median number of failed tests remains one. This shows both the benchmark’s ability to reflect model progress and why a small number of failed tests alone does not establish that the benchmark is close to saturation. We also added an evaluation using the native Codex CLI.

  6. Giving the merger access to tests (Reviewer ucoY). We evaluated this suggested change on all 206 tasks. The merger ran tests on 142 tasks, but the solved set remained unchanged at 47/206, and the final patches were identical on 205/206 tasks. These results highlight the difficulty of using tests effectively to verify cross-PR interactions.

These additional experiments and analyses address Reviewer KKu4’s concerns about cross-PR interactions, tasks involving more than two PRs, and realistic development workflows. They also address related concerns discussed with Reviewer RTto, who explicitly acknowledged that “the new evidence has strengthened my confidence in the experimental soundness.” Unfortunately, despite our follow-ups, we have not yet received Reviewer KKu4’s assessment of these additions.

We believe the new evidence substantially addresses the main concerns raised during review and strengthens the case for CoDev-Bench as a useful benchmark for multi-PR development. We hope these additions, together with the feedback received during the discussion, will be helpful in your final assessment.

Thank you again for your time and consideration.

Sincerely,

The Authors of Submission 10586

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.