- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
CoDev-Bench evaluates coding agents on pairs of pull requests (PRs) independently developed from a shared repository base. The benchmark contains 206 PR pairs from 137 open-source repositories across six languages. Given the base repository and two task descriptions, an agent must produce a unified patch that passes tests associated with both PRs. The authors evaluate four models using single-agent, split-and-merge, and runtime subagent approaches. The best configuration achieves a 21.8% resolution rate, with no consistent improvement from multi-agent approaches. Further analyses identify insufficient verification and implicit cross-PR dependencies as potential causes of failure. The paper presents naturally concurrent PR pairs as a challenging evaluation setting for coding agents.
Critical strengths
S1. Relevant and underexplored evaluation setting. The benchmark uses naturally concurrent PRs rather than synthetic tasks, capturing realistic software development requirements that are underrepresented in existing coding benchmarks.
S2. Comprehensive empirical analysis. The evaluation covers four models, multiple agent architectures, and three trials. The failure analysis provides useful insights into verification gaps, cross-PR dependencies, and limitations of existing multi-agent approaches.
Critical weaknesses
W1. Unclear construct validity of concurrent development. The primary task requires an agent to jointly implement two known requirements, rather than independently developing changes and subsequently integrating them. Thus, it mainly evaluates joint multi-task coding, without fully capturing concurrency-specific challenges such as stale contexts, independently made implementation decisions, and integration conflicts. The existing split-and-merge baselines only partially address this gap.
W2. Insufficient isolation of concurrency-specific difficulty. The low resolution rate may largely reflect increased task complexity rather than concurrency itself. Moreover, the reported cross-PR dependencies may partly arise from tests tied to specific implementation details (e.g., the nginx case in Appendix E). These issues weaken the evidence that the observed failures specifically result from concurrent development.
Questions for authors
Q1. What capabilities does CoDev-Bench measure beyond jointly solving two coding tasks? How does the formulation capture the unique challenges of concurrent development?
Q2. Can the authors compare independent per-PR success rates with joint-resolution rates under comparable budgets to quantify the additional difficulty?
Q3. Among the 83 pairs exhibiting cross-PR test dependencies, how many involve genuine functional dependencies rather than implementation-specific test expectations?
6 comments
Response by Authors (2/2)
Questions and Comments
Q1
What capabilities does CoDev-Bench measure beyond jointly solving two coding tasks? How does the formulation capture the unique challenges of concurrent development?
Beyond solving two tasks, CoDev-Bench measures whether an agent recognizes where two independently developed changes meet in the repository and keeps them consistent there. The task provides only the base repository and the two PRs’ requirements, so these meeting points are left for the agent to find. Two results show that the difficulty is not just doing two tasks at once. When each PR can be evaluated on its own (92 pairs), one agent implementing both PRs resolves only 1.1–3.3 points fewer pairs than two agents implementing them separately (Q2). When the two PRs are coupled, the Resolved Rate falls from 22.4% to 9.5% over the paper's 28 configurations, and from 25.2–26.8% to 10.8–13.3% in the four workflows of W1.
Q2
Can the authors compare independent per-PR success rates with joint-resolution rates under comparable budgets to quantify the additional difficulty?
Thank you for suggesting this comparison. Per-side success is defined only where each side can be scored alone, which holds for 92 of the 206 pairs. These 92 exclude all 83 coupled pairs, so the comparison below leaves out the extra difficulty of cross-PR dependencies. On these pairs we compare implementing the two PRs separately (two workers, 250 steps / 5 USD each) with implementing them jointly (one agent, 350 steps / 10 USD ) for the two strongest models under Mini-SWE-Agent, at the same 10 USD total cost cap. Joint completion is 3.3 (Opus) and 1.1 (Grok) points lower.
| Model | A alone resolved | B alone resolved | Implemented separately, both pass | Implemented jointly, both pass | Difference (pp) |
|---|---|---|---|---|---|
| Opus 4.8 | 48.9% | 46.7% | 27.2% | 23.9% | +3.3 |
| Grok 4.5 | 45.7% | 48.9% | 26.1% | 25.0% | +1.1 |
The remaining 114 pairs (the 83 coupled pairs and 31 whose tests span both sides) resolve jointly at rates 7.2–9.2 points lower:
| Model | 92 independently evaluable pairs | Remaining 114 pairs | Remaining − evaluable (pp) |
|---|---|---|---|
| Opus 4.8 | 23.9% | 16.7% | −7.2 |
| Grok 4.5 | 25.0% | 15.8% | −9.2 |
Q3
Among the 83 pairs exhibiting cross-PR test dependencies, how many involve genuine functional dependencies rather than implementation-specific test expectations?
Thank you for suggesting a finer distinction among the sources of cross-PR dependencies. The two PRs of a pair are developed independently from the same base commit, so one PR’s functionality does not build on the other’s; a dependency between them appears only where the two changes meet in the repository. To locate these meeting points, we read both sides’ gold patches and hidden tests for each of the 83 pairs.
| Source of the dependency | Pairs | Share | Example |
|---|---|---|---|
| Changes to shared code or interfaces | 18 | 21.7% | Playwright: both PRs edit toHaveScreenshot in toMatchSnapshot.ts |
| Synchronization of generated code and companion files | 4 | 4.8% | Descheduler: zz_generated.conversion.go references the fields added by both PRs |
| Shared tests depending on both sides’ changes | 49 | 59.0% | Locust: test_runners.py uses PR A’s RPCError and PR B’s test_start event |
| Joint expectations over complete outputs | 12 | 14.5% | nginx-ingress: tests compare the complete configuration generated by virtualserver.go (see W2) |
In all four cases the two PRs meet in a part of the repository they share, and that part has to be consistent with both changes at once. The hidden tests pass the fairness audit of Appendix C, which discards pairs whose assertions mostly check implementation details. These 83 pairs resolve at 9.5% across the paper’s 28 configurations, against 22.4% for the other 123.
Thank you again for your thoughtful feedback. Please feel free to let us know if anything remains unclear or if you have further questions.
Thanks for your response!
Q1: For the 49 shared-test-dependent pairs, how many reflect genuine functional interactions between the PRs, rather than tests that simply reference both independently implemented features? Does the performance gap persist after excluding test-induced dependencies?
Q2: Does "implemented separately, both pass" mean that the two independently generated patches also pass the full combined test suite after merging? If not, could the authors report this comparison under the same evaluation criterion?
Response by Authors (1/2)
Thank you for the careful questions. Below we respond point by point: we clarify what capability the benchmark measures, compare independent with joint completion, and trace where cross-PR dependencies come from.
Weaknesses
W1
Concurrency in CoDev-Bench lives in the task structure rather than in the solving procedure. To test this, we add two workflows that realize the challenges you list, and compare all four workflows on the same 206 pairs (Copilot CLI, Opus 4.8).
In the sequential workflow, agent A’s PR is committed first; agent B reads it through the git history and builds PR B on top, handling the interaction itself. In the independent workflow, two sessions implement one PR each without seeing each other, and a third session merges them with the paper’s merger instructions. The coupled / non-coupled split below uses the test from W2: one side’s gold patch alone fails its own hidden tests. The table lists each workflow’s resolved rate, overall and by coupling:
All four workflows land within 2 points of each other, and in every workflow coupled pairs resolve at less than half the rate of the rest. The paper’s 28 configurations show the same pattern: 9.5% versus 22.4%.
The three challenges you list map onto this comparison:
git applyruns report (Opus 4.8 × Mini-SWE-Agent, first run)The difficulty sits in the task itself: the dependency hides outside both sides’ requirements and code, survives a clean merge, and the agent must recognize it on its own in every workflow.
W2
You raise two possibilities: the low Resolved Rate may come from task complexity rather than concurrency, and the reported cross-PR dependencies may come from tests tied to implementation details. We check both, with a size-controlled comparison and a case-by-case audit of the coupling. A pair counts as coupled when one side’s gold patch alone fails that side’s own hidden tests. With task size controlled, coupled pairs still resolve at a lower rate. The 83 coupled pairs are similar in size to the rest: 16.4 non-test code files on average, versus 17.8. Splitting all 28 configurations by code-file count and by overlap type:
The coupled group resolves lower in every row of the table, and the size-weighted gap is 11.7 percentage points. Only among the largest tasks (≥19 code files) do the two groups come close; in every other row the gap is at least 4.8 points. The nginx case does carry a dependency. With either side applied alone, what is missing is exactly the other PR’s change: the
Hostheader that PR B adds, and the[]versus[""]value from PR A. The location the test expects is not arbitrary either: the shared header list is where the repository already handles request headers, and PR B’s own requirement thatrequestHeaders.setoverride the default points to the same place. Descheduler shows the same at build time: the generated conversion code refers to the fields that both PRs add, so with either side alone the build stops. Q3 below breaks down where all 83 dependencies come from.