- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ParallACT, a benchmark for evaluating decentralized collaboration among coding agents working on a shared repository. It includes complementary tasks requiring interface agreement and contradictory tasks requiring reconciliation of incompatible edits. The authors compare single-agent, sequential, parallel, peer-aware, communication, and reconciliation strategies.
The experiments show that parallel agents achieve substantially lower joint success than a single-agent oracle with access to both requirements. Communication and synchronization recover part of the performance gap but generally incur higher token costs. The failure analysis suggests that coordination failures, rather than individual implementation failures, are the dominant bottleneck.
Critical strengths
Realistic problem formulation. The paper distinguishes centralized multi-agent planning from independently motivated agents operating in a shared codebase. Comprehensive evaluation. It evaluates joint success, partial completion, communication strategies, and token cost rather than only reporting final pass rates. Useful failure analysis. Many failures arise from interface mismatches, conflicting edits, and unclear ownership rather than from inability to implement the individual tasks.
Critical weaknesses
Limited dataset scale and synthetic construction. ParallACT contains only 85 tasks, and many complementary or contradictory requirements are authored by the researchers or GPT-5.6 Sol. The results may not represent the conflict distribution of real software teams. Selection bias in the cross-model evaluation. DeepSeek and GLM are evaluated only on 40 instances solved by the Claude oracle, so their results are not representative of the full benchmark. The oracle comparison is not fully controlled. The oracle sees both requirements, while parallel agents initially see only their own requirement. The gap therefore mixes coordination difficulty with information-access asymmetry. Communication strategies are somewhat artificial. File, Redis, messaging, and synchronization conditions depend on specific prompts and read/write mechanisms, so their conclusions may not transfer directly to realistic issue, PR, or code-review workflows.
Questions for authors
How many GPT-generated silver instances, tests, and reconciliation solutions were manually reviewed, and what was the agreement between reviewers? Why is the cross-model evaluation restricted to 40 instances solved by the Claude oracle? Can results on the full benchmark or difficulty-stratified subsets be reported? Can the authors add an information-matched oracle in which both parallel agents see both requirements but must still coordinate independently? Are the contradictory requirements grounded in real development histories, or are most conflicts manually constructed?
7 comments
W3: Oracle comparison.
We believe the suggested control you describe is exactly the communication oracle we have (Section 3.2): it extends parallel messaging by giving each agent its partner's requirement verbatim, so the two scenarios differ only in that information. With Claude Code + Opus 4.8, the communication oracle reaches 56% / 83% (contradictory / complementary), close to the single-agent oracle's 62% / 86%, against 20% / 63% for parallel messaging; this difference measures the information-access asymmetry you describe. In the multi-principal setting, however, this asymmetry is part of the problem rather than an artifact of the comparison: each agent knows only its own principal's requirement, and the communication scenarios give the agents a channel through which they can obtain the partner's requirement themselves. Yet the best channel reaches only 32% on contradictory instances, against 56% for the communication oracle: as Section 4.2 shows, the agents discuss the right code and often agree on a design, but their agreements do not prevent conflicting edits. Section 3.2 describes both oracles as diagnostic upper bounds rather than deployable strategies.
To isolate what coordination costs when information is matched, we added a new scenario, communication oracle with gold tests (Appendix K.1): with both requirements and unlimited access to their own gold tests, two agents reach 84%, against 98% for a single agent with the same information and tests (). Since both scenarios have the same information, the remaining difference comes from the two agents having to coordinate their edits.
W4: The communication strategies may not transfer to realistic workflows.
We ran two new scenarios built on realistic workflows: Git and Git + PR review (Claude Code + Opus 4.8, the original 85 instances; setup, results and analysis in Appendix K.2, Table 24; prompts in Appendix M.1). Each agent implements its requirement in parallel on its own branch; the branch that is finished first is merged into the main branch, and the other agent then pulls the updated main branch, resolves any conflicts, and merges. In Git + PR review, each agent opens a pull request and reviews the other's, and the author revises for up to two rounds before merging. Git reaches 68% / 83% (contradictory / complementary), comparable to the single-agent oracle (62% / 86%; on contradictory instances) and far above parallel-unaware (16% / 57%), and PR review does not change this (58% / 80%; not significantly different from Git).
These workflows coordinate the agents through a human-designed tool rather than through communication. The agents have no channel through which to talk to each other and exchange information only through interfaces built for code (commits, merges and reviews), so the need to communicate is offloaded to the tool. For code, this is a practical solution, but it does not show whether agents can coordinate by communicating, which is what ParallACT is designed to test and what settings without such tools require. Our channel scenarios are therefore deliberately minimal: a plain send/receive channel with no imposed workflow, so that any coordination must come from what the agents communicate.
The channel results do not depend on a specific prompt or mechanism. The four variants share an identical send/receive interface and differ only in medium (shared file or Redis) and framing (free-form messaging or synchronization), and the pattern holds across both harnesses and all six open-weight configurations: on contradictory instances, no channel variant recovers more than 43% of the gap between parallel-unaware and the oracle in any configuration or run.
Q1: How many of the generated dataset instances were manually reviewed?
All of them. One author reviewed every one of the 128 generated instances by hand, including their tests and reconciliation solutions, as well as every seed instance (Section 3.1; Appendix B). Before this review, each generated instance goes through three automatic stages.
- Construction: requirement A is always a real SWE-Bench Pro task, and requirement B is written as a separately motivated ticket that a different developer could receive, with candidates spread across languages, repositories and subsystems before generation.
- Executable soundness checks: we run every instance in its original SWE-Bench Pro environment, where each reference solution must pass its own tests alone and the pair must interact exactly as its family requires; instances that still fail after two correction rounds are discarded. These checks verify each conflict without relying on reviewer judgment.
- Independence audit: a separate GPT 5.6 Sol pass enforces our blind rule (no problem statement mentions the other requirement, its files or its tests) and rejects pairs that read as two halves of one decomposed task.
Q2: Selection bias in the cross-model evaluation.
We restricted it only to limit compute; the cross-model evaluation now covers the full benchmark, and the six open-weight configurations run on all 155 instances (Table 2; Appendix D). We report two difficulty-stratified views. On the 19 contradictory instances that Opus's oracle fails, the six open-weight configurations' oracles reach 21–63%, while their parallel-unaware runs reach only 0–5%. The 70 new instances are also harder than the original 85, yet show the same pattern (Appendix D.2, Table 9): averaged over the six configurations, the contradictory oracle reaches 72% on the 50 original and 43% on the 35 new contradictory instances, sequential A→B 69% and 46%, and parallel-unaware only 22% and 15%. Please see W2 for the full results.
Q3: Adding information-matched oracle
This is in fact our communication oracle (Section 3.2). Each parallel agent gets its partner's requirement verbatim, works independently on its own copy, and can message the other. With Claude Code + Opus 4.8 it reaches 56% / 83% (contradictory / complementary), against 62% / 86% for the single-agent oracle. We also added a stricter information-matched scenario, communication oracle with gold tests (Appendix K.1), which matches the single-agent oracle with gold tests in both information and test access: two agents reach 84% against 98% for a single agent (). Please see W3 for the discussion.
Q4: Are the conflicts manually constructed or are they grounded in real development histories?
Requirement A always comes from a real SWE-Bench Pro task, which is an actual change from the repository's history. Requirement B is a separately motivated code change written by us for the 27 seed instances, and written by GPT 5.6 Sol for the 128 generated ones. It is built to either conflict with A's code (contradictory) or to depend on a shared interface with it (complementary). The conflicts are therefore constructed, and we chose this on purpose. Merged history records only the final, already reconciled change, so the conflict between two independent changes is not observable there. Constructing them lets us verify with executable tests that each instance has a real conflict (Section 3.1).
We hope that these results and clarifications address your comments and will allow you to revisit your score.
Thank you for engaging so closely with our work and for your insightful questions. We are pleased that you highlight our "realistic problem formulation," which "distinguishes centralized multi-agent planning from independently motivated agents operating in a shared codebase," our "comprehensive evaluation," which covers "joint success, partial completion, communication strategies, and token cost rather than only reporting final pass rates," and our "useful failure analysis," showing that "many failures arise from interface mismatches, conflicting edits, and unclear ownership rather than from inability to implement the individual tasks." In response to your comments, we have extended ParallACT from 85 to 155 instances with 70 new ones, evaluated six open-weight configurations on all of them with no selection, added Git and pull request review scenarios as realistic coordination workflows, and strengthened the information-matched control with a new scenario. On contradictory instances, the gap between the oracle and parallel agents persists on the instances the original selection excluded, no communication channel recovers more than 43% of it, and two agents with the same information and tests as a single agent still fall significantly short of it. With Git, agents coordinate at the level of the oracle, but by offloading the need to communicate to a human-designed tool; ParallACT is designed to test whether agents can coordinate by communicating, which they must do wherever no such tool exists.
We extended the benchmark from 85 to 155 instances with 70 new ones, built from SWE-Bench Pro tasks disjoint from the original 85 with the same pipeline and verification (Section 3.1; Appendix B), and evaluated six configurations (Claude Code and Codex, each with Qwen, DeepSeek and GLM) on all of them (Table 2; Appendix D). The new instances are harder but show the same pattern (Appendix D.2): on their contradictory instances, averaged over the six configurations, the oracle reaches 43% and a single sequential hand-off 46%, against 15% for parallel-unaware.
Construction: in every instance, requirement A is a real SWE-Bench Pro task; only its counterpart, requirement B, is written, by hand for the 27 seed instances and by GPT 5.6 Sol for the 128 generated ones. Every instance must pass the soundness checks (Section 3.1) and is reviewed by hand, and generated instances also pass a GPT 5.6 Sol independence audit (Appendix B). The gap between the oracle and parallel agents appears on both seed and generated instances: on contradictory instances, averaged over the six configurations, the oracle reaches 74% on seed and 58% on generated instances, against 18% and 19% for parallel-unaware (for Claude Code + Opus 4.8, 67% and 61% against 0% and 21%; Table 4).
Representativeness: ParallACT is a diagnostic benchmark. It isolates the conflicts that require coordination rather than sampling how often they occur in real teams, as the revised Section 3.1 now states.
The cross-model evaluation now covers the full benchmark. We added a second harness (Codex), a new model (Qwen) and 70 new instances, which brings ParallACT from 85 to 155 instances, and the six open-weight configurations (Claude Code and Codex, each with Qwen, DeepSeek and GLM) run on all of them (Table 2; all scenarios in Appendix D). Appendix D.1 compares all seven configurations, including Claude Code + Opus 4.8, on the 85 instances they share. We originally chose the subset only to limit compute. Every configuration shows the same pattern: on the 85 contradictory instances (50 original and 35 new), parallel-unaware falls 33–54 points below the oracle in each of the six configurations (all p<10−6), and a single sequential hand-off recovers at least 82% of that gap. The full runs also cover the instances the original subset left out: on the 19 contradictory instances that Opus's oracle fails, the six configurations' oracles reach 21–63%, while their parallel-unaware runs reach only 0–5%. The gap holds on exactly these instances.