acceptodds
Under review as a conference paper at ICLR 2027

Decentralized Multi-Agent Coordination and Communication in Shared Codebases

Abstract

Coding agents working in shared repositories may receive separate, independently-motivated requests from human developers. When this happens, their individual implementations may work separately but conflict when combined. To study this setting, we introduce ParallACT, a benchmark of orchestrator-free multi-agent coding tasks. ParallACT contains both complementary and contradictory tasks. In complementary tasks, each agent’s task is separately useful and implementations must share an interface to be productively combined. In contradictory pairs, both agents’ tasks require editing the same code in incompatible ways, and naively combining independent solutions fails to produce a useful implementation. We compare a single-agent oracle given the combined set of tasks against multi-agent settings that vary execution order, visibility, awareness of other agents, and access to a communication channel. We find that the oracle solves most tasks while parallel agents perform substantially worse. Their performance drop is notable for complementary tasks, but sharp (e.g., 62% → 16%) for contradictory tasks where merged code typically satisfies only one specification. Simply telling an agent that a peer exists does not help significantly, and giving agents access to a communication channel only recovers part of the performance when it enforces synchronization, which comes at an additional token cost. ParallACT thus identifies and isolates a core challenge that will become more common as multi-agent coding by independent principals becomes standard, and provides a controlled test of which coordination strategies help, and at what cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent LLMs, Coding Agents, Multi Agent Coordination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_asy4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ParallACT, a benchmark for evaluating decentralized collaboration among coding agents working on a shared repository. It includes complementary tasks requiring interface agreement and contradictory tasks requiring reconciliation of incompatible edits. The authors compare single-agent, sequential, parallel, peer-aware, communication, and reconciliation strategies.

The experiments show that parallel agents achieve substantially lower joint success than a single-agent oracle with access to both requirements. Communication and synchronization recover part of the performance gap but generally incur higher token costs. The failure analysis suggests that coordination failures, rather than individual implementation failures, are the dominant bottleneck.

Critical strengths

Realistic problem formulation. The paper distinguishes centralized multi-agent planning from independently motivated agents operating in a shared codebase. Comprehensive evaluation. It evaluates joint success, partial completion, communication strategies, and token cost rather than only reporting final pass rates. Useful failure analysis. Many failures arise from interface mismatches, conflicting edits, and unclear ownership rather than from inability to implement the individual tasks.

Critical weaknesses

Limited dataset scale and synthetic construction. ParallACT contains only 85 tasks, and many complementary or contradictory requirements are authored by the researchers or GPT-5.6 Sol. The results may not represent the conflict distribution of real software teams. Selection bias in the cross-model evaluation. DeepSeek and GLM are evaluated only on 40 instances solved by the Claude oracle, so their results are not representative of the full benchmark. The oracle comparison is not fully controlled. The oracle sees both requirements, while parallel agents initially see only their own requirement. The gap therefore mixes coordination difficulty with information-access asymmetry. Communication strategies are somewhat artificial. File, Redis, messaging, and synchronization conditions depend on specific prompts and read/write mechanisms, so their conclusions may not transfer directly to realistic issue, PR, or code-review workflows.

Questions for authors

How many GPT-generated silver instances, tests, and reconciliation solutions were manually reviewed, and what was the agreement between reviewers? Why is the cross-model evaluation restricted to 40 instances solved by the Claude oracle? Can results on the full benchmark or difficulty-stratified subsets be reported? Can the authors add an information-matched oracle in which both parallel agents see both requirements but must still coordinate independently? Are the contradictory requirements grounded in real development histories, or are most conflicts manually constructed?

7 comments

Thank you for engaging so closely with our work and for your insightful questions. We are pleased that you highlight our "realistic problem formulation," which "distinguishes centralized multi-agent planning from independently motivated agents operating in a shared codebase," our "comprehensive evaluation," which covers "joint success, partial completion, communication strategies, and token cost rather than only reporting final pass rates," and our "useful failure analysis," showing that "many failures arise from interface mismatches, conflicting edits, and unclear ownership rather than from inability to implement the individual tasks." In response to your comments, we have extended ParallACT from 85 to 155 instances with 70 new ones, evaluated six open-weight configurations on all of them with no selection, added Git and pull request review scenarios as realistic coordination workflows, and strengthened the information-matched control with a new scenario. On contradictory instances, the gap between the oracle and parallel agents persists on the instances the original selection excluded, no communication channel recovers more than 43% of it, and two agents with the same information and tests as a single agent still fall significantly short of it. With Git, agents coordinate at the level of the oracle, but by offloading the need to communicate to a human-designed tool; ParallACT is designed to test whether agents can coordinate by communicating, which they must do wherever no such tool exists.

W1: Dataset scale and synthetic construction.

We extended the benchmark from 85 to 155 instances with 70 new ones, built from SWE-Bench Pro tasks disjoint from the original 85 with the same pipeline and verification (Section 3.1; Appendix B), and evaluated six configurations (Claude Code and Codex, each with Qwen, DeepSeek and GLM) on all of them (Table 2; Appendix D). The new instances are harder but show the same pattern (Appendix D.2): on their contradictory instances, averaged over the six configurations, the oracle reaches 43% and a single sequential hand-off 46%, against 15% for parallel-unaware.

  • Construction: in every instance, requirement A is a real SWE-Bench Pro task; only its counterpart, requirement B, is written, by hand for the 27 seed instances and by GPT 5.6 Sol for the 128 generated ones. Every instance must pass the soundness checks (Section 3.1) and is reviewed by hand, and generated instances also pass a GPT 5.6 Sol independence audit (Appendix B). The gap between the oracle and parallel agents appears on both seed and generated instances: on contradictory instances, averaged over the six configurations, the oracle reaches 74% on seed and 58% on generated instances, against 18% and 19% for parallel-unaware (for Claude Code + Opus 4.8, 67% and 61% against 0% and 21%; Table 4).

  • Representativeness: ParallACT is a diagnostic benchmark. It isolates the conflicts that require coordination rather than sampling how often they occur in real teams, as the revised Section 3.1 now states.

W2: Selection bias in the cross-model evaluation.

The cross-model evaluation now covers the full benchmark. We added a second harness (Codex), a new model (Qwen) and 70 new instances, which brings ParallACT from 85 to 155 instances, and the six open-weight configurations (Claude Code and Codex, each with Qwen, DeepSeek and GLM) run on all of them (Table 2; all scenarios in Appendix D). Appendix D.1 compares all seven configurations, including Claude Code + Opus 4.8, on the 85 instances they share. We originally chose the subset only to limit compute. Every configuration shows the same pattern: on the 85 contradictory instances (50 original and 35 new), parallel-unaware falls 33–54 points below the oracle in each of the six configurations (all p<10−6p < 10^{-6}), and a single sequential hand-off recovers at least 82% of that gap. The full runs also cover the instances the original subset left out: on the 19 contradictory instances that Opus's oracle fails, the six configurations' oracles reach 21–63%, while their parallel-unaware runs reach only 0–5%. The gap holds on exactly these instances.

W3: Oracle comparison.

We believe the suggested control you describe is exactly the communication oracle we have (Section 3.2): it extends parallel messaging by giving each agent its partner's requirement verbatim, so the two scenarios differ only in that information. With Claude Code + Opus 4.8, the communication oracle reaches 56% / 83% (contradictory / complementary), close to the single-agent oracle's 62% / 86%, against 20% / 63% for parallel messaging; this difference measures the information-access asymmetry you describe. In the multi-principal setting, however, this asymmetry is part of the problem rather than an artifact of the comparison: each agent knows only its own principal's requirement, and the communication scenarios give the agents a channel through which they can obtain the partner's requirement themselves. Yet the best channel reaches only 32% on contradictory instances, against 56% for the communication oracle: as Section 4.2 shows, the agents discuss the right code and often agree on a design, but their agreements do not prevent conflicting edits. Section 3.2 describes both oracles as diagnostic upper bounds rather than deployable strategies.

To isolate what coordination costs when information is matched, we added a new scenario, communication oracle with gold tests (Appendix K.1): with both requirements and unlimited access to their own gold tests, two agents reach 84%, against 98% for a single agent with the same information and tests (p=0.04p = 0.04). Since both scenarios have the same information, the remaining difference comes from the two agents having to coordinate their edits.

W4: The communication strategies may not transfer to realistic workflows.

We ran two new scenarios built on realistic workflows: Git and Git + PR review (Claude Code + Opus 4.8, the original 85 instances; setup, results and analysis in Appendix K.2, Table 24; prompts in Appendix M.1). Each agent implements its requirement in parallel on its own branch; the branch that is finished first is merged into the main branch, and the other agent then pulls the updated main branch, resolves any conflicts, and merges. In Git + PR review, each agent opens a pull request and reviews the other's, and the author revises for up to two rounds before merging. Git reaches 68% / 83% (contradictory / complementary), comparable to the single-agent oracle (62% / 86%; p=0.51p = 0.51 on contradictory instances) and far above parallel-unaware (16% / 57%), and PR review does not change this (58% / 80%; not significantly different from Git).

These workflows coordinate the agents through a human-designed tool rather than through communication. The agents have no channel through which to talk to each other and exchange information only through interfaces built for code (commits, merges and reviews), so the need to communicate is offloaded to the tool. For code, this is a practical solution, but it does not show whether agents can coordinate by communicating, which is what ParallACT is designed to test and what settings without such tools require. Our channel scenarios are therefore deliberately minimal: a plain send/receive channel with no imposed workflow, so that any coordination must come from what the agents communicate.

The channel results do not depend on a specific prompt or mechanism. The four variants share an identical send/receive interface and differ only in medium (shared file or Redis) and framing (free-form messaging or synchronization), and the pattern holds across both harnesses and all six open-weight configurations: on contradictory instances, no channel variant recovers more than 43% of the gap between parallel-unaware and the oracle in any configuration or run.

Q1: How many of the generated dataset instances were manually reviewed?

All of them. One author reviewed every one of the 128 generated instances by hand, including their tests and reconciliation solutions, as well as every seed instance (Section 3.1; Appendix B). Before this review, each generated instance goes through three automatic stages.

  • Construction: requirement A is always a real SWE-Bench Pro task, and requirement B is written as a separately motivated ticket that a different developer could receive, with candidates spread across languages, repositories and subsystems before generation.
  • Executable soundness checks: we run every instance in its original SWE-Bench Pro environment, where each reference solution must pass its own tests alone and the pair must interact exactly as its family requires; instances that still fail after two correction rounds are discarded. These checks verify each conflict without relying on reviewer judgment.
  • Independence audit: a separate GPT 5.6 Sol pass enforces our blind rule (no problem statement mentions the other requirement, its files or its tests) and rejects pairs that read as two halves of one decomposed task.

Q2: Selection bias in the cross-model evaluation.

We restricted it only to limit compute; the cross-model evaluation now covers the full benchmark, and the six open-weight configurations run on all 155 instances (Table 2; Appendix D). We report two difficulty-stratified views. On the 19 contradictory instances that Opus's oracle fails, the six open-weight configurations' oracles reach 21–63%, while their parallel-unaware runs reach only 0–5%. The 70 new instances are also harder than the original 85, yet show the same pattern (Appendix D.2, Table 9): averaged over the six configurations, the contradictory oracle reaches 72% on the 50 original and 43% on the 35 new contradictory instances, sequential A→B 69% and 46%, and parallel-unaware only 22% and 15%. Please see W2 for the full results.

Q3: Adding information-matched oracle

This is in fact our communication oracle (Section 3.2). Each parallel agent gets its partner's requirement verbatim, works independently on its own copy, and can message the other. With Claude Code + Opus 4.8 it reaches 56% / 83% (contradictory / complementary), against 62% / 86% for the single-agent oracle. We also added a stricter information-matched scenario, communication oracle with gold tests (Appendix K.1), which matches the single-agent oracle with gold tests in both information and test access: two agents reach 84% against 98% for a single agent (p=0.04p = 0.04). Please see W3 for the discussion.

Q4: Are the conflicts manually constructed or are they grounded in real development histories?

Requirement A always comes from a real SWE-Bench Pro task, which is an actual change from the repository's history. Requirement B is a separately motivated code change written by us for the 27 seed instances, and written by GPT 5.6 Sol for the 128 generated ones. It is built to either conflict with A's code (contradictory) or to depend on a shared interface with it (complementary). The conflicts are therefore constructed, and we chose this on purpose. Merged history records only the final, already reconciled change, so the conflict between two independent changes is not observable there. Constructing them lets us verify with executable tests that each instance has a real conflict (Section 3.1).

We hope that these results and clarifications address your comments and will allow you to revisit your score.

Reviewer_skDjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ParallACT, a benchmark of coding tasks to be completed by pairs of agents. The authors curate 85 task pairs from SWEBench Verified -- 27 manual and 58 automatic, with 50 contradictory (the code edits involve changes that will not merge by default) and 35 complementary (the code edits involve changes that should work together by default). The authors test a suite of 19 diverse settings (18 for complementary) spanning various ways of orchestrating multi-agent systems/single-agent systems, and varying the communication channels, knowledge of other agents, and access to test cases. Using these various settings the authors provide analysis about building multi-agent collaboration systems including: sequential repetition of the agent work (e.g. ABAB) is no more effective than one pass (e.g. AB), agents should be made aware of their peers, and one of the most effective strategies the agents pursued was giving the whole task to just one of the two agents and letting it implement for both.

Critical strengths

I found a few clear strengths in this paper:

  1. The sweep over the settings to evaluate was quite comprehensive. The authors tried many different settings one might think of when building a multi-agent system such as different communication channels, different knowledge of other agents, various orchestration patterns, etc. This leads to some useful insights for builders of such multi-agent systems.
  2. The motivation of the paper makes sense: as multi-agent repository management becomes more common we need to study these communication/coordination deficiencies.
  3. The paper was well written and clear
  4. The benchmark seems to be a useful resource for the community to build on, and manual labor went into crafting the benchmark which is appreciated for quality assurance when it is often much easier to generate benchmarks.
  5. The metrics for cost seem well thought out and the ideas for comparison across models by using a ratio against oracle single-agent cost is a clever solution.
Critical weaknesses

I also do feel the paper has a few critical weaknesses:

  1. Despite the comprehensiveness of the 19 situations, I do feel like one critical setting is missing which is perhaps the most common way multi-agent systems will interact in a repo. As the authors describe in L345-L346 this may be relevant "if hundreds or thousands of developers independently deploy agents on a shared codebase". However the realistic setting in which this would happen is in a git repository (or some other version control). In such a system agents would check out an independent work tree, implement their feature, and commit it (likely with a brief commit message). Then when agents try to commit in the future they could see that their commit would lead to a merge conflict, they can pull the changes into their worktree, and agentically resolve merge conflicts before once again committing. This today is how teams of 1000 developers work on a monorepo. The setting in which both agents cannot see the work of the other but need to communicate to make a single submission feels like it is not motivated in a very realistic design for multi-agent collaboration. I fear by not testing this way it limits some of the claims that can be made about how agents fail on real world multi-agent deployments.
  2. The evaluation ends up being a bit small, only 20 contradictory and 20 complementary tasks are evaluated between all models. It is unclear which proportion of these 40 are human generated versus the AI generated tasks. Sometimes the winning approach wins by 5 or 10 points which is only 1 or 2 tasks. If it is within budget it would be nice to add confidence intervals on these evaluations and run for more trials.
  3. I found myself a bit disappointed that the analysis primarily takes place in the appendix. In fact a lot of the details of the paper are in the appendix (13 appendix sections!!). Section 4.2 "analysis" is one of the most important parts of the paper (the only reason to do the super comprehensive 19 setting evaluation is to get the results/analysis from it), however it ends up only being around 65-70% of a page while citing appendix J for details on how the analysis was actually done. This would benefit from promotion to main paper material with a figure, perhaps by being a bit more terse on some of the explanations on data collection.
Questions for authors
  1. In Figure 2 I didn't understand the resolution to the "Contradictory" case. It doesn't seem to satisfy either of the code proposals. In any case, the first two figures feel a bit "AI generated", so a suggestion might be to take an edit pass on them for clarity. Concretely my question is how does the resolution actually fix the contradictory cases in figure 2?
  2. I would suggest changing the phrasing L232 "We run 19 contradictory and 18 complementary scenarios". On first read I thought this was number of evaluation tasks you ran, and I was confused when later you said 20 of each.
  3. Something about filtering to the cases where Open 4.8 succeeds and doing the cross model evaluation there feels a bit weird. What if there is a systematic reason Opus fails, but GLM 5.3 would succeed? Feels like we are getting a skewed look.
  4. Would you add confidence intervals to your tables to make it easier to assess significance?
5 comments

We are grateful for the time and care you put into this review and for your detailed, practical suggestions. We appreciate that you find our sweep over settings "quite comprehensive," leading to "some useful insights for builders of such multi-agent systems," that "the motivation of the paper makes sense," that the paper "was well written and clear," and that the benchmark is "a useful resource for the community to build on." Building on your feedback, we have added the version-control workflow you describe, with Git and Git + PR review scenarios, extended the benchmark from 85 to 155 instances with 70 new ones, run six configurations on all of them with no subset selection, added confidence intervals from a second full run, and moved the communication analysis into the main paper. On contradictory instances, the gap between the oracle and parallel agents is far larger than the confidence intervals in every configuration and run. With Git, agents coordinate at the level of the oracle, so for code, version control is a practical solution; it does so, however, by offloading the need to communicate to the tool, while ParallACT is designed to test whether agents can coordinate by communicating, which they must do wherever no such tool exists.

W1: Realistic setting such as Git version control.

In our parallel scenarios, each agent works on its own copy of the repository and cannot see the other's work until both have finished, when the two changes are combined automatically (by concatenation, Git merge or a reconciler model). As you note, this leaves out the step in which an agent pulls the merged work and resolves conflicts itself. Based on this comment, we ran two new scenarios that add this step, Git and Git + PR review, with Claude Code + Opus 4.8 on the original 85 instances (setup, results and analysis in Appendix K.2, Table 24; prompts in Appendix M.1). In Git, both agents implement in parallel on their own branches, commit and push, with no order assigned in advance. The branch that is finished first is merged into the main branch; the other agent then pulls the updated main branch, resolves any conflicts, and merges. In Git + PR review, each agent also reviews the other's pull request, and the author revises for up to two rounds before merging.

Git performs comparably to the single-agent oracle: 68% / 83% (contradictory / complementary), against 62% / 86% for the oracle and 16% / 57% for parallel-unaware; on contradictory instances, it is not significantly different from the oracle (p=0.51p = 0.51). The gain comes from the integration turn, in which the second agent works against the peer's finished code: merging the parallel-unaware patches with Git merge alone, without this turn, reaches only 14% (Table 13). PR review does not change this (58% / 80%; not significantly different from Git). Like developers, agents can therefore use version control to integrate independently written changes, and for code, this is a practical solution.

Git provides coordination, but it does not test communication, which is what ParallACT is designed to study. In Git, the agents have no channel through which to talk to each other and exchange information only through interfaces built for code (commits, merges and reviews), so the need to communicate is offloaded to the tool. In our communication scenarios, the agents have no such workflow, so any coordination must come from what they communicate. We use code as the testbed because joint correctness can be verified with executable tests, but the question extends beyond code: many settings in which agents must work together have no equivalent of version control, and there agents must coordinate by communicating, which is where our channel scenarios show they perform poorly.

W2: Scale of evaluations.

We ran additional experiments for each point; the full results are in the revised paper.

Scale: we built 70 new instances from SWE-Bench Pro tasks disjoint from the original 85, with the same pipeline and verification (Section 3.1; Appendix B), bringing ParallACT to 155 instances. All open-weight configurations now run on all 155 instances instead of the 40-instance subset, and the evaluation now has six configurations, adding a second harness (Codex) and a new model (Qwen): Claude Code and Codex, each with Qwen, DeepSeek and GLM (Table 2; all scenarios in Appendix D). On the 85 contradictory instances, parallel-unaware falls 33–54 points below the oracle in each of the six configurations (all p<10−6p < 10^{-6}).

Composition: of the 155 instances, 27 are human-written seed instances (12 contradictory, 15 complementary) and 128 are generated: 58 among the original 85 (38 / 20) and the 70 new ones (35 / 35). Table 3 (Appendix A) gives the per-repository counts for all 155 instances, split by seed and generated and by construction round. Table 4 (Appendix D) already reports Opus's results separately on seed and generated instances for every scenario, and Table 1 below gives the same split for the six configurations. The gap between the oracle and parallel agents on contradictory instances appears on both seed and generated instances, for Opus and across the six configurations.

Table 1. Joint pass rate (%) of the six open-weight configurations (mean) on seed vs. generated instances, contradictory / complementary (12 / 15 seed and 73 / 55 generated instances).

SeedGenerated
Oracle74/7658/71
Sequential A→B67/8658/67
Parallel unaware18/6419/50
Parallel Sync (Redis)25/7628/60
Comm. oracle53/8947/67

Confidence intervals and trials: within the rebuttal period, we ran a full second run of the original 85 instances for three configurations (Claude Code + Opus 4.8, Claude Code + DeepSeek and Codex + DeepSeek) and report both runs with means and 95% bootstrap confidence intervals in Appendix I.1 (Table 22). The intervals are about ±9–14 points, so, as you note, differences of 5–10 points between similar scenarios are within run-to-run variation. The central gaps are much larger: on contradictory instances, parallel-unaware stays 38–54 points below the oracle in every run of every configuration, and these comparisons remain significant after Holm–Bonferroni correction.

W3: Extensive analysis performed but large parts are in the appendix

Thank you for this suggestion. We moved the communication analysis from the appendix into Section 4.2, which now describes both how the analysis is done and what it finds, without referring to an appendix. We analyze the message logs and transcripts of all Claude Code + Opus 4.8 channel and communication-oracle runs, track when each message is read relative to the recipient's first edit, and label each conversation with an outcome-blind LLM judge, which agrees with an author's independent labels on 90% of a stratified sample of 30 conversations (Cohen's κ=0.67\kappa = 0.67). The main text now reports what was previously in the appendix: agents reach a concrete agreement more often under Sync than under free-form messaging (72% vs. 46% of contradictory runs), yet 61% of runs with an agreement still produce a merge conflict, and only 3% of messages go unread while the recipient is still working.

Q1: Clarification on Figure 2

We have updated Figure 2 and its caption to make it more clear. In the contradictory case, both requirements change the second argument of getObjects(). Requirement A makes it a list of fields to return, and requirement B makes it an options object for compact output. Each patch passes its own tests alone, but both rewrite the same argument, so merging them conflicts. The joint solution in green gives each requirement its own parameter. A’s field selection keeps the second position, and B’s compact output becomes a separate keyword argument. Both behaviors are then available through one signature, so both sets of tests pass. The reconciliation is neither agent’s original proposal and is rather what a coordinated pair of agents would agree on.

Q2: Clarification on evaluation scenario count

Thank you. We have rephrased it in Section 3.2.

Q3: Filtered subset for evaluation

The cross-model evaluation now covers the full benchmark: the six open-weight configurations (Claude Code and Codex, each with Qwen, DeepSeek and GLM) run on all 155 instances (Table 2; all scenarios in Appendix D), and Appendix D.1 compares all seven configurations, including Claude Code + Opus 4.8, on the 85 instances they share. This also tests your hypothesis directly. The open-weight configurations do solve instances that Opus fails: their oracles solve 14 of the 19 contradictory instances that Opus's oracle fails, reaching 21–63% on them. Yet on those same instances, parallel-unaware reaches only 0–5% in every configuration. The parallel gap holds on exactly the instances the original selection excluded.

Q4: Can you add confidence intervals to the tables to make significance easier to assess?

Yes. Appendix I.1 (Table 22) now reports both runs, their mean, and a 95% bootstrap confidence interval over instances for the three configurations we ran twice (Claude Code + Opus 4.8, Claude Code + DeepSeek and Codex + DeepSeek), for six main scenarios. The intervals are about ±9–14 points, so differences of 5–10 points between similar scenarios are within run-to-run variation, while the gap between the oracle and parallel-unaware on contradictory instances, 38–54 points in every run, is far outside them. We are running repeated runs for the remaining configurations and will add their confidence intervals to the results tables in the final version.

We hope these additions address your comments and will allow you to revisit your score.

Reviewer_NGttOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces a benchmark for evaluating coding agents' ability to coordinate with each other when the requirements for how they contribute naturally conflict. They build a dataset and evaluation environment, and they show that even state-of-the-art agents, such as Claude opus, suffer from dramatic performance drops. They've also shown that communication techniques, even with multiple different designs, remedied very little. They only partially recover performance but require significant token cost.

Critical strengths
  1. The formulation and evaluation address an understudied problem about coordination in multi-agent coding.
  2. The paper documents the detailed construction methodology of the benchmark, which is of value to the community.
  3. They show a comprehensive evaluation across multiple scenarios, providing a systematic comparison showing the performance drop and the remedy of different communication techniques. The observation that communication only helped a little is a helpful finding.
Critical weaknesses
  1. The evaluation limits the number of re-attempts and debug iterations that each agent can have, which is a strong limitation in terms of how it limits the impact of the observation. It would be helpful to show a curve of how the success rate increases in contrast to the increased cost when agents are able to continuously communicate with each other and refine their own contributions.
  2. any discussions about how structured coordination mechanisms such as pull request reviews might affect the results? I see that it is mentioned the introduction section, but don't see an empirical or further detailed discussion later on in the paper.
Questions for authors
  1. the paper mentions "about 1/3 of messages are never read" how fundamentally unsolvable is this problem in state-of-the-art coding agents? How much of the failures are contributed/explained by this factor?
6 comments

Thank you for your careful and constructive review. We are glad you find that our work addresses "an understudied problem about coordination in multi-agent coding," that it "documents the detailed construction methodology of the benchmark, which is of value to the community," and that it provides "a comprehensive evaluation across multiple scenarios," where "the observation that communication only helped a little is a helpful finding." Following your suggestions, we ran three new scenarios. The first one lets agents communicate and debug without an iteration limit and gives the requested success–cost curve. The other two are built on structured coordination through Git and pull request review. With unlimited debugging, two agents still fall significantly short of a single agent with the same information (84% vs. 98%). Git lets agents coordinate at the level of the single-agent oracle, and pull request review does not change this. For code, a human-designed workflow is a practical way to coordinate. However, it does so without the agents themselves communicating because the need to communicate is offloaded to the tool. ParallACT is designed to test whether agents can coordinate by communicating, and they must do so wherever no such tool exists. Our results show that they perform poorly.

W1: Limitation on re-attempts and debug iterations.

We ran a new scenario that removes this limit: communication oracle with gold tests on Claude Code + Opus 4.8 over the 50 contradictory instances (Appendix K.1), and all the prompts are in Appendix M.1. Each agent receives both requirements, a two-way message channel, and its own gold tests, which it can run as often as it chooses against its current changes. The agents are told to keep working together until both requirements hold after the merge with no limit on test runs or messages, and stop when they judge the work is done. We record the joint state at every test run, which gives the requested curve of joint success against debug iterations (Table 1) and token cost (Table 2). Both are also in Table 23 of the revised paper.

Table 1. Joint success (%) after the first k test runs (both agents combined), and median token cost spent so far, relative to the single-agent oracle. Success stays at 82% from four test runs on (median 3 test runs per run, at most 9).

k12345
Joint success1256688282
Cost so far0.53×0.99×1.18×1.26×1.30×

Table 2. Joint success (%) within a token budget, relative to the single-agent oracle.

Budget0.5×1×1.5×2×3×≥ 4×
Joint success23856668084

Success rises over the first four test runs and then levels off at 82% (84% when the agents stop). That is still significantly below the 98% reached by a single agent given both requirements and the same gold tests (the oracle with gold tests scenario; p=0.04p = 0.04). The two agents also spend more: 1.90x the tokens of the single-agent oracle without tests, against 0.74x for the single agent with tests. More iterations do not close the gap: beyond four test runs, extra cost barely improves success.

W2: Discussion of structured coordination mechanisms such as pull request reviews.

Based on this comment, we ran two new scenarios built on Git, with and without PR review, using Claude Code + Opus 4.8 on the original 85 instances (setup and analysis in Appendix K.2, Table 24; prompts in Appendix M.1).

Setup. In Git, each agent works in a clone of a shared repository on its own branch, and is told only that other developers are working on their own changes, and that the main branch is protected. Both agents implement in parallel, commit and push; no order is set in advance. The branch of the agent that finishes first is merged into the main branch, and the other agent then gets an integration turn: it brings the updated main branch into its own branch, resolves any merge conflicts so that its requirement holds and the work already on the main branch keeps working, and merges. In Git + PR review, code changes reach the main branch only through pull requests. Each agent opens one and reviews the other's request knowing only its own requirement, and the author revises for up to two rounds before merging.

Table 3. Joint pass rate (%) and token cost relative to the single-agent oracle, Claude Code + Opus 4.8, contradictory / complementary (50 / 35 instances).

ScenarioContra.Compl.Cost
Oracle62861.00 / 1.00
Sequential A→B54771.18 / 1.12
Parallel unaware16571.17 / 1.14
Git (new)68832.14 / 1.42
Git + PR review (new)58805.22 / 2.50

Git provides coordination, but does not test communication. Like developers, agents can use a version-control workflow to integrate independently written changes, so for code, a human-designed tool is a practical solution. However, the agents have no channel through which to talk to each other and exchange information only through interfaces built for code (commits, merges and reviews), so the need to communicate is offloaded to the tool. ParallACT uses code as a testbed for multi-agent communication and coordination because joint correctness can be verified with executable tests, and the questions it asks (whether agents exchange the right information, read it in time and act on what they agree; Section 4.2) remain open whichever scenario succeeds. Version control is also specific to coding. In many other settings, agents must work together and coordinate by communicating, which is where our scenarios on communication channels show they perform poorly.

Q1: How much of the failures are contributed/explained by messages not being read by the agents and do state-of-the-art agent harnesses solve it?

Unread messages are not the predominant cause of failure. We evaluate agents in state-of-the-art harnesses, Claude Code and Codex, where the channel is a tool the agent calls, so an agent sees a message only when it chooses to check. We checked every message in a full run of Claude Code + Opus 4.8 on all 85 instances: most unread messages arrive after the receiving agent has already finished, and only 3% go unread while the receiver is still working. Such a missed message occurs in 16% of failing and 24% of passing contradictory runs (17% and 18% pooled over both families), so unread messages explain little of the failures. Better message delivery alone would therefore not solve the problem. Parallel Sync, which prompts agents to announce and reconcile each change, raises concrete agreement from 46% of contradictory runs under free-form messaging to 72% of contradictory runs, yet 61% of runs with an agreement still produce a merge conflict and only 19% of those conflicting runs pass. The harder problem is acting on what was communicated: the conflict still persists even when agents read on time and agree. We incorporated this additional result in Section 4.2.

We hope that our response addresses your comments and will allow you to revisit your score.

Dear Reviewer NGtt,

As two days remain in the discussion period, we would like to check whether our response addressed your comments. Following your suggestions, we ran a scenario without an iteration limit, added Git and Git + PR review scenarios, and analyzed how much unread messages contribute to the failures. These results show that unlimited debugging does not close the gap (success levels off at 82%, against 98% for a single agent with the same information), that Git reaches the oracle's level by offloading communication to the tool while PR review does not improve on it, and that only 3% of messages go unread while the recipient is still working. We would appreciate it if you would consider these results when revisiting your score. If anything is still unclear, we are glad to answer any further questions in the remaining time.

Reviewer_jbifOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces ParallACT, a benchmark of 85 paired coding tasks where agents independently work on the same repository without a central planner. It studies both conflicting and complementary requirements across different execution schedules and communication settings.

The main finding is that parallel agents perform much worse than a single agent with access to both requirements, especially on conflicting tasks. Sequential execution and full task disclosure recover much of this gap, while ordinary communication provides limited improvement.

Critical strengths

1.Interesting setting. The paper studies decentralized coding with independently assigned requirements, which is more realistic than the usual centrally orchestrated multi-agent setup.

2.Extensive evaluation. The authors compare multiple execution and communication strategies, with useful analyses of merge conflicts, communication failures, and token costs.

Critical weaknesses

1.The main result is somewhat expected. The single-agent oracle sees both requirements, while parallel agents only see their own. The benchmark also selects tasks where independent solutions fail joint tests. The large performance gap therefore reflects missing information as much as coordination failure. When both agents receive both requirements, the gap becomes small and statistically insignificant. This weakens the paper's central claim.

2.Some experimental choices make the conclusions less convincing. The evaluation appears to use only one run per setting, with many statistical comparisons but no multiple-testing correction. The cross-model subset is selected based on Claude's oracle success, and the main metric takes the best outcome across three merge policies. These choices make it difficult to judge how robust the results are.

Questions for authors

1.Given the small gap between the single-agent oracle and communication oracle, isn't the main bottleneck information availability rather than coordination? 2.Were multiple stochastic runs performed? How would the results change with multiple-testing correction? 3.Why does the main metric count success under any of three merge policies? What happens with one fixed policy? 4.How were the generated task pairs checked for naturalness and independence? Were multiple human annotators involved? 5.Can the authors evaluate other models on an independently sampled subset rather than one selected using Claude's oracle performance? 6.How does ParallACT differ substantially from CooperBench beyond the task construction and mutual-unawareness setting? Also, a few related-work citations appear to need correction. For example, Cemri et al., Why Do Multi-Agent LLM Systems Fail?, NeurIPS 2026 should be NeurIPS 2025

7 comments

Thank you for taking the time to review our work and for your thoughtful feedback. We are encouraged that you find our setting "more realistic than the usual centrally orchestrated multi-agent setup" and our evaluation "extensive," with "useful analyses of merge conflicts, communication failures, and token costs." To address your comments, we have added a new information-matched scenario with unlimited test access, extended the benchmark from 85 to 155 instances with 70 new ones, run six configurations on all of them with no subset selection, repeated the full evaluation of the original 85 for three configurations with confidence intervals, and applied a multiple-testing correction. Across all of these, our central result holds: on contradictory instances, parallel agents fall far below the single-agent oracle, the mechanisms available to them recover only part of this gap, and the gap remains significant after Holm–Bonferroni correction in every configuration and run.

We also noticed that you rated the presentation as difficult to follow. Since other reviewers responded more positively to the presentation (Reviewer NGtt rated it 3, "Clear and easy to follow," and Reviewer skDj listed "The paper was well written and clear" among the paper's strengths), we would be grateful if you could point us to the specific sections or passages that were hard to follow, so that we can revise them. In the meantime, we have revised the paper for clarity: Section 3.1 is more concise, the communication analysis is now in the main text (Section 4.2), and a new Figure 3 summarizes the soundness checks for each instance.

W1: The gap between parallel agents and the single-agent oracle reflects missing information in addition to coordination failure.

Our central claim is that when agents receive independently written, conflicting requirements, parallel agents fail to produce a jointly correct solution, and the mechanisms available to them (peer awareness, communication channels, and merge policies) recover at most part of the performance gap. The revised Introduction states this claim explicitly. The oracle is not a competing strategy; it shows that the tasks are solvable when one agent knows both requirements. Giving the agents the gold tests removes any ambiguity about what the tests check: a single agent then solves 98% of contradictory instances (oracle + gold tests), yet two parallel agents, each with its own gold tests, reach only 38% (parallel + gold tests; 96% vs. 32% averaged over the six open-weight configurations, significant in all six; Table 21).

You are correct that the oracle has more information than the parallel agents. In the multi-principal setting, each principal writes their own requirement, so each agent knows only its own. Obtaining the peer's information is therefore itself part of the coordination problem. Giving the agents every requirement up front would require someone who knows all of them; in an organization with 10k developers, centralizing all requirements before the agents act is infeasible. We therefore use the oracle and the communication oracle as diagnostic upper bounds rather than deployable strategies (Section 3.2).

The communication oracle result supports this claim. Given the peer's requirement, two parallel agents come close to the oracle (56% vs. 62% on contradictory instances with Claude Code + Opus 4.8), so what parallel agents mainly lack is that requirement. In the communication scenarios, agents can obtain exactly this through the channel, yet the best of them reaches only 32% on contradictory instances against 56% for the communication oracle with Claude Code + Opus 4.8 (28% vs. 48% averaged over the six open-weight configurations; significant in 5 of the 6; Table 21). Section 4.2 shows why: the agents discuss the right code and often agree on a design, but their agreements do not prevent conflicting edits. Even when both agents have the same information and unlimited access to their gold tests, two agents still fall significantly short of a single agent (84% vs. 98%, p=0.04p = 0.04; Appendix K.1). Selecting tasks where independent solutions fail the joint tests is deliberate: ParallACT is a diagnostic benchmark that isolates conflicts requiring coordination, rather than estimating how often such conflicts arise in practice (Section 3.1).

W2: Robustness of results.

We ran additional experiments for each of these points. The full results are in the revised paper; we summarize the main points here.

No subset selection, more instances, a second harness and a new model. The cross-model evaluation now covers the full benchmark. We also built 70 new instances from SWE-Bench Pro tasks disjoint from the original 85, with the same pipeline and verification (Section 3.1; Appendix B), and added a second harness (Codex) and a new model (Qwen3.8 Flash Next). This gives six open-weight configurations (Claude Code and Codex, each with Qwen, DeepSeek and GLM), all run on the 155 instances (Table 2; all scenarios in Appendix D), summarized in Table A. Appendix D.1 compares all seven configurations on the 85 instances they share, and Appendix D.2 splits the results into the original 85 and the 70 new instances.

Table A. Joint pass rate (%). Opus: Claude Code + Opus 4.8 on the original 85 instances. Open-weight: mean over the six configurations on all 155 instances, with the range across configurations in parentheses.

Opus, contra.Opus, compl.Open-weight, contra.Open-weight, compl.
Oracle628660 (46–71)72 (61–84)
Sequential A→B547760 (42–76)71 (63–81)
Parallel unaware165719 (12–26)53 (49–57)
Parallel Sync (Redis)327728 (21–40)64 (50–74)
Comm. oracle568348 (33–61)71 (60–80)

The patterns we reported in the original submission hold in every configuration: on the 85 contradictory instances (50 original and 35 new), parallel-unaware falls 33–54 points below the oracle in each of the six configurations (all p<10−6p<10^{-6}), a single sequential hand-off recovers at least 82% of that gap, and communication recovers only part of it.

Repeated runs. We repeated the full evaluation of the original 85 instances for three configurations (Claude Code + Opus 4.8, Claude Code + DeepSeek and Codex + DeepSeek) with the same harness, model, settings and prompts, so only the agents' sampled outputs differ (Appendix I.1, Table 22). The 95% confidence intervals are about ±9–14 points, so differences of 5–10 points between similar scenarios are within run-to-run variation. The central gap is much larger. On contradictory instances, parallel-unaware stays 38–54 points below the oracle in every run of every configuration (Table B). These runs take significant time because of API rate limits, the number of scenarios and the complexity of each task; we are running the remaining configurations and will include them in the final version.

Table B. Contradictory instances (50), mean of the two runs with 95% bootstrap confidence intervals over instances, and the oracle − parallel-unaware gap in each run.

OracleParallel unawareGap (Run 1 / Run 2)
Claude Code + Opus 4.864 [51, 76]20 [10, 30]46 / 42
Claude Code + DeepSeek62 [50, 74]23 [14, 33]40 / 38
Codex + DeepSeek78 [67, 88]26 [15, 37]50 / 54

Multiple testing. We applied a Holm–Bonferroni correction over the 35 comparisons in Appendix I, separately for each configuration and run; Appendix I.1 reports the corrected results. The comparisons behind our central claim (W1) remain significant: on contradictory instances, oracle vs. parallel-unaware and oracle + gold tests vs. parallel + gold tests in all 10 configuration-runs (Opus, the six open-weight configurations, and the three second runs), and sequential A→B vs. parallel-unaware in 9 of 10 (Claude Code + DeepSeek Run 2: p=0.004p = 0.004 before correction, 0.130.13 after). The three Opus comparisons that lose significance are not part of that claim. Two are between parallel scenarios: parallel Sync (Redis) vs. parallel-unaware, which only makes the gain from communication less certain and is consistent with communication recovering at most part of the gap, and communication oracle vs. parallel Sync (Redis) (p=0.06p = 0.06), a diagnostic comparison. The third is the smaller oracle vs. parallel-unaware gap on complementary instances (86% vs. 57%), which points the same way in every configuration and is significant when pooled over the six open-weight configurations (p<0.001p < 0.001; Table 21).

Merge policies. Appendix F reports each fixed policy separately (Tables 13 and 14 for Opus, Table 15 for the open-weight configurations). For Opus, none of the 28 paired policy comparisons is significant after Holm correction, and the best of the three is at most 6 points above the reconciler alone. In the open-weight configurations, the reconciler does recover failures (contradictory parallel-unaware rises from 9% with Git merge to 19% with the reconciler, averaged over the six configurations), yet the best parallel scenario reaches 28%, against 60% for a single sequential hand-off. Taking the best of the three can only raise the parallel scenarios, so the reported gap is conservative.

Q1: Given the small gap between the single-agent oracle and the communication oracle, isn't the main bottleneck information availability rather than coordination?

Please see W1: in the multi-principal setting, obtaining the peer's information is itself part of the coordination problem. To separate information from coordination directly, we ran a new scenario, communication oracle with gold tests, with Claude Code + Opus 4.8 (Appendix K.1; prompts in Appendix M.1).

Setup. Each agent receives its own requirement and its partner's verbatim, a two-way message channel (it can send, read, or wait up to 90 seconds for a reply), and its own gold tests, which it can run at any time against its current changes. The agents are told to keep working together until both requirements hold after the merge, with no limit on test runs or messages; as in the communication oracle, the two changes are then merged and evaluated on the combined tests. This matches the single-agent oracle with gold tests in information and test access, so the only difference is that the work is split across two agents. We run it on the 50 contradictory instances, since a complementary agent's tests exercise its partner's code and cannot be run on one agent's changes alone.

Results. Two agents reach 84% (42 of 50) against 98% for the single agent with the same information and tests (p=0.04p = 0.04, exact McNemar; also p=0.04p = 0.04 against our second run of the single agent). They improve on the communication oracle without tests (56%, p=0.007p = 0.007) and on parallel agents with gold tests but without the partner's requirement (38%, p<0.001p < 0.001), so information and tests each help, but together they still do not close the gap. The two agents spend 1.9x the oracle's tokens, against 0.7x for the single agent. Recording the joint state at every test run, success rises over the first four test runs and then levels off at 82% (Table 23), so beyond a few test runs, additional tokens barely improve success. Since both scenarios have the same information, the remaining difference comes from the two agents having to coordinate their edits.

Q2: Were multiple stochastic runs performed? How would the results change with multiple-testing correction?

The original submission used one run per setting. We have now run a full second run of the original 85 instances for three configurations (Claude Code + Opus 4.8, Claude Code + DeepSeek and Codex + DeepSeek; Appendix I.1, Table 22): the 95% confidence intervals are about ±9–14 points, while on contradictory instances parallel-unaware stays 38–54 points below the oracle in every run. With Holm–Bonferroni correction over the 35 comparisons, separately for each configuration and run, the comparisons behind our central claim remain significant (oracle vs. parallel-unaware and oracle + gold tests vs. parallel + gold tests in all 10 configuration-runs; sequential A→B vs. parallel-unaware in 9 of 10). The three Opus comparisons that lose significance are between parallel scenarios or on complementary instances, and our central claim does not rely upon them. Please see W2 for the details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.