acceptodds
Under review as a conference paper at ICLR 2027

OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime

Abstract

Multi-agent coding systems tackle complex software engineering tasks through specialized roles, typically measuring their collaboration benefits against a single-agent baseline. However, existing evaluations blindly assume the configured organization is faithfully executed, whereas agents often solve tasks entirely in isolation without any actual delegation. Furthermore, confounding factors like models, toolsets, and token budgets frequently vary across systems, preventing observed gains from being cleanly attributed to the organization itself. To address this, we introduce OpenCollab, a framework for controlled multi-agent evaluation. OpenCollab defines adherence to quantify whether the declared organization is actually realized during execution. It provides a unified runtime to hold all key experimental factors strictly constant, and records a fine-grained event stream to enable per-run verification between assigned and realized organizations. On SWE-bench Verified, we find that adherence is strictly zero across 40 tasks when the model is free to choose whether to collaborate. By altering only one closing instruction in the role prompt to mandate delegation, adherence jumps to 0.925 on the same tasks. These findings demonstrate that multi-agent performance claims are uninterpretable without measuring adherence, establishing OpenCollab as a necessary infrastructure for rigorous, causal multi-agent comparisons.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Multi-Agent Collaboration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sP4tOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OpenCollab separates agent orchestration from a shared execution runtime, supporting single agents, model-directed teams, and scripted workflows. It combines declarative roles and communication graphs, runtime restrictions, isolated workspaces, and structured event records. The paper defines structural Adherence and explains when an adherence-adjusted outcome difference identifies a configuration-level complier effect. In the reported Team experiments, Adherence reduces to whether every declared teammate spends tokens and produces output; it does not require incorporation of that output into the final solution. Experiments examine 36 SWE-bench Pro tasks and compare coding harnesses on 193 SWE-bench Pro, 89 Terminal-Bench 2.1, and 113 DeepSWE tasks. A two-solver workflow improves DeepSWE success over the single-agent configuration while consuming more resources.

Critical strengths
  1. Useful infrastructure for auditable organizational experiments. The common runtime, declarative role/tool/topology controls, role-attributed events, and worktree provenance provide a coherent basis for comparing organizations and diagnosing failed handoffs (Sections 3.1–3.4; Appendix C). This contribution has value independently of leaderboard superiority: it makes otherwise opaque execution differences observable and supports both model-chosen and scripted coordination on the same substrate.

  2. A substantive diagnostic result with unusually explicit limits. Appendix A separates the Wald ratio from the complier effect, states the exclusion and coupling assumptions, and checks the relevant paired contrasts. For the Open configuration, Table 6 reports a +11.8-point Wald ratio but a −5.9-point paired contrast among adherent runs and +15.8 among non-adherent runs. The text appropriately treats these as descriptive and acknowledges that rerunning Single changes 5 of 36 outcomes. The adherence experiments also include confidence intervals and paired comparisons (Tables 2 and 10). These analyses make the warning against attributing gains from configuration labels alone useful and credible.

Critical weaknesses
  1. The claimed isolation of organizational dimensions exceeds several experimental contrasts. Table 2 is presented as single-dimension ablations relative to one reference, yet Appendix B.1.1 (pp. 19–20) shows that moving from Star to Ring changes the roster, role cards, and which teammates must participate, as well as the graph. The reference and the “2M / role” Budget row already have the same allowance; their difference is two prompt sentences about available and historical token use. The other budget rows change both allowance and that prompt information. These differences are disclosed, which is valuable, and the Ring-with-shortcut versus Ring comparison does isolate an edge. Nevertheless, the broader framing does not establish separate topology and resource-capacity effects. Since controlled attribution is a central contribution, the conclusions should distinguish those effects from effects of bundled configurations and budget-awareness prompts.

  2. Duo's advantage does not establish the additional value of its organization over repeated solving and selection. Appendix B.1.3 describes two complete candidates, different solver instructions, and a selector with an optional adjudicator. On DeepSWE, Duo uses 26.11M average tokens versus Base's 11.95M (Table 4). The 21-versus-5 discordant outcomes and reported exact test, p = 0.0025, support an advantage for the full workflow; the authors also correctly acknowledge the inconclusive Terminal-Bench comparison, p = 0.51 (Table 9). However, no two-run Base control with the same selector and comparable total budget separates role design or information exchange from extra sampling. The manuscript also calls the selector a “fixed rule” without specifying that rule. These limitations narrow the introduction's collaboration-benefit claim, while leaving the demonstrated higher-compute workflow result intact.

Questions for authors
  1. Clarification of experimental attribution: Which existing Table 2 comparisons support separate effects of topology and token capacity when role rosters, role cards, and budget-information prompts are held fixed? Which conclusions should instead be limited to the bundled configurations?
  2. Clarification for reproducibility: What exact rule selects between Duo's candidates, what triggers the adjudicator, and what candidate evidence is available at each decision? This would help assess how much of the improvement the selection procedure could explain.
  3. Limited additional validation: On DeepSWE, can the authors compare Duo with two independent Base candidates under the same selection procedure and comparable total token allowance, reporting actual cost? Alternatively, would they limit the claim to the effectiveness of the complete, more expensive workflow rather than additional value attributable to its role design or information exchange?
Reviewer_nVvoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OpenCollab, a multi-agent coding framework that supports programmable collaboration, controlled runtime execution, and fine-grained event tracing. It proposes an Adherence metric to measure whether configured agent organizations are realized during execution and investigates how different configurations affect collaboration. Experiments show substantial variations in Adherence across settings. On three coding benchmarks, OpenCollab (Duo) achieves competitive or state-of-the-art performance, while its single-agent configuration demonstrates strong token efficiency.

Critical strengths
  1. OpenCollab provides a unified framework for configuring and evaluating multi-agent collaboration under controlled runtime conditions. Its structured event streams and Adherence metric enable systematic analysis of actual agent behavior.
  2. The framework demonstrates strong empirical performance across three coding benchmarks. In particular, OpenCollab (Duo) substantially improves DeepSWE performance over its single-agent counterpart, while OpenCollab (Base) achieves the lowest token consumption among the evaluated harnesses.
Critical weaknesses
  1. Unclear attribution of performance gains. OpenCollab (Duo) runs two solving processes and selects between their outputs. Without a compute-matched repeated single-agent baseline, it is difficult to distinguish collaboration benefits from additional inference computation.
  2. Limited interpretation of Adherence. The experimental metric primarily measures whether teammates participate rather than whether they contribute meaningfully. Moreover, the causal interpretation of CACE depends on assumptions that are not supported by the Open-card diagnostic, limiting the strength of causal conclusions.
Questions for authors
  1. How does OpenCollab (Duo) compare with a compute-matched baseline using two independent single-agent runs and the same selection mechanism?
  2. Could the authors clarify how Adherence captures meaningful collaboration beyond agent participation and how this affects the causal interpretation of the results?
Reviewer_Q6HtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of the "illusion of collaboration" in multiagent systems. They present Opencollab a framework for multi-agent coding in which any configurations (single agent, model-directed Team, scripted Workflow) can be scripted on one shared runtime. Their framework enforces topologies, token budgets, and logs with role information for enable programatic analysis. Using these logs the are able to evaluate how well the agents follow the mulitagent topology using Adherence metrics (which they introduce). On 36 SWE-bench Pro tasks, changing the model, tools, budget, prompt or topology moves Adherence between 47.2% and 97.2%. In a cross-harness comparison with GPT-5.6-Luna, a two-coder Workflow (they call Duo) achieves the highest Pass@1 on SWE-bench Pro, Terminal-Bench 2.1 and DeepSWE among the five harnesses evaluated.

Critical strengths
  1. The "illusion of collaboration" framing is compelling and identifies a real, under-measured problem: configured multi-agent organizations often are not realized at runtime, and they show that Adherence varies widely with the model, tools, budget, prompt and topology.
  2. The runtime is carefully engineered, enforcing tool and messaging boundaries, enforcing role token budgets, and recording a role-attributed event stream that makes controlled comparisons possible.
  3. The appendices are unusually transparent, reporting answer leakage, hand-stopped runs, rerun variance and the limits of their own CACE estimates, and the execution traces (Figures 9–11) clearly illustrate how delegation succeeds or fails. (Though I only know this because I got lost and asked an AI which read the appendix. Drop the mostly useless figures 1/2 in the paper and bring contents from the appendix.)
Critical weaknesses
  1. The SOTA claims are not well supported. The intro's SOTA claims rest on a single-model (GPT-5.6-Luna), single-run comparison against four harnesses. On SWE-bench Pro, the evaluation uses 193 of the 731 public tasks without explaining how they were selected, and the margin over Codex CLI is one task (64.25% vs. 63.73%).

  2. The gains of Duo over Base are not shown to come from collaboration. The claim that collaboration provides substantial gains over a single agent is not well supported. For example, on Terminal-Bench 2.1 the +3.4 points is a 6:3 split of discordant tasks, and the authors own sign test gives a p of 0.51. Section 4.3 itself says this is "too few to separate the two."

  3. The adherence study is limited and the metric seems shallow. It uses 36 tasks with one run per configuration, and in practice (if I am reading the appendix correctly?) Adherence only checks that each teammate produced at least one model output, not whether the teammates' work affected the result?

  4. While the writing is better then many of the Opus5 papers I have read, Figure 1 is mostly icons and labels and does not convey the problem or a result, and Figure 2 is dense enough that it is hard to follow before reading Section 3. A trace like Figure 3 or Figure 10, which shows a team failing to collaborate, would motivate the "illusion of collaboration" much more directly. The prompt "cards" need more explanation in the main text. I think Open card defines the reference configuration? But I don't really understand the role or how cards work (this is where I started to explore the appendix which says more but some of this needs to be in the paper.)

Questions for authors

I did not understand the motivation for CACE as presented. Could the authors clarify the following? What question is CACE meant to answer, and how would a practitioner use it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.