- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OpenCollab separates agent orchestration from a shared execution runtime, supporting single agents, model-directed teams, and scripted workflows. It combines declarative roles and communication graphs, runtime restrictions, isolated workspaces, and structured event records. The paper defines structural Adherence and explains when an adherence-adjusted outcome difference identifies a configuration-level complier effect. In the reported Team experiments, Adherence reduces to whether every declared teammate spends tokens and produces output; it does not require incorporation of that output into the final solution. Experiments examine 36 SWE-bench Pro tasks and compare coding harnesses on 193 SWE-bench Pro, 89 Terminal-Bench 2.1, and 113 DeepSWE tasks. A two-solver workflow improves DeepSWE success over the single-agent configuration while consuming more resources.
Critical strengths
-
Useful infrastructure for auditable organizational experiments. The common runtime, declarative role/tool/topology controls, role-attributed events, and worktree provenance provide a coherent basis for comparing organizations and diagnosing failed handoffs (Sections 3.1–3.4; Appendix C). This contribution has value independently of leaderboard superiority: it makes otherwise opaque execution differences observable and supports both model-chosen and scripted coordination on the same substrate.
-
A substantive diagnostic result with unusually explicit limits. Appendix A separates the Wald ratio from the complier effect, states the exclusion and coupling assumptions, and checks the relevant paired contrasts. For the Open configuration, Table 6 reports a +11.8-point Wald ratio but a −5.9-point paired contrast among adherent runs and +15.8 among non-adherent runs. The text appropriately treats these as descriptive and acknowledges that rerunning Single changes 5 of 36 outcomes. The adherence experiments also include confidence intervals and paired comparisons (Tables 2 and 10). These analyses make the warning against attributing gains from configuration labels alone useful and credible.
Critical weaknesses
-
The claimed isolation of organizational dimensions exceeds several experimental contrasts. Table 2 is presented as single-dimension ablations relative to one reference, yet Appendix B.1.1 (pp. 19–20) shows that moving from Star to Ring changes the roster, role cards, and which teammates must participate, as well as the graph. The reference and the “2M / role” Budget row already have the same allowance; their difference is two prompt sentences about available and historical token use. The other budget rows change both allowance and that prompt information. These differences are disclosed, which is valuable, and the Ring-with-shortcut versus Ring comparison does isolate an edge. Nevertheless, the broader framing does not establish separate topology and resource-capacity effects. Since controlled attribution is a central contribution, the conclusions should distinguish those effects from effects of bundled configurations and budget-awareness prompts.
-
Duo's advantage does not establish the additional value of its organization over repeated solving and selection. Appendix B.1.3 describes two complete candidates, different solver instructions, and a selector with an optional adjudicator. On DeepSWE, Duo uses 26.11M average tokens versus Base's 11.95M (Table 4). The 21-versus-5 discordant outcomes and reported exact test, p = 0.0025, support an advantage for the full workflow; the authors also correctly acknowledge the inconclusive Terminal-Bench comparison, p = 0.51 (Table 9). However, no two-run Base control with the same selector and comparable total budget separates role design or information exchange from extra sampling. The manuscript also calls the selector a “fixed rule” without specifying that rule. These limitations narrow the introduction's collaboration-benefit claim, while leaving the demonstrated higher-compute workflow result intact.
Questions for authors
- Clarification of experimental attribution: Which existing Table 2 comparisons support separate effects of topology and token capacity when role rosters, role cards, and budget-information prompts are held fixed? Which conclusions should instead be limited to the bundled configurations?
- Clarification for reproducibility: What exact rule selects between Duo's candidates, what triggers the adjudicator, and what candidate evidence is available at each decision? This would help assess how much of the improvement the selection procedure could explain.
- Limited additional validation: On DeepSWE, can the authors compare Duo with two independent Base candidates under the same selection procedure and comparable total token allowance, reporting actual cost? Alternatively, would they limit the claim to the effectiveness of the complete, more expensive workflow rather than additional value attributable to its role design or information exchange?