- Rating
- 2(+1)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CoordLedger to distinguish the impacts brought about by information access, inference opportunities, and coordination in multi-agent systems. The paper distinguishes between centralized deployment and structurally distributed deployment, and designs three single-agent references: SCM, SDM, and SAA. On five tracks and six model configurations, the authors found that: when centralized access is allowed, communication usually cannot exceed the corresponding centralized reference; while in distributed access, although communication is helpful, it may still lag behind the all-access reference. The paper also attributes the losses further through information-flow traces.
Critical strengths
- I believe that the framing of access contract is valuable. By distinguishing between deployment-constrained reference and all-access counterfactual, it is possible to avoid directly attributing the improvement brought about by additional information access to the benefits of coordination.
- The experiment as a whole is also quite systematic. The paper includes multiple model series, paired comparisons, trace analysis, and controlled interventions.
Critical weaknesses
- I think the conceptual novelty of the paper compared to recent distributed-information benchmarks, especially SILO-BENCH, is relatively incremental. SILO-BENCH has used complete-information single-agent reference and has also reached similar conclusions: agents can obtain distributed information, but still fail in the reasoning/integration stage. Therefore, I believe the actual contribution points of this paper seem smaller than the actual situation.
- My explanation of the coordination tax is not entirely convincing. This decomposition is essentially an accounting identity, not a causal decomposition. At the same time, since SAA is not a performance upper bound, the realization rate does not necessarily fall within [0,1]. Moreover, the paper emphasizes 48% itself clearly depends on topology: on the same F2, the average coordination residual of Star is much smaller than that of Chain. Finally, the so-called Compute-Matched actually only matches provider-call opportunities and generation budget, and does not match the actual token number or FLOPs.
Questions for authors
- What scientific conclusions can CoordLedger obtain that were previously impossible? I hope this paper is not merely reporting a SILO-BENCH style full-information single-agent baseline and a deployment-constrained/no-communication baseline. If this paper can provide specific examples to illustrate that the SDM/SAA ledger actually changed the conclusion, rather than merely reorganizing the existing comparison, it will significantly enhance novelty.
- When , how should be interpreted? Since SAA is not an oracle, is it too strong to call it "fraction of available value"? I also hope the authors can clearly report the realization rate of Star in the same way as they do for F2/Chain.
- Can a sensitivity analysis that truly matches by token or compute be added, or can the term "Compute-Matched" be modified? In Appendix H, the actual token usage differences between SCM and multi-agent protocols are quite significant, so I hope to know whether the main conclusions of the centralizable setting still hold under a true token cap.
- How sensitive are the coordination tax and integration failure on F2/F3 to simple coordination scaffolds? For example, will the results still hold after adding a clear evidence tracking or reconciliation step?
8 comments
What the additional comparisons tell us 1
1. What changes relative to the earlier benchmarks?
SILO-BENCH already compares distributed agents with a single model given all the information, and HiddenBench reports local decisions before discussion, decisions after discussion, and full-information performance. We will make those precedents clearer in the related work. We are not claiming that a pooled single-agent baseline or a communication–integration gap was first observed here.
What CoordLedger adds is the deployment access contract as part of the evaluation specification. If one execution is allowed to read the full information, a centralized single model is an available alternative. If no execution can directly pool the role views, the relevant deployment comparison is a single model that operates under that restriction. We use SDM for that comparison and SAA as a separate all-access counterfactual, with the inference opportunity specified for each arm.
The F2 results show why it helps to keep both comparisons. Across six configurations, Chain scores 33.50%. Against its paired all-access SAA (49.50%), that is a 16.00-point shortfall. Against the restricted SDM (18.75%), the same Chain has a 14.75-point gain. Even without the Llama-3-8B configuration, whose SDM has a formatting failure mode, the gain over SDM is 11.90 points.
We can also see the difference on individual questions. In 22 of the 200 Qwen3-8B F2 cases, Chain and SAA-Chain both answer correctly while SDM is incorrect. On one question, for instance, Chain and SAA-Chain both give the English Channel, while SDM returns AMBIGUOUS. Comparing Chain only with SAA records a tie on that question. Comparing it with the allowed restricted single-agent implementation shows the gain from communication. On the 170 F2 items whose annotated required evidence spans shards, the six-configuration average still shows Chain 13.04 points above SDM and 17.55 points below SAA-Chain.
The other deployment regime gives a different reading of a team gain. In centralizable Controlled Communication, Star goes from 23.1% without communication to 78.0% with communication, while the permitted centralized SCM reaches 88.4%. Star clearly helps relative to isolated roles, but the centralized alternative scores higher when the deployment permits it.
These are the conclusions we want the ledger to make easy to read. We do not see the three accuracy differences as new mathematics. The value is in specifying which alternatives the deployment allows and reporting the differences that answer those distinct questions.
2. How should we read the residual and the realization rate?
Your point about is correct. SAA is a measured single-model implementation, not an oracle or a guaranteed upper bound. Equation 4 is an accounting identity over the evaluated outcomes. When the empirical access gap, , is positive, we report . It can exceed 1 if MA outperforms SAA, and it can be negative if MA performs below SDM. We report N/A when the denominator is zero or negative.
Here are the aggregate F2 results for the two protocols:
| F2, six-configuration mean | SDM | MA | Paired SAA | |||
|---|---|---|---|---|---|---|
| Chain | 18.75% | 33.50% | 49.50% | +14.75 pp | −16.00 pp | 0.4797 |
| Star | 18.75% | 20.42% | 22.92% | +1.67 pp | −2.50 pp | 0.4000 |
So the 47.97% figure is specifically the F2/Chain comparison, not a general rate for multi-agent systems. Star's small absolute residual needs some care as well: its paired SAA scores only 22.92%, compared with 49.50% for SAA-Chain. SAA-Star uses pooled evidence in four working calls but a notes-only final call; SAA-Chain has three pooled working calls and a final call that directly sees pooled evidence. Closeness to these two different references does not by itself rank Star and Chain as coordination mechanisms.
We also calculated for each Star configuration, as you requested:
| F2/Star model | |||
|---|---|---|---|
| Qwen3-8B | +3.50 pp | +2.00 pp | 1.75 |
| Qwen3-32B | −3.00 pp | −0.50 pp | N/A |
| Llama-3-8B | +23.50 pp | +24.50 pp | 0.96 |
| Gemma-3-12B | 0.00 pp | 0.00 pp | N/A |
| Ministral-3-14B | −15.00 pp | +1.00 pp | −15.00 |
| Phi-4 | +1.00 pp | −2.00 pp | N/A |
The Qwen3-8B and Ministral denominators are small, and their paired-bootstrap intervals include zero; those ratios should be read alongside the signed accuracy differences. The Llama row also includes the known SDM formatting failure. We will use the more precise description “deployment gain normalized by the empirical access gap”, rather than implying the ratio must be a percentage between zero and one.
What the additional comparisons tell us 2
3. What happens under a shared token ceiling?
In the original experiments, “compute-matched” refers to the configured model-call opportunities and generation limits; it does not mean identical realized token counts or FLOPs. We ran an additional sensitivity analysis that gives the compared arms the same nominal episode ceiling on provider input-plus-output tokens and scores every scheduled episode.
On Controlled Communication, with a 2,432-token ceiling, SCM has the highest point accuracy among SCM, Star, and Chain for both Qwen models. is −4.0 [−9.2, 1.2] points for 8B and −3.5 [−9.4, 2.1] for 32B; is −26.5 [−31.7, −21.0] and −35.0 [−40.0, −29.6], respectively. With tighter ceilings, however, many repeated-pooling SCM episodes cannot fit and its score falls below the communicating protocols. The one-call full-information baseline has a different cost profile: it remains the highest-scoring of the four evaluated arms at every tested ceiling, including 400/480 correct for Qwen3-8B and 462/480 for Qwen3-32B at 2,432 tokens.
On Natural Transfer, at an 11,136-token ceiling, Star scores 10.2 [7.6, 12.9] points above SCM for Qwen3-8B and 5.7 [2.9, 8.5] above SCM for Qwen3-32B. But 241 and 247 of SCM's scheduled episodes, respectively, cannot fit within that ceiling. On the subset where all four arms finish within budget, the contrasts are +1.59 [−1.06, 4.24] and −5.59 [−8.24, −3.06] points.
The rankings clearly depend on the budget and on whether a pooled-input schedule can actually run. That is part of the evaluation story, not something a call-count match can settle. We will describe the original setup as matched call opportunity and generation limits and report the shared token-ceiling results separately. Realized token use still differs, and we have not measured FLOPs.
4. Do simple coordination scaffolds change the result?
We tested evidence tracking, reconciliation, tracking plus reconciliation, and a generic check on F2/F3 using the two Qwen models and both topologies.
Across the tested conditions, evidence tracking and reconciliation did not produce a positive final-accuracy gain that survived Holm correction. On F2/Chain, reconciliation changes accuracy by −2.5 [−7.0, 2.0] points for Qwen3-8B and −4.0 [−9.0, 1.0] for Qwen3-32B. Relative to the same fixed SAA-Chain, the corresponding residuals move from −18.0 to −20.5 and from −16.0 to −20.0 points.
Some changes are strongly implementation-dependent. On Qwen3-8B/F2/Chain, adding the tracking condition reduces accuracy from 32.0% to 6.5%, while parse flags rise from 1 to 182 out of 200 runs. On Qwen3-32B/F3/Star, the generic check raises accuracy from 0% to 6% (+6.0 [3.0, 9.5] points, Holm-adjusted ), while parse flags fall from 97 to zero. These are changes in observed task success, but they do not all reflect better integration of evidence. In particular, the generic check's gain coincides with a large improvement in output validity.
There is one related point about the diagnostic categories. The original 52.8% on F2 and 73.7% on F3 count wrong outcomes without a recorded parse or budget failure. The archived records do not verify complete required-evidence visibility at the actual decision input, so we will not describe those percentages as verified integration-stage failures.
Where the traces are complete, we can say more. In the historical controlled intervention, Qwen3-32B carry-forward Chain improves from 207/480 to 441/480 correct answers. Of the remaining failed runs, 36 have all required evidence visible at the final decision, while 3 fail in format or execution. This is the clearest instance-level evidence we have for separating evidence routing from the later decision.
Sorry for the slow reply. We spent some time checking the paired outcomes and finishing additional token-budget and coordination-scaffold experiments.
Thanks for bringing up SILO-BENCH. We agree that a full-information single-agent reference, and the observation that agents can receive information without using it well, are not new on their own. HiddenBench also reports before-discussion, after-discussion, and full-information results. The part we wanted to make explicit in CoordLedger is a little different: which single-agent comparison is actually meaningful under the information access that a deployment permits?
We went back through the F2 cases to see what changes when we add that deployment reference. We also checked the Star realization rates, the token-ceiling results, and the simple coordination scaffolds you suggested. Here is what we found.