acceptodds
Under review as a conference paper at ICLR 2027

CoordLedger: The Coordination Tax of Multi-Agent Systems

Abstract

Multi-agent LLM systems are usually judged by whether they outperform a single agent. Yet a multi-agent win can come from three different sources—more inference calls, access to more information, or better coordination—and standard comparisons cannot tell them apart. We introduce , a benchmark that compares each communicating system against the single-agent references implied by its deployment access contract. When all task-critical information could be given to one model, the deployment is centralizable, and the reference is a compute-matched single model. When no single role may see all of it, the deployment is structurally distributed, and we use two references: a single agent under the same access restriction, and a compute-matched single agent that sees everything. contains 1,957 instances across five tracks, with 58,248 frozen core runs over six model configurations from five model families. The two regimes behave very differently. On centralizable tasks, the compute-matched single model matches or beats both the Star and Chain topologies in five of six configurations. On structurally distributed tasks, communication clearly helps: Chain raises mean accuracy from 0.188 for the access-restricted agent to 0.335. But the all-access agent reaches 0.495, so communication recovers less than half (48%) of the value that pooling information makes available. We call the remaining gap the coordination tax. Stage-level traces show where it is paid: as information becomes distributed, the dominant failure shifts from delivering evidence to integrating it at the final decision. A multi-agent gain, in short, is only interpretable against the reference its deployment permits.

Primary area:
datasets and benchmarks
Keywords:
Multi-Agent LLMs, Multi-Agent Systems, Agent Evaluation, Benchmarking, Coordination, Access Constraints, Communication Protocols, Information Flow, Coordination Tax

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_w6UbOpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CoordLedger to distinguish the impacts brought about by information access, inference opportunities, and coordination in multi-agent systems. The paper distinguishes between centralized deployment and structurally distributed deployment, and designs three single-agent references: SCM, SDM, and SAA. On five tracks and six model configurations, the authors found that: when centralized access is allowed, communication usually cannot exceed the corresponding centralized reference; while in distributed access, although communication is helpful, it may still lag behind the all-access reference. The paper also attributes the losses further through information-flow traces.

Critical strengths
  1. I believe that the framing of access contract is valuable. By distinguishing between deployment-constrained reference and all-access counterfactual, it is possible to avoid directly attributing the improvement brought about by additional information access to the benefits of coordination.
  2. The experiment as a whole is also quite systematic. The paper includes multiple model series, paired comparisons, trace analysis, and controlled interventions.
Critical weaknesses
  1. I think the conceptual novelty of the paper compared to recent distributed-information benchmarks, especially SILO-BENCH, is relatively incremental. SILO-BENCH has used complete-information single-agent reference and has also reached similar conclusions: agents can obtain distributed information, but still fail in the reasoning/integration stage. Therefore, I believe the actual contribution points of this paper seem smaller than the actual situation.
  2. My explanation of the coordination tax is not entirely convincing. This decomposition is essentially an accounting identity, not a causal decomposition. At the same time, since SAA is not a performance upper bound, the realization rate �0��0� does not necessarily fall within [0,1]. Moreover, the paper emphasizes 48% itself clearly depends on topology: on the same F2, the average coordination residual of Star is much smaller than that of Chain. Finally, the so-called Compute-Matched actually only matches provider-call opportunities and generation budget, and does not match the actual token number or FLOPs.
Questions for authors
  1. What scientific conclusions can CoordLedger obtain that were previously impossible? I hope this paper is not merely reporting a SILO-BENCH style full-information single-agent baseline and a deployment-constrained/no-communication baseline. If this paper can provide specific examples to illustrate that the SDM/SAA ledger actually changed the conclusion, rather than merely reorganizing the existing comparison, it will significantly enhance novelty.
  2. When �0��0�, how should �1��1� be interpreted? Since SAA is not an oracle, is it too strong to call it "fraction of available value"? I also hope the authors can clearly report the realization rate of Star in the same way as they do for F2/Chain.
  3. Can a sensitivity analysis that truly matches by token or compute be added, or can the term "Compute-Matched" be modified? In Appendix H, the actual token usage differences between SCM and multi-agent protocols are quite significant, so I hope to know whether the main conclusions of the centralizable setting still hold under a true token cap.
  4. How sensitive are the coordination tax and integration failure on F2/F3 to simple coordination scaffolds? For example, will the results still hold after adding a clear evidence tracking or reconciliation step?
8 comments

Sorry for the slow reply. We spent some time checking the paired outcomes and finishing additional token-budget and coordination-scaffold experiments.

Thanks for bringing up SILO-BENCH. We agree that a full-information single-agent reference, and the observation that agents can receive information without using it well, are not new on their own. HiddenBench also reports before-discussion, after-discussion, and full-information results. The part we wanted to make explicit in CoordLedger is a little different: which single-agent comparison is actually meaningful under the information access that a deployment permits?

We went back through the F2 cases to see what changes when we add that deployment reference. We also checked the Star realization rates, the token-ceiling results, and the simple coordination scaffolds you suggested. Here is what we found.

AuthorsOpenReview ↗

What the additional comparisons tell us 1

1. What changes relative to the earlier benchmarks?

SILO-BENCH already compares distributed agents with a single model given all the information, and HiddenBench reports local decisions before discussion, decisions after discussion, and full-information performance. We will make those precedents clearer in the related work. We are not claiming that a pooled single-agent baseline or a communication–integration gap was first observed here.

What CoordLedger adds is the deployment access contract as part of the evaluation specification. If one execution is allowed to read the full information, a centralized single model is an available alternative. If no execution can directly pool the role views, the relevant deployment comparison is a single model that operates under that restriction. We use SDM for that comparison and SAA as a separate all-access counterfactual, with the inference opportunity specified for each arm.

The F2 results show why it helps to keep both comparisons. Across six configurations, Chain scores 33.50%. Against its paired all-access SAA (49.50%), that is a 16.00-point shortfall. Against the restricted SDM (18.75%), the same Chain has a 14.75-point gain. Even without the Llama-3-8B configuration, whose SDM has a formatting failure mode, the gain over SDM is 11.90 points.

We can also see the difference on individual questions. In 22 of the 200 Qwen3-8B F2 cases, Chain and SAA-Chain both answer correctly while SDM is incorrect. On one question, for instance, Chain and SAA-Chain both give the English Channel, while SDM returns AMBIGUOUS. Comparing Chain only with SAA records a tie on that question. Comparing it with the allowed restricted single-agent implementation shows the gain from communication. On the 170 F2 items whose annotated required evidence spans shards, the six-configuration average still shows Chain 13.04 points above SDM and 17.55 points below SAA-Chain.

The other deployment regime gives a different reading of a team gain. In centralizable Controlled Communication, Star goes from 23.1% without communication to 78.0% with communication, while the permitted centralized SCM reaches 88.4%. Star clearly helps relative to isolated roles, but the centralized alternative scores higher when the deployment permits it.

These are the conclusions we want the ledger to make easy to read. We do not see the three accuracy differences as new mathematics. The value is in specifying which alternatives the deployment allows and reporting the differences that answer those distinct questions.

2. How should we read the residual and the realization rate?

Your point about ρ\rho is correct. SAA is a measured single-model implementation, not an oracle or a guaranteed upper bound. Equation 4 is an accounting identity over the evaluated outcomes. When the empirical access gap, SAA−SDM\mathrm{SAA - SDM}, is positive, we report ρ=(MA−SDM)/(SAA−SDM)\rho = (\mathrm{MA - SDM})/(\mathrm{SAA - SDM}). It can exceed 1 if MA outperforms SAA, and it can be negative if MA performs below SDM. We report N/A when the denominator is zero or negative.

Here are the aggregate F2 results for the two protocols:

F2, six-configuration meanSDMMAPaired SAAMA−SDM\mathrm{MA - SDM}MA−SAA\mathrm{MA - SAA}ρ\rho
Chain18.75%33.50%49.50%+14.75 pp−16.00 pp0.4797
Star18.75%20.42%22.92%+1.67 pp−2.50 pp0.4000

So the 47.97% figure is specifically the F2/Chain comparison, not a general rate for multi-agent systems. Star's small absolute residual needs some care as well: its paired SAA scores only 22.92%, compared with 49.50% for SAA-Chain. SAA-Star uses pooled evidence in four working calls but a notes-only final call; SAA-Chain has three pooled working calls and a final call that directly sees pooled evidence. Closeness to these two different references does not by itself rank Star and Chain as coordination mechanisms.

We also calculated ρ\rho for each Star configuration, as you requested:

F2/Star modelMA−SDM\mathrm{MA - SDM}SAA−SDM\mathrm{SAA - SDM}ρ\rho
Qwen3-8B+3.50 pp+2.00 pp1.75
Qwen3-32B−3.00 pp−0.50 ppN/A
Llama-3-8B+23.50 pp+24.50 pp0.96
Gemma-3-12B0.00 pp0.00 ppN/A
Ministral-3-14B−15.00 pp+1.00 pp−15.00
Phi-4+1.00 pp−2.00 ppN/A

The Qwen3-8B and Ministral denominators are small, and their paired-bootstrap intervals include zero; those ratios should be read alongside the signed accuracy differences. The Llama row also includes the known SDM formatting failure. We will use the more precise description “deployment gain normalized by the empirical access gap”, rather than implying the ratio must be a percentage between zero and one.

AuthorsOpenReview ↗

What the additional comparisons tell us 2

3. What happens under a shared token ceiling?

In the original experiments, “compute-matched” refers to the configured model-call opportunities and generation limits; it does not mean identical realized token counts or FLOPs. We ran an additional sensitivity analysis that gives the compared arms the same nominal episode ceiling on provider input-plus-output tokens and scores every scheduled episode.

On Controlled Communication, with a 2,432-token ceiling, SCM has the highest point accuracy among SCM, Star, and Chain for both Qwen models. Star−SCM\mathrm{Star - SCM} is −4.0 [−9.2, 1.2] points for 8B and −3.5 [−9.4, 2.1] for 32B; Chain−SCM\mathrm{Chain - SCM} is −26.5 [−31.7, −21.0] and −35.0 [−40.0, −29.6], respectively. With tighter ceilings, however, many repeated-pooling SCM episodes cannot fit and its score falls below the communicating protocols. The one-call full-information baseline has a different cost profile: it remains the highest-scoring of the four evaluated arms at every tested ceiling, including 400/480 correct for Qwen3-8B and 462/480 for Qwen3-32B at 2,432 tokens.

On Natural Transfer, at an 11,136-token ceiling, Star scores 10.2 [7.6, 12.9] points above SCM for Qwen3-8B and 5.7 [2.9, 8.5] above SCM for Qwen3-32B. But 241 and 247 of SCM's scheduled episodes, respectively, cannot fit within that ceiling. On the subset where all four arms finish within budget, the Star−SCM\mathrm{Star - SCM} contrasts are +1.59 [−1.06, 4.24] and −5.59 [−8.24, −3.06] points.

The rankings clearly depend on the budget and on whether a pooled-input schedule can actually run. That is part of the evaluation story, not something a call-count match can settle. We will describe the original setup as matched call opportunity and generation limits and report the shared token-ceiling results separately. Realized token use still differs, and we have not measured FLOPs.

4. Do simple coordination scaffolds change the result?

We tested evidence tracking, reconciliation, tracking plus reconciliation, and a generic check on F2/F3 using the two Qwen models and both topologies.

Across the tested conditions, evidence tracking and reconciliation did not produce a positive final-accuracy gain that survived Holm correction. On F2/Chain, reconciliation changes accuracy by −2.5 [−7.0, 2.0] points for Qwen3-8B and −4.0 [−9.0, 1.0] for Qwen3-32B. Relative to the same fixed SAA-Chain, the corresponding residuals move from −18.0 to −20.5 and from −16.0 to −20.0 points.

Some changes are strongly implementation-dependent. On Qwen3-8B/F2/Chain, adding the tracking condition reduces accuracy from 32.0% to 6.5%, while parse flags rise from 1 to 182 out of 200 runs. On Qwen3-32B/F3/Star, the generic check raises accuracy from 0% to 6% (+6.0 [3.0, 9.5] points, Holm-adjusted p=.00342p = .00342), while parse flags fall from 97 to zero. These are changes in observed task success, but they do not all reflect better integration of evidence. In particular, the generic check's gain coincides with a large improvement in output validity.

There is one related point about the diagnostic categories. The original 52.8% on F2 and 73.7% on F3 count wrong outcomes without a recorded parse or budget failure. The archived records do not verify complete required-evidence visibility at the actual decision input, so we will not describe those percentages as verified integration-stage failures.

Where the traces are complete, we can say more. In the historical controlled intervention, Qwen3-32B carry-forward Chain improves from 207/480 to 441/480 correct answers. Of the remaining failed runs, 36 have all required evidence visible at the final decision, while 3 fail in format or execution. This is the clearest instance-level evidence we have for separating evidence routing from the later decision.

Reviewer_PZp5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CoordLedger, a benchmark for evaluating when multi-agent collaboration is useful. It distinguishes settings where one agent can access all necessary information from settings where access is restricted across roles, and selects single-agent comparisons accordingly.

Critical strengths

The paper addresses a meaningful evaluation problem. A team may outperform a single agent simply because it collectively has access to more information. Explicitly considering access restrictions makes these comparisons more useful and easier to interpret.

The evaluation covers multiple tasks and model configurations. The differences between Star and Chain are particularly informative: they show that the way agents exchange information can substantially affect how much benefit collaboration provides.

Critical weaknesses

My main reservation concerns the depth of the insights. The proposed measures are straightforward comparisons of performance, and much of their value comes from organizing these comparisons clearly.

Questions for authors

The paper describes interventions that preserve evidence or carry it forward through the conversation. Could the authors clarify how much these interventions improve final task accuracy, alongside their effects on the corresponding failure categories?

3 comments

Thank you for your review. You highlighted two parts of CoordLedger that we care about: taking information access into account when comparing agents, and looking at the difference between Star and Chain. Both come from the same question: when a team does better, is it because the agents collectively have more information, or because their communication helps them use it?

You also asked whether the evidence-preserving interventions improve final task accuracy, alongside the changes in failure categories. We checked the archived paired runs for both.

Final accuracy increases by 23.54–90.00 percentage points in five of six comparisons. The remaining Llama-3-8B Star comparison changes by +0.83 points.

In the next response, we share the accuracy and failure-stage results together. We then return to how these observations fit into the broader evaluation question.

AuthorsOpenReview ↗

What changed in the final answers and the failure stages

1. Final task accuracy

We went back to the archived runs for the two interventions described in §5.3 and Appendix I: evidence-preserving Star and carry-forward Chain. We compared each intervention with its corresponding baseline on the same instances and measured the change in final-answer accuracy.

Intervention / modelInstancesAccuracy before → afterGain in percentage points [95% CI]
Star / Qwen3-32B12010.00% → 100.00%+90.00 [84.17, 95.00]
Star / Qwen3-8B12042.50% → 92.50%+50.00 [40.83, 59.17]
Star / Llama-3-8B12084.17% → 85.00%+0.83 [−4.17, 5.83]
Chain / Qwen3-32B48043.13% → 91.88%+48.75 [43.96, 53.33]
Chain / Qwen3-8B48046.25% → 80.00%+33.75 [28.96, 38.75]
Chain / Llama-3-8B48037.92% → 61.46%+23.54 [19.38, 27.71]

The intervals use 10,000 instance-paired bootstrap resamples. Star is evaluated on the 120 Task B graph-reachability instances, and Chain on the 480 controlled synthetic instances, so these are within-protocol intervention comparisons.

Five of the six comparisons show substantial gains in final accuracy. The largest changes appear for the two Qwen models, while Llama-3-8B under Star changes little. That difference is interesting in itself: making evidence available can matter enormously, but its effect depends on what was limiting the final decision in the first place.

2. Changes in the corresponding failure categories

We also followed the required evidence through the recorded traces and compared the failure categories before and after each intervention.

Topology / modelMissing evidencePropagation lossEvidence visible, answer wrong
Star / Qwen3-32B108 → 00 → 00 → 0
Star / Qwen3-8B69 → 00 → 00 → 9
Star / Llama-3-8B13 → 00 → 06 → 18
Chain / Qwen3-32B32 → 0241 → 00 → 36
Chain / Qwen3-8B19 → 0239 → 00 → 94
Chain / Llama-3-8B0 → 0178 → 00 → 65

For Star, the intervention directly forwards the public evidence available to each worker. With Qwen3-32B, the 108 missing-evidence cases disappear, and accuracy rises from 12/120 to 120/120. Qwen3-8B shows a similar pattern: missing-evidence cases fall from 69 to zero, and accuracy rises from 42.5% to 92.5%.

Llama-3-8B is different. Its 13 missing-evidence cases also disappear, but final accuracy barely changes. More runs now reach the decision stage with the required evidence visible and still produce an incorrect answer. So the experiment separates two things that can otherwise look alike: getting the evidence to the decision-maker, and using that evidence to make the right decision.

For Chain, carry-forward preserves previously received evidence as the conversation moves from one agent to the next. The effect is particularly clear for Qwen3-32B. The original 241 propagation-loss cases fall to zero; among those cases, 208 become correct, while 30 reach the final decision with the evidence visible but still answer incorrectly. Across the full 480 paired instances, 239 answers change from wrong to right and five change from right to wrong, giving the net +48.75-point gain above.

We find this progression useful. In several configurations, losing information along the communication path is a major obstacle to answering correctly. Once that path is repaired, the remaining errors are often at the stage where the model has the evidence but still needs to put it together. This connects the failure-stage analysis to the actual task outcomes, which was the point of your question.

AuthorsOpenReview ↗

How this fits into CoordLedger

Stepping back, the main idea behind CoordLedger is simple: whether multi-agent collaboration is useful depends partly on what information the deployment allows a single agent to see.

We start by making that access rule explicit. If one execution is allowed to see all the task-critical information, we compare the communicating team with a single agent given that information and a matched inference opportunity (SCM). If the information has to remain distributed across roles, we use two different references: a single agent under the same access restriction (SDM), and an all-access single-agent counterfactual (SAA). These comparisons answer different questions, and we think it matters to keep them separate.

For example, in Controlled Communication, average accuracy rises from 23.1% without communication to 78.0% with Star. So sharing evidence clearly helps compared with leaving the roles isolated. But the matched centralized reference reaches 88.4%. When centralization is allowed, that changes how we interpret the team's improvement.

F2 gives the other side of the picture. With Qwen3-8B, the access-restricted SDM scores 21.5%, Chain reaches 32.0%, and the all-access SAA reaches 50.0%. Here, communication has a clear practical benefit under the restriction, even though there is still room between the team and the all-access reference.

That is what we wanted CoordLedger to capture: the value communication adds under the actual access rules, and the opportunity that remains when information has to move between agents. The evidence-flow analysis and the interventions are a further step, helping us see where information is lost and how changing its route affects the final answer.

This also connects to your observation about Star and Chain. Their differences are interesting because collaboration is not only about whether agents exchange information, but also about how that information reaches the final decision.

We hope this gives a clearer picture of how the pieces of the paper fit together. If you have any other questions, or if there is a part you would like to discuss further, we would be happy to continue the conversation.

Reviewer_dU17OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce an evaluation framework that separates multiagent gains from advantages in information access and inference budgets. It distinguishes deployments where one agent may access all relevant information from those requiring separate role views, comparing teams against appropriate single agent references. Across many instances in tracks and model configurations, it evaluates Star and Chain communication protocols and diagnoses failures in evidence delivery, propagation, visibility, and integration.

Critical strengths

The paper is easy to read and follow. The empirical study shows broad evaluation and actionable diagnostics. The experiments are designed well and answers the research questions the authors ask. The paper makes information access an explicit part of fair multi agent evaluation. Comparing communicating teams with both an access restricted single agent and an all access reference distinguishes useful communication from advantages gained simply by pooling information.

Critical weaknesses

The coordination tax seems like a descriptive performance gap rather than an isolated measure of coordination failure. The all access reference changes how information is presented and processed, including context length and reasoning, while matching call opportunities and generation budgets does not equalize actual computation.

And the paper's finding that communication recovers less than half of the available value depends strongly on the chosen protocol. On the same distributed evidence track, Star approaches the all access reference in five of six configurations, whereas Chain exhibits a substantial shortfall. This just makes the strongest quantitative conclusion primarily a result about Chain under the evaluated constraints.

Some runs lack message contents and therefore cannot support precise failure attribution, while the integration category identifies an incorrect decision despite visible evidence without explaining why integration failed. So the claimed shift from delivery failures to integration failures provides less complete explanation of causes. It seems that the diagnostic evidence does not fully establish the mechanisms behind the reported gaps.

Questions for authors

How sensitive are the main conclusions to matching actual token use or computational cost rather than call opportunities and generation limits? Will differences in context processing explain part of the measured "coordination tax"?

4 comments

Sorry for getting back to you a little late. We were finishing a few additional token-budget experiments and going back through the saved traces.

Thanks for the detailed review. It was good to see that the access-aware comparison came across clearly. You also made a useful distinction about the coordination tax: the performance gap tells us how far apart two implementations are, but not by itself why they differ. We think that's the right way to read the measure.

We looked again at the F2 Star/Chain references and at what happens under shared token ceilings. A few details turned out to matter quite a lot, especially that Star and Chain use different SAA interfaces and that the ranking changes as pooled inputs become feasible. We've put the numbers and trace results below.

AuthorsOpenReview ↗

The reference, the budget, and the observed failures 1

1. What the coordination tax measures

The way we read Equation 4 is as an accounting identity. By coordination tax, we mean the observed difference between a communicating system and its paired all-access single-agent reference under the benchmark contract. That comparison gives us the size of the gap; the trace analysis is where we look at what happened along the information path.

For F2/Chain, averaged over six configurations, SDM scores 18.75%, Chain 33.50%, and its paired SAA 49.50%. Chain therefore gains 14.75 points over the access-restricted deployment reference, while remaining 16.00 points below the all-access reference. The realization rate is 47.97% for this particular comparison.

You also pointed out that Star is much closer to SAA. There is an important detail behind that observation: Star and Chain use different paired SAA interfaces.

F2, six-configuration meansSDMMulti-agentPaired SAAMA − SDMMA − SAAρ\rho
Chain18.75%33.50%49.50%+14.75 pp−16.00 pp47.97%
Star18.75%20.42%22.92%+1.67 pp−2.50 pp40.00%

In SAA-Star, four working calls receive pooled evidence, followed by a notes-only final call, with a 256-token output limit per call. SAA-Chain has three pooled-evidence working calls and a final call that can read the pooled evidence directly, capped at 384 output tokens. The Star gap is smaller partly because its paired SAA itself scores much lower. Star's absolute accuracy is 20.42%, compared with Chain's 33.50%; the small Star–SAA-Star difference is not evidence that Star reaches the same all-access performance as Chain's reference.

So we agree that 48% should be stated as an F2/Chain result, not a general percentage for multi-agent collaboration. We would also distinguish a protocol's shortfall against its own reference from a comparison that holds one common reference fixed. Those are different experiments.

2. What changes under shared token ceilings

The original compute-matched setup matched provider-call opportunities and generation limits, not realized token totals. We therefore ran an additional sensitivity analysis with shared episode ceilings on input-plus-output tokens. Each row below uses the same 200 F2 instances, and the intervals are instance-paired bootstrap 95% intervals.

Qwen3 model / episode ceilingCorrect Chain / SAA-ChainChain − SAA, pp [95% CI]Budget failures Chain / SAA
8B / 3,584 tokens47 / 0+23.5 [17.5, 29.5]59 / 200
8B / 9,728 tokens64 / 12+26.0 [19.5, 33.0]0 / 164
8B / 13,056 tokens64 / 86−11.0 [−18.0, −3.5]0 / 31
32B / 3,584 tokens60 / 0+30.0 [24.0, 36.5]58 / 200
32B / 9,728 tokens79 / 22+28.5 [21.0, 36.0]0 / 161
32B / 12,928 tokens79 / 96−8.5 [−16.0, −0.5]0 / 31

The pattern is easy to see: at tight ceilings, the pooled SAA input often cannot fit, so Chain finishes more tasks. As the ceiling increases, SAA becomes feasible on more instances and overtakes Chain. At the largest evaluated ceilings, its lead is 11.0 points for Qwen3-8B and 8.5 points for Qwen3-32B. Both SAA arms still have 31 input-infeasible episodes, so these are budget-constrained comparisons, not equal-FLOP estimates.

At the two higher ceilings, Chain continues to outperform the deployment-restricted SDM on F2: +10.5 points [4.0, 17.0] for Qwen3-8B and +16.0 points [10.0, 22.5] for Qwen3-32B, with no recorded budget failures in either Chain–SDM pair. The practical gain under restricted access remains visible, even as the gap to SAA changes with the budget.

We also checked the centralizable tracks. On Controlled Communication, at a 2,432-token ceiling, SCM has the highest point accuracy of SCM, Star, and Chain for both Qwen models; at tighter ceilings, large numbers of SCM episodes cannot fit. On Natural1000 with an 11,136-token ceiling, Star initially leads SCM by 10.2 points for Qwen3-8B and 5.7 points for Qwen3-32B, but SCM has 241 and 247 input-infeasible episodes, respectively. On the common subset where all four arms finish within budget, the contrasts become +1.59 points [−1.06, 4.24] and −5.59 points [−8.24, −3.06].

What this tells us is that fitting the pooled input into the budget matters a lot. A centralized reference can work well when its input fits, but become unusable under a tighter ceiling. The access rule and the inference budget really have to be read together with the score.

AuthorsOpenReview ↗

The reference, the budget, and the observed failures 2

3. What the traces can tell us

On the controlled synthetic tasks, we can directly follow the required evidence into the final decision. The archived carry-forward intervention on Qwen3-32B Chain raises final accuracy from 207/480 to 441/480, or +48.75 points [43.96, 53.33].

Outcome on 480 paired runsOriginal ChainCarry-forward Chain
Correct answer207441
Missing required evidence320
Propagation loss2410
Required evidence visible, answer wrong036
Format/runtime failure03

Of the 241 original propagation-loss cases, 208 become correct, while 30 now reach the decision with the required evidence visible but still answer incorrectly. The routing change has a large effect on final accuracy, and the remaining errors occur later in the information path.

The evidence-preserving Star intervention shows the same distinction from another angle. On 120 Task B instances, Qwen3-32B improves from 12/120 to 120/120 (+90.0 points), while Qwen3-8B gains 50.0 points. Llama-3-8B gains only 0.83 points, even though the required evidence reaches the final decision. Getting the evidence there and making the correct decision with it are related, but separate, problems.

For F2/F3, the archived traces are less specific: the reported 52.8% and 73.7% correspond to incorrect outcomes without recorded parse or budget failures, but the final visibility of every required evidence unit is not established in those records. We therefore separate those outcome counts from the stage-level observations that we can verify in the controlled traces. The controlled interventions provide the clearest account of where an evidence-routing change affects the final answer.

AuthorsOpenReview ↗

What we wanted CoordLedger to capture

The reason we built CoordLedger is a fairly simple question: what should we compare a communicating team with, given the information that the deployment actually allows each agent to see?

We start by specifying that access rule. If the information can be centralized, we compare the team with a single model that can read it with a matched inference opportunity (SCM). If access has to remain distributed, we use both the restricted single-agent deployment (SDM) and an all-access counterfactual (SAA). One comparison tells us whether communication helps under the restriction; the other shows what happens when we allow information to be pooled.

The F2/Chain result shows why we use both: Chain improves by 14.75 points over SDM and remains 16.00 points below its paired SAA. Both numbers tell us something different. The token-ceiling experiments add a practical detail: we also need to know whether the pooled input can actually fit within the available budget.

Once we know which comparison makes sense, we can look at how the evidence moves. The traces and interventions help us see when evidence was lost along the way and when it arrived but was not used successfully. That's also what makes the Star/Chain difference interesting to us: the route matters, and so does the information available to the reference.

We've tried to make those distinctions clearer here. If you'd like to look at any particular result or assumption in more detail, we'd be happy to keep discussing it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.