- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies where answer-critical information is lost or altered in distributed LLM systems. It proposes an evidence-flow audit that follows benchmark-annotated supports through sharding, extraction, transmission, consolidation, and final aggregation. The audit distinguishes observable evidence losses from candidate aggregation failures, where an answer is incorrect despite complete measured lexical coverage. Across eleven protocols, two benchmarks, and ten models, the authors report extraction and selection losses on Silo-Bench and frequent errors under complete measured coverage on distributed MuSiQue. Additional experiments examine evidence deletion, transmission fidelity, and communication budgets. On 326 audit-flagged failures, replacing the final aggregator with a stronger model produces larger gains than upgrading one local extraction step. The paper presents these results as a basis for diagnosing failures and identifying repair targets, while acknowledging that lexical coverage does not establish semantic sufficiency.
Critical strengths
The paper offers a useful fact-level perspective on distributed reasoning failures. Tracking critical evidence across intermediate stages helps distinguish information loss from potential failures to use available evidence, which final accuracy alone cannot reveal.
Critical weaknesses
-
The paper does not clearly establish a central contribution beyond organizing several related diagnostic analyses. Its motivation is to explain where distributed reasoning fails and use that understanding to identify repair targets. However, the experiments address different pieces of this problem without sufficiently connecting them. The deletion study establishes that annotated supports matter, the budget sweeps examine evidence selection, and Table 9 compares two model-upgrade locations on selected failures. These are useful observations, but they do not collectively establish how reliably the proposed audit distinguishes failure sources or what additional insight its diagnoses provide. The presentation reinforces this gap by introducing successive metrics and findings without consistently explaining how each advances the central argument. Consequently, the framework’s contribution remains less clear than the breadth of the evaluation suggests.
-
The main diagnostic findings depend on a fact-coverage measure whose validity remains uncertain. Appendix B.6 counts any positive token or numerical overlap as coverage, including when generated atomic statements are compared with whole supporting paragraphs. Tables 41–42 show that changing the threshold from 0 to 0.5 reduces MuSiQue’s measured recall from 1.000 to 0.267 and its score-ranked relay’s candidate aggregation-failure rate from 68% to 3%, while the answers remain unchanged. The authors explicitly acknowledge this sensitivity and avoid claiming that lexical coverage proves semantic sufficiency. Nevertheless, without independent validation, the audit cannot reliably establish the distinction central to the paper’s motivation: whether critical information was lost or reached the aggregator but was not successfully used. This uncertainty limits the substantive conclusions that can be drawn from the reported failure patterns.
Questions for authors
None
3 comments
Response to Reviewer CR8J - Part 2/2
W2-a: Independent validation of the fact-coverage measure
We share the concern that word overlap is too coarse to separate a lost fact from an unused one. We have replaced it with an LLM label of whether a message states each critical fact, and validated that label against an independent model on both benchmarks.
Table R4: Coverage under word matching and under the LLM label. Recall: share of critical facts counted as delivered to the final step. Agreement: share of facts on which the LLM label (Claude Haiku 5.5) matches an independent model (Gemini 3.6 Flash), checked on 189 MuSiQue and 515 Silo-Bench facts.
| Benchmark | Recall, word matching | Recall, LLM label | Agreement | 95% CI |
|---|---|---|---|---|
| MuSiQue | 100.0% | 90.2% | 95% | 91–97 |
| Silo-Bench | 84.4% | 93.0% | 91% | 89–94 |
The label is validated and corrects word matching. It agrees with the independent model on more than 90% of facts on both benchmarks. It also corrects word matching in both directions: it rejects MuSiQue facts that matched only on shared words, and it accepts Silo-Bench facts stated in other words.
W2-b: Whether the headline survives the validated measure
The threshold sensitivity the reviewer points out, a drop from 68% to 3% for the score-ranked relay, is a property of word matching. To check the headline without any threshold, we recounted the outcomes of Table R1 under both measures.
Table R5: MuSiQue trace outcomes under word matching and under the LLM label (1,200 traces). The answers are the same under both measures; only the attribution changes.
| Outcome | Word matching | LLM label |
|---|---|---|
| Correct answer | 31% | 31% |
| Wrong: a critical fact was lost before the final step | 0% | 19% |
| Wrong: all critical facts arrived (aggregation failure) | 69% | 50% |
The headline holds. The LLM label moves a fifth of all traces from aggregation failure to lost facts, which are losses that word matching could not see. Even so, aggregation failure remains the largest source of failure, at half of all traces against a fifth for lost facts.
Appreciation and Summary
We deeply appreciate your close reading, especially of our threshold analysis, which pinpointed the paper's most important weakness: word matching could not separate a lost fact from one that arrived unused. We have replaced it with an LLM label that agrees with an independent model on more than 90% of facts, and the central finding holds without any threshold, with aggregation failure remaining the largest source at half of all traces.
Your point on the central contribution also led us to connect the experiments into one argument, from where failures arise, to how the audit tells causes apart, to a targeted repair for each failure mode. Thank you for helping the paper's claims rest on firmer ground.
Response to Reviewer CR8J - Part 1/2
We thank the reviewer for seeing the value of the fact-level view, which separates lost evidence from evidence that arrived but went unused. Following the review, we have built the experiments into one argument, from where failures arise to how the audit locates and repairs them, and we have replaced word matching with a validated LLM label.
W1: The central contribution and how the experiments connect
We agree that the central contribution deserved a clearer statement. That contribution is a fact-level diagnosis that is reliable enough to act on. The revised paper supports it in three steps, each with its own experiment.
Step 1: where failures arise.We sorted 1,200 MuSiQue traces by whether every critical fact reached the final step, using the LLM label validated in W2-a.Half of all traces fail after every fact has arrived, so final accuracy alone cannot show where to intervene.
Step 2: telling causes apart.We built 41 failures whose cause we know, by injecting a distorted message or deleting a critical fact. The audit, Who&When and Who&When Pro each diagnosed every failure with Gemini 3.6 Flash and with Qwen3-235B, and we pooled the two models.The audit takes a different route from existing methods: it traces facts instead of judging the whole trace. It names the failure stage comparably and is the only method that outputs the lost fact as a checked item. The two routes are complementary, since the audit's evidence gives Who&When its most accurate result.
Step 3: from diagnosis to repair.For each failure mode, we applied one repair aimed only at the fact and stage the audit names.One argument.Together, these steps show that a multi-agent system can be debugged one failure at a time, by fixing the stage that lost the fact instead of upgrading every component.