Where Does Reasoning Fail? Attributing Distributed LLM Failures through Critical Evidence
Abstract
Distributed LLM systems often fail even when inter-agent communication is permitted, yet final accuracy alone cannot diagnose where reasoning breaks. We introduce an **evidence-flow audit**, a failure attribution framework that tracks answer-critical evidence through a five-stage pipeline to distinguish extraction and selection losses, transmission distortion, and candidate aggregation failures. The unit of analysis is the fact needed for an answer, rather than message volume alone; benchmark supports and lexical alignment enable systematic tracking of these facts. Auditing eleven protocols across two benchmarks and ten LLMs reveals complementary diagnostic patterns: Silo-Bench exhibits measurable losses before aggregation, while MuSiQue often yields incorrect answers despite complete measured coverage. This pattern highlights the value of examining both evidence fidelity and downstream reasoning. The framework maps these diagnoses to targeted interventions, such as re-extraction, revised selection, and evidence integration. A one-call model-placement comparison on audit-flagged cases finds larger gains from upgrading final aggregation than from upgrading one local extraction step across three tasks, illustrating how the audit can inform intervention placement.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.