acceptodds
Under review as a conference paper at ICLR 2027

Where Does Reasoning Fail? Attributing Distributed LLM Failures through Critical Evidence

Abstract

Distributed LLM systems often fail even when inter-agent communication is permitted, yet final accuracy alone cannot diagnose where reasoning breaks. We introduce an **evidence-flow audit**, a failure attribution framework that tracks answer-critical evidence through a five-stage pipeline to distinguish extraction and selection losses, transmission distortion, and candidate aggregation failures. The unit of analysis is the fact needed for an answer, rather than message volume alone; benchmark supports and lexical alignment enable systematic tracking of these facts. Auditing eleven protocols across two benchmarks and ten LLMs reveals complementary diagnostic patterns: Silo-Bench exhibits measurable losses before aggregation, while MuSiQue often yields incorrect answers despite complete measured coverage. This pattern highlights the value of examining both evidence fidelity and downstream reasoning. The framework maps these diagnoses to targeted interventions, such as re-extraction, revised selection, and evidence integration. A one-call model-placement comparison on audit-flagged cases finds larger gains from upgrading final aggregation than from upgrading one local extraction step across three tasks, illustrating how the audit can inform intervention placement.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.