acceptodds
Under review as a conference paper at ICLR 2027

Where Does Reasoning Fail? Attributing Distributed LLM Failures through Critical Evidence

Abstract

Distributed LLM systems often fail even when inter-agent communication is permitted, yet final accuracy alone cannot diagnose where reasoning breaks. We introduce an **evidence-flow audit**, a failure attribution framework that tracks answer-critical evidence through a five-stage pipeline to distinguish extraction and selection losses, transmission distortion, and candidate aggregation failures. The unit of analysis is the fact needed for an answer, rather than message volume alone; benchmark supports and lexical alignment enable systematic tracking of these facts. Auditing eleven protocols across two benchmarks and ten LLMs reveals complementary diagnostic patterns: Silo-Bench exhibits measurable losses before aggregation, while MuSiQue often yields incorrect answers despite complete measured coverage. This pattern highlights the value of examining both evidence fidelity and downstream reasoning. The framework maps these diagnoses to targeted interventions, such as re-extraction, revised selection, and evidence integration. A one-call model-placement comparison on audit-flagged cases finds larger gains from upgrading final aggregation than from upgrading one local extraction step across three tasks, illustrating how the audit can inform intervention placement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent LLMs, distributed reasoning, evidence flow, communication protocols, reasoning evaluation, failure analysis, multi-hop reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CR8JOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies where answer-critical information is lost or altered in distributed LLM systems. It proposes an evidence-flow audit that follows benchmark-annotated supports through sharding, extraction, transmission, consolidation, and final aggregation. The audit distinguishes observable evidence losses from candidate aggregation failures, where an answer is incorrect despite complete measured lexical coverage. Across eleven protocols, two benchmarks, and ten models, the authors report extraction and selection losses on Silo-Bench and frequent errors under complete measured coverage on distributed MuSiQue. Additional experiments examine evidence deletion, transmission fidelity, and communication budgets. On 326 audit-flagged failures, replacing the final aggregator with a stronger model produces larger gains than upgrading one local extraction step. The paper presents these results as a basis for diagnosing failures and identifying repair targets, while acknowledging that lexical coverage does not establish semantic sufficiency.

Critical strengths

The paper offers a useful fact-level perspective on distributed reasoning failures. Tracking critical evidence across intermediate stages helps distinguish information loss from potential failures to use available evidence, which final accuracy alone cannot reveal.

Critical weaknesses
  1. The paper does not clearly establish a central contribution beyond organizing several related diagnostic analyses. Its motivation is to explain where distributed reasoning fails and use that understanding to identify repair targets. However, the experiments address different pieces of this problem without sufficiently connecting them. The deletion study establishes that annotated supports matter, the budget sweeps examine evidence selection, and Table 9 compares two model-upgrade locations on selected failures. These are useful observations, but they do not collectively establish how reliably the proposed audit distinguishes failure sources or what additional insight its diagnoses provide. The presentation reinforces this gap by introducing successive metrics and findings without consistently explaining how each advances the central argument. Consequently, the framework’s contribution remains less clear than the breadth of the evaluation suggests.

  2. The main diagnostic findings depend on a fact-coverage measure whose validity remains uncertain. Appendix B.6 counts any positive token or numerical overlap as coverage, including when generated atomic statements are compared with whole supporting paragraphs. Tables 41–42 show that changing the threshold from 0 to 0.5 reduces MuSiQue’s measured recall from 1.000 to 0.267 and its score-ranked relay’s candidate aggregation-failure rate from 68% to 3%, while the answers remain unchanged. The authors explicitly acknowledge this sensitivity and avoid claiming that lexical coverage proves semantic sufficiency. Nevertheless, without independent validation, the audit cannot reliably establish the distinction central to the paper’s motivation: whether critical information was lost or reached the aggregator but was not successfully used. This uncertainty limits the substantive conclusions that can be drawn from the reported failure patterns.

Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer CR8J - Part 1/2

We thank the reviewer for seeing the value of the fact-level view, which separates lost evidence from evidence that arrived but went unused. Following the review, we have built the experiments into one argument, from where failures arise to how the audit locates and repairs them, and we have replaced word matching with a validated LLM label.


W1: The central contribution and how the experiments connect

We agree that the central contribution deserved a clearer statement. That contribution is a fact-level diagnosis that is reliable enough to act on. The revised paper supports it in three steps, each with its own experiment.

Step 1: where failures arise. We sorted 1,200 MuSiQue traces by whether every critical fact reached the final step, using the LLM label validated in W2-a.

Table R1: Where MuSiQue failures arise (1,200 traces: three models × four protocols × 100 questions).

OutcomeShare
Correct answer31%
Wrong: a critical fact was lost before the final step19%
Wrong: all critical facts arrived (aggregation failure)50%

Half of all traces fail after every fact has arrived, so final accuracy alone cannot show where to intervene.

Step 2: telling causes apart. We built 41 failures whose cause we know, by injecting a distorted message or deleting a critical fact. The audit, Who&When and Who&When Pro each diagnosed every failure with Gemini 3.6 Flash and with Qwen3-235B, and we pooled the two models.

Table R2: Diagnosis of 82 failures with a known cause. Failure stage: share of diagnoses naming the stage where the fact was lost. Lost fact (checked): whether the method outputs the lost critical fact as a checked item, not free text; for the audit, the share of named facts that are correct.

MethodFailure stage95% CILost fact (checked)
Who&When83%73–90❌
Who&When Pro67%56–76❌
Audit (ours)77%67–85✅ 92% correct
Who&When + audit evidence (ours)87%78–92✅ from the audit

The audit takes a different route from existing methods: it traces facts instead of judging the whole trace. It names the failure stage comparably and is the only method that outputs the lost fact as a checked item. The two routes are complementary, since the audit's evidence gives Who&When its most accurate result.

Step 3: from diagnosis to repair. For each failure mode, we applied one repair aimed only at the fact and stage the audit names.

Table R3: One targeted repair per failure mode (MuSiQue). Recovered: share of audit-diagnosed failures answered correctly after the repair; there are 39 extraction, 92 distortion, 17 redundancy and 152 aggregation failures. The aggregation row re-scores the case study of Paper Table 9 (Table 13 in the revision) in the same way. Its cases were flagged by the submission's word-matching audit; on the one model also checked with the LLM label, about a quarter had in fact lost a fact, so this share is conservative.

Failure modeTargeted repairRecovered95% CI
Extraction lossAsk the agent to re-read its documents for the named fact36%23–52
Evidence distortionRestore the source sentence; re-read other flagged facts55%45–65
Redundancy waste (2 slots per agent)Redundancy-aware selection94%73–99
Aggregation failureOne call to a stronger aggregation model20%15–27

One argument. Together, these steps show that a multi-agent system can be debugged one failure at a time, by fixing the stage that lost the fact instead of upgrading every component.

  • Failures hide from accuracy. Many failures arise where final accuracy cannot see them.
  • The audit locates them and names the fact. It finds the failure stage as well as existing methods and also outputs the lost fact as a checked item.
  • The diagnosis guides repairs that work. Each failure mode has a targeted repair that recovers a share of its failures.
AuthorsOpenReview ↗

Response to Reviewer CR8J - Part 2/2

W2-a: Independent validation of the fact-coverage measure

We share the concern that word overlap is too coarse to separate a lost fact from an unused one. We have replaced it with an LLM label of whether a message states each critical fact, and validated that label against an independent model on both benchmarks.

Table R4: Coverage under word matching and under the LLM label. Recall: share of critical facts counted as delivered to the final step. Agreement: share of facts on which the LLM label (Claude Haiku 5.5) matches an independent model (Gemini 3.6 Flash), checked on 189 MuSiQue and 515 Silo-Bench facts.

BenchmarkRecall, word matchingRecall, LLM labelAgreement95% CI
MuSiQue100.0%90.2%95%91–97
Silo-Bench84.4%93.0%91%89–94

The label is validated and corrects word matching. It agrees with the independent model on more than 90% of facts on both benchmarks. It also corrects word matching in both directions: it rejects MuSiQue facts that matched only on shared words, and it accepts Silo-Bench facts stated in other words.


W2-b: Whether the headline survives the validated measure

The threshold sensitivity the reviewer points out, a drop from 68% to 3% for the score-ranked relay, is a property of word matching. To check the headline without any threshold, we recounted the outcomes of Table R1 under both measures.

Table R5: MuSiQue trace outcomes under word matching and under the LLM label (1,200 traces). The answers are the same under both measures; only the attribution changes.

OutcomeWord matchingLLM label
Correct answer31%31%
Wrong: a critical fact was lost before the final step0%19%
Wrong: all critical facts arrived (aggregation failure)69%50%

The headline holds. The LLM label moves a fifth of all traces from aggregation failure to lost facts, which are losses that word matching could not see. Even so, aggregation failure remains the largest source of failure, at half of all traces against a fifth for lost facts.

AuthorsOpenReview ↗

Appreciation and Summary

We deeply appreciate your close reading, especially of our threshold analysis, which pinpointed the paper's most important weakness: word matching could not separate a lost fact from one that arrived unused. We have replaced it with an LLM label that agrees with an independent model on more than 90% of facts, and the central finding holds without any threshold, with aggregation failure remaining the largest source at half of all traces.

Your point on the central contribution also led us to connect the experiments into one argument, from where failures arise, to how the audit tells causes apart, to a targeted repair for each failure mode. Thank you for helping the paper's claims rest on firmer ground.

Reviewer_UEGhOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an evidence-flow audit, a failure diagnosis framework for distributed reasoning systems. Failures in these systems can happen across stages: (1) an agent failed to extract the critical unit (required evidence to answer the problem, as in the paper's own terms), (2) the critical unit was treated as unimportant by the agent and not transmitted for global aggregation, (3) the meaning of the critical unit was distorted during transmission, (4) duplicate evidence consumes message transmission budget without adding information, or (5) the aggregator failed to derive the correct answer given that all necessary critical units arrive. Existing failure attribution systems do not identify whether the error occurs during extraction, transmission, or aggregation. In addition to defining failure modes above, the paper proposes targeted fixes for better recovery.

The evaluation covers two distributed reasoning workloads, 11 protocols falling into four categories, and diverse LLM families. It analyzes extraction and aggregation failures (4.2), whether different ranking criteria select the correct critical unit, and how the budget affects performance (4.3), and presents a case study motivating the use of targeted fixes to improve failure recovery (4.4).

Critical strengths

S1. The paper addresses explainability and failure diagnosis for distributed reasoning systems, a critical and timely problem. A proper automated failure attribution and recovery system enables sustainable, self-governed multi-agent systems.

S2. The case study in Section 4.4 strengthens the motivation for aligning repair with the failure mode to improve the recovery rate.

Critical weaknesses

W1. The paper's organization is difficult to follow. The failure diagnosis framework should be independent of the particular communication protocol. Yet the paper places the audited protocols under Section 3. Moreover, the paper claims that mapping failure patterns to repairs is a contribution, but Section 3 only discusses it briefly at the end of the communication protocol subsection 3.4. The experiments section begins with a Key Finding 1 highlighting the token and accuracy tradeoffs of protocols, which is not directly related to the paper's main contribution of an evidence-flow-based failure diagnosis system.

W2. The experiments do not cover all failure modes. Table 3 shows six metrics per LLM-task-protocol setting, but Section 4's analysis only covers extraction, selection, and aggregation failures. Evidence distortion and Redundancy waste metrics are underdiscussed.

W3. The paper would benefit from a more direct empirical comparison with existing failure attribution frameworks for multi-agent systems. Highlighting a few failure examples where existing frameworks fail to differentiate these failures, but the evidence-flow audit identifies different underlying root causes (e.g., an evidence distortion or aggregation problem) could strengthen the contributions of the work.

W4. While mapping diagnostic patterns to repair targets is one claimed contribution, the experiment only covers one such mapping in Section 4.4. It would help to see how other failure modes can be fixed through targeted repair.

Questions for authors

Q1. The evidence-flow audit seems to require pre-labeled critical units. In Silo-Bench and MuSiQue, these labels come from pre-labeled supporting evidence. Such labels are often not available in practice. Can the authors briefly comment on how this audit system can be applied to real-life workloads?

4 comments
AuthorsOpenReview ↗

Response to Reviewer UEGh - Part 1/3

We thank the reviewer for recognizing failure diagnosis in distributed reasoning as a timely problem, and our case study as support for failure-specific repair. Following the review, we have reorganised the paper around the audit and added experiments on each failure mode and its repair, on existing attribution methods, and on use without gold labels.


W1: Organisation and independence from the audited protocols

We share the view that the framework should stand apart from the protocols it audits. The revised paper is organised as follows.

  • Motivation from practice. Agents increasingly hand tasks to collaborating sub-agents, so more failures arise in the collaboration and must be located quickly.
  • Failure modes. Each critical fact passes through five stages. At each stage, one failure mode can occur: the fact is lost, distorted, crowded out or left unused.
  • Protocol-independent audit. The audit is defined only by checkpoints that track each fact. The protocols move to the experimental setup, and their trade-offs (former Key Finding 1) take one sentence.
  • From diagnosis to repair. A new subsection maps each failure mode to a repair: re-reading for extraction loss, restoring the source sentence for distortion, redundancy-aware selection for redundancy waste, and a stronger aggregator for aggregation failure.
  • Experiments by failure mode. After a reliability comparison with existing methods, the experiments locate where facts are lost, test distortion and redundancy in their own subsections, and apply one targeted repair per failure mode. The introduction now opens with the evidence that every checkpoint is needed.

The paper now makes one argument. Collaboration failures are common, so the audit locates where each critical fact is lost, and each failure mode is then measured and repaired in turn. The protocols serve only as the systems under test.

AuthorsOpenReview ↗

Response to Reviewer UEGh - Part 2/3

W2-a: How often evidence distortion occurs and whether it matters

Thank you for pointing out that evidence distortion appeared only as a metric in Paper Table 3 (Table 5 in the revision). We answer the two questions behind it with two experiments.

Experiment 1: how often. On Silo-Bench relay traces, we compared each extracted critical fact with the message that relays it. We labelled whether the message keeps the meaning, keeps only part of it, or changes it.

Table R1: How often relayed facts lose meaning (845 fact–message pairs, Silo-Bench).

LabelShare
Distorted: meaning changed4.5%
Partly lost or distorted (e.g., a qualifier dropped)29.0%

Experiment 2: whether it causes failures. On MuSiQue failures where the audit found a distorted fact, we replaced the distorted message with its source sentence and re-ran only the final step. As a control, we re-ran that step with the distortion kept.

Table R2: Failed runs answered correctly after the final step is re-run (92 MuSiQue failures with a distorted fact).

Final step re-run withCorrect95% CI
Source sentence restored45%35–55
Distortion kept (control)5%2–12

Distortion is rare but decisive. Only about one relayed fact in twenty is outright distorted, yet undoing a distortion fixes nearly half of the failures it affects. The audit therefore checks fidelity at a checkpoint of its own.


W2-b: Redundancy waste and its cost

We appreciate the reviewer raising redundancy waste, which the paper measured but did not test. Each agent can send only a few items to the final step, so a duplicate can take a slot that a needed fact would otherwise fill.

How agents select. Each agent scores its candidate facts and sends the top ones, up to a fixed number of slots. Score-ranked selection, used in the paper, takes the highest scores. Redundancy-aware selection also skips any candidate that repeats one already chosen.

Experiment setup. On MuSiQue, we gave each agent one, two or three slots. We then added a duplicate of each agent's top candidate and compared accuracy before and after. With one slot, the duplicate can only replace its identical original. With two or three slots, score-ranked selection spends one slot on the duplicate.

Table R3: Accuracy before → after adding one duplicate per agent (MuSiQue, 90 questions, exact match). Slots per agent: items each agent may send, with the share the duplicate takes under score-ranked selection. Δ: change in percentage points.

Slots per agent (taken by duplicate)Score-rankedΔRedundancy-awareΔ
1 (0%)58% → 60%+257% → 59%+2
2 (50%)79% → 61%−1880% → 80%0
3 (33%)92% → 82%−1093% → 90%−3

Redundancy costs accuracy once budgets grow. Its effect depends on how many slots each agent has.

  • One slot. A duplicate is harmless, since it can only replace its identical original.
  • Two or three slots. Under score-ranked selection, the duplicate pushes out a needed fact and costs up to 18 points.
  • Repair. Redundancy-aware selection, the repair for this failure mode, removes most of that loss.
AuthorsOpenReview ↗

Response to Reviewer UEGh - Part 3/3

W3: Whether existing attribution methods can tell the causes apart

The concern. A direct comparison is indeed the clearest test of what the audit adds: do existing frameworks separate the root causes that our audit separates?

Experiment setup. We built 41 MuSiQue failures of known cause, each from an injected distorted message or a deleted critical fact. Who&When, Who&When Pro and our audit each diagnosed every failure with Gemini 3.6 Flash and with Qwen3-235B. Pooled, this gives 82 diagnoses per method.

What we measure. Whether each method names the failure stage, where the fact was lost (e.g., extraction or transmission), and whether it outputs the lost fact.

Table R4: Diagnosis of 82 failures with a known cause. Failure stage: share of diagnoses naming the correct stage. Lost fact (checked): whether the method outputs the lost critical fact as a checked item, not free text; for the audit, the share of named facts that are correct.

MethodFailure stage95% CILost fact (checked)
Who&When83%73–90❌
Who&When Pro67%56–76❌
Audit (ours)77%67–85✅ 92% correct
Who&When + audit evidence (ours)87%78–92✅ from the audit

What the comparison shows. Who&When and Who&When Pro judge the whole trace, while the audit follows each critical fact through the checkpoints. These two routes lead to three results.

  • Comparable on the failure stage. The routes perform comparably, with overlapping intervals.
  • Only the audit outputs the lost fact as a checked item. It is correct in 92% of cases, and a targeted repair needs exactly this information (W4).
  • Complementary routes. Given the audit's evidence, Who&When is the most accurate on the failure stage, and the combined diagnosis also carries the lost fact.

W4: Targeted repairs for the other failure modes

This is a good point: one case study, the stronger aggregator in Paper Table 9 (Table 13 in the revision), does not show that the diagnosis-to-repair mapping holds in general. We now pair each remaining failure mode with one targeted repair that acts only on the fact and stage the audit names, and we count the failures it turns into correct answers.

Table R5: One targeted repair per failure mode (MuSiQue). Recovered: share of audit-diagnosed failures answered correctly after the repair (39 extraction, 92 distortion, 28 redundancy and 152 aggregation failures). The redundancy failures are answers that a duplicate broke (W2-b). The aggregation row re-scores the case study of Paper Table 9 in the same way. Its cases were flagged by the submission's word-matching audit; on the one model also checked with the LLM label, about a quarter had in fact lost a fact, so this share is conservative.

Failure modeTargeted repairRecovered95% CI
Extraction loss: the agent had the fact but did not extract itAsk the agent to re-read its documents for the named fact36%23–52
Evidence distortion: a relayed fact changed meaningRestore the source sentence; re-read other flagged facts55%45–65
Redundancy waste, 2 slots per agentRedundancy-aware selection94%73–99
Redundancy waste, 3 slots per agentRedundancy-aware selection73%43–90
Aggregation failure: all facts arrived, answer wrongOne call to a stronger aggregation model20%15–27

Every failure mode has a repair. These repairs recover a third of extraction losses, over half of distortions, most redundancy failures and a fifth of aggregation failures. Because the audit says which fact failed and where, a system can fix the one failing stage instead of upgrading every component.


Q1: Using the audit without pre-labelled critical facts

We agree that gold critical facts are rarely available in practice. The audit, however, only needs a list of the facts an answer depends on, and at run time a model can write this list from the question and the source documents.

Experiment setup. For 81 failed MuSiQue runs, Qwen3-235B wrote the critical facts without seeing any gold label. We compared the audit's diagnosis on these facts with its diagnosis on gold facts, focusing on early losses: facts lost at sharding or extraction, before any message is sent.

Table R6: Run-time audit (model-written facts) against the gold-fact audit (81 failed MuSiQue runs). Alarm precision: share of run-time early-loss alarms that the gold-fact audit confirms. Catch rate: share of gold-fact early losses that the run-time audit also flags.

MeasureShare95% CI
Alarm precision92%82–97
Catch rate83%71–90

Works with or without gold labels. Gold facts give the most precise diagnosis, but they are not required: with model-written facts, the audit's alarms are almost always confirmed and it catches most gold-fact losses. The audit can use gold labels where they exist and run without them elsewhere.

AuthorsOpenReview ↗

Appreciation and Summary

We deeply appreciate your careful review and its concrete suggestions.

You saw failure diagnosis in distributed reasoning as a timely problem and our case study as support for failure-specific repair, and your request to extend that repair to every failure mode shaped much of our revision. Each suggestion became an experiment:

  • evidence distortion and redundancy waste now have their own tests,
  • a comparison with Who&When and Who&When Pro on failures of known cause shows that only the audit outputs the lost fact,
  • and model-written facts let the audit run without gold labels.

Every failure mode now has one targeted repair, recovering 20–94% of its failures, and the paper is reorganised so that the audit stands apart from the protocols it tests. We hope this addresses your concerns and would welcome any further questions.

Reviewer_chSuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an LLM-based five-stage pipeline for tracking evidence that is critical to producing correct answers. The proposed pipeline is evaluated using 11 protocols across two benchmarks. The paper has provided a lot of details to demonstrate the ideas and vailiatye its effectiveness.

Critical strengths

The underlying research idea appears reasonable.

Critical weaknesses
  • The manuscript is not sufficiently well organised or clearly written, making the proposed approach difficult to follow.

  • The motivation for the proposed pipeline is not adequately established, particularly in relation to existing approaches.

Questions for authors
  • How does the proposed pipeline compare with existing approaches to tracking and attributing agent failures, such as [R1]? Could the authors clarify the distinctive advantages of the proposed pipeline, particularly in terms of the information it captures and the insights it provides beyond existing methods?

  • What motivates the inclusion of each stage in the proposed pipeline? Could the authors provide further justification or experimental evidence demonstrating the contribution and necessity of each stage?

[R1] https://arxiv.org/pdf/2607.09996

5 comments
AuthorsOpenReview ↗

Response to Reviewer chSu - Part 1/2

We thank the reviewer for finding the research idea reasonable and the evaluation detailed. Following the review, we have reorganised the paper around the audit, positioned it against existing attribution methods, and added experiments that compare it with them directly and test each of its stages.


W1: Organisation and clarity of the manuscript

We have revised the organisation so that the audit can be followed from motivation to repair.

  • Motivation from practice. Agents increasingly hand tasks to collaborating sub-agents, so more failures arise in the collaboration and must be located quickly.
  • Failure modes. Each critical fact passes through five stages, and at each stage one failure mode can occur: the fact is lost, distorted, crowded out or left unused.
  • Protocol-independent audit. The audit is defined only by checkpoints that track each fact. The protocols move to the experimental setup as the systems under test.
  • From diagnosis to repair. A new subsection maps each failure mode to a targeted repair, such as re-reading for a missed fact or restoring a distorted sentence.
  • Experiments by question. The experiments test the audit's reliability against existing methods, then examine each failure mode and its repair. The introduction now opens with the evidence that every checkpoint is needed (Q2).

The paper now makes one argument: collaboration failures are common, the audit locates where each critical fact is lost, and each failure mode is measured and repaired in turn.


W2: Motivation relative to existing approaches

We agree that the paper should state plainly why a new pipeline is needed when attribution methods already exist. To make the gap explicit, we compared what each method's diagnosis tells a user, based on each method's own paper.

Table R1: What each method's diagnosis reports. ✅ reported; ❌ not reported; – not applicable.

The diagnosis tells you…Who&WhenMASTWho&When ProOurs
Which agent or step failed✅❌✅✅
What kind of failure it wasFree text✅ 14 modes✅ error mode✅
Which fact was lostFree text❌Free text✅
Where in the pipeline it was lostVia step❌Via step✅
Whether it never arrived or arrived unused❌❌❌✅
Guides a working repair–Category level–✅

The gap the audit fills. Existing methods say who failed or what kind of failure it was, and may describe the lost fact in free text. None reports which fact was lost, where, and whether it arrived as checked outputs, which is exactly what a targeted repair needs.

AuthorsOpenReview ↗

Response to Reviewer chSu - Part 2/2

Q1: What the audit adds beyond existing methods such as Who&When Pro

A direct comparison is indeed the clearest way to show what the audit captures beyond existing methods.

Experiment setup. We built 41 MuSiQue failures of known cause, each from an injected distorted message or a deleted critical fact. Who&When, Who&When Pro and our audit each diagnosed every failure with Gemini 3.6 Flash and with Qwen3-235B; pooled, this gives 82 diagnoses per method.

What we measure. Whether each method names the failure stage, where the fact was lost (e.g., extraction or transmission), and whether it outputs the lost fact.

Table R2: Diagnosis of 82 failures with a known cause. Failure stage: share of diagnoses naming the correct stage. Lost fact (checked): whether the method outputs the lost critical fact as a checked item, not free text; for the audit, the share of named facts that are correct.

MethodFailure stage95% CILost fact (checked)
Who&When83%73–90❌
Who&When Pro67%56–76❌
Audit (ours)77%67–85✅ 92% correct
Who&When + audit evidence (ours)87%78–92✅ from the audit

What the comparison shows. Who&When and Who&When Pro judge the whole trace, while the audit follows each critical fact through the checkpoints. The two routes lead to three results.

  • Comparable on the failure stage. The routes perform comparably, with overlapping intervals.
  • Only the audit outputs the lost fact as a checked item. It is correct in 92% of cases, and a targeted repair needs exactly this information.
  • Complementary routes. Given the audit's evidence, Who&When is the most accurate on the failure stage, and the combined diagnosis also carries the lost fact.

Q2: Why each stage of the pipeline is needed

Thank you for asking for evidence on each stage. Each stage ends in a check that catches one failure mode, so a stage is needed if, once its check is removed, the audit misplaces or misses failures.

Experiment setup. On relay traces, 2,395 on MuSiQue (Gemma-4-31B, GPT-4o-mini and Qwen3-235B, pooled) and 156 on Silo-Bench (Qwen3-235B), we switched off one check at a time and recorded which traces change diagnosis and where they go. This set differs from the 1,200 traces in our other responses, and distorted facts form their own group here.

Table R3: Effect of switching off one check. The benchmark columns give the share of all traces whose diagnosis changes. The fidelity check is not run on Silo-Bench, which has no validated distortion label.

Check removedFailures only it catchesMuSiQueSilo-BenchWithout it, they are…
ExtractionFact never picked up31%17%Blamed on selection (MuSiQue) or the final step (Silo-Bench)
FidelityFact arrived distorted6%–Blamed on the final step, or missed
Final answerFact arrived but went unused33%67%Missed

Every stage is needed. Each check catches failures that no other check sees. Without it, up to a third of MuSiQue traces are blamed on the wrong stage or missed entirely.

AuthorsOpenReview ↗

From failure to repair: end-to-end evidence - Part 1/2

To show that the reorganised paper (W1) forms one argument, we bring together the experiments added in this rebuttal. They follow it in four steps: failures arise where accuracy cannot see them, the audit locates them reliably, each failure mode matters, and each has a targeted repair.

Step 1: where failures arise. We sorted 1,200 MuSiQue traces by whether every critical fact reached the final step, using the LLM label validated in Step 2.

Table R4: Where MuSiQue failures arise (1,200 traces: three models × four protocols × 100 questions).

OutcomeShare
Correct answer31%
Wrong: a critical fact was lost before the final step19%
Wrong: all critical facts arrived (aggregation failure)50%

Failures hide from accuracy. Half of all traces fail after every fact has arrived, and a fifth lose a fact on the way. Final accuracy alone cannot tell these apart or show where to intervene.

Step 2: the audit is reliable. The audit decides whether a message states each critical fact with an LLM label, which replaces the submission's word matching. We checked this label against an independent model on both benchmarks.

Table R5: Agreement of the audit's fact label (Claude Haiku 5.5) with an independent model (Gemini 3.6 Flash). 189 MuSiQue and 515 Silo-Bench facts.

BenchmarkAgreement95% CI
MuSiQue95%91–97
Silo-Bench91%89–94

Run-time use. Gold critical facts are rarely available in practice. For 81 failed MuSiQue runs, Qwen3-235B wrote the critical facts from the question and documents alone, and we compared the audit's diagnosis on these facts with its diagnosis on gold facts. We focus on early losses: facts lost at sharding or extraction, before any message is sent.

Table R6: Run-time audit (model-written facts) against the gold-fact audit (81 failed MuSiQue runs). Alarm precision: share of run-time early-loss alarms that the gold-fact audit confirms. Catch rate: share of gold-fact early losses that the run-time audit also flags.

MeasureShare95% CI
Alarm precision92%82–97
Catch rate83%71–90

Reliable, with or without gold labels. The label agrees with an independent model on more than 90% of facts, and with model-written facts the audit's alarms are almost always confirmed. Q1 and Q2 above add that the audit matches existing methods on the failure stage, and that each of its checks is needed.

Step 3: each failure mode matters. The submission measured evidence distortion and redundancy waste but did not test them. For distortion, on MuSiQue failures where the audit found a distorted fact, we replaced the distorted message with its source sentence and re-ran only the final step. As a control, we re-ran that step with the distortion kept.

Table R7: Failed runs answered correctly after the final step is re-run (92 MuSiQue failures with a distorted fact). The repair in Table R9-Row 2 also re-reads other flagged facts.

Final step re-run withCorrect95% CI
Source sentence restored45%35–55
Distortion kept (control)5%2–12

Redundancy setup. Each agent scores its candidate facts and sends the top ones, up to a fixed number of slots. Score-ranked selection, used in the paper, takes the highest scores; redundancy-aware selection also skips any candidate that repeats one already chosen. On MuSiQue, we gave each agent one, two or three slots, added a duplicate of each agent's top candidate, and compared accuracy before and after.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.