acceptodds
Under review as a conference paper at ICLR 2027

Localization is not Attribution: Diagnosing Who-and-When Blame in Multi-Agent LLM Systems

Abstract

When a multi-agent LLM system fails, two questions must be answered to enable repair: when did the trajectory go wrong, and who is responsible. We show that these two questions are far from equally hard. Across four frontier LLM judges on the Who&When benchmark of real multi-agent failures, every method we test—including transcript-reading baselines, structured step-finding, and a strong leave-one-out evidence attribution—localizes the failing step far more often than it correctly attributes the responsible agent. We quantify this gap with a diagnostic metric, rsw: the fraction of correctly localized cases in which the blamed agent is nonetheless wrong. rsw stays stubbornly high (0.19–0.24) for all methods and, crucially, rises on the annotation subsets that multiple judges agree are most trustworthy—evidence that rsw reflects a genuine model limitation, not label noise. Diagnosing the failure, we trace the error to role-charter collapse: when two agents in a real trajectory play overlapping roles, judges summarize them into indistinguishable profiles and then misattribute blame. We propose Role-Op Attribution (ROA), which builds distinctive role charters via leave-one-out contrast and aligns them to per-agent operation signatures. ROA is the first method to significantly reduce rsw (-0.036; 95% CI [-0.068,-0.003], paired bootstrap, n=632) while nudging joint who+when accuracy upward (+0.027; 95% CI [-0.003,0.059]), at 40% lower judge-input cost than the strongest baseline. Component ablation shows the two mechanisms are coupled rather than independent tricks, and self-consistency voting fails to close the gap, confirming that attribution demands a structural fix rather than more sampling. Finally, the same localization–attribution dissociation generalizes to a vision–language multi-agent setting: perception-layer faults are systematically mis-blamed on the reasoner across five visual modalities, both under controlled injection and on 665 naturally occurring VLM failures (a 0.36 who-attribution gap whose 95% CI excludes zero). We release all code and per-case verdicts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent LLM systems, failure attribution, LLM-as-judge, credit assignment, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_k9DXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that in failure attribution for multi-agent LLM systems, locating the decisive step ("when") is much easier than identifying the responsible agent ("who"). It introduces the right-step-wrong-agent rate (rsw) and attributes the gap to "role-charter collapse": agents with overlapping behavior receive near-identical role summaries. It proposes Role-Op Attribution (ROA), which builds rarity-weighted operation fingerprints as distinctive charters and asks the judge to match the located step's operation signature to them. On Who&When, pooled over four frontier judges, ROA reports the lowest rsw (0.174 vs. 0.210 for naive reading) and the highest joint accuracy (0.135 vs. 0.109). Supporting analyses include gold-label verification and human re-annotation arguing that the gap is not label noise, a component ablation, a self-consistency control, a synthetic negative result, and an extension to a vision–language multi-agent system.

Critical strengths
  1. Separating localization errors from attribution errors is a sensible diagnostic goal, and the paper reports per-judge results and a negative synthetic result rather than only pooled numbers.
Critical weaknesses
  1. The central claim is contradicted by the paper's own main results, and the primary metric is ill-defined. In Table 1, "when" accuracy is 0.308–0.320 and "who" accuracy is 0.277–0.418; for evidence-LOO and ROA, who exceeds when. This is also the opposite of the original Who&When finding (best agent-level accuracy 53.5% vs. best step-level accuracy 14.2%, from different methods), which the paper does not discuss, and Figure 2(a)'s "when" accuracies of 0.69–1.00 cannot be reconciled with Table 1. The abstract describes rsw as the fraction of correctly localized cases with the wrong agent (a conditional rate), whereas Section 3.2 defines it as Pr[t̂ = t*, â ≠ a* | located] = when − joint. Under that definition, a method that localizes worse obtains a lower rsw: ROA's "when" is 0.012 below naive's, about a third of its 0.036 rsw reduction, while its joint gain is not significant. Since both the headline phenomenon and the main improvement rest on this metric, this is decisive for my rating.

  2. The mechanism and the statistical claims do not hold up, and many reported numbers are inconsistent. The only quantitative link between charter overlap and rsw comes from the synthetic set (Fig. 7), where ROA hurts (rsw 0.201 → 0.290, Sec. 4.9); on real data, where Sec. 3.3 reports collapse in over a third of tasks, the overlap signal neither separates rsw cases nor tracks ROA's gain (Sec. 4.3, Limitations), and Appendix A states that the synthetic set has no collapse. The label-noise argument stratifies by whether the same judges endorse the gold label, and because rsw = when − joint, it is mechanically higher wherever judges localize better, yet Tables 2–3 report rsw without the corresponding when and joint accuracies; the human who-κ of 0.39 is itself evidence of label ambiguity. The n = 632 pooled cases are four judges on the same 160 tasks; the task-clustered CI [−0.078, +0.001] includes zero, and ROA makes Claude worse (rsw 0.228 → 0.245, joint 0.114 → 0.101) while leaving Doubao flat (Table 5). Further inconsistencies: Fig. 2(b)'s per-judge rsw (e.g., Doubao 0.36) vs. 0.167 in Table 5, with the axis labeled "wrong editor"; a cross-modal gap of 0.812 in the text vs. 0.76 in Fig. 5; and 665 real VLM failures in the text vs. 53 in Appendix C. The cross-modal study also reports only who-accuracy by fault type, so it does not test the who/when dissociation.

Questions for authors
  1. What is the denominator of "when" accuracy in Table 1 and in Figure 2(a), and how do you reconcile Table 1 with the claim that every method localizes far better than it attributes?
  2. Which rsw definition underlies all reported numbers? Please report the conditional rate Pr[â ≠ a* | t̂ = t*] for every method with task-clustered intervals.
  3. Why are 160 tasks used when Who&When contains 184 annotated failure logs, and, since the judges receive the ground-truth answer (Sec. 3.1), do the findings hold in Who&When's setting without it?
Reviewer_bWtcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

“Localization Is Not Attribution” introduces a diagnostic metric and a method for attributing failures in multi-agent systems to individual agents. When a multi-agent system fails, two main questions arise: when did the error occur, and which agent was responsible? Because agents often perform different but overlapping tasks, identifying the step at which an error occurred does not necessarily identify the responsible agent.

The paper uses the WHO&WHEN dataset to evaluate RSW (Right Step, Wrong Agent) and ROA (Role-Op Attribution). RSW measures how often an error is correctly localized but attributed to the wrong agent; lower values are therefore preferable. ROA is the proposed attribution method, designed to reduce RSW.

The authors show that ROA can reduce RSW and improve joint accuracy, meaning that both the error step and the responsible agent are correctly identified. However, the error rate remains high, and ROA reduces accuracy on synthetic errors involving clearly separated roles.

Critical strengths
  1. The paper addresses an important problem and proposes a useful way to distinguish error localization from agent attribution in multi-agent systems.

  2. The authors clearly acknowledge the limitations of their approach and present it as a partial solution.

  3. ROA has lower computational costs than comparable methods.

  4. The paper highlights that both identifiable role profiles and reconstructable operations are needed to improve RSW and joint accuracy.

Critical weaknesses
  1. ROA’s effectiveness depends strongly on the model used as the judge.

  2. The paper does not adequately explain why ROA reduces accuracy in scenarios involving clearly separated roles and synthetic errors.

  3. Elements and labels overlap in Figure 3, making it difficult to read.

  4. The “when” values in Figure 2 range from 0.69 to 1.00, whereas Table 1 reports a pooled value of 0.32. The relationship between these results is unclear. The same issue applies to the “who” values.

  5. Section 4.4 states that “ROA is the only method to move both primary metrics in the right direction (Table 1).” However, Table 1 appears to show that Evidence-LOO also reduces RSW and increases joint accuracy.

Overall Assessment

I recommend a weak reject. The RSW metric appears useful, and ROA offers a promising approach to improving attribution. However, the unresolved discrepancies and insufficiently explained results make the findings difficult to interpret. Combined with ROA’s dependence on the judge model, these issues leave me unconvinced that the improvements are robust.

Questions for authors
  1. Could you clarify which methods and data subsets are represented in Figure 2 and Table 1, and explain how their results relate to each other?

  2. What explains the differences in ROA’s effectiveness across judge models?

  3. Why does ROA reduce accuracy on synthetic errors involving clearly separated roles? The explanation that ROA is unnecessary in this setting does not fully explain the decline.

  4. Could you clarify the claim that ROA is the only method that improves both primary metrics? Table 1 appears to show that Evidence-LOO also reduces RSW and increases joint accuracy.

  5. Could you adjust the spacing in Figure 3 to prevent elements and labels from overlapping?

Reviewer_MKgbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

When a multi-agent LLM system fails, finding when it went wrong is much easier than finding who caused it — this paper names that gap, introduces the rsw (right-step-wrong-agent) diagnostic metric, traces the failure to role-charter collapse (agents with overlapping roles get indistinguishable judge summaries), and proposes Role-Op Attribution (ROA), which builds contrastive operation fingerprints to make charters distinctive. On the WHO&WHEN benchmark, ROA is the first method to significantly reduce rsw (−0.036) while nudging joint accuracy upward, and a vision-language generalization experiment shows the localization–attribution dissociation extends across modalities. The core limitation is that absolute joint accuracy remains modest (0.135), ROA hurts on one of four judges (CLAUDE-OPUS-4.8), and the causal diagnosis holds cleanly only on synthetic data.

Critical strengths
  • the gold-verification study. rsw goes up on the labels everyone agrees on (0.21 to 0.465), which kills the "its just noisy labels" counterargument. clever and convincing, this is the best thing in the paper.
  • honest reporting of negatives. they show ROA hurts on clean synthetic roles and is mixed per judge. most papers hide this, so it buys trust.
Critical weaknesses
  1. Missing citations:
  • Who&When Pro (arXiv:2607.09996): a direct successor to the benchmark used here — its absence from related work is conspicuous.
  • LongRCA Bench (arXiv:2608.15242): the most recent large-scale benchmark; if it supersedes WHO&WHEN in size and annotation quality, claims built on the 160-task real set need reconsideration.
  • Rethinking Failure Attribution (arXiv:2603.25001): proposes a multi-perspective benchmark explicitly covering "who" attribution; direct competitor not compared.
  1. the main "significant" claim leans on an inflated n=632. with the correct unit of analysis (160 tasks) their own one-sided p is 0.029, which is roughly 0.058 two-sided under symmetry, so the one headline result is basicaly on the non-significant side of 0.05.
  2. The mechanistic diagnosis (role-charter collapse) is only confirmed on synthetic data — not on the real WHO&WHEN set where ROA is evaluated and claimed to work.
  3. The CLAUDE-OPUS-4.8 regression (+0.017 rsw, −0.013 joint) is a substantive negative affecting 25% of the judge pool and should appear in the abstract and introduction.
Questions for authors

can you fix weaknesses from above?

  • (clarification) if you make the cluster-robust bootstrap your primary analysis, does the rsw reduction survive as significant, yes or no?
  • (clarification) the CLAUDE-OPUS-4.8 regression (+0.017 rsw, -0.013 joint): is the judge x method interaction significant, and why is claude different?
  • (clarification) on the real set, does anything correlate with rsw (agent count, trajectory lenght, shared tool calls)? if not, how do you defend the charter-collapse story?
  • (suggestion, not scoring) have you tried ROA on the VLM setting, or is only the phenomenon transfered and not the fix?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.