- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that in failure attribution for multi-agent LLM systems, locating the decisive step ("when") is much easier than identifying the responsible agent ("who"). It introduces the right-step-wrong-agent rate (rsw) and attributes the gap to "role-charter collapse": agents with overlapping behavior receive near-identical role summaries. It proposes Role-Op Attribution (ROA), which builds rarity-weighted operation fingerprints as distinctive charters and asks the judge to match the located step's operation signature to them. On Who&When, pooled over four frontier judges, ROA reports the lowest rsw (0.174 vs. 0.210 for naive reading) and the highest joint accuracy (0.135 vs. 0.109). Supporting analyses include gold-label verification and human re-annotation arguing that the gap is not label noise, a component ablation, a self-consistency control, a synthetic negative result, and an extension to a vision–language multi-agent system.
Critical strengths
- Separating localization errors from attribution errors is a sensible diagnostic goal, and the paper reports per-judge results and a negative synthetic result rather than only pooled numbers.
Critical weaknesses
-
The central claim is contradicted by the paper's own main results, and the primary metric is ill-defined. In Table 1, "when" accuracy is 0.308–0.320 and "who" accuracy is 0.277–0.418; for evidence-LOO and ROA, who exceeds when. This is also the opposite of the original Who&When finding (best agent-level accuracy 53.5% vs. best step-level accuracy 14.2%, from different methods), which the paper does not discuss, and Figure 2(a)'s "when" accuracies of 0.69–1.00 cannot be reconciled with Table 1. The abstract describes rsw as the fraction of correctly localized cases with the wrong agent (a conditional rate), whereas Section 3.2 defines it as Pr[t̂ = t*, â ≠ a* | located] = when − joint. Under that definition, a method that localizes worse obtains a lower rsw: ROA's "when" is 0.012 below naive's, about a third of its 0.036 rsw reduction, while its joint gain is not significant. Since both the headline phenomenon and the main improvement rest on this metric, this is decisive for my rating.
-
The mechanism and the statistical claims do not hold up, and many reported numbers are inconsistent. The only quantitative link between charter overlap and rsw comes from the synthetic set (Fig. 7), where ROA hurts (rsw 0.201 → 0.290, Sec. 4.9); on real data, where Sec. 3.3 reports collapse in over a third of tasks, the overlap signal neither separates rsw cases nor tracks ROA's gain (Sec. 4.3, Limitations), and Appendix A states that the synthetic set has no collapse. The label-noise argument stratifies by whether the same judges endorse the gold label, and because rsw = when − joint, it is mechanically higher wherever judges localize better, yet Tables 2–3 report rsw without the corresponding when and joint accuracies; the human who-κ of 0.39 is itself evidence of label ambiguity. The n = 632 pooled cases are four judges on the same 160 tasks; the task-clustered CI [−0.078, +0.001] includes zero, and ROA makes Claude worse (rsw 0.228 → 0.245, joint 0.114 → 0.101) while leaving Doubao flat (Table 5). Further inconsistencies: Fig. 2(b)'s per-judge rsw (e.g., Doubao 0.36) vs. 0.167 in Table 5, with the axis labeled "wrong editor"; a cross-modal gap of 0.812 in the text vs. 0.76 in Fig. 5; and 665 real VLM failures in the text vs. 53 in Appendix C. The cross-modal study also reports only who-accuracy by fault type, so it does not test the who/when dissociation.
Questions for authors
- What is the denominator of "when" accuracy in Table 1 and in Figure 2(a), and how do you reconcile Table 1 with the claim that every method localizes far better than it attributes?
- Which rsw definition underlies all reported numbers? Please report the conditional rate Pr[â ≠ a* | t̂ = t*] for every method with task-clustered intervals.
- Why are 160 tasks used when Who&When contains 184 annotated failure logs, and, since the judges receive the ground-truth answer (Sec. 3.1), do the findings hold in Who&When's setting without it?