- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces the idea of multi-perspective failure attribution of multi-agent systems, unlike the prior work that identifies a single step as failure inducing. Based on expert annotation, a benchmark of 289 annotated trajectories is constructed and analyzed, which includes per-step failure labels, rationale and an ideal action. Findings show that there is substantial disagreement between human annotators and based on a manual analysis of sampled conflicts, disagreements mostly reflect multiple valid perspectives. Further, they evaluate five LLMs on this task. For this, multiple samples are generated and ranked based on consensus. NDCG score is used to compare the predicted ranking against human ranking, which are also produced based on consensus. Results show that LLMs perform better than a random baseline on this task.
Critical strengths
- The paper introduces a novel and useful multi-perspective failure attribution benchmark.
- The benchmark consists of logs annotated by multiple experts with detailed analyses of the annotation quality.
- Experimental results show that when sampled multiple times, LLMs perform well at this task.
Critical weaknesses
- The downstream utility of multi-perspective failure attribution compared to a single failure is not yet established.
- The annotation instructions ask whether each step is correct, not whether it lead to the overall trajectory failure. This is different from failure attribution defined in Who&When benchmark and in Section 3.2.
- The reasoning evaluation judge is not rigorously validated by humans.
Questions for authors
- How are ties handles in computing NDCG? Since both the GT and the predictions are coarse-grained (=1,2 or 3 for N=3), how you handle ties could substantially impact the NDCG scores.
- What is the random baseline reported in Table 3?
3 comments
Response to W3
We thank the reviewer for raising this important concern. To validate our reasoning evaluation judge, we conducted an additional human evaluation study involving 100 (model reasoning, human reasoning) pairs, stratified by dataset, model backbone, and judge score.
An expert annotator evaluated each pair in two stages. First, without access to the judge's output, the annotator independently assigned a coarse rating (low / mid / high). For the blind agreement analysis, we mapped the judge's 1–10 scores into three categories (low: 1–3, mid: 4–6, high: 7–10) and compared them with the annotator's ratings using weighted Cohen's κ and Spearman's ρ. Second, after reviewing the judge's score and rationale, the annotator assessed whether the judge's evaluation was appropriate.
| Metric | Value (95% CI) |
|---|---|
| Blind agreement (weighted κ) | 0.81 [0.70, 0.89] |
| Blind agreement (Spearman ρ) | 0.81 [0.71, 0.89], p < 10⁻²⁰ |
| Judge assessment deemed appropriate | 88% |
The results demonstrate strong agreement between the reasoning judge and independent human assessments, with both weighted κ and Spearman ρ reaching 0.81. Furthermore, the annotator considered 88% of the judge's assessments appropriate after reviewing their scores and rationales.
Most disagreements occurred in the mid-score range. The judge also showed a slight tendency toward stricter scoring, with 8% of cases under-scored versus 4% over-scored relative to the human assessment. In particular, disagreements often arose when the judge penalized mismatches in fine-grained details despite agreement on the high-level failure reason and ideal action.
These findings provide empirical support for the reliability of our reasoning evaluation judge, while acknowledging that the validation is based on a sample evaluated by a single expert annotator. We have included the validation protocol and detailed results in the revised manuscript.
Revised manuscript: Section 6 (Attribution Reasoning Evaluation, summary of the validation); Appendix H.6 (protocol and Table 11); Appendix G.2 (pointer).
Response to Q1
We thank the reviewer for highlighting this important issue. In our original evaluation, we resolved ties chronologically, prioritizing earlier steps based on the intuition that errors occurring earlier in an execution trajectory may have a greater influence on subsequent agent behaviors and therefore warrant higher priority for system refinement.
Motivated by the reviewer's concern, we further investigated an alternative evaluation approach that explicitly accounts for ties, which frequently arise in our vote-based rankings. Specifically, we explored a tie-aware nDCG formulation following [5]. For ground truth, steps receiving equal votes are assigned the average relevance of their occupied ranks. For predictions, tied steps are evaluated using the expected DCG over their possible orderings, avoiding an arbitrary preference among equally ranked steps.
| Model | Hand-Crafted nDCG@5 | Hand-Crafted nDCG@full | Automatic nDCG@5 | Automatic nDCG@full |
|---|---|---|---|---|
| GPT-4.1 | 0.4908 | 0.5337 | 0.6785 | 0.6821 |
| GPT-5.1 | 0.4813 | 0.5746 | 0.8062 | 0.8116 |
| Sonnet 4.5 | 0.5073 | 0.5811 | 0.8029 | 0.8130 |
| Qwen3-8B | 0.3648 | 0.4040 | 0.6690 | 0.6857 |
| GPT-OSS-120B | 0.4861 | 0.5475 | 0.6978 | 0.7112 |
The results remain broadly consistent with our original findings, with Sonnet 4.5 demonstrating consistently strong performance across all evaluation settings, while Qwen3-8B generally exhibits the weakest performance. This additional analysis demonstrates that our main findings remain consistent under an evaluation approach that does not rely on chronological tie-breaking. We retain the original results for comparability and have included the tie-aware evaluation as an additional robustness analysis in the revised manuscript, along with a clarification of the original tie-breaking procedure.
Revised manuscript: Section 6 (Multi-Perspective Failure Attribution Evaluation, tie-breaking procedure and its rationale); Appendix G.2 (Tie-breaking); Appendix H.4 (Table 9).
[5] McSherry and Najork, Computing Information Retrieval Performance Measures Efficiently in the Presence of Tied Scores, ECIR 2008.
Response to Q2
The random baseline in Table 3 is constructed by randomly selecting failure steps and randomly ranking the selected steps. Specifically, for each execution trajectory, we first determine the number of steps in the ground-truth ranking. We then randomly sample the same number of steps from the trajectory and randomly permute their order to construct a predicted ranking. The baseline therefore matches the label density of each log.
Revised manuscript: Appendix G.2 (Random Baseline) and the caption of Table 3.
Dear Reviewer,
As the author–reviewer discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal and additional experimental results.
We hope our responses have addressed your concerns. If you have any remaining questions or concerns, we would be happy to provide further clarification before the discussion period closes.
Thank you again for your time and consideration.
We thank the reviewer for raising these important concerns. We address each concern below.
Response to W1
We agree with the reviewer that the ultimate goal of failure attribution is to guide system refinement and improve MAS reliability. However, as explicitly stated in Section 3.2, our work focuses on offline, post hoc failure attribution from completed execution logs, while downstream MAS refinement is discussed as future work in Section 8.
Conceptually, multi-perspective attribution can provide a more comprehensive basis for system refinement than relying on a single attribution. For example, in Figure 1, one perspective suggests that refinement should address the orchestrator's inappropriate delegation. Another perspective additionally highlights the data analyst's improper introduction of synthetic data. Considering both perspectives could reveal complementary opportunities for improving different agent behaviors that might otherwise be overlooked under a single attribution.
Nevertheless, translating these diverse attributions into effective system modifications requires additional mechanisms to prioritize potential fixes, reconcile conflicting recommendations, and validate their downstream effects. Developing and evaluating such refinement methods is beyond the scope of this benchmark-focused work. We have clarified these potential benefits and challenges in the revised manuscript.
Revised manuscript: Section 3.2 (Scope); Section 8 (Future Work).
Response to W2
About Step-Level Correctness Annotation
As the reviewer correctly noted, we instructed annotators to determine whether each agent's behavior was correct or incorrect at each step. Specifically, our objective was to identify steps where an agent failed to properly fulfill its assigned role or tasks. The rationale behind this annotation scheme is that MAS failure attribution extends beyond identifying a single decisive cause of task failure to diagnosing problematic agent behaviors that should be corrected to improve system reliability. This formulation differs from Who&When, which focuses on identifying the earliest decisive failure step. However, recent studies such as [1,2,3,4] also consider attributing failures to multiple agent behaviors rather than restricting responsibility to a single step.
Revised manuscript: Section 3.2 (new Failure Steps paragraph defining the attribution target and citing related work that labels multiple erroneous steps or agents per trajectory; revised Definition); Section 4 (Annotation Fields, clarifying that each annotator labels each step as correct or erroneous and may flag multiple steps).
[1] Aegis: Automated Error Generation and Attribution for Multi-Agent Systems, ICLR 2026
[2] Tracing Agentic Failure from the Flow of Success, NeurIPS 2026
[3] Agentrx: Diagnosing ai agent failures from execution trajectories, EMNLP 2026 Findings
[4] Agentprocessbench: Diagnosing step-level process quality in tool-using agents, KDD 2026
About Multi-Step Attribution vs. Multi-Perspective Attribution
We emphasize that multi-step attribution and multi-perspective attribution address two distinct dimensions of failure attribution:
Our primary contribution concerns the latter. While multi-step attribution allows multiple failure steps within a trajectory, it does not account for differences among experts in determining which steps constitute failures. To illustrate this distinction, consider Figure 1, where different analysts may interpret the same execution trajectory as follows:
Thus, even when multiple failure steps are permitted, evaluation against a single annotator's labels may penalize alternative attributions considered valid by other experts. Multi-perspective evaluation is designed to address this limitation of single-reference evaluation.
Revised manuscript: Section 3.2 (new Multi-Step vs. Multi-Perspective Attribution paragraph).