acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Evaluation

Abstract

Existing benchmarks and methods for MAS failure attribution largely assume a single deterministic root cause for each execution failure. We challenge this assumption, arguing that MAS failures often admit multiple plausible attributions depending on an observer’s analytical perspective. To address this, we propose multi-perspective failure attribution and introduce MP-BENCH, the first benchmark and evaluation protocol designed for this setting. Through extensive experiments, we show that the perceived limitations of LLMs in failure attribution stem not from insufficient model capability, but from evaluation assumptions that fail to account for the inherently multi-perspective nature of MAS failures.

Primary area:
datasets and benchmarks
Keywords:
multi-agent system, agentic system evaluation, failure attribution

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vUZvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces the idea of multi-perspective failure attribution of multi-agent systems, unlike the prior work that identifies a single step as failure inducing. Based on expert annotation, a benchmark of 289 annotated trajectories is constructed and analyzed, which includes per-step failure labels, rationale and an ideal action. Findings show that there is substantial disagreement between human annotators and based on a manual analysis of sampled conflicts, disagreements mostly reflect multiple valid perspectives. Further, they evaluate five LLMs on this task. For this, multiple samples are generated and ranked based on consensus. NDCG score is used to compare the predicted ranking against human ranking, which are also produced based on consensus. Results show that LLMs perform better than a random baseline on this task.

Critical strengths
  1. The paper introduces a novel and useful multi-perspective failure attribution benchmark.
  2. The benchmark consists of logs annotated by multiple experts with detailed analyses of the annotation quality.
  3. Experimental results show that when sampled multiple times, LLMs perform well at this task.
Critical weaknesses
  1. The downstream utility of multi-perspective failure attribution compared to a single failure is not yet established.
  2. The annotation instructions ask whether each step is correct, not whether it lead to the overall trajectory failure. This is different from failure attribution defined in Who&When benchmark and in Section 3.2.
  3. The reasoning evaluation judge is not rigorously validated by humans.
Questions for authors
  1. How are ties handles in computing NDCG? Since both the GT and the predictions are coarse-grained (=1,2 or 3 for N=3), how you handle ties could substantially impact the NDCG scores.
  2. What is the random baseline reported in Table 3?
3 comments

We thank the reviewer for raising these important concerns. We address each concern below.


Response to W1

We agree with the reviewer that the ultimate goal of failure attribution is to guide system refinement and improve MAS reliability. However, as explicitly stated in Section 3.2, our work focuses on offline, post hoc failure attribution from completed execution logs, while downstream MAS refinement is discussed as future work in Section 8.

Conceptually, multi-perspective attribution can provide a more comprehensive basis for system refinement than relying on a single attribution. For example, in Figure 1, one perspective suggests that refinement should address the orchestrator's inappropriate delegation. Another perspective additionally highlights the data analyst's improper introduction of synthetic data. Considering both perspectives could reveal complementary opportunities for improving different agent behaviors that might otherwise be overlooked under a single attribution.

Nevertheless, translating these diverse attributions into effective system modifications requires additional mechanisms to prioritize potential fixes, reconcile conflicting recommendations, and validate their downstream effects. Developing and evaluating such refinement methods is beyond the scope of this benchmark-focused work. We have clarified these potential benefits and challenges in the revised manuscript.

Revised manuscript: Section 3.2 (Scope); Section 8 (Future Work).


Response to W2

About Step-Level Correctness Annotation

As the reviewer correctly noted, we instructed annotators to determine whether each agent's behavior was correct or incorrect at each step. Specifically, our objective was to identify steps where an agent failed to properly fulfill its assigned role or tasks. The rationale behind this annotation scheme is that MAS failure attribution extends beyond identifying a single decisive cause of task failure to diagnosing problematic agent behaviors that should be corrected to improve system reliability. This formulation differs from Who&When, which focuses on identifying the earliest decisive failure step. However, recent studies such as [1,2,3,4] also consider attributing failures to multiple agent behaviors rather than restricting responsibility to a single step.

Revised manuscript: Section 3.2 (new Failure Steps paragraph defining the attribution target and citing related work that labels multiple erroneous steps or agents per trajectory; revised Definition); Section 4 (Annotation Fields, clarifying that each annotator labels each step as correct or erroneous and may flag multiple steps).

[1] Aegis: Automated Error Generation and Attribution for Multi-Agent Systems, ICLR 2026

[2] Tracing Agentic Failure from the Flow of Success, NeurIPS 2026

[3] Agentrx: Diagnosing ai agent failures from execution trajectories, EMNLP 2026 Findings

[4] Agentprocessbench: Diagnosing step-level process quality in tool-using agents, KDD 2026

About Multi-Step Attribution vs. Multi-Perspective Attribution

We emphasize that multi-step attribution and multi-perspective attribution address two distinct dimensions of failure attribution:

  • Multi-Step Attribution: Identifying multiple steps involving erroneous agent behaviors within a single execution trajectory.
  • Multi-Perspective Attribution: Accounting for the fact that different analysts may reasonably identify different sets of failure steps for the same trajectory.

Our primary contribution concerns the latter. While multi-step attribution allows multiple failure steps within a trajectory, it does not account for differences among experts in determining which steps constitute failures. To illustrate this distinction, consider Figure 1, where different analysts may interpret the same execution trajectory as follows:

  • Perspective 1 (Steps 2 and 4): Step 2 is erroneous because the orchestrator should not have invoked the data analyst at that stage. Step 4 is also erroneous because the orchestrator executed the code despite recognizing that it contained synthetic data.
  • Perspective 2 (Steps 3 and 4): Step 3 is erroneous because the agent improperly introduced synthetic data into the code. Step 4 is also erroneous because the orchestrator executed the code despite being aware of this issue.

Thus, even when multiple failure steps are permitted, evaluation against a single annotator's labels may penalize alternative attributions considered valid by other experts. Multi-perspective evaluation is designed to address this limitation of single-reference evaluation.

Revised manuscript: Section 3.2 (new Multi-Step vs. Multi-Perspective Attribution paragraph).

Response to W3

We thank the reviewer for raising this important concern. To validate our reasoning evaluation judge, we conducted an additional human evaluation study involving 100 (model reasoning, human reasoning) pairs, stratified by dataset, model backbone, and judge score.

An expert annotator evaluated each pair in two stages. First, without access to the judge's output, the annotator independently assigned a coarse rating (low / mid / high). For the blind agreement analysis, we mapped the judge's 1–10 scores into three categories (low: 1–3, mid: 4–6, high: 7–10) and compared them with the annotator's ratings using weighted Cohen's κ and Spearman's ρ. Second, after reviewing the judge's score and rationale, the annotator assessed whether the judge's evaluation was appropriate.

MetricValue (95% CI)
Blind agreement (weighted κ)0.81 [0.70, 0.89]
Blind agreement (Spearman ρ)0.81 [0.71, 0.89], p < 10⁻²⁰
Judge assessment deemed appropriate88%

The results demonstrate strong agreement between the reasoning judge and independent human assessments, with both weighted κ and Spearman ρ reaching 0.81. Furthermore, the annotator considered 88% of the judge's assessments appropriate after reviewing their scores and rationales.

Most disagreements occurred in the mid-score range. The judge also showed a slight tendency toward stricter scoring, with 8% of cases under-scored versus 4% over-scored relative to the human assessment. In particular, disagreements often arose when the judge penalized mismatches in fine-grained details despite agreement on the high-level failure reason and ideal action.

These findings provide empirical support for the reliability of our reasoning evaluation judge, while acknowledging that the validation is based on a sample evaluated by a single expert annotator. We have included the validation protocol and detailed results in the revised manuscript.

Revised manuscript: Section 6 (Attribution Reasoning Evaluation, summary of the validation); Appendix H.6 (protocol and Table 11); Appendix G.2 (pointer).


Response to Q1

We thank the reviewer for highlighting this important issue. In our original evaluation, we resolved ties chronologically, prioritizing earlier steps based on the intuition that errors occurring earlier in an execution trajectory may have a greater influence on subsequent agent behaviors and therefore warrant higher priority for system refinement.

Motivated by the reviewer's concern, we further investigated an alternative evaluation approach that explicitly accounts for ties, which frequently arise in our vote-based rankings. Specifically, we explored a tie-aware nDCG formulation following [5]. For ground truth, steps receiving equal votes are assigned the average relevance of their occupied ranks. For predictions, tied steps are evaluated using the expected DCG over their possible orderings, avoiding an arbitrary preference among equally ranked steps.

ModelHand-Crafted nDCG@5Hand-Crafted nDCG@fullAutomatic nDCG@5Automatic nDCG@full
GPT-4.10.49080.53370.67850.6821
GPT-5.10.48130.57460.80620.8116
Sonnet 4.50.50730.58110.80290.8130
Qwen3-8B0.36480.40400.66900.6857
GPT-OSS-120B0.48610.54750.69780.7112

The results remain broadly consistent with our original findings, with Sonnet 4.5 demonstrating consistently strong performance across all evaluation settings, while Qwen3-8B generally exhibits the weakest performance. This additional analysis demonstrates that our main findings remain consistent under an evaluation approach that does not rely on chronological tie-breaking. We retain the original results for comparability and have included the tie-aware evaluation as an additional robustness analysis in the revised manuscript, along with a clarification of the original tie-breaking procedure.

Revised manuscript: Section 6 (Multi-Perspective Failure Attribution Evaluation, tie-breaking procedure and its rationale); Appendix G.2 (Tie-breaking); Appendix H.4 (Table 9).

[5] McSherry and Najork, Computing Information Retrieval Performance Measures Efficiently in the Presence of Tied Scores, ECIR 2008.

Response to Q2

The random baseline in Table 3 is constructed by randomly selecting failure steps and randomly ranking the selected steps. Specifically, for each execution trajectory, we first determine the number of steps in the ground-truth ranking. We then randomly sample the same number of steps from the trajectory and randomly permute their order to construct a predicted ranking. The baseline therefore matches the label density of each log.

Revised manuscript: Appendix G.2 (Random Baseline) and the caption of Table 3.

Dear Reviewer,

As the author–reviewer discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal and additional experimental results.

We hope our responses have addressed your concerns. If you have any remaining questions or concerns, we would be happy to provide further clarification before the discussion period closes.

Thank you again for your time and consideration.

Reviewer_f5RdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes changes in assumption made in prior MAS failure attribution benchmarks that each failed execution has a single deterministic failure-inducing step, arguing that a failure often admits multiple plausible attributions depending on the analyst's perspective. The authors introduce MP-Bench consisting of 289 logs from MAS systems annotated by three experts with a failure label, failure reason and ideal action per step.

The paper's evaluation ranks steps by annotator consensus and compares this ranking with aggregated LLM predictions, alongside an LLM-based reasoning assessment. Experiments across five models examine repeated sampling and combinations of models. The paper also analyzes annotator disagreement and develops a filtering procedure to distinguish plausible alternative attributions from annotation noise.

Critical strengths
  • I appreciate the finding that disagreement can reflect different points along the same failure chain, including planning, execution, and verification. Collecting separate reasons and ideal actions from three annotators makes these differences inspectable. The human audit also acknowledges that disagreement includes annotation noise, which is useful for interpreting the benchmark. The paper also addresses identifying such annotation noise, using a LLM-judge based heuristic, that achieved high agreement with independent human review.
  • The paper provides useful evidence that combining models can be more effective than repeatedly sampling one model, for multi-perspective MAS evaluation.
Critical weaknesses
  • The main conclusion that previously observed limitations of LLMs come from deterministic evaluation rather than model capability is not established by the experiments. Table 3 shows above-random performance on MP-Bench, however it does not compare single-reference and multi-perspective evaluation on the same logs and model outputs. From my understanding, the effects of changing the benchmark, metric, and inference setup remain mixed together.
  • From my reading, the benchmark labels erroneous steps rather than failure-inducing steps. Figure 7 asks annotators whether "each agent action was correct (0) or incorrect (1)", and Figure 9 asks the model to "Identify all agents who made errors". Several errors along one chain are then co-occurring errors recalled differently, not competing perspectives. This makes the task easier than decisive-error attribution.
Questions for authors
  1. Could the authors provide a matched comparison scoring same predictions on the same executions against a single reference and multiple validated references? How much of the claimed improvement remains when the model and sampling budget are fixed?
  2. Have the authors considered counterfactual replay from flagged steps with the identified ideal action, to separate failure-inducing steps from non-decisive errors?
  3. Can the authors report a human reference, e.g., leave-one-annotator-out nDCG against the other two annotators?
  4. Was the 95.9% filter agreement measured on annotations held out from rule development?
6 comments

We thank the reviewer for raising these important concerns. We address each concern below.

About Step-Level Correctness Annotation

As the reviewer correctly noted, we instructed annotators to determine whether each agent's behavior was correct or incorrect at each step. Specifically, our objective was to identify steps where an agent failed to properly fulfill its assigned role or tasks. The rationale behind this annotation scheme is that MAS failure attribution extends beyond identifying a single decisive cause of task failure to diagnosing problematic agent behaviors that should be corrected to improve system reliability.

This formulation differs from Who&When, which focuses on identifying the earliest decisive failure step. However, recent studies such as [1,2,3,4] also consider attributing failures to multiple agent behaviors rather than restricting responsibility to a single step.

Revised manuscript: Section 3.2 (new Failure Steps paragraph defining the attribution target and citing related work that labels multiple erroneous steps or agents per trajectory; revised Definition); Section 4 (Annotation Fields, clarifying that each annotator labels each step as correct or erroneous and may flag multiple steps).

[1] Aegis: Automated Error Generation and Attribution for Multi-Agent Systems, ICLR 2026

[2] Tracing Agentic Failure from the Flow of Success, NeurIPS 2026

[3] Agentrx: Diagnosing ai agent failures from execution trajectories, EMNLP 2026 Findings

[4] Agentprocessbench: Diagnosing step-level process quality in tool-using agents, KDD 2026


Multi-Step Attribution vs. Multi-Perspective Attribution

We emphasize that multi-step attribution and multi-perspective attribution address two distinct dimensions of failure attribution:

  • Multi-Step Attribution: Identifying multiple steps involving erroneous agent behaviors within a single execution trajectory.
  • Multi-Perspective Attribution: Accounting for the fact that different analysts may reasonably identify different sets of failure steps for the same trajectory.

Our primary contribution concerns the latter. While multi-step attribution allows multiple failure steps within a trajectory, it does not account for differences among experts in determining which steps constitute failures. To illustrate this distinction, consider Figure 1, where different analysts may interpret the same execution trajectory as follows:

  • Perspective 1 (Steps 2 and 4): Step 2 is erroneous because the orchestrator should not have invoked the data analyst at that stage. Step 4 is also erroneous because the orchestrator executed the code despite recognizing that it contained synthetic data.
  • Perspective 2 (Steps 3 and 4): Step 3 is erroneous because the agent improperly introduced synthetic data into the code. Step 4 is also erroneous because the orchestrator executed the code despite being aware of this issue.

Thus, even when multiple failure steps are permitted, evaluation against a single annotator's labels may penalize alternative attributions considered valid by other experts. Multi-perspective evaluation is designed to address this limitation of single-reference evaluation.

Revised manuscript: Section 3.2 (new Multi-Step vs. Multi-Perspective Attribution paragraph).

Additional Experiments: Disentangling Multi-Step and Multi-Perspective Attribution

To address the reviewer's concerns, we conducted two additional analyses.

1. Multi-Step Attribution without Multi-Perspective Evaluation

We first evaluated LLM performance using a single annotator as the reference, thereby removing multi-perspective evaluation. Since this setting does not yield a consensus-based relevance ranking, we used set-based precision, recall, and F1 instead of nDCG, with a single LLM sampling.

ModelHand-Crafted Micro F1 (LLM / Random)Automatic Micro F1 (LLM / Random)
GPT-4.10.353 / 0.1420.487 / 0.250
GPT-5.10.343 / 0.1620.555 / 0.276
Sonnet 4.50.385 / 0.1540.570 / 0.282
Qwen3-8B0.227 / 0.1180.440 / 0.233
GPT-OSS-120B0.334 / 0.1500.489 / 0.236

As shown above, LLMs consistently outperform the random baseline even without multi-perspective evaluation. Across five models, their micro-F1 scores are approximately 2.25× and 1.99× those of the random baseline on hand-crafted and automatic MAS, respectively.

Revised manuscript: Section 7.3 (new RQ2) and Table 4.

We agree with the reviewer that the performance gap relative to prior work cannot be attributed solely to multi-perspective evaluation, as differences in task formulation, including the allowance of multiple failure steps, may also contribute. We have revised our original claim accordingly (see below).

2. Does Single-Reference Evaluation Overlook Alternative Attributions?

We further examined how frequently single-reference evaluation overlooks failure attributions recognized by other human annotators. Specifically, we treated each annotator in turn as the sole ground-truth reference. We identified LLM-predicted steps that were not labeled as failures by the reference annotator (i.e., false positives under single-reference evaluation) and measured the proportion recognized as failures by at least one of the other two annotators.

ModelOther-Annotator Support among FPs
GPT-4.136.8%
GPT-5.134.6%
Sonnet 4.539.0%
Qwen3-8B21.2%
GPT-OSS-120B33.1%

Across the five models, 21.2–39.0% of LLM predictions considered false positives under single-reference evaluation were recognized as failures by other annotators. Notably, approximately 33–37% of these alternative attributions were supported by both remaining annotators, despite being rejected by the selected single reference.

These findings demonstrate that a substantial proportion of LLM predictions penalized under single-reference evaluation align with alternative human judgments. Thus, even when multiple failure steps are allowed, multi-perspective evaluation captures alternative failure attributions identified by other human annotators that would otherwise be overlooked under single-reference evaluation.

Taken together, our additional analyses suggest that the performance differences cannot be explained solely by relaxing the task to multi-step error detection. While allowing multiple failure steps relaxes the evaluation setting, single-reference evaluation still overlooks a substantial fraction of LLM predictions supported by other annotators. Accounting for multiple perspectives therefore addresses an additional source of evaluation mismatch beyond multi-step attribution alone.

Revised manuscript: Section 7.3 (RQ2, Single-reference evaluation penalizes attributions that other experts consider valid) and Table 4 (FP Support row). Throughout the paper, we have revised our claims about LLM performance as follows: “deterministic, single-reference evaluation overlooks a substantial fraction of LLM predictions that other experts consider valid, and therefore underestimates LLMs' failure attribution capability” (Abstract; Section 1, including the contribution list; Section 7.2, summary paragraph; Section 8).

Have you considered counterfactual replay?

As clarified in our responses above, our benchmark aims to identify incorrect agent behaviors that warrant system refinement, rather than exclusively identifying causally decisive steps responsible for the final task outcome. Therefore, distinguishing failure-inducing steps from non-decisive errors through counterfactual replay is not necessary for our task formulation, as both types of errors are relevant to diagnosing and improving MAS reliability. Even errors that do not determine the final task outcome may introduce inefficiencies or reveal problematic agent behaviors that should be corrected. While counterfactual replay could provide additional insights into the causal contribution of individual errors, such a distinction would not change which erroneous behaviors our benchmark aims to identify.


Human Reference Report

We thank the reviewer for this valuable suggestion. We conducted an additional leave-one-annotator-out (LOO) nDCG@5 evaluation, where each human annotator is evaluated against a reference ranking constructed from the other two annotators. We also evaluated LLM predictions against the same two-annotator references for comparison.

Hand-Crafted MASAutomatic MAS
Human (LOO)0.4970.744
Best Model0.456 (GPT-OSS-120B)0.819 (Sonnet 4.5)

The results reveal different patterns across the two settings. The human reference exceeds the reported model score on hand-crafted MAS, whereas the best-performing LLM exceeds the human reference on automatic MAS.

We emphasize that these results serve only as an approximate human reference. Unlike LLM predictions, which aggregate multiple sampling runs to construct a ranking, each human annotator provides only binary failure labels. Consequently, all steps identified as failures by a single annotator have equal relevance, and their annotations do not provide an explicit ranking among those steps. To compute nDCG, we therefore broke ties by step order, prioritizing earlier steps when constructing the human ranking. However, this ordering reflects an evaluation heuristic rather than the annotator's actual assessment of relative failure importance. Furthermore, the two-annotator reference rankings are coarser than the three-annotator consensus used in our main evaluation.

A more rigorous human reference would require collecting additional independent human annotations and obtaining explicit rankings of failure steps, allowing human performance to be evaluated against a separate set of annotators without relying on heuristic tie-breaking. We have included the LOO analysis and these limitations in the revised manuscript.

Revised manuscript: Appendix H.5 (Table 10), referenced from Section 7.2.


About Rule-Based Filtering Agreement

Yes. The 95.9% filtering agreement was measured on annotations held out from rule development. The filtering rules were developed based on recurring noise patterns observed during preliminary data analysis, while the agreement was evaluated on a separate set of annotations not used for rule development.

Reviewer_RLCYOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper challenges the standard assumption in multi-agent failure attribution benchmarks that every failed execution trace has a single deterministic root cause. The authors argue that different human analysts naturally blame different stages of a failure chain. They build MP-BENCH, a dataset of 289 failed traces annotated by three experts who label all failure steps along with failure reasons and ideal actions. By sampling LLMs multiple times and aggregating their predictions into consensus rankings evaluated with nDCG and an LLM judge, the authors claim that current LLMs are actually quite good at failure attribution once evaluated from multiple perspectives.

Critical strengths
  1. The proposed challenge is realistic and well motivated. Forcing a single ground-truth error step on a long multi-agent trajectory can feel arbitrary because errors cascade across orchestrators and workers. Framing attribution as a ranking problem where high-consensus steps represent perspective-invariant bugs makes sense.
  2. Valuable human disagreement dataset and analysis. Getting three qualified experts to annotate 289 traces with step labels, failure reasons, and ideal actions takes real effort and I appreciate it.
Critical weaknesses
  1. Conflation of multi-perspective attribution with multi-step error detection, which undermines the main claim about LLM capability. The central claim of the paper is that LLMs only looked bad on prior benchmarks because of single-label deterministic assumptions. However, both the human annotation guide and the LLM prompt instruct the annotator to flag all steps where an agent made a mistake, rather than asking each annotator for their perspective on the root cause. As a result, a single annotator flags multiple steps per trace, and a median of 50 percent of all steps in the automatic MAS logs end up labeled as ground-truth failures. Getting an nDCG score of 0.80 when half the trajectory is a positive label is much easier than pinpointing a single root cause with top-1 accuracy. The jump in numbers comes largely from relaxing the task into dense multi-step error detection and using a softer ranking metric, not just from accounting for multiple perspectives.
  2. Model bias in ground-truth filtering and lack of downstream repair validation. Nearly half of the raw failure labels come from a single annotator, and the authors use Claude Sonnet 4.5 to filter out 43 percent of those labels as noise. Later, Claude Sonnet 4.5 is evaluated on this filtered dataset and comes out as the top-performing model, which can introduce circularity in favor of Claude models.
Questions for authors
  1. Clarification on task definition: Since each individual annotator and each individual LLM run is instructed to output all erroneous steps in a trace, how do you separate genuine perspective disagreement on why a task failed from simply listing every cascading error along the trajectory?
  2. I wanted to learn more about the potential bias using a single Claude model to filter the data. How much does Sonnet 4.5's ranking in Table 3 change if you evaluate on the unfiltered human annotations or filter using an independent model family?
3 comments

We thank the reviewer for raising these important concerns. We address each concern below.

About Step-Level Correctness Annotation

As the reviewer correctly noted, we instructed annotators to determine whether each agent's behavior was correct or incorrect at each step. Specifically, our objective was to identify steps where an agent failed to properly fulfill its assigned role or tasks. The rationale behind this annotation scheme is that MAS failure attribution extends beyond identifying a single decisive cause of task failure to diagnosing problematic agent behaviors that should be corrected to improve system reliability.

This formulation differs from Who&When, which focuses on identifying the earliest decisive failure step. However, recent studies such as [1,2,3,4] also consider attributing failures to multiple agent behaviors rather than restricting responsibility to a single step.

Revised manuscript: Section 3.2 (new Failure Steps paragraph defining the attribution target and citing related work that labels multiple erroneous steps or agents per trajectory; revised Definition); Section 4 (Annotation Fields, clarifying that each annotator labels each step as correct or erroneous and may flag multiple steps).

[1] Aegis: Automated Error Generation and Attribution for Multi-Agent Systems, ICLR 2026

[2] Tracing Agentic Failure from the Flow of Success, NeurIPS 2026

[3] Agentrx: Diagnosing ai agent failures from execution trajectories, EMNLP 2026 Findings

[4] Agentprocessbench: Diagnosing step-level process quality in tool-using agents, KDD 2026


Multi-Step Attribution vs. Multi-Perspective Attribution

We emphasize that multi-step attribution and multi-perspective attribution address two distinct dimensions of failure attribution:

  • Multi-Step Attribution: Identifying multiple steps involving erroneous agent behaviors within a single execution trajectory.
  • Multi-Perspective Attribution: Accounting for the fact that different analysts may reasonably identify different sets of failure steps for the same trajectory.

Our primary contribution concerns the latter. While multi-step attribution allows multiple failure steps within a trajectory, it does not account for differences among experts in determining which steps constitute failures. To illustrate this distinction, consider Figure 1, where different analysts may interpret the same execution trajectory as follows:

  • Perspective 1 (Steps 2 and 4): Step 2 is erroneous because the orchestrator should not have invoked the data analyst at that stage. Step 4 is also erroneous because the orchestrator executed the code despite recognizing that it contained synthetic data.
  • Perspective 2 (Steps 3 and 4): Step 3 is erroneous because the agent improperly introduced synthetic data into the code. Step 4 is also erroneous because the orchestrator executed the code despite being aware of this issue.

Thus, even when multiple failure steps are permitted, evaluation against a single annotator's labels may penalize alternative attributions considered valid by other experts. Multi-perspective evaluation is designed to address this limitation of single-reference evaluation.

Revised manuscript: Section 3.2 (new Multi-Step vs. Multi-Perspective Attribution paragraph).

Additional Experiments: Disentangling Multi-Step and Multi-Perspective Attribution

To address the reviewer's concerns, we conducted two additional analyses.

1. Multi-Step Attribution without Multi-Perspective Evaluation

We first evaluated LLM performance using a single annotator as the reference, thereby removing multi-perspective evaluation. Since this setting does not yield a consensus-based relevance ranking, we used set-based precision, recall, and F1 instead of nDCG, with a single LLM sampling.

ModelHand-Crafted Micro F1 (LLM / Random)Automatic Micro F1 (LLM / Random)
GPT-4.10.353 / 0.1420.487 / 0.250
GPT-5.10.343 / 0.1620.555 / 0.276
Sonnet 4.50.385 / 0.1540.570 / 0.282
Qwen3-8B0.227 / 0.1180.440 / 0.233
GPT-OSS-120B0.334 / 0.1500.489 / 0.236

As shown above, LLMs consistently outperform the random baseline even without multi-perspective evaluation. Across five models, their micro-F1 scores are approximately 2.25× and 1.99× those of the random baseline on hand-crafted and automatic MAS, respectively.

Revised manuscript: Section 7.3 (new RQ2) and Table 4.

We agree with the reviewer that the performance gap relative to prior work cannot be attributed solely to multi-perspective evaluation, as differences in task formulation, including the allowance of multiple failure steps, may also contribute. We have revised our original claim accordingly (see below).

2. Does Single-Reference Evaluation Overlook Alternative Attributions?

We further examined how frequently single-reference evaluation overlooks failure attributions recognized by other human annotators. Specifically, we treated each annotator in turn as the sole ground-truth reference. We identified LLM-predicted steps that were not labeled as failures by the reference annotator (i.e., false positives under single-reference evaluation) and measured the proportion recognized as failures by at least one of the other two annotators.

ModelOther-Annotator Support among FPs
GPT-4.136.8%
GPT-5.134.6%
Sonnet 4.539.0%
Qwen3-8B21.2%
GPT-OSS-120B33.1%

Across the five models, 21.2–39.0% of LLM predictions considered false positives under single-reference evaluation were recognized as failures by other annotators. Notably, approximately 33–37% of these alternative attributions were supported by both remaining annotators, despite being rejected by the selected single reference.

These findings demonstrate that a substantial proportion of LLM predictions penalized under single-reference evaluation align with alternative human judgments. Thus, even when multiple failure steps are allowed, multi-perspective evaluation captures alternative failure attributions identified by other human annotators that would otherwise be overlooked under single-reference evaluation.

Taken together, our additional analyses suggest that the performance differences cannot be explained solely by relaxing the task to multi-step error detection. While allowing multiple failure steps relaxes the evaluation setting, single-reference evaluation still overlooks a substantial fraction of LLM predictions supported by other annotators. Accounting for multiple perspectives therefore addresses an additional source of evaluation mismatch beyond multi-step attribution alone.

Revised manuscript: Section 7.3 (RQ2, Single-reference evaluation penalizes attributions that other experts consider valid) and Table 4 (FP Support row). Throughout the paper, we have revised our claims about LLM performance as follows: “deterministic, single-reference evaluation overlooks a substantial fraction of LLM predictions that other experts consider valid, and therefore underestimates LLMs' failure attribution capability” (Abstract; Section 1, including the contribution list; Section 7.2, summary paragraph; Section 8).

About Model Bias

We thank the reviewer for raising this important concern. To investigate potential bias introduced by using Claude Sonnet 4.5 for noise filtering, we conducted an additional robustness analysis using unfiltered human annotations and annotations independently filtered by GPT-4.1 and GPT-5.1. We report nDCG@5 aggregated across hand-crafted and automatic MAS.

ModelUnfilteredGPT-4.1 FilterGPT-5.1 FilterSonnet Filter
GPT-4.10.55280.55690.55290.5618
GPT-5.10.57330.57120.56710.5734
Sonnet 4.50.61560.61010.60000.6100
Qwen3-8B0.46440.46580.46740.4707
GPT-OSS-120B0.55310.55990.56040.5627

Notably, Sonnet 4.5 consistently ranks first across all four filtering conditions, including evaluation without any LLM-based filtering. Furthermore, the relative ranking of all five models remains unchanged across filtering conditions. These results suggest that Sonnet 4.5's superior performance is not primarily driven by the use of Claude-based noise filtering, as its ranking remains robust when filtering is removed or performed using independent model families.

Revised manuscript: Section 4 (Annotation Quality Control, final sentence) and Appendix H.3 (Table 8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.