acceptodds
Under review as a conference paper at ICLR 2027

GAMA: Graph-Augmented Multi-Source Attribution for LLM-Based Multi-Agent Systems

Abstract

Large language model (LLM)-based multi-agent systems solve complex tasks through collaboration, but errors can propagate across agents and jointly cause task failure. Existing failure-attribution methods typically identify a single responsible agent or error step, overlooking multiple error sources within one execution. We study multi-source failure attribution: identifying all agent-step pairs that introduce errors contributing to task failure. This is challenging because multiple errors and their effects can become intertwined, requiring both structural reasoning to trace error propagation and semantic reasoning to identify error sources. We propose GAMA, a Graph-Augmented Multi-Source Attribution framework that combines these two forms of reasoning. GAMA represents each execution as a temporal graph with dependency and same-agent relations. A graph encoder integrates upstream dependencies and downstream consequences to produce source-error scores, which guide an LLM to predict the final attribution set. We also introduce Whos & Whens, a benchmark with step-level multi-source annotations constructed through context-dependent error injection and failure validation. Experiments show that GAMA improves failure attribution across single- and multi-source benchmarks. On Whos & Whens, GAMA improves exact match by 7.58% over the best baseline across three LLM backbones. On CORRECT-Error, it improves exact-step accuracy over CORRECT by 18.23% on average across seven tasks. Code is available at https://anonymous.4open.science/r/GAMA-54FB.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multi-Agent Systems, Failure Attribution, Graph Neural Networks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_r1JROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies multi-source failure attribution in multi-agent systems, distinguishing error origins from downstream propagated errors. It introduces GAMA, which combines execution-graph scoring with LLM reasoning, and Whos & Whens, a benchmark containing 1,913 failed trajectories. Experiments demonstrate improved multi-source attribution across three LLM backbones and strong localization results on a single-source benchmark.

Critical strengths

The paper addresses the important problem of identifying multiple error sources in multi-agent failures and contributes a benchmark with execution-based validation and human assessment. GAMA combines structural dependencies with LLM reasoning to distinguish error origins from propagated errors. Evaluation across three LLM backbones and a separate single-source benchmark supports its effectiveness, while ablations demonstrate the contributions of the graph relations, bidirectional updates, and terminal-context scoring.

Critical weaknesses
  1. The paper shows that dependency edges improve attribution, but does not directly evaluate their extraction quality or sensitivity to incorrect and missing edges. A controlled edge-addition/deletion experiments would help assess the reliability of this central component.
Questions for authors
  1. In execution graph construction, the four dependency categories are plausible, but their definitions and coverage are insufficiently explained. For example, tool outputs may also constitute data dependencies, while shared-state or resource dependencies may be implicit in the logs. Since these categories appear to be merged into a single dependency relation, please clarify their role in extraction and how ambiguous cases are handled. An analysis of extracted edge quality would help establish whether the graph reliably captures the dependencies needed for failure attribution.

  2. MAS structure shapes agent collaboration and interaction, influencing how errors propagate and affect task outcomes. How does GAMA perform across different organizational structures, such as centralized, decentralized, and hierarchical systems? Are there qualitative differences in its ability to distinguish source errors from propagated errors across these structures?

  3. In the second panel of Figure 2, the roles of the node and edge features are unclear because their connections to the computation are not explicitly shown. Please indicate how node features initialize the node representations and how edge features enter message construction before the GRU updates. More generally, please carefully review the figures to ensure that they accurately convey the methodology and help readers understand the corresponding text.

3 comments
AuthorsOpenReview ↗

Response to Reviewer r1JR (1/2)

We sincerely thank you for your detailed comments.

Weakness 1: Dependency-edge quality and sensitivity to missing or incorrect edges

The graph scorer is more sensitive to missing dependency edges, while adding noisy edges has a smaller effect. The graph scorer uses a GNN to combine information along execution relations. Removing dependency edges reduces the structural information available for attribution. In comparison, adding noisy edges preserves the original dependencies, so the model can still use these relations to gather useful information.

We conduct controlled dependency-edge perturbation experiments by deleting existing dependency edges or adding extra edges to introduce noise. All added edges point from earlier to later steps. We evaluate the Graph Scorer without the final LLM. All metrics are percentages.

Graph conditionEMMacro-F1Micro-F1
Original Graph Scorer18.0353.4549.08
Delete 10% dependency edges17.8353.4349.30
Delete 30% dependency edges16.7852.7449.11
Delete 50% dependency edges15.6851.7048.54
Add 10% noisy edges17.9853.0548.50
Add 30% noisy edges17.8353.0648.58
Add 50% noisy edges17.6252.8848.95

EM and Macro-F1 decrease as more dependency edges are removed, highlighting the importance of retaining dependency information for source-error attribution. With different levels of added edge noise, all three metrics remain close to their original values, indicating that the graph scorer maintains relatively stable attribution performance despite extra noisy connections.

Question 1: Dependency categories, overlapping cases, and coverage of implicit dependencies

We list four dependency categories in the prompt to guide the LLM to identify dependencies through different types of information, covering as many dependencies visible in the log as possible and reducing omissions. These labels guide extraction but all map to a single dependency relation during graph construction and scoring. The graph scorer therefore uses only two relation types: dependency and same-agent.

LabelMeaning
DataA later step uses facts, numerical values, or other data provided by an earlier step.
ToolA later step uses the result returned by an earlier tool call.
ReasoningA later step relies on a conclusion or judgment from earlier reasoning.
ControlThe execution or action choice of a later step depends on an earlier plan, task assignment, or control decision.

When categories overlap, such as when a tool result also serves as data input, the extractor selects one main label without adding duplicate dependency edges for the same step pair. The label choice does not change graph encoding as long as the two endpoints remain the same.

Shared-state or resource dependencies are included when the log provides evidence of a dependency between the corresponding steps. Implicit dependencies that cannot be identified from the log remain outside the current coverage.

AuthorsOpenReview ↗

Response to Reviewer r1JR (2/2)

Question 2: Performance across organizational structures

GAMA outperforms both baselines across the evaluated centralized hierarchical, decentralized, and hybrid systems, although attribution accuracy varies across systems.

We group the systems by their main control flow and communication mechanism. MacNet and Debate are decentralized systems. AgentVerse and DyLAN are hybrid systems that use dynamic teams and dynamic routing. SmolAgents and Magentic-One are centralized hierarchical systems. Dynamic teams change which agents take part in a task, while dynamic routing changes which agents interact and how information passes between them during execution.

The following results are EM (%).

MethodMacNetDebateAgentVerseDyLANSmolAgentsMagentic-OneOverall
CORRECT21.6120.7526.167.7723.7727.9622.37
EDGE22.6020.2229.654.8523.7727.9623.42
GAMA38.9827.4934.8810.6830.3337.6333.51

Accuracy differences across systems. All three methods have low attribution accuracy on DyLAN. Its dynamic interactions can make error propagation more complex, making it harder to trace error origins and distinguish source errors from propagated errors.

Question 3: Clarifying node initialization, message construction, and GRU updates

We have revised the framework figure to explicitly connect the features and computation modules, clarifying the following steps:

  • Node initialization: Node features xux_u and xvx_v pass through the node-projection MLP to obtain the initial representations huh_u and hvh_v.
  • Message construction: The current node representations hu,hvh_u,h_v, edge features xex_e, and elapsed-time feature Δtℓ\Delta t_\ell enter the message MLP together to produce mℓm_\ell.
  • Updates to both endpoints: The message mℓm_\ell and each endpoint's current state enter the corresponding GRU to update the source and target nodes.

Thank you again for your careful reading and helpful suggestions.

Thanks for your response. My concern has been addressed. Please revise the manuscript accordingly.

Reviewer_L9dsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates multi-source failure attribution in LLM-based multi-agent systems, aiming to identify all agent-step pairs that independently introduce root-cause errors while filtering out downstream propagated errors. To address the limitations of conventional single-source assumptions and unstructured log inspection, the authors propose GAMA, a framework that couples structural graph tracing with LLM semantic reasoning. GAMA models multi-agent execution trajectories as temporal dependency graphs and employs a lightweight temporal graph network with bidirectional memory updates to produce soft prior scores for candidate steps, which subsequently guide an LLM to predict the final source-error set via counterfactual verification. Additionally, authors construct Whos & Whens, a benchmark containing ~2k failure trajectories across 6 domains and 6 multi-agent frameworks, annotated through step-level error injections. Experimental results show that GAMA consistently outperforms baselines across diverse LLM backbones, achieving improvement on both multi-source Whos & Whens and single-source CORRECT-Error benchmark.

Critical strengths
  1. The motivation and problem formulation is clear. Formally distinguishing source errors from propagated errors via counterfactual necessity, addresses a fundamental limitation in prior single-agent/single-step attribution works. Targeting complete multi-source set recovery is critical for real-world multi-agent systems.
  2. The architecture balances structural inductive bias with rich semantic verification. By using a lightweight TGN to score steps and feeding these scores as soft priors into an LLM, GAMA avoids both the context degradation of flat trajectory prompting and the semantic rigidity of pure graph modeling.
  3. Proposed Whos & Whens benchmark is helpful to the community, offering a resource for multi-agent system debugging. Its synthetic construction pipeline is well-controlled, covers diverse task domains and agent architectures, and has been validated by human judges for label reliability and naturalness.
  4. Ablation studies are informative with systematically proving the utility of each proposed component.
Critical weaknesses
  1. Lack of source-level and domain-level breakdown: The ~2k trajectories in Whos & Whens span different task domains (e.g., math vs. search) and varying numbers of concurrent error sources (k=1k=1 to k≥4k \ge 4). However, the paper only reports aggregate macro performance. It remains unclear how GAMA scales as the concurrent error count kk increases, or whether the effectiveness of graph priors varies substantially between different task domains.
  2. Lack of OOD validation: The evaluation strictly relies on grouped 5-fold cross-validation within the synthetic Whos & Whens benchmark (and CORRECT-Error). Because training and testing folds share the same underlying MAS architectures, task types, and perturbation manipulators, this represents a purely in-domain evaluation. The paper does not evaluate whether the learned graph scorer transfers to unseen MAS frameworks, unseen task distributions, or naturally occurring execution logs.
  3. Disconnect from Downstream Closed-Loop System Repair: The central practical motivation of the paper is that identifying all source errors allows developers or systems to repair multiple failure sources together, avoiding the overhead of repeated trial-and-error diagnosis and re-execution. However, the paper evaluates only attribution metrics (EM, F1, Acc@k) and provides no closed-loop repair experiments to quantify whether accurate multi-source attribution actually leads to higher recovery rates or lower costs in practice.
Questions for authors
  1. Can the authors provide ood or natural evaluation results (e.g., zero-shot transfer across unseen task with leave-one-out (leave one domain/mas) method, transfer to another error-attribution benchmark) to verify the transferability of the learned priors?

  2. Can the authors provide granular breakdown by error counts (kk) and task domains, same as weakness q1?

  3. In Table 1, though GAMA achieves much better performance than other baselines, it still gets error on more than half predictions. Could the authors provide a qualitative error analysis analyzing these failure causes? For false positives, is the bottleneck primarily distinguishing propagated errors from source errors? For false negatives, are errors missed due to low TGN prior scores or LLM verification mistakes? Does the TGN scorer ever produce confident false priors that mislead the LLM's final decision? I believe more failure case analysis will help the community better understand the key challenges and inspire future improvements on this task.

2 comments
AuthorsOpenReview ↗

Response to Reviewer L9ds (1/2)

We sincerely thank you for your comments.

Weakness 1 & Question 2: Breakdown by source-error count and task domain

GAMA outperforms the baselines in every source-error-count group and across all six task domains.

1. Different source-error counts.

We report EM (%) grouped by the ground-truth number of source errors, k.

Methodk = 1k = 2k = 3k >= 4
All-at-Once22.6913.616.3110.50
CORRECT28.4814.1416.5011.42
EDGE29.2018.0613.5912.79
GAMA41.5924.8722.8217.81

The improvement is larger in single-source cases, while all methods have lower accuracy in multi-source cases. The true source count is unknown at inference time, so the model must determine both how many source errors exist and which steps introduce them. Recovering the complete set becomes harder as the source count increases.

2. Different task domains.

The following results are EM (%). "Leave-one" denotes training the Graph Scorer without the target domain and evaluating on that held-out domain.

MethodMATHGSM8KMMLU-ProSciBenchHumanEvalGAIAOverall
CORRECT23.3826.5320.8320.0016.3127.9122.37
EDGE23.6327.7023.2122.3915.9627.9123.42
GAMA32.8439.9432.4430.7531.2133.4933.51
GAMA (leave-one)32.0939.0731.8431.3430.1433.0232.98

Although the gains vary across domains, GAMA consistently outperforms both baselines, showing that its effectiveness is not limited to a particular task domain.

Weakness 2 & Question 1: Generalization to unseen domains, unseen systems, and real failure logs

GAMA generalizes across task domains, multi-agent systems, and external real failure logs. Different tasks and systems can share similar error-propagation patterns. The graph scorer learns these patterns from inter-step dependencies, providing reusable structural priors. Our synthetic data come from actual multi-agent executions through context-dependent error injection and execution-based failure validation. These trajectories capture how errors propagate through agent interactions, allowing the scorer to learn patterns that may also arise in natural failures.

1. Generalization to unseen task domains.

Under leave-one-domain-out evaluation, overall EM drops from 33.51% to 32.98%, a decrease of 0.53%. Performance remains close to the standard setting across individual domains, supporting transfer to task domains unseen during training.

2. Generalization to unseen multi-agent systems.

We exclude the target system from Graph Scorer training and evaluate on that held-out system. The following results are EM (%); "leave-one" denotes leave-one-system-out evaluation.

MethodMacNetAgentVerseDebateSmolAgentsDyLANMagentic-OneOverall
CORRECT21.6126.1620.7523.777.7727.9622.37
EDGE22.6029.6520.2223.774.8527.9623.42
GAMA38.9834.8827.4930.3310.6837.6333.51
GAMA (leave-one)36.4432.5525.6128.6810.6836.5631.42

Overall EM drops by 2.09%, but GAMA still outperforms both baselines on every system. The larger drop suggests that cross-system transfer is more challenging than cross-domain transfer.

3. Generalization to real failure logs.

We evaluate transfer to Who&When [1], an external benchmark containing real multi-agent failure cases. Since it includes GAIA tasks and Magentic-One, we separately exclude their trajectories from Graph Scorer training to assess dependence on training data from the same domain or system.

The following results are accuracy (%). HC denotes hand-crafted systems, and AG denotes algorithm-generated systems.

MethodHC: Agent-levelHC: Step-levelAG: Agent-levelAG: Step-level
All-at-Once48.2810.3459.5230.95
CORRECT53.4518.9763.4938.89
GAMA (all training data)65.5229.3172.2254.76
GAMA (w/o GAIA)63.7929.3170.6353.97
GAMA (w/o Magentic-One)63.7927.59----

GAMA outperforms the baselines at both agent and step levels. It retains this advantage on both subsets after excluding GAIA, and on the hand-crafted subset after excluding Magentic-One, supporting transfer to external real failure logs.

[1] Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems.

AuthorsOpenReview ↗

Response to Reviewer L9ds (2/2)

Weakness 3: Practical impact on closed-loop repair

Joint multi-source repair achieves a higher recovery rate with fewer repair attempts, but a gap remains between accurate attribution and successful repair. GAMA achieves an attribution EM of 37.63% on the evaluated Magentic-One subset, while joint repair achieves a success rate of 18.3%. Turning attribution results into effective joint interventions remains a challenge. Our future work will focus on joint repair of multiple error sources and on using repair outcomes as feedback to guide and improve multi-source attribution.

To evaluate the practical value of attribution, we follow DoVer's intervention-and-re-execution protocol, extend it to multi-source repair, and conduct experiments on Magentic-One failed trajectories. GAMA uses held-out attribution predictions.

We compare three repair strategies:

  • DoVer: Sequentially attempt candidate interventions until the task succeeds or the candidates are exhausted.
  • GAMA (joint repair): Predict the complete source-error set in one pass, generate a joint intervention addressing the predicted sources, and re-execute once.
  • GAMA (single-source repair): Repair one predicted source per round. If the task still fails, repeat attribution and repair, allowing up to 10 attempts per task.
Repair strategySuccess Rate (%)Total Attempts
DoVer11.8357
GAMA (joint repair)18.393
GAMA (single-source repair)17.2819

Total attempts counts intervention-and-re-execution attempts across all tasks, including unsuccessful cases. Under the respective stopping rules, joint repair achieves the highest recovery rate with the fewest attempts, supporting the practical value of identifying and addressing multiple error sources together.

Question 3: How graph priors and LLM verification affect false positives and false negatives

The LLM rejects most high-score non-source steps and recovers some low-score gold sources, but it still selects some non-source steps and misses some true sources.

LLM decisionHigh-score gold sourceHigh-score non-sourceLow-score gold sourceLow-score non-source
Selected70.20%26.75%30.78%3.50%
Rejected29.80%73.25%69.22%96.50%

False positives and incorrect high-score priors. The LLM rejects most high-score non-source steps, but still selects some of them. In comparison, only a small fraction of low-score non-source steps are selected.

False negatives and semantic verification. The LLM recovers some low-score gold sources, but misses most of them. Most high-score gold sources are retained, while some are still missed.

Case study. We provide a case-study figure in the "Case Study" section of the paper's appendix. Step 29 introduces unsupported winner records, while Step 42 introduces a new incorrect year and violates the task's temporal constraint. DeepSeek alone identifies only Step 42. With graph scores as soft priors, GAMA recovers the complete annotated source-error set {29, 42}, while rejecting the high-scoring non-source Steps 33 and 36. This case illustrates the complementary roles of the two modules: graph priors help recover a missed source, while LLM verification rejects incorrect candidates.

Reviewer_BWKjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper discusses multi-source failure attribution which refers to the issue of which agent-step pair is the source of the error vs propagated error in a multi-agent system. To address this issue they propose a method called GAMA and a benchmark called Whos & Whens to look at utilizing graphs to determine which steps would cause an error/introduce a new error. They identify execution graphs to show each component of the multi-agent execution and then uses these with a GNN to assign a score that is used for LLM determination of source-error steps.

Critical strengths
  1. I think the benchmark is pretty comprehensive across different general reasoning tasks.
  2. Improvements over different LLM backbones to find the exact step attribution accuracy.
  3. Well-motivated and I think the separation of source vs propagated is useful.
Critical weaknesses
  1. The graph structure isn't sufficiently justified in the various experiments done, is the GNN the best use case over this graph structure or is it possible to achieve similar performance using a transformer or MLP on the same step representations.
  2. Is the LLM scorer necessary as well, it seems like a heavy weight step and i'm wondering if some lightweight classifier could perform similarly? Such as GNN + MLP classifier.
Questions for authors

What is the performance like over different numbers of source errors? I wonder if single-source is dominating the results.

Are there any applications on real multi-agent failure cases? Does it generalize?

3 comments
AuthorsOpenReview ↗

Response to Reviewer BWKj (1/2)

We sincerely thank you for your comments.

Weakness 1: Contribution of the graph encoder

We use execution graphs and GNN to explicitly model dependencies between steps and combine step information along these relations, enabling the model to learn error-propagation patterns. In comparison, an MLP has difficulty tracing errors across steps. A Transformer can model connections between steps, but its attention weights do not directly reflect the dependencies between them.

To test these alternatives, we replace the graph encoder with an MLP or a Transformer while keeping the step input representations, final DeepSeek attribution module, and evaluation protocol unchanged.

MethodEMMacro-F1Micro-F1
MLP + LLM28.5953.4451.61
Transformer + LLM28.0752.8851.57
GAMA (Graph Scorer + LLM)33.5157.5059.00

The graph encoder performs better than both alternatives across all three metrics, supporting the effectiveness of explicitly constructing execution graphs and using a GNN for source-error attribution.

Weakness 2: Necessity of the final LLM and lightweight classifier alternatives

The final LLM is necessary because error attribution requires understanding each step in the context of the complete trajectory to determine whether it introduces a new error or inherits an earlier one.

We first clarify that the Graph Scorer itself is a lightweight GNN + MLP model: the GNN encodes execution relations between steps, while the MLP combines candidate-step and terminal-step representations to predict source-error scores. The final LLM uses these scores together with the complete trajectory to perform semantic verification and determine the source-error set.

To test whether lightweight classifiers can replace the final LLM, we compare direct attribution predictions from MLP, Transformer, and Graph Scorer classifiers. We also increase the parameter count of the original Graph Scorer's MLP while retaining the GNN.

MethodEMMacro-F1Micro-F1
MLP15.0050.8849.16
Transformer15.4752.5749.67
Graph Scorer18.0353.4549.08
GNN + larger MLP18.1453.5549.12
GAMA (Graph Scorer + LLM)33.5157.5059.00

Lightweight classifiers perform well below full GAMA, and increasing the MLP size brings only a small improvement. This supports the important role of the final LLM's semantic understanding and verification in GAMA.

Question 1: Performance across source-error counts

GAMA outperforms the baselines in every source-error-count group. We report EM (%) grouped by the ground-truth number of source errors, k.

Methodk = 1k = 2k = 3k >= 4
All-at-Once22.6913.616.3110.50
CORRECT28.4814.1416.5011.42
EDGE29.2018.0613.5912.79
GAMA41.5924.8722.8217.81

The improvement is larger in single-source cases, while all methods have lower accuracy in multi-source cases, reflecting the difficulty of multi-source attribution. The true source count is unknown at inference time, so the model must determine both how many source errors exist and which steps introduce them. Recovering the complete set therefore becomes harder as the number of source errors increases.

AuthorsOpenReview ↗

Response to Reviewer BWKj (2/2)

Question 2: Generalization to unseen domains, unseen systems, and real failure logs

GAMA demonstrates the ability to generalize across task domains and multi-agent systems, as different tasks and systems can share similar error-propagation patterns. The graph scorer learns these patterns from inter-step dependencies, providing reusable structural priors. Our synthetic data are generated from actual multi-agent executions through context-dependent error injection and execution-based failure validation. The resulting trajectories capture how errors propagate through agent interactions, allowing the scorer to learn patterns that may also arise in natural failures.

To validate this generalization, we conduct leave-one-domain-out and leave-one-system-out experiments within Whos & Whens and evaluate transfer to Who&When [1], an external benchmark containing real multi-agent failure cases.

1. Generalization to unseen task domains.

We exclude the target domain from Graph Scorer training and evaluate on that held-out domain. The following results are EM (%); "leave-one" denotes leave-one-domain-out evaluation.

MethodMATHGSM8KMMLU-ProSciBenchHumanEvalGAIAOverall
CORRECT23.3826.5320.8320.0016.3127.9122.37
EDGE23.6327.7023.2122.3915.9627.9123.42
GAMA32.8439.9432.4430.7531.2133.4933.51
GAMA (leave-one)32.0939.0731.8431.3430.1433.0232.98

Under leave-one-domain-out evaluation, GAMA's overall EM drops by 0.53 %, with performance across individual domains remaining close to the standard setting. These results support the transferability of the learned graph priors to task domains unseen during training.

2. Generalization to unseen multi-agent systems.

We further exclude the target multi-agent system from Graph Scorer training and evaluate on that held-out system. The following results are EM (%); "leave-one" denotes leave-one-system-out evaluation.

MethodMacNetAgentVerseDebateSmolAgentsDyLANMagentic-OneOverall
CORRECT21.6126.1620.7523.777.7727.9622.37
EDGE22.6029.6520.2223.774.8527.9623.42
GAMA38.9834.8827.4930.3310.6837.6333.51
GAMA (leave-one)36.4432.5525.6128.6810.6836.5631.42

Under leave-one-system-out evaluation, GAMA's overall EM drops by 2.09 %, but it still outperforms both baselines on every system. The larger drop compared with leave-one-domain-out evaluation suggests that cross-system transfer is more challenging than cross-domain transfer.

3. Generalization to real failure logs.

We evaluate GAMA on real execution failures in the external Who&When benchmark. Since Who&When includes GAIA tasks and the Magentic-One system, we separately exclude their trajectories from Graph Scorer training to assess dependence on training data from the same domain or system.

The following results are accuracy (%). HC denotes hand-crafted systems, and AG denotes algorithm-generated systems.

MethodHC: Agent-levelHC: Step-levelAG: Agent-levelAG: Step-level
All-at-Once48.2810.3459.5230.95
CORRECT53.4518.9763.4938.89
GAMA (all training data)65.5229.3172.2254.76
GAMA (w/o GAIA)63.7929.3170.6353.97
GAMA (w/o Magentic-One)63.7927.59----

GAMA outperforms the baselines in both agent-level and step-level accuracy. It retains this advantage on both subsets after excluding GAIA, and on the hand-crafted subset after excluding Magentic-One. These results support GAMA's transfer to external real failure logs.

[1] Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems.

Thanks for addressing my concerns, especially with the ablations on the graph encoder. I will raise my rating to 3 accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.