- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies multi-source failure attribution in multi-agent systems, distinguishing error origins from downstream propagated errors. It introduces GAMA, which combines execution-graph scoring with LLM reasoning, and Whos & Whens, a benchmark containing 1,913 failed trajectories. Experiments demonstrate improved multi-source attribution across three LLM backbones and strong localization results on a single-source benchmark.
Critical strengths
The paper addresses the important problem of identifying multiple error sources in multi-agent failures and contributes a benchmark with execution-based validation and human assessment. GAMA combines structural dependencies with LLM reasoning to distinguish error origins from propagated errors. Evaluation across three LLM backbones and a separate single-source benchmark supports its effectiveness, while ablations demonstrate the contributions of the graph relations, bidirectional updates, and terminal-context scoring.
Critical weaknesses
- The paper shows that dependency edges improve attribution, but does not directly evaluate their extraction quality or sensitivity to incorrect and missing edges. A controlled edge-addition/deletion experiments would help assess the reliability of this central component.
Questions for authors
-
In execution graph construction, the four dependency categories are plausible, but their definitions and coverage are insufficiently explained. For example, tool outputs may also constitute data dependencies, while shared-state or resource dependencies may be implicit in the logs. Since these categories appear to be merged into a single dependency relation, please clarify their role in extraction and how ambiguous cases are handled. An analysis of extracted edge quality would help establish whether the graph reliably captures the dependencies needed for failure attribution.
-
MAS structure shapes agent collaboration and interaction, influencing how errors propagate and affect task outcomes. How does GAMA perform across different organizational structures, such as centralized, decentralized, and hierarchical systems? Are there qualitative differences in its ability to distinguish source errors from propagated errors across these structures?
-
In the second panel of Figure 2, the roles of the node and edge features are unclear because their connections to the computation are not explicitly shown. Please indicate how node features initialize the node representations and how edge features enter message construction before the GRU updates. More generally, please carefully review the figures to ensure that they accurately convey the methodology and help readers understand the corresponding text.
3 comments
Response to Reviewer r1JR (2/2)
Question 2: Performance across organizational structures
GAMA outperforms both baselines across the evaluated centralized hierarchical, decentralized, and hybrid systems, although attribution accuracy varies across systems.
We group the systems by their main control flow and communication mechanism. MacNet and Debate are decentralized systems. AgentVerse and DyLAN are hybrid systems that use dynamic teams and dynamic routing. SmolAgents and Magentic-One are centralized hierarchical systems. Dynamic teams change which agents take part in a task, while dynamic routing changes which agents interact and how information passes between them during execution.
The following results are EM (%).
| Method | MacNet | Debate | AgentVerse | DyLAN | SmolAgents | Magentic-One | Overall |
|---|---|---|---|---|---|---|---|
| CORRECT | 21.61 | 20.75 | 26.16 | 7.77 | 23.77 | 27.96 | 22.37 |
| EDGE | 22.60 | 20.22 | 29.65 | 4.85 | 23.77 | 27.96 | 23.42 |
| GAMA | 38.98 | 27.49 | 34.88 | 10.68 | 30.33 | 37.63 | 33.51 |
Accuracy differences across systems. All three methods have low attribution accuracy on DyLAN. Its dynamic interactions can make error propagation more complex, making it harder to trace error origins and distinguish source errors from propagated errors.
Question 3: Clarifying node initialization, message construction, and GRU updates
We have revised the framework figure to explicitly connect the features and computation modules, clarifying the following steps:
- Node initialization: Node features and pass through the node-projection MLP to obtain the initial representations and .
- Message construction: The current node representations , edge features , and elapsed-time feature enter the message MLP together to produce .
- Updates to both endpoints: The message and each endpoint's current state enter the corresponding GRU to update the source and target nodes.
Thank you again for your careful reading and helpful suggestions.
Thanks for your response. My concern has been addressed. Please revise the manuscript accordingly.
Response to Reviewer r1JR (1/2)
We sincerely thank you for your detailed comments.
The graph scorer is more sensitive to missing dependency edges, while adding noisy edges has a smaller effect. The graph scorer uses a GNN to combine information along execution relations. Removing dependency edges reduces the structural information available for attribution. In comparison, adding noisy edges preserves the original dependencies, so the model can still use these relations to gather useful information.
We conduct controlled dependency-edge perturbation experiments by deleting existing dependency edges or adding extra edges to introduce noise. All added edges point from earlier to later steps. We evaluate the Graph Scorer without the final LLM. All metrics are percentages.
EM and Macro-F1 decrease as more dependency edges are removed, highlighting the importance of retaining dependency information for source-error attribution. With different levels of added edge noise, all three metrics remain close to their original values, indicating that the graph scorer maintains relatively stable attribution performance despite extra noisy connections.
We list four dependency categories in the prompt to guide the LLM to identify dependencies through different types of information, covering as many dependencies visible in the log as possible and reducing omissions. These labels guide extraction but all map to a single dependency relation during graph construction and scoring. The graph scorer therefore uses only two relation types: dependency and same-agent.
When categories overlap, such as when a tool result also serves as data input, the extractor selects one main label without adding duplicate dependency edges for the same step pair. The label choice does not change graph encoding as long as the two endpoints remain the same.
Shared-state or resource dependencies are included when the log provides evidence of a dependency between the corresponding steps. Implicit dependencies that cannot be identified from the log remain outside the current coverage.