- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Conditional State Decoding (CSD), a training-free framework for failure attribution in LLM-based multi-agent systems. It uses a frozen language model to assess four source-to-step correctness relationships (i.e., normal behavior, error onset, propagation, and recovery) and combines these assessments through exact dynamic programming over a dependency forest to identify the earliest error source. Experiments on CORRECT-Error and Who&When evaluate its attribution accuracy against prompting and voting-based baselines, along with analyses of decoding strategies and dependency constraints.
Critical strengths
S1. Structured Approach to Agent Failure Attribution
While I feel that the formulation and evaluation lacks enough rigor, I appreciate the overall approach of moving beyond direct LLM-based error localization toward structured trajectory analysis for agent failure attribution.
Critical weaknesses
W1. Limited Benchmark Coverage and Insufficient Comparison with Existing Methods
The experimental evaluation is limited to two relatively small datasets (80 CORRECT-Error traces and 126 Who&When Algorithm-Generated traces), without considering other Who&When subsets or more recent benchmarks such as AgentRx. Moreover, the comparisons primarily focus on direct prompting and voting-based baselines, omitting stronger recent failure-attribution methods such as AgentRx. Given the relatively modest improvements over the strongest reported voting baselines, the limited benchmark coverage and lack of competitive comparisons make it difficult to establish the effectiveness and generalizability of the proposed method.
W2. Restrictive Structural Assumptions and Unclear Methodological Generality
The proposed method assumes that each execution step has a single principal source and can be represented by a binary correctness state, resulting in four predefined source-to-step relations. However, realistic multi-agent workflows may involve multiple upstream dependencies, concurrent errors, or steps that simultaneously repair one error while introducing another. These structural assumptions may oversimplify complex failure propagation patterns, potentially limiting the method's applicability to diverse agent systems. Furthermore, the dependency forest is extracted using an LLM without direct validation of its accuracy, leaving the robustness of the overall framework to incorrectly inferred dependencies insufficiently examined.
Questions for authors
Q1. Benchmark Selection and Ground-Truth Reliability
The evaluation is limited to 80 CORRECT-Error traces and the 126 Algorithm-Generated traces from Who&When. Why are the other Who&When subsets excluded, and why are more recent benchmarks, such as AgentRx, not considered? Furthermore, given the potential ambiguity of single-step failure attribution, how reliable are the ground-truth labels, and how might alternative valid attributions affect the reported results?
Q2. Significance of Improvements and Comparison with Stronger Baselines
The improvements over the strongest voting baselines are relatively modest, corresponding to only two additional correct predictions on CORRECT-80 and six on Who-AG. Are these improvements statistically significant? Moreover, why are stronger recent failure-attribution methods, such as AgentRx, not included in the comparison?
Q3. Generalizability of the Dependency Representation
CSD assumes that each step has a single principal source and represents its correctness using a binary state, enabling four source-to-step relations. However, realistic multi-agent workflows often involve multiple upstream dependencies, concurrent errors, or steps that simultaneously correct one error and introduce another. How well can this representation capture such scenarios, and to what extent do these structural assumptions limit the applicability of CSD to more complex agent systems?
Q4. Reliability and Impact of Dependency Forest Extraction
The dependency forest is a fundamental component of CSD, yet it is constructed through an LLM-based extraction procedure without direct evaluation of its accuracy. How reliable are the extracted parent-child dependencies? Could the authors evaluate dependency extraction against manually verified relationships and compare attribution performance using automatically extracted versus ground-truth dependency structures to quantify the impact of structural errors?