- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates failure attribution in multi-agent LLM systems, focusing on identifying both the responsible agent and the root-cause step. It introduces the concept of symptom attraction, where attribution methods tend to identify downstream failure symptoms rather than earlier omitted actions.
To address this problem, the authors propose GCCA, which constructs a goal-derived necessary-precondition chain and uses hierarchical localization to identify the earliest unmet condition. Experiments on Who&When demonstrate improvements on long trajectories, particularly for omission errors, while maintaining relatively low token consumption.
Critical strengths
-
The paper identifies an interesting and practically relevant failure mode. The distinction between omission errors and downstream symptoms helps explain why existing attribution methods struggle with long trajectories.
-
GCCA is intuitive, interpretable, and computationally efficient. Its goal-conditioned formulation provides a useful alternative to trajectory-only reasoning, and the reported results demonstrate promising improvements in long-trajectory attribution.
Critical weaknesses
-
Insufficient theoretical justification. The information-theoretic analysis largely depends on assumed properties of local evidence. In particular, the monotonic information assumption is not independently established. Moreover, the accuracy-based probe does not directly estimate conditional mutual information, and its reported improvement has limited statistical support.
-
Limited empirical validation. The main improvements are demonstrated on only 58 long trajectories. The exact-step accuracy increases from 17.8% to 23.0%, but statistical significance against the strongest baselines is not established. Furthermore, the near-zero aggregate signed bias may partially result from cancellation between positive and negative errors. The lack of an independent long-trajectory evaluation also limits conclusions about generalizability.
Questions for authors
-
Can the authors provide paired statistical significance tests and confidence intervals for the main accuracy improvements over the strongest baselines?
-
What empirical evidence supports the monotonic information assumption in Eq. (2), and how does the accuracy-based probe substantiate the conditional mutual information claim in Eq. (3)?
-
How sensitive is GCCA to incorrect or unnecessary preconditions? Given the approximately 68% unconditional chain-faithfulness rate reported for Hand-Crafted trajectories, can the authors provide additional validation of precondition quality and its impact on attribution accuracy?
3 comments
Thanks for your response. But I still have other problems.
- Execution omissions do not necessarily lead to task failure, nor are they always the root cause, as certain steps of a task may already be known conditions (or implicit knowledge) to the model.
- There may be more complex graph-structured relationships between preconditions and execution steps, making the current approach overly coarse.
- What is the rationale behind the selection of the routing threshold separating short and long trajectories?
Thank you for the follow-up. Brief answers to the three points; numbers that are not in the PDF are marked as such.
1. Omissions need not cause failure or be the root cause. We agree, and the method assumes neither. (i) GCCA does not blame a skipped action. It asks which goal-derived condition is left unmet and blames the first step that leaves one unmet. If the agents skip a step because the fact is already known, the condition is satisfied and nothing is blamed. The inference runs from the failure to an unmet necessary condition, not from an omission to the failure. (ii) The root-cause step is the benchmark's human label, not ours; "omission" is a two-annotator category of that labelled step (κ = 0.69), used only to slice results. (iii) The same rule covers failures that are not omissions, since a wrong action also leaves a condition unmet, and we report both types (Table 3): on omissions GCCA is the most accurate (26.2 exact, 62.7 at ±5); on commissions it is at 29.2 against A2P's 37.5 (Limitation 5); on Algorithm-Generated, where 4% of failures are omissions, it ties A2P (33.9 vs. 33.6). (iv) If GCCA treats an implicitly satisfied condition as unmet, its prediction misses the labelled step and counts as an error. Such cases are already inside the reported accuracies, under the same backbone and inputs as every baseline.
2. Graph-structured relations. A fair point: the chain is a linear simplification and does not model AND/OR relations among preconditions (Limitation 4). What we can say about its cost:
- In the paper: the chain order is only a hint, because the rule takes the earliest violating step, not the first condition in list order; an extra condition that the trajectory satisfies is never blamed; and pruning the chain lowers accuracy (27.0 → 20.7, App. C). CHIEF, which builds a hierarchical causal graph, reaches 9.8 on Hand-Crafted against 27.0, at about 15 times the tokens (Table 2).
- From development, not in the paper: we tried three departures from the plain chain. (a) Conditions as an unordered set, each judged on its own: Hand-Crafted 61.5 / 22.4 and Algorithm-Generated 46.8 / 28.8 (agent / step), against 70.1 / 23.0 and 51.1 / 33.9. (b) A chain aligned to the path the agents took: 67.2 / 22.4 and 53.2 / 34.4, no gain. Both are switches in the released code (
--variant orderfree,pathcond); they were run with GPT-4o before the parser correction, each against the baseline under the same parser. (c) Keeping CHIEF's graph and checking each necessary condition against it (DeepSeek-V3.2-thinking, 3 runs): step accuracy is unchanged on Algorithm-Generated (30.2) and falls from 15.5 to 6.9 on Hand-Crafted.
Our reading is that richer structure asks the model for many separate judgments, which were unreliable on long trajectories in these attempts, whereas one judgment over the whole trajectory and the whole chain held up. We do not claim that a precondition graph cannot help; it is a natural next step.
3. The routing threshold. Two questions are involved. Why two paths: a short trajectory is already a single window, so the direct path reads it once (2 calls); the hierarchical path (6 calls) first narrows a long trajectory to a window. Forcing the single pass on every trajectory lowers Hand-Crafted from 67.8 / 27.0 to 64.4 / 25.3 (Table 4b), mostly on the agent. Why 15: all 126 Algorithm-Generated trajectories have at most 10 steps, so any k ≥ 10 sends them to the direct path; on Hand-Crafted, 7 of 58 trajectories have at most 15 steps. For k from 10 to 30, Hand-Crafted step accuracy stays within 21.3–27.0, and agent accuracy is 69.0 and 67.8 for k ≤ 15 against 60.9–64.4 for k ≥ 20 (App. H). Split by length (not in the PDF; alignment applied to every method): on the 7 short Hand-Crafted trajectories the methods are close (GCCA 42.9, A2P 47.6, ECHO 42.9); on the 51 long ones GCCA is at 24.8 against 17.6 for A2P and 18.3 for ECHO.
Thank you for the review. The revision adds the paired tests and confidence intervals of Q1, answers the cancellation concern with error sizes (W2), and tests sensitivity to unnecessary preconditions (Q3). One correction comes first; it concerns the 68% in your Q3 and changes the Hand-Crafted numbers.
Correction. App. C of the submission disclosed a chain-parsing fault, reported runs made before the fix, and described the accuracies as unaffected. They are not. The fault: markdown in the model's chain output made the parser return a placeholder on 16% of long Hand-Crafted chains. With the fix as the default (one line at each of three parse sites, in the supplementary code), we re-ran every GCCA Hand-Crafted experiment from scratch. No chain degrades (0 of 58), and Hand-Crafted is now Who 67.8 (was 70.1), When 27.0 (23.0), ±5 59.2 (56.3), bias +0.76 (+1.03). All Hand-Crafted GCCA numbers below come from the corrected code; our "Summary of changes" comment gives the details.
Q1, W2 (paired tests and confidence intervals). New Table 10. The step alignment that GCCA uses is now applied to every method, so the comparison is like for like (Table 8): Hand-Crafted step accuracy is 27.0 for GCCA and 21.3 for A2P and ECHO. GCCA vs. A2P over the 58 trajectories (difference; one-sided McNemar on the per-item majority vote; paired-bootstrap 95% CI): exact +5.7, p = .035 [−1.7, 13.2]; ±5 +11.5, p = .018 [3.4, 19.5]; on the 42 omissions, exact +11.1, p = .008 [3.2, 19.8] and ±5 +15.1, p = .011 [5.6, 24.6]. Table 10 reports the same tests against ECHO and CHIEF.
W2 (cancellation). The aggregate +0.76 does combine +5.4 on omissions with −11.3 on commissions, and §4.2 now says so. It is not large errors cancelling. On omissions GCCA's mean absolute error is 11.7 steps with a median of 4, against 12.4 for A2P, 21.7 for ECHO and 23.3 for CHIEF (main-text Table 3). Table 9 shows where the omission predictions land: within one step of the root cause on 34% for GCCA, and past the symptom on 14%, against 32% for ECHO and 48% for CHIEF.
W2 (independent long-trajectory evaluation). We agree this is the main limit of the evidence. We found no other public benchmark that pairs long trajectories with root-cause labels. For the closest candidate, AFTraj-Agentic, two annotators independently labelled all 470 failures with the Hand-Crafted protocol (App. F). 51.5% of its failures are omissions, but its symptoms are immediate: the annotated symptom coincides with the root cause on 80% of failures, and the mean cause–symptom gap is 0.3 steps against 9.2 on Hand-Crafted omissions. It tests the boundary, not the target regime, and behaves as the mechanism predicts: no method has a late bias there (largest +0.6 steps), and GCCA is on par at the step (52.5 against 53.3 for the best).
W1, Q2 (the assumptions; what the probe shows). We agree with both points. Eq. (2) is assumption A2 of a stylized model (App. A); we do not measure it directly. We test its consequences on the annotated failures. Direction: ECHO and CHIEF land nearer the symptom than the root cause on 76% and 82% of gap cases, against 57.2% for a random predictor (p = .011, .001; App. G). Location: 32.5% of ECHO's and 48.4% of CHIEF's omission predictions fall past the annotated symptom (Table 9). Gap: where the annotated cause–symptom gap is 9.2 steps their omission bias is +19 to +22, and where it is 0.3 steps no late bias appears. Because these readers overshoot the first symptom, Proposition 1 is now a lower bound, E[r^−R⋆]≥μprop, and the text says the direction is tested, not that the model is verified. On Eq. (3): the probe is an accuracy difference between two matched conditions, and the revision calls it an operational proxy for the conditional term, not an estimate of mutual information. Its support is directional: +5.6 points on omissions (one-sided p = .055), +4.8 in a rerun three months later, +5.2 pooled (95% CI [−0.8, 12.7], one-sided p = .048). At this effect size p < .05 needs about 50–60 omissions, and Hand-Crafted has 42 (App. C).
Q3 (incorrect or unnecessary preconditions; the 68%). Unnecessary conditions: a variant that prunes the chain to the conditions the model judges necessary lowers Hand-Crafted accuracy, 27.0 → 20.7 at exact and 59.2 → 54.0 at ±5 (paired-bootstrap 95% CIs [−12.1, −1.1] and [−10.3, −0.6]; App. C). GCCA blames the first unmet condition, so a spurious condition that the trajectory satisfies is never blamed, whereas pruning can remove a needed one. Missing chain: without the chain, Hand-Crafted falls from 67.8 / 27.0 to 62.6 / 21.3 and the late bias returns (+0.76 → +12.5; Table 4b, §4.3). The 68%: that figure counted the 16% placeholder chains as misses. With the fix no chain is a placeholder (0 of 58); the 81.4% is an LLM-judge estimate on the chains that parsed, and App. C says so. The method is thus insensitive to extra conditions and sensitive to missing ones (Limitations 3 and 4).