acceptodds
Under review as a conference paper at ICLR 2027

Finding Root Causes, Not Symptoms: A Mechanism and Method for Agent Failure Attribution

Abstract

When a multi-agent LLM system fails over a long run, naming the responsible agent is often the easy part. Pinpointing the step where the failure began is hard, and hardest on the long trajectories that matter most. We find that on such runs the dominant error is an omission: a necessary action is silently skipped, so it leaves no local trace and surfaces only later as a downstream symptom. Attributors that score steps by local evidence are therefore pulled toward that late symptom, a systematic bias we call symptom attraction. But what the run needed to do is determined by the task, not by the trajectory. From the question and its correct answer, which we call the goal, the conditions that any successful run must meet can be listed. The condition the skipped action would have met is on that list, even though no step in the trajectory shows it. GCCA (Goal-Conditioned Causal Attribution) turns the goal into a chain of necessary preconditions and blames the first step that leaves one unmet. On Who&When, GCCA is the best zero-shot attributor at both naming the agent (70.1%) and pinpointing the step (23.0%) on the long, omission-heavy subset. It also cuts the late bias from as much as +15.7 steps to a near-zero +1.03. The bias that reveals the problem is thus the bias GCCA removes. GCCA stays competitive on short trajectories, uses the fewest tokens at 4.7k per sample against 8–67k, and holds up across five backbones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM multi-agent systems, failure attribution, root cause analysis, error localization, LLM agents, long-horizon trajectories, goal-conditioned reasoning, training-free

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qr32OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates failure attribution in multi-agent LLM systems, focusing on identifying both the responsible agent and the root-cause step. It introduces the concept of symptom attraction, where attribution methods tend to identify downstream failure symptoms rather than earlier omitted actions.

To address this problem, the authors propose GCCA, which constructs a goal-derived necessary-precondition chain and uses hierarchical localization to identify the earliest unmet condition. Experiments on Who&When demonstrate improvements on long trajectories, particularly for omission errors, while maintaining relatively low token consumption.

Critical strengths
  1. The paper identifies an interesting and practically relevant failure mode. The distinction between omission errors and downstream symptoms helps explain why existing attribution methods struggle with long trajectories.

  2. GCCA is intuitive, interpretable, and computationally efficient. Its goal-conditioned formulation provides a useful alternative to trajectory-only reasoning, and the reported results demonstrate promising improvements in long-trajectory attribution.

Critical weaknesses
  1. Insufficient theoretical justification. The information-theoretic analysis largely depends on assumed properties of local evidence. In particular, the monotonic information assumption is not independently established. Moreover, the accuracy-based probe does not directly estimate conditional mutual information, and its reported improvement has limited statistical support.

  2. Limited empirical validation. The main improvements are demonstrated on only 58 long trajectories. The exact-step accuracy increases from 17.8% to 23.0%, but statistical significance against the strongest baselines is not established. Furthermore, the near-zero aggregate signed bias may partially result from cancellation between positive and negative errors. The lack of an independent long-trajectory evaluation also limits conclusions about generalizability.

Questions for authors
  1. Can the authors provide paired statistical significance tests and confidence intervals for the main accuracy improvements over the strongest baselines?

  2. What empirical evidence supports the monotonic information assumption in Eq. (2), and how does the accuracy-based probe substantiate the conditional mutual information claim in Eq. (3)?

  3. How sensitive is GCCA to incorrect or unnecessary preconditions? Given the approximately 68% unconditional chain-faithfulness rate reported for Hand-Crafted trajectories, can the authors provide additional validation of precondition quality and its impact on attribution accuracy?

3 comments

Thank you for the review. The revision adds the paired tests and confidence intervals of Q1, answers the cancellation concern with error sizes (W2), and tests sensitivity to unnecessary preconditions (Q3). One correction comes first; it concerns the 68% in your Q3 and changes the Hand-Crafted numbers.

Correction. App. C of the submission disclosed a chain-parsing fault, reported runs made before the fix, and described the accuracies as unaffected. They are not. The fault: markdown in the model's chain output made the parser return a placeholder on 16% of long Hand-Crafted chains. With the fix as the default (one line at each of three parse sites, in the supplementary code), we re-ran every GCCA Hand-Crafted experiment from scratch. No chain degrades (0 of 58), and Hand-Crafted is now Who 67.8 (was 70.1), When 27.0 (23.0), ±5 59.2 (56.3), bias +0.76 (+1.03). All Hand-Crafted GCCA numbers below come from the corrected code; our "Summary of changes" comment gives the details.

Q1, W2 (paired tests and confidence intervals). New Table 10. The step alignment that GCCA uses is now applied to every method, so the comparison is like for like (Table 8): Hand-Crafted step accuracy is 27.0 for GCCA and 21.3 for A2P and ECHO. GCCA vs. A2P over the 58 trajectories (difference; one-sided McNemar on the per-item majority vote; paired-bootstrap 95% CI): exact +5.7, p = .035 [−1.7, 13.2]; ±5 +11.5, p = .018 [3.4, 19.5]; on the 42 omissions, exact +11.1, p = .008 [3.2, 19.8] and ±5 +15.1, p = .011 [5.6, 24.6]. Table 10 reports the same tests against ECHO and CHIEF.

W2 (cancellation). The aggregate +0.76 does combine +5.4 on omissions with −11.3 on commissions, and §4.2 now says so. It is not large errors cancelling. On omissions GCCA's mean absolute error is 11.7 steps with a median of 4, against 12.4 for A2P, 21.7 for ECHO and 23.3 for CHIEF (main-text Table 3). Table 9 shows where the omission predictions land: within one step of the root cause on 34% for GCCA, and past the symptom on 14%, against 32% for ECHO and 48% for CHIEF.

W2 (independent long-trajectory evaluation). We agree this is the main limit of the evidence. We found no other public benchmark that pairs long trajectories with root-cause labels. For the closest candidate, AFTraj-Agentic, two annotators independently labelled all 470 failures with the Hand-Crafted protocol (App. F). 51.5% of its failures are omissions, but its symptoms are immediate: the annotated symptom coincides with the root cause on 80% of failures, and the mean cause–symptom gap is 0.3 steps against 9.2 on Hand-Crafted omissions. It tests the boundary, not the target regime, and behaves as the mechanism predicts: no method has a late bias there (largest +0.6 steps), and GCCA is on par at the step (52.5 against 53.3 for the best).

W1, Q2 (the assumptions; what the probe shows). We agree with both points. Eq. (2) is assumption A2 of a stylized model (App. A); we do not measure it directly. We test its consequences on the annotated failures. Direction: ECHO and CHIEF land nearer the symptom than the root cause on 76% and 82% of gap cases, against 57.2% for a random predictor (p = .011, .001; App. G). Location: 32.5% of ECHO's and 48.4% of CHIEF's omission predictions fall past the annotated symptom (Table 9). Gap: where the annotated cause–symptom gap is 9.2 steps their omission bias is +19 to +22, and where it is 0.3 steps no late bias appears. Because these readers overshoot the first symptom, Proposition 1 is now a lower bound, E[r^−R⋆]≥μpropE[\hat r - R^\star] \ge \mu_{prop}, and the text says the direction is tested, not that the model is verified. On Eq. (3): the probe is an accuracy difference between two matched conditions, and the revision calls it an operational proxy for the conditional term, not an estimate of mutual information. Its support is directional: +5.6 points on omissions (one-sided p = .055), +4.8 in a rerun three months later, +5.2 pooled (95% CI [−0.8, 12.7], one-sided p = .048). At this effect size p < .05 needs about 50–60 omissions, and Hand-Crafted has 42 (App. C).

Q3 (incorrect or unnecessary preconditions; the 68%). Unnecessary conditions: a variant that prunes the chain to the conditions the model judges necessary lowers Hand-Crafted accuracy, 27.0 → 20.7 at exact and 59.2 → 54.0 at ±5 (paired-bootstrap 95% CIs [−12.1, −1.1] and [−10.3, −0.6]; App. C). GCCA blames the first unmet condition, so a spurious condition that the trajectory satisfies is never blamed, whereas pruning can remove a needed one. Missing chain: without the chain, Hand-Crafted falls from 67.8 / 27.0 to 62.6 / 21.3 and the late bias returns (+0.76 → +12.5; Table 4b, §4.3). The 68%: that figure counted the 16% placeholder chains as misses. With the fix no chain is a placeholder (0 of 58); the 81.4% is an LLM-judge estimate on the chains that parsed, and App. C says so. The method is thus insensitive to extra conditions and sensitive to missing ones (Limitations 3 and 4).

Thanks for your response. But I still have other problems.

  1. Execution omissions do not necessarily lead to task failure, nor are they always the root cause, as certain steps of a task may already be known conditions (or implicit knowledge) to the model.
  2. There may be more complex graph-structured relationships between preconditions and execution steps, making the current approach overly coarse.
  3. What is the rationale behind the selection of the routing threshold separating short and long trajectories?

Thank you for the follow-up. Brief answers to the three points; numbers that are not in the PDF are marked as such.

1. Omissions need not cause failure or be the root cause. We agree, and the method assumes neither. (i) GCCA does not blame a skipped action. It asks which goal-derived condition is left unmet and blames the first step that leaves one unmet. If the agents skip a step because the fact is already known, the condition is satisfied and nothing is blamed. The inference runs from the failure to an unmet necessary condition, not from an omission to the failure. (ii) The root-cause step is the benchmark's human label, not ours; "omission" is a two-annotator category of that labelled step (κ = 0.69), used only to slice results. (iii) The same rule covers failures that are not omissions, since a wrong action also leaves a condition unmet, and we report both types (Table 3): on omissions GCCA is the most accurate (26.2 exact, 62.7 at ±5); on commissions it is at 29.2 against A2P's 37.5 (Limitation 5); on Algorithm-Generated, where 4% of failures are omissions, it ties A2P (33.9 vs. 33.6). (iv) If GCCA treats an implicitly satisfied condition as unmet, its prediction misses the labelled step and counts as an error. Such cases are already inside the reported accuracies, under the same backbone and inputs as every baseline.

2. Graph-structured relations. A fair point: the chain is a linear simplification and does not model AND/OR relations among preconditions (Limitation 4). What we can say about its cost:

  • In the paper: the chain order is only a hint, because the rule takes the earliest violating step, not the first condition in list order; an extra condition that the trajectory satisfies is never blamed; and pruning the chain lowers accuracy (27.0 → 20.7, App. C). CHIEF, which builds a hierarchical causal graph, reaches 9.8 on Hand-Crafted against 27.0, at about 15 times the tokens (Table 2).
  • From development, not in the paper: we tried three departures from the plain chain. (a) Conditions as an unordered set, each judged on its own: Hand-Crafted 61.5 / 22.4 and Algorithm-Generated 46.8 / 28.8 (agent / step), against 70.1 / 23.0 and 51.1 / 33.9. (b) A chain aligned to the path the agents took: 67.2 / 22.4 and 53.2 / 34.4, no gain. Both are switches in the released code (--variant orderfree, pathcond); they were run with GPT-4o before the parser correction, each against the baseline under the same parser. (c) Keeping CHIEF's graph and checking each necessary condition against it (DeepSeek-V3.2-thinking, 3 runs): step accuracy is unchanged on Algorithm-Generated (30.2) and falls from 15.5 to 6.9 on Hand-Crafted.

Our reading is that richer structure asks the model for many separate judgments, which were unreliable on long trajectories in these attempts, whereas one judgment over the whole trajectory and the whole chain held up. We do not claim that a precondition graph cannot help; it is a natural next step.

3. The routing threshold. Two questions are involved. Why two paths: a short trajectory is already a single window, so the direct path reads it once (2 calls); the hierarchical path (6 calls) first narrows a long trajectory to a window. Forcing the single pass on every trajectory lowers Hand-Crafted from 67.8 / 27.0 to 64.4 / 25.3 (Table 4b), mostly on the agent. Why 15: all 126 Algorithm-Generated trajectories have at most 10 steps, so any k ≥ 10 sends them to the direct path; on Hand-Crafted, 7 of 58 trajectories have at most 15 steps. For k from 10 to 30, Hand-Crafted step accuracy stays within 21.3–27.0, and agent accuracy is 69.0 and 67.8 for k ≤ 15 against 60.9–64.4 for k ≥ 20 (App. H). Split by length (not in the PDF; alignment applied to every method): on the 7 short Hand-Crafted trajectories the methods are close (GCCA 42.9, A2P 47.6, ECHO 42.9); on the 51 long ones GCCA is at 24.8 against 17.6 for A2P and 18.3 for ECHO.

Reviewer_ZLQCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that on long multi-agent trajectories many failures are omissions, which leave no local trace, so attributors that score steps by local evidence drift toward the later step where the failure becomes visible ("symptom attraction"). It measures this as signed step bias on Who&When Hand-Crafted (+15.21 for ECHO, +15.72 for CHIEF) and gives a stylized model of it (Proposition 1). GCCA derives a chain of necessary preconditions from the question and its correct answer, finds the first step that leaves one unmet, and maps an orchestrator's dispatching step to the step that carries it out. With GPT-4o, it reports 70.1% agent and 23.0% step accuracy on Hand-Crafted, against 64.4 and 17.8 for the best zero-shot baselines, a bias of +1.03, and the lowest token cost. The appendix adds human error-type and symptom-step labels for the 58 Hand-Crafted failures and results on five backbones.

Critical strengths

(1) The diagnostic part is what I value most. The human labeling of all 58 Hand-Crafted failures by error type and symptom step, together with the decision to test the direction of errors (nearer the symptom or nearer the root) against a random-predictor null instead of 50% (App. G, l.947–956), turns "attributors point late" into something one can measure. Under that null, ECHO and CHIEF land nearer the symptom on 74% and 75% of the gap cases (p = 0.027 and 0.012), and their bias on omissions (+18.9 and +22.1, Table 7) is above their overall Hand-Crafted bias (+15.21 and +15.72), so the late bias sits mostly in the omissions. Other work on step attribution can reuse both the labels and the test. For me this is what gives the paper a real research question.

(2) The method follows from the diagnosis. If an omission leaves no trace in the trajectory, reasoning backward from what the task required is a sensible way to find it, and the ablation shows that the precondition chain does real work (removing it drops Hand-Crafted to 54.6/14.4, Table 3b). GCCA is also the cheapest method in Table 2, and its agent-level gain on Hand-Crafted (70.1 vs 64.4) does not pass through the alignment step discussed below (l.747).

Critical weaknesses

(1) The paper's main empirical claim, that GCCA is the best zero-shot method at pinpointing the step on Hand-Crafted, is not yet established. The final alignment step (§3.2, l.288–292) is part of GCCA, and I could not find a statement that baseline predictions receive the same post-processing; §4.1 only says that all methods get the same inputs (l.310–312). Without alignment, GCCA's Hand-Crafted step accuracy is 18.4 (Table 3b, w/o Disambig.), next to A2P's 17.8.

I don't want to read too much into that row. Its agent column also moves (70.1 to 69.5, and 51.1 to 53.2 on Algorithm-Generated) although alignment does not touch agent predictions (l.747), so it is probably a separate run, and GCCA's own run-to-run std on this metric is 3.3. But that is the point: the 5.2-point lead rests on 58 items, has no between-method significance test on accuracy (the tests in App. G concern bias and error direction), and depends on a step that, as far as the paper describes, only GCCA receives. The w/o-GT row (20.7) and the five-backbone results (Table 5, no baselines) presumably keep the alignment as well, since ablations remove one component at a time (l.399) and the backbones run GCCA "unchanged" (l.408), so they cannot settle the question either.

(2) The mechanism is the paper's main conceptual contribution and is presented as verified empirically (l.653), but its quantitative prediction does not fit the paper's own measurements. In Proposition 1 the attributor returns the symptom step, so the expected bias equals the mean cause-to-symptom gap, and both relaxations only shrink it (l.670–679). The annotated symptom steps in App. G, which the paper uses to confirm the model's premise (l.915–916), give a mean gap of +7.1 steps with no symptom before its root cause (l.913–914). Since every gap is non-negative, the mean gap over the 42 omissions is at most 7.1 × 58 / 42 ≈ 9.8. ECHO and CHIEF have omission biases of +18.9 and +22.1 (Table 7), about twice that bound, so on average they land well past the annotated symptom, which the model does not predict. The nearer-symptom-or-nearer-root test (l.953–956) cannot tell stopping at the symptom from overshooting it.

The effect is also narrower than the abstract suggests. A2P is indistinguishable from random (50%, p = 0.41); on omissions GCCA's +5.8 is close to All-at-Once (+6.5) and A2P (+7.1), and its aggregate +1.03 averages +5.8 on omissions and −11.4 on commissions (l.373–374). App. G itself limits the effect to the methods that read the whole trajectory (l.924–928), and I believe the late bias of those two methods is real. The abstract and introduction (l.17–18, l.43–44), however, state it for local-evidence attributors in general and say that GCCA removes it.

Questions for authors

(1) Could you apply the same orchestrator-to-executor alignment to the cached outputs of all baselines and re-report the Hand-Crafted columns of Table 2, with an item-level paired test (for example McNemar or a paired bootstrap over the 58 items) for both agent and step accuracy? This needs no new model calls, and it is the result most likely to change my score: if GCCA keeps a significant step-level lead, I would move toward acceptance.

(2) Are the rows of Table 3b separate runs? Step accuracy with and without alignment on the same GCCA runs would show how much of the lead the alignment accounts for.

(3) (Clarification) How do you reconcile omission biases of +18.9 and +22.1 with the annotated gaps, whose mean of +7.1 over all 58 failures bounds the mean omission gap at about 9.8? If the symptom step r_m in Proposition 1 is meant to differ from the annotated R_sym (both are described as the step where the failure becomes visible, l.665 and l.908), what is it, and how was the mechanism verified empirically (l.653)?

Minor (no response needed): Figure 4 shows ECHO predicting step 5 once, while l.419–420 says that CHIEF and ECHO never reach it.

3 comments

Thank you for the careful review. Your Q1 is done: with the same alignment applied to every method, GCCA keeps its Hand-Crafted step lead (27.0 vs. 21.3), significant against A2P at the item level (McNemar p = .035 exact, .018 at ±5, .008 on omissions). One correction to the submission comes first, because it changes the Hand-Crafted numbers.

Correction. App. C of the submission disclosed a chain-parsing fault (markdown in the model's chain output made the parser return a placeholder on 16% of long Hand-Crafted chains), reported runs made before the fix, and described the accuracies as unaffected. They are not. With the fix as the default (one line at each of three parse sites, in the supplementary code), we re-ran every GCCA Hand-Crafted experiment from scratch. No chain degrades (0 of 58), and Hand-Crafted is now Who 67.8 (was 70.1), When 27.0 (23.0), ±5 59.2 (56.3), bias +0.76 (+1.03). All Hand-Crafted GCCA numbers below come from the corrected code; our "Summary of changes" comment gives the details.

Q1 / W1 (alignment for every method; item-level tests). §4.1 now states the convention, Tables 2, 3 and 4a apply it, new Table 8 gives both versions and new Table 10 the tests. The alignment is applied to the cached outputs of every baseline, as you suggested; no baseline was re-run. Hand-Crafted step accuracy, 3-run mean:

All-at-OnceA2PStep-by-StepBinary SearchECHOCHIEFGCCA
raw6.917.817.26.915.58.021.8
aligned9.221.317.26.321.39.827.0

GCCA vs. A2P over the 58 items (difference in points; one-sided McNemar on the per-item majority vote; paired-bootstrap 95% CI): step exact +5.7, p = .035 [−1.7, 13.2]; step ±5 +11.5, p = .018 [3.4, 19.5]; omissions (n = 42) exact +11.1, p = .008 [3.2, 19.8]; omissions ±5 +15.1, p = .011 [5.6, 24.6]. ECHO ties A2P at 21.3 after alignment; Table 10 reports the same tests against ECHO and CHIEF. The agent column is untouched by the alignment, as you noted; its tests are in Table 10 (+8.0 over A2P, CI [−1.7, 17.8]).

Q2 (same runs with and without alignment). The revision does exactly this. The "w/o align." row of Table 4b scores the same three runs before the alignment: 21.8 → 27.0, agent unchanged at 67.8. The rule fires on 20, 19 and 20 of 58 trajectories, corrects 5, 4, 3 and breaks 2, 0, 1 (Table 8 caption). The submitted "w/o Disambig." row was a separate run, as you suspected; the other rows of Table 4b are independent 3-run experiments, and the caption now says so. The same rule gives A2P +3.5 and ECHO +5.8, so it does not create the lead: 21.8 vs. 17.8 raw, 27.0 vs. 21.3 aligned. The other ablations and the backbone table keep the alignment, as you inferred; Table 8 is the comparison that settles it.

W2 / Q3 (the bound). You are right, and thank you for working it out from the appendix numbers: your bound is the reason for the main change to the mechanism section. rmr_m is the annotated RsymR_{sym}. The mean gap on the 42 omissions is 9.2 steps (your bound: 9.8), and ECHO and CHIEF land well past it. The submitted proposition (the attributor returns the symptom step, so the bias equals the gap) was too strong, and "verified empirically" was the wrong phrase. App. A now states a lower bound: under the same assumptions r^≥rm\hat r \ge r_m, so E[r^−R⋆]≥μpropE[\hat r - R^\star] \ge \mu_{prop}, with equality only for a reader that stops at the first symptom, and the text says the direction is tested (ECHO and CHIEF land nearer the symptom on 76% and 82% of gap cases, against 57.2% for a random predictor; 74% and 75% on raw outputs in the submission). New Table 9 measures what the nearer-symptom test could not separate. On omissions, ECHO and CHIEF land more than one step past the annotated symptom on 32% and 48% of predictions; averaged over all omission predictions they sit +10.1 and +13.0 steps past it. A2P averages −1.7 and GCCA −3.8, and GCCA is within one step of the root cause on 34%, the most of any method.

W2 (scope). Agreed. The abstract and §1 sentences you cite now say "read the whole trajectory for local evidence", and "removes" is now "reduces". That the reduction comes from the chain is what the ablation shows: without the chain the late bias returns (+0.76 → +12.5, §4.3). A2P is not symptom-attracted (61% nearer the symptom on gap cases against 57.2% for a random predictor, p = .40), and the revision says so. On omissions GCCA's bias (+5.4) is indeed close to All-at-Once (+6.6) and A2P (+7.4). What separates GCCA there is accuracy: 26.2 / 62.7 at exact / ±5, against 15.1 / 47.6 for A2P and 8.7 / 40.5 for All-at-Once (now main-text Table 3). The aggregate +0.76 is +5.4 on omissions and −11.3 on commissions (§4.2); commissions are where GCCA loses to A2P (29.2 vs. 37.5, Limitations).

Minor. Fixed: §4.4 now matches Figure 4.

We wanted to make sure the two items from your Q1 are easy to find in the revision: the alignment applied to every method is in Table 8, and the item-level tests are in Table 10. We are glad to clarify anything further before the discussion period closes.

Thanks for the detailed reply. The same-run ablation answers Q2, and stating Prop. 1 as a lower bound takes care of Q3. On Q1 though, the exact-step gain over A2P is +5.7 with a bootstrap CI of [-1.7, 13.2], so it's only significant one-sided. The ±5 and omission numbers look better, but the main claim is about pinpointing the exact step, and on 58 trajectories that still isn't settled for me. I'll keep my score.

Reviewer_gnQVOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

When a multi agent LLM system fails, two questions follow: which agent is responsible (Who) and at which step the failure began (When). This paper first documents a phenomenon it calls symptom attraction. On long runs, the dominant error type is omission: a necessary action is silently skipped. An omitted action leaves no trace in the log, so methods that score each step by local evidence are pulled toward the late step where the failure becomes visible, the symptom, not the root cause. On the longest benchmark subset, two thorough baseline readers miss by +15.2 and +15.7 steps on average.

A small information theoretic model (explicitly labeled an idealization) predicts this direction, and posits that the lost signal survives in the goal, the question and its correct answer (G = (q, a)), because the goal says which conditions had to hold even if no step addressed them. The method, GCCA, turns this into an algorithm: an LLM reverse decomposes the goal into a chain of necessary preconditions, from the goal back to the most basic prerequisite, and then names the first step that leaves a condition unmet. Short trajectories use a direct path (2 calls). Long ones use a hierarchical path (3 calls): temporally annotated chain, coarse localization over a per step summary, fine localization inside the window.

Results on the Who&When benchmark: on the long Hand Crafted subset (58 trajectories), GCCA is the best zero shot attributor on both metrics, Who 70.1% vs 64.4% for the best baseline, and When 23.0% vs 17.8%, at 4.7k tokens per sample versus 8 to 67k for alternatives, and it cuts the late bias to +1.03 steps. On the short Algorithm Generated subset it is competitive but not best on Who (51.1 vs 57.1). It is stable across five LLM backbones, and transfers on Who to a second benchmark.

Critical strengths
  1. A well defined, well measured phenomenon. Symptom attraction is crisply stated, measured with a signed bias metric, and observed in two independent benchmark sources. The narrative closes a loop: the bias that reveals the problem (+15.7) is the bias the method removes (+1.03), and the tolerance matched accuracy says this is genuine localization, not bias balancing.
  2. Honest treatment of its own mechanism. The information model is frankly called an idealization, not a theorem. The key premise (the goal adds localizing information) is tested with a matched probe, and the probe is reported at p = 0.055 with n = 42, explicitly labeled "directional evidence" instead of spun.
  3. Strong cost and robustness profile. Zero shot, 3.1 calls per sample on average, an order of magnitude cheaper than the thorough readers it beats. Five backbones without collapse. Prompts released verbatim (Appendix B). An ablation without the ground truth answer (62.6/20.7) shows the signal is in the goal structure, not the answer key.
  4. A mature limitations section. It reports where GCCA loses (commissions: 25.0 vs A2P's 37.5), why it is not first on short sets, and that chain faithfulness (81 to 86%) far exceeds exact step accuracy (23%), so localization, not elicitation, is the bottleneck.
Critical weaknesses
  1. The evidence base is smallest where the claim is biggest. Hand Crafted has only 58 trajectories (42 omissions), and 23.0% vs 17.8% is a difference of about three trajectories. Three run means are reported, but per sample overlap matters more than run variance. No paired per sample significance test is given for the main table.
  2. Mechanism evidence is suggestive, not causal. Eq. (3) is asserted, and the only direct test is the small probe at p = 0.055. Chain faithfulness is measured by an LLM judge, is 81.4% only among chains that parse, and drops to about 68% unconditionally on Hand Crafted after an admitted formatting fault. This operational fragility belongs in the main text.
  3. The near zero average bias hides opposite signed per type biases (+5.8 on omissions, minus 11.4 on commissions). On the error type where local evidence methods already work, GCCA both loses accuracy and becomes early biased. The paper should foreground that GCCA is an omission specialist.
  4. A benchmark matched post processing step supplies part of the gain. The deterministic "alignment" that moves orchestrator dispatch predictions to the executing step follows the benchmark's annotation guideline. Removing it cuts When from 23.0 to 18.4, so about a fifth of the method's advantage is guideline matching, not root cause detection.
  5. Single benchmark family. The step level advantage disappears on AFTraj's short trajectories, and there is no evaluation on traces from other agent frameworks (MetaGPT, AutoGen style workloads).
Questions for authors
  1. Please add a per sample paired bootstrap over trajectories for the main comparison: is 70.1/23.0 vs 64.4/17.8 significant on Hand Crafted?
  2. How large would the omission probe need to be to reach p < 0.05 at the observed effect (5.6% to 11.1%)? Can you replicate the probe on AFTraj omissions as an independent source?
  3. How often does the deterministic alignment step fix versus break a correctly localized step? Is it configurable per benchmark, and what happens on a benchmark with different annotation guidelines?
  4. Chains can contain spurious, non necessary conditions (your Example 2, C4 C5). If you ablate spurious conditions before attribution, do predictions change on Hand Crafted?
  5. Reverse test: does withholding the question and keeping only the answer a destroy the effect? Which part of G actually carries the signal?
2 comments

Thank you for the careful review. Each of your five questions led to a new result in the revision. This comment (1/2) covers a correction to the submission and Q1–Q3, with W1, W4 and W5; a second comment covers Q4, Q5, W2 and W3.

Correction (your W2). App. C of the submission disclosed a chain-parsing fault, reported runs made before the fix, and described the accuracies as unaffected. They are not. The fault: markdown in the model's chain output made the parser return a placeholder on 16% of long Hand-Crafted chains. With the fix as the default (one line at each of three parse sites, in the supplementary code), we re-ran every GCCA Hand-Crafted experiment from scratch. No chain degrades (0 of 58), and Hand-Crafted is now Who 67.8 (was 70.1), When 27.0 (23.0), ±5 59.2 (56.3), bias +0.76 (+1.03). All Hand-Crafted GCCA numbers below come from the corrected code; our "Summary of changes" comment gives the details.

Q1, W1 (paired tests for the main comparison). Added as Table 10, with the step alignment applied to every method (see Q3). Step, GCCA 27.0 vs. A2P 21.3 over the 58 trajectories (difference in points; one-sided McNemar on the per-item majority vote; paired-bootstrap 95% CI): exact +5.7, p = .035 [−1.7, 13.2]; ±5 +11.5, p = .018 [3.4, 19.5]; on the 42 omissions, exact +11.1, p = .008 [3.2, 19.8] and ±5 +15.1, p = .011 [5.6, 24.6]. Agent: +3.4 over Step-by-Step [−7.5, 14.4] and +8.0 over A2P [−1.7, 17.8].

Q2 (probe size; replication on AFTraj). Size: at the observed effect, a one-sided p < .05 needs about 50–60 omissions and 80% power about 110–140 (App. C); Hand-Crafted has 42. Replication on Hand-Crafted: a rerun three months later on a different endpoint gave +4.8 on omissions (first run +5.6); pooled over the two, +5.2, 95% CI [−0.8, 12.7], one-sided p = .048.

For AFTraj we did the work the probe needs. AFTraj-Agentic has no omission labels (the 72% in the submission was an automatic estimate, not a human annotation), so two annotators independently labelled all 470 failures, error type and first symptom step, with the Hand-Crafted protocol. The 109 cases without an agreed type were adjudicated under written rules (κ = 0.73 on the 375 cases that carry a reason text), and the labels are in the supplementary material. What they show is that this set cannot test the mechanism. 51.5% of its failures are omissions, but the symptom coincides with the root cause on 80% of failures, and the mean cause–symptom gap is 0.3 steps against 9.2 on Hand-Crafted omissions: an omitted action there is visible at the next step. The probe agrees. The trajectory alone already locates 44.2% of the omission root causes (5.6% on Hand-Crafted), and adding the goal changes that by −3.0 points (95% CI [−7.9, 1.7], n = 242); on commissions it adds +7.9 ([1.3, 14.5], n = 228). The premise of Eq. (1), a locally silent root cause, does not hold on this set, so Eq. (3) predicts no omission-specific gain there. The attribution results are consistent with that: no method has a late bias on it (largest +0.6 steps), and GCCA is on par at the step (52.5 against 53.3 for the best) and first on the agent (80.9). We looked for a second long-trajectory set with root-cause labels and found none; App. F says so.

Q3, W4 (alignment: fix vs. break; configurable; other guidelines). Over GCCA's three runs the rule fires on 20, 19 and 20 of 58 trajectories, corrects 5, 4, 3 predictions and breaks 2, 0, 1 (Table 8 caption). The 18.4 you cite came from a separate run; scored on the same runs, GCCA is 21.8 before the alignment and 27.0 after (Table 4b). The revision applies the same rule to every method (§4.1), and new Table 8 gives both versions: A2P 17.8 → 21.3, ECHO 15.5 → 21.3. GCCA leads before the rule and after it. The rule is configurable (--ablation wo_disambig in the code). It fires only on a step that carries the orchestrator's dispatch marker, so it never fires on Algorithm-Generated or AFTraj; on a benchmark that charges the dispatcher instead of the executor, one turns it off and gets the raw column of Table 8.

W5 (other frameworks). Who&When is itself AutoGen-style: Algorithm-Generated comes from AG2 CaptainAgent teams and Hand-Crafted from Magentic-One, which runs on AutoGen; AFTraj is a third source. We have no MetaGPT traces, and the step-level advantage is specific to long trajectories with delayed symptoms, as Q2 shows.

(2/2) Q4, Q5, W2 and W3.

Q4 (ablating spurious conditions). Predictions do change, for the worse. We added a variant with one extra call that keeps only the conditions the model judges necessary and drops the rest (App. C). On Hand-Crafted: agent 67.8 → 64.9, exact step 27.0 → 20.7, ±5 59.2 → 54.0; item-level paired bootstrap, −6.3 at exact (95% CI [−12.1, −1.1]) and −5.2 at ±5 ([−10.3, −0.6]). Our reading: GCCA blames the first unmet condition, so an extra condition that the trajectory satisfies is never blamed and costs nothing, whereas pruning can only remove conditions, and when it removes the true one the root cause is no longer in the chain.

Q5 (question withheld, answer kept). It weakens the effect and does not destroy it (new row "w/o q" in Table 4b). Hand-Crafted is 62.1 / 20.7 against 67.8 / 27.0 for the full model, ±5 accuracy falls from 59.2 to 48.3, and the late bias rises from +0.76 to +4.8. Withholding the answer instead gives 64.9 / 21.3, with ±5 at 56.9. Both parts matter at the exact step; the question carries more of the signal, since removing it costs more at ±5 and brings part of the late bias back. That is why we describe the signal as living in the goal structure rather than in the answer key.

W2 (mechanism evidence; chain faithfulness). We agree the evidence for Eq. (3) is directional. The revision keeps that wording, calls the probe's accuracy difference an operational proxy for the conditional term, and adds the replication and the power statement (Q2). On faithfulness: the 68% counted the 16% placeholder chains as misses. With the fix no chain is a placeholder (0 of 58), so that share is gone. The 81.4% is an LLM-judge estimate on the chains that parsed, and App. C says so. The dependence on chain quality is stated in the main text as Limitations 3 and 4.

W3 (omission specialist). Agreed, and the revision foregrounds it. The by-error-type table is now in the main text (Table 3), §4.2 has a paragraph titled "The advantage is concentrated on omissions", and the bias paragraph states the split: +5.4 on omissions, −11.3 on commissions. On commissions GCCA is at 29.2 against A2P's 37.5, which is Limitation 5. On omissions it is the most accurate method, 26.2 / 34.1 / 62.7 at exact / ±1 / ±5 against at most 19.8 / 24.6 / 52.4 for any baseline. Its mean absolute error there is 11.7 steps with a median of 4 (A2P 12.4, ECHO 21.7, CHIEF 23.3), so the near-zero aggregate bias is not large errors cancelling.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of changes in the revised PDF

Summary of changes in the revised PDF (changes in blue; main text 10 pages)

We thank all three reviewers. The method and its prompts are unchanged; the only code change is the parser correction of item 1, and every added analysis answers a request in the reviews. Labels: reviewer, weakness W / question Q in the order written.

1. Correction: chain-parsing fault; GCCA Hand-Crafted results regenerated (gnQV-W2, qr32-Q3). App. C of the submission disclosed a formatting fault in the chain parser, reported runs made before the fix, and described the reported accuracies as unaffected. They are not. The fault: when the model wrote its chain with markdown emphasis, the pattern that reads the conditions matched nothing, and attribution fell back to a one-line placeholder condition (16% of long Hand-Crafted chains). The fix strips markdown before parsing, one line at each of three parse sites; it was in the submitted code behind --parsefix and is now the default (--no_parsefix restores the old behaviour). We re-ran every GCCA Hand-Crafted experiment from scratch with the fix (main table, ablations, tolerance, backbones, k-sweep). No chain degrades (0 of 58), and Hand-Crafted becomes Who 70.1→67.8, When 23.0→27.0, ±5 56.3→59.2, bias +1.03→+0.76, tokens 4.7k→4.6k. Every Hand-Crafted GCCA attribution result in the revision comes from the corrected code (abstract, §1, §4, §6, Tables 2–6 and 11, App. C–I). In Table 6 the Qwen backbone is now Qwen3.8-Max, a stronger model than the Qwen3-235B used before.

2. Step alignment applied to every method (ZLQC-W1/Q1/Q2, gnQV-W4/Q3). §4.1 states the convention: Tables 2, 3 and 4a apply the alignment to all methods; Table 1 and Fig. 2 report raw outputs. New Table 8 gives raw vs. aligned step accuracy for all seven methods, with GCCA scored on the same runs before its own alignment (21.8→27.0; A2P 17.8→21.3, ECHO 15.5→21.3). The "w/o align." row of Table 4b is now those same runs. §3.2 describes the rule as a switch: it fires on about 20 of 58 trajectories, corrects 4 and breaks 1 per run (counts in the Table 8 caption), never fires on Algorithm-Generated or AFTraj, and can be turned off.

3. Item-level significance (ZLQC-Q1, gnQV-W1/Q1, qr32-W2/Q1). New Table 10: one-sided McNemar on the per-item majority vote and paired-bootstrap 95% CIs over the 58 trajectories, GCCA vs. A2P, ECHO and CHIEF, for step exact / ±5, agent, and the omission subset. §4.2 and the Table 2 caption report the step results against A2P (p = .035 exact, .018 at ±5, .008 on omissions).

4. Error-type analysis promoted and extended (ZLQC-W2/Q3, gnQV-W3, qr32-W2). Table 3 (localization by human-annotated error type) moves to the main text, with ±1 and MAE columns. New Table 9 shows where omission predictions land relative to the root cause and the symptom: ECHO and CHIEF overshoot the symptom on 32% / 48% of omissions, A2P does not on average, and GCCA is within one step of the root cause most often (34%). §4.2 is rewritten accordingly; the bias paragraph adds MAE and median error.

5. Proposition 1 restated as a lower bound; scope (ZLQC-W2/Q3, qr32-W1/Q2). App. A: r^≥rm\hat r \ge r_m and E[r^−R⋆]≥μpropE[\hat r - R^\star] \ge \mu_{prop}, with equality for a reader that stops at the first symptom; "verified empirically" becomes "direction tested empirically". Abstract and §1 scope the effect to attributors that read the whole trajectory, and "removes" becomes "reduces". §2.3 and App. C call the probe's accuracy difference an operational proxy for the conditional term.

6. Second human annotation: AFTraj-Agentic (gnQV-W5/Q2, qr32-W2). App. F: all 470 failures annotated with the same two-rater protocol (51.5% omissions; κ = 0.73 on the 375 with a reason text; the 109 without an agreed type adjudicated under written rules), replacing the submission's 72%, an automatic estimate. The symptom coincides with the root cause on 80% of failures (mean gap 0.3 steps vs. 9.2 on Hand-Crafted omissions), so the set is a short-trajectory analogue of the target regime. GCCA, re-run with the corrected code: 80.9 / 52.5. Labels are in the supplementary material.

7. Probe (gnQV-Q2, qr32-W1). App. C: a replication of the Eq. (3) probe three months later (+4.8 on omissions, same sign reversal on commissions) and a power statement (p < .05 needs about 50–60 omissions, 80% power about 110–140).

8. Non-necessary conditions (gnQV-Q4, qr32-Q3). App. C: a pruning variant that keeps only necessary conditions: Hand-Crafted 67.8 / 27.0 / 59.2 → 64.9 / 20.7 / 54.0, with item-level CIs and the reason.

9. Question-withheld ablation (gnQV-Q5). Table 4b row "w/o q" (Hand-Crafted 62.1 / 20.7, ±5 48.3) and §4.3.

10. Corrections. §4.4 now matches Fig. 4 (ZLQC, minor). App. E explains the low DeepSeek agent cell on Algorithm-Generated. App. H: k text corrected.

Supplementary material: the parser fix is on by default; new scripts reproduce Tables 2–4, 8–10 and the probe; the AFTraj-Agentic labels are added.

AuthorsOpenReview ↗

Closing summary

We thank the Area Chair and the reviewers. As the discussion closes, here is where each main request stands, with pointers into the revision. All Hand-Crafted GCCA numbers come from the corrected code (item 1 of our "Summary of changes").

  • Reviewer ZLQC. Q1 is the request the review names as most likely to change the score. It is in the revision as asked: the alignment is applied to the cached outputs of every baseline, and Hand-Crafted step accuracy is then 27.0 for GCCA against 21.3 for A2P and ECHO (Table 8). Item-level McNemar against A2P gives p = .035 at the exact step, .018 at ±5 and .008 on omissions; Table 10 has the intervals and the agent tests. For W2/Q3, Proposition 1 is restated as a lower bound, following the reviewer's own calculation (App. A, Table 9).
  • Reviewer gnQV. Each of Q1–Q5 has a new result: paired tests (Table 10); a probe replication and a two-annotator labelling of all 470 AFTraj-Agentic failures (App. C, F); alignment counts on the same runs (Table 8); a pruning variant (App. C); a question-withheld ablation (Table 4b).
  • Reviewer qr32. The first-round points (paired tests, error cancellation, the model's assumptions, incorrect preconditions) are answered in Tables 3, 9 and 10 and App. A, C and F. The follow-up raised three design questions (omissions as root causes, graph-structured preconditions, the routing threshold); our second reply answers each.

We would be grateful if these could be taken into account in the discussion that follows.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.