acceptodds
Under review as a conference paper at ICLR 2027

From Anomalies to Failures: Constructing Causal Error Graphs for Agentic Trace Diagnosis

Abstract

LLM-driven agents are increasingly deployed in complex applications, where long agentic traces make failures difficult to diagnose. Existing trace diagnosis methods often conflate anomalies, errors, and failures, making diagnostic targets ambiguous; they also lack structured modeling of how causally relevant errors propagate and amplify into final task failures, resulting in unreliable failure attribution. To address these problems, we propose **CEG-Agent**, a tool-augmented agentic framework for causal diagnosis of agentic traces. Specifically, CEG-Agent introduces an explicit taxonomy of anomalies, errors, and failures, and constructs **Causal Error Graphs** (CEGs), a unified typed representation that links execution events, diagnostic nodes, and failure outcomes through causal relations. To evaluate causal trace diagnosis, we further construct **CEG-Bench**, a fully agent-annotated benchmark with high-confidence, consensus-derived CEG annotations obtained through an **Adversarial Agentic Adjudication Protocol** (AAAP). We validate the resulting annotations against an expert-curated human gold set, which shows close agreement with the automatic annotations. Experiments on CEG-Bench demonstrate that CEG-Agent achieves state-of-the-art performance under both semantically relaxed and structurally exact evaluation criteria. Our code is publicly available.

Primary area:
datasets and benchmarks
Keywords:
Causal Error Graphs, Agentic Trace Diagnosis, Adversarial Agentic Adjudication Protocol, Anomalies, Errors, Failures

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RocyOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies failure diagnosis for long agentic traces. The authors divide diagnostic signals into three levels, namely anomalies, errors, and failures, and on this basis define the Causal Error Graph (CEG), which uses typed edges to connect execution events, process-level errors, and final failures. Building on this, they propose CEG-Agent, a tool-augmented diagnosis framework that incrementally constructs, validates, and revises a CEG while estimating a repair value for each error. For evaluation, the authors build CEG-Bench, a set of 100 traces annotated through AAAP, an adversarial multi-agent annotation protocol, and additionally have 8 experts annotate a human gold set, CEG-Bench-Gold, to check the reliability of the automatic annotations. Experiments show that CEG-Agent outperforms same-backbone Claude Code and Codex baselines under both the semantically relaxed (SRE) and structurally exact (SEE) evaluations.

Critical strengths
  1. Separating surface anomalies, causally relevant errors, and final failures is a reasonable modeling direction with practical value for debugging.

  2. The benchmark and the framework fit together well: they share the same schema, and the evaluation covers both the semantic and the structural level.

  3. The authors verify the automatic annotations against a human gold set. With the human gold set as the reference, CEG-Agent still achieves the highest composite score within both backbones, and its improvement over the strongest baseline (SRE +7.3 and SEE +3.1 for CEG-Claude, Tables 10/11) is even larger than under the automatic reference. This alleviates the concern that the results depend on the automatically

Critical weaknesses

W1. The paper does not show that the causal structure, its central contribution, is reliably annotated, recovered, or useful.

  1. The benchmark is least reliable on the causal edges. Against the expert gold set, AAAP reaches only 20.1 on SEE causes and 36.2 on contributes_to (Table 9), and no inter-annotator agreement among the experts is reported, so the claim of "strong agreement" in the main text is overstated. Against the same gold set, every system scores between 0.5 and 2.3 on SEE causes (Table 11), so improvements in causal wiring cannot be confirmed.
  2. In the main tables, CEG-Claude improves over Claude Code (+Process) by only +2.0 SRE and +1.2 SEE (Tables 1 and 2), and no ablation removes the critic, the sub-agent, anomaly detection, or repair-value estimation, so the contribution of each module is unknown.
  3. The repair hints in Section 5.4 are derived from the AAAP annotations rather than from CEG-Agent's predictions, and none of the three CEG hints contains causal edges (Appendix F), so the experiment tests neither the predicted graphs nor the graph structure. On Tau2, the best hint is the one that states only the failure (91.7 vs. 77.8 for the generic retry). On SWE, the priority-error hint (8.6) falls below the generic retry (14.3), which contradicts the claim that CEG hints "consistently win". Each source has only 29 to 36 traces, with a single attempt per trace.

W2. The formalization, another main contribution, is partly inconsistent and hard to follow.

  1. The legal tuples Ω in Eq. (2) do not include (A, ATTACHED-TO, E), although the text states that anomalies are grounded in events through such edges. Section 3.3.1 and Appendix A.5 claim that the projection Π guarantees W1∧W2∧W3, but none of the five rules in Appendix A.4 makes an unreachable error reach a failure, and A.4 itself classifies this case as a semantic issue (I_sem). Under Eq. (8), the numerator excludes r while the denominator includes it, so σ can never reach 1, which contradicts the statement that σ = 1 marks a unique bottleneck, and σ is always 0 when there is only one error.
  2. Section 3 introduces many symbols (ε(r), μ(r), ν(r), ρ(r), d(r), θ(f), Ω, Reach, W1–W4) without a running example, does not explain how CAUSES and AMPLIFIES are distinguished in practice, and only names the 8 mechanism categories and 6 failure types without defining them. The structural role ρ(r) and the repair value ρ♯(r) are written almost identically, and K denotes both the agent's iteration budget and the number of AAAP rounds.
Questions for authors
  1. How does CEG-Agent differ from DoVer (arXiv:2512.06749), which verifies failure hypotheses through interventions, CORRECT (arXiv:2509.24088), which identifies decisive errors in multi-agent systems, and A2P (arXiv:2509.10401), which formulates failure attribution as abduction, intervention, and prediction? Since Section 5 states that no like-for-like external baseline exists, could at least one of them be compared on the error dimensions of SRE and SEE?
  2. Could you run the repair experiment with CEG-Agent's predicted graphs and, keeping the error nodes and their descriptions fixed, compare hints with and without explicit causal edges, using more than one attempt per trace?
  3. What is the inter-annotator agreement among the eight experts before the consensus discussion, and could the experts see the AAAP annotations while annotating, given that Appendix D describes their task as to "annotate and verify" the traces?
  4. Which rule in Π guarantees W3 when an error cannot reach any failure, and should the numerator of Eq. (8) include r?
  5. Given that the repair-value tool and the number of critic rounds can already be configured (CE_ENABLE_REPAIR and CE_CRITIC_ROUNDS, Appendix A.2), how do the results change when the critic, sub-agent delegation, anomaly detection, or repair-value estimation is removed?
9 comments
AuthorsOpenReview ↗

Response to Reviewer Rocy (W1)

Weakness 1: Reliability, recovery, and usefulness of causal structure

Thank you for the detailed comments. We agree that exact causal wiring is the hardest part of the task, but we do not think the current results imply that causal structure is unreliable or unrecovered.

First, the 20.1 and 36.2 numbers come from the strictest exact-matching dimensions, SEE causes and SEE contributes to. These require exact matching of error nodes by event grounding and mechanism, followed by exact edge multiset matching. A single mismatch in grounding, mechanism label, or edge endpoint makes an otherwise similar causal relation count as incorrect. Under the semantic evaluator, agreement is substantially higher: against the expert-audited human reference, AAAP reaches 71.5 overall CEG Sim, with 76.3 error agreement, 76.3 failure agreement, 88.1 anomaly agreement, 58.0 causal-edge agreement, and 61.9 main-chain agreement. SEE agreement also increases with trace complexity, from 43.0 on small traces to 45.8 on medium traces and 52.4 on large traces.

Thus, exact edge reconstruction is the least reliable dimension, which is why we report both SRE and SEE. We will avoid the overly broad phrase “strong agreement” and state the result more precisely: high agreement on node-level and semantic diagnostic content, moderate agreement on causal chains, and low agreement on exact edge tuples.

Second, although SEE causes is low for all systems against the expert reference, causal-structure recovery is supported by more discriminative dimensions. CEG-Agent improves on SRE main chain, which measures root-to-failure propagation recovery, as well as SRE causal edge and SEE attached to, which measure causal structure and grounding. It also improves propagated-role recall, which requires identifying downstream errors together with their upstream causes. Exact SEE causes is therefore an important difficulty indicator, but not the only evidence for causal reconstruction.

Third, the main-table CEG-Claude gain over Claude Code plus Process is the least favorable comparison. It is measured against the AAAP reference and diluted by the 15 gold-empty traces, where abstaining systems are mechanically favored. On the non-empty subset, CEG-Claude improves over Claude Code plus Process by 5.6 SRE and 1.4 SEE. Against the expert-audited human reference, the same contrast widens to 7.3 SRE and 3.1 SEE. The GPT-5.5 backbone shows larger gains: 13.6 SRE and 6.5 SEE against AAAP, and 14.2 SRE and 9.1 SEE against the expert reference.

We also ran leave-one-module-out ablations using the existing switches for critic rounds and repair value, plus registration switches for anomaly detection and sub-agent delegation. This directional pilot was run on CEG-Bench with GLM-4.6 at temperature 0:

ConfigSRESEEerror F1main chainΔSREΔSEE
full0.1980.3440.3560.133
no critic0.1580.2520.2220.067-0.040-0.092
no sub-agent0.1620.2670.2440.067-0.036-0.077
no anomaly0.1220.2140.1110.000-0.076-0.130
no repair0.1210.1750.1110.000-0.077-0.169

This pilot is small, so we treat the magnitudes as directional rather than definitive. Still, the full system is best on both composite metrics, and removing the critic, sub-agent, anomaly detection, or repair-value tool lowers performance. These tools affect the whole agentic construction trajectory, not only their nominal output fields.

Finally, on downstream usefulness, we added experiments using CEG-Agent’s own predicted graphs as feedback, rather than only AAAP annotations:

CEG Source Used as FeedbackFailureErrorError-HM
AAAP CEG91.780.683.3
CEG-Agent predicted86.186.183.3
Claude Code predicted86.186.180.6
Codex predicted77.886.180.6

CEG-Agent predicted feedback outperforms the generic retry baseline of 77.8 under all three projections, showing that downstream repair is not limited to oracle AAAP annotations.

We agree that the repair results should be stated carefully. On Tau2Bench, failure-only feedback is strongest, 91.7 vs. 77.8 for generic retry. This does not contradict CEG, because the failure node is part of the CEG, and Tau2 is an interactive task where naming the outcome-level failure gives the agent a clear correction target. On SWE-bench, priority-error filtering hurts, 8.6 vs. 14.3. However, concise CEG hints still help: failure and error feedback both reach 17.1. The drop is specific to aggressive repair-value filtering in single-shot code patching, where filtering can remove contextual errors needed for a complete patch.

We therefore scope the claim as follows: CEGs provide useful diagnostic feedback, but the best projection depends on the downstream setting.

AuthorsOpenReview ↗

Response to Reviewer Rocy (W2)

Weakness 2: Formalization inconsistencies and readability

Thank you for the careful comments. We have clarified the formalization and improved the presentation in the revised version.

First, regarding anomaly grounding, our original design de-emphasized anomaly-to-event edges because they are non-causal provenance links, not part of the causal propagation structure. Anomaly grounding mainly indicates where an anomaly is observed in the trace, and can be represented either as node-level grounding metadata or as a non-causal ATTACHED_TO edge from an anomaly node A to an event node E. Since these links do not affect causal reachability, repair-value computation, or causal-edge evaluation, they were not emphasized in the main causal edge set. To improve schema clarity, we now explicitly include optional non-causal anomaly grounding edges in the legal edge-type set and clarify that they are used for grounding and visualization.

Second, the deterministic projection operator is intended to guarantee the mechanical validity of the graph, including legal edge types, deduplication, event-next edges, attached-to grounding, and role-topology consistency. Failure reachability is different: it is a semantic validity condition, not something that should be repaired by blindly inserting a causal edge. If an error cannot reach any failure, the validator and critic loop treat it as an invalid retained causal-error hypothesis, and the graph must remove, rewrite, or regenerate it. We have clarified that deterministic projection enforces the mechanical parts of the schema, while failure reachability is enforced through semantic validation and regeneration.

Third, we thank the reviewer for identifying the issue in the repair-value severance score. The denominator should exclude the target error rr, because rr is deleted in the counterfactual graph and should be excluded from both numerator and denominator. If rr is the sole contributor to the failure, the severance score is defined as 1. With this correction, a score of 1 correctly indicates that rr is a unique bottleneck.

Fourth, we thank the reviewer for the suggestion on readability. We have added a running example through the method section. The example instantiates event nodes, anomaly nodes, error nodes, failure nodes, grounding attributes, typed edges, reachability, and well-formedness checks, making the notation easier to follow.

Fifth, we have added operational definitions for CAUSES and AMPLIFIES. A CAUSES edge means that the upstream error enables the downstream error to occur: if the upstream error were removed, the downstream error would likely disappear. An AMPLIFIES edge means that the downstream error would still exist, but the upstream error increases its severity, likelihood, or downstream impact.

Sixth, we have supplemented the vocabulary definitions for the eight error mechanisms and six failure types.

  • Error mechanisms
MechanismDefinition
RepresentationThe agent forms an incorrect or incomplete representation of the task, state, user request, environment, or evidence.
PlanningThe agent chooses an inappropriate strategy, decomposes the task incorrectly, or follows an invalid plan.
ExecutionThe agent fails while carrying out an otherwise plausible plan, such as using a tool incorrectly or making an implementation mistake.
Evidence integrationThe agent observes relevant evidence but fails to integrate, compare, or use it correctly.
ControlThe agent mismanages the interaction process, such as looping, stopping too early, taking the wrong branch, or failing to recover.
Self-evaluationThe agent incorrectly judges its own progress, answer quality, or whether the task has been solved.
OmissionThe agent skips a necessary step, constraint, check, tool call, or piece of information.
EnvironmentThe failure is affected by external conditions such as tool errors, unavailable resources, or environment mismatch.
  • Failure types
Failure TypeDefinition
CompletionThe agent fails to complete the task or stops before producing the required outcome.
CorrectnessThe final answer or action is factually, logically, or functionally incorrect.
ConstraintThe agent violates an explicit instruction, format requirement, policy, or task constraint.
EfficiencyThe agent completes or partially completes the task but with unnecessary cost, delay, or inefficient behavior.
SafetyThe agent produces or attempts unsafe, harmful, or policy-violating behavior.
OtherThe failure does not fit cleanly into the above categories or combines multiple failure modes.

Finally, we have removed the notation collisions by renaming the repair-value symbol to distinguish it from the structural-role symbol, and by renaming the AAAP round budget to avoid confusion with the CEG-Agent inference budget.

AuthorsOpenReview ↗

Response to Reviewer Rocy (Q1/2)

Q1: Difference from DoVer, CORRECT, and A2P, and comparison on shared dimensions

DoVer, CORRECT, and A2P are closely related, but they target a different diagnostic object. DoVer verifies failure hypotheses through interventions; A2P formulates failure attribution through abduction, action, and prediction; and CORRECT identifies decisive errors in multi-agent settings. These methods typically output a decisive step, responsible agent, or failure hypothesis. In contrast, CEG-Agent constructs a full typed graph over a long trace, explicitly separating anomalies, process-level errors, and outcome-level failures, while also reconstructing error propagation and failure attribution.

Since these methods do not output CEGs, a full schema-level comparison is not well-defined. We therefore compare on a shared subtask: error-step localization and root-error top-1. Using official prompts from Who and When and A2P, we obtain:

SystemPRF1root@1
Who and When All-at-Once, official prompt, GLM-4.60.500.070.120.30
A2P Abduct-Act-Predict, official prompt, GLM-4.60.400.080.120.40
Claude Code flat, Claude0.140.090.110.20
Codex flat, GPT0.900.330.420.50
CEG-Agent, Claude0.440.560.480.60

The fairest metric for single-step attribution methods is root@1, because they output one decisive step by design. CEG-Agent achieves the best root@1 score, 0.60, compared with 0.40 for A2P and 0.30 for Who and When. CEG-Agent also achieves the best F1 score, 0.48. Codex flat has higher precision, 0.90, but much lower recall, 0.33, indicating conservative under-generation.

Thus, while DoVer, A2P, and CORRECT are not like-for-like CEG baselines, shared sub-dimension comparison is possible and supports the advantage of CEG-Agent on root-error localization.


Q2: Predicted graphs and the role of causal edges in repair

We added a new Tau2Bench repair condition that explicitly includes causal edges. In this setting, we keep the error feedback format comparable to the flat error-list condition, but additionally provide the causal relations among the errors. The results are:

Feedback HintTau2Bench
Failure feedback91.7
Flat error feedback80.6
Error + causal-edge feedback88.9
High/medium repair-error feedback83.3

Adding explicit causal edges improves repair over the flat error list, 88.9 vs. 80.6, and also remains well above the generic retry baseline of 77.8. This suggests that graph topology can provide repair-relevant information beyond simply listing mistakes.

At the same time, the best projection still depends on the task. On Tau2Bench, failure-only feedback remains strongest, 91.7, likely because Tau2 is an interactive task where naming the outcome-level failure gives the agent a clear correction target. This does not undermine the graph design: the failure node is itself part of the CEG, and the new edge condition shows that causal structure improves over flat error feedback.

We also ran predicted-feedback experiments using CEG-Agent’s own predicted graphs, rather than only AAAP annotations. CEG-Agent predicted feedback reaches 86.1 with failure feedback, 86.1 with error feedback, and 83.3 with high/medium repair-error feedback, all above the generic retry baseline of 77.8. Thus, the repair effect is not limited to oracle AAAP annotations.

We therefore state the claim more precisely: CEG-derived feedback improves downstream repair, and explicit causal edges can improve over flat error lists, although the best feedback projection is task-dependent. Graph topology is useful both as repair-relevant structure and as a diagnostic representation: it makes propagation chains readable, distinguishes upstream causes from downstream symptoms, supports repair-value computation, and provides a basis for future model-side use such as diagnosis-guided training and credit assignment.

Reviewer_kTRzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that debugging long LLM agent trajectories is difficult because existing tools either flag a single decisive step, or dump flat error categories, which mixes up harmless quirks with actual mistakes that snowball into task failure. To fix this, the authors build a tool-calling diagnostic agent that reads an execution log and constructs a directed causal graph separating harmless surface behaviors, intermediate process mistakes, and final task failures, backed by rule-based schema checks and an LLM critic. They test this on a new benchmark of 100 failed agent trajectories across web browsing, tool use, and coding, where reference graphs are created through multi-agent debate and validated against human expert annotations.

Critical strengths
  1. The idea of separating harmless execution issues from cascading mistakes is interesting.
Critical weaknesses
  1. The contribution feels like an incremental repackaging of TraceSIR dressed up in unnecessarily heavy mathematical notation, while bypassing external benchmarks and baselines. The paper reuses the exact 150-trace candidate pool, the deterministic trace parser, and even the analyzer agent from TraceSIR, yet never compares against baseline attribution methods or evaluates on established step-localization benchmarks like Who&When. Instead, simple engineering steps like deduplicating edges and checking node types are framed as constrained MAP estimation over a well-formedness manifold with limit-based projection operators. Testing this schema-enforcing tool loop only against general coding CLIs on its own custom JSON schema makes the main tables predictable and offers very little scientific insight.
  2. The experimental section mainly just report metric numbers, and does not provide any insights into why it works. The downstream retry study fails to justify the complex graph design. Most of the evaluation just breaks down two custom matching metrics across backbones and trace lengths. When the paper finally tests whether these graphs actually help fix failed runs, simply telling the agent the high-level task failure beats giving it the process errors on Tau2Bench, and filtering by the proposed repair priority scores actually hurts coding performance on SWE-bench compared to a generic retry prompt.
Questions for authors
  1. Even if prior methods or step-scoring baselines do not output your full graph schema, they still identify root-cause steps and error categories. Why not evaluate your method against existing baselines on step-localization benchmarks like Who&When, or compare root-error detection accuracy directly against other baseline methods?
  2. In the retry experiments in Section 5.4, you only feed the agent the outcome failure or a flat list of error descriptions, without the causal edges. Does the graph topology itself provide any measurable benefit for agent repair over a flat list of mistakes?
  3. Why did you hide the ground truth task outcome from the annotator agents during benchmark creation, which caused 15 out of 100 known failed traces to have empty reference graphs and turned part of the evaluation into an abstention test?
4 comments
AuthorsOpenReview ↗

Response to Reviewer kTRz (W1)

Weakness 1: TraceSIR repackaging, heavy notation, and lack of external baselines

Thank you for the careful comments. We respectfully disagree that CEG-Agent is a repackaging of TraceSIR.

TraceSIR produces a structured but essentially flat diagnostic report. In contrast, CEG-Agent changes the diagnostic object itself: it constructs a typed causal graph over events, anomalies, errors, and failures, with explicit propagation, amplification, and contributes-to relations. The benchmark target is also different: CEG-Bench annotates full causal error graphs, not only trace reports or error categories.

Reusing the 150-trace candidate pool and deterministic parser is deliberate. It controls for data and preprocessing, ensuring that performance differences come from causal diagnosis rather than from a different trace source or parser. The TraceSIR analyzer is used only as an independent reviewer inside AAAP, not as the final annotator and not as a component of CEG-Agent. Final annotations must still pass adversarial cross-examination by two heterogeneous agentic annotators and are further audited against CEG-Bench-Gold.

The formalization is intended to make the CEG object and evaluation target precise. We do not view implementation details such as edge deduplication or type checking as the core contribution; they are safeguards for schema validity. The main technical and empirical challenge is identifying causally relevant errors and reconstructing their propagation structure.

Regarding the concern that the main tables compare only against generic coding CLIs, we added a targeted sub-dimension comparison with attribution baselines that do not output full CEGs but do output root or decisive steps. Since these methods are not schema-compatible with CEG, we evaluate the shared subtask: error-step localization and root-error top-1.

SystemPRF1root@1
Who and When All-at-Once, official prompt, GLM-4.60.500.070.120.30
A2P Abduct-Act-Predict, official prompt, GLM-4.60.400.080.120.40
Claude Code flat, Claude0.140.090.110.20
Codex flat, GPT0.900.330.420.50
CEG-Agent, Claude0.440.560.480.60

This comparison uses the official prompts from the corresponding repositories and maps each predicted decisive step to CEG-Bench event IDs. The fairest comparison with single-point attribution methods is root@1, because they emit one decisive step by design. CEG-Agent obtains the best root@1 score, 0.60, compared with 0.40 for A2P and 0.30 for Who and When. CEG-Agent also obtains the best F1 score, 0.48.

Codex flat has the highest precision, 0.90, because it is highly conservative and predicts few error steps. However, its recall is much lower, 0.33, and its root@1 is also below CEG-Agent, 0.50 vs. 0.60. This indicates that Codex often identifies a correct step when it fires, but misses many relevant errors. CEG-Agent provides a better precision-recall balance and stronger root-error localization.

We also verified the step-localization harness on an external TRAIL-GAIA subset using TRAIL span labels. On the five smallest traces, it obtains 0.67/0.80/0.70 P/R/F1 and 0.60 decisive-hits-HIGH. This is a small sanity check rather than a full CEG evaluation, but it shows that the sub-dimension comparison can be instantiated on established step-localization annotations.

Finally, the main scientific insight is not only that CEG-Agent scores higher, but also why it scores higher. CEG-Agent better matches the scale of the gold graphs, producing about 4.6 errors and 6.9 causal edges per trace, close to the gold averages of 3.9 errors and 7.5 causal edges. In contrast, baselines tend to under-generate errors, anomalies, and causal edges. The gains concentrate on hard mechanisms such as representation, evidence integration, and planning, as well as on propagated errors that require causal-chain reconstruction. CEG-Agent also remains stronger on non-empty substantive traces: CEG-Claude improves over the same-backbone baseline by 5.6 SRE and 1.4 SEE, while CEG-Codex improves by 13.0 SRE and 6.2 SEE.

AuthorsOpenReview ↗

Response to Reviewer kTRz (W2)

Weakness 2: Experiments lack insight and the retry study does not justify graph design

The experiments are designed to separate several distinct questions: semantic diagnosis, exact graph reconstruction, graph-size calibration, mechanism and role recall, trace complexity, robustness against human gold, and downstream repair. The two main metrics are not arbitrary duplicates. SRE measures semantic diagnostic correctness under natural-language variation, while SEE measures strict graph reconstruction. Their divergence is informative: some backbones produce fluent descriptions but weaker exact structure, while others recover exact edges better despite lower semantic similarity.

The “why it works” evidence is visible in the fine-grained analyses. CEG-Agent improves most on main-chain recovery, causal-edge recovery, anomaly detection, and exact error grounding. Baselines often fail by under-generating graph structure, especially anomalies and causal edges. CEG-Agent also improves on propagated errors, which are precisely the cases where a flat local-step diagnosis is insufficient.

For downstream retry, the results support a more precise claim: CEGs provide useful diagnostic feedback, but the best projection depends on the task. On Tau2Bench, all compact CEG hints outperform generic retry: failure feedback reaches 91.7, error feedback reaches 80.6, and priority-error feedback reaches 83.3, compared with 77.8 for generic retry. The fact that failure-only feedback is strongest on Tau2 does not undermine CEG; the failure node is part of the CEG, and Tau2 is an interactive task where naming the outcome failure gives the agent a clear correction target.

On SWE-bench, the reviewer is right that priority-error filtering hurts, 8.6 vs. 14.3 for generic retry. The important detail is that concise CEG hints still help: failure and error feedback both reach 17.1, above generic. The drop is specific to aggressive repair-value filtering in single-shot code patching, where removing contextual errors can make the patch incomplete or brittle. This suggests that repair-value calibration and feedback formatting matter, not that CEG diagnosis lacks downstream value.

We also ran new feedback experiments using automatically predicted CEGs, rather than only AAAP annotations:

CEG Source Used as FeedbackFailureErrorError-HM
AAAP CEG91.780.683.3
CEG-Agent predicted86.186.183.3
Claude Code predicted86.186.180.6
Codex predicted77.886.180.6

CEG-Agent predicted feedback outperforms generic retry at 77.8 under all three projections. This shows that the repair effect is not limited to oracle annotations.

We do not claim that this table isolates the marginal repair effect of causal edges alone. In this work, graph topology is primarily valuable as a human-readable and visualizable diagnostic structure: it shows which errors are upstream causes, which are downstream symptoms, and how they contribute to the final failure. It also provides a structural basis for future model-side use, such as diagnosis-guided training or credit assignment, where models can exploit not only which steps are wrong but also how errors propagate.

As additional evidence that fine-grained diagnosis is useful beyond retry, we also tested CEG diagnosis-driven training. We compiled localized diagnoses into token-level supervision for a DeepSeek-R1-Distill-Qwen-1.5B student on MATH-500:

Settingmean@16maj@16
Base student78.181.1
Diagnosis-driven training80.786.9
Gain+2.6+5.8

Fine-grained localization also reduces wrongly penalized correct tokens by about 66–70% compared with coarse span localization. This is not the main evidence for edge-level causal correctness, but it shows the potential value of event-grounded diagnosis as a credit-assignment signal for model optimization.

AuthorsOpenReview ↗

Response to Reviewer kTRz (Q1/2/3)

Q1: Why not compare against step-localization or root-cause baselines?

We address this by reducing both CEG-Agent and prior attribution methods to a shared subtask: error-step localization and root-error top-1. The full results are reported under Weakness 1. Briefly, using official prompts from Who and When and A2P, CEG-Agent obtains the best root@1 score, 0.60, compared with 0.40 for A2P and 0.30 for Who and When. CEG-Agent also obtains the best F1, 0.48. Codex flat has higher precision, 0.90, but much lower recall, 0.33, showing a conservative under-generation pattern.

A full evaluation on Who and When is not directly compatible with CEG-Agent because Who and When is a multi-agent responsible-agent plus decisive-step benchmark, while CEG-Bench is a long-trace typed graph construction benchmark. The shared sub-dimension comparison is therefore the appropriate evaluation: it compares root-error localization without forcing prior methods to emit a graph schema they were not designed for.


Q2: Does graph topology itself improve repair over a flat list?

The new predicted-feedback experiments show that CEG-derived information improves repair over generic feedback. On Tau2Bench, a flat predicted error list from CEG-Agent reaches 86.1, compared with 77.8 for generic retry. Structured CEG projections are also useful: failure-only feedback reaches 86.1, and high/medium repair-error feedback reaches 83.3.

We do not overstate this as proof that causal edges alone improve repair. The current feedback experiments test CEG-derived slices, not a controlled edge-only toggle. In this paper, graph topology is primarily valuable as a diagnostic structure: it makes propagation chains readable, distinguishes upstream causes from downstream symptoms, supports repair-value computation, and gives humans a visualizable explanation of failure formation. It also provides a basis for future model-side use, such as diagnosis-guided training and credit assignment.

Thus, the current evidence supports the usefulness of CEG-derived feedback for repair, while the strongest demonstrated value of topology is interpretability, visualizable propagation structure, and a foundation for future model-side credit assignment.


Q3: Why hide the ground-truth task outcome during annotation?

We did not provide the oracle task outcome to annotator agents because CEG-Bench is designed to evaluate diagnosis from the trace itself, which is the setting available in most real debugging deployments. If annotator agents were given the oracle final outcome or evaluator feedback, the benchmark would depend on information unavailable to a deployed trace diagnoser, and the annotation process would be less reproducible outside benchmark environments.

The cost is that 15 out of 100 traces receive empty CEGs because no causal error can be confirmed from the trace alone. We explicitly analyze this effect rather than hiding it. On the non-empty subset of 85 traces, where substantive causal diagnosis is required, CEG-Agent’s advantage becomes clearer: CEG-Claude improves over Claude Code plus Process by 5.6 SRE and 1.4 SEE, and CEG-Codex improves over Codex plus Process by 13.0 SRE and 6.2 SEE.

This shows that the main results are not driven by an abstention test. The gold-empty subset is reported separately precisely to make this tradeoff transparent.

Thanks for the new experiments, I will keep my score.

Reviewer_t9EuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CEG-Agent, a framework for diagnosing failures in LLM-driven agent traces by constructing Causal Error Graphs (CEGs). The framework distinguishes anomalies, process-level errors, and task-level failures, and represents their relationships using typed graph structures. The authors also introduce CEG-Bench, a benchmark of 100 failed agent traces annotated through a multi-agent adjudication protocol.

Critical strengths
  • The paper addresses an important problem. Understanding how errors emerge and propagate through long agentic executions is valuable for debugging and improving LLM agents.
  • Distinguishing anomalies, errors, and failures provides a useful conceptual organization for trace diagnosis and also the framing as a graph.
Critical weaknesses
  • Insufficient positioning against closely related work and no comparison to existing baselines. The related-work discussion overlooks several highly relevant approaches, including GraphTracer (2025), AgentTrace (2026), AgentRx (2026), and REFLECT (2026). In particular, GraphTracer already investigates error propagation and root-cause attribution using information-dependency graphs, while AgentTrace reconstructs causal graphs from execution logs for root-cause analysis. AgentRx provides evidence-grounded failure diagnosis, and REFLECT evaluates failure hypotheses through interventions. These works overlap substantially with the motivation and technical contributions of CEG-Agent. The authors should clarify the distinctions, establish what is genuinely novel about their causal graph formulation, and provide appropriate comparisons or justify why such comparisons are infeasible. Even for the baselines shown, the empirical improvements are relatively modest for the strongest baseline in the paper. How to vouch for the effectiveness of the framework?

Relevant papers:

  • GraphTracer: https://arxiv.org/abs/2510.10581

  • AgentTrace: https://arxiv.org/abs/2603.14688

  • AgentRx: https://arxiv.org/abs/2602.02475

  • DoVer: https://arxiv.org/abs/2512.06749

  • TRAIL: https://arxiv.org/abs/2505.08638

  • Tokengeist: https://arxiv.org/abs/2607.22610

  • The causal claims are insufficiently justified. The framework infers causal relationships primarily through LLM judgments over observational traces. It remains unclear how the proposed method distinguishes genuine causal dependencies from plausible explanatory relationships. The paper would benefit from intervention-based or counterfactual validation of the inferred causal edges.

  • The reliability of the benchmark annotations is questionable. Although the authors use adversarial multi-agent consensus and human validation, the exact agreement with human annotations is only 20.1% for causal edges and 36.2% for error-to-failure relationships in the lowest setting.

  • Comparison with "flat" baselines: Do the automatically predicted CEGs improve downstream repair outcomes, and how do they compare with much simpler error summaries?

Questions for authors

Please see weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer t9Eu (W1)

Weakness 1: Insufficient positioning against related work and no comparison to existing baselines.

Thank you for pointing out these closely related works. They share our high-level goal of failure attribution, but differ in the diagnostic object being constructed. Many prior methods focus on decisive-step or responsible-agent attribution, producing a culprit step, responsible agent, or flat error description. Graph-based methods such as GraphTracer and AgentTrace are closer, but mainly model information dependency or log-level causal structure. In contrast, CEG-Agent constructs a typed diagnostic graph over events, anomalies, process-level errors, and outcome-level failures. Its edges encode diagnostic causal relations such as causes, amplifies, and contributes to, rather than only information or log dependency. Tokengeist is related as a token-level analysis method, but does not target typed trace-level causal graph construction.

MethodOutput UnitAEF Sep.GraphProp.GroundRepairInterv.BenchViz
Who and WhenResponsible agent and decisive step✗✗✗✓✗✗✓✗
TRAILSpan-level error list✗✗✗✓✗✗✓✗
TraceSIRStructured trace diagnosis report✗◐✗✓✗✗◐✗
CORRECTDecisive error via schema transfer✗✗✗✓✗◐✓✗
A2PDecisive step via abduction, action, and prediction✗✗✗✓✗✓✗✗
DoVerFailure hypothesis with intervention verification✗✗✗✓◐✓✗✗
REFLECTIntervention-based error attribution✗✗✗✓✗✓◐✗
GraphTracerRoot-cause node on information-dependency graph✗✓◐◐✗✗✗✓
AgentTraceCausal graph from execution logs✗✓◐◐✗✗◐✓
AgentRxEvidence-grounded failure diagnosis✗✗✗✓✗✗◐✗
CEG-AgentTyped causal error graph✓✓✓✓✓◐✓✓

Legend: ✓ supported, ◐ partially supported, ✗ not supported.
AEF Sep. means explicit anomaly/error/failure separation. Prop. means error propagation or amplification. Repair means repair signal or prioritization. Interv. means intervention or counterfactual validation. CEG-Agent is partial on Interv. because it uses graph-structural counterfactuals and closed-loop repair, but not per-edge executed interventions.

More concretely, GraphTracer builds an information-dependency graph to localize root-cause nodes and dependency paths. CEG differs in three ways: its nodes are typed as events, anomalies, errors, and failures; its edges encode diagnostic causal relations with mechanism, role, and failure-type vocabularies; and it additionally separates harmless anomalies from causally relevant errors and estimates repair value.

AgentTrace also reconstructs causal graphs from logs, but CEG turns root-cause analysis into a typed, evaluable, and repair-prioritizable diagnostic object. It attaches event grounding, mechanism and role labels, failure attribution, anomaly nodes, and repair-value estimates within the same graph.

AgentRx and REFLECT are complementary. AgentRx emphasizes evidence-grounded failure diagnosis but does not construct typed propagation graphs. REFLECT, DoVer, and A2P provide stronger executed intervention or counterfactual validation. CEG-Agent instead focuses on typed graph construction from long traces, with structural counterfactual scoring and closed-loop repair as indirect validation. Per-edge executed intervention validation remains future work.

Thus, the novelty of CEG-Agent is not simply using a graph, but combining event grounding, anomaly/error/failure separation, typed error propagation and amplification, failure attribution, repair-value estimation, graph benchmark evaluation, and visualization in one schema. A full like-for-like comparison with decisive-step methods is not possible because they do not output CEGs, but their outputs can still be compared on subdimensions such as error localization and root-error detection.

Regarding the modest gains over the strongest baseline, the CEG-Claude versus Claude Code plus Process result is the least favorable setting. It is measured against the AAAP reference and diluted by 15 gold-empty traces, where abstaining systems receive a mechanical advantage. On the non-empty subset, CEG-Claude improves over Claude Code plus Process by 5.6 SRE and 1.4 SEE. Against the independent human gold reference, the gap widens to 7.3 SRE and 3.1 SEE. The GPT-5.5 backbone shows larger gains: 13.6 SRE and 6.5 SEE against AAAP, and 14.2 SRE and 9.1 SEE against the human reference. These results suggest that CEG-Agent is most useful when substantive causal structure must be recovered, rather than when the trace admits an empty or near-empty diagnosis.

AuthorsOpenReview ↗

Response to Reviewer t9Eu (W2/3)

Weakness 2: The causal claims are insufficiently justified.

The causal edges are inferred from failed traces, so we do not claim that every edge is experimentally verified causal truth. Our claim is more specific: CEG constrains causal attribution beyond free-form plausible explanation.

First, every causal relation must be grounded in concrete execution events and must connect process-level errors to outcome-level failures through typed edges. This rules out explanations that sound plausible but are not tied to the actual trace.

Second, repair value is based on a graph-structural counterfactual. For an error node ee, the severance score removes ee from the causal graph and checks whether reachability to the primary failure is cut. Thus, bottleneck errors are not identified only by natural-language judgment, but by an explicit counterfactual operation on the graph.

Third, the closed-loop repair study provides behavioral evidence that CEG-derived diagnostic information is actionable. On Tau2Bench, all CEG hints outperform generic retry: failure-level feedback reaches 91.7, error feedback reaches 80.6, and priority-error feedback reaches 83.3, compared with 77.8 for generic retry. On BrowseComp, error-based CEG hints improve performance from 6.9 to 17.2. On SWE-bench, concise failure and error hints improve from 14.3 to 17.1, although priority filtering drops to 8.6.

These results do not validate each individual causal edge, but they show that the diagnosed failures and errors can change downstream outcomes. We will make this scope explicit: CEG-Agent provides trace-grounded causal graph inference, graph-structural counterfactual scoring, and closed-loop behavioral evidence, but not per-edge executed intervention validation. Per-edge interventions, in the spirit of DoVer, A2P, and REFLECT, are complementary and remain an important future extension.


Weakness 3: The reliability of the benchmark annotations is questionable.

The 20.1 and 36.2 numbers correspond to the strictest exact-matching dimensions, namely SEE causes and SEE contributes to. These are the hardest dimensions because they require exact matching of error nodes followed by exact edge multiset matching. A single disagreement in event grounding, mechanism label, or edge endpoint makes an otherwise similar causal relation count as incorrect.

The broader agreement picture is stronger. Against the expert-curated human gold set, AAAP reaches 71.5 overall CEG Sim under SRE, with 76.3 error agreement, 76.3 failure agreement, and 88.1 anomaly agreement. Causal structure is also much higher under the semantic criterion than under exact tuple matching: SRE causal-edge agreement is 58.0, and main-chain agreement is 61.9. SEE agreement further increases with trace complexity, from 43.0 on small traces to 45.8 on medium traces and 52.4 on large traces.

These numbers do not imply that the annotations are unreliable. Rather, they show that exact causal wiring is the hardest part of the task. We will avoid the overly broad phrase “strong agreement” and report reliability more precisely: high agreement on semantic diagnostic content and node types, moderate agreement on causal chains, and low agreement on exact edge reconstruction. This is also why we report both SRE and SEE instead of relying only on exact edge scores.

AuthorsOpenReview ↗

Response to Reviewer t9Eu (W4)

Weakness 4: Comparison with flat baselines and downstream repair.

We ran new downstream experiments using automatically predicted CEGs as feedback. On Tau2Bench, CEG-Agent predicted feedback reaches 86.1 with failure feedback, 86.1 with error feedback, and 83.3 with high/medium repair-error feedback, all above the generic retry baseline of 77.8. This shows that the repair gain is not limited to AAAP oracle annotations.

CEG Source Used as FeedbackFailureErrorError-HM
AAAP CEG91.780.683.3
CEG-Agent predicted86.186.183.3
Claude Code predicted86.186.180.6
Codex predicted77.886.180.6

This also compares CEGs with simpler summaries. A flat predicted error list already improves over generic feedback, 86.1 vs. 77.8. CEG additionally supports structured projections: failure-only feedback also reaches 86.1, and repair-filtered errors reach 83.3. With AAAP CEGs, repair filtering improves over the full error list, 83.3 vs. 80.6. With predicted CEGs, the full error list is slightly better, 86.1 vs. 83.3, indicating that predicted repair-value calibration can still improve.

Thus, the supported claim is that CEGs provide useful repair feedback and multiple actionable projections, rather than merely a verbose error dump.

We do not claim that this table isolates the marginal repair effect of causal edges alone. In this work, graph topology is primarily valuable as a human-readable and visualizable diagnostic structure: it shows which errors are upstream causes, which are downstream symptoms, and how they contribute to the final failure. It also provides a structural basis for future model-side use, such as diagnosis-guided training or credit assignment, where models can exploit not only which steps are wrong but also how errors propagate. A strict edge-only repair ablation would require holding error text fixed and toggling only the edges.

Beyond retry, fine-grained CEG diagnosis can also drive training optimization. In a MATH-500 diagnosis-driven training study, token-level supervision from localized diagnoses improves a DeepSeek-R1-Distill-Qwen-1.5B student:

Settingmean@16maj@16
Base student78.181.1
Diagnosis-driven training80.786.9
Gain+2.6+5.8

Fine-grained localization also reduces wrongly penalized correct tokens by about 66–70% compared with coarse span localization, supporting the value of event-grounded diagnosis for credit assignment. This confirms the potential future value of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.