- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies failure diagnosis for long agentic traces. The authors divide diagnostic signals into three levels, namely anomalies, errors, and failures, and on this basis define the Causal Error Graph (CEG), which uses typed edges to connect execution events, process-level errors, and final failures. Building on this, they propose CEG-Agent, a tool-augmented diagnosis framework that incrementally constructs, validates, and revises a CEG while estimating a repair value for each error. For evaluation, the authors build CEG-Bench, a set of 100 traces annotated through AAAP, an adversarial multi-agent annotation protocol, and additionally have 8 experts annotate a human gold set, CEG-Bench-Gold, to check the reliability of the automatic annotations. Experiments show that CEG-Agent outperforms same-backbone Claude Code and Codex baselines under both the semantically relaxed (SRE) and structurally exact (SEE) evaluations.
Critical strengths
-
Separating surface anomalies, causally relevant errors, and final failures is a reasonable modeling direction with practical value for debugging.
-
The benchmark and the framework fit together well: they share the same schema, and the evaluation covers both the semantic and the structural level.
-
The authors verify the automatic annotations against a human gold set. With the human gold set as the reference, CEG-Agent still achieves the highest composite score within both backbones, and its improvement over the strongest baseline (SRE +7.3 and SEE +3.1 for CEG-Claude, Tables 10/11) is even larger than under the automatic reference. This alleviates the concern that the results depend on the automatically
Critical weaknesses
W1. The paper does not show that the causal structure, its central contribution, is reliably annotated, recovered, or useful.
- The benchmark is least reliable on the causal edges. Against the expert gold set, AAAP reaches only 20.1 on SEE
causesand 36.2 oncontributes_to(Table 9), and no inter-annotator agreement among the experts is reported, so the claim of "strong agreement" in the main text is overstated. Against the same gold set, every system scores between 0.5 and 2.3 on SEEcauses(Table 11), so improvements in causal wiring cannot be confirmed. - In the main tables, CEG-Claude improves over Claude Code (+Process) by only +2.0 SRE and +1.2 SEE (Tables 1 and 2), and no ablation removes the critic, the sub-agent, anomaly detection, or repair-value estimation, so the contribution of each module is unknown.
- The repair hints in Section 5.4 are derived from the AAAP annotations rather than from CEG-Agent's predictions, and none of the three CEG hints contains causal edges (Appendix F), so the experiment tests neither the predicted graphs nor the graph structure. On Tau2, the best hint is the one that states only the failure (91.7 vs. 77.8 for the generic retry). On SWE, the priority-error hint (8.6) falls below the generic retry (14.3), which contradicts the claim that CEG hints "consistently win". Each source has only 29 to 36 traces, with a single attempt per trace.
W2. The formalization, another main contribution, is partly inconsistent and hard to follow.
- The legal tuples Ω in Eq. (2) do not include (A, ATTACHED-TO, E), although the text states that anomalies are grounded in events through such edges. Section 3.3.1 and Appendix A.5 claim that the projection Π guarantees W1∧W2∧W3, but none of the five rules in Appendix A.4 makes an unreachable error reach a failure, and A.4 itself classifies this case as a semantic issue (I_sem). Under Eq. (8), the numerator excludes r while the denominator includes it, so σ can never reach 1, which contradicts the statement that σ = 1 marks a unique bottleneck, and σ is always 0 when there is only one error.
- Section 3 introduces many symbols (ε(r), μ(r), ν(r), ρ(r), d(r), θ(f), Ω, Reach, W1–W4) without a running example, does not explain how CAUSES and AMPLIFIES are distinguished in practice, and only names the 8 mechanism categories and 6 failure types without defining them. The structural role ρ(r) and the repair value ρ♯(r) are written almost identically, and K denotes both the agent's iteration budget and the number of AAAP rounds.
Questions for authors
- How does CEG-Agent differ from DoVer (arXiv:2512.06749), which verifies failure hypotheses through interventions, CORRECT (arXiv:2509.24088), which identifies decisive errors in multi-agent systems, and A2P (arXiv:2509.10401), which formulates failure attribution as abduction, intervention, and prediction? Since Section 5 states that no like-for-like external baseline exists, could at least one of them be compared on the error dimensions of SRE and SEE?
- Could you run the repair experiment with CEG-Agent's predicted graphs and, keeping the error nodes and their descriptions fixed, compare hints with and without explicit causal edges, using more than one attempt per trace?
- What is the inter-annotator agreement among the eight experts before the consensus discussion, and could the experts see the AAAP annotations while annotating, given that Appendix D describes their task as to "annotate and verify" the traces?
- Which rule in Π guarantees W3 when an error cannot reach any failure, and should the numerator of Eq. (8) include r?
- Given that the repair-value tool and the number of critic rounds can already be configured (CE_ENABLE_REPAIR and CE_CRITIC_ROUNDS, Appendix A.2), how do the results change when the critic, sub-agent delegation, anomaly detection, or repair-value estimation is removed?
9 comments
Response to Reviewer Rocy (W2)
Weakness 2: Formalization inconsistencies and readability
Thank you for the careful comments. We have clarified the formalization and improved the presentation in the revised version.
First, regarding anomaly grounding, our original design de-emphasized anomaly-to-event edges because they are non-causal provenance links, not part of the causal propagation structure. Anomaly grounding mainly indicates where an anomaly is observed in the trace, and can be represented either as node-level grounding metadata or as a non-causal ATTACHED_TO edge from an anomaly node A to an event node E. Since these links do not affect causal reachability, repair-value computation, or causal-edge evaluation, they were not emphasized in the main causal edge set. To improve schema clarity, we now explicitly include optional non-causal anomaly grounding edges in the legal edge-type set and clarify that they are used for grounding and visualization.
Second, the deterministic projection operator is intended to guarantee the mechanical validity of the graph, including legal edge types, deduplication, event-next edges, attached-to grounding, and role-topology consistency. Failure reachability is different: it is a semantic validity condition, not something that should be repaired by blindly inserting a causal edge. If an error cannot reach any failure, the validator and critic loop treat it as an invalid retained causal-error hypothesis, and the graph must remove, rewrite, or regenerate it. We have clarified that deterministic projection enforces the mechanical parts of the schema, while failure reachability is enforced through semantic validation and regeneration.
Third, we thank the reviewer for identifying the issue in the repair-value severance score. The denominator should exclude the target error , because is deleted in the counterfactual graph and should be excluded from both numerator and denominator. If is the sole contributor to the failure, the severance score is defined as 1. With this correction, a score of 1 correctly indicates that is a unique bottleneck.
Fourth, we thank the reviewer for the suggestion on readability. We have added a running example through the method section. The example instantiates event nodes, anomaly nodes, error nodes, failure nodes, grounding attributes, typed edges, reachability, and well-formedness checks, making the notation easier to follow.
Fifth, we have added operational definitions for CAUSES and AMPLIFIES. A CAUSES edge means that the upstream error enables the downstream error to occur: if the upstream error were removed, the downstream error would likely disappear. An AMPLIFIES edge means that the downstream error would still exist, but the upstream error increases its severity, likelihood, or downstream impact.
Sixth, we have supplemented the vocabulary definitions for the eight error mechanisms and six failure types.
- Error mechanisms
| Mechanism | Definition |
|---|---|
| Representation | The agent forms an incorrect or incomplete representation of the task, state, user request, environment, or evidence. |
| Planning | The agent chooses an inappropriate strategy, decomposes the task incorrectly, or follows an invalid plan. |
| Execution | The agent fails while carrying out an otherwise plausible plan, such as using a tool incorrectly or making an implementation mistake. |
| Evidence integration | The agent observes relevant evidence but fails to integrate, compare, or use it correctly. |
| Control | The agent mismanages the interaction process, such as looping, stopping too early, taking the wrong branch, or failing to recover. |
| Self-evaluation | The agent incorrectly judges its own progress, answer quality, or whether the task has been solved. |
| Omission | The agent skips a necessary step, constraint, check, tool call, or piece of information. |
| Environment | The failure is affected by external conditions such as tool errors, unavailable resources, or environment mismatch. |
- Failure types
| Failure Type | Definition |
|---|---|
| Completion | The agent fails to complete the task or stops before producing the required outcome. |
| Correctness | The final answer or action is factually, logically, or functionally incorrect. |
| Constraint | The agent violates an explicit instruction, format requirement, policy, or task constraint. |
| Efficiency | The agent completes or partially completes the task but with unnecessary cost, delay, or inefficient behavior. |
| Safety | The agent produces or attempts unsafe, harmful, or policy-violating behavior. |
| Other | The failure does not fit cleanly into the above categories or combines multiple failure modes. |
Finally, we have removed the notation collisions by renaming the repair-value symbol to distinguish it from the structural-role symbol, and by renaming the AAAP round budget to avoid confusion with the CEG-Agent inference budget.
Response to Reviewer Rocy (Q1/2)
Q1: Difference from DoVer, CORRECT, and A2P, and comparison on shared dimensions
DoVer, CORRECT, and A2P are closely related, but they target a different diagnostic object. DoVer verifies failure hypotheses through interventions; A2P formulates failure attribution through abduction, action, and prediction; and CORRECT identifies decisive errors in multi-agent settings. These methods typically output a decisive step, responsible agent, or failure hypothesis. In contrast, CEG-Agent constructs a full typed graph over a long trace, explicitly separating anomalies, process-level errors, and outcome-level failures, while also reconstructing error propagation and failure attribution.
Since these methods do not output CEGs, a full schema-level comparison is not well-defined. We therefore compare on a shared subtask: error-step localization and root-error top-1. Using official prompts from Who and When and A2P, we obtain:
| System | P | R | F1 | root@1 |
|---|---|---|---|---|
| Who and When All-at-Once, official prompt, GLM-4.6 | 0.50 | 0.07 | 0.12 | 0.30 |
| A2P Abduct-Act-Predict, official prompt, GLM-4.6 | 0.40 | 0.08 | 0.12 | 0.40 |
| Claude Code flat, Claude | 0.14 | 0.09 | 0.11 | 0.20 |
| Codex flat, GPT | 0.90 | 0.33 | 0.42 | 0.50 |
| CEG-Agent, Claude | 0.44 | 0.56 | 0.48 | 0.60 |
The fairest metric for single-step attribution methods is root@1, because they output one decisive step by design. CEG-Agent achieves the best root@1 score, 0.60, compared with 0.40 for A2P and 0.30 for Who and When. CEG-Agent also achieves the best F1 score, 0.48. Codex flat has higher precision, 0.90, but much lower recall, 0.33, indicating conservative under-generation.
Thus, while DoVer, A2P, and CORRECT are not like-for-like CEG baselines, shared sub-dimension comparison is possible and supports the advantage of CEG-Agent on root-error localization.
Q2: Predicted graphs and the role of causal edges in repair
We added a new Tau2Bench repair condition that explicitly includes causal edges. In this setting, we keep the error feedback format comparable to the flat error-list condition, but additionally provide the causal relations among the errors. The results are:
| Feedback Hint | Tau2Bench |
|---|---|
| Failure feedback | 91.7 |
| Flat error feedback | 80.6 |
| Error + causal-edge feedback | 88.9 |
| High/medium repair-error feedback | 83.3 |
Adding explicit causal edges improves repair over the flat error list, 88.9 vs. 80.6, and also remains well above the generic retry baseline of 77.8. This suggests that graph topology can provide repair-relevant information beyond simply listing mistakes.
At the same time, the best projection still depends on the task. On Tau2Bench, failure-only feedback remains strongest, 91.7, likely because Tau2 is an interactive task where naming the outcome-level failure gives the agent a clear correction target. This does not undermine the graph design: the failure node is itself part of the CEG, and the new edge condition shows that causal structure improves over flat error feedback.
We also ran predicted-feedback experiments using CEG-Agent’s own predicted graphs, rather than only AAAP annotations. CEG-Agent predicted feedback reaches 86.1 with failure feedback, 86.1 with error feedback, and 83.3 with high/medium repair-error feedback, all above the generic retry baseline of 77.8. Thus, the repair effect is not limited to oracle AAAP annotations.
We therefore state the claim more precisely: CEG-derived feedback improves downstream repair, and explicit causal edges can improve over flat error lists, although the best feedback projection is task-dependent. Graph topology is useful both as repair-relevant structure and as a diagnostic representation: it makes propagation chains readable, distinguishes upstream causes from downstream symptoms, supports repair-value computation, and provides a basis for future model-side use such as diagnosis-guided training and credit assignment.
Response to Reviewer Rocy (W1)
Weakness 1: Reliability, recovery, and usefulness of causal structure
Thank you for the detailed comments. We agree that exact causal wiring is the hardest part of the task, but we do not think the current results imply that causal structure is unreliable or unrecovered.
First, the 20.1 and 36.2 numbers come from the strictest exact-matching dimensions, SEE causes and SEE contributes to. These require exact matching of error nodes by event grounding and mechanism, followed by exact edge multiset matching. A single mismatch in grounding, mechanism label, or edge endpoint makes an otherwise similar causal relation count as incorrect. Under the semantic evaluator, agreement is substantially higher: against the expert-audited human reference, AAAP reaches 71.5 overall CEG Sim, with 76.3 error agreement, 76.3 failure agreement, 88.1 anomaly agreement, 58.0 causal-edge agreement, and 61.9 main-chain agreement. SEE agreement also increases with trace complexity, from 43.0 on small traces to 45.8 on medium traces and 52.4 on large traces.
Thus, exact edge reconstruction is the least reliable dimension, which is why we report both SRE and SEE. We will avoid the overly broad phrase “strong agreement” and state the result more precisely: high agreement on node-level and semantic diagnostic content, moderate agreement on causal chains, and low agreement on exact edge tuples.
Second, although SEE causes is low for all systems against the expert reference, causal-structure recovery is supported by more discriminative dimensions. CEG-Agent improves on SRE main chain, which measures root-to-failure propagation recovery, as well as SRE causal edge and SEE attached to, which measure causal structure and grounding. It also improves propagated-role recall, which requires identifying downstream errors together with their upstream causes. Exact SEE causes is therefore an important difficulty indicator, but not the only evidence for causal reconstruction.
Third, the main-table CEG-Claude gain over Claude Code plus Process is the least favorable comparison. It is measured against the AAAP reference and diluted by the 15 gold-empty traces, where abstaining systems are mechanically favored. On the non-empty subset, CEG-Claude improves over Claude Code plus Process by 5.6 SRE and 1.4 SEE. Against the expert-audited human reference, the same contrast widens to 7.3 SRE and 3.1 SEE. The GPT-5.5 backbone shows larger gains: 13.6 SRE and 6.5 SEE against AAAP, and 14.2 SRE and 9.1 SEE against the expert reference.
We also ran leave-one-module-out ablations using the existing switches for critic rounds and repair value, plus registration switches for anomaly detection and sub-agent delegation. This directional pilot was run on CEG-Bench with GLM-4.6 at temperature 0:
This pilot is small, so we treat the magnitudes as directional rather than definitive. Still, the full system is best on both composite metrics, and removing the critic, sub-agent, anomaly detection, or repair-value tool lowers performance. These tools affect the whole agentic construction trajectory, not only their nominal output fields.
Finally, on downstream usefulness, we added experiments using CEG-Agent’s own predicted graphs as feedback, rather than only AAAP annotations:
CEG-Agent predicted feedback outperforms the generic retry baseline of 77.8 under all three projections, showing that downstream repair is not limited to oracle AAAP annotations.
We agree that the repair results should be stated carefully. On Tau2Bench, failure-only feedback is strongest, 91.7 vs. 77.8 for generic retry. This does not contradict CEG, because the failure node is part of the CEG, and Tau2 is an interactive task where naming the outcome-level failure gives the agent a clear correction target. On SWE-bench, priority-error filtering hurts, 8.6 vs. 14.3. However, concise CEG hints still help: failure and error feedback both reach 17.1. The drop is specific to aggressive repair-value filtering in single-shot code patching, where filtering can remove contextual errors needed for a complete patch.
We therefore scope the claim as follows: CEGs provide useful diagnostic feedback, but the best projection depends on the downstream setting.