- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes EvoCause, an LLM-guided framework for refining causal graphs using expert-labeled historical incidents to improve root cause analysis (RCA). The method first constructs an initial alarm propagation graph using existing causal discovery algorithms. It then iteratively compares graph-based root cause predictions with expert annotations and employs an LLM to propose edge additions, removals, or reversals. These modifications undergo deterministic structural validation, and the best-performing graph is selected based on RCA accuracy over a labeled alignment set. During inference, root causes are identified directly from the refined graph without additional LLM calls.
The paper also introduces TeleRCA, a production telecommunications dataset containing 10,922 expert-annotated incidents and 485,681 alarm events. Experiments on synthetic data and TeleRCA report improvements in RCA performance across different causal discovery backbones. Additional analyses examine graph reconstruction quality, LLM configurations, and the contribution of alarm-name semantics.
Critical strengths
- Valuable real-world data: TeleRCA contains a large number of expert-annotated incidents from a production telecom network, addressing the scarcity of realistic RCA datasets.
- Clear problem formulation: The paper explicitly characterizes root-cause labels as indirect constraints and acknowledges that they cannot uniquely identify the underlying causal graph.
- Practical framework design: LLM-guided graph refinement is combined with deterministic validation and selection, enabling interpretable RCA without LLM calls at inference time.
- Meaningful experimental analysis: Experiments across different causal discovery methods demonstrate promising improvements, while synthetic graph reconstruction and alarm-title anonymization studies provide additional insights into the method.
Critical weaknesses
-
Missing dataset and appendices: Although the paper claims to release TeleRCA, no accessible dataset link is provided. The repeatedly referenced appendices are also missing, limiting reproducibility and verification of experimental details.
-
Unclear necessity of LLMs: The paper lacks comparisons with simpler non-LLM graph refinement strategies using the same expert feedback, making it difficult to isolate the actual contribution of LLM-guided editing.
-
Restrictive graph assumptions: A single stationary alarm-type DAG may not capture resource-specific or context-dependent causal relationships. Moreover, disconnected incident subgraphs can produce multiple spurious root cause predictions, but graph connectivity is not analyzed.
-
Limited causal validity: Optimizing root cause prediction accuracy does not necessarily recover correct causal relationships. The real-world experiments lack ground-truth causal graphs to validate the learned structures.
-
Limited generalization evaluation: The real-world evaluation relies on random incident-level splits from a single telecom network, without testing temporal shifts, unseen resources, or cross-system generalization.
-
Limited evaluation on external datasets: Although the paper discusses existing RCA datasets, EvoCause is evaluated only on the authors' synthetic data and TeleRCA, without validation on independent real-world datasets. Combined with the lack of a publicly accessible TeleRCA release, this limits reproducibility and makes it difficult to assess the method's generalizability across different systems.
-
Insufficient ablation studies: Although the paper evaluates different LLM backbones and alarm-name anonymization, the contributions of key components, such as batch-based feedback, optimization history, and graph selection, are not sufficiently examined in the provided manuscript. The missing appendices further prevent verification of the claimed component ablations.
Questions for authors
-
Dataset availability: When will TeleRCA and the missing appendices be made available? Could the authors provide details on dataset construction, expert annotation procedures, and preprocessing?
-
Necessity of LLMs: How does EvoCause compare with simpler non-LLM graph refinement methods that optimize the same objective using expert labels? What additional benefit does LLM-guided editing provide?
-
Graph connectivity: How does EvoCause handle disconnected incident subgraphs or isolated alarm nodes, which may introduce spurious root cause predictions? Could the authors report connectivity statistics for the learned graphs?
-
Generalizability: Why is EvoCause not evaluated on existing external RCA datasets discussed in the paper? Could the authors provide additional experiments using independent datasets or temporal splits to demonstrate generalization?
-
Causal validity: Since the refinement objective optimizes RCA accuracy rather than causal correctness, how can the authors assess whether the modified edges represent meaningful causal relationships rather than overfitting to expert annotations?
-
Ablation studies: Could the authors provide component-level ablations to quantify the contributions of batch-based feedback, optimization history, and graph selection? How sensitive is the method to batch size and the number of refinement rounds?