acceptodds
Under review as a conference paper at ICLR 2027

AVERT: A Tool-Verified Agent with Responsibility-Routed Rule Evolution for Telecom Root Cause Analysis

Abstract

Telecom root cause analysis (RCA) is critical for service restoration and repair planning during multi-site outages. LLM agents struggle because evidence is scattered across long alarm streams and topology graphs, candidate causes may lack evidential support, and free-form reasoning is difficult to verify. To address these challenges, we present AVERT, an RCA agent that follows an iterative hypothesis–diagnosis–verification workflow. Given the incident evidence, AVERT first forms an initial root-cause hypothesis. The Rule-Constraint Layer then structures the diagnosis into three stages: candidate enumeration, causal and topological constraint checking, and verdict selection, producing a provisional diagnosis. The Verification-Feedback Layer tests this diagnosis against service-path impact and feeds blocked paths, surviving sites, and counterexamples back to revise the hypothesis and rerun the rule stages. To keep the rule layer adaptive as incident patterns evolve, we introduce offline Responsibility-Routed Rule Evolution, which uses evaluator feedback and execution traces to identify the responsible stage and update only the corresponding rules. We evaluate AVERT on TelcoOutageBench, comprising 131 real-world telecom incidents. Compared with frontier model baselines, AVERT improves the mean RCA Score by 2.71–13.81 percentage points across all eight backbones, including Claude Opus 4.6, GPT-5.4, and GLM-5.1.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Root Cause Analysis, Telecom Networks, Tool-Augmented Reasoning, Rule Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.60 over 5 reviews, above 10% of the others.
Reviewer_NTPqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AVERT, an agent for telecom root cause analysis (RCA), with two main components. The first is a tool-assisted diagnostic loop. The authors design three tools to assist the LLM in root cause analysis. In particular, diagnose tool checks candidate hypotheses by simulating the impact of hypothesized node or link failures on service paths and returns feedback for revising the diagnosis. The second is a rule-constraint layer that can be optimized offline. Three sets of rules in the prompt guide the LLM through candidate enumeration, constraint checking, and final root-cause set selection. A stronger model analyzes execution traces and failure patterns to iteratively revise these rules offline. On the private TelcoOutageBench dataset, which contains 131 real-world incidents, AVERT achieves higher RCA Scores than direct prompting baseline across all eight backbones. Ablation experiments further support the contributions of the core components.

Critical strengths
  1. Topology-based deterministic hypothesis verification has practical value. Relying solely on an LLM to interpret the topology and reason about and verify fault hypotheses can lead to errors. The paper uses a deterministic tool to simulate the impact of hypothesized failures on service paths, providing more accurate external feedback for diagnosis and helping filter out incorrect candidate hypotheses.
Critical weaknesses
  1. The method figures are closer to conceptual illustrations than precise framework diagrams. They are overly simplified and may have been generated directly using AI tools without sufficient refinement. Their details contain inaccuracies and ambiguous meanings. High-quality method diagrams are essential for readers to understand the system’s components and how they interact. These presentation issues obscure the proposed workflow, contributing to my recommendation to reject.

  2. The experimental evidence is insufficient to support the conclusions. Evaluation on a single private dataset restricts independent reproducibility and leaves generalization untested. Additional, appropriately adapted evaluations on public benchmarks such as TN-RCA530 would help address this concern. Without broader experimental validation, the method’s effectiveness beyond the evaluated dataset remains unclear. Also need to compare with existing RCA agents.

Questions for authors
  1. To reduce the context burden of the raw alarm table, the paper initially provides the LLM with a processed alarm table. What preprocessing rules are applied, and what criteria determine which raw alarm records are retained?

  2. How does your work compare with existing RCA agents? Any experiments.

Reviewer_5SqSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes AVERT, a framework for root cause analysis (RCA) in telecom networks. AVERT contributes an iterative hypothesis–diagnosis–verification agentic workflow, and the optimization of the RCA harness via offline in-context learning. The AVERT framework consists of a tool layer, a verification-feedback layer that validates predicted causes against simulated outcomes, and a policy layer that constrain via rules expressed in natural language how a model enumerates new hypothesis, excludes them, and issues the final verdict. AVERT improves the accuracy of RCA compared to vanilla prompting and a deterministic baseline implemented by the authors.

Critical strengths
  • None
Critical weaknesses
  1. Limited novelty and underspecified problem formulation. AVERT primarily combines existing tool-use and prompt-engineering techniques without establishing a sufficiently novel methodological contribution. The verification-feedback mechanism relies on a custom offline verifier that maps candidate root causes to blocked service paths and affected sites. However, the paper does not adequately specify the assumptions underlying this mechanism, particularly whether network configurations are considered or whether verification relies solely on topology. The formulation also leaves unclear whether AVERT supports failures that do not manifest as path disruptions, such as transient packet loss and gray failures. These limitations make it difficult to assess the framework's generality and technical contribution. The paper also insufficiently positions itself against existing work on network fault localization and LLM-based troubleshooting.

  2. Insufficient evaluation and weak baselines. AVERT is evaluated on a proprietary, purpose-built benchmark whose representativeness and complexity are insufficiently characterized, despite the availability of public network RCA benchmarks. The evaluation relies on a direct-prompting baseline without equivalent tool access and a handcrafted deterministic baseline, rather than established network fault-localization methods or modern general-purpose agent harnesses. Furthermore, the rule-evolution mechanism is not compared against existing harness-optimization approaches, nor is its offline optimization cost adequately accounted for. Consequently, the experiments do not convincingly establish the significance of AVERT's improvements or its advantages over existing alternatives.

Questions for authors
  • [Clarify] Why the authors have not considered open-source network RCA benchmark such as NIKA, NetOpsBench, Cornetto, FauIT-Bench, CTBench? Can the authors provide evidence that AVERT generalizes to unseen network types or failure modes?
  • [Clarify] How many network topologies are present in TelcoOutageBench?
  • [Clarify] Can the authors classify the complexity of the tasks in the dataset, having access to the ground truth? If feasible, can the authors provide a breakdown of the performance (e.g., Tab. 2, Tab. 3, Fig. 4) by task complexity? On this benchmark, small models appear to perform quite well, so it is unclear whether the reported improvements are significant.
  • [Clarify] How does AVERT compare with general-purpose coding agents equipped with equivalent RCA tools (e.g., Codex, Claude Code, OpenCode)? What capabilities of AVERT cannot be reproduced by memory-first harnesses such as Letta or mem0?
  • [Clarify] Why authors have implemented a custom the graph-based network fault-localization methods, and not compared with baselines such as NetDiagnoser (CoNEXT 2007) or network tomography methods?
  • [Clarify] Did the authors come across any of the following works?
  • [Clarify] Does the offline verifier account for network configurations and routing behavior, or does it rely solely on topology? How does AVERT handle failures that do not manifest as complete service-path disruptions (e.g., gray failures or transient packet loss)? Why was a custom verifier preferred over established network verification tools such as Batfish?
  • [Future work suggestion] Can the authors quantify how well AVERT harness can generalize across RCA benchmarks? What happens for unseen network types and failure modes?
  • [Future work suggestion] AVERT uses tokes during the offline process of rule evolution to optimize the AVERT harness. An alternative is to spend the same tokens at inference time for test-time scaling. How many tokens are required for offline rule evolution? How does its cost-performance trade-off compare with spending an equivalent compute budget at inference time, for example by sampling and verifying multiple candidate solutions? Does the evolved harness generalize sufficiently across tasks to amortize its offline optimization cost?
Reviewer_2RJGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces AVERT, an LLM agent for multi-cause RCA for telecommunications networks.

AVERT combines targeted alarm retrieval, service-path reconstruction, and deterministic topology-impact verification with a diagnostic policy organized into Enumeration, Constraint, and Verdict layers (EL/CL/VL). An offline optimization procedure uses evaluator feedback and execution traces to identify a responsible layer and restrict rule updates to that layer. Experiments use 131 real incidents, with 51 for rule evolution and 80 for testing. The complete system improves RCA Score over backbone-matched direct prompting across eight models.

Critical strengths
  1. A practically meaningful and well-defined task in the real world. The paper distinguishes affected sites from actual repair locations and explicitly handles multiple independent root causes. The structured output and matching-aware evaluation are better aligned with operational diagnosis than a single-label accuracy metric

  2. Integration of executable feedback. The verifier returns surviving paths and unexplained sites, providing concrete evidence for hypothesis revision. This is a stronger feedback mechanism than normal self-reflection, and the cumulative ablations show improvements from adding verification beyond evidence-access tools..

Critical weaknesses
  1. Limited methodological novelty. AVERT primarily integrates established techniques: tool-augmented reasoning, executable verification, and feedback-driven prompt optimization. All of them are common in agent systems, the authors just applied them within a predefined EL/CL/VL framework for telecom RCA. The contribution therefore appears to be a domain-specific integration of existing methods, with insufficient evidence that the framework offers a generalizable methodological advance over prior RCA agents and modular prompt optimizers.

2.The distinctive contribution of responsibility routing is not sufficiently isolated. Figure 4 compares monolithic rewriting, unrestricted EL/CL/VL editing, and responsibility-routed editing, but the evolution comparison is reported for a single run. These conditions also differ in their editing constraints. Consequently, the results do not clearly establish whether the benefit comes from accurate responsibility attribution or simply from restricting the scope of each update.

  1. The scope of topology verification is narrower than some of the main-text framing suggests. The paper acknowledges that topology consistency is necessary but insufficient, but it should more clearly distinguish structural consistency from causal correctness and quantify the effect of incomplete path reconstruction.

  2. The evidence for stable improvement and broader generalization remains limited. Evaluation uses 80 incidents from one industry partner, while the evolution procedure repeatedly optimizes against 51 training incidents. This scale is not reliable enough.

Questions for authors

RCAgent (Wang et al., CIKM 2024) has already used autonomous tool calls, log/code experts, context management, and trajectory-level self-consistency to complete RCA, so there is a precedent for "LLM calling tools for fault diagnosis" itself. In addition, I believe there are many pre-defined frameworks for analyzing RCA tasks in detail. So how does AVERT advance beyond RCAgent, which already supports autonomous tool-augmented RCA? Could you clarify the methodological differences, particularly in topology verification and responsibility-routed rule evolution, and compare against an adapted RCAgent baseline using the same backbone, evidence access, and tool budget?

Reviewer_NfDeOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AVERT, an LLM agent for telecom root cause analysis with multiple cause lables. AVERT combines three CLI tools (alarm filtering, service-path restoration, and a deterministic topology-impact verifier), there rules (EL/CL/VL), and an evolving offline procedure in which Codex (GPT-5.4) edits one rule layer per round using evaluator scores and execution traces. On TelcoOutageBench (131 real incidents: 51 for evolution, 80 for test), AVERT outperforms direct prompting by 2.71 to 13.81 RCA Score points across eight backbones (Table 2), and the evolved rules add 6.16 points on Qwen3.5-27B (Table 3).

Critical strengths

The benchmark is built from 131 production group-outage work orders with engineer-annotated, multi-cause labels (Sec. 3.2, Table 1), which is a realistic setting that most public RCA benchmarks lack, although the data cannot be released (App. F.4).

The topology-impact verifier turns a diagnosis into a falsifiable check, and the cumulative ablation credits it with the largest pre-evolution gain on Qwen3.5-27B (70.05 to 75.25 RCA Score, Table 3).

The comparison is backbone-matched over eight models with three runs each (Table 2), the ablation is repeated for every backbone (Table 4), and the evolution procedure never sees the 80 test incidents (Sec. 5.1, App. E).

Critical weaknesses

The figures have significant presentation problems: the overview flow chart (Fig. 1) does not show the control flow described in Sec. 4, the three method figures (Figs. 1 to 3) contradict one another, and Fig. 3 contains visible editing artifacts.

The evaluation in Table 2 and Sec. 5.1 includes too few baselines and no domain-specific ones, and Related Work (Sec. 2) leaves out the recent line of hypothesize-then-verify and graph-guided LLM RCA agents [R1-R4] that the core idea overlaps with.

"Tool-verified agent", the concept in the paper's title, is never defined in Sec. 1 or Sec. 4: the verifier in App. D.3 checks only one necessary topological condition while contribution 2 (Sec. 1) presents every diagnosis as "falsifiable", and neither Sec. 4 nor App. F (Table 9) states whether verification is enforced or how many agents or LLM roles AVERT contains.

Questions for authors

Responsibility-Routed Rule Evolution is in the title and is listed as a contribution, but Sec. 4.3 leaves most of the explanation to App. E, Table 3 does not separate its gain from adding the EL/CL/VL rules themselves, and the case for routing rests on one single-run experiment on one backbone (Fig. 4). The dataset is under-specified: Sec. 3.2 never defines the "processed alarm table" every method receives, Table 1 reports row counts but no input sizes in tokens, and the claim in Sec. 3.2 and App. A.1 that the full evidence does not fit in one prompt is never tested.

Summary of novelty:

The components are individually known: a ReAct-style tool agent on a mini-SWE-agent harness, hypothesis generation followed by verification and revision (HERO [R1], SpecRCA [R2], GALA [R3, R4]), graph-bounded investigation (GALA [R3, R4]), and LLM-driven prompt evolution (TextGrad (Yuksekgonul et al., 2024), AvaTaR (Wu et al., 2024), eARCO [R5]). What is new is the telecom-specific topology-impact verifier and the restriction of each evolution edit to one of three rule layers. The second is a modest change to existing prompt optimizers, and since the paper compares against neither RCA agents nor existing optimizers, the share of the gain that comes from AVERT's specific design cannot be determined.

Detailed comments to the authors:

The verifier and the TelcoOutageBench evaluation are solid engineering work, and the per-backbone ablation in Table 4 is more thorough than is usual for agent papers. The comments below concern how the method and data are presented and what the method is compared against.

  1. Fig. 1 places three "layers" around a robot icon and does not show the actual sequence described in Sec. 4 (hypothesis, then EL, CL and VL, then verification and revision), and its CL box uses a different font and border color from EL and VL. Fig. 2 does not match Fig. 1 or Sec. 4.1, for example by calling diagnose before the evidence tools and leaving the dashed "Feedback: failed verification" arrow unexplained, and Fig. 3 shows visible editing artifacts (a pasted label in box 4, a patched line in box 2 in a different font, and swapped ACCEPT/ROLLBACK arrowhead colors).

  2. Table 2 compares AVERT only with Direct Prompt and one in-house heuristic, and neither the cited telecom agent TN-AutoRCA (Wu et al., 2025) nor any existing prompt optimizer (e.g., eARCO [R5], TextGrad (Yuksekgonul et al., 2024)) is run. Related Work (Sec. 2) also misses earlier cloud RCA agents [R6-R9], and recent RCA agents built on the same hypothesize-then-verify idea with graph guidance, such as HERO [R1], SpecRCA [R2] and GALA [R3, R4], and these last make the verifier-in-the-loop contribution less distinctive than claimed.

  3. The term "tool-verified agent" is never defined, and the paper does not say how many agents or LLM calls AVERT uses or whether the agent must call the verifier before answering. The verifier itself (App. D.3) only checks that the predicted failures cut all paths of the affected sites, so it cannot tell cause types apart or reject an over-broad hypothesis, and the "falsifiable" claim in contribution 2 is therefore overstated.

  4. Rule Evolution is in the title, but Sec. 4.3 leaves the acceptance rule and routing details to App. E, and the "routing" is GPT-5.4's judgment rather than a defined credit-assignment procedure. Table 3 goes straight from no rules to evolved rules, so the gain from evolution is not separated from adding the hand-written EL/CL/VL prompt, and the case for routing rests on a single run on one backbone (Fig. 4).

  5. The paper never explains how the "processed alarm table" is built from the raw alarms or whether that step used the labels, and it gives no collection period, selection criteria or annotator agreement. It also gives no token counts for the claim that raw alarms do not fit in one prompt.

  6. The gains are not uniform, and the paper does not discuss this: for example, AVERT lowers Match F1 on Qwen3.5-9B, and the evolved rules lower RCA Score on Claude Opus-4.6 and DeepSeek-v3.2 (Table 2, Sec. 5.2). The benchmark contains only 22 hardware failures, and no per-cause-type results are reported, so it is unclear whether the method works beyond the power and fiber majority.

References (new; works already in the submission are cited by its own author-year keys):

R1. J. Lyu, J. Yi, P. Chen, S. Liang, L. Pan. HERO: Hypothesis-Centered Root-Cause Analysis for Microservice Incidents. ASE 2026. https://doi.org/10.1145/3832783.3837509

R2. L. Zhang et al. Hypothesize-Then-Verify: Speculative Root Cause Analysis for Microservices with Pathwise Parallelism (SpecRCA). ICSE-NIER 2026. https://arxiv.org/abs/2601.02736

R3. Y. Tian, Y. Liu, Z. Chong, Z. Huang, H.-A. Jacobsen. GALA: Can Graph-Augmented Large Language Model Agentic Workflows Elevate Root Cause Analysis? arXiv preprint, 2025. https://arxiv.org/abs/2508.12472

R4. Y. Tian, Y. Liu, Z. Chong, Z. Huang, Y. Li, H.-A. Jacobsen. GALA: Graph-Augmented LLM Agents for Root Cause Analysis and Incident Response in Microservices. ASE 2026. https://arxiv.org/abs/2608.08968

R5. D. Goel, R. Magazine, S. Ghosh, A. Nambi, P. Deshpande, X. Zhang, C. Bansal, S. Rajmohan. eARCO: Efficient Automated Root Cause Analysis with Prompt Optimization. ASE 2025 Industry Showcase. https://arxiv.org/abs/2504.11505

R6. Z. Wang et al. RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models. CIKM 2024. https://doi.org/10.1145/3627673.3680016

R7. Y. Chen et al. Automatic Root Cause Analysis via Large Language Models for Cloud Incidents (RCACopilot). EuroSys 2024. https://doi.org/10.1145/3627703.3629553

R8. W. Zhang et al. mABC: Multi-Agent Blockchain-Inspired Collaboration for Root Cause Analysis in Micro-Services Architecture. Findings of EMNLP 2024. https://arxiv.org/abs/2404.12135

R9. J. Xu et al. OpenRCA: Can Large Language Models Locate the Root Cause of Software Failures? ICLR 2025. https://proceedings.iclr.cc/paper_files/paper/2025/file/d29b8d53678015079e1d245c023e49d2-Paper-Conference.pdf

Reviewer_GsHyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces AVERT, an LLM-based agent for identifying multiple root causes of telecom network outages from incident tickets, network topology, and alarm records. Instead of relying on free-form reasoning, AVERT uses tools to retrieve relevant alarms and reconstruct service paths, organizes diagnosis into candidate enumeration, causal and topological constraint checking, and final verdict selection, then verifies whether the proposed failures would actually disconnect the affected sites. It also proposes an offline method that analyzes past diagnostic errors and updates only the rule stage responsible for each error. Evaluated on real-world incidents, AVERT consistently improves root-cause-analysis scores across eight LLM backbones.

Critical strengths
  1. The paper addresses an important and practically challenging problem using a well-motivated combination of targeted evidence retrieval, structured diagnostic rules, and deterministic topology-based verification.
  2. The paper presents evaluations covering real-world incidents on which the authors show competitive gains.
Critical weaknesses
  1. A major limitation is that the evaluation is restricted to a single closed, proprietary dataset. Although the results demonstrate AVERT’s effectiveness in the studied operational environment, it remains unclear whether the method generalizes to other telecom networks, incident distributions, alarm schemas, or organizational settings. The lack of evaluation on additional datasets also makes it difficult to determine whether the reported gains reflect broadly applicable reasoning capabilities or adaptation to the characteristics of this particular dataset.
  2. The headline Direct Prompt comparison is not controlled for information access: Direct Prompt receives only processed alarms. AVERT can retrieve raw alarms through filter_alarms. The authors acknowledge explicitly that Table 2 is an end-to-end comparison rather than a pure estimate of agent design. They do not evaluate a “Direct Prompt + all raw alarms” condition because some alarm tables would exceed the usable prompt budget.
  3. Generalization of the method may be limited: The evolved prompt is long and contains many highly specific mappings. These rules may be valuable operational expertise, but they complicate the novelty claim. The learned object is essentially a textual rule policy specialized to one partner’s network representation and alarm vocabulary. The key scientific question is whether “responsibility routing” offers a general agent-optimization principle beyond this hand-structured policy space which is weakly validated in the paper.
Questions for authors

Please refer weakness section above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.