acceptodds
Under review as a conference paper at ICLR 2027

ContextCraft: Root Cause Analysis with Tool-Augmented Agents

Abstract

Root Cause Analysis (RCA) for modern distributed systems requires synthesizing gigabytes of heterogeneous observability signals during active incidents where engineers must restore service quickly despite noisy, delayed, or missing telemetry data. Recent LLM-guided RCA systems demonstrate that agents can iteratively query telemetry and propose diagnoses, yet they typically commit to a single conclusion via unstructured scratchpads with no mechanism to compare or audit competing hypotheses. We propose ContextCraft—a well designed agentic RCA system to operationalize Analysis of Competing Hypotheses (ACH) for AI agents augmented with a curated set of tools that are efficient in root cause localization. Across our evaluations, ContextCraft demonstrates that structured hypothesis–evidence reasoning can improve the accuracy–efficiency trade-off over free-form agentic RCA, achieving up to a 76.3% relative accuracy gain and 2.14 faster execution in our primary comparison, while producing fully auditable, evidence-grounded reports.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Root cause analysis, RCA, OpenRCA, AI agents, tool-augmented reasoning, observability, distributed systems, incident diagnosis, hypothesis evaluation, anomaly detection, reliability engineering, context engineering, telemetry

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_V9nxOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing RCA agents typically accumulate evidence in free-form text and commit to a single hypothesis. This makes it difficult to compare alternative hypotheses, track supporting and refuting evidence, and determine when to stop gathering evidence. To address these limitations, this paper proposes ContextCraft, which generates a diverse set of hypotheses, gathers evidence, and ultimately identifies the root cause. Experiments show that ContextCraft outperforms OpenRCA across multiple models.

Critical strengths
  1. The paper identifies key limitations of existing RCA agents: difficulty comparing alternative hypotheses, tracking supporting and refuting evidence, and determining when sufficient evidence has been collected.
  2. The paper proposes a new RCA agent workflow that generates a diverse hypothesis set and then gathers evidence to identify the root cause.
Critical weaknesses
  1. The experimental evaluation is insufficient. Table 1 includes only three models, and ContextCraft underperforms OpenRCA on Gemini 3.5 Flash.
  2. The method imposes a rigid workflow that lacks flexibility to handle unexpected situations.
  3. The paper lacks comprehensive ablation studies and comparative experiments, such as analyses of hypothesis-set size and ablations of different telemetry modalities.
Questions for authors
  1. Why does ContextCraft underperform OpenRCA on Gemini 3.5 Flash?
  2. How does hypothesis-set size affect performance, and what are the results of ablations across different telemetry modalities?
  3. How does ContextCraft handle unexpected situations given the rigidity of its prescribed workflow?
Reviewer_Ko3aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ContextCraft, a tool‑augmented LLM agent built as a LangGraph state‑machine for distributed‑system root‑cause analysis. Targeting confirmation bias, unauditable free‑form reasoning and unbounded tool calls in prior RCA agents, it operationalizes Analysis of Competing Hypotheses (ACH): competing root‑cause hypotheses are preserved, telemetry data is collected through a fixed set of observability tools, and each piece of evidence receives a structured verdict against every hypothesis and is logged in an evidence matrix. An evidence‑driven stopping rule ends data collection and produces fully auditable diagnosis reports. Its main contributions lie in realizing ACH for LLM agents with the evidence‑matrix mechanism, constructing a structured RCA workflow with hypothesis expansion, discriminative tool planning, multi‑dimensional scoring and early‑stopping logic, and employing a fixed telemetry toolchain for improved reproducibility, alongside open prompt templates for empirical research. Evaluated on the OpenRCA Bank dataset across Gemini 2.5 Pro, Gemini 3.5 Flash and Qwen3 against the OpenRCA baseline, ContextCraft greatly cuts LLM invocation and token overhead; under Gemini 2.5 Pro it delivers improved accuracy and a 2.14× speedup. An ACH ablation demonstrates performance gains despite overlapping confidence intervals. The work admits notable limitations including inconsistent cross‑model performance, heuristic hand‑tuned hyperparameters, reliance on only one benchmark dataset, absence of quantitative comparison against other structured multi‑agent RCA systems, and lack of human‑subject user studies.

Critical strengths
  1. Existing LLM agents for root‑cause analysis mostly record evidence using free‑form scratchpads. The authors introduce Analysis of Competing Hypotheses (ACH), a methodology from intelligence analysis, into the agent workflow. Multiple competing root‑cause hypotheses are evaluated simultaneously, and each piece of evidence is assigned a verdict of SUPPORTS / WEAK SUPPORT / NOT RELEVANT / REFUTES against all hypotheses to construct an evidence matrix.
  2. A curated set of fixed, validated observability tools is adopted, including wavelet‑based anomaly detection, STL decomposition, Granger causality, log‑burst analysis, trace latency analysis and others. These tools produce structured outputs fed into the evidence matrix and yield auditable root‑cause reports, where every conclusion can be traced back to tool invocations and evidence verdicts.
  3. The experimental design is relatively rigorous, with comprehensive ablation studies and multi‑model validation.
Critical weaknesses
  1. The methodological novelty is limited. Innovations reside in prompt workflows, state machines, and structured intermediate representations (evidence matrix). Many modules are heuristic engineering designs with manually tuned parameters lacking justifications, including the verdict vocabulary, scoring weights, λ, and Nmax=3N_{max}=3, all empirically set.
  2. Direct quantitative comparisons against other structured multi‑agent RCA approaches such as RCAFlow, Flow‑of‑Action and Praxis are missing. These prior works also emphasize structured reasoning. The authors only provide qualitative comparison, stating that runtime artifacts of baselines are unavailable for numerical evaluation. Without quantitative benchmarks, it is difficult to situate ContextCraft against state‑of‑the‑art methods.
  3. Inconsistent performance across different models raises concerns about generalizability. Specifically, Gemini 2.5 Pro achieves improved accuracy alongside a 2.14× speedup; Gemini 3.5 Flash yields reduced accuracy compared with OpenRCA despite substantial token savings; Qwen3 brings marginal accuracy gains but slower wall‑clock runtime.
  4. The evaluation relies solely on the Bank system from the OpenRCA benchmark, which limits external validity. This dataset suffers from sparse logs, missing severity annotations, and multiple plausible explanations for certain cases. Validation on additional datasets is absent.
Questions for authors
  1. Since quantitative comparison against related structured multi‑agent RCA systems (RCAFlow, Flow‑of‑Action, Praxis) is not available, can you elaborate on concrete, testable behavioural differences between ContextCraft and those alternatives beyond high‑level qualitative description? For example, what failure modes would ContextCraft avoid that those systems may suffer from?
  2. Only the OpenRCA Bank dataset is used for evaluation. What known artefacts of this benchmark (e.g. sparse logs, multiple valid ground‑truth explanations) might skew your reported metrics, and to what degree could these artefacts inflate or deflate ContextCraft’s real‑world expected performance?
  3. Many hyperparameters including verdict‑mapping rules, scoring weights, λ coefficient and the hard round cap Nmax=3N_{max}=3 are empirically set. Could you provide sensitivity analysis results showing how model performance varies when these key parameters change?
  4. Your ACH ablation shows accuracy improvement yet overlapping bootstrap confidence intervals. What statistical test was applied to assess whether the observed gain from the ACH loop is statistically significant? How confident are we that this performance lift will persist across new data?
  5. Why does ContextCraft degrade accuracy for Gemini 3.5 Flash while bringing benefits for Gemini 2.5 Pro? Is this performance drop driven by the heavier reasoning burden imposed by the evidence‑matrix evaluation on weaker‑capability LLMs? What minimum model capability is required for reliable operation of your ACH workflow?
Reviewer_Du1AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ContextCraft, an automated pipeline to localize root causes of agent system failures. ContextCraft uses a predefined set of telemetry tools to enable more logical and detailed hypotheses about possible root causes, and then uses another set of diagnostic tools to investigate potential evidence corresponding to each hypothesis. With the above tools, LLM analyzers can produce much more logical, structural, and interpretable root cause analyses. Experimental results show that ContextCraft outperforms the existing baseline OpenRCA.

Critical strengths
  • Clear problem formulation: the paper directly targets drawbacks of existing RCA methods (over-simple design), and proposes a more complex framework to solve the issues.
  • Clear framework design: the framework leverages multiple carefully organized tools to enable better root cause analysis by LLM.
Critical weaknesses
  • Relatively shallow: in most cases, the telemetry tools already exist in the agent workflow and cannot reflect the innovative design of the method; the diagnostic tools are strongly dependent on LLMs. The design of the diagnostic tools themselves may not matter much.
Questions for authors
  • Can the authors demonstrate the method on more complex or realistic scenarios? Does the method generalize well? How should the framework be adapted?
  • There are a line of work named agent debugging, which also aims to localize causes of agent failures. Multiple methods (e.g., AgentRx, AgentDebug) also have specific design to enable better LLM analyses. Should these methods be considered as baselines? If not, what are the fundamental differences between these methods and ContextCraft?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.