acceptodds
Under review as a conference paper at ICLR 2027

OpenRCA 2.0: Benchmarking Process Level Incident Root Cause Analysis for Microservices

Abstract

Root cause analysis (RCA) poses a holistic test of LLM agentic capabilities, such as long-context understanding, multi-step reasoning, and tool use. However, existing RCA benchmarks label only the root cause, leaving the agent’s reasoning about how the fault produces the observed symptoms unverified. To enable process-level evaluation, we introduce PAVE, a step-wise labeling protocol that uses deliberately injected faults as known interventions to reconstruct verified propagation paths. PAVE has access to fault-injection records unavailable to the agent, enabling it to verify propagation paths from known causes to observed symptoms. Applying PAVE yields OpenRCA 2.0, an RCA benchmark for process-level evaluation of LLM agents, comprising 454 instances across 3 microservice systems. Across 16 frontier LLMs, agents recover the complete set of root causes with an average exact-match accuracy of only 47.6%. Analysis of agent trajectories shows that, for 97.1% of missed root causes, the faulty service had already appeared in a query or tool result. Process-level evaluation reveals that agents often attribute failures to services that interact with the faulty service or are affected by it.

Primary area:
datasets and benchmarks
Keywords:
benchmark, root cause analysis, distributed systems, microservice

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_beCtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OpenRCA 2.0 is a process-level benchmark for evaluating LLM agents on microservice root cause analysis (RCA). Its core contribution is PAVE (Path Annotation via Verified Effects), an annotation protocol that treats fault injection as a known intervention to reconstruct verified propagation paths, thereby producing ground truth over the agent's reasoning process rather than just the final root-cause label. The benchmark contains 454 instances across three architecturally distinct microservice systems, 23 fault types, and multiple data stores, and was evaluated on 16 frontier LLMs with a unified tool set. Evaluation shows an exact-match accuracy of only 47.6% on average, while the novel AnySvc metric reaches 88.7%, and edge-level F1 (77.0%) is systematically lower than node-level F1 (86.1%). A key finding is that for 97.1% of missed root causes, the faulty service had already appeared in the query or tool results, and agents often attribute faults to services that interacted with, or were affected by, the faulty service — evidence that the bottleneck lies in causal reasoning rather than evidence retrieval.

Critical strengths

• The problem is novel and has genuine value. The paper points out that existing RCA benchmarks label only the root cause, not the reasoning process, and cannot distinguish the two failure modes of "never examined the faulty service" vs. "examined it but misjudged the causal direction." The gap is clearly defined and the motivation is convincing.

• The two-phase verification design is clear and reproducible. Structural Pruning (topology + rule set) plus Causal Verification (baseline deviation, call evidence, timing alignment), with the rule set, 18 propagation rules, detection thresholds, and hyperparameters all disclosed, meeting the reproducibility requirements of benchmark work.

• The dataset has reasonable diversity. Three systems with distinct architectures, covering both horizontal and vertical propagation channels, 23 fault types, and multiple data stores.

Critical weaknesses

• The scale is small and generalizability is in question. Of the 454 instances, 274 come from TrainTicket while Otel Demo has only 32; moreover, all are controlled faults under synthetic load, and the paper itself states they do not represent production incidents.

• The blind audit has limited independence and a small sample. Only 28 instances/138 edges, conducted by the authors themselves with LLM assistance, with one instance also used for development/debugging; two rounds agree on 134/138, and the disagreements are adjudicated by the authors.

• The agent configuration is single. Only one tool-augmented ReAct-style architecture is evaluated, without providing the dependency graph; no comparison across different agent frameworks, retrieval strategies, or whether a topology prior is given, limiting the scope of the conclusions.

• The results rely on manually constructed priors. Structural Pruning depends heavily on the dependency graph G and the propagation rule set R; propagation mechanisms outside R's vocabulary can be silently pruned without being detected.

Questions for authors

Q1. Only one tool-augmented ReAct-style architecture is evaluated, without providing the dependency graph. Have the authors compared different agent frameworks, retrieval strategies, and whether a topology prior is given?

Q2 PAVE's results depend heavily on the dependency graph G and the propagation rule set R; mechanisms outside R's vocabulary are silently pruned by Structural Pruning. Have the authors conducted a coverage/sensitivity analysis of R (e.g., adding or removing rule types and observing how the verified paths and final metrics change)? If a genuine propagation mechanism is not in R, how can it be prevented from being mislabeled as "non-propagation" and dropped?

Q3. Of the 454 instances, 274 come from TrainTicket while Otel Demo has only 32; the released set also contains 4 frontend root causes, 129 single-hop, and 22 instances with no service edge. Why are these instances that fail the original filtering criteria retained? What is their impact on metrics such as EM/PR? Should results be reported per-system/per-fault-family with a weighted aggregate, to avoid a single system dominating the conclusions?

4 comments

We thank the reviewer for recognizing the problem definition and the reproducibility of the two-phase design. All new results below will be added to the final version.

W1 / Q3. Scale, balance, and the retained instances.

Scale. Among RCA and cloud-operations benchmarks with LLM-agent evaluation at ML venues, OpenRCA 2.0 is the largest in instances and contains the largest open-source system:

BenchmarkInstancesLargest open-source system
OpenRCA 1.0 (ICLR 2025)335none (enterprise telemetry)
AIOpsLab (MLSys 2025)48SocialNetwork, 28 microservices
ITBench (ICML 2025)42 SRE scenarios (11 public)OpenTelemetry Demo
OpenRCA 2.0454TrainTicket, 44 services

Balance. Instances are concentrated on TrainTicket by design. On the smaller systems models are already near ceiling (top 8 above 95% EM on Hotel Reservation), so instances are concentrated where they still separate models. Per-system leaderboards are in Appendix Table 8.

Subsets. On TrainTicket alone (274 instances) and on instances with at least two propagation hops (306), both main findings hold. AnySvc exceeds EM by 57.5 and 50.7 points, and Edge F1 is significantly below Node F1 for all 16 models. Weighting the three systems equally gives mean EM 53.6 and Edge F1 78.8.

Synthetic faults. Instances are controlled injections under synthetic load, as the Datasheet states (App. A). This is what makes the process labels possible: the injection record is the intervention the annotator conditions on, and production incidents do not come with one.

Why instances failing the original filters are retained. Structural filtering was applied to the earlier annotation, before effect verification. PAVE's re-annotation changed some graphs. We kept these instances rather than re-filter on the new labels, because their outcome labels remain valid. Graph metrics already exclude instances with no service edge, and PR excludes instances without an alarm downstream of a root cause (§3.2).

Removing all instances with at most one hop, which include the four frontend-root instances, leaves 306 instances. Mean EM changes from 47.6 to 41.2 and PR from 84.0 to 86.1. The EM order is essentially unchanged (Kendall τ = 0.90 with the full set), and Claude Opus 5 remains first.

W2. Audit independence and size.

We audited 30 new instances not used in development, 10 per system, sampled with a fixed seed and labeled by the pipeline exactly as submitted. Three annotators with 2, 3, and 6 years of SRE or related experience judged every observed call edge between services whose behavior changed in the fault window. None of them is an author of this paper. Because they judged every such edge, not only the pipeline's candidates, recall is measured directly.

SystemPrecisionRecall
TrainTicket93.873.2
OTel Demo87.5100.0
Hotel Reservation100.093.3
All92.3 (95% CI 86.4–97.7)85.7 (95% CI 75.7–97.5)

The annotators judged independently (Fleiss' κ = 0.73 on whether each edge is real). Final labels are the majority verdict, or, for edges examined a second time, the label all three annotators agreed on. Confidence intervals come from a bootstrap over instances.

W3 / Q1. Other agent frameworks and a topology prior.

The benchmark's object of evaluation is the LLM, so the framework is held fixed to keep model comparisons controlled, as in the SWE-bench Bash Only track, which runs every model in the same minimal scaffold.

To measure what the framework contributes, we ran DeepSeek V4.1 Flash 1017 on all 454 instances in two frameworks: ours, and Claude Code, a general coding agent that runs shell and Python on the raw telemetry. Claude Code is a different framework and a different retrieval strategy: free-form code with no row limit instead of one SQL tool. Telemetry, ground truth, the 20-turn budget, and the answer format are the same in both.

FrameworkEMAnySvcPRNode F1Edge F1
Ours (SQL tool)45.895.486.885.675.3
Claude Code59.792.588.788.079.8

The paper's conclusions hold under the coding agent. AnySvc still exceeds EM by 32.8 points, and Edge F1 stays 8.2 points below Node F1 (both 95% CIs exclude zero). Locating the root service does not improve (AnySvc 92.5 vs 95.4 in our framework), and PR does not change significantly (88.7 vs 86.8).

What the framework changes is how precisely the answer is stated, and with it the level of EM (+13.9 points, 95% CI 9.9–18.1). An extra root cause appears in 14.8% of Claude Code's answers (ours: 28.2%), and a wrong fault kind in 27.3% (ours: 36.8%). Shifts in level like this are why Table 2 gives every model the same framework.

The dependency graph is withheld on purpose (App. F.4.3, "No topology leakage"). Inferring which service calls which from traces is part of the diagnosis. A provided graph would turn edge prediction into lookup over a known structure.

W4 / Q2. Dependence on G\mathcal{G} and R\mathcal{R}; mechanisms outside R\mathcal{R}.

R\mathcal{R} is applied per fault type. The implementation instantiates it as one manifest per fault type. Each manifest lists, layer by layer, which transitions of R\mathcal{R} a path from that fault may follow and the feature ranges that define the destination state (e.g. the latency ratio for slow). A hop outside these is not proposed, so this is the layer where a mechanism outside the vocabulary could be silently pruned.

We tested this directly by rerunning PAVE with a permissive R that admits every call edge at every layer for every fault type, keeping Phase 2 unchanged. An edge that this run admits and G* lacks is a candidate mechanism outside the vocabulary that Phase 2's call evidence still supports. There are 80 such edges (4.5% of the 1,770 in G∗\mathcal{G}^*), all in 26 TrainTicket instances. Rescoring all 16 models against the permissive graphs changes mean Edge F1 by 0.01 points (largest change for any model 0.56), and the Edge F1 ranking is identical (Kendall τ = 1.00). The vocabulary therefore prunes few supported edges, and those it prunes do not change model comparisons.

Dear Reviewer beCt,

Thank you very much for your help with reviewing for ICLR'27!

The authors have now responded to your reviews. As the discussion period is approaching the end, I'm wondering if you could have a look at the authors' responses and see whether they have addressed your concerns. And could you also leave a comment to let the authors know your thoughts? If any concerns remain, please explain which ones have not been addressed and why.

Warm regards, Your AC

Dear Reviewer beCt,

Thank you again for your thoughtful review. With only a few hours remaining in the discussion period, we would greatly appreciate it if you could take a moment to revisit our clarifications. If they have addressed your concerns, we would be grateful if you could revisit your overall assessment in light of our responses.

We sincerely appreciate your time and consideration.

Best regards,
Authors

Reviewer_6qUBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that existing RCA benchmarks label only the root cause and therefore cannot verify how LLM agents reason about fault propagation. The authors propose PAVE, which uses fault-injection records to reconstruct verified propagation paths. Phase1 enumerates candidate paths from the dependency graph and 18 propagation rules. Phase2 uses pre-injection data as the baseline, checks each hop for service anomalies, call evidence, and temporal ordering, and removes injections that did not take effect. The resulting OpenRCA 2.0 contains 454 instances from 3 microservice systems and covers 23 fault types. The authors evaluate 16 LLMs with a shared ReAct-style agent that accesses telemetry only through SQL, report outcome-level metrics (EM, AnySvc) and process-level metrics (PR, Node/Edge F1, SQL Exec), and analyze missed root causes based on agent trajectories.

Critical strengths
  1. The problem is well defined, and the labeling idea is natural. Outcome-only labels cannot distinguish an agent that never examined the faulty service from one that examined it but inverted the causal direction. The authors exploit the annotator's access to injection records, which the agent lacks, to construct process-level labels. Effect verification removes 37 instances whose injections did not take effect, correcting a common source of label noise in fault-injection benchmarks.

  2. The evaluation is large in scale and statistically careful. The paper evaluates 37 models in total, reports bootstrap confidence intervals for every result, and uses a paired bootstrap to test the ordering between models. Based on this test, the authors explicitly state that the main table supports tiers rather than a strict ranking.

  3. The case study is informative. In Figure4, the callee's server spans all return 200, while the caller receives 503. Claude Opus 5 cites this contrast as evidence yet still reports the callee as the root cause, and 12 of the 16 models make the same attribution. Such inverted causal judgments can only be revealed by process-level analysis.

Critical weaknesses
  1. The reliability of the process-level labels is not sufficiently validated. PAVE infers propagation edges from call relations, threshold-based anomaly detection, and temporal ordering, which the authors admit does not guarantee causality. Label quality thus rests on an audit that covers only 28 instances, is not broken down by system, and was performed by the authors with LLM assistance, with first-round results fed back into the pipeline. Its recall of 85.3% means agents are penalized for real edges the labels miss, so it is unclear how much of the reported 29.1% of exact-match answers with mismatched graphs reflects label gaps. The sensitivity of labels and rankings to the detection thresholds is also not examined.

  2. The dataset is small and imbalanced, with most discriminative instances from TrainTicket. The three systems are open-source testbeds with only 9 to 44 services. OTel Demo contributes just 32 instances, and Hotel Reservation is nearly saturated, with the top 8 models above 95% EM. In addition, 129 instances have only one-hop propagation and 22 have no service-level edge. The headline average EM of 47.6% pools all systems, while on TrainTicket the best model reaches only 53.3% and the average 26.6%, so the pooled numbers say little about complex propagation scenarios.

  3. The process-level metrics can hardly support the claim of process-level evaluation. Edge F1 trailing Node F1 largely follows from the definitions, since an edge requires both endpoints to be correct; given correct endpoints, edge precision is already 95.7%. PR is bounded by AnySvc and nearly equal to it (e.g., 90.4 vs. 93.2 for Claude Opus 5), suggesting that finding the right service almost always yields a valid path. SQL Exec exceeds 90% for every model and does not check whether results support the claims. The findings about reasoning therefore rely mainly on heuristic trajectory attribution, validated manually on only 23 samples.

  4. The results cannot be separated from the agent framework's settings. All models use one SQL-only framework whose settings may drive the results. Query outputs are truncated to about 10 rows, which may partly explain why 40.1% of missed root causes lack necessary evidence. Runs that hit the 20-turn limit (294 in total) missed 20.7% of root causes versus 11.5% for the rest. The prompt makes propagation optional and hints at multiple root causes, which may explain why GPT-5.6 Sol scores 81.2 EM but 42.3 PR on OTel Demo, and Qwen3.8-Max 93.8 AnySvc but 24.0 EM. With a single framework and no ablations, model failures cannot be distinguished from framework effects.

Questions for authors
  1. Could the authors have experienced SREs blindly audit a larger, system-stratified sample and report per-system edge precision, recall, and inter-annotator agreement? Does the model ranking remain stable when the main thresholds vary within a reasonable range?

  2. Could the authors report results on TrainTicket alone, or only on instances with at least two hops of propagation, to check whether the main conclusions still hold?

  3. Could the authors report Edge F1 and PR conditioned on a correct root-cause service, or add a metric that checks whether the SQL results support the corresponding claims, to show that the process-level metrics capture information beyond the outcome-level metrics?

  4. The current framework accesses data only through SQL, and query results are truncated. Could the authors re-evaluate several representative models on a TrainTicket subset with an agent that can execute Python, such as a general coding agent, require the propagation path in the output, and report how EM, PR, and the share of missed root causes lacking necessary evidence change?

2 comments

We thank the reviewer for the detailed reading and the concrete numbers. All new results below will be added to the final version.

W1 / Q1. Label reliability: independent, stratified audit; threshold sensitivity.

Audit. We audited 30 new instances not used in development, 10 per system, sampled with a fixed seed and labeled by the pipeline exactly as submitted. Three annotators with 2, 3, and 6 years of SRE or related experience judged every observed call edge between services whose behavior changed in the fault window. They did not know which edges the pipeline had produced. Because they judged every such edge, not only the pipeline's candidates, recall is measured directly.

SystemEdges judgedPrecisionRecallFleiss' κ
TrainTicket16293.873.20.77
OTel Demo5887.5100.00.75
Hotel Reservation22100.093.30.40
All24292.3 (95% CI 86.4–97.7)85.7 (95% CI 75.7–97.5)0.73

κ is computed on the independent judgments, on whether each edge is real. Final labels are the majority verdict, or, for edges examined a second time, the label all three annotators agreed on. Confidence intervals come from a bootstrap over instances.

Note that κ is lower on Hotel Reservation. Most of its audited faults are pod failures, after which requests to the failed service vanish. Whether an edge exists then cannot be judged from the data, so annotators often marked such edges UNCLEAR. At least one annotator did so for 12 of its 22 edges (55%), against 14% on TrainTicket and 31% on OTel Demo.

How much of the 29.1% reflects label gaps. A missing edge in G∗\mathcal{G}^* can only penalize an agent that draws it, not one that omits it. Most of the 29.1% are omissions: 22.4% of exact-match answers leave out at least one G∗\mathcal{G}^* edge. On the 30 audited instances, crediting the agent edges the audit finds supported lowers the mismatch only from 20.7% to 20.1%.

Thresholds. We shifted the hop-admission thresholds (latency ratio and increase, error and call-volume tests) and the node-level anomaly thresholds by ±20% and ±50%, reran the pipeline, and rescored all 16 models. At ±20%, at most 20 of the 1,770 reference edges change and mean Edge F1 moves by at most 0.11 points. At ±50%, at most 55 edges change. The Edge F1 ranking is identical in every threshold variant (Kendall τ = 1.00), and the PR order moves by at most one place.

W2 / Q2. Dataset size and balance; results on TrainTicket alone and on ≥2-hop instances.

Among RCA and cloud-operations benchmarks with LLM-agent evaluation at ML venues, OpenRCA 2.0 is the largest in instances, contains the largest open-source system, and is the only one with process-level labels:

BenchmarkInstancesLargest open-source system
OpenRCA 1.0 (ICLR 2025)335none (enterprise telemetry)
AIOpsLab (MLSys 2025)48SocialNetwork, 28 microservices
ITBench (ICML 2025)42 SRE scenarios (11 public)OpenTelemetry Demo
OpenRCA 2.0454TrainTicket, 44 services

The distribution across systems follows difficulty. On the smaller systems frontier models are already near ceiling: on Hotel Reservation the top 8 models exceed 95% EM, as the reviewer notes. Additional instances there would add little power to separate models, so we concentrate instances on TrainTicket, the largest system, where models still fail (274 of 454 instances). Per-system leaderboards are in Appendix Table 8.

These are results on the two subsets the reviewer asks for:

SubsetnAnySvcEMNode F1Edge F1Node–Edge gap significant
TrainTicket only27484.126.681.268.016/16 models
≥2 hops30691.941.285.174.616/16 models
All (Table 2)45488.747.686.177.016/16 models

Both findings hold on both subsets, and more strongly than in the pooled numbers. The AnySvc–EM gap widens from 41.1 points (all) to 57.5 (TrainTicket), and the Node–Edge gap from 9.1 to 13.2. Claude Opus 5 has the highest EM on both subsets (53.3 and 64.4), and the EM order largely agrees with the pooled order (Kendall τ = 0.72 and 0.90).

W3 / Q3. Do process metrics carry information beyond the outcome?

The process-level information lies in (a)–(c) below. As Finding 2 notes, node errors contribute substantially to the Edge–Node gap itself (95.7% conditional edge precision).

(a) Conditioned on a correct root-cause service, Edge F1 ranges from 70.4 to 88.4 across models, and PR from 83.9 to 97.5. A correct service does not by itself satisfy PR, which also requires a directed path of G∗\mathcal{G}^* edges from that service to an alarm (§3.2). PR is close to AnySvc for the strongest models (Claude Opus 5: 93.2 vs 90.4), but the gap reaches 10.8 points (MiniMax M3: 79.7 vs 68.9) and is significant under a paired bootstrap for all 16 models.

The two levels also order models differently: Kendall τ between the EM order and the conditional Edge F1 order is 0.53. For example, Qwen3.8-Max ranks last on EM (24.0) but has conditional Edge F1 82.0, above GPT-5.6 Sol (80.7), whose EM is 65.2.

(b) Correct outcome with wrong explanation: 29.1% of exact-match answers have graphs that differ from G∗\mathcal{G}^*, which no outcome metric can register.

(c) A non-LLM baseline: a trace-topology and anomaly heuristic reaches Node F1 67.5 (below every LLM), Edge F1 64.5 (LLM mean 77.0), and PR 31.5; on TrainTicket its Edge F1 is 47.4 vs the LLM mean of 68.0.

(d) SQL Exec is a validity control, not a measure of the explanation. It rules out tool failure as the cause of wrong diagnoses (96.5% on wrong vs 97.1% on correct diagnoses; query success 87.0% vs 85.9%). None of (a)–(c) uses the trajectory attribution.

W4 / Q4. Separating model failures from framework effects.

The benchmark evaluates LLMs, so the framework is held fixed as a controlled variable, as in the SWE-bench Bash Only track (mini-SWE-agent). Differences between rows of Table 2 therefore reflect the backbone.

Coding agent. As suggested, we ran DeepSeek V4.1 Flash 1017 on all 454 instances in two frameworks: ours, and Claude Code, a general coding agent that runs shell and Python on the raw telemetry with no row limit on query results. Telemetry, ground truth, the 20-turn budget, and the answer format are the same in both.

FrameworkEMEM (TrainTicket)AnySvcPRNode F1Edge F1
Ours (SQL tool)45.822.395.486.885.675.3
Claude Code59.738.392.588.788.079.8

The two runs separate framework effects from model failures. The failures the paper reports persist under the coding agent: AnySvc still exceeds EM by 32.8 points, and Edge F1 stays 8.2 points below Node F1 (both 95% CIs exclude zero). Locating the root service does not improve (AnySvc 92.5 vs 95.4 in our framework), and PR does not change significantly (88.7 vs 86.8).

What the framework changes is how precisely the answer is stated, and with it the level of EM (+13.9 points, 95% CI 9.9–18.1). An extra root cause appears in 14.8% of Claude Code's answers (ours: 28.2%), and a wrong fault kind in 27.3% (ours: 36.8%). Shifts in level like this are why Table 2 gives every model the same framework.

Most misses do not come from data access: for 58.1% of missed root causes in the original runs, the retrieved results already contained the evidence needed (§3.3).

Turn limit. The 20-turn budget is part of the task: it models incident response, where an SRE must localize a fault quickly. At the cap the model is asked to submit, and the answer is scored as usual. 9 of the 16 models did not reach it in any run, so the 20.7% vs 11.5% contrast mostly compares different models.

Truncation. The row count is an argument the model sets on each call (default 10). 73.1% of queries set it, and 76.3% aggregate in SQL, so the limit bounds rows returned, not what a query computes. Removing the limit in Claude Code does not raise AnySvc.

Optional propagation. Omission is rare: every model includes a propagation graph in at least 96.7% of its answers, and Claude Code did in all 454 without being required to. GPT-5.6 Sol's low OTel PR (42.3) has another cause: its graphs stop at frontend and leave out the last hop to frontend-proxy, where the alarm is observed. The hop is missing in 13 of its 14 OTel PR failures with a correct root service; Claude Opus 5 includes it in 24 of 25.

Multiple root causes. Under the same prompt, about one in five answers from the other 15 models contain a root cause that matches no injected fault. For Qwen3.8-Max it is more than half, which accounts for most of its AnySvc–EM gap. Reporting a root cause where no fault was injected is a diagnostic error, and EM counts it as one.

Reviewer_Jp2HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces OpenRCA 2.0, a benchmark for evaluating microservice root cause analysis beyond final diagnosis accuracy. Its PAVE pipeline combines known fault injections, dependency rules, and telemetry checks to annotate propagation graphs. The benchmark contains 454 instances across three systems. Evaluations of 16 LLM agents examine root-cause identification, propagation-graph agreement, and investigation trajectories to distinguish different diagnostic failures.

Critical strengths

Propagation annotations provide a useful extension to outcome-only evaluation, allowing correct diagnoses with incomplete or incorrect explanations to be distinguished. The benchmark combines controlled fault injection with normal and abnormal telemetry across three systems, under a shared agent interface

Critical weaknesses
  1. The reliability of the reference graphs needs stronger independent validation. Appendix E.1 reports 85.3% edge recall and acknowledges that agents can be penalized for real edges missing from the annotations. The audit also informed pipeline fixes and version selection before rechecking the same instances, so its reported precision does not establish performance on unseen cases. Since graph agreement is central to the benchmark, the impact of these omissions and development-dependent auditing on model scores needs to be quantified.
  2. The process-level evaluation only partially verifies evidence-based reasoning. Graph metrics assess the submitted explanation, while SQL Exec checks whether cited queries execute and return rows, not whether those rows support the claims. The trajectory analysis adds useful diagnostic information, but it does not close this gap. An agent could therefore obtain strong graph and SQL scores while attaching irrelevant evidence. The distinction between explanation agreement and verified reasoning should be clearer.
Questions for authors
  1. Can the final, frozen PAVE pipeline be audited on a fresh sample not used for development, including an explicit search for missing propagation edges?
  2. How often are agent-predicted edges marked incorrect despite being supported by the telemetry, and how much does adjudicating these edges change scores or model comparisons?
  3. On a representative sample, do the cited SQL results actually support the associated root-cause and propagation claims? How well does SQL Exec track that assessment?
  4. Can a simple trace-topology and anomaly-based baseline achieve strong graph scores? This would help establish how much the process metrics discriminate causal diagnosis from reconstructing plausible dependency structure.
3 comments

We thank the reviewer for the positive assessment and the concrete questions. All new results below will be added to the final version.

W1 / Q1. Independent audit of the pipeline on unseen instances, with an explicit search for missing edges.

We audited 30 new instances not used in development, 10 per system, sampled with a fixed seed and labeled by the pipeline exactly as submitted. Three annotators with 2, 3, and 6 years of SRE or related experience judged every observed call edge between services whose behavior changed in the fault window. None of them is an author of this paper. They did not know which edges the pipeline had produced. Because they judged every such edge, not only the pipeline's candidates, recall is measured directly.

SystemPrecisionRecall
TrainTicket93.873.2
OTel Demo87.5100.0
Hotel Reservation100.093.3
All92.3 (95% CI 86.4–97.7)85.7 (95% CI 75.7–97.5)

The annotators judged independently (Fleiss' κ = 0.73 on whether each edge is real; pairwise Cohen's κ 0.72–0.74). Final labels are the majority verdict, or, for edges examined a second time, the label all three annotators agreed on. Confidence intervals come from a bootstrap over instances.

Q2. Agent edges marked incorrect but supported by telemetry, and effect on scores.

The audit above includes every edge that any of the 16 models predicted on the 30 audited instances. Of the 67 predicted edges absent from G∗\mathcal{G}^*, the annotators judged 11 (16.4%) supported by telemetry.

Crediting each model for the supported edges it predicted raises mean Edge F1 on these instances from 86.3 to 87.8. The largest change for any model is 3.0 points, and the ranking is nearly unchanged (Kendall τ = 0.95).

Changing the pipeline itself also leaves the ranking unchanged. Shifting the hop-admission and anomaly thresholds by ±20% changes at most 20 of the 1,770 reference edges and moves mean Edge F1 by at most 0.11 points. Letting Phase 1 admit every call edge adds 80 edges and moves mean Edge F1 by 0.01. The Edge F1 ranking is identical in each of these variants (Kendall τ = 1.00).

Missing edges in G∗\mathcal{G}^* also shift all models together. Dropping 10% of its edges at random (5 seeds) lowers all models' Edge F1 together, and the ranking stays nearly fixed (τ ≥ 0.98).

W2 / Q3. SQL Exec versus evidence that supports the claim.

SQL Exec is a validity control, not a measure of reasoning. It rules out tool failure as the cause of wrong diagnoses. If queries had broken or returned nothing, SQL Exec would be lower on wrong answers. It is not: 96.5% on wrong vs 97.1% on correct diagnoses, and query success is 87.0% vs 85.9% (§3.3).

We agree the distinction should be explicit, and the final version will state it where the metrics are defined (§3.2). Node F1, Edge F1, and PR measure whether the explanation agrees with G∗\mathcal{G}^*, not whether the cited rows support it. Irrelevant evidence cannot raise them, because G∗\mathcal{G}^* is built from the injection record, which the agent does not see. The trajectory analysis (§3.3) reads what the agent retrieved, not what it cited.

Q4. A trace-topology and anomaly baseline.

We implemented the baseline the reviewer describes. It uses only the traces of the two windows, with no LLM, injection record, or G∗\mathcal{G}^*. It builds the call graph from cross-service spans. A service is anomalous when its error rate, p95/p99 latency, or request volume deviates from the normal window, with thresholds fixed before scoring. The root is the anomalous service with no degraded edge into an anomalous callee, and the propagation graph is the degraded call edges among anomalous services. We scored it with the paper's scorer, which reproduces all 7,263 submitted LLM scores exactly.

AnySvcPRNode F1Edge F1Edge F1 (TrainTicket)Edge F1 (Hotel Res.)
Topology + anomaly baseline34.431.567.564.547.498.6
LLMs, mean (range)88.7 (77.5–95.6)84.0 (68.9–92.5)86.1 (75.2–91.9)77.0 (61.4–87.2)68.095.2

The baseline does not reach strong graph scores. Its Node F1 is below every LLM, and its Edge F1 is 12.5 points below the LLM mean.

The gap is largest where propagation is deep. On TrainTicket the baseline's Edge F1 is 47.4 vs 68.0, and it finds a correct root in 21.2% of instances vs 84.1%. Even given the true root, its TrainTicket Edge F1 is 66.7. LLM answers with a correct root reach 71.8.

On Hotel Reservation, whose graphs average 1.5 edges and are mostly pod failures, the baseline recovers the graph from call topology and caller errors (Edge F1 98.6). It still finds the root in only 60.1% of instances (LLMs 98.4%).

Dear Reviewer Jp2H,

Thank you very much for your help with reviewing for ICLR'27!

The authors have now responded to your reviews. As the discussion period is approaching the end, I'm wondering if you could have a look at the authors' responses and see whether they have addressed your concerns. And could you also leave a comment to let the authors know your thoughts? If any concerns remain, please explain which ones have not been addressed and why.

Warm regards, Your AC

Thanks for the additional experiments. The independent audit addresses my main concern. I also read the discussion with Reviewer 6qUB and found the rescoring helpful, as it shows how annotation errors affect the scores and that the main conclusions still hold after correction. Despite the remaining limitations, I think this is a useful benchmark. I’m keeping my score of 3 and support acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.