- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OpenRCA 2.0 is a process-level benchmark for evaluating LLM agents on microservice root cause analysis (RCA). Its core contribution is PAVE (Path Annotation via Verified Effects), an annotation protocol that treats fault injection as a known intervention to reconstruct verified propagation paths, thereby producing ground truth over the agent's reasoning process rather than just the final root-cause label. The benchmark contains 454 instances across three architecturally distinct microservice systems, 23 fault types, and multiple data stores, and was evaluated on 16 frontier LLMs with a unified tool set. Evaluation shows an exact-match accuracy of only 47.6% on average, while the novel AnySvc metric reaches 88.7%, and edge-level F1 (77.0%) is systematically lower than node-level F1 (86.1%). A key finding is that for 97.1% of missed root causes, the faulty service had already appeared in the query or tool results, and agents often attribute faults to services that interacted with, or were affected by, the faulty service — evidence that the bottleneck lies in causal reasoning rather than evidence retrieval.
Critical strengths
• The problem is novel and has genuine value. The paper points out that existing RCA benchmarks label only the root cause, not the reasoning process, and cannot distinguish the two failure modes of "never examined the faulty service" vs. "examined it but misjudged the causal direction." The gap is clearly defined and the motivation is convincing.
• The two-phase verification design is clear and reproducible. Structural Pruning (topology + rule set) plus Causal Verification (baseline deviation, call evidence, timing alignment), with the rule set, 18 propagation rules, detection thresholds, and hyperparameters all disclosed, meeting the reproducibility requirements of benchmark work.
• The dataset has reasonable diversity. Three systems with distinct architectures, covering both horizontal and vertical propagation channels, 23 fault types, and multiple data stores.
Critical weaknesses
• The scale is small and generalizability is in question. Of the 454 instances, 274 come from TrainTicket while Otel Demo has only 32; moreover, all are controlled faults under synthetic load, and the paper itself states they do not represent production incidents.
• The blind audit has limited independence and a small sample. Only 28 instances/138 edges, conducted by the authors themselves with LLM assistance, with one instance also used for development/debugging; two rounds agree on 134/138, and the disagreements are adjudicated by the authors.
• The agent configuration is single. Only one tool-augmented ReAct-style architecture is evaluated, without providing the dependency graph; no comparison across different agent frameworks, retrieval strategies, or whether a topology prior is given, limiting the scope of the conclusions.
• The results rely on manually constructed priors. Structural Pruning depends heavily on the dependency graph G and the propagation rule set R; propagation mechanisms outside R's vocabulary can be silently pruned without being detected.
Questions for authors
Q1. Only one tool-augmented ReAct-style architecture is evaluated, without providing the dependency graph. Have the authors compared different agent frameworks, retrieval strategies, and whether a topology prior is given?
Q2 PAVE's results depend heavily on the dependency graph G and the propagation rule set R; mechanisms outside R's vocabulary are silently pruned by Structural Pruning. Have the authors conducted a coverage/sensitivity analysis of R (e.g., adding or removing rule types and observing how the verified paths and final metrics change)? If a genuine propagation mechanism is not in R, how can it be prevented from being mislabeled as "non-propagation" and dropped?
Q3. Of the 454 instances, 274 come from TrainTicket while Otel Demo has only 32; the released set also contains 4 frontend root causes, 129 single-hop, and 22 instances with no service edge. Why are these instances that fail the original filtering criteria retained? What is their impact on metrics such as EM/PR? Should results be reported per-system/per-fault-family with a weighted aggregate, to avoid a single system dominating the conclusions?
4 comments
W3 / Q1. Other agent frameworks and a topology prior.
The benchmark's object of evaluation is the LLM, so the framework is held fixed to keep model comparisons controlled, as in the SWE-bench Bash Only track, which runs every model in the same minimal scaffold.
To measure what the framework contributes, we ran DeepSeek V4.1 Flash 1017 on all 454 instances in two frameworks: ours, and Claude Code, a general coding agent that runs shell and Python on the raw telemetry. Claude Code is a different framework and a different retrieval strategy: free-form code with no row limit instead of one SQL tool. Telemetry, ground truth, the 20-turn budget, and the answer format are the same in both.
| Framework | EM | AnySvc | PR | Node F1 | Edge F1 |
|---|---|---|---|---|---|
| Ours (SQL tool) | 45.8 | 95.4 | 86.8 | 85.6 | 75.3 |
| Claude Code | 59.7 | 92.5 | 88.7 | 88.0 | 79.8 |
The paper's conclusions hold under the coding agent. AnySvc still exceeds EM by 32.8 points, and Edge F1 stays 8.2 points below Node F1 (both 95% CIs exclude zero). Locating the root service does not improve (AnySvc 92.5 vs 95.4 in our framework), and PR does not change significantly (88.7 vs 86.8).
What the framework changes is how precisely the answer is stated, and with it the level of EM (+13.9 points, 95% CI 9.9–18.1). An extra root cause appears in 14.8% of Claude Code's answers (ours: 28.2%), and a wrong fault kind in 27.3% (ours: 36.8%). Shifts in level like this are why Table 2 gives every model the same framework.
The dependency graph is withheld on purpose (App. F.4.3, "No topology leakage"). Inferring which service calls which from traces is part of the diagnosis. A provided graph would turn edge prediction into lookup over a known structure.
W4 / Q2. Dependence on and ; mechanisms outside .
is applied per fault type. The implementation instantiates it as one manifest per fault type. Each manifest lists, layer by layer, which transitions of a path from that fault may follow and the feature ranges that define the destination state (e.g. the latency ratio for slow). A hop outside these is not proposed, so this is the layer where a mechanism outside the vocabulary could be silently pruned.
We tested this directly by rerunning PAVE with a permissive R that admits every call edge at every layer for every fault type, keeping Phase 2 unchanged. An edge that this run admits and G* lacks is a candidate mechanism outside the vocabulary that Phase 2's call evidence still supports. There are 80 such edges (4.5% of the 1,770 in ), all in 26 TrainTicket instances. Rescoring all 16 models against the permissive graphs changes mean Edge F1 by 0.01 points (largest change for any model 0.56), and the Edge F1 ranking is identical (Kendall τ = 1.00). The vocabulary therefore prunes few supported edges, and those it prunes do not change model comparisons.
Dear Reviewer beCt,
Thank you very much for your help with reviewing for ICLR'27!
The authors have now responded to your reviews. As the discussion period is approaching the end, I'm wondering if you could have a look at the authors' responses and see whether they have addressed your concerns. And could you also leave a comment to let the authors know your thoughts? If any concerns remain, please explain which ones have not been addressed and why.
Warm regards, Your AC
Dear Reviewer beCt,
Thank you again for your thoughtful review. With only a few hours remaining in the discussion period, we would greatly appreciate it if you could take a moment to revisit our clarifications. If they have addressed your concerns, we would be grateful if you could revisit your overall assessment in light of our responses.
We sincerely appreciate your time and consideration.
Best regards,
Authors
We thank the reviewer for recognizing the problem definition and the reproducibility of the two-phase design. All new results below will be added to the final version.
W1 / Q3. Scale, balance, and the retained instances.
Scale. Among RCA and cloud-operations benchmarks with LLM-agent evaluation at ML venues, OpenRCA 2.0 is the largest in instances and contains the largest open-source system:
Balance. Instances are concentrated on TrainTicket by design. On the smaller systems models are already near ceiling (top 8 above 95% EM on Hotel Reservation), so instances are concentrated where they still separate models. Per-system leaderboards are in Appendix Table 8.
Subsets. On TrainTicket alone (274 instances) and on instances with at least two propagation hops (306), both main findings hold. AnySvc exceeds EM by 57.5 and 50.7 points, and Edge F1 is significantly below Node F1 for all 16 models. Weighting the three systems equally gives mean EM 53.6 and Edge F1 78.8.
Synthetic faults. Instances are controlled injections under synthetic load, as the Datasheet states (App. A). This is what makes the process labels possible: the injection record is the intervention the annotator conditions on, and production incidents do not come with one.
Why instances failing the original filters are retained. Structural filtering was applied to the earlier annotation, before effect verification. PAVE's re-annotation changed some graphs. We kept these instances rather than re-filter on the new labels, because their outcome labels remain valid. Graph metrics already exclude instances with no service edge, and PR excludes instances without an alarm downstream of a root cause (§3.2).
Removing all instances with at most one hop, which include the four frontend-root instances, leaves 306 instances. Mean EM changes from 47.6 to 41.2 and PR from 84.0 to 86.1. The EM order is essentially unchanged (Kendall τ = 0.90 with the full set), and Claude Opus 5 remains first.
W2. Audit independence and size.
We audited 30 new instances not used in development, 10 per system, sampled with a fixed seed and labeled by the pipeline exactly as submitted. Three annotators with 2, 3, and 6 years of SRE or related experience judged every observed call edge between services whose behavior changed in the fault window. None of them is an author of this paper. Because they judged every such edge, not only the pipeline's candidates, recall is measured directly.
The annotators judged independently (Fleiss' κ = 0.73 on whether each edge is real). Final labels are the majority verdict, or, for edges examined a second time, the label all three annotators agreed on. Confidence intervals come from a bootstrap over instances.