OpenRCA 2.0: Benchmarking Process Level Incident Root Cause Analysis for Microservices

Root cause analysis (RCA) poses a holistic test of LLM agentic capabilities, such as long-context understanding, multi-step reasoning, and tool use. However, existing RCA benchmarks label only the root cause, leaving the agent’s reasoning about how the fault produces the observed symptoms unverified. To enable process-level evaluation, we introduce PAVE, a step-wise labeling protocol that uses deliberately injected faults as known interventions to reconstruct verified propagation paths. PAVE has access to fault-injection records unavailable to the agent, enabling it to verify propagation paths from known causes to observed symptoms. Applying PAVE yields OpenRCA 2.0, an RCA benchmark for process-level evaluation of LLM agents, comprising 454 instances across 3 microservice systems. Across 16 frontier LLMs, agents recover the complete set of root causes with an average exact-match accuracy of only 47.6%. Analysis of agent trajectories shows that, for 97.1% of missed root causes, the faulty service had already appeared in a query or tool result. Process-level evaluation reveals that agents often attribute failures to services that interact with the faulty service or are affected by it.