- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ORCA-bench, a new benchmark for root cause analysis (RCA) that incorporates greater task complexity, including time-to-detection (TTD) and report-time offsets, along with source-code access and human verification. The evaluation yields several interesting findings, such as reduced RCA accuracy when input context is limited or source-code access is removed.
Critical strengths
- The paper is well organized and easy to follow.
- The paper provides a comprehensive experimental evaluation of different models and configurations on ORCA-bench.
Critical weaknesses
- The novelty needs to be articulated more clearly. Source-code access, comprehensive telemetry interfaces and data, and human verification have already been explored in prior work. The explicit modeling of TTD and report-time offsets appears to be a distinctive contribution and could be emphasized further.
- RCA performance depends heavily on agent harness design, including skills, context management, and tool engineering. Therefore, the results do not sufficiently support the broad conclusion that current agents are not ready for oncall.
- The motivation for another RCA benchmark is not sufficiently clear. The paper should better explain the unmet need that ORCA-bench addresses and why existing benchmarks are insufficient.
Questions for authors
- Have the code and data been publicly released?
- How do different agent harnesses perform on ORCA-bench?
Thank you for your thoughtful and constructive review. We were delighted that you found the paper is well organized and easy to follow and provides a comprehensive experimental evaluation of different models and configurations. We address each question below.
Public release of code and data: We have publicly released the code on Github and 755 of the 1,079 tasks through Harbor Hub. The release includes task construction and evaluation scripts, access to the telemetry snapshots, and instructions for reproducing experiments and submitting leaderboard results. The remaining 324 tasks form a separate held-out evaluation split to monitor data contamination. Their task packages omit the reference answers, and submissions are scored by a judge run by the maintainer, keeping the answers private.
Effect of Harness on Performance: We have conducted a new experiment comparing three open-source harnesses: Terminus-2, RCA-agent from OpenRCA, and OpenSRE. For RCA-agent, we export the metrics, logs, and traces from ORCA-bench as CSV files, following the conventions of the Market dataset in OpenRCA. For OpenSRE, we use ORCA-bench’s original snapshotted telemetry environment and retain OpenSRE’s native investigation workflow and tools, including its telemetry-query tools, shell commands, and Python execution. For all harnesses, we score the final reports using ORCA-bench’s LLM judge verifier. Below we report RCA accuracy across the incident tasks in ORCA-bench Verified.
mean ± 1 std error with N=32
For OpenSRE runs, we set a 64-iteration limit that was exhausted on 19 of 32 incident tasks for GPT-5.5, which explains the agent’s performance gap compared to other harnesses.