acceptodds
Under review as a conference paper at ICLR 2027

ORCA-bench: How Ready Are Language Model Agents for Oncall?

Abstract

Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began. We introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. ORCA-bench pairs 1,079 RCA tasks with six days of metrics, logs, and traces collected from an OpenTelemetry-instrumented microservice system under continuous simulated user load. Agents investigate this recorded history through real observability interfaces—Prometheus, Jaeger, and OpenSearch via Grafana—with full access to application source code. Tasks systematically vary report specificity, time-to-detection, and co-occurring fault scenarios. Ground-truth symptoms are curated and signed off by expert SREs, and our LLM-as-judge is independently re-scored by humans (Cohen's ). Across five frontier agents, the best RCA Accuracy is 25.3% on Medium-difficulty tasks (the realistic-input setting) and 10.0% on Hard—a gap that remains even with Claude Fable 5. The weakest model hallucinates an implausible root cause in 40% of incident reports, and removing source-code access reduces RCA accuracy and increases the hallucination rate for every evaluated model. Crucially, these are performances on a curated 50 GB / six-day testbed with tasks investigated in isolation on a system whose code and instrumentation are public. Since real production systems are order of magnitudes larger, more dynamic, and more idiosyncratic, the gap we report in our controlled setting underscores the engineering work still needed before agents can be entrusted with production reliability.

Primary area:
datasets and benchmarks
Keywords:
large language models, LLM agents, agent evaluation, benchmarks, root cause analysis, site reliability engineering, AIOps, observability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9PEbOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ORCA-bench, a new benchmark for root cause analysis (RCA) that incorporates greater task complexity, including time-to-detection (TTD) and report-time offsets, along with source-code access and human verification. The evaluation yields several interesting findings, such as reduced RCA accuracy when input context is limited or source-code access is removed.

Critical strengths
  1. The paper is well organized and easy to follow.
  2. The paper provides a comprehensive experimental evaluation of different models and configurations on ORCA-bench.
Critical weaknesses
  1. The novelty needs to be articulated more clearly. Source-code access, comprehensive telemetry interfaces and data, and human verification have already been explored in prior work. The explicit modeling of TTD and report-time offsets appears to be a distinctive contribution and could be emphasized further.
  2. RCA performance depends heavily on agent harness design, including skills, context management, and tool engineering. Therefore, the results do not sufficiently support the broad conclusion that current agents are not ready for oncall.
  3. The motivation for another RCA benchmark is not sufficiently clear. The paper should better explain the unmet need that ORCA-bench addresses and why existing benchmarks are insufficient.
Questions for authors
  1. Have the code and data been publicly released?
  2. How do different agent harnesses perform on ORCA-bench?
1 comment

Thank you for your thoughtful and constructive review. We were delighted that you found the paper is well organized and easy to follow and provides a comprehensive experimental evaluation of different models and configurations. We address each question below.

Public release of code and data: We have publicly released the code on Github and 755 of the 1,079 tasks through Harbor Hub. The release includes task construction and evaluation scripts, access to the telemetry snapshots, and instructions for reproducing experiments and submitting leaderboard results. The remaining 324 tasks form a separate held-out evaluation split to monitor data contamination. Their task packages omit the reference answers, and submissions are scored by a judge run by the maintainer, keeping the answers private.

Effect of Harness on Performance: We have conducted a new experiment comparing three open-source harnesses: Terminus-2, RCA-agent from OpenRCA, and OpenSRE. For RCA-agent, we export the metrics, logs, and traces from ORCA-bench as CSV files, following the conventions of the Market dataset in OpenRCA. For OpenSRE, we use ORCA-bench’s original snapshotted telemetry environment and retain OpenSRE’s native investigation workflow and tools, including its telemetry-query tools, shell commands, and Python execution. For all harnesses, we score the final reports using ORCA-bench’s LLM judge verifier. Below we report RCA accuracy across the incident tasks in ORCA-bench Verified.

mean ± 1 std error with N=32

HarnessOpus 4.7Sonnet 4.6GPT-5.5DeepSeek-V4-Pro
Terminus-221.9% ± 7.0%25.0% ± 6.2%20.8% ± 6.5%26.0% ± 6.8%
RCA-agent28.1% ± 7.0%29.2% ± 7.1%31.2% ± 7.2%17.7% ± 5.6%
OpenSRE25.0% ± 7.8%28.1% ± 8.1%12.5% ± 5.9%15.6% ± 6.5%

For OpenSRE runs, we set a 64-iteration limit that was exhausted on 19 of 32 incident tasks for GPT-5.5, which explains the agent’s performance gap compared to other harnesses.

Reviewer_VZNtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ORCA-bench, a benchmark for evaluating language-model agents on root-cause analysis in an on-call setting. It runs the OpenTelemetry Astronomy Shop under simulated load and exposes six days of recorded metrics, logs, traces, and source code through realistic observability interfaces. The benchmark contains 1,079 tasks constructed from 11 feature-flag faults and varies report specificity, time-to-detection, and concurrent-failure structure. Experiments show that frontier agents perform poorly, particularly with vague reports and multiple possible causes.

Critical strengths
  1. On-call root-cause analysis is an important and challenging task in real-world production environments. Evaluating agents’ capabilities in this setting is therefore critical. By combining metrics, logs, traces, source code, and realistic query interfaces, this paper provides a substantially richer evaluation environment than existing static RCA benchmarks.
  2. The benchmark explicitly studies important operational factors such as vague user reports, delayed detection, concurrent incidents, and no-incident controls. Also, the symptom-level rubrics provide useful diagnostic supervision beyond a single root-cause label.
  3. The paper carefully validates its LLM judge against human scoring and reports strong agreement; and the harness comparison and source-code ablation also provide useful evidence that both agent infrastructure and available context affect performance.
Critical weaknesses
  1. Despite the large number of tasks, the underlying diversity is limited: the 1,079 tasks are derived from only 11 feature flags, 42 fault occurrences, and one six-day execution of a public demo application. Many tasks are therefore correlated variants rather than distinct incidents.

  2. The incidents and user reports are generated rather than collected from real on-call practice. The paper does not compare their distribution, difficulty, or failure taxonomy with real tickets or postmortems.

  3. Ground-truth construction depends substantially on GPT-5.4, while only 40 tasks receive complete manual verification. It is unclear whether the plausible-cause sets are consistently complete across the full benchmark.

  4. There is no human-SRE baseline, making it difficult to distinguish model limitations from intrinsic ambiguity or benchmark-interface difficulty.

  5. I think that this paper's claim about agents not being “ready for oncall” is stronger than the evidence supports. Also, the achived results can only establish difficulty on this controlled environment, but not necessarily on the distribution of real production incidents.

Questions for authors
  1. Why did you choose not to include even a small anonymized set of real on-call incidents or postmortems for external-validity analysis?

  2. How representative are the 11 feature flags of the root-cause distribution encountered by practicing SREs?

  3. Were all 1,079 plausible-root-cause sets reviewed by humans, or only the 40-task verified subset? How was label completeness assessed?

  4. How much effective task diversity remains after grouping tasks that share the same telemetry window and underlying incident?

  5. Do you expect performance on ORCA-bench to predict performance on larger, evolving, and privately instrumented production systems?

  6. Could the proposed synthesis methodology be used to construct a dataset for post-training? If so, would models trained on this synthetic data generalize to and improve performance on real-world on-call tasks?

2 comments

Thank you for your thoughtful and constructive review. We were delighted that you found the paper provides a substantially richer evaluation environment than existing static RCA benchmarks, explicitly studies important operational factors, and carefully validates its LLM judge against human scoring. We address each question below.

External-validity analysis: To check that our results transfer, we evaluate the same models on 50 real on-call incidents from an early-stage startup. We use the Terminus-2 harness and expose source code, metrics, logs, and traces, matching the ORCA-bench setup as closely as possible. On the real incidents, the models differ significantly (Friedman χ2(3)=23.7\chi^2(3) = 23.7, p<0.001p < 0.001). Opus 4.7, GPT-5.5 and Sonnet 4.6 all outperform DeepSeek-V4-Pro, and Opus 4.7 also outperforms the other two (Wilcoxon signed-rank tests with Holm correction, α=0.05\alpha = 0.05), matching ORCA-bench, where DeepSeek-V4-Pro also ranks last on both RCA accuracy and depth. As on ORCA-bench, the most common failure is locking onto a louder symptom and missing the true trigger. The agent usually blames a nearby commit, PR, or deploy, and sometimes a co-occurring fault such as an upstream outage, much like the case discussed in the paper (Sec. 5).

Label completeness: We checked whether each feature flag within the task’s temporal window could plausibly explain the user-reported issue. When generating both the user-facing issues and the ground-truth answers, we provided the LLM with the mechanism and symptoms, which humans thoroughly verified. By providing the mechanism and symptoms as context, we enabled the LLMs to reliably establish the mapping between feature flags and issues. We verified these mappings on a representative subset of 40 tasks—sampled at random and stratified by issue specificity, scenario, and offset/TTD parameterization, so that every difficulty level, scenario type, and timing setting is covered.

Effective task diversity: The 884 evaluated tasks come from 13 scenarios and 42 fault occurrences. Each fault occurrence appears under several user reports and up to 9 time-to-detection and report-window variants. To check that our conclusions do not rely on these correlated variants, we average each model’s scores within each fault occurrence and repeat the analysis (N=42N = 42). The ranking is unchanged. On RCA accuracy the models differ significantly (Friedman χ2(3)=36.8\chi^2(3) = 36.8, p<0.001p < 0.001); moreover, Sonnet 4.6 outperforms GPT-5.5, and all three of Sonnet 4.6, Opus 4.7 and GPT-5.5 outperform DeepSeek-V4-Pro (Wilcoxon signed-rank tests with Holm correction, α=0.05\alpha = 0.05). On RCA depth the models differ significantly (χ2(3)=66.4\chi^2(3) = 66.4, p<0.001p < 0.001); moreover, Opus 4.7 and GPT-5.5 outperform Sonnet 4.6, which in turn outperforms DeepSeek-V4-Pro (Wilcoxon signed-rank tests with Holm correction, α=0.05\alpha = 0.05). Even when grouping by the 13 scenarios, DeepSeek-V4-Pro ranks last on both metrics.

Post-training experiments: We post-train Qwen3.6-35B-A3B using SFT on ORCA-bench. We split the tasks into training and test sets and train on selected Terminus-2 traces generated by DeepSeek-V4-Pro and GLM-5 from the training split. On held-out tasks evaluated with Terminus-2, SFT yields a 22.2% relative increase in RCA accuracy (33.33% to 40.74%) and a 75.1% relative increase in RCA depth (29.44% to 51.54%).

  • Cross-harness transfer: To assess cross-harness transfer, we evaluate the same fine-tuned model using OpenSRE; SFT yields a 100% relative increase in RCA accuracy (18.52% to 37.04%) and a 161.7% relative increase in RCA depth (20.19% to 52.84%). These gains support transfer across harnesses.
  • Cross-dataset transfer: We evaluate the fine-tuned model on a dataset of 50 real on-call incidents from an early-stage startup. We found that difference in accuracy is not statistically significant under a two-sided paired sign-flip permutation test.

As the scope of this work is primarily in evaluation, we leave post-training as an exciting direction for future work.

Reviewer_VZNtOpenReview ↗

Partially resolved my concerns

Thanks authors for their response. However, only some of my concerns have been addressed, and several weaknesses and questions remain unresolved. Also, after reading the other reviewers' comments, I decided to maintain my original score.

Reviewer_Ab1EOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ORCA-bench, a benchmark for evaluating agents’ performance in root cause analysis (RCA) in cloud incidents. Compared to previous RCA benchmarks, ORCA-bench mainly features its task complexity in terms of delay and ambiguity in human report requesting for RCA. The construction of ORCA-bench starts with a demo app in OpenTelemetry. Then the authors manually cause incidents in the app and generate task instances with different levels of delay for starting RCA and request ambiguity, along with the ground-truth. The authors then evaluate 5 LLMs with Terminus-2 as agent harness with ORCA-bench and using LLM-as-a-Judge. The results show that agents are far from perfect in RCA tasks. Removing context and source code during RCA can further degrade the performance.

Critical strengths
  1. Integrating two real-world factors (delay and ambiguity) into RCA task formulation, which previous benchmarks didn’t cover.
  2. Artifact is available for validating the reproducibility.
Critical weaknesses
  1. Finding 4 reports that the agent fails to construct 26-40% valid telemetry call. But it is unclear to what extent these failures reflect difficulties in using the observability interfaces rather than limitations in investigation ability. Since current LLMs are probably not specifically trained to use these tools, it would be more natural for agents to process JSON files exported from these observability tools. This is similar to the scenarios where agents are good at using CLI rather than GUI interfaces.
  2. The primary novelty and significance of this paper lie in finding 2 and finding 3, especially in the quantitative analysis, where removing context leads to a decrease in performance. It is unclear how the qualitative findings differ from previous works, it is also unclear how the quantitative finding can be generalized to broader RCA scenarios. In addition, given that the delay to detection (i.e., TDD in this paper) is the key design compared to previous benchmarks, there is limited discussion around this factor.
Questions for authors
  1. Could the authors provide a finer-grained breakdown of these unsuccessful queries and evaluate whether a telemetry setup (like OpenRCA) with Python-based analysis improves RCA performance?
  2. Could the authors elaborate more about the distinct findings compared to the existing works?
4 comments

Thank you for your thoughtful and constructive review. We are pleased that you appreciated the paper’s integration of real-world factors into RCA task formulation. We address each question in turn.

Fine-grained breakdown of unsuccessful queries: We attribute every unsuccessful telemetry call to a cause using deterministic rules:

CauseOpus 4.7Sonnet 4.6GPT-5.5GLM-5DeepSeek-V4-Pro
failed in the sandbox (missing tool or module, shell quoting or globbing), not at the backend2.7%2.3%28.1%6.2%11.5%
nonexistent or disabled path, host, index, datasource or dashboard1.0%1.5%7.9%1.3%7.0%
rejected by the backend as invalid2.6%4.5%10.1%7.6%5.9%
exceeded a server-side size, resolution or memory limit0.1%0.1%0.9%0.1%0.8%
backend unreachable or credentials rejected0.0%0.0%0.2%0.0%0.0%
the backend returned data, but the agent's own code failed or printed nothing1.1%1.5%9.7%1.0%0.7%
no stored data in the queried range24.3%15.9%1.8%26.1%18.6%
metric, label, field or service not in the data8.4%8.8%1.4%11.1%9.5%
a required filter value never occurs11.1%13.3%0.2%9.0%9.9%
valid query that matched nothing (an upper bound; includes true absences)46.4%50.5%35.0%36.8%34.7%
cause not visible in the output, even on replay2.2%1.7%4.8%0.8%1.3%
n unsuccessful telemetry queries12,66817,0747,74829,55430,769
n total telemetry queries43,77454,39929,45973,42780,878

Consistent with Fig. 7b, GPT-5.5 genuinely struggles to make valid telemetry queries, whereas all other models struggle with valid but empty queries.

Additional experiment using OpenRCA setup: To further expand the scope of our evaluation, we have conducted a new experiment using the telemetry setup of OpenRCA. We first exported the metrics, logs, and traces from ORCA-bench to CSV files, following the conventions of the Market dataset in OpenRCA, then evaluated four models in the RCA-agent harness. In the revision, we will include the following comparison between Terminus-2 and RCA-agent across the incident tasks in ORCA-bench Verified:

mean ± 1 std error with N=32

HarnessOpus 4.7Sonnet 4.6GPT-5.5DeepSeek-V4-Pro
Terminus-221.9% ± 7.0%25.0% ± 6.2%20.8% ± 6.5%26.0% ± 6.8%
RCA-agent28.1% ± 7.0%29.2% ± 7.1%31.2% ± 7.2%17.7% ± 5.6%

Novelty of finding #2: We designed the benchmark to specifically study how the uncertainty introduced by the user report impacts RCA quality. Prior benchmarks vary other parts of the agent's context, but none vary the report itself. Notably, reports with different specificity require different ground-truth answers, and on average the number of plausible answers grows from 2.00 on Easy to 4.41 on Hard tasks. Even with the number of root causes held fixed, RCA accuracy and depth still fall as the report gets less specific (Fig. L.4). In contrast, the ablations in prior work keep the incident and its labels fixed: OpenRCA (Tab. 2) varies which metrics the model sees and finds that providing golden KPIs improves accuracy; Cloud-OpsBench (Tab. 5) adds past incidents or runbooks and finds gains in certain settings; SREGym (Tab. 3) injects irrelevant background faults and finds that the diagnosis success rate drops.

Novelty of finding #3: No prior work has directly studied the impact of having source code access. Most benchmarks do not expose code at all (AIOpsLab, OpenRCA, ITBench, SREGym, OpenRCA 2.0). Cloud-OpsBench (Table 3) exposes source code through two tools (ListCodeFiles, GetSourceCode) for application-level faults but does not ablate them. The closest experiment is ITBench's trace masking (Tab. 19 & 20), which lowers GPT-4o's diagnosis pass@1 from 18.1% to 9.5%, but it removes a telemetry source rather than code. We run all five models with and without code access on the same 884 incident tasks (Fig. 6). When we average metrics across the five models, we find that removing code lowers RCA accuracy by 47% relative and doubles the hallucination rate (+103%). Incident-time accuracy falls by only 8%, since timing information lives in telemetry, but root-cause-mechanism accuracy drops by 43% (Fig. J.1 & J.5).

Discussion of TTD: In Fig. L.7, we show that the average RCA depth across the five models increases by 5% relative as TTD increases from 15 minutes to 1 hour and decreases by 7% relative as TTD increases from 15 minutes to 8 hours. This impact is small compared to the 49% relative drop when varying the report specificity from Easy to Hard difficulty (finding #2).

I appreciate the authors for the additional experiment and clarification. They address my first concern regarding whether the issue was attributable to the port itself. However, my second concern regarding the novelty and significance of Findings 2 and 3 remains.

Similar observations have been studied in closely related software engineering tasks, particularly bug diagnosis and automated program repair [1, 2]. Thus, the main contribution here appears to be demonstrating that these phenomena also hold in production-style SRE RCA, rather than establishing fundamentally new insights. Moreover, the analysis of Finding 2 does not sufficiently disentangle the effects of different types of missing information or explain how their absence affects agents' diagnostic processes. Overall, these findings provide a moderate extension of existing insights to the SRE RCA setting, although their novelty appears relatively incremental.

[1] Khatib et al., What Makes a Good Bug Report for an AI Agent?, 2026.

[2] Meng et al., LLM-based Agents for Automated Bug Fixing: How Far Are We?, ICSE 2026.

Thank you for the follow-up. We're glad the additional experiment resolved your first concern.

Software engineering and root cause analysis naturally overlap in many ways; indeed, our Finding #3 shows how important source-code access is for RCA. The connections you draw between our work and prior work, namely [1] and [2], add helpful nuance. That said, LLM agents do not yet own the reliability side of the software development life cycle, and compared with existing RCA benchmarks, ORCA-bench takes a meaningful step toward understanding the remaining gaps.

"The analysis of Finding 2 does not sufficiently disentangle the effects of different types of missing information."

Thank you for raising this. By design, going from Easy to Medium removes the symptom/error message from the user-reported issue, and going from Medium to Hard further removes the affected service. Each removal lowers both RCA accuracy and RCA depth for every model. Averaged across the five models, RCA accuracy falls from 41.6% to 21.3% to 6.7%, and RCA depth falls from 51.0% to 37.6% to 26.1%.

"...or explain how their absence affects agents' diagnostic processes."

Interestingly, while each removal increases the number of ground-truth root causes (2.00→2.84→4.412.00 \to 2.84 \to 4.41), the number of root causes agents attribute stays flat at about 1.2, as does the number of output tokens, and the number of agent steps slightly decreases. In other words, agents do not investigate harder when the user-reported issue is vaguer: they attribute about the same number of root causes and therefore cover a shrinking fraction of them, which is consistent with the drop in RCA recall from Easy to Hard.

The table below reports the mean ±\pm 1 standard error across all tasks at each difficulty level, pooled across the five models.

Difficulty# GT root causes# root causes attributed# agent steps# output tokens
Easy (n=288n=288)2.00±0.032.00 \pm 0.031.20±0.021.20 \pm 0.0246.7±0.646.7 \pm 0.624.824.8k ±\pm 0.20.2k
Medium (n=316n=316)2.84±0.042.84 \pm 0.041.20±0.021.20 \pm 0.0245.1±0.645.1 \pm 0.625.225.2k ±\pm 0.20.2k
Hard (n=280n=280)4.41±0.054.41 \pm 0.051.16±0.031.16 \pm 0.0343.5±0.543.5 \pm 0.524.824.8k ±\pm 0.20.2k

Thank you for the further explanations, which have clarified the mechanism behind Finding 2, but this does not materially change my assessment of the novelty/significance of Findings 2 and 3. The benchmark itself may serve as a meaningful new evaluation setting (by integrating two real-world factors), but Findings 2 and 3, as the paper's main insights, remain largely incremental relative to prior work. I therefore regret that I must maintain my original assessment. And I encourage the authors to enhance the paper by conducting more in-depth studies regarding the two new factors.

Comments on the paper

1 comment

We thank all the reviewers for their helpful and detailed feedback on our work. We highlight the following additions to our original submission:

  1. A fine-grained breakdown of unsuccessful queries
  2. Evaluation of other harness, including the telemetry setup from OpenRCA with Python-based analysis and OpenSRE
  3. External-validity analysis comparing the findings on ORCA-bench with findings on real on-call incidents from an early staged startup
  4. Post-training experiments and generalization to other harnesses

We now respond to specific comments and questions from each review below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.