acceptodds
Under review as a conference paper at ICLR 2027

Incident-Arena: Getting agents to the last nine of reliability

Abstract

AI coding agents are ubiquitous in engineering workflows amongst industry and academia. Yet, despite their use in app coding, relatively less attention has been paid to their ability to execute on production incident response. This emerging field, termed agentic site-reliability-engineering (SRE) contains benchmarks limited by (1) unrealistic environments, typically toy repositories (2) non-standard framework implementations and (3) simple static verifiers. We introduce Incident-Arena, a human-built benchmark of 20 carefully selected tasks grounded in real-world deployed open source software. Each task deploys a production application to an ephemeral Kubernetes cluster, injecting a fault from the config layer through underlying images, and a sustained load profile given the task requirements. We also present a novel verification method, going beyond static checks to functional verifiers, holding systems level metrics stable, while ensuring repairs are done safely. Agent trials run an average of 2.81M tokens and 41 turns, going beyond existing benchmarks, demonstrating agentic long horizon reasoning. Across 20 tasks and 3 application substrates, frontier models score below 60%, with failures extending from diagnosis/localization errors, through incomplete repairs and unsafe regressions.

Primary area:
datasets and benchmarks
Keywords:
SRE, Benchmark, Long Horizon, DevOps, Incident Repair, On-Call, Agents, Agentic Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oBisOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Incident-Arena, a new benchmark for testing agents on Site Reliability Engineering tasks. The benchmark differs from existing ones in three main ways, namely Operational Validity, meaning faults are injected in more realistic applications, not toy ones, and evaluation is done using deterministic verifiers that validate the environment state; Diagnostic Resolution is provided by measuring agent capabilities across varying reasoning capabilities; and Reproducibility and Interoperability is achieved using the Harbor Framework. The benchmark 20 human-authored tasks across three substrates, Slack, Saelor, and FrappeERP. Verification is done along two axes, namely an Outcome Gate that measures whether the service-level indicators such as, latency, improve indicating recovering, and a Safety Gate that measures whether the repair persists through a restart, protected data is preserved and operational constraints. Experimental results across a range of frontier models demonstrate the difficulty of the benchmark. The paper also presents a failure taxonomy, and analyzes reward hacking and reasoning effort.

Critical strengths
  1. The benchmark design is sold and realistic. The tasks are human-authored tasks and have been improved via adversarial testing. Evaluation is done via verifiers that measure both outcome success and safety.
  2. Experiments across a range of frontier models demonstrates the complexity of the benchmark.
  3. Detailed analysis of failures is useful.
Critical weaknesses
  1. Benchmark size is small. This is understandable to some extent given the high complexity involved in authoring the tasks but introduces issues, such as the large confidence intervals noted in the paper.
  2. Section 5.2 needs a major rewrite. a. How was this analysis done? b. How can a trial pass without a full fix? c. Why don't the numbers here match the rest of the paper (e.g., declared too early has 376 in Section 5.1 vs. 340 in Section 5.2)
  3. Some writing inconsistencies exist. For example, 2.81M tokens and 41 turns stated in the abstract are not further supported in the paper. Similarly, it states diagnosis/localization errors but that is not a failure category.
  4. There are several typos, such as, "measurincluded duringin."
  5. The reward hacking analysis relies on GPT-6 Astra, which is both an evaluated model and has poor kappa.
Questions for authors
  1. Please provide more details on the QA process used for the failure taxonomy.
  2. What is est. failures in Table 3?
  3. Was the agent informed about the following: a. constraints that the safety gate measures, b. that it must undo the damage done by the fault, c. it must verify that the fault has been cleared after applying the fix.
1 comment
AuthorsOpenReview ↗

R3 (oBis) Response

We sincerely and genuinely thank R3, oBis, for their feedback and support in this work.

  1. Benchmark size : This concern is addressed above in the general comment.
  2. Sec 5.2 Analysis: (a) Method. The original analysis compared actions in agent trajectories with the scripted reference repair (oracle). We will rewrite this section to distinguish oracle coverage and verified recovery.

(b) Passing without a “full fix.” “Full fix” referred to matching all reference actions (oracle), not satisfying the complete repair contract. An alternative valid repair can pass without matching that action list in the oracle. An oracle is only needed to show that the task is solvable, and in RL task creation, we are not prescriptive about the corrective actions. We will remove this misleading terminology.

(c) Inconsistent counts. Sections 5.1 and 5.2 apply different selection criteria, including reference-action coverage in Section 5.2. We will replace the unreconciled 376/340 comparison with consistently defined results from one frozen trial manifest, stating each denominator explicitly.

  1. Writing inconsistencies: We will report the calculated averages: 2.88M recorded input + output tokens across 3,000 trials, and 39.08 turns. We will also remove the unsupported “diagnosis/localization errors” characterization from the abstract.

  2. Typos: We thank the reviewer & sincerely apologies for typos. We will scrub & fix the typos in the camera-ready.

  3. RH analysis:Appendix K compares four judges: GPT-6 Astra, Claude Opus 5, Claude Opus 5.5, and Gemini 3.8 Flash. Rank concordance is higher, Kendall's W = 0.756. We agree that Astra judging its own trajectories could introduce bias. We will make clear that these are exploratory LLM annotations and benchmark pass/fail is determined independently by executable verifiers.

  4. QA process: We will distinguish task-level QA from validation from the post-trial trajectory labels. Revised camera-ready Section 5.1 will report categories derived directly from recorded outcome and safety outputs, while Section 5.2 will provide selected trajectory examples. Our QA process includes (1) adversarial QA (2) LLM-as-a-judge QA (3) pretrial-check QA and (4) post-trail check QA.

  5. Table 3 We will replace it with recorded outcomes

\text{Outcome / safety} & \text{Trials} & \text{Share (\\%)} \\\\ \hline \text{Pass / pass} & 983 & 32.77 \\\\ \text{Pass / fail} & 850 & 28.33 \\\\ \text{Fail / pass} & 408 & 13.60 \\\\ \text{Fail / fail} & 759 & 25.30 \\\\ \hline \text{Total} & 3{,}000 & 100.00 \end{array}$$ Regarding Table 3: Categories are assigned in priority order, so each failure contributes to one category even when several conditions apply. We will distinguish observed counts and make the denominator explicit. $$\begin{array}{lrr} \text{Category} & \text{Count} & \text{Share} \\\\ \hline \text{Never declared} & 394 & 19.5\\% \\\\ \text{Off permitted list} & 428 & 21.2\\% \\\\ \text{Cause or damage left} & 334 & 16.6\\% \\\\ \text{Repair did not last} & 53 & 2.6\\% \\\\ \text{Declared too early} & 375 & 18.6\\% \\\\ \text{Other failures} & 433 & 21.5\\% \\\\ \hline \text{Total} & 2{,}017 & 100\\% \end{array}$$ We will release full trajectories, and annotations for the camera-ready. 8. **Agent instruction constraints**: Examples from the agent-visible instructions used in the main collection. | Question | Exact instruction excerpt | |---|---| | Safety rules | “Preserve queued work and keep queue capacity finite and loss-intolerant through a broker restart.” | | Undo damage | “Make sure both problems are resolved.” This refers to record creation and queued jobs; full state restoration is not specified. | | Verify recovery | “Before handing this back, make sure checkouts stay healthy through another busy period and across new database sessions.” | Therefore, the agents are given the gates but only insofar as it models human-style approaches to prompting, without being too specific on the verifier gates.
Reviewer_gboBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Incident-Arena, a benchmark for evaluating agents on incident recovery in running application environments. It comprises 20 human-authored tasks across Saleor, FrappeERP, and a custom Slack-like system, with a deterministic verifier that checks both operational recovery under sustained traffic and the durability and safety of repairs. The authors conduct 3,000 trials across different model–harness configurations and reasoning levels, and analyze the resulting trajectories to characterize failure modes.

Critical strengths
  1. Practically relevant evaluation setting. Evaluating agents on running systems moves beyond static diagnosis or repository-level changes and addresses an important problem in operational reliability.
  2. Comprehensive success criteria. The outcome and safety gates check not only whether service behavior recovers, but also whether repairs persist and preserve protected data and operational boundaries. Reference-solution and no-operation checks further strengthen the evaluation.
Critical weaknesses

Limited task scale and coverage. The benchmark contains only 20 tasks, with 13 concentrated in the custom Slack-like system, six in FrappeERP, and only one in Saleor. Although constructing executable incident-recovery tasks requires substantial effort, this small and uneven task set raises concerns about its coverage of production failure modes and the generalizability of the findings. The 3,000 trials provide repeated measurements on these tasks but do not increase the diversity of underlying scenarios. The authors acknowledge this limitation, but a clearer account of task selection, coverage, and sensitivity to individual tasks would strengthen the contribution.

Questions for authors
  1. What are the main bottlenecks to expanding the benchmark, and is there a concrete process for adding tasks from additional applications or documented production incidents?
2 comments
AuthorsOpenReview ↗

R2 (gboB) Response

We sincerely and genuinely thank R2, gboB, for their feedback and support in this work.

  1. Limited Task Scale & Coverage: We discuss the size of the task set and the sensitivity of the results to removing Slack in the general response. We agree that 20 tasks cannot cover the entirety & full range of production incidents, and will call this out as a limitation for the camera ready.
  2. Task Selection, Coverage & Task sensitivity: Contributors and the authors built tasks to test SRE incidents and agent capabilities across configuration, runtime, and image faults. For each task, we ran a scripted reference repair and a no-operation trial. In the reported experiment, the reference repair passed and the no-operation trial failed on all 20 tasks. Every task also had at least one passing agent trial. We will describe the selection and qualification process more clearly in the paper.
  3. Bottleneck in expansion Process: Each new task needs a reproducible workload, a repair path available to the agent, and checks for recovery, data preservation, and durability. Developing and qualifying these components takes substantial manual work, which limits how quickly the benchmark can grow. For a senior SWE, this is typically a minimum of 1 week to develop each task, outside of substrate development. Consequently, developing these tasks at scale is difficult. Nonetheless, we will document this limitation in the core paper as well.

Thank you for the response and clarifications. I have reviewed the rebuttal. The size of this benchmark is rather small. While I look forward to the authors’ future improvements, I believe my original assessment remains appropriate. Therefore, I will maintain my original score.

Reviewer_3YHuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Incident-Arena, a benchmark designed to evaluate AI agents' ability to diagnose and repair incidents in live software systems. The benchmark consists of 20 human-authored tasks across three application substrates, with faults injected at the configuration, runtime, and image layers of Kubernetes-based environments. The authors develop a deterministic, LLM-free verification framework that evaluates both service recovery under sustained workloads and the safety and durability of repairs. Through 3,000 evaluation trials across 10 model–harness configurations and multiple reasoning effort levels, the paper finds that even the strongest agents achieve at most 60% Pass@1, with common failure modes including non-durable repairs, premature completion declarations, and modifications outside permitted scopes.

Critical strengths
  1. Meaningful research problem. The paper investigates an important and relatively underexplored capability of AI agents: diagnosing and repairing incidents in live software systems. Extending agent evaluation beyond conventional coding tasks to production-oriented incident response is a meaningful research direction.

  2. Realistic and executable benchmark environments. Instead of relying solely on static repositories or simplified diagnostic scenarios, Incident-Arena deploys multi-service applications in Kubernetes environments with configurable faults and sustained workloads. This design enables the evaluation of agents' ability to interact with and repair running systems.

  3. Comprehensive verification of repair outcomes. The proposed dual-gate verification framework considers not only whether a service recovers but also whether the repair is durable, respects operational constraints, and preserves system integrity. The deterministic, LLM-free evaluation also avoids the uncertainty associated with LLM-based grading.

  4. Detailed empirical analysis of agent failures. Beyond reporting aggregate success rates, the paper analyzes 3,000 agent trajectories to identify different failure patterns, including incomplete repairs, premature completion declarations, and unsafe modifications. The analysis of reasoning effort and post-fix behavior provides additional insights into the limitations of current agents in incident-response tasks.

Critical weaknesses
  1. Limited benchmark coverage and diversity. The benchmark contains only 20 tasks across three application substrates, with 13 tasks concentrated in the authors' self-developed Slack-like system. Moreover, fault types, onset conditions, and application substrates are confounded. This limited and imbalanced coverage makes it difficult to determine how well the findings generalize to broader production systems and incident scenarios.

  2. Insufficient validation of task solvability under agent access constraints. The benchmark imposes different information and permission constraints on agents, ranging from restricted operational APIs to source-code access. Although the authors validate tasks using golden solutions and no-op trials, it remains unclear whether these checks ensure that each task is solvable using only the information, tools, and permissions available to the agent. Without such validation, some failures may result from insufficient observability or inaccessible repair actions rather than limitations in agent capabilities, potentially affecting the validity of the benchmark.

  3. Evaluation Metrics and Statistical Reporting. The evaluation metrics and reporting could be further improved. First, while Pass@1 is used to represent the average success rate across repeated trials, a clearer distinction between single-trial success rates and aggregated scores would improve interpretability. Second, since the authors conduct three independent trials per setting, reporting Pass@3 alongside Pass@1 could provide additional insights into agent reliability across repeated attempts. Third, Table 2 averages Pass@1 across different reasoning effort levels, which may obscure performance differences and complicate comparisons between models with different effort configurations. Reporting results separately for each model–effort configuration would provide a more informative and transparent comparison.

Questions for authors
  1. Reliability of failure classification. Section 5.1 presents a failure taxonomy and mentions a rule-based classification procedure supported by human-in-the-loop QA. Could the authors clarify how the failure labels were assigned to individual agent trajectories? Was the classification fully automated, or did it involve LLM-based or human judgments? If automated or LLM-assisted methods were used, was the classification accuracy validated against human annotations, and what level of agreement was achieved?

  2. Impact of domain-specific skills. Since incident response is a specialized domain requiring substantial operational knowledge, have the authors considered evaluating agents equipped with SRE-specific skills or application-specific guidance (e.g., system architecture, diagnostic procedures, and operational best practices)? It would be interesting to investigate whether such knowledge augmentation could improve agent performance compared with the current settings.

1 comment
AuthorsOpenReview ↗

R1 (3YHu) Response

We genuinely thank R1, 3YHu, for their feedback and support in this work.

  1. Limited benchmark coverage and diversity: We address this concern above in the general comment.

  2. Insufficient validation of task solvability under agent access constraints.: Yes, all 20 tasks have been validated with agent-surface area specific oracle & nops. We have audited every oracle and both the agent trajectories to prove this out. Furthermore, every single task has at least one passing agent trial under the evaluated setting, showing every task the passing criteria available.

  3. Evaluation Metrics and Statistical Reporting We recognize this gap, and will add the following table to the paper: https://pastebin.com/j43tcZTM .The full table below reports all 50 model–effort settings, with Pass@1, Pass@3, and 95% confidence intervals. Pass@3 means at least one success in three attempt. (The full table is extremely large and cannot fit in this comment, please view the pastebin).

  4. Reliability of Failure Classification: We will revise Section 5.1 to report categories specifically from recorded outcome and safety outputs. Section 5.2 will use examples illustrate repair progress and incomplete verification. The separate LLM-based reward-hacking analysis and its inter-judge agreement will remain explicitly identified as post-trial annotation.

  5. SRE-Skills: We ablate the use of the skills from [1] using GPT-6-Astra on xhigh across all 20 tasks. The gap between the baseline agent and the skills-attached agent was only an increase of +1.8% ±\pm 5.2% (95% CI) which is functionally no effect. We can add the specified tables and information to the Camera-ready Appendix.

[1] https://github.com/Rootly-AI-Labs/SRE-skills-bench

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Author Response

We sincerely and genuinely thank the reviewers for their thoughtful & responses. Reviewing is difficult work, and we appreciate your careful feedback.

We will address the shared concerns here:

R1 (3YHu), R2 (gboB), R3 (oBis) : Coverage & Reliability

We agree that 20 tasks is incomparable to other synthetic or automated benchmarks but we argue that this range of tasks is typical. Especially for manually human-created tasks, 20 tasks is quite common, seen in : [1] SWE-Marathon has 20 tasks, [2] APEX-SWE observability track has 25 open sourced tasks and [3] FrontierSWE which has 34 tasks total (14 of which are implementation tasks) , and finally [4] SRE-Gym-Lite has 17 tasks (recently used in Devin Oncall's benchmark [5])

As such we show that this range of tasks is typical, especially for hand-made tasks.

Individual task & Substrate sensitivity

We further address the concern that the individual task reliability is a core concern, by ablating the individual tasks and recording the Spearman rank correlation. The lowest individual task spearman correlation was ρ=0.986\mathbf{\rho = 0.986} . This is an extremely strong correlation between the final ranking and the individual task reliability.

On further concerns on the fat tail of slack task, we ablate the substrates (Slack, FrappeERP, Saleor) to demonstrate the there is still a correlation despite the fat tail, as seen below:

\text{Application} & \text{Task weight} & \text{Rank correlation if omitted} \\\\ \hline \text{Frappe} & 30\\% & 0.943 \\\\ \text{Saleor} & 5\\% & 0.995 \\\\ \text{Slack} & 65\\% & 0.636 \end{array}$$ The remaining spearman correlation for removing Slack is $\mathbf{\rho = 0.636}$, which despite the rest of the substrates compromising 35% of the dataset demonstrates positive correlation. We will add the per-substrate results & call this out as a limitation for the camera ready. [1] SWE-Marathon (https://arxiv.org/pdf/2606.07682) [2] Apex-SWE (https://arxiv.org/pdf/2601.08806) [3] Frontier-SWE (https://www.frontierswe.com/tasks) [4] SRE-Gym (https://arxiv.org/pdf/2605.07161) [5] https://devin.ai/blog/introducing-devin-oncall

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.