- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents Incident-Arena, a new benchmark for testing agents on Site Reliability Engineering tasks. The benchmark differs from existing ones in three main ways, namely Operational Validity, meaning faults are injected in more realistic applications, not toy ones, and evaluation is done using deterministic verifiers that validate the environment state; Diagnostic Resolution is provided by measuring agent capabilities across varying reasoning capabilities; and Reproducibility and Interoperability is achieved using the Harbor Framework. The benchmark 20 human-authored tasks across three substrates, Slack, Saelor, and FrappeERP. Verification is done along two axes, namely an Outcome Gate that measures whether the service-level indicators such as, latency, improve indicating recovering, and a Safety Gate that measures whether the repair persists through a restart, protected data is preserved and operational constraints. Experimental results across a range of frontier models demonstrate the difficulty of the benchmark. The paper also presents a failure taxonomy, and analyzes reward hacking and reasoning effort.
Critical strengths
- The benchmark design is sold and realistic. The tasks are human-authored tasks and have been improved via adversarial testing. Evaluation is done via verifiers that measure both outcome success and safety.
- Experiments across a range of frontier models demonstrates the complexity of the benchmark.
- Detailed analysis of failures is useful.
Critical weaknesses
- Benchmark size is small. This is understandable to some extent given the high complexity involved in authoring the tasks but introduces issues, such as the large confidence intervals noted in the paper.
- Section 5.2 needs a major rewrite. a. How was this analysis done? b. How can a trial pass without a full fix? c. Why don't the numbers here match the rest of the paper (e.g., declared too early has 376 in Section 5.1 vs. 340 in Section 5.2)
- Some writing inconsistencies exist. For example, 2.81M tokens and 41 turns stated in the abstract are not further supported in the paper. Similarly, it states diagnosis/localization errors but that is not a failure category.
- There are several typos, such as, "measurincluded duringin."
- The reward hacking analysis relies on GPT-6 Astra, which is both an evaluated model and has poor kappa.
Questions for authors
- Please provide more details on the QA process used for the failure taxonomy.
- What is est. failures in Table 3?
- Was the agent informed about the following: a. constraints that the safety gate measures, b. that it must undo the damage done by the fault, c. it must verify that the fault has been cleared after applying the fix.
R3 (oBis) Response
We sincerely and genuinely thank R3, oBis, for their feedback and support in this work.
(b) Passing without a “full fix.” “Full fix” referred to matching all reference actions (oracle), not satisfying the complete repair contract. An alternative valid repair can pass without matching that action list in the oracle. An oracle is only needed to show that the task is solvable, and in RL task creation, we are not prescriptive about the corrective actions. We will remove this misleading terminology.
(c) Inconsistent counts. Sections 5.1 and 5.2 apply different selection criteria, including reference-action coverage in Section 5.2. We will replace the unreconciled 376/340 comparison with consistently defined results from one frozen trial manifest, stating each denominator explicitly.
-
-
-
-
-
\text{Outcome / safety} & \text{Trials} & \text{Share (\\%)} \\\\ \hline \text{Pass / pass} & 983 & 32.77 \\\\ \text{Pass / fail} & 850 & 28.33 \\\\ \text{Fail / pass} & 408 & 13.60 \\\\ \text{Fail / fail} & 759 & 25.30 \\\\ \hline \text{Total} & 3{,}000 & 100.00 \end{array}$$ Regarding Table 3: Categories are assigned in priority order, so each failure contributes to one category even when several conditions apply. We will distinguish observed counts and make the denominator explicit. $$\begin{array}{lrr} \text{Category} & \text{Count} & \text{Share} \\\\ \hline \text{Never declared} & 394 & 19.5\\% \\\\ \text{Off permitted list} & 428 & 21.2\\% \\\\ \text{Cause or damage left} & 334 & 16.6\\% \\\\ \text{Repair did not last} & 53 & 2.6\\% \\\\ \text{Declared too early} & 375 & 18.6\\% \\\\ \text{Other failures} & 433 & 21.5\\% \\\\ \hline \text{Total} & 2{,}017 & 100\\% \end{array}$$ We will release full trajectories, and annotations for the camera-ready. 8. **Agent instruction constraints**: Examples from the agent-visible instructions used in the main collection. | Question | Exact instruction excerpt | |---|---| | Safety rules | “Preserve queued work and keep queue capacity finite and loss-intolerant through a broker restart.” | | Undo damage | “Make sure both problems are resolved.” This refers to record creation and queued jobs; full state restoration is not specified. | | Verify recovery | “Before handing this back, make sure checkouts stay healthy through another busy period and across new database sessions.” | Therefore, the agents are given the gates but only insofar as it models human-style approaches to prompting, without being too specific on the verifier gates.Writing inconsistencies: We will report the calculated averages: 2.88M recorded input + output tokens across 3,000 trials, and 39.08 turns. We will also remove the unsupported “diagnosis/localization errors” characterization from the abstract.
Typos: We thank the reviewer & sincerely apologies for typos. We will scrub & fix the typos in the camera-ready.
RH analysis:Appendix K compares four judges: GPT-6 Astra, Claude Opus 5, Claude Opus 5.5, and Gemini 3.8 Flash. Rank concordance is higher, Kendall's W = 0.756. We agree that Astra judging its own trajectories could introduce bias. We will make clear that these are exploratory LLM annotations and benchmark pass/fail is determined independently by executable verifiers.
QA process: We will distinguish task-level QA from validation from the post-trial trajectory labels. Revised camera-ready Section 5.1 will report categories derived directly from recorded outcome and safety outputs, while Section 5.2 will provide selected trajectory examples. Our QA process includes (1) adversarial QA (2) LLM-as-a-judge QA (3) pretrial-check QA and (4) post-trail check QA.
Table 3 We will replace it with recorded outcomes