- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents SRE-Marathon, a benchmark that evaluates SRE agents under continuous operation. Other SRE benchmarks mostly involve single-fault episodes, so I believe this is a major difference. There are different fault types injected, including chaos engineering, hand-written code, or configuration changes. Each run is recorded and scored offline without a model judge. The Marathon-Score gives each fault ordered credit for correlation, localization, and repair, and halves the score if the agent attempts any forbidden operation. The evaluation covers ten methods on two models, Kimi K2.5 and GPT-5.6 Luna.
Critical strengths
- SRE-Marathon considers scenarios where faults keep happening in a continuously running system. This setting is more realistic and makes SRE tasks more challenging than in existing benchmarks. Existing benchmarks give the agent one incident, score one answer, and finish, and agents already do well on them (e.g., 72.6% diagnosis success on SREGym). SRE-Marathon instead injects faults into a single live system continuously, so multiple faults exist at the same time. It also evaluates correlation as an SRE agent capability, since one fault can trigger a huge volume of alerts across services, and alerts from different faults mix together.
- There are some interesting findings. I believe these findings are useful for building more autonomous and effective SRE agents.
Critical weaknesses
- The low repair rate may come from the benchmark’s timing, not because the agents are not able to fix them. Each fault lasts only 4 to 12 minutes. After that, the benchmark removes the fault by itself. A repair counts only if the agent fixes the fault before then. But the agent runs only once every 2 minutes, so it gets just two to six chances per fault. Finding the fault already uses one or two of those chances, and deploying a fix can take minutes, since a fix may be naturally harder than detection/correlation. This is my main concern, because "agents cannot repair well" is the paper’s main conclusion. Also, it seems that the score will reward correlation more. If an agent can localize and repair the incidents but correlates wrongly, it will be rewarded with nothing.
Questions for authors
- Can two active faults hit the same service?
- How does SRE-Marathon check whether the injection succeeds? What if the agents perturb the systems and make the service run abnormally instead of injecting the faults successfully?
- Have you observed agents gaming the benchmark? Beyond the forbidden operations and the policy layer, what stops an agent from, for example, opening one incident per alert or rolling back every recently changed service?
- It is possible that the agents repair but do not diagnose correctly (the paper also mentions this, I believe). But I am curious about the reasons behind.
- How are fault durations chosen?
5 comments
Q4. Why can agents repair faults without correctly diagnosing them?
A diagnosis is correct only when both the faulty service and the cause of failure are correctly identified. The cause refers to the responsible artifact for change-driven faults or the failure mechanism for chaos faults. An agent may identify the right service but misidentify the cause, yet still repair it through rollback, which can resolve different types of faults. Such an intervention is effective, but does not demonstrate that the agent correctly understood why the failure occurred. We have clarified this recurring pattern in Appendix J.
Q5. How are fault durations chosen?
Fault lifetimes define the period available for autonomous remediation before a scheduled fallback intervention that models human takeover. We use the approximately five-minute paging-response target for user-facing services (Beyer et al., 2016) as an initial reference for this deadline. We then determine each fault subtype’s duration through offline validation of the full injection-and-reversion lifecycle, allowing additional time as needed to complete deployment, symptom manifestation, and recovery. The resulting durations range from 4 to 12 minutes, with code bugs generally receiving longer windows to accommodate image builds and rollouts:
| Category and defect subtype | Duration (min) |
|---|---|
| Code bug | 8–12 |
| Feature or logic error | 12 |
| Incorrect constant or flag | 12 |
| Component incompatibility | 12 |
| Type, validation, or exception error | 12 |
| Backward incompatibility | 12 |
| Latent defect enabled by a feature flag | 8 |
| Config bug | 4–12 |
| Invalid setting | 12 |
| Uncoordinated change | 12 |
| Inconsistent names or values | 12 |
| Deployed setting | 4–8 |
| Chaos test | 4–10 |
| Deployment error | 4–10 |
| Dependency failure | 8 |
| Infrastructure | 8 |
| Database / network | 8 |
| Auth failure | 8 |
We have added these durations to Table 4 and included a detailed breakdown of “Deployed setting” bug and “Deployment error” chaos test durations in Appendix B. Our response to W1 reports how extending them affects repair success.
Reference: Beyer et al. (2016). Site Reliability Engineering. Chapter 11, “Being On-Call.”
Thank you again for your detailed review. We have posted our responses and uploaded a revised PDF with the changes highlighted in blue.
As the discussion period is drawing to a close, we would be grateful if you could let us know whether our responses and revisions address your concerns. If any point remains unclear or would benefit from further evidence, we are happy to provide additional clarification or results.
Thank you for your time and consideration.
Thank you for the thoughtful review and constructive feedback. We address your concerns about repair performance and scoring, followed by your questions about the benchmark’s design and observed agent behavior. The submission PDF has been updated to address all reviewer feedback, with changes highlighted in blue.
W1. Does the fault duration explain the low repair rate?
Correlation, localization, and repair do not require separate invocations. Within a single invocation, an agent can retrieve evidence, diagnose a fault, and execute a repair through successive LLM inferences and tool calls. Diagnosis therefore does not necessarily consume one or two invocations before remediation begins.
To assess the effect of fault duration, we evaluated a new 56-fault Astronomy Shop schedule with Kimi K2.5, extending every fault’s lifetime by 0, 2, or 4 minutes while keeping fault order, injection times, and agent settings unchanged. The extensions allow one or two additional agent invocations. Repair rates are:
Extending fault lifetimes by four minutes increases repair rates for both methods, supporting the reviewer’s concern that timing affects measured success. However, under this extension, interventions only outside the affected scope or incorrect actions still account for 66.0% of faults for ReAct and 80.4% for Keep-agent. Additional time therefore improves repair outcomes, but does not resolve substantial limitations in selecting appropriate intervention targets and remedies. Because injection times remain fixed, extending fault lifetimes also increases fault overlap.
Our trace analysis across the original backbones likewise identifies both intervention errors and timing constraints: 37.3–68.7% of faults involve actions outside the affected scope, incorrect interventions, or invalid tool arguments, while 4.0–14.7% receive correct remedies too late for verification. Our conclusion concerns timely repair under sustained operation, rather than an inability to repair faults given sufficient time.
We have added the failure analysis and lifetime-extension results to Appendix J.1. Our response to Q5 explains how fault durations are chosen.
W2. Reward for correlation.
Marathon-Score conditions localization and repair credit on successful preceding stages to assess reliable end-to-end incident handling. A repair without correct incident attribution or diagnosis may restore service, but does not establish that the agent correctly connected the symptoms, cause, and remedy. Such repairs still receive credit in the separately reported repair stage metric (Rep). We have added a brief justification for this design in Section 3.
Q1. Can two active faults target the same service?
No. Each fault targets a single service, and each service can have at most one active injected fault. However, their downstream effects can overlap, making alert correlation and diagnosis challenging. We’ve added a clarification to Section 2.
Q2. How is successful fault injection verified?
We validate injections at two levels. Before admitting a fault to the catalog, we verify that injection produces the expected symptoms and that reverting it restores service health (Appendix C). During each run, we verify that the intended change is in place, such as confirming that the patched image is deployed or the configuration update is applied. Thus, injection success is established directly from the system state, even when agent actions also contribute to the observed symptoms. We found no injections that failed due to agent actions across the Table 2 runs. The verification mechanism is now described in Section 2.
Q3. Have agents exhibited strategies that could game the benchmark?
These strategies are not explicitly prohibited, and we observed related behaviors. ORBIT opens essentially one incident per alert, achieving high correlation but near-zero localization and Marathon-Scores of only 25.6–25.9. RAG with Kimi K2.5 executed 63 rollbacks across 11 services in one run without any credited repair; its overall service degradation was also worse than the no-action baseline (150.5%). Thus, these strategies did not yield strong overall performance. We’ve added this analysis to Appendix J.3.
Response to be continued below.