acceptodds
Under review as a conference paper at ICLR 2027

SRE-Marathon: A Continuous, Change-Driven Benchmark for Autonomous Site Reliability Agents

Abstract

Benchmarks for site reliability engineering (SRE) agents are typically episodic: one fault is injected, the agent receives an incident task, and its response is scored. Production operation is not. Incidents surface through noisy alerts, overlap in time, and often originate from code or configuration changes. We present SRE-Marathon, a benchmark for long-horizon, continuous SRE operation. An agent is invoked at a fixed cadence with cumulative alert history and a persistent workspace while operating a live two-zone Kubernetes deployment as a fault orchestrator injects overlapping faults according to a seeded, production-calibrated schedule. Curated code and configuration changes deployed through the same build pipeline available for repair. Each run is recorded into a sealed bundle and scored offline: Marathon-Score credits each injected fault for ordered progress through correlation, localization, and repair, with all metrics computed deterministically from recorded system evidence. Across three applications and about sixty faults per run, the best of 10 methods reaches only 41.3 out of 100. Agents often correlate and localize faults, but almost never complete repairs while the faults remain active.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, benchmark, AIOps, long-horizon evaluation, site reliability engineering, root cause analysis, automated remediation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_kzHGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SRE-Marathon, a benchmark that evaluates SRE agents under continuous operation. Other SRE benchmarks mostly involve single-fault episodes, so I believe this is a major difference. There are different fault types injected, including chaos engineering, hand-written code, or configuration changes. Each run is recorded and scored offline without a model judge. The Marathon-Score gives each fault ordered credit for correlation, localization, and repair, and halves the score if the agent attempts any forbidden operation. The evaluation covers ten methods on two models, Kimi K2.5 and GPT-5.6 Luna.

Critical strengths
  1. SRE-Marathon considers scenarios where faults keep happening in a continuously running system. This setting is more realistic and makes SRE tasks more challenging than in existing benchmarks. Existing benchmarks give the agent one incident, score one answer, and finish, and agents already do well on them (e.g., 72.6% diagnosis success on SREGym). SRE-Marathon instead injects faults into a single live system continuously, so multiple faults exist at the same time. It also evaluates correlation as an SRE agent capability, since one fault can trigger a huge volume of alerts across services, and alerts from different faults mix together.
  2. There are some interesting findings. I believe these findings are useful for building more autonomous and effective SRE agents.
Critical weaknesses
  1. The low repair rate may come from the benchmark’s timing, not because the agents are not able to fix them. Each fault lasts only 4 to 12 minutes. After that, the benchmark removes the fault by itself. A repair counts only if the agent fixes the fault before then. But the agent runs only once every 2 minutes, so it gets just two to six chances per fault. Finding the fault already uses one or two of those chances, and deploying a fix can take minutes, since a fix may be naturally harder than detection/correlation. This is my main concern, because "agents cannot repair well" is the paper’s main conclusion. Also, it seems that the score will reward correlation more. If an agent can localize and repair the incidents but correlates wrongly, it will be rewarded with nothing.
Questions for authors
  1. Can two active faults hit the same service?
  2. How does SRE-Marathon check whether the injection succeeds? What if the agents perturb the systems and make the service run abnormally instead of injecting the faults successfully?
  3. Have you observed agents gaming the benchmark? Beyond the forbidden operations and the policy layer, what stops an agent from, for example, opening one incident per alert or rolling back every recently changed service?
  4. It is possible that the agents repair but do not diagnose correctly (the paper also mentions this, I believe). But I am curious about the reasons behind.
  5. How are fault durations chosen?
5 comments

Thank you for the thoughtful review and constructive feedback. We address your concerns about repair performance and scoring, followed by your questions about the benchmark’s design and observed agent behavior. The submission PDF has been updated to address all reviewer feedback, with changes highlighted in blue.

W1. Does the fault duration explain the low repair rate?

Correlation, localization, and repair do not require separate invocations. Within a single invocation, an agent can retrieve evidence, diagnose a fault, and execute a repair through successive LLM inferences and tool calls. Diagnosis therefore does not necessarily consume one or two invocations before remediation begins.

To assess the effect of fault duration, we evaluated a new 56-fault Astronomy Shop schedule with Kimi K2.5, extending every fault’s lifetime by 0, 2, or 4 minutes while keeping fault order, injection times, and agent settings unchanged. The extensions allow one or two additional agent invocations. Repair rates are:

MethodOriginal lifetime+2 min+4 min
ReAct3.6%3.6%10.7%
Keep-agent7.1%3.6%14.3%

Extending fault lifetimes by four minutes increases repair rates for both methods, supporting the reviewer’s concern that timing affects measured success. However, under this extension, interventions only outside the affected scope or incorrect actions still account for 66.0% of faults for ReAct and 80.4% for Keep-agent. Additional time therefore improves repair outcomes, but does not resolve substantial limitations in selecting appropriate intervention targets and remedies. Because injection times remain fixed, extending fault lifetimes also increases fault overlap.

Our trace analysis across the original backbones likewise identifies both intervention errors and timing constraints: 37.3–68.7% of faults involve actions outside the affected scope, incorrect interventions, or invalid tool arguments, while 4.0–14.7% receive correct remedies too late for verification. Our conclusion concerns timely repair under sustained operation, rather than an inability to repair faults given sufficient time.

We have added the failure analysis and lifetime-extension results to Appendix J.1. Our response to Q5 explains how fault durations are chosen.

W2. Reward for correlation.

Marathon-Score conditions localization and repair credit on successful preceding stages to assess reliable end-to-end incident handling. A repair without correct incident attribution or diagnosis may restore service, but does not establish that the agent correctly connected the symptoms, cause, and remedy. Such repairs still receive credit in the separately reported repair stage metric (Rep). We have added a brief justification for this design in Section 3.

Q1. Can two active faults target the same service?

No. Each fault targets a single service, and each service can have at most one active injected fault. However, their downstream effects can overlap, making alert correlation and diagnosis challenging. We’ve added a clarification to Section 2.

Q2. How is successful fault injection verified?

We validate injections at two levels. Before admitting a fault to the catalog, we verify that injection produces the expected symptoms and that reverting it restores service health (Appendix C). During each run, we verify that the intended change is in place, such as confirming that the patched image is deployed or the configuration update is applied. Thus, injection success is established directly from the system state, even when agent actions also contribute to the observed symptoms. We found no injections that failed due to agent actions across the Table 2 runs. The verification mechanism is now described in Section 2.

Q3. Have agents exhibited strategies that could game the benchmark?

These strategies are not explicitly prohibited, and we observed related behaviors. ORBIT opens essentially one incident per alert, achieving high correlation but near-zero localization and Marathon-Scores of only 25.6–25.9. RAG with Kimi K2.5 executed 63 rollbacks across 11 services in one run without any credited repair; its overall service degradation was also worse than the no-action baseline (150.5%). Thus, these strategies did not yield strong overall performance. We’ve added this analysis to Appendix J.3.

Response to be continued below.

Q4. Why can agents repair faults without correctly diagnosing them?

A diagnosis is correct only when both the faulty service and the cause of failure are correctly identified. The cause refers to the responsible artifact for change-driven faults or the failure mechanism for chaos faults. An agent may identify the right service but misidentify the cause, yet still repair it through rollback, which can resolve different types of faults. Such an intervention is effective, but does not demonstrate that the agent correctly understood why the failure occurred. We have clarified this recurring pattern in Appendix J.

Q5. How are fault durations chosen?

Fault lifetimes define the period available for autonomous remediation before a scheduled fallback intervention that models human takeover. We use the approximately five-minute paging-response target for user-facing services (Beyer et al., 2016) as an initial reference for this deadline. We then determine each fault subtype’s duration through offline validation of the full injection-and-reversion lifecycle, allowing additional time as needed to complete deployment, symptom manifestation, and recovery. The resulting durations range from 4 to 12 minutes, with code bugs generally receiving longer windows to accommodate image builds and rollouts:

Category and defect subtypeDuration (min)
Code bug8–12
Feature or logic error12
Incorrect constant or flag12
Component incompatibility12
Type, validation, or exception error12
Backward incompatibility12
Latent defect enabled by a feature flag8
Config bug4–12
Invalid setting12
Uncoordinated change12
Inconsistent names or values12
Deployed setting4–8
Chaos test4–10
Deployment error4–10
Dependency failure8
Infrastructure8
Database / network8
Auth failure8

We have added these durations to Table 4 and included a detailed breakdown of “Deployed setting” bug and “Deployment error” chaos test durations in Appendix B. Our response to W1 reports how extending them affects repair success.

Reference: Beyer et al. (2016). Site Reliability Engineering. Chapter 11, “Being On-Call.”

Thank you again for your detailed review. We have posted our responses and uploaded a revised PDF with the changes highlighted in blue.

As the discussion period is drawing to a close, we would be grateful if you could let us know whether our responses and revisions address your concerns. If any point remains unclear or would benefit from further evidence, we are happy to provide additional clarification or results.

Thank you for your time and consideration.

Reviewer_95nLOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SRE-Marathon, a benchmark for long-horizon, continuous operations of site reliability engineering (SRE) agents. In SRE-Marathon, an agent is invoked at a fixed cadence with cumulative alert history and a persistent workspace to fix issues in a live Kubernetes deployment that another fault orchestrator injects based on a seeded scheduling system. The agents are scored based on their progress on correlation, localization, and repair. Across three applications and ten methods for evaluation with Kimi K2.5 and GPT-5.6 Luna, the best method scores 41.3 out of 100.

Critical strengths
  1. Compared to existing SRE benchmarks, SRE-Marathon evaluates agents in a more realistic continuous serving operation scenario, where agents need to infer incidents from noisy alert streams and the faults arrive continuously overlapping across services.
Critical weaknesses
  1. Faults are automatically reverted after a scheduled 4–12-minute lifetime. This limits realism for persistent code and configuration defects and may disadvantage repairs requiring builds and deployments.
  2. Evaluation uses only two non-frontier model backbones (Kimi K2.5 and GPT-5.6 Luna), leaving it unclear how much the reported failure modes generalize to stronger models and how model capability interacts with the harness constraints.
Questions for authors
  1. How did you choose the 4--12 minute window for faults to be reverted? Can you discuss how realistic this design is?
  2. How much of the limitations discussed in the paper would be resolved if we switch to stronger models?
3 comments

Thank you for the thoughtful review and constructive comments. We address your concerns about fault durations and model capability below, including new results with a stronger backbone. The submission PDF has been updated to address all reviewer feedback, with changes highlighted in blue.

W1 / Q1. Choice and realism of fault durations.

Fault durations are informed by the approximately five-minute paging-response target for user-facing services (Beyer et al., 2016), which specifies when an on-call engineer begins responding. Code bugs receive longer durations to accommodate image builds and rollouts. We then validate injection and reversion offline and extend durations where needed for deployment, symptom manifestation, and recovery.

CategoryDuration (min)
Code bug8–12
Config bug4–12
Chaos test4–10

Scheduled reversion represents an external fallback intervention, rather than suggesting that persistent defects disappear naturally. We acknowledge that the resulting deadlines can disadvantage slower repairs; the benchmark evaluates timely remediation within these windows. We have added the duration of each fault subtype to Table 4 in Appendix B and clarified this design choice.

Reference: Beyer et al. (2016). Site Reliability Engineering. Chapter 11, “Being On-Call.”

W2 / Q2. Would stronger models resolve the observed limitations?

We reran all nine LLM methods with GPT-6.1 Sol, with reasoning set to none as for GPT-5.6 Luna, using the same schedules, harness, and $8 per-run budget. Results below are averaged over the three applications.

MethodCorrLocRepMarathon-ScoreΔ vs Kimi K2.5Δ vs GPT-5.6 Luna
ReAct†0.6250.4670.06236.9+11.5+20.6
RAG0.6550.4840.02235.0+11.5+26.8
ORBIT†0.0450.0450.0002.3−23.3−23.6
OpenSRE†0.2030.0390.0287.5+3.8−0.8
Keep-agent0.8170.2500.00635.2−6.1+3.9
Codex†0.4470.1310.02318.5+10.4−1.9
Claude Code†0.2150.0680.0288.9−4.5−5.4
OpsMem0.6260.1190.00024.8+5.0+3.8
COCA0.6550.3590.02331.0+11.4+3.0
All methods0.4770.2180.02122.2+2.2+2.9

† Exhausted the budget before completion in at least one application. These runs remain scored over the full schedule, with no additional credit earned during the remaining portion.

The stronger backbone improves the average Marathon-Score from 20.0 (Kimi K2.5) and 19.3 (GPT-5.6 Luna) to 22.2. The main gain is localization, which increases from approximately 12–15% to 21.8%; repair increases from approximately 0.6% to 2.1%, but remains rare. ReAct and RAG benefit most, reaching localization rates of 46.7% and 48.4%.

The improvements depend on how each method uses the stronger model. With Sol, ORBIT spends 84–96% of its model calls repeatedly validating incidents and publishes only about five incidents per run. This reduces its correlation score from 0.77 to 0.05 and exhausts the budget in some runs. Claude Code attempts more interventions, but approximately one-third of faults receive an incorrect intervention. Overall, stronger models substantially improve localization, while effective repair still depends on choosing appropriate actions and using the available budget efficiently.

Reviewer_95nLOpenReview ↗

Stronger-model results address my main concern, but repair deadlines still matter

I’m maintaining my weak accept. The benchmark makes a useful contribution by testing whether agents can identify and repair overlapping faults from a shared stream of noisy alerts. The authors addressed my main empirical question by rerunning all nine LLM methods with a stronger model.

My remaining reservation concerns the 4–12-minute repair windows. A five-minute paging target tells us when an engineer should begin responding, but offers little basis for when a repair should be complete. On one schedule, adding four minutes roughly doubled or tripled repair rates for the two methods reported. I appreciate that the authors now frame their conclusion around timely repair under the tested deadlines.

Thank you for the thoughtful follow-up. Our fault lifetimes define the period available for autonomous remediation before a scheduled fallback intervention, intended to model human intervention. The longer-window results show that these deadlines materially affect repair rates, so our conclusions concern sustained incident handling under the specified time constraints. We appreciate your constructive feedback on this distinction.

Reviewer_RdwvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a new benchmark: SRE-Marathon. It addresses the problem that current benchmarks follow a pattern in which a single fault is injected, the agent receives one incident task, and its response is scored. However, this state-of-the-art does not reflect a real production environment, for example due to parallel incidents that may occur -- SRE-Marathon aims to close this gap. Three applications (Astronomy Shop, Online Boutique, Train Ticket) were tested with a total of 700 incidents. Ten different methods were tested with two different models. The best model combined with the best method achieved 41.3 out of 100 possible points.

Critical strengths

The new benchmark addresses the problem that current benchmarks do not cover a real, running production environment. As a result, benchmark results are only transferable to a limited extent. The SRE-Marathon benchmark newly takes up this problem and attempts to be a more realistic benchmark whose results and insights transfer better to a real production environment.

Critical weaknesses
  • A baseline is missing showing how the models and methods perform when no faults are injected, or just single, individual faults. Only by comparing "What were the models and methods able to achieve when multiple faults were not present in parallel?" and "What were the models and methods able to achieve when multiple faults were present in parallel?" can the question be answered whether such a benchmark is needed at all. Currently, the paper claims that existing benchmarks do not reflect reality, but whether the benchmark yields better insights is not supported by the paper's results. Although Appendix I provides some initial insights, a stronger focus should be placed on this.

  • According to Section 2, the orchestrator reverts every fault after 4–12 minutes. It remains unclear how agent interventions that are already live are handled. An example: At time t1, the agent attempts to fix a fault. The change does not solve the problem but is deployed. At time t2, the agent attempts to fix a different fault, based on the system state modified by t1. This change also goes live. If the fault from t1 is then reverted, it remains open what happens to the faulty change from t1 and whether it continues to affect the fault from t2. A clarification of how the revert ensures a consistent system state in such cases would be desirable.

  • Table 3 lists all "Category and defect subtype" combinations. A more in-depth analysis of the results showing where the various categories and defect subtypes perform well or poorly on Corr Loc Rep would provide meaningful insights into how the models and methods currently fail.

  • The model does not have the option to take the system offline or stop new deployments until existing issues are resolved, even though a human team could respond in such a way in reality.

Minor:

  • A breakdown of tokens into input tokens, tokens read from cache, and output tokens would be better. For the two models "kimi-k2.5" and "gpt-5.6-luna", no reasoning level was specified. However, the reasoning level affects cost, latency, and result quality.

  • Not all abbreviations are properly introduced, e.g., MTTD, MTTL, MTTR, Astra, OB, TT.

  • In line 255, the applications should be mentioned before the numbers in order to improve readability.

Questions for authors

What is the goal of the benchmark? To push the models to their limits or to create a realistic scenario? Are the fault frequencies realistic, representative for specific situations? And how do you know, how can you assess this?

Are the solutions to all injected faults always independent of one another? Could it also happen that, because two injected faults occur simultaneously, a different (and possibly better) solution exists than when both injected faults occur individually?

Where does d = 1 come from when a run contains an attempted forbidden operation? This has no impact on the results of this study, since according to the paper such a case never occurred, but it currently appears to be chosen arbitrarily because no explanation is given. Furthermore, this is treated as binary: if an attempted forbidden operation exists, it is reduced by 50%; if no attempted forbidden operation exists, it is not. If a model performs an attempted forbidden operation ten times, it is penalized just as much as if it had attempted it only once. Why was this decision made?

6 comments

Thank you for the thoughtful review and constructive suggestions. We address each concern and question below, with additional experiments and clarifications incorporated into the revised PDF paper. The changes are highlighted in blue.

W1. Comparison with individual faults.

We compare two settings using the same faults from an Astronomy Shop marathon schedule: Concurrent, the original schedule with overlapping faults, and Individual, where faults are injected one at a time in the same order and with the same lifetimes. In the Individual setting, the application must pass two health checks after each reversion before the next fault is injected. Both settings use the same alert rules, invocation interval, methods, and Kimi K2.5 backbone, with agent state preserved throughout.

MethodSettingCorrLocRepMarathon-Score
Keep-agentConcurrent0.8250.4560.01842.1
Individual0.8600.4910.03544.4
Claude CodeConcurrent0.1750.0350.0007.0
Individual0.5090.1400.07020.5
CodexConcurrent0.0880.0180.0182.9
Individual0.5440.0530.07021.1
ReActConcurrent0.1580.0530.0535.8
Individual0.7370.2280.17535.7

Removing overlap substantially improves ReAct, Claude Code, and Codex, particularly in correlation, while Keep-agent changes relatively little. This comparison shows that concurrent faults introduce an additional challenge and reveals differences in how methods separate incidents from a shared alert stream.

Performance remains limited even when faults occur individually: repair rates range from 3.5% to 17.5%. Although this setting eliminates concurrent faults, agents must still handle diverse failures within resource limits and complete correlation, localization, and repair. These results show that concurrency adds difficulty to an already challenging incident-resolution task, supporting the benchmark’s focus on sustained operation with overlapping faults.

We have added the main findings to the evaluation section and the full comparison and experimental details to Appendix I.

W2. Handling agent interventions when faults are reverted.

The orchestrator restores the fault’s target object to its pre-injection state, overwriting subsequent agent edits to that object. Changes to other objects persist. In your example, the changes made at t1 and t2 are overwritten if they modify the reverted object; otherwise, they remain in effect and may influence subsequent incidents. Reversion therefore removes the injected fault without resetting the entire application or automatically undoing all agent interventions. We have clarified this behavior in Section 2.

W3. Performance analysis by fault category and subtype.

We have added a breakdown of correlation, localization, and repair by fault category and subtype to Appendix J. The category-level results below are calculated across all fault instances in the runs reported in Table 2:

CategoryCorrLocRep
Code bug23.7%0.0%0.5%
Config bug59.0%16.8%0.3%
Chaos test45.7%15.2%1.1%
Code bugs are particularly challenging: agents correlate fewer of them and achieve no successful localization. Configuration and chaos faults are more often correlated and localized, but repair remains rare across all three categories. The subtype analysis further shows that dependency failures have the highest repair rate (6.9%), while several configuration subtypes receive no correlation, localization, or repair credit.

W4. Shutdown and deployment-freeze options.

Each fault is validated offline to confirm that reverting it removes its symptoms, and its recovery can be performed through the tools available to agents. Thus, no fault in the current catalog requires taking the service offline or freezing new deployments to resolve it. Nevertheless, these are realistic containment strategies outside our current scope. We have clarified this limitation and will consider supporting them as we extend the benchmark to more complex faults.

Response to be continued below.

W5. Minor issues.

  • Token consumption and reasoning configuration. We have added the full token breakdown to Appendix G, with four methods shown below (mean tokens per run). Both models were accessed through AWS Bedrock, which did not record cache-read tokens. Kimi K2.5 used the default configuration without an explicit reasoning parameter, while GPT-5.6 Luna used reasoning none to enable tool calling in our setup.
MethodKimi K2.5 input / outputGPT-5.6 Luna input / output
ReAct6.46M / 47k6.27M / 51k
Keep-agent556k / 129k610k / 132k
Codex7.13M / 59k7.91M / 54k
Claude Code6.60M / 58k3.29M / 90k
  • Abbreviations. We have expanded MTTD, MTTL, and MTTR at first use in Section 3(f), and introduced Astro, OB, and TT in Section 2 and the Table 1 caption.
  • Application names. We have revised the sentence to name each application before its percentage: “39% (Online Boutique), 76% (Train Ticket), and 47% (Astronomy Shop).”

Q1. The goal of the SRE-Marathon benchmark.

SRE-Marathon evaluates whether agents can sustain incident detection, correlation, localization, and repair in a continuously running system. Its fault mechanisms and category distribution are calibrated using production studies, while the main schedule deliberately accelerates fault arrivals to test performance under increasing concurrency.

The resulting alert volume is comparable to production observations: 98.9% of run-minutes in SRE-marathon contain at most ten alerts, compared with 96.9% in a three-year study at a large commercial bank (Zhao et al., 2020). Our busiest minute contains 33 alerts, well below the hundreds to thousands per minute observed during alert storms in that study. This comparison contextualizes the alert workload; the fault-arrival rate itself is not calibrated to a typical production hour or a specific production scenario.

The repair windows are motivated by timely automated response before human intervention, using the approximately five-minute paging-response target for user-facing services as a reference (Beyer et al., 2016). Scheduled reversion represents an external fallback intervention. We have clarified these design choices and their scope in Section 2.

References:

  • Zhao et al. (2020). Understanding and Handling Alert Storm for Online Service Systems. ICSE-SEIP, 162–171.
  • Beyer et al. (2016). Site Reliability Engineering. Chapter 11, “Being On-Call.”

Q2. Can concurrent faults admit a joint solution?

Our injection mechanism (Section 2) guarantees that concurrently active faults target different services, making a single remedy that resolves both less likely. However, their effects may overlap, so we do not assume that their solutions are always independent. In our evaluation, repair is scored by outcome rather than by matching a specific solution. Therefore, if a joint intervention satisfies the repair criteria for both faults, it receives repair credit for both.

Q3. Penalty for forbidden operations.

The factor was initially intended to penalize a run containing any forbidden-operation attempt, regardless of frequency. However, we agree that the 50% penalty lacks sufficient justification and have removed the (1-d/2) factor from Marathon-Score. All reported results remain unchanged because no such attempts occurred. In future evaluations, we will report the number of forbidden-operation attempts separately.

Thank you again for your detailed review. We have posted our responses and uploaded a revised PDF with the changes highlighted in blue.

As the discussion period is drawing to a close, we would be grateful if you could let us know whether our responses and revisions address your concerns. If any point remains unclear or would benefit from further evidence, we are happy to provide additional clarification or results.

Thank you for your time and consideration.

Reviewer_sdgkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SRE-Marathon evaluates agents operating live microservice systems under continuous, overlapping faults, including code and configuration changes. It provides a 700-fault catalog across three applications, a persistent intervention interface, and recorded evidence for offline evaluation of alert correlation, localization, repair, operational impact, and safety. The main marathon runs inject approximately 60 faults per run and evaluate ten methods, with two LLM backbones where applicable. The benchmark exposes a large gap between reporting or diagnosis and successful repair under the tested protocol.

Critical strengths
  1. The continuous operation protocol and validated change-fault catalog address an important evaluation problem. Combining persistent agent state, overlapping incidents, live interventions, and code/configuration changes creates a useful setting for studying how SRE agents manage sustained operational demands. The contribution is the integrated protocol and its fault construction, rather than individual features such as live systems or noisy alerts in isolation.

  2. Recorded evidence and stage-level outcomes make failures interpretable. The benchmark separates correlation, localization, and repair and supports offline recomputation from the run record instead of relying on an LLM judge. Appendix J already distinguishes missed faults, correlation-only outcomes, localization without action, and ineffective interventions. The sequential Marathon-Score is a defensible measure of lifecycle progress, and the separately reported stages help prevent it from being mistaken for a repair-success rate.

Critical weaknesses
  1. The main comparative evaluation provides limited evidence of stability across fault schedules. Table 2 averages one shared-seed marathon run per application for each method. A common schedule is useful for within-schedule fairness, but it does not establish how representative those comparisons are across the catalog. Appendix H reports ReAct scores of 20.1 and 7.3 on two single-host Astronomy Shop seeds; the additional 17.0 result also changes venue and cannot isolate seed variation. These observations motivate uncertainty estimates for representative methods. They do not prove that leaderboard rankings reverse, nor should a benchmark require identical rankings under every scenario. The present sampling nevertheless leaves the reliability of the central method comparisons unclear.

  2. Near-zero repair is established for the tested protocol, but its interpretation as an agent-capability limitation remains underdetermined. Outcomes depend jointly on the two budget-oriented backbones, fault lifetimes, the 120-second invocation interval and 110-second action budget, tool workflows, and load. Appendix J provides useful descriptive failure analysis, but does not fully isolate these factors. The shorter runs in Appendix I change several settings together, as the authors acknowledge, and therefore cannot establish concurrency as the cause. This matters for a benchmark intended to measure operational capability: the evaluation should clarify how often unsuccessful repair reflects an incorrect intervention, execution constraints, or a correct remedy arriving after the eligible window.

Questions for authors
  1. Can representative leading methods be evaluated over a small number of additional fault seeds on the same venue, with uncertainty for Marathon-Score and handled/repair outcomes? If this is not available, which comparative conclusions are supported beyond the specific shared schedules?

  2. Can the existing traces further separate unsuccessful interventions into incorrect remedies, tool or policy failures, and correct remedies completed too late? A limited controlled check of the action window or load would also help explain the measured capability gap without requiring a broad new model sweep.

2 comments

Thank you for the thoughtful review and constructive comments. We address the two concerns and corresponding questions below. The submission PDF has been updated to address all reviewer feedback, with changes highlighted in blue.

W1 / Q1. Stability across fault schedules.

We evaluated four representative methods with Kimi K2.5 on ten Astronomy Shop schedules using the same single-host setup as Table 2. The schedules share the same injection-rate settings but vary the selected faults, their order, and their timing. Each schedule is shared across methods. The table reports mean ± standard deviation across ten seeds.

MethodCorrLocRepMarathon-Score
Keep-agent0.765 ± 0.1040.339 ± 0.1830.049 ± 0.04035.7 ± 8.1
Claude Code0.256 ± 0.0730.048 ± 0.0640.014 ± 0.0189.5 ± 2.7
Codex0.225 ± 0.1460.080 ± 0.1080.010 ± 0.0188.7 ± 6.5
ReAct0.207 ± 0.1280.074 ± 0.0660.033 ± 0.0338.1 ± 5.3

Keep-agent ranks first on all ten seeds. Its paired score advantage is 26.1 ± 10.0 points over Claude Code, 27.0 ± 8.6 over Codex, and 27.5 ± 12.6 over ReAct. Repair also remains limited across schedules: only 2.6% of fault instances are repaired, and only 4 fault instances are handled end to end across the 40 runs.

However, the ordering of Claude Code, Codex, and ReAct changes across seeds. Much of this variation comes from correlation: for example, Codex declares between 1 and 24 incidents per run, with Marathon-Scores ranging from 0.6 to 18.6. These results support Keep-agent's lead among the four methods in this setting, but show that small score differences on a single schedule do not establish a reliable ranking.

We have added the stability summary to the main evaluation section and the per-seed results to Appendix H, and qualified comparisons between methods with similar scores.

W2 / Q2. What prevents successful repair?

We re-analyzed the recorded traces across all three applications for the LLM-based methods in Table 2. Below are ReAct and Keep-agent on both original backbones; percentages are over all injected fault instances.

Outcome / failure categoryReAct Kimi K2.5ReAct GPT-5.6 LunaKeep-agent Kimi K2.5Keep-agent GPT-5.6 Luna
Repaired0.6%0.0%1.7%1.7%
No intervention attempted39.0%36.2%6.1%27.0%
Interventions only outside the affected scope24.9%29.9%34.8%33.9%
Incorrect action on the affected service12.4%14.7%13.9%5.7%
Invalid tool arguments0.0%0.0%20.0%5.2%
Invocation or action timeout0.0%0.0%10.4%19.5%
Correct remedy, but symptoms persisted8.5%9.0%4.3%2.9%
Correct remedy with insufficient verification time14.7%10.2%8.7%4.0%

An incorrect action targets the affected service but selects an unsuitable remedy. Invalid tool arguments cause rejection, while timeouts interrupt an invocation or action, such as an image build. The final two categories distinguish remedies whose symptom probes continue to fail, usually because another concurrent fault remains unresolved, from remedies applied too late to complete two verification probes before reversion.

Failures in selecting the intervention target, choosing the remedy, or supplying valid arguments account for 37.3–68.7% of faults across these configurations. However, the remaining categories show that unsuccessful repair can also reflect execution and verification constraints.

Effect of longer fault lifetimes. We additionally evaluated a new 56-fault Astronomy Shop schedule with Kimi K2.5, extending every fault’s lifetime by 0, 2, or 4 minutes while keeping fault order, injection times, and agent settings unchanged. The extensions allow one or two additional agent invocations. Repair rates are:

MethodOriginal lifetime+2 min+4 min
ReAct3.6%3.6%10.7%
Keep-agent7.1%3.6%14.3%

With four additional minutes, repair rates increase for both methods, indicating that longer fault lifetimes can improve measured success. Nevertheless, interventions only outside the affected scope or incorrect actions still account for 66.0% of faults for ReAct and 80.4% for Keep-agent. Longer lifetimes therefore help, but substantial difficulties in choosing where and how to intervene remain. Because injection times are fixed, extending fault lifetimes also increases overlap between faults.

We have added the category definitions, full trace analysis, and lifetime-extension results for all four evaluated methods to Appendix J.1.

Thank you again for your detailed review. We have posted our responses and uploaded a revised PDF with the changes highlighted in blue.

As the discussion period is drawing to a close, we would be grateful if you could let us know whether our responses and revisions address your concerns. If any point remains unclear or would benefit from further evidence, we are happy to provide additional clarification or results.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.