acceptodds
Under review as a conference paper at ICLR 2027

When Does Snapshot Evaluation Predict Live Microservice RCA?

Abstract

Microservice root-cause analysis (RCA) agents are commonly evaluated on frozen incident snapshots because snapshot evaluation is reproducible and inexpensive. Deployed agents, however, query running systems whose telemetry evolves during diagnosis. It is therefore unclear whether capability estimates obtained from snapshots predict performance in live environments, or which properties of an incident make that extrapolation fail. We introduce a strictly paired evaluation protocol that instantiates each incident under three conditions: Snapshot, with frozen evidence; Live-fixed, with a live backend but the same evidence surface as Snapshot; and Live, with telemetry that grows during investigation. The protocol holds the incident, agent, tool interface, query budget, observation cutoff, and scoring contract fixed, allowing backend interaction to be separated from the value of additional evidence. We further define three machine-checkable validity conditions: evidence sufficiency, execution-identity consistency, and scoring comparability. These conditions must hold before a paired result is attributable. Runs that violate them are classified as undiagnosable or incomparable rather than counted as model failures. On a cross-application benchmark spanning multiple fault mechanisms, we use paired score differences, strict-outcome agreement, and case-level flips to measure Snapshot-to-Live transfer across models, treating independent case families rather than repeated calls as the statistical unit. A complementary failure analysis demonstrates how omitted validity conditions can leave apparently successful evaluations without a defensible comparison. Together, the protocol and benchmark turn the assumption that snapshot evaluation proxies live RCA into a controlled, auditable, and falsifiable empirical question.

Primary area:
datasets and benchmarks
Keywords:
root cause analysis, LLM agents, evaluation methodology, benchmark validity, offline vs online evaluation, AIOps, microservices

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HeqvOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether snapshot-based evaluation reflects the live performance of microservice root-cause analysis (RCA) agents. It introduces a three-arm protocol—Snapshot, Live-fixed, and Live—along with validity checks for evidence sufficiency, execution provenance, and scoring consistency. From a software engineering perspective, the contribution is primarily an evaluation methodology and an experience report on failures in agent-evaluation infrastructure, rather than a new RCA technique.

Critical strengths
  • Relevant research question. Understanding whether inexpensive offline evaluations reflect live diagnostic performance is practically important.
  • Useful experimental design. Introducing Live-fixed helps distinguish backend differences from the effects of an expanding observation window.
Critical weaknesses
  • Insufficient evidence to answer the central question. The primary analysis covers only four cases across two applications. It does not establish when snapshot evaluation is predictive or identify incident characteristics associated with transfer failures. Mean score differences and outcome agreement alone do not establish predictive validity.

  • The validity claims require stronger verification. The relationship between “machine-checkable” evidence sufficiency and the reviewer-based admissibility process is unclear. The paper also needs a clearer account of how the Live-fixed cutoff-recording defect was addressed and verified in P6. The absence of accompanying code, data, and audit artifacts limits independent assessment.

  • The methodological contribution is useful but limited in scope. The validity conditions largely operationalize established evaluation and software-integrity principles for RCA. The infrastructure failures are informative, but arise within the authors’ own pipeline, and their effects on scores or rankings are not demonstrated. Extensive operational history also obscures the main contribution.

Questions for authors
  1. Could the authors clarify the actionable implications of this work for practitioners evaluating RCA agents?
  2. The protocol requires per-run evidence-sufficiency checks. How were these implemented for fresh live realizations in P6, beyond the pre-run P4 review?
Reviewer_yGbmOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a strictly paired three-arm protocol with three validity conditions to test whether snapshot evaluation of root-cause analysis agents predicts their performance on live systems.

Critical strengths

A strictly paired three-arm evaluation protocol that separates the effect of the backend from the effect of additional observability (Live-fixed vs. Live) while holding the incident, agent, tools, budget and scoring fixed.

Three machine-checkable validity conditions that decide whether a paired result can be attributed, so that undiagnosable or incomparable runs are not counted as model failures.

A failure taxonomy that maps each discovered pipeline defect to the specific comparison it invalidates, together with a fully disclosed pre-registration and execution record.

Critical weaknesses

The paper does not answer the question it poses. Only four cases in two applications pass admissibility, the primary layers fall below the authors' own eight-cluster threshold, and every hypothesis verdict is "not_evaluated".

The paper is very hard to read. The main text relies heavily on internal code names (P4, P6, r15, x2u, J1–J5, DEV-S2) and constant qualifications, and it cannot be understood without the 30-page appendix.

According to the AI use statement, LLM agents proposed the hypotheses and protocol, made most analysis and amendment decisions, ran the experiments and drafted the paper. Together with the controller omissions disclosed in Appendix F, this raises concerns about the level of human oversight.

Questions for authors

See the above weaknesses

Reviewer_hDBUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether snapshot-based evaluation provides a reliable estimate of the performance of root-cause analysis (RCA) agents in live microservice environments. The authors propose a three-arm evaluation protocol: Snapshot, Live-fixed, and Live. They also define three validity conditions for making meaningful comparisons: evidence sufficiency, execution-identity consistency, and scoring comparability.

The paper's main contribution is its focus on evaluation validity rather than a new RCA agent or benchmark. The failure analysis is useful, particularly the examples where integrity checks pass even though the executed code differs from the declared artifact.

However, the experimental results are far from conclusive. Only four cases across two applications pass the evidence-sufficiency checks in all three arms. The primary comparison is therefore descriptive, and the reported score differences do not establish whether snapshot evaluation systematically overestimates or underestimates live performance. More importantly, the paper documents unresolved problems with execution identity, scoring consistency, and observation-cutoff recording in its own evaluation pipeline. These issues raise questions about how much of the proposed protocol has been validated in practice.

Critical strengths

An important evaluation problem. Snapshot benchmarks are convenient, but their ability to predict performance in live environments should not be taken for granted. The paper addresses a practical question that is relevant to the evaluation of tool-using RCA agents.

A useful decomposition of the comparison. Separating Snapshot, Live-fixed, and Live helps distinguish backend effects from the effects of additional observability. This is more informative than comparing frozen and live evaluations directly, although differences between incident realizations remain a confounding factor.

Attention to evaluation validity. The three proposed conditions address real sources of invalid comparisons. In particular, the execution-identity examples show why a successful integrity check is not sufficient evidence that the declared artifact was actually executed.

Appropriate caution in interpreting the results. The authors explicitly acknowledge the small number of admissible cases and avoid making population-level, ranking, or equivalence claims based on the primary experiment. Separating undiagnosable cases from model failures is also a sensible design choice.

Critical weaknesses

The empirical evidence is too limited to answer the main research question. The primary analysis contains only four cases across two applications, with four independent clusters in the J3 analysis. The reported Snapshot − Live-fixed difference of +9.0 percentage points is descriptive and cannot establish a general transfer relationship. The larger J5 analysis includes cases that fail the evidence-sufficiency requirement and therefore cannot resolve this limitation.

The evaluation pipeline has unresolved validity problems. The paper reports three deterministic reproductions in which the integrity check passes despite discrepancies between the declared and executed artifacts. It also reports dirty worktrees in three of four snapshot runs in the earlier pilot. These findings are important, but they leave uncertainty about the reliability of the experimental infrastructure. The paper needs to demonstrate that the proposed checks can reliably prevent such failures in the final evaluation, rather than mainly documenting cases in which they fail.

Scoring comparability remains insufficiently established. The authors acknowledge that 10% of the rubric uses different predicates across the snapshot and live arms. A unified scorer exists, but rescoring the pilot data fails because the required evidence contract is missing and the time contracts are inconsistent. This weakens confidence in the earlier pilot results and highlights an unresolved implementation issue in the broader evaluation methodology.

The paired design does not fully isolate the effects being studied. Snapshot and Live-fixed use different realizations of the same incident specification. Consequently, the primary contrast includes realization variation in addition to the backend difference. The authors acknowledge this limitation, but the current experiment does not quantify its contribution. Repeated runs of the same cases do not compensate for the small number of independent incident families.

The methodological novelty needs a clearer demonstration. The paper distinguishes itself from prior work through its explicit validity conditions and attributable comparisons, rather than through the construction of a live RCA environment. However, the authors should more clearly establish what the proposed protocol enables that existing benchmark-auditing and reproducibility practices cannot already provide. The failure taxonomy is useful, but its general value beyond the authors' own pipeline remains uncertain.

Reproducibility is not yet demonstrated at the level needed to validate the claims. The paper provides extensive implementation details and reports plans to release the tooling, harness, and validity checks, but no separate code or data supplement accompanies the submission. Given that several central findings concern failures in the evaluation infrastructure itself, independent verification is particularly important.

Questions for authors

Can the authors provide a complete end-to-end evaluation in which all three arms pass the evidence-sufficiency, execution-identity, and scoring-comparability checks, with the Live-fixed observation cutoff explicitly recorded for every run?

How sensitive are the reported Snapshot − Live-fixed differences to variation between live incident realizations? Is it possible to repeat the comparison across multiple realizations of the same incident while keeping the execution artifact, scoring rubric, and observation window consistent?

Why could the unified scorer not be used to produce comparable scores for the pilot data? Can the missing evidence contracts and inconsistent time contracts be resolved, or does doing so require rebuilding the affected cases?

How were the evidence-sufficiency checks validated, and how sensitive is the set of admissible cases to the acceptance criteria? Could independent adjudication or a second diagnostic assessment help establish that these checks do not systematically exclude difficult cases?

What is the main methodological advance over existing live-agent benchmarks and evaluation-integrity practices? Can the authors provide a concrete example in which the proposed protocol changes the conclusion that would be drawn from a conventional snapshot-versus-live comparison?

Will the benchmark cases, execution harness, validity checks, and analysis scripts be released with sufficient information to reproduce the reported results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.