- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether snapshot-based evaluation reflects the live performance of microservice root-cause analysis (RCA) agents. It introduces a three-arm protocol—Snapshot, Live-fixed, and Live—along with validity checks for evidence sufficiency, execution provenance, and scoring consistency. From a software engineering perspective, the contribution is primarily an evaluation methodology and an experience report on failures in agent-evaluation infrastructure, rather than a new RCA technique.
Critical strengths
- Relevant research question. Understanding whether inexpensive offline evaluations reflect live diagnostic performance is practically important.
- Useful experimental design. Introducing Live-fixed helps distinguish backend differences from the effects of an expanding observation window.
Critical weaknesses
-
Insufficient evidence to answer the central question. The primary analysis covers only four cases across two applications. It does not establish when snapshot evaluation is predictive or identify incident characteristics associated with transfer failures. Mean score differences and outcome agreement alone do not establish predictive validity.
-
The validity claims require stronger verification. The relationship between “machine-checkable” evidence sufficiency and the reviewer-based admissibility process is unclear. The paper also needs a clearer account of how the Live-fixed cutoff-recording defect was addressed and verified in P6. The absence of accompanying code, data, and audit artifacts limits independent assessment.
-
The methodological contribution is useful but limited in scope. The validity conditions largely operationalize established evaluation and software-integrity principles for RCA. The infrastructure failures are informative, but arise within the authors’ own pipeline, and their effects on scores or rankings are not demonstrated. Extensive operational history also obscures the main contribution.
Questions for authors
- Could the authors clarify the actionable implications of this work for practitioners evaluating RCA agents?
- The protocol requires per-run evidence-sufficiency checks. How were these implemented for fresh live realizations in P6, beyond the pre-run P4 review?