- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This is a well-motivated and carefully measured analysis framework that reframes RCA-agent debugging from "whether the final diagnosis is right" to "where in the diagnostic process the ground truth was lost," with an attribution scheme (FS0/FS1/FS2) that is exhaustive, mutually exclusive, and free of any LLM judge. Its most valuable contributions are the engineering-level failure mechanisms and the counter-intuitive findings that end-to-end accuracy hides (stage migration behind near-zero net change, a bottleneck that shifts rather than clears). However, its practical reach is bounded by the enumerable-solution-space assumption and the reliability of self-reported candidate records, its evaluation rests on a single synthetic testbed of limited scale, and the intervention gains are modest and uneven, with several interventions hurting accuracy. On balance this is a solid, original contribution to RCA-agent observability; whether it should be accepted depends on how convincingly the authors can address the generality limits and the nondeterminism confound behind the migration conclusions.
Critical strengths
• Existing work (hallucination checks, execution-error checks) can only catch failures with a flaw in the reasoning chain; this paper points out that a large class of failures are sound at every step, with every tool call succeeding and conclusions consistent with observations, yet the true root cause was lost from the very beginning — such failures cannot be caught by existing trace-level checks. This is the most direct motivation of the work.
• It proposes three stages, FS0 / FS1 / FS2 (observation → nomination → selection), which are exhaustive and mutually exclusive by construction, attributing a vague "wrong" to "where it was lost", each mapping to a different remediation (broadening observation / improving recognition / strengthening adjudication).
• Attribution needs no LLM judge, relying only on candidate-set membership matching plus tool outputs for string matching, avoiding the unreliable and non-reproducible problem of an LLM judge, and being auditable and reproducible.
Critical weaknesses
• Limited applicability — requires an enumerable solution space; it only holds for scenarios where (component, reason) come from an enumerable list, and agents producing free-form text are directly inapplicable. This is a very large weakness.
• The FS0 definition is a "weakened" one and may mislead attribution: FS0 is defined as "the true component never appearing", rather than "appearing but with insufficient evidence". The authors admit this causes many cases to err toward FS1, thus possibly overestimating failures at the nomination stage and underestimating observation capability — a credibility issue of the attribution itself.
• The test dataset is single and limited in scale: based on a modified OpenRCA, with 3 microservice systems and 261 queries, and synthetic / simplified scenarios; transferability to production-grade RCA is not validated.
• Intervention gains are limited and uneven: multiple interventions show negative gains on multiple models (under PI-1, the first three models range from −2.68% to −4.21%).
Questions for authors
- How large a fraction of real production RCA agents satisfy the enumerable-solution-space requirement? In cases where the agent's output is free-form prose, is there any validated way to still apply GLA (e.g., closed-vocabulary extraction)? Without this, the practical reach of the framework is unclear.
- How faithful are the candidate records to the agent's actual hypothesis space? Only 20 runs/model were sampled for consistency — could you report whether attribution labels are stable across multiple runs of the same query, and how often a run is mis-attributed because the model under-reports or mis-reports a hypothesis?
- The FS0 condition uses component appearance rather than sufficient evidence, and you state this errs toward FS1. Could you quantify this bias — e.g., among runs labeled FS1, how many would shift to FS0 under a stricter observation criterion, and would the headline finding (FS1 dominance) survive that shift?
- Several interventions decrease accuracy (e.g., PI-1 drops three models by −2.68% to −4.21%). Given that the migration matrices also show harm, what is the concrete, quantifiable benefit of GLA for a developer — i.e., does acting on GLA's guidance improve outcomes over blind prompting, on average?