acceptodds
Under review as a conference paper at ICLR 2027

Ground-Truth Lifetime Analysis: Localizing Failures in Agentic Root Cause Analysis

Abstract

Root cause analysis (RCA) agents are judged by whether their final diagnosis is correct, which tells a developer that a run failed but not where. Recent work focuses on identifying hallucination and execution errors in reasoning traces. However, we find that many failures are well-formed on their own terms: every tool call succeeds and every claim is grounded in what the agent observed, yet the true cause was already lost somewhere earlier in the run. We therefore introduce Ground-Truth Lifetime Analysis (GLA), which localizes a failed diagnosis to the stage at which the ground truth was lost — never observed, observed but never raised as a candidate, or raised but rejected. The underlying philosophy is to make the agent's hypotheses auditable, so GLA instruments the agent loop to record its operations on a candidate set at every step, and attributes each failure by tracking the true cause's membership in that set, with no LLM judge. Measuring four frontier models on microservice systems, we find that one stage dominates: up to 83% of failed runs observed the relevant telemetry but never formed the true cause as a candidate, while the observation gap reaches 42% in the weakest model-system pairing. GLA further exposes how agents respond to interventions, in terms that end-to-end metrics cannot capture: the variation across models, the substantial migration of runs between stages behind an unchanged total, and the bottleneck that shifts rather than clears. Our intervention improves accuracy by up to 13.03%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, root cause analysis, failure attribution, AI Ops, empirical study

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iE6gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This is a well-motivated and carefully measured analysis framework that reframes RCA-agent debugging from "whether the final diagnosis is right" to "where in the diagnostic process the ground truth was lost," with an attribution scheme (FS0/FS1/FS2) that is exhaustive, mutually exclusive, and free of any LLM judge. Its most valuable contributions are the engineering-level failure mechanisms and the counter-intuitive findings that end-to-end accuracy hides (stage migration behind near-zero net change, a bottleneck that shifts rather than clears). However, its practical reach is bounded by the enumerable-solution-space assumption and the reliability of self-reported candidate records, its evaluation rests on a single synthetic testbed of limited scale, and the intervention gains are modest and uneven, with several interventions hurting accuracy. On balance this is a solid, original contribution to RCA-agent observability; whether it should be accepted depends on how convincingly the authors can address the generality limits and the nondeterminism confound behind the migration conclusions.

Critical strengths

• Existing work (hallucination checks, execution-error checks) can only catch failures with a flaw in the reasoning chain; this paper points out that a large class of failures are sound at every step, with every tool call succeeding and conclusions consistent with observations, yet the true root cause was lost from the very beginning — such failures cannot be caught by existing trace-level checks. This is the most direct motivation of the work.

• It proposes three stages, FS0 / FS1 / FS2 (observation → nomination → selection), which are exhaustive and mutually exclusive by construction, attributing a vague "wrong" to "where it was lost", each mapping to a different remediation (broadening observation / improving recognition / strengthening adjudication).

• Attribution needs no LLM judge, relying only on candidate-set membership matching plus tool outputs for string matching, avoiding the unreliable and non-reproducible problem of an LLM judge, and being auditable and reproducible.

Critical weaknesses

• Limited applicability — requires an enumerable solution space; it only holds for scenarios where (component, reason) come from an enumerable list, and agents producing free-form text are directly inapplicable. This is a very large weakness.

• The FS0 definition is a "weakened" one and may mislead attribution: FS0 is defined as "the true component never appearing", rather than "appearing but with insufficient evidence". The authors admit this causes many cases to err toward FS1, thus possibly overestimating failures at the nomination stage and underestimating observation capability — a credibility issue of the attribution itself.

• The test dataset is single and limited in scale: based on a modified OpenRCA, with 3 microservice systems and 261 queries, and synthetic / simplified scenarios; transferability to production-grade RCA is not validated.

• Intervention gains are limited and uneven: multiple interventions show negative gains on multiple models (under PI-1, the first three models range from −2.68% to −4.21%).

Questions for authors
  1. How large a fraction of real production RCA agents satisfy the enumerable-solution-space requirement? In cases where the agent's output is free-form prose, is there any validated way to still apply GLA (e.g., closed-vocabulary extraction)? Without this, the practical reach of the framework is unclear.
  2. How faithful are the candidate records to the agent's actual hypothesis space? Only 20 runs/model were sampled for consistency — could you report whether attribution labels are stable across multiple runs of the same query, and how often a run is mis-attributed because the model under-reports or mis-reports a hypothesis?
  3. The FS0 condition uses component appearance rather than sufficient evidence, and you state this errs toward FS1. Could you quantify this bias — e.g., among runs labeled FS1, how many would shift to FS0 under a stricter observation criterion, and would the headline finding (FS1 dominance) survive that shift?
  4. Several interventions decrease accuracy (e.g., PI-1 drops three models by −2.68% to −4.21%). Given that the migration matrices also show harm, what is the concrete, quantifiable benefit of GLA for a developer — i.e., does acting on GLA's guidance improve outcomes over blind prompting, on average?
Reviewer_uJjQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that a wrong root-cause diagnosis is usually not explained by hallucination or a failed tool call: the run may be well-formed throughout and still wrong, because the true cause was lost earlier. Ground-Truth Lifetime Analysis instruments an RCA agent to record, at every step, a (component, reason, action) triple over an explicit candidate set, where action is propose, confirm or reject, and then uses the known ground truth to assign each incorrect run to the first stage the true cause did not survive: FS0 observation failure, FS1 candidate-nomination failure, FS2 candidate-selection failure. Attribution is pure string matching over candidate records and logged tool outputs, with no LLM judge, which makes the labels deterministically recomputable.

The measurement covers 261 single-root-cause tasks from a modified OpenRCA across Bank, Market and Telecom, with a ReAct-style main-LLM-plus-executor agent capped at 25 steps and four models (Opus-5, GPT-5.6, an internal model IM, DS-V4). The headline results are that FS1 is the largest stage in eleven of twelve model-system pairings, ranging from 39% to 83% of incorrect runs; that FS0 grows as capability falls (pooled 4, 7, 24, 70 runs; 42% for DS-V4 on Telecom); and that four prompt interventions move runs between stages in large opposing flows that aggregate accuracy conceals. Reported gains are up to 13.03 accuracy points.

Two scoping notes belong here because the abstract does not carry them. The 83% and 42% figures are shares within the incorrect runs of one pairing — 65 of IM's 78 Bank failures and 20 of DS-V4's 48 Telecom failures — not shares of all tasks or averages over models. And "improves accuracy by up to 13.03%" is 13.03 percentage points, from DS-V4's 9.20% to 22.22% under PI-1; PI-3 and PI-4 are measured against PI-2 rather than the original prompt.

Critical strengths

1. The attribution is judge-free, exhaustive by construction, and the paper's arithmetic holds up under checking. The three stages partition failures by a decidable test on recorded state rather than by an LLM's opinion about a trace, which is what makes the composition comparable across models and interventions, and lets a developer see which records triggered a label. I verified the reported quantities and they reconcile exactly: every bar in Figure 2 has counts summing to its Fail total with shares matching to rounding (IM on Bank, 3 + 65 + 10 = 78 with 4%/83%/13%); the three systems' task counts sum to 261 (118 + 92 + 51); the pooled FS0 progression 4, 7, 24, 70 matches the per-system counts and Table 1; all twelve cells of Table 3 satisfy (net stage change) = (accuracy change) ×\times 261, including the headline −45−4+15=−34-45 - 4 + 15 = -34 with 34/261=13.03%34/261 = 13.03\%; and every migration matrix row in Figure 3 sums to that model's baseline stage total (DS-V4: 70, 157, 10, 24 summing to 261). I also reproduced the one cell-level significance test the paper gives, 20 correct-to-FS2 against 7 returning, exact p=0.0192p = 0.0192 against the reported 0.019. The paper is also candid in the right places: Section 3 states plainly that the FS0 criterion is weaker than sufficient-evidence and "errs toward FS1", Section 4.2 states the FS1 disjunction correctly, Section 5 concedes that nondeterminism and genuine intervention effects cannot be separated, and the Discussion concedes that GLA localizes a stage but not the site and supplies no fix.

2. The migration matrix is better experimental practice than reporting accuracy, and the paper does not hide negative or inconsistent results. An unchanged total can mean nothing happened or that many runs were fixed while many others broke, and Figure 3 separates those two cases case by case; Finding 5's large opposing flows behind a flat net, and Finding 6's bottleneck that shifts to the next stage instead of clearing, are claims end-to-end metrics cannot express. Supplying the reverse flow for Finding 6's "40 runs moving FS0$\to$FS1" gives exact p=1.3×10−4p = 1.3\times10^{-4}, so that one holds. The paper also reports that the same prompt edit moves different models in opposite directions, that PI-3 degrades all four, and that its interventions are compared against PI-2 rather than a flattering baseline — none of which is the shape of a paper selecting its best story. The engineering mechanisms it surfaces are concrete and useful: that deviation relative to a per-metric threshold is not comparable across metrics, and that a consecutivity rule can filter out a genuinely short fault, are the kind of findings a practitioner can act on.

Critical weaknesses

1. The instrument is not a passive observer, and the single model carrying the headline intervention result has no instrumentation control at all. Section 4.1 describes GLA as "minimally intrusive" and supports this with Appendix C's finding of no systematic accuracy degradation. But Table 5 shows IM moving from 20.7% to 26.8%, which the authors themselves attribute to "being required to keep an explicit candidate set" — a 6.1-point gain. That is a good property for an agent design and a problem for a measurement: the compositions reported throughout the paper are those of an agent the instrument has already changed, as footnote 1 concedes. Worse, Table 5 covers only OPUS-5, GPT-5.6 and IM — DS-V4 is absent. DS-V4 is the model with by far the worst observation stage (70 of the pooled 105 FS0 runs; 42% on Telecom) and the largest intervention gain in the paper (9.20% to 22.22% under PI-1, the 13.03-point headline). The one configuration whose numbers do the most work is the one with no measurement-perturbation check.

Three further mechanisms point the same way. (i) The recording requirement carries a format burden that falls hardest on weak models: footnote 2 concedes the prompt specifies each record as a tuple while "JSON has no tuple type, [so] a stricter instruction would specify a list of lists"; a failed format check discards the entire step and consumes step budget; and the paper's own error statistics count invalid candidate records among those failures. So some share of the exploration failures being measured in the weakest models may be produced by the measurement apparatus rather than observed by it. I cannot show that it is, and neither does the paper exclude it. (ii) The candidate records are generated by the model under test, and the fidelity check — 20 runs per model, records "consistent with, or a superset of" the reasoning trace — compares the model's records against the same model's own prose, so it cannot detect a hypothesis the agent acted on but never wrote down. (iii) Because FS2 requires only that the tuple appear "under any action", including an outright rejection with no prior proposal, a model that enumerates candidates more liberally is pushed toward later stages by construction. That bears directly on Finding 2, "as diagnosis capability falls, failure tends to move toward earlier stages": the ordering is confounded with how verbosely each model records, and the paper reports no candidate counts, candidate-set sizes or record-coverage rates that would separate the two. Relatedly, the title's "Lifetime" promises more than the statistics deliver — attribution turns on ever-appearance, while first proposal, first rejection, last elimination and final non-selection are different events on a lifetime, and candidates may be rejected and reconfirmed.

Finally, Appendix C's validity argument is of the wrong type for this paper. It checks whether two configurations agree in aggregate, while GLA's unit of analysis is the individual run's stage. Pairing the released tests/vanilla/ runs against the instrumented base runs query by query, under the paper's own row filter, shows why the margins agree so exactly: over 522 comparable queries, 90 (17.2%) change between correct and incorrect, 45 in each direction. Two runs of the same model under the same prompt disagree on one query in six. Per-run agreement, not marginal agreement, is the check the contribution requires.

2. Two of the paper's load-bearing claims are contradicted by its own methods section and appendix, and the migration magnitudes cannot be read against noise. (a) The abstract and introduction both state that "up to 83% of failed runs observed the relevant telemetry but never formed the true cause as a candidate". Section 3 operationalizes FS0 as the ground-truth component never appearing in observations, explicitly declines to require evidence sufficient for the ground-truth reason, and states that the resulting condition "errs toward FS1 when a run retrieved the component but not the decisive signal". So FS1 pools two failures that call for opposite fixes — the decisive signal was retrieved and no hypothesis formed, versus the decisive signal was never retrieved at all, which is an observation failure in all but name. If the true cause is packet loss on component A and the agent retrieved only A's CPU series, the run is scored FS1. Credit where due: Section 4.2 states the disjunction correctly ("the agent either fails to retrieve the critical signals implicating the true cause, or to qualify them into candidates"), so the body is honest and the abstract is not. That is exactly what makes it consequential — the sentence readers will cite asserts the stronger of two readings that the paper's own method section says it cannot distinguish.

(b) Finding 7 reads "Deleting a harmful rule need not unblock the stage it obstructed: without the knowledge it encoded, the cost reappears later in the process", and Section 4.3 states that "PI-3 removes the rejecting rules without replacement", explaining the degradation as "a knowledge regression". Listing 6 contradicts this. PI-3's prompt carries an Additional Rule 2 absent from PI-2 — Listing 5 shows PI-2's Additional Rules consist of Rule 1 only, and Listing 6's caption confirms "Additional Rule 1 is the rule PI-2 adds, which PI-3 carries" — which supplies substantive new domain knowledge: that KPIs whose names contain both "TCP" and "WAIT" are connection counts rather than wait durations, that sustained increases are evidence of network faults such as latency or packet loss, and that they should be correlated with traffic, error, trace and log evidence. PI-4's Listing 7 does not carry it. So PI-3 is a deletion of three rules plus an addition of knowledge in the fault family the paper elsewhere identifies as hardest to diagnose, and the net effect cannot be attributed to the deletion. This is not a compound-intervention quibble: the added rule points in the direction that should help, so Finding 7's mechanism is not isolated and the phrase "without replacement" is incorrect as written.

(c) What the migration matrices can bear is smaller than the prose implies. Against the 17.2% churn floor above (and per-model floors of 17.6% and 16.9%), all twelve intervention arms cross at 15.3% to 22.6% on the correct/incorrect axis, so the "27% to 55% of runs change stage" movement is in substantial part the churn of re-running an identical configuration — and a four-category floor can only be larger than a two-category one. Section 5's retreat to reporting only McNemar-significant pairs is the right call, and five of twelve arms are significant on the accuracy axis; but Figure 3 is 16 matrices, each four-state matrix admits up to six directional comparisons, and no multiplicity control is reported, so individual p<0.05p < 0.05 cells should not carry mechanism claims. Two further gaps compound this: the interventions were designed by reading failure traces and then re-measured on the same full query set with no development/test split, so the reported gains may not transfer; and Table 1's mechanism evidence is co-occurrence without base rates — it reports how often small-kk truncation, budget exhaustion or errored steps appear in FS0 runs without the corresponding frequency in non-FS0 or correct runs, and the appendix concedes the truncation counts include preview-only truncations and are therefore "an upper bound on truncations that remove data from the analysis". The stage-level floor cannot be recovered from the release at all: tests/vanilla/ ships eight bare result CSVs and no candidate records.

Questions for authors
  1. Please correct Section 4.3 and re-examine Finding 7. Listing 6 shows PI-3 adds an Additional Rule 2 on TCP/WAIT KPIs that PI-2 does not contain and PI-4 does not carry, so "removes the rejecting rules without replacement" is not accurate, and the accuracy drop is the net of a deletion and an addition of network-fault knowledge. What are the results for the clean decomposition — delete the three rules only; add the TCP/WAIT rule only; both — and does the knowledge-regression mechanism survive it? PI-4 should likewise be separated into its ranking change and its zero-threshold handling.
  2. Please scope the abstract's FS1 claim to what the criterion measures. Section 3 states the FS0 condition "errs toward FS1 when a run retrieved the component but not the decisive signal", and Section 4.2 states the disjunction correctly, but the abstract and introduction assert that those runs "observed the relevant telemetry". Within FS1, what share of runs retrieved telemetry that actually contains the decisive signal for the ground-truth reason, as opposed to only the component? A hand-labelled stratified sample of 20 to 30 runs per model, reported as agreement with the string criterion, would either sharpen or soften the headline, and would also show whether FS1 still dominates under the stricter criterion.
  3. Would you add the DS-V4 instrumentation ablation, and a base-prompt re-run? DS-V4 carries the heaviest FS0 and the 13.03-point headline yet is the one model missing from Table 5. Separately, one instrumented re-run of the base prompt for one model (261 runs) would give the four-category stage-migration matrix of a configuration against itself; the released vanilla pairing already gives a 17.2% correct/incorrect churn floor, above which no arm separates on that axis, and the stage-level floor needs candidate traces that tests/vanilla/ does not carry.
  4. Can you report, per model, the number of candidate records, the candidate-set size, record coverage, and the breakdown of discarded steps into candidate-format errors, other format errors and executor errors, together with the step budget each consumed? Finding 2's ordering — weaker agents losing the ground truth earlier — is currently confounded with how verbosely each model records, since FS2 is reached by the tuple appearing under any action including rejection; and the format burden conceded in footnote 2 falls hardest on the weakest models.
  5. Section 5 attributes the nondeterminism to the models accepting no temperature setting, but the released config.json sets "temperature": 1.0 for all three models and rca/api_router.py:138 places it in every request unconditionally, with no branch removing it and no fallback on an invalid-parameter error. Were the endpoints in fact accepting temperature=1.0, and if so should the stated reason be revised?
  6. For Table 1's mechanisms, what are the base rates in non-FS0 and in correct runs? Without them, "many FS0 runs show small-kk truncation" has little explanatory force, especially as the counts are an acknowledged upper bound. The convincing version would be a counterfactual: identify a truncation that provably removed the ground-truth component or decisive signal, restore it with other steps held fixed, and show a repeatable change downstream.
  7. Does GLA produce debugging value that the alternatives do not? A fair comparison would give equal-budget developers raw traces only, simple component-coverage statistics, and full GLA records, and measure time-to-localize, fix success rate and rate of harmful edits — with all arms given the same ground-truth information, so that any advantage is not simply from using the answer.
  8. How are near-misses attributed, given that matching is exact on the (component, reason) tuple with no partial credit? How many failures have the right component with a synonymous or neighbouring reason, and are they FS1 or FS2?
  9. Two scope items. The main LLM and the executor default to the same model, so "capability" conflates the main model's reasoning, the executor's code generation, and the information lost between them — can one be varied with the other fixed? And how many tasks are ones where the available telemetry cannot distinguish the true reason from a competitor, as Section 5 allows: flagging evidence-insufficient cases and reporting them separately would separate agent incapability from data non-identifiability.
  10. On cost and reproducibility: Table 5 reports 0.88M, 0.92M and 1.38M total tokens per run for the three instrumented models, but the interventions' costs are not given, and PI-1 returns more raw evidence while PI-2 requires inspecting every KPI. Could per-intervention token, latency and step-budget costs be reported alongside the accuracy change? And since IM is an internal model with neither code nor run artifacts released, could the IM-dependent findings be restated with a public model, or marked as unauditable?
Reviewer_cJeZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Ground-Truth Lifetime Analysis (GLA) is a debugging framework for AI agents that diagnose root causes of system failures. It uses the known correct answer and the agent’s recorded hypothesis to classify three failure types. 1) Observation: the correct component never appeared in tool outputs; 2) Nomination: the component appeared, but the correct component–cause pair was never recorded as a hypothesis; 3) Selection: the correct hypothesis was recorded, but the agent selects another (wrong) answer.

Across 261 incidents and four models, the paper finds that nomination failures dominate many settings. By testing prompt changes and tracking movement between failure categories, GLA reveals improvements and regressions that overall accuracy hides. Based on GLA, the intervention improves accuracy by up to 13.03%.

Critical strengths
  • The observation–nomination–selection decomposition provides a useful framework for investigating where a diagnosis failed, rather than relying only on final accuracy.

  • The analysis is simple. Hypothesis records and classification make the assigned failure categories inspectable, without relying on another LLM judge.

  • The migration analysis is informative. Tracking how individual cases move between failure categories after a prompt change reveals both improvements and regressions that an aggregate accuracy number cannot disclose.

  • The experiments include diversity and negative results. The study covers 261 incidents across three systems and four model backbones. It reports interventions that hurt as well as help, providing a more complete picture than success-only examples.

Critical weaknesses

GLA provides interpretable failure categories, but stronger validation is needed to show that they reliably identify what to fix. More specifically.

  • Do the labels capture the intended failures?

    • Seeing the correct component does not mean the relevant diagnostic evidence was available. Although acknowledged, this gap is not quantified, which weakens the interpretation of FS1 dominance as a hypothesis-generation bottleneck rather than an evidence-retrieval problem.

    • The manual audit is useful, but does not quantify missing or unsupported candidate records.

    • Instrumentation may change the behavior being measured. More trajectory-level and repeated-run checks are needed to demonstrate the representativeness of the instrumented agent.

  • Do the interventions support the explanations?

    • PI-3 combines rule deletion with an added TCP/WAIT rule, preventing clean attribution to knowledge removal.

    • Repeated unchanged-prompt runs are also needed to distinguish intervention effects from randomness.

  • Does GLA improve debugging?

Evidence is limited to one modified benchmark and one harness, without a controlled demonstration that GLA improves debugging over ordinary trace inspection. Known ground truth further restricts its applicability.

Questions for authors
  • Among FS1 cases, how often did the observations contain evidence relevant to the true cause, rather than just the correct component’s name? Does FS1 remain dominant when this distinction is accounted for?

  • Was the added TCP/WAIT rule in Listing 6 active in the reported PI-3 experiment? If so, can its effect be separated from rule deletion before attributing the outcome to knowledge removal?

  • How much stage migration occurs across repeated runs with an unchanged prompt, and which intervention effects remain distinguishable from this baseline variability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.