acceptodds
Under review as a conference paper at ICLR 2027

Agentic RCA under Partial Observability: From Root Observation to Diagnosis

Abstract

LLM-based root-cause analysis (RCA) agents iteratively inspect metrics, logs, and traces to identify the source of an incident, but fault propagation and concurrent activity can make non-root components plausible competing explanations. Final-answer accuracy alone cannot reveal whether telemetry from the root-cause component was retrieved, whether the root was considered or favored, or whether it reached the final diagnosis. We analyze 335 OpenRCA queries by aligning incident-side signal conditions with retrieved telemetry, expressed diagnostic states, and final diagnoses. Later root signals and especially stronger non-root competition accompany poorer diagnosis, whereas broader anomalous activity does not show the same pattern. Root retrieval shows no consistent association with non-root competition. After observing the root, however, agents facing stronger competition are less likely to favor it or select it in the final diagnosis. We further evaluate three targeted agent designs on 22 queries. Although these designs improve component accuracy, residual failures show that additional structure can leave competition unresolved. These findings motivate cross-hypothesis discrimination—using evidence to distinguish among causal hypotheses—as a design target for agentic RCA.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Root Cause Analysis, LLM Agents, Partial Observability, Diagnostic Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EB49OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that the limited accuracy of current root cause analysis (RCA) agents under partial observability stems from non-root signals interfering with root-cause signals. It identifies and empirically examines three incident-side signal conditions: relative peak timing, rival prominence, and concurrent anomaly breadth. It also proposes three agent designs to mitigate these effects and improve RCA accuracy.

Critical strengths
  1. The paper addresses an important question: why current RCA agents perform poorly across many scenarios. It investigates interference from non-root signals and identifies three incident-side signal conditions, supported by empirical experiments.
  2. The paper proposes three agent designs to mitigate this interference and improve RCA accuracy.
Critical weaknesses
  1. The experimental evaluation is insufficiently comprehensive and covers too few models. For example, the results in Table 2 rely on a single model, which is insufficient to establish the general effectiveness of the proposed agent designs.
  2. The agent designs appear overly simple and heuristic. Given the extensive preceding analysis, the designs should more directly address the specific problems identified. How do the three designs follow from the analysis?
  3. The writing is difficult to follow, and the presentation lacks clarity and organization.
Questions for authors
  1. In Figure 4(c), why does GPT-5.5 perform better under low rival prominence than under high rival prominence, and why does GPT-4o perform worse than GPT-5.5?
  2. Similarly, in Figure 4(d), why are the values higher for GPT-4o in some categories but higher for GPT-5.5 in others?
  3. Why do the three proposed agent designs improve RCA accuracy?
  4. What should be prioritized to address interference from non-root signals: better agent design or more comprehensive observations? This remains an open question that requires empirical investigation.
7 comments
AuthorsOpenReview ↗

Response to Reviewer EB49 (1/4)

Thank you for the review. The table summarizes our answers and where the revision changed. Changes made for your comments are marked in orange in the revised PDF.

ConcernOur answer in briefWhere in the revision
W1On 139 incidents, SDC and LGD improve over Agent-RCA, and LGD matches or exceeds it with two more backbones.Section 5.5; Appendix A.5 (Tables 6 and 8)
W2, Q3Each design supplies one kind of support the analysis finds missing, and it gains where that difficulty limits the agent.Appendix A.5 (Table 7); Appendix F.5
W3One name per term, a terms table, redrawn figures, and an explicit analysis-to-design mapping.Appendix A; Section 2.2; Figures 1 and 4
Q1, Q2Both differences arise after the root is observed, and the flip shows where each model loses the root.Figure 4; Appendix D.2; Appendix E.1
Q4Observation must first bring in the root's own evidence. After that, design that directs observation at separating the root from its rivals matters most.Section 5.4; Appendix C.2

1. More models (W1).

W1. The experimental evaluation is insufficiently comprehensive and covers too few models. For example, the results in Table 2 rely on a single model, which is insufficient to establish the general effectiveness of the proposed agent designs.

We agree that one model on 22 incidents cannot establish general effectiveness, and we extended the evaluation in both directions you raise. Each design targets a difficulty the analysis identifies, and we evaluate the designs as diagnostic probes of whether such targeted support is enough.

For more incidents, we ran the designs, with GPT-4o as in Table 2, on the 139 single-root component incidents shared across Bank, Market, and Telecom.

Table 6 (abridged, full table in Appendix A.5). Component accuracy on 139 shared incidents. Differences are paired, with intervals from a within-dataset bootstrap.

MethodCorrect / 139Recovered / RegressedDifference (pp)
Agent-RCA22––
SDC4635 / 11+17.3 [+8.6, +25.9]
LGD4436 / 14+15.8 [+6.5, +25.2]
HGI1914 / 17−2.2 [−10.1, +5.8]

At this scale, SDC and LGD improve reliably over Agent-RCA, whereas HGI does not, which 22 incidents could not show.

For more models, we ran the same three designs with GPT-5.5 and with Qwen3.5-27B in non-thinking mode.

Table 8 (Appendix A.5). Correct component answers on the 22 matched Telecom incidents, by backbone.

BackboneAgent-RCALGDHGISDC
GPT-4o4765
GPT-5.591044
Qwen3.5-27B11053

LGD, which compares candidates after each builds local support, matches or exceeds Agent-RCA with every backbone. With Qwen3.5-27B it raises a 27B open model from 1 to 10 correct answers, the level of GPT-5.5 with Agent-RCA.

More backbones also show when each kind of support helps, because this follows where a backbone loses the root (Appendix A.5). GPT-4o loses about as much before concrete root observation as after it (16.9 and 16.2 points), so support that helps the agent reach the root's own evidence pays off, as SDC does on the 139 incidents. GPT-5.5 already observes the root in 68.4% of histories, against 23.4% for GPT-4o, and it loses about twice as much after observation as before it (28.5 against 14.6 points). There, SDC and HGI do not improve on Agent-RCA, while LGD holds its level. Stronger backbones thus make post-observation competition the main remaining difficulty, which is the target of cross-hypothesis discrimination that our paper proposes.

AuthorsOpenReview ↗

Response to Reviewer EB49 (2/4)

2. How the designs follow from the analysis, and why accuracy rises (W2/Q3).

W2. The agent designs appear overly simple and heuristic. Given the extensive preceding analysis, the designs should more directly address the specific problems identified. How do the three designs follow from the analysis?

Q3. Why do the three proposed agent designs improve RCA accuracy?

The designs are deliberately simple. Each is built to supply one kind of support that our analysis finds missing, and we evaluate it as a diagnostic probe of whether that support is enough. Table 7 maps each design to the condition it targets, the mechanism it is meant to supply, and what we observed.

Table 7 (Appendix A.5). Analysis-to-design mapping. Hypothesized mechanisms are design intents. Observed results, on the 22 Telecom incidents, are exploratory.

DesignTargeted conditionHypothesized mechanismObserved
LGDRival prominence; concurrent breadthComponents establish local support before global comparisonLocal root support in 6 of 7 successes and 2 of 15 failures; 3 failures lost the root at the Reviewer's candidate limit
HGIEarly preference for a non-root hypothesisExplicit beliefs and targeted probes revise a preferenceRecovered 2 of 4 Agent-RCA failures in which the root had been observed; repeated updates took in support for the eventually selected component in 12 of 16 failures and 3 of 6 successes
SDCCandidates not directly investigatedStructured coverage, updates, and review keep candidates under examinationRoot-directed coverage returned evidence in all 5 successes; in 6 of 17 failures it did but the root never led numerically, and in the other 11 the root was never directly targeted; 14 of 17 failures ended on a higher-signal component

Each design gains where the agent faces the difficulty that the design targets, and the 139 incidents show this pattern. Coverage pays off where the root is already the strongest signal. There SDC recovers 16 Agent-RCA failures without a regression, because reaching the root's own evidence is enough. Comparison pays off where a rival is stronger, when each candidate first builds local support. There LGD recovers more failures than SDC (23 against 18, with 10 regressions each). HGI's revisions correct some early preferences, but its updates can also strengthen the candidate already favored. The gains thus follow the incident-side signal conditions our analysis measures, which is how the designs follow from the analysis.

The residual failures show what is still missing. Under high rival prominence, neither SDC nor HGI shows a reliable gain (Appendix A.5), and the residual failures of all three designs include cases in which evidence enters intermediate judgments, belief updates, or review without resolving competition with a plausible alternative hypothesis (Section 5.5). Appendix F.5 tests what these cases lack. Given the same evidence, an instruction to contrast the candidates did not select the root more often than a review of the same candidate list (7.1% vs. 11.3%), and given extra steps it did not beat continued investigation. What they need is evidence that separates the root from its rivals. The next design step is therefore support that seeks such evidence before committing, which is the target we call cross-hypothesis discrimination.

3. Writing clarity (W3)

W3. The writing is difficult to follow, and the presentation lacks clarity and organization.

We agree, and we revised the paper around its main thread. The argument follows the labeled root through four stages: concrete root observation, candidate inclusion, diagnostic leadership, and final root selection. Each stage now has one name used throughout, and a new table at the start of Appendix A defines every term of the analysis. The Figure 1 caption now walks through the pipeline and states which research question uses each part, and an error in the figure itself is corrected. Figures 4(c) and 4(d) are redrawn (our answers to Q1 and Q2). The link from the analysis to the designs is now explicit in Table 7, and the role of the designs as diagnostic probes is stated consistently in the Abstract, Contribution 3, Section 5.5, the Discussion, and the Conclusion.

AuthorsOpenReview ↗

Response to Reviewer EB49 (3/4)

4. Figure 4(c): low vs. high prominence, and GPT-4o vs. GPT-5.5 (Q1).

Q1. In Figure 4(c), why does GPT-5.5 perform better under low rival prominence than under high rival prominence, and why does GPT-4o perform worse than GPT-5.5?

We agree that Figure 4(c) was hard to read, and we redrew it so that each stage shows each model's low and high groups side by side. Both differences you ask about arise after the root has been observed.

For low versus high rival prominence, the root enters GPT-5.5's candidate set about equally often in both groups. The gap opens when the root should become the leading diagnosis. Under high rival prominence, the root usually enters the candidate set behind a stronger rival and rarely overtakes it later (Appendix E.2). This follows from how agents weigh candidates. In the failures we reviewed, relative anomaly magnitude was used to favor a non-root component or discount the root, without a recorded discriminating check, in 30 of 39 GPT-4o and 50 of 71 GPT-5.5 histories (Appendix F.2), and anomaly magnitude favors the stronger rival. High rival prominence only guarantees that stronger non-root components exist. Ranking them ahead of the root is the agents' behavior, and it is the competition our analysis describes.

For GPT-4o versus GPT-5.5, across all histories with a concrete root observation, GPT-4o falls behind at the two ends of this progression. It keeps an observed root as a candidate less often (57.1% against 89.7%), and once the root is the leading diagnosis, it selects it in the final diagnosis less often (60.7% against 90.1%). In between, the two models are close in whether a candidate root becomes the leading diagnosis (58.3% and 67.0%). A stronger model is thus better at keeping an observed root and at committing to it, but competition among the candidates remains hard for both. This is why a stronger backbone alone does not close the gap.

5. Figure 4(d): why the model ranking flips across paths (Q2).

Q2. Similarly, in Figure 4(d), why are the values higher for GPT-4o in some categories but higher for GPT-5.5 in others?

We agree that Figure 4(d) did not show that its paths partition the omissions, which is why the ranking seems to flip. We redrew it as one bar per model and group that sums to 100%: the share in which the root was selected, and the shares lost at each stage. A model that loses more roots at an early stage has fewer left to lose later. GPT-4o most often loses the observed root before it ever enters the candidate set. GPT-5.5 keeps more roots as candidates, so its losses move to the later stage in which the root remains a candidate but never becomes the leading diagnosis, which is also where its losses grow under high rival prominence. The flip therefore reflects where each model loses the root, not two different behaviors on the same path (Appendix E.1).

Reviewer_zFZ6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors analyzes executions on telemetry and do an analysis on the root cause and how it can be seen and found, without being confused with other subsequent reasons. They writers ask questions in order to improve the search for where the failure happened exactly.

Critical strengths

Strengths:

S1. The authors have comprehensive analysis and divide into key questions from which implications can be made into understanding root cause for errors in telemetry. S2. Problem formulation is explained clearly and related looks comprehensive, with works from the last year and positioning is good. Partial observability is an important problem to work on.

Critical weaknesses

Weaknesses:

W1. Novelty is about the analysis, and it feels to me with little novelty compared to related work [1] and contribution on the analyses parts. W2. RCA with partial observability is interesting problem, but I feel like partial observability is not significant part of the paper and is not evaluated as I would expect.

[1] JustDiag!: A Diagnostic Justification Engine for Accountable Root Cause Analysis (Bi et al., 2026)

Questions for authors

Questions: Q1. I would like to know how partial observability can be measured, and how partiality affects results. I assume that the more sparse the information is, we should see less success in diagnosing. Q2. Is it possible to take a method from the related work (for example JustDiag [1]) and work our a comparison against a stronger method? And if possible, on more than 22 cases of the Telecom incident.

5 comments
AuthorsOpenReview ↗

Response to Reviewer zFZ6 (1/3)

Thank you for the review. Your questions all ask what this paper adds, so we state it first. Our contribution is evidence of where and when RCA agents lose the root cause, obtained by measuring incident-side signal conditions independently of the agent. After a concrete root observation, greater rival prominence is associated with a lower likelihood that the root becomes the leading diagnosis or reaches the final answer. This post-observation diagnostic gap lies in distinguishing the root from plausible rivals, which is why we propose cross-hypothesis discrimination as a design target.

The table summarizes our answers and where the revision changed. Changes made for your comments are marked in green in the revised PDF.

ConcernOur answer in briefWhere in the revision
W1JustDiag proposes a system for accountable adjudication. Our contribution is an empirical finding about where existing agents lose the root.Related Work (Section 3)
W2, Q1Partial observability is measured from two sides, and masking confirms your expectation for the root's own evidence: final root selection is 56.7% with full telemetry, 30.0% with half hidden, and none with all hidden.Appendix C.1; Appendix C.2 (Table 11)
Q2JustDiag cannot be run here. On 139 incidents across Bank, Market, and Telecom, SDC and LGD improve over Agent-RCA (+17.3 and +15.8 points), whereas HGI does not.Appendix A.5

1. Novelty of the analysis (W1)

W1. Novelty is about the analysis, and it feels to me with little novelty compared to related work [1] and contribution on the analyses parts.

[1] JustDiag!: A Diagnostic Justification Engine for Accountable Root Cause Analysis (Bi et al., 2026)

We agree that JustDiag is closely related. It shares our motivation that RCA should be evaluated beyond final-answer accuracy and that competing hypotheses should be considered explicitly. The two works, however, ask different questions and make different contributions. JustDiag is a system-design paper. It introduces a diagnostic-justification architecture that structures evidence, findings, hypotheses, claims, and assessments to support accountable adjudication and calibrated non-closure.

Our work instead studies empirically how competition among plausible causes manifests in existing agentic RCA. We measure incident-side signal competition independently of agent behavior and align it with retrieved telemetry, expressed candidate states, diagnostic leadership, and final diagnoses, which locates the post-observation diagnostic gap stated above. Across three targeted agent designs, we further show that additional investigation, belief updating, local comparison, and review can still accumulate support for a candidate without resolving why it should be preferred over plausible alternatives.

Our novelty relative to JustDiag therefore lies not in proposing hypothesis comparison, which JustDiag implements. It lies in showing empirically that post-observation competition is a distinct diagnostic bottleneck, and on that basis in motivating cross-hypothesis discrimination as a design and evaluation target beyond generic evidence acquisition and process structure alone. Related Work (Section 3) now states this difference.

AuthorsOpenReview ↗

Response to Reviewer zFZ6 (2/3)

2. Measuring and testing partial observability (W2/Q1).

W2. RCA with partial observability is interesting problem, but I feel like partial observability is not significant part of the paper and is not evaluated as I would expect.

Q1. I would like to know how partial observability can be measured, and how partiality affects results. I assume that the more sparse the information is, we should see less success in diagnosing.

We agree that partial observability should be evaluated explicitly. In RCA, an agent sees only the telemetry within its observation scope, and its sequential investigation decides which part of it is returned (Section 2.2). The paper already measures this investigation-dependent view from two sides, but did not present these measures as an evaluation of partial observability. We do so below and add masking experiments. They confirm your expectation for the evidence about the root: the sparser it is, the lower final root selection.

How partial observability is measured. The first measure is what the investigation returns. Concrete root observation records whether a return contains the root's identifier with a telemetry datum (Appendix C.1). The root is concretely observed in 23.4% of GPT-4o and 68.4% of GPT-5.5 histories, and the two models select the root in 30.9% and 58.3% of histories with such an observation, against 8.8% and 12.2% without. Broader coverage of the candidates alone is not associated with final root selection (Appendix C.2). The second measure is what the incident brings into the returns. Because faults propagate, non-root components can show stronger or earlier anomalies than the root, and we measure this from each incident's full telemetry, independently of any agent, for example as rival prominence, the share of non-root components whose peak anomaly exceeds the root's (Appendix B.2). Among histories with a concrete root observation, the final diagnosis omits the root 17.2 points more often under high rival prominence for GPT-4o and 33.7 points more often for GPT-5.5 (RQ2).

How sparser evidence affects results. We masked telemetry in GPT-5.5 executions and compared each with the recorded full-telemetry histories of the same incidents. Masking removes a component's rows from every telemetry file while its name stays in the candidate list, as for an unmonitored component.

Table 11 (abridged, full table in Appendix C.2). Final root selection and concrete root observation under masking on the same 20 incidents (GPT-5.5). †The 14 incidents with a rival, whose recorded histories selected the root in 40.5%.

Hidden telemetryRoot selectionConcrete root observation
None (recorded)56.7%78.3%
Root's own telemetry, half30.0%55.0%
Root's own telemetry, all0.0%0.0%
Non-root components, half50.0%65.0%
Strongest rivals†57.1%85.7%

The sparser the root's own evidence, the less often the root is selected. Hiding half of the non-root components cost far less (6.7 points, against 26.7 for half of the root's own telemetry), and hiding the strongest rivals raised selection from 40.5% to 57.1%, the direction expected if rivals mislead. Less telemetry is thus not uniformly worse: what matters is whose evidence is missing.

Your expectation therefore holds for the evidence about the root. Under partial observability the agent must first obtain the root's own evidence and then separate the root from plausible rivals with it. Sparser root evidence makes both harder, and rival prominence makes the second harder even when the root is observed, which is why the paper targets cross-hypothesis discrimination.

AuthorsOpenReview ↗

Response to Reviewer zFZ6 (3/3)

3. A stronger comparison and more incidents (Q2).

Q2. Is it possible to take a method from the related work (for example JustDiag [1]) and work our a comparison against a stronger method? And if possible, on more than 22 cases of the Telecom incident.

A direct comparison with JustDiag is not possible at present. Its data are proprietary, we found no public code, and a reimplementation would not be faithful. If its code is released, we will run it on OpenRCA and analyze its traces.

For more incidents, we extended the evaluation beyond Telecom. Table 2 used 22 Telecom incidents because the designs were built as probes of our analysis and Telecom has only 23 single-root component queries. Applying the same incident rule to Bank and Market gives 139 shared incidents.

Table 6 (abridged, full table in Appendix A.5). Component accuracy on 139 shared incidents. Recovered and Regressed count recoveries of Agent-RCA failures and regressions of its successes. Differences are paired, with intervals from a within-dataset bootstrap.

MethodCorrect / 139Recovered / RegressedDifference (pp)
Agent-RCA22––
SDC4635 / 11+17.3 [+8.6, +25.9]
LGD4436 / 14+15.8 [+6.5, +25.2]
HGI1914 / 17−2.2 [−10.1, +5.8]

At this scale, SDC and LGD improve reliably over Agent-RCA, whereas HGI does not, which 22 incidents could not show. The gains also arise where the agent faces the difficulty that each design targets. Where at least one rival exceeds the root (94 incidents), LGD, which compares candidates after each builds local support, recovers 23 Agent-RCA failures against 10 regressions (SDC: 18 and 10). Where the root is already the strongest signal (33 incidents), covering the candidates is enough, and SDC recovers 16 failures without a regression. The incident-side signal conditions therefore indicate which support an incident needs: coverage where the root already stands out, and comparison where a rival is stronger.

Reviewer_k6ViOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a retrospective study of LLM-based root-cause analysis on 335 OpenRCA queries. It characterizes incidents using relative peak timing, rival prominence, and concurrent anomaly breadth, then aligns these properties with retrieved telemetry, expressed diagnostic states, and final diagnoses. The analysis finds that stronger non-root signals are associated with poorer diagnostic outcomes, particularly in whether an observed root becomes the leading diagnosis. Three exploratory agent designs evaluated on 22 Telecom incidents provide additional examples of successful recoveries and unresolved hypothesis competition.

Critical strengths

1.Finer-grained analysis beyond final accuracy. Separating root observation, candidate inclusion, diagnostic leadership, and final selection provides a useful descriptive framework for examining where diagnostic trajectories diverge from the labeled root. This distinguishes executions that would otherwise receive the same final-error label. 2.Careful statistical treatment and transparent scope. The study includes repeated executions for two models, accounts for shared incident windows through clustered uncertainty estimates, and applies multiple-testing correction. It also explicitly distinguishes expressed diagnostic states from latent beliefs and acknowledges that root observation does not establish evidential sufficiency.

Critical weaknesses

The analysis does not sufficiently distinguish diagnostic failure from uncertainty inherent in sequential investigation.

  1. Anomaly salience as an investigative cue versus a causal criterion. Showing that the most anomalous component is often not the root cause does not establish that prioritizing it is misguided. A prominent downstream symptom may provide valuable evidence for tracing the underlying cause. The analysis should distinguish investigating salient anomalies from treating salience itself as sufficient evidence of causation.
  2. Root observation versus root identifiability. A concrete root observation establishes that telemetry from the labeled root was displayed, not that the available evidence distinguished it from competing hypotheses. Consequently, root inclusion without diagnostic leadership may reflect justified uncertainty rather than failed evidence use. Evaluating intermediate rankings against the retrospectively known root risks hindsight bias unless the sufficiency of contemporaneous evidence is assessed.
  3. Descriptive findings versus actionable mechanisms. Temporarily favoring incorrect hypotheses and subsequently eliminating them can be a necessary part of sequential diagnosis. Without identifying available but unused discriminating evidence—or feasible investigations that would resolve the ambiguity—the observed trajectories do not establish an avoidable reasoning failure. The three architectural probes do not isolate this distinction, leaving the connection between the reported phenomenon, the proposed cross-hypothesis discrimination target, and practical improvement insufficiently demonstrated.
Questions for authors

1.Can you distinguish the value of investigating salient symptoms from the risk of directly attributing causation to them? For example, do the trajectories show when investigating a prominent non-root component helps recover the root through dependency tracing, versus when salience substitutes for causal evidence? 2.In cases where the root is observed but never becomes the leading diagnosis, was it identifiable from the evidence available at the relevant decision point? Could an independently adjudicated subset distinguish insufficient evidence from available but unused discriminating evidence? 3.How do the architectural probes validate the specific bottleneck identified by the analysis? Could a controlled intervention, with comparable information and investigation budgets, test whether explicitly contrasting competing hypotheses improves diagnosis beyond generic additional investigation or review?

10 comments
AuthorsOpenReview ↗

Response to Reviewer k6Vi (1/4)

Thank you for a careful review. The table summarizes our answers and where the revision changed. Changes made for your comments are marked in blue in the revised PDF, and the comments below give the details.

ConcernOur answer in briefWhere in the revision
W1, Q1Successes and failures alike start from the salient rival. Those that reach the root more often relate the rival to the root (19 of 31 vs. 18 of 48).Appendix F.6
W2, Q2In a label-blind review, the evidence favored the root in none of 30 assessed failures. In 18 of them the agent still committed, on evidence that favored no candidate.RQ2 Answer (Section 5.4); Appendix F.4
W3, Q3In the controlled test you propose, an instruction to contrast did not recover the root from the same evidence, whereas further investigation recovered more. The designs recover 7 of the 16 failures on 139 incidents in which the observed root never led.Section 5.5; Appendix A.5; Appendix F.5

Overall. The analysis does not sufficiently distinguish diagnostic failure from uncertainty inherent in sequential investigation.

Our measures do not penalize the provisional preferences that sequential investigation needs. We count only a final diagnosis that omits an observed root, not a temporary preference for another component. The root also counts as the leading diagnosis if it leads in any state after the concrete root observation, and such recoveries occur: for GPT-5.5, the root later became the leading diagnosis in 23 of the 106 histories where it entered the candidate set as non-leading (Appendix E.2). RQ2 locates where the loss occurs. Whether that loss reflects uncertainty that the evidence cannot resolve, and whether it could have been resolved, are your W2 and W3.

AuthorsOpenReview ↗

Response to Reviewer k6Vi (2/4)

1. Salience as an investigative cue vs. a causal criterion (W1/Q1).

W1. Anomaly salience as an investigative cue versus a causal criterion. Showing that the most anomalous component is often not the root cause does not establish that prioritizing it is misguided. A prominent downstream symptom may provide valuable evidence for tracing the underlying cause. The analysis should distinguish investigating salient anomalies from treating salience itself as sufficient evidence of causation.

Q1. Can you distinguish the value of investigating salient symptoms from the risk of directly attributing causation to them? For example, do the trajectories show when investigating a prominent non-root component helps recover the root through dependency tracing, versus when salience substitutes for causal evidence?

We agree that investigating a salient symptom can be sound, because tracing it can lead to the cause. Our trajectories show both cases you describe. Investigating a salient rival helps when the agent traces how it relates to the root. When the rival is weighed by its anomaly alone, salience stands in for causal evidence, and this becomes more common as rival prominence grows.

We annotated the 151 histories in which the agent concretely observed a salient rival, that is, a non-root component whose peak anomaly magnitude exceeds the root's (the components counted by rival prominence, Appendix B.2), without access to the outcome or to which component was the root (LLM annotators, adjudication, and agreement in Appendix F.6). In every history that weighed the rival, the agent compared the candidates by anomaly size. The salient rival is therefore where the investigation starts in successes and failures alike. Table 21 gives the paths these histories took.

Table 21 (Appendix F.6). Post-observation histories with a concretely observed salient rival, by path and rival prominence, with the number in which the agent related the rival to the root (All includes the 6 bottom-quartile histories).

PathAll: historiesAll: relatedMiddle: historiesMiddle: relatedTop quartile: historiesTop quartile: related
Weighed the rival, then selected the root31191691510
The rival became the final answer48181453211
Selected the root without weighing the rival19117110
Failed on another component533302201

What differed was whether the agent related the rival to the root, through a call or dependency, propagation, or which anomaly came first. It did so in 19 of the 31 histories that went on to select the root, but in 18 of the 48 in which the rival became the final answer. Two Bank incidents with the same pair of components illustrate the difference. In one, the agent found that "Mysql01 slow logs identify Tomcat01 as the client producing slow INSERT operations" and selected Tomcat01, the root. In the other, it selected the database because "the strongest and most coherent metric evidence is around Mysql01."

Under greater rival prominence, the second path is more likely. In the top quartile, the rival became the final answer in 32 histories and the investigation went on to the root in 15, whereas in the middle range the two were close (14 and 16). This is the post-observation loss that RQ2 associates with rival prominence, whereas prominence is not consistently associated with whether the root's telemetry is retrieved (Appendix C.1).

2. Root observation vs. root identifiability (W2/Q2).

W2. Root observation versus root identifiability. A concrete root observation establishes that telemetry from the labeled root was displayed, not that the available evidence distinguished it from competing hypotheses. Consequently, root inclusion without diagnostic leadership may reflect justified uncertainty rather than failed evidence use. Evaluating intermediate rankings against the retrospectively known root risks hindsight bias unless the sufficiency of contemporaneous evidence is assessed.

Q2. In cases where the root is observed but never becomes the leading diagnosis, was it identifiable from the evidence available at the relevant decision point? Could an independently adjudicated subset distinguish insufficient evidence from available but unused discriminating evidence?

We agree that a concrete root observation does not establish identifiability, which is why our diagnostic progression separates concrete root observation from the leading diagnosis. Your concern is whether the gap between them reflects justified uncertainty or failed evidence use. In the failures we assessed, uncertainty explains why the observed root did not become the leading diagnosis. That uncertainty is not external to the agent, however. Under partial observability, the evidence in hand is what the agent's own investigation returned, and the agent decides when to stop investigating (Section 2.1). Uncertainty therefore does not explain the commitment.

Was the root identifiable at the decision point? In these failures, it was not. For 36 histories in which the root was concretely observed, never led, and was omitted from the final answer, two LLM judges and then an author, all blind to the label, assessed which of five candidates (the root, the agent's leading diagnosis, its next-ranked candidate when there was one, and other components) the evidence returned before the final answer favored. A larger anomaly alone did not count as discriminating evidence. Appendix F.4 gives the procedure.

Table 19 (Appendix F.4). Candidate favored by the evidence returned before the final answer, as confirmed by an author blind to the label.

HistoriesnRootAnotherNot decisiveNot assessed
Failed: root observed, never led, not selected36012186
Succeeded127230

In none of the 30 failures we could assess did the evidence favor the root, whereas it did in 7 of the 12 successes. These failures therefore reflect insufficient evidence for the root at the point of commitment, not discriminating evidence left unused, and the RQ2 Answer now says "favoring" rather than "recognizing". The same contrast addresses hindsight. The stages we track are defined with the label, yet they agree with a reading of the evidence made without it.

Is the loss then justified uncertainty? Uncertainty explains why the root did not lead in these failures, but not the commitment. In 18 of the 30 failures, the review found no evidence favoring any of the five candidates, yet the agent named one of them as the cause, after a median of 8 of 25 steps and without reaching the step limit. This judgment needs no label. In the other 12, the evidence pointed to another component, which is consistent with the competition from non-root signals that our incident-side analysis measures (RQ1).

The agents thus had root telemetry but no grounds to prefer the root, and in most cases no grounds to prefer the candidate they chose. Cross-hypothesis discrimination, as defined in our RQ3 Answer, includes determining whether the available evidence establishes such a preference at all. In these failures it did not, and the agents committed regardless. What fails after observation is therefore establishing that preference before committing, not observing the root or reading evidence that favored it. Our answer to W3 tests what supplies such evidence.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful and constructive reviews. The analyses we added in answering the reviewers' questions together strengthen the paper's central finding, the post-observation diagnostic gap: diagnosis depends on observing the root's own evidence, but agents can still commit without evidence that separates the root from the other candidates. This is why we propose cross-hypothesis discrimination as a design and evaluation target.

In the revised PDF, changes are colored by reviewer: blue for k6Vi, green for zFZ6, orange for EB49, and pink for general corrections. The table lists the main changes and the analyses added for this response, the concerns they answer, and where to find them. Each reviewer's response then opens with a table of their own concerns.

Change or new analysisAnswersWhere
Agent designs stated as diagnostic probes, not a general accuracy gaink6Vi W3; EB49 W1, W2Abstract, Contribution 3, Section 5.5, Discussion, Conclusion
Coverage of what investigations return, as a measure of partial observabilityzFZ6 W2, Q1Appendix C.2
How our study differs from JustDiagzFZ6 W1Section 3
Terms table, one name per stage, pipeline walkthroughEB49 W3Appendix A, Section 2.2, Figure 1
Figure 4(c) and 4(d) redrawnEB49 Q1, Q2Figure 4
New: paths through a salient rival, annotated blind to the outcomek6Vi W1, Q1Appendix F.6
New: label-blind review of the evidence at the final decisionk6Vi W2, Q2Appendix F.4
New: controlled tests of an instruction to contrast (replay; matched budget)k6Vi W3, Q3Appendix F.5
New: masking of the root's own telemetry (half or all), of non-root components, and of the strongest rivalszFZ6 W2, Q1; EB49 Q4Appendix C.2
New: the three designs on 139 incidents across Bank, Market, and TelecomzFZ6 Q2; EB49 W1, W2Appendix A.5
New: the three designs with GPT-5.5 and Qwen3.5-27BEB49 W1Appendix A.5
AuthorsOpenReview ↗

Rebuttal Summary for Area Chair

Dear Area Chair,

We thank you for overseeing the discussion and all reviewers for their constructive engagement. Our paper empirically examines where agentic RCA trajectories diverge from the labeled root, distinguishing root observation, candidate inclusion, diagnostic leadership, and final selection. It finds that agents can observe the root's own telemetry yet commit before separating it from the other candidates, which motivates cross-hypothesis discrimination, using the available evidence to determine whether and why one root-cause hypothesis should be preferred over plausible alternatives, as a design target. The discussion led us to clarify an important boundary: observing telemetry from the root does not establish that the available evidence identifies it. As the discussion period ends, we briefly summarize the concerns raised by each reviewer, how they were addressed, and the reviewers' final responses.

ReviewerMain concernsHow addressedFinal response
k6ViSalience as a cue versus a causal criterion (W1, Q1); whether an observed root was identifiable, and whether failures with and without evidence for it are separated (W2, Q2, follow-up); whether feasible investigations could resolve the ambiguity (W3, Q3).W1, Q1: paths reaching the root more often relate the salient rival to it, so salience helps as a cue but misleads as a criterion. W2, Q2, follow-up: labeled failures mostly committed without evidence for the root, often with steps remaining; Table 19 separates the two kinds of failure in 68 histories. W3, Q3: in the reviewer's controlled test, further investigation recovered the root more often than contrasting on the same evidence, yet agents stopped early even when told to seek separating evidence, committing before the candidates were separated.Appreciated the rebuttal and the additional analyses; retained the rating.
zFZ6Novelty relative to JustDiag (W1); how partial observability is measured and affects results (W2, Q1); comparison with a stronger method and on more than 22 incidents (Q2).W1: our contribution is empirical evidence of where existing agents lose the root, not a system for accountable adjudication like JustDiag. W2, Q1: partial observability is measured from what investigations return and what incidents bring into them, and masking experiments confirmed its effect on root selection. Q2: JustDiag cannot be run here; on 139 incidents, SDC and LGD improve over Agent-RCA.Acknowledged the strengthened evaluation of partial observability and raised the informativeness score to 2; retained the overall rating.
EB49Evaluation with a single model on 22 incidents (W1); how the designs follow from the analysis and why they improve accuracy (W2, Q3); clarity of writing (W3); the differences in Figure 4(c) and 4(d) (Q1, Q2); whether better design or more observation should come first (Q4).W1: extended the evaluation to 139 incidents and two more backbones. W2, Q3: each design supplies support the analysis finds missing, so its gains follow the incident-side signal conditions, and their residual failures point to cross-hypothesis discrimination as the next step. W3, Q1, Q2: clarified the writing and explained the differences in Figure 4(c) and 4(d). Q4: the root's own evidence must be observed first, after which design that seeks evidence separating the root from its rivals matters most.Stated that the response addressed some concerns and that the paper offers a fresh perspective and a valuable contribution; raised the rating to Weak Accept.

Overall, the added analyses strengthen the empirical contribution while clarifying its limits. Root observation should not be equated with root identifiability, and post-observation errors should not uniformly be interpreted as avoidable reasoning failures. The controlled comparison specifies what cross-hypothesis discrimination requires: when the evidence in hand does not distinguish the hypotheses, the agent must seek evidence that does before committing. We hope the paper will be evaluated primarily as an empirical study of agentic RCA, with the architectural probes serving as exploratory evidence rather than definitive validation of a mechanism.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.