- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper argues that the limited accuracy of current root cause analysis (RCA) agents under partial observability stems from non-root signals interfering with root-cause signals. It identifies and empirically examines three incident-side signal conditions: relative peak timing, rival prominence, and concurrent anomaly breadth. It also proposes three agent designs to mitigate these effects and improve RCA accuracy.
Critical strengths
- The paper addresses an important question: why current RCA agents perform poorly across many scenarios. It investigates interference from non-root signals and identifies three incident-side signal conditions, supported by empirical experiments.
- The paper proposes three agent designs to mitigate this interference and improve RCA accuracy.
Critical weaknesses
- The experimental evaluation is insufficiently comprehensive and covers too few models. For example, the results in Table 2 rely on a single model, which is insufficient to establish the general effectiveness of the proposed agent designs.
- The agent designs appear overly simple and heuristic. Given the extensive preceding analysis, the designs should more directly address the specific problems identified. How do the three designs follow from the analysis?
- The writing is difficult to follow, and the presentation lacks clarity and organization.
Questions for authors
- In Figure 4(c), why does GPT-5.5 perform better under low rival prominence than under high rival prominence, and why does GPT-4o perform worse than GPT-5.5?
- Similarly, in Figure 4(d), why are the values higher for GPT-4o in some categories but higher for GPT-5.5 in others?
- Why do the three proposed agent designs improve RCA accuracy?
- What should be prioritized to address interference from non-root signals: better agent design or more comprehensive observations? This remains an open question that requires empirical investigation.
7 comments
Response to Reviewer EB49 (2/4)
2. How the designs follow from the analysis, and why accuracy rises (W2/Q3).
W2. The agent designs appear overly simple and heuristic. Given the extensive preceding analysis, the designs should more directly address the specific problems identified. How do the three designs follow from the analysis?
Q3. Why do the three proposed agent designs improve RCA accuracy?
The designs are deliberately simple. Each is built to supply one kind of support that our analysis finds missing, and we evaluate it as a diagnostic probe of whether that support is enough. Table 7 maps each design to the condition it targets, the mechanism it is meant to supply, and what we observed.
Table 7 (Appendix A.5). Analysis-to-design mapping. Hypothesized mechanisms are design intents. Observed results, on the 22 Telecom incidents, are exploratory.
| Design | Targeted condition | Hypothesized mechanism | Observed |
|---|---|---|---|
| LGD | Rival prominence; concurrent breadth | Components establish local support before global comparison | Local root support in 6 of 7 successes and 2 of 15 failures; 3 failures lost the root at the Reviewer's candidate limit |
| HGI | Early preference for a non-root hypothesis | Explicit beliefs and targeted probes revise a preference | Recovered 2 of 4 Agent-RCA failures in which the root had been observed; repeated updates took in support for the eventually selected component in 12 of 16 failures and 3 of 6 successes |
| SDC | Candidates not directly investigated | Structured coverage, updates, and review keep candidates under examination | Root-directed coverage returned evidence in all 5 successes; in 6 of 17 failures it did but the root never led numerically, and in the other 11 the root was never directly targeted; 14 of 17 failures ended on a higher-signal component |
Each design gains where the agent faces the difficulty that the design targets, and the 139 incidents show this pattern. Coverage pays off where the root is already the strongest signal. There SDC recovers 16 Agent-RCA failures without a regression, because reaching the root's own evidence is enough. Comparison pays off where a rival is stronger, when each candidate first builds local support. There LGD recovers more failures than SDC (23 against 18, with 10 regressions each). HGI's revisions correct some early preferences, but its updates can also strengthen the candidate already favored. The gains thus follow the incident-side signal conditions our analysis measures, which is how the designs follow from the analysis.
The residual failures show what is still missing. Under high rival prominence, neither SDC nor HGI shows a reliable gain (Appendix A.5), and the residual failures of all three designs include cases in which evidence enters intermediate judgments, belief updates, or review without resolving competition with a plausible alternative hypothesis (Section 5.5). Appendix F.5 tests what these cases lack. Given the same evidence, an instruction to contrast the candidates did not select the root more often than a review of the same candidate list (7.1% vs. 11.3%), and given extra steps it did not beat continued investigation. What they need is evidence that separates the root from its rivals. The next design step is therefore support that seeks such evidence before committing, which is the target we call cross-hypothesis discrimination.
3. Writing clarity (W3)
W3. The writing is difficult to follow, and the presentation lacks clarity and organization.
We agree, and we revised the paper around its main thread. The argument follows the labeled root through four stages: concrete root observation, candidate inclusion, diagnostic leadership, and final root selection. Each stage now has one name used throughout, and a new table at the start of Appendix A defines every term of the analysis. The Figure 1 caption now walks through the pipeline and states which research question uses each part, and an error in the figure itself is corrected. Figures 4(c) and 4(d) are redrawn (our answers to Q1 and Q2). The link from the analysis to the designs is now explicit in Table 7, and the role of the designs as diagnostic probes is stated consistently in the Abstract, Contribution 3, Section 5.5, the Discussion, and the Conclusion.
Response to Reviewer EB49 (3/4)
4. Figure 4(c): low vs. high prominence, and GPT-4o vs. GPT-5.5 (Q1).
Q1. In Figure 4(c), why does GPT-5.5 perform better under low rival prominence than under high rival prominence, and why does GPT-4o perform worse than GPT-5.5?
We agree that Figure 4(c) was hard to read, and we redrew it so that each stage shows each model's low and high groups side by side. Both differences you ask about arise after the root has been observed.
For low versus high rival prominence, the root enters GPT-5.5's candidate set about equally often in both groups. The gap opens when the root should become the leading diagnosis. Under high rival prominence, the root usually enters the candidate set behind a stronger rival and rarely overtakes it later (Appendix E.2). This follows from how agents weigh candidates. In the failures we reviewed, relative anomaly magnitude was used to favor a non-root component or discount the root, without a recorded discriminating check, in 30 of 39 GPT-4o and 50 of 71 GPT-5.5 histories (Appendix F.2), and anomaly magnitude favors the stronger rival. High rival prominence only guarantees that stronger non-root components exist. Ranking them ahead of the root is the agents' behavior, and it is the competition our analysis describes.
For GPT-4o versus GPT-5.5, across all histories with a concrete root observation, GPT-4o falls behind at the two ends of this progression. It keeps an observed root as a candidate less often (57.1% against 89.7%), and once the root is the leading diagnosis, it selects it in the final diagnosis less often (60.7% against 90.1%). In between, the two models are close in whether a candidate root becomes the leading diagnosis (58.3% and 67.0%). A stronger model is thus better at keeping an observed root and at committing to it, but competition among the candidates remains hard for both. This is why a stronger backbone alone does not close the gap.
5. Figure 4(d): why the model ranking flips across paths (Q2).
Q2. Similarly, in Figure 4(d), why are the values higher for GPT-4o in some categories but higher for GPT-5.5 in others?
We agree that Figure 4(d) did not show that its paths partition the omissions, which is why the ranking seems to flip. We redrew it as one bar per model and group that sums to 100%: the share in which the root was selected, and the shares lost at each stage. A model that loses more roots at an early stage has fewer left to lose later. GPT-4o most often loses the observed root before it ever enters the candidate set. GPT-5.5 keeps more roots as candidates, so its losses move to the later stage in which the root remains a candidate but never becomes the leading diagnosis, which is also where its losses grow under high rival prominence. The flip therefore reflects where each model loses the root, not two different behaviors on the same path (Appendix E.1).
Response to Reviewer EB49 (1/4)
Thank you for the review. The table summarizes our answers and where the revision changed. Changes made for your comments are marked in orange in the revised PDF.
1. More models (W1).
We agree that one model on 22 incidents cannot establish general effectiveness, and we extended the evaluation in both directions you raise. Each design targets a difficulty the analysis identifies, and we evaluate the designs as diagnostic probes of whether such targeted support is enough.
For more incidents, we ran the designs, with GPT-4o as in Table 2, on the 139 single-root component incidents shared across Bank, Market, and Telecom.
Table 6 (abridged, full table in Appendix A.5). Component accuracy on 139 shared incidents. Differences are paired, with intervals from a within-dataset bootstrap.
At this scale, SDC and LGD improve reliably over Agent-RCA, whereas HGI does not, which 22 incidents could not show.
For more models, we ran the same three designs with GPT-5.5 and with Qwen3.5-27B in non-thinking mode.
Table 8 (Appendix A.5). Correct component answers on the 22 matched Telecom incidents, by backbone.
LGD, which compares candidates after each builds local support, matches or exceeds Agent-RCA with every backbone. With Qwen3.5-27B it raises a 27B open model from 1 to 10 correct answers, the level of GPT-5.5 with Agent-RCA.
More backbones also show when each kind of support helps, because this follows where a backbone loses the root (Appendix A.5). GPT-4o loses about as much before concrete root observation as after it (16.9 and 16.2 points), so support that helps the agent reach the root's own evidence pays off, as SDC does on the 139 incidents. GPT-5.5 already observes the root in 68.4% of histories, against 23.4% for GPT-4o, and it loses about twice as much after observation as before it (28.5 against 14.6 points). There, SDC and HGI do not improve on Agent-RCA, while LGD holds its level. Stronger backbones thus make post-observation competition the main remaining difficulty, which is the target of cross-hypothesis discrimination that our paper proposes.