acceptodds
Under review as a conference paper at ICLR 2027

LOCA-RCA: A Label-Free Counterfactual Benchmark for Dependence on Cited Evidence in RCA Agents

Abstract

LLM agents increasingly produce root-cause diagnoses accompanied by citations to observability telemetry, yet nothing checks whether the diagnosis actually depends on the evidence it cites. We introduce LOCA-RCA (Label-free Observability Citation Ablation for Root Cause Analysis), a training-free and label-free benchmark that removes exactly the evidence an agent itself cited, re-runs it, and compares the outcome against removing an equal number of non-cited items. Across 7 models and 3 deployments (335 OpenRCA incidents, of which Market and Bank are two schemas of one testbed), citations are counterfactually load-bearing well beyond a size-matched removal of evidence the agent did not cite, for most models, and the dependence runs through two channels with different operational meanings: switching to a different root cause (flip) or refusing to answer (abstain). Reporting them separately reverses the naive ranking. Whole-pool citation makes specificity structurally unmeasurable rather than low, which we report as a benchmark output rather than a missing row; a citation budget removes the behaviour on the one deployment and two models where we tested it. A salience-matched control leaves the contrast intact in 7 of 9 model-deployment cells. Item-level human reliability is established under none of four annotation rounds, so the metric stands on its construction; and base accuracy against gold is 0-11% on two deployments, so this is an audit signal about grounding, not a filter for right answers. We release the construction scripts, probes, metrics, per-incident traces and annotation material.

Primary area:
datasets and benchmarks
Keywords:
root cause analysis, RCA agents, AIOps, observability, telemetry, LLM agents, counterfactual evaluation, evidence ablation, citation attribution, label-free benchmark, abstention, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rAQNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces LOCA-RCA, a label-free counterfactual benchmark for measuring whether an LLM-based root-cause diagnosis actually depends on the evidence cited by the model. Given a baseline diagnosis and its cited evidence set, the benchmark re-runs the model after (1) removing the cited evidence, (2) removing a size-matched set of non-cited evidence, and (3) retaining only the cited evidence. I have concerns about the novelty of the core intervention, the construct being measured, and the extent to which the current setting supports a general-purpose benchmark for RCA agents.

Critical strengths
  1. The paper asks a clear and practically relevant question. Instead of evaluating only whether a diagnosis is correct, the paper asks whether the diagnosis actually changes when the evidence that supposedly supports it is removed.

  2. The matched control is good. The paper does not interpret raw sensitivity to evidence deletion as evidence dependence. The random-removal control shows why this distinction matters.

Critical weaknesses

1. The conceptual contribution is narrower than the scale of the paper suggests.
The main intervention closely follows prior sufficiency and random-removal designs; the primary contribution is adapting and packaging them for label-free RCA evaluation.

2. The presentation is dense and very difficult to follow.
The paper relies heavily on long and partially redundant tables, while relatively few figures are used to summarize the main findings; several figures also contain small labels and annotations that are difficult to read. For example, Tables 1, 12, 13, and 14 repeatedly summarize the status, limitations, and follow-up requirements of largely overlapping claims. More importantly, some interpretations change across sections: the main text emphasizes that small models express dependence mainly through FLIP while frontier models do so through ABSTAIN, whereas Appendix F.10 later shows that this ordering reverses after conditioning on non-abstention and concludes that the apparent ranking is partly induced by censoring rather than reflecting an intrinsic model-level property.

3. The practical meaning of a high dependence score remains unclear.
A diagnosis can strongly depend on its cited evidence while still being incorrect, and the metric is not shown to be an effective correctness screen.

4. The measured score is strongly conditional on the evaluation setup.
Dependence scores change considerably with evidence-pool construction and citation elicitation, which limits their comparability across settings.

5. The current evaluation is still far from realistic RCA-agent deployment.
The benchmark uses single-shot diagnosis over a fixed evidence pool rather than an iterative agent that retrieves evidence, invokes tools, and updates its diagnosis over time.

6. The practically recommended setting is not evaluated at the same scale as the main benchmark.
The paper recommends budgeted citations and stronger controls, but the top-kk setup is evaluated only on a limited subset of models and deployments.

Questions for authors

Q1. What is the main additional capability provided by LOCA-RCA compared with existing comprehensiveness/sufficiency and random-removal evaluations?
It would be helpful to clarify what can be measured or concluded in the proposed RCA formulation that could not be obtained by directly applying these existing counterfactual evaluation methods.

Q2. What concrete operational decision should an engineer make based on a high or low LOCA-RCA score?
Since high evidence dependence does not imply a correct diagnosis, it is unclear whether the intended use is trust calibration, escalation, diagnosis rejection, or primarily post-hoc auditing.

Q3. How should LOCA-RCA scores be compared across different evidence-pool construction and citation-elicitation settings?
Given the substantial variation in absolute scores under different setups, is there a principled normalization that makes scores comparable across systems and deployments?

Q4. Does the proposed metric remain informative in a realistic agentic RCA workflow?
An evaluation involving iterative retrieval, tool use, and hypothesis revision would help establish whether the observed evidence-dependence behavior transfers beyond the current fixed-pool, single-shot setting.

Q5. Can the authors evaluate the recommended seetings more systematically?
It would strengthen the practical claims to apply the top-k citation budget, salience-matched controls, and repeated base-step evaluation across substantially more model × deployment combinations.

Reviewer_CcMeOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether an LLM's root-cause diagnosis actually depends on the telemetry it cites, and proposes a probe that can answer this without a gold root cause. For each OpenRCA incident, the authors build a bounded evidence pool of at most 24 anomalous log, metric, and trace items (in practice often 16), and ask the model to return a root cause together with the ids of the evidence it relies on. They then re-run the model three times: with exactly the cited items removed, with an equal number of non-cited items removed at random, and with only the cited items kept. The reported quantity is the contrast between the first two arms, so a model that is simply sensitive to a smaller pool scores zero. The contrast is split into a flip channel (a different root cause) and an abstain channel (refusal), and the authors argue the two call for different operational responses. Experiments cover seven models on three OpenRCA deployments (Market, Bank, Telecom; 335 incidents, with Market and Bank being two schemas of one testbed). The main findings are that the contrast excludes zero for every measurable model except Mistral-7B on Telecom; that small models express dependence mostly through flips and larger or frontier models mostly through abstention; that Claude-Opus-5 cites almost the entire pool, so its contrast cannot be estimated, which a k = 3 citation cap fixes on Telecom; and that the contrast largely survives a salience-matched control (7 of 9 cells). The paper also reports four human annotation rounds, none of which establishes item-level reliability, and notes that base accuracy against gold is 0 to 11% on Market and Bank, so the metric is offered as an audit signal rather than a correctness filter.

Critical strengths
  • The question matters for anyone deploying LLM diagnosis with citations, and the probe is built carefully. Pairing the cited removal with a size-matched random removal, adding a sufficiency arm, and checking against a null re-ask control (on Telecom the random arm changes the conclusion at exactly the null rate, while the cited arm sits 0.833 and 0.579 above it) is the right set of controls for this kind of claim. The salience-matched control is a good addition, and it does real work: it removes about half of Gemini-Flash's apparent advantage over Qwen-32B on Market.

  • The cite-everything result is the most practically useful finding in the paper. Claude-Opus-5 cites 97.8% of the pool on average and the entire pool on 88.7% of Market incidents, which leaves nothing to form a control. An aggregate faithfulness score would report this as perfect dependence. A simple "cite at most k = 3" budget brings its whole-pool rate on Telecom from 0.975 to 0.000 and makes the contrast estimable (+0.429). Anyone building citation-based audits for agents should know about this failure mode and the simple fix. I also appreciated that cross-model claims are recomputed on matched populations and on incident clusters, and that withdrawn readings are listed rather than quietly dropped.

Critical weaknesses
  • The novelty is narrower than the framing suggests. The authors acknowledge that the three-arm design comes from ERASER's comprehensiveness and sufficiency, with the random-rationale baseline from Carton et al., and that Terrenzi et al. use the same design concurrently. The paper then sets itself apart mainly by being label-free. However, ERASER's comprehensiveness is already defined on the model's own predicted class rather than on a gold label, so "remove the rationale and see whether the model's own prediction moves, relative to a random baseline" is not new either. What is left is applying this to OpenRCA, splitting the change outcome by whether the new answer is a refusal, and the cite-everything observation. The flip/abstain split is a reasonable reporting choice, but it is a partition of an existing binary outcome. Its claimed operational meaning is also exactly what the human study was meant to support and did not.

  • It is unclear how much of the contrast measures dependence and how much follows mechanically from the pool construction. After the cited items are removed, the diagnosed component is still mentioned by some remaining item in only 68%, 29%, and 35% of probed cases on Market, Bank, and Telecom. In most Bank and Telecom cases, then, the cited removal deletes every mention of the component the model named, and the model can hardly keep that answer. On Telecom, POST-HOC occurs once in 168 erased cases. A random removal of non-cited items will almost never do this. The paper reports that the contrast stays positive in both strata on Market and Bank, which helps, but the headline numbers mix the two. The abstain channel has a similar issue. It appears to track each model's base-step refusal habit: Mistral-7B never abstains at the base step on any deployment and has an abstain contrast of 0.000, while GPT-5.5 abstains on 78 of 148 Market incidents at base and has an abstain contrast of 0.671. "The two channels reorder the ranking" may largely mean "some models refuse more readily", which the paper itself says seven models cannot settle. Finally, every intervention is subtractive, and Appendix G concedes that a model re-answering near-randomly under any information loss is indistinguishable from one whose citations are load-bearing.

  • The metric is not connected to anything an operator could act on. Base accuracy is 0.000 to 0.070 on Market and 0.027 to 0.104 on Bank, so almost every measured dependence is the dependence of a wrong answer on its evidence. On Telecom, the case most worth flagging (a wrong diagnosis) is mostly missed: 57.3% of rows are wrong diagnoses the net contrast does not flag. The gain in precision against gold comes from the control arm, and the cited removal adds only +0.017 [-0.009, +0.043] on top of it. Only one model-deployment cell (Telecom Qwen-32B, n = 12) has enough correct-at-base incidents to check the contrast on correct answers. Human validity is not established either: Fleiss κ = 0.018 in round 2, κ = 0.000 at the round-3 gate, and κ = 0.369 on the round-4 main set against its pre-registered gate. The metric therefore lacks both a link to correctness and human agreement, and "stands on its construction", which is the very thing the second weakness questions. The setting is also a single-shot answer over a fixed, pre-built pool of at most 24 items rather than an agent that chooses what to retrieve, so "RCA agents" in the title claims more than the experiments test.

  • The evidence is thinner than "7 models, 335 incidents, 3 deployments" suggests, and the writing makes this hard to see. The 148 Market rows share only 14 distinct incident ids. Market and Bank are the same testbed. On Telecom, GPT-5.5 and Gemini-Flash are eligible on 19 of 51 incidents, and Claude-Opus-5 has one paired incident. Only 6 of 148 Market incidents are eligible for all seven models. After clustering and correction, the frontier/open separation survives only against the weakest open model (Holm 0.040 and 0.046). GPT-5.5's eligible set moved by 8 of 51 incidents between two identical runs. Apart from "the contrast is positive for most models" and the cite-everything result, most comparative findings end up exploratory, withdrawn, or unsettled. The paper is honest about this, but the 49 pages, and a main text dominated by status labels, pre-registration history, and reversed directions, leave a reader knowing in detail what is not claimed and with little sense of what to take away. A shorter paper built around the two robust findings would be easier to evaluate.

Questions for authors
  • Can you report, for every model on all three deployments, the contrast stratified by whether the diagnosed component survives the cited removal, and add a control that removes non-cited items mentioning the same component? This would show how much of the headline contrast is left once mechanical erasure of the answer is ruled out.

  • How do you separate the abstain channel from base-step refusal propensity? For example, does the abstain contrast still differ across models after conditioning on incidents where both models answered at base with similar confidence, or after normalising by each model's base abstention rate?

  • Have you tried a non-subtractive intervention, such as replacing the cited items with neutral placeholders of the same modality, or swapping in an item that points to a different component? Either would help separate dependence on specific evidence from a general response to losing information, which the current design cannot do.

  • How does your contrast differ from ERASER comprehensiveness minus a random-rationale baseline computed on the model's own prediction? If it is the same quantity, the contribution should be framed as an application and analysis study rather than a new probe.

Reviewer_CUxyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper creates and tests a mechanism to check weather a model actually relies on evidence it cites within its own diagnostic decision making.

Critical strengths

I liked the idea if this paper, and I think the mechanism is quite neat. It's an interesting and prephas even useful approach, if the right problem is matched to it.

The paper is clear and easy to read and most things are clearly explained.

Critical weaknesses

First, the paper is full of very LLM-y sentences, which I found a bit in your face. The paper is clear, but it feels like it came straight out if Claude and it makes it feel almost 'cheap' style wise.

More importantly, I feel like the authors are really missing out on potential by not identifying a setting where models do not use the cited evidence. This could be due to deception, ambiguity, conflicting information... I don't know and it's not really my job to find out. However, saying oh yes, models do use citations, we checked (while somewhat reassuring) makes the paper feel a bit like a "ok then" rather than something exciting. I would currently place at as a very nice workshop paper but a bit below what I would expect to see in a top tier conference.

Questions for authors

What motivated you to do this work? Was it a real problem you encountered or just curiousity?

What do you envision work building on this world do?

What do you think your readers should take away from this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.