- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces LOCA-RCA, a label-free counterfactual benchmark for measuring whether an LLM-based root-cause diagnosis actually depends on the evidence cited by the model. Given a baseline diagnosis and its cited evidence set, the benchmark re-runs the model after (1) removing the cited evidence, (2) removing a size-matched set of non-cited evidence, and (3) retaining only the cited evidence. I have concerns about the novelty of the core intervention, the construct being measured, and the extent to which the current setting supports a general-purpose benchmark for RCA agents.
Critical strengths
-
The paper asks a clear and practically relevant question. Instead of evaluating only whether a diagnosis is correct, the paper asks whether the diagnosis actually changes when the evidence that supposedly supports it is removed.
-
The matched control is good. The paper does not interpret raw sensitivity to evidence deletion as evidence dependence. The random-removal control shows why this distinction matters.
Critical weaknesses
1. The conceptual contribution is narrower than the scale of the paper suggests.
The main intervention closely follows prior sufficiency and random-removal designs; the primary contribution is adapting and packaging them for label-free RCA evaluation.
2. The presentation is dense and very difficult to follow.
The paper relies heavily on long and partially redundant tables, while relatively few figures are used to summarize the main findings; several figures also contain small labels and annotations that are difficult to read. For example, Tables 1, 12, 13, and 14 repeatedly summarize the status, limitations, and follow-up requirements of largely overlapping claims. More importantly, some interpretations change across sections: the main text emphasizes that small models express dependence mainly through FLIP while frontier models do so through ABSTAIN, whereas Appendix F.10 later shows that this ordering reverses after conditioning on non-abstention and concludes that the apparent ranking is partly induced by censoring rather than reflecting an intrinsic model-level property.
3. The practical meaning of a high dependence score remains unclear.
A diagnosis can strongly depend on its cited evidence while still being incorrect, and the metric is not shown to be an effective correctness screen.
4. The measured score is strongly conditional on the evaluation setup.
Dependence scores change considerably with evidence-pool construction and citation elicitation, which limits their comparability across settings.
5. The current evaluation is still far from realistic RCA-agent deployment.
The benchmark uses single-shot diagnosis over a fixed evidence pool rather than an iterative agent that retrieves evidence, invokes tools, and updates its diagnosis over time.
6. The practically recommended setting is not evaluated at the same scale as the main benchmark.
The paper recommends budgeted citations and stronger controls, but the top- setup is evaluated only on a limited subset of models and deployments.
Questions for authors
Q1. What is the main additional capability provided by LOCA-RCA compared with existing comprehensiveness/sufficiency and random-removal evaluations?
It would be helpful to clarify what can be measured or concluded in the proposed RCA formulation that could not be obtained by directly applying these existing counterfactual evaluation methods.
Q2. What concrete operational decision should an engineer make based on a high or low LOCA-RCA score?
Since high evidence dependence does not imply a correct diagnosis, it is unclear whether the intended use is trust calibration, escalation, diagnosis rejection, or primarily post-hoc auditing.
Q3. How should LOCA-RCA scores be compared across different evidence-pool construction and citation-elicitation settings?
Given the substantial variation in absolute scores under different setups, is there a principled normalization that makes scores comparable across systems and deployments?
Q4. Does the proposed metric remain informative in a realistic agentic RCA workflow?
An evaluation involving iterative retrieval, tool use, and hypothesis revision would help establish whether the observed evidence-dependence behavior transfers beyond the current fixed-pool, single-shot setting.
Q5. Can the authors evaluate the recommended seetings more systematically?
It would strengthen the practical claims to apply the top-k citation budget, salience-matched controls, and repeated base-step evaluation across substantially more model × deployment combinations.