- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper evaluates existing claim-level conformal factuality filtering in multi-hop retrieval-augmented generation. It combines iterative retrieval with claim decomposition, relevance scoring, and split-conformal calibration, and reports results on three QA datasets using two language models. The main finding is that higher factuality targets increase response-level support while substantially reducing claim retention and the frequency of non-empty answers.
Critical strengths
The paper addresses a relevant practical question: whether reliable filtered outputs remain informative in multi-hop RAG.
Critical weaknesses
-
The contribution beyond existing conformal factuality studies is limited. The scoring function and calibration procedure are inherited from Conformal-RAG; the principal change is the upstream retrieval pipeline. Dependencies between retrieval hops do not themselves present a new obstacle to response-level conformal validity, provided calibration and test questions remain exchangeable under the same fixed pipeline. Moreover, the reported increase in full-support rates as filtering becomes more stringent follows directly from the nested filtering rule when scores and annotations are held fixed. The substantive contribution therefore needs to come from new empirical insights specific to multi-hop retrieval. However, the reliability-informativeness trade-off and the role of vacuous outputs have already been studied by Chen et al. (2026), and the additional insight provided here is not sufficiently clear.
-
The experimental design does not adequately isolate the effect of multi-hop retrieval. The only single-hop reference uses Llama on HotpotQA. There is no systematic paired comparison across retrieval settings or examination of how intermediate retrieval errors affect the retained claims. Consequently, the results do not establish whether the observed information loss is specific to multi-hop retrieval or reflects the general behavior of conformal filtering.
-
The independent evaluation sample is too small to support broad empirical conclusions. Each dataset contains only 60 questions, split into 30 calibration and 30 test examples. The 100 repetitions reuse the same generated responses and therefore measure partition sensitivity rather than performance across independent question samples. With 30 calibration examples, the standard conformal order-statistic index at the 95% target is 30 , so calibration depends on the largest score. This does not invalidate conformal coverage, but it makes the information-retention results particularly sensitive to extreme examples and calibration-set size. The current experiments do not distinguish this small-sample effect from the intrinsic difficulty of multi-hop factuality control.
Questions for authors
-
What finding is specific to multi-hop retrieval and goes beyond the reliability–informativeness trade-offs already documented in prior work?
-
Can the existing retrieval logs clarify how many questions actually use multiple hops and whether additional hops contribute evidence needed for the final answer?