acceptodds
Under review as a conference paper at ICLR 2027

RAG Control Is Not Answer Fidelity: An Auditable Protocol for the Sufficiency–Fidelity Gap in Multi-Hop RAG

Abstract

Retrieval-augmented generation (RAG) fails on multi-hop questions when later support is indexed under an intermediate answer rather than the original query, and inference-time controllers are typically tuned to retrieve more, plan better, and stop smarter. We audit whether those control objectives actually track answer quality. Five-seed paired experiments with Holm correction are run on the official HotpotQA distractor split through PR-RAG, a training-free bounded protocol that exposes dependency-conditioned planning, evidence accumulation, a sufficiency gate, typed failure recovery, and per-call cost as logged, inspectable state. Three findings define what inference-time RAG control does and does not buy. (i) The headline: at the deployed operating point, 208/500 stopping decisions had evidence sufficient for the gold chain yet wrong answers, and 0/6 registered EM contrasts among the evaluated control variants survive Holm correction—evidence sufficiency is necessary but not sufficient for correctness. (ii) Control primitives act locally: planning improves chain coverage; verification changes stopping and cost, not answer quality; added retrieval budget raises coverage without EM, so evidence coverage and answer accuracy are separable control targets. (iii) The audit instrument is itself measurable: a blind two-annotator audit puts the offline gold-chain proxy's false-negative rate at 43.2%, while audit telemetry still contains a learnable sufficiency signal (question-disjoint offline calibrator). The protocol and per-question traces are released as a reusable auditing resource, and the structure replicates across five Llama-3.1-8B seeds and a dense retriever. Closed-pool scope, non-established EM superiority, and cross-dataset generalization remain open.

Primary area:
generative models
Keywords:
retrieval-augmented generation, multi-hop question answering, inference-time control, evidence sufficiency, evaluation protocol, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Hir6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how evidence accumulation and stopping judgments relate to answer correctness in multi-hop RAG. The authors introduce PR-RAG, a training-free bounded protocol with four interfaces for planning, retrieval, verification, and typed reflection. Across five decoding seeds on a fixed 500-question HotpotQA distractor set with Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, the authors evaluate PR-RAG against five baselines to test how full-chain coverage relates to normalized EM and token-level F1.

Critical strengths
  • Across five decoding seeds on the fixed 500-question HotpotQA distractor set, PR-RAG increases full-chain coverage over NaiveRAG from 54.2% to 73.8% and token-level F1 from 55.4 to 60.9 on Qwen2.5-7B-Instruct, and increases full-chain coverage from 54.2% to 84.7%, F1 from 52.8 to 61.7, and EM from 41.2 to 47.6 on Llama-3.1-8B-Instruct.
  • In the five-seed no-verification ablation on Qwen2.5-7B-Instruct, verification reduces retrieval calls per question from 5.18 to 2.34 and LLM calls per question from 8.50 to 5.97 while keeping EM at 47.8 versus 46.7 without verification.
Critical weaknesses
  • The paper focuses on the sufficiency-fidelity gap and a coverage-fidelity control plane, but it evaluates answer correctness only with normalized EM and token-level F1 and never audits human-rated fidelity. While 34.32% of five-seed PR-RAG outcomes fall into Category B with complete coverage and incorrect EM, Appendix G wrote that Category B includes normalization discrepancies and cannot be labelled unsupported without a fidelity audit.
  • Most mechanism variants and the 2WikiMultiHoQA evaluation use only a single decoding seed. The w/o Planning ablation in Table 9, the reflection variants in Table 14, the end-to-end learned-gate replay in Section 6.3, and the 2WikiMultiHoQA subset in Section 5.5 run only on seed 42 and lack the five-seed paired comparisons used in the main protocol.
Questions for authors
  • Can the authors conduct a small human evaluation on Category B outcomes with complete coverage and incorrect EM to show how many are normalization discrepancies versus unsupported answers?
  • How do the w/o Planning ablation, the reflection variants, the end-to-end learned-gate replay, and the 2WikiMultiHoQA evaluation perform when run across all five decoding seeds 42 to 46?
Reviewer_Cwx6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits the gap between evidence sufficiency control and answer correctness in multi-hop RAG using PR-RAG, a training-free protocol logging planning, retrieval, verification, and reflection trajectories. Across 5 seeds on a 500-question HotpotQA distractor subset (Qwen2.5-7B and Llama-3.1-8B), the authors show that while planning boosts evidence coverage and F1, 34.32% of runs have complete gold-chain coverage yet wrong EM, and 0/6 EM gains over baselines survive Holm correction. A human audit further reveals a 43.2% false-negative rate in the standard gold-chain sufficiency proxy.

Critical strengths
  1. Uses 5-seed paired bootstrap and Holm correction, transparently reporting null EM results (0/6 significant) rather than over-claiming SOTA.

  2. Cleanly decouples module roles—Planning drives coverage, Verification controls cost, and Reflection fails at the reading/synthesis bottleneck.

Critical weaknesses

1.Evaluates only on HotpotQA's 10-paragraph distractor pool (k=3). Retrieving ~2.3 times per question samples most of the 10 paragraphs, inflating coverage and failing to reflect open-corpus retrieval.

  1. Proposes a "Fidelity Gate" (Gate 2) as the central takeaway but leaves it completely unimplemented.

  2. PR-RAG uses ~6 LLM calls vs. 1–3 for baselines without compute-matched controls. Official CRAG/Self-RAG/S2G-RAG checkpoints were replaced by prompt approximations due to missing environment/Hub access.

  3. How does PR-RAG (~6 LLM calls) compare against a 2-call baseline paired with 4 calls of self-consistency under a matched budget?

Questions for authors

Please refer to the above Weaknesses.

Reviewer_UA3POpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper analyzes RAG systems and how coverage and other metrics correspond to EM and fidelity etc. This is a very interesting topic but I am unable to recommend that the paper is accepted as the writing makes it very hard to understand and also to understand the exact contributions of the paper. Already the abstact is written badly:

"We audit this mismatch with PR-RAG, a training-free bounded protocol ex- posing dependency state, verification, typed failures, and per-call cost. " This is the second sentence of the abstract.

It then follows by an statement regarding a specific seed of a randomized algorithm (it seems more inforamtive to look at the typical case).

Another indication of the inappropriate writing is Figure 1 where text and boxes are unaligned and overlap etc.

Critical strengths

Didn't understand if there are major strengths.

Critical weaknesses

The writing is not at the standard one would expect for a submission.

Questions for authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.