- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines how evidence accumulation and stopping judgments relate to answer correctness in multi-hop RAG. The authors introduce PR-RAG, a training-free bounded protocol with four interfaces for planning, retrieval, verification, and typed reflection. Across five decoding seeds on a fixed 500-question HotpotQA distractor set with Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, the authors evaluate PR-RAG against five baselines to test how full-chain coverage relates to normalized EM and token-level F1.
Critical strengths
- Across five decoding seeds on the fixed 500-question HotpotQA distractor set, PR-RAG increases full-chain coverage over NaiveRAG from 54.2% to 73.8% and token-level F1 from 55.4 to 60.9 on Qwen2.5-7B-Instruct, and increases full-chain coverage from 54.2% to 84.7%, F1 from 52.8 to 61.7, and EM from 41.2 to 47.6 on Llama-3.1-8B-Instruct.
- In the five-seed no-verification ablation on Qwen2.5-7B-Instruct, verification reduces retrieval calls per question from 5.18 to 2.34 and LLM calls per question from 8.50 to 5.97 while keeping EM at 47.8 versus 46.7 without verification.
Critical weaknesses
- The paper focuses on the sufficiency-fidelity gap and a coverage-fidelity control plane, but it evaluates answer correctness only with normalized EM and token-level F1 and never audits human-rated fidelity. While 34.32% of five-seed PR-RAG outcomes fall into Category B with complete coverage and incorrect EM, Appendix G wrote that Category B includes normalization discrepancies and cannot be labelled unsupported without a fidelity audit.
- Most mechanism variants and the 2WikiMultiHoQA evaluation use only a single decoding seed. The w/o Planning ablation in Table 9, the reflection variants in Table 14, the end-to-end learned-gate replay in Section 6.3, and the 2WikiMultiHoQA subset in Section 5.5 run only on seed 42 and lack the five-seed paired comparisons used in the main protocol.
Questions for authors
- Can the authors conduct a small human evaluation on Category B outcomes with complete coverage and incorrect EM to show how many are normalization discrepancies versus unsupported answers?
- How do the w/o Planning ablation, the reflection variants, the end-to-end learned-gate replay, and the 2WikiMultiHoQA evaluation perform when run across all five decoding seeds 42 to 46?