- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies evidence selection in retrieval-augmented generation (RAG) and argues that guaranteeing the retention of each relevant passage individually does not guarantee that the final context retains all required support. To address this, the authors calibrate a single threshold on the maximum deletion score over each question’s annotated supporting items, which authorizes all retained-subset proposals at once with an O(K) scan, and they characterize when this reduction holds through minimal harmful sets. They further show that when retrieval itself misses supporting evidence, every downstream gate restricted to the retrieved candidates faces a provable coverage ceiling, and they evaluate the approach on HotpotQA, 2WikiMultiHopQA, and MuSiQue, including a comparison between BM25 and title-linked retrieval on 2Wiki. However, the theoretical contribution is modest, since the main reduction is standard order-statistic calibration combined with a simple monotonicity argument and the coverage ceiling follows from a short probabilistic argument, although the minimal-harmful-set characterization and the oracle lower bounds are a reasonable addition. The linear-time guarantee also relies on annotated supporting sets with singleton witnesses; the paper acknowledges in theory that alternative reasoning paths break this reduction, but the experiments only use the single support annotations provided by the datasets and do not test incomplete or multi-path annotations. In addition, the fixed-frame authorizer only controls support omission, and in the corresponding MuSiQue grid the answer-error upper bounds remain above 60%, so answer correctness is certified only separately through the open-retrieval audits. Finally, much of the empirical improvement comes from the learned scorer and the stronger retriever rather than the calibration method itself, as the authors’ own scorer-by-calibration comparison suggests, while the BM25 baseline misses complete support on most questions and the linked retriever is developed on the target corpus.
Critical strengths
- Rigorous and transparent statistical auditing. The confirmations use frozen protocols and ID-disjoint roles, and the multi-policy audits allocate confidence simultaneously. The paper also reports failed audits (Table 13). The key endpoints are reproducible from the reported counts (e.g., 83/2000 -> 4.96%; =.64746 -> 37.11% ceiling; 10.12%/3.95%/43.04% for the linked joint gate). Why it matters: the theory is modest, so the empirical evidence is the paper's main claim to acceptance. This rigor makes that evidence trustworthy for the stated populations.
- A clear demonstration that per-snippet guarantees do not compose. The matched scorer x calibration design (Table 3) separates validity, which comes from question-level calibration (~10% -> ~4% omission), from compression and accuracy gains, which come from the scorer. Why it matters: this is a useful and correct diagnostic for the RAG risk-control community, and it is the paper's most transferable insight.
Critical weaknesses
- The theoretical contribution is modest. The main reduction is standard order-statistic calibration combined with a simple monotonicity argument: Eq. 8 always returns a superset of , so calibrating suffices. The coverage ceiling (Eq. 13) follows from a short probabilistic argument. The minimal-harmful-set characterization and the oracle lower bounds are a reasonable addition, but they do not change this overall picture. Why it matters: the case for acceptance therefore rests largely on the empirical contribution.
- The guarantee is narrow, and much of the empirical gain is not attributable to the method. (a) The linear-time guarantee relies on annotated supporting sets with singleton witnesses. The paper acknowledges in theory (Theorem 1(ii)) that alternative reasoning paths break this reduction, but the experiments only use the single support annotations provided by the datasets and do not test incomplete or multi-path annotations. (b) The fixed-frame authorizer only controls support omission. In the corresponding MuSiQue grid (Table 8), the answer-error upper bounds remain above 60%, so answer correctness is certified only separately, through the open-retrieval audits. (c) Much of the empirical improvement comes from the learned scorer and the stronger retriever rather than the calibration method itself, as the authors' own scorer-by-calibration comparison (Table 3) suggests. The BM25 baseline misses complete support on most questions (13,082/20,000 draws), and the linked retriever is developed on the target corpus. Why it matters: the headline results say little about the value of the proposed calibration in realistic settings.
Questions for authors
- Beyond standard order-statistic calibration of and the monotonicity of Eq. 8, what does Theorem 1(i)/(iv) establish that existing risk-control frameworks (e.g., conformal risk control and learn-then-test, cited in Section 2) do not already imply? A precise answer could change my assessment of the theoretical contribution.
- When a question admits several sufficient support sets, what exactly does the authorizer certify? How often do the datasets' single annotations miss valid alternative reasoning paths?
- What is the end-to-end inference cost of the MuSiQue authorizer, including the per-paragraph Qwen2.5-7B necessity features used by the learned scorer? The reported prompt-token savings (e.g., 37.71% in Table 3) count only generator-input tokens.
2 comments
Thank you for the clarifications and additional experiments. The new calibration comparison is helpful. With the scores and marginal-quantile convention fixed, the reduction in support omission from 10.20% to 4.85% provides clearer evidence for the effect of question-level calibration. The timing study also answers my cost question.
My main concern is still the extent of the contribution. I understand the revised theoretical positioning, and I find the minimal-witness characterization useful. However, the main reduction for annotated-support omission remains fairly direct, so my assessment of the theoretical novelty has not changed substantially.
The examples of alternative reasoning paths help clarify the guarantee. What remains unclear is how often preserving the annotated support differs from preserving sufficient evidence in practice. The added examples establish that this distinction matters, but do not quantify its impact. Similarly, the 62.20% EM error in the MuSiQue cap-20 setting does not contradict the support guarantee, but shows that controlling this loss alone is far from ensuring reliable answers. I acknowledge that the separate open-retrieval audits provide stronger answer-risk results for their evaluated policies.
The cost measurements also make the practical tradeoff clearer: once scoring is included, the learned pipeline takes 4.50 times as long as full evidence on the measured sample. The shorter context therefore does not give an overall speedup here. In addition, the timed online implementation changes some selected contexts and cannot directly use the frozen-score risk certificate. I appreciate that the revision reports this explicitly, but it leaves the certified quality and measured online cost attached to different execution settings.
The response has addressed several of my questions, and I give the authors credit for the additional checks and clearer claims. Nevertheless, the remaining concerns about novelty and practical scope are substantial enough that I will keep my original recommendation.
The endpoint check exposes a real answer-risk failure. We retain that result and have narrowed the theoretical claims to the event representation, computation limits and fixed-candidate coverage bound. Blue text in the revised manuscript marks the main substantive additions and changes relative to the original submission; wording-only edits are not marked individually.
Beyond CRC/LTT. Existing frameworks apply once the all-action loss is supplied; CRC already considers suprema over perturbations. Theorem 1(i) gives the exact event identity for our restoration rule. With scores fixed, the empty proposal attains the worst event, covering all 2K proposals without subset enumeration. Part (iv), c(1−rs)≤1−Pr(B=1), is a sharp bound for every subset gate sharing those candidates, including an oracle. Neither part provides a new statistical calibration guarantee. Their purpose is to identify the event to calibrate and the coverage limit that gate tuning cannot overcome. Parts (ii)-(iii) characterize minimal harmful witnesses and the difference between monotone and unrestricted exact oracle access. The revision states these roles without presenting the elementary reductions as difficult proofs.
Multiple paths and annotations. Our loss records failure to retain S. When S is sufficient, preserving it keeps one valid path, but omitting it may still leave an alternative. When S is insufficient, preserving the annotation does not ensure sufficient evidence. Actual candidate frames illustrate both issues. For MuSiQue
2hop__61273_20093, the annotated writer-to-death-year route uses 2,11. Non-support paragraph 15 also names Mercury as the writer, so 11,15 supplies the recorded answer 1991 without the annotated writer paragraph. A Star Wars example replaces the annotated creator paragraph with a non-support infobox. A further case links Bamberg in Germany to Bamberg, South Carolina, exposing an entity mismatch in the annotated chain. Same-title support/non-support overlap occurs in 421/2,000 questions; this page-level count cannot estimate the still-unknown frequency of alternative reasoning paths. All 98,160 finite event tests pass. Independent programs check case provenance, indices and textual anchors, which verify the records but do not measure semantic prevalence.Answer risk and attribution. At cap-20, only 756 of 2,000 MuSiQue answers are correct: empirical EM error is 62.20%. A tighter confidence bound would not resolve this failure, which remains reported. This answer loss uses benchmark strings and aliases rather than independent factual adjudication. For support risk, a new replay holds the marginal-quantile convention fixed. Grouping by question lowers learned omission from 10.20% to 4.85%, followed by 4.15% under the stronger high-probability rule. The marginal point's 5.72% test upper endpoint still fails the original 5% high-confidence gate. Compression and EM gains depend on scoring. In the retrieval experiment, the developed systems have different fitted selectors. The oracle ceiling covers all selector capacities on the original BM25 candidates, but its support loss and population remain those of the audit; it supplies no unseen-corpus generalization claim.
Actual MuSiQue cost. We timed 64 fixed questions with three repetitions across five arms, yielding 960 complete runs. The interval includes online embeddings, per-paragraph Qwen2.5-7B necessity inference, lexical/graph features, HGB, authorization and generation. Mean learned latency is 3.228 seconds against 0.718 for full evidence: ratio 4.50, descriptive question-paired 95% interval [4.39,4.60]. Dense and necessity scoring take 1.105 and 1.641 seconds; authorization alone takes about 8 microseconds. Scoring outweighs the generator-token savings. Candidates are preloaded, and loading and fitting are excluded. Prompt and token checks pass for every run, with agreement across repetitions. Online scoring changes one non-support paragraph in each of three learned contexts relative to the frozen quality trace. Replaying the original batches recovers the frozen scores, and substituting necessity features reproduces the context changes. These results require separate online sample-quality reporting; the original 4.96% endpoint cannot be transferred. Section 6.4 and Appendix B give the MuSiQue breakdown and execution conditions separately from the older 2Wiki timings.