acceptodds
Under review as a conference paper at ICLR 2027

Risk Control for Complete Evidence Actions: Uniform Authorization and Retrieval Limits

Abstract

Retrieval-augmented generation (RAG) systems must decide both which evidence to retain and whether to answer. Preserving each relevant snippet with high probability does not ensure that an answer retains all required support. RC-POBP uses one calibrated envelope to authorize all 2^K retained-subset proposals, including data-dependent choices. Given item scores, authorization requires an O(K) scan. We characterize when minimal harmful sets permit this reduction. With incomplete retrieval, the envelope separates missing from deleted support and gives a sharp coverage ceiling for every gate restricted to the retrieved candidates. Experiments on HotpotQA, 2Wiki, and MuSiQue distinguish calibrated support preservation, learned evidence compression, and selective answer quality. On MuSiQue, the authorizer removes 47.40% of evidence with a 4.96% support-omission upper bound while improving answer quality over full evidence. In a frozen 2Wiki audit, a linked-retrieval policy meets the 5% support-risk target with an answer-error upper bound of 10.12%. With joint 95% confidence, its coverage is at least 43.04%, above the 37.11% ceiling for every gate on the original BM25 candidates satisfying the same support-risk target. Its correct-answer yield also exceeds this ceiling. The comparison identifies a retrieval bottleneck that downstream gate tuning alone cannot overcome.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
retrieval-augmented generation, risk control, evidence selection, selective prediction, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6ibXOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies evidence selection in retrieval-augmented generation (RAG) and argues that guaranteeing the retention of each relevant passage individually does not guarantee that the final context retains all required support. To address this, the authors calibrate a single threshold on the maximum deletion score over each question’s annotated supporting items, which authorizes all retained-subset proposals at once with an O(K) scan, and they characterize when this reduction holds through minimal harmful sets. They further show that when retrieval itself misses supporting evidence, every downstream gate restricted to the retrieved candidates faces a provable coverage ceiling, and they evaluate the approach on HotpotQA, 2WikiMultiHopQA, and MuSiQue, including a comparison between BM25 and title-linked retrieval on 2Wiki. However, the theoretical contribution is modest, since the main reduction is standard order-statistic calibration combined with a simple monotonicity argument and the coverage ceiling follows from a short probabilistic argument, although the minimal-harmful-set characterization and the oracle lower bounds are a reasonable addition. The linear-time guarantee also relies on annotated supporting sets with singleton witnesses; the paper acknowledges in theory that alternative reasoning paths break this reduction, but the experiments only use the single support annotations provided by the datasets and do not test incomplete or multi-path annotations. In addition, the fixed-frame authorizer only controls support omission, and in the corresponding MuSiQue grid the answer-error upper bounds remain above 60%, so answer correctness is certified only separately through the open-retrieval audits. Finally, much of the empirical improvement comes from the learned scorer and the stronger retriever rather than the calibration method itself, as the authors’ own scorer-by-calibration comparison suggests, while the BM25 baseline misses complete support on most questions and the linked retriever is developed on the target corpus.

Critical strengths
  1. Rigorous and transparent statistical auditing. The confirmations use frozen protocols and ID-disjoint roles, and the multi-policy audits allocate confidence simultaneously. The paper also reports failed audits (Table 13). The key endpoints are reproducible from the reported counts (e.g., 83/2000 -> 4.96%; bLb_L=.64746 -> 37.11% ceiling; 10.12%/3.95%/43.04% for the linked joint gate). Why it matters: the theory is modest, so the empirical evidence is the paper's main claim to acceptance. This rigor makes that evidence trustworthy for the stated populations.
  2. A clear demonstration that per-snippet guarantees do not compose. The matched scorer x calibration design (Table 3) separates validity, which comes from question-level calibration (~10% -> ~4% omission), from compression and accuracy gains, which come from the scorer. Why it matters: this is a useful and correct diagnostic for the RAG risk-control community, and it is the paper's most transferable insight.
Critical weaknesses
  1. The theoretical contribution is modest. The main reduction is standard order-statistic calibration combined with a simple monotonicity argument: Eq. 8 always returns a superset of {i:s(i)≤γ}\{i:s(i)\le\gamma\}, so calibrating max⁡i∈Ss(i)\max_{i\in S}s(i) suffices. The coverage ceiling (Eq. 13) follows from a short probabilistic argument. The minimal-harmful-set characterization and the oracle lower bounds are a reasonable addition, but they do not change this overall picture. Why it matters: the case for acceptance therefore rests largely on the empirical contribution.
  2. The guarantee is narrow, and much of the empirical gain is not attributable to the method. (a) The linear-time guarantee relies on annotated supporting sets with singleton witnesses. The paper acknowledges in theory (Theorem 1(ii)) that alternative reasoning paths break this reduction, but the experiments only use the single support annotations provided by the datasets and do not test incomplete or multi-path annotations. (b) The fixed-frame authorizer only controls support omission. In the corresponding MuSiQue grid (Table 8), the answer-error upper bounds remain above 60%, so answer correctness is certified only separately, through the open-retrieval audits. (c) Much of the empirical improvement comes from the learned scorer and the stronger retriever rather than the calibration method itself, as the authors' own scorer-by-calibration comparison (Table 3) suggests. The BM25 baseline misses complete support on most questions (13,082/20,000 draws), and the linked retriever is developed on the target corpus. Why it matters: the headline results say little about the value of the proposed calibration in realistic settings.
Questions for authors
  1. Beyond standard order-statistic calibration of max⁡i∈Sqsq(i)\max_{i\in S_q}s_q(i) and the monotonicity of Eq. 8, what does Theorem 1(i)/(iv) establish that existing risk-control frameworks (e.g., conformal risk control and learn-then-test, cited in Section 2) do not already imply? A precise answer could change my assessment of the theoretical contribution.
  2. When a question admits several sufficient support sets, what exactly does the authorizer certify? How often do the datasets' single annotations miss valid alternative reasoning paths?
  3. What is the end-to-end inference cost of the MuSiQue authorizer, including the per-paragraph Qwen2.5-7B necessity features used by the learned scorer? The reported prompt-token savings (e.g., 37.71% in Table 3) count only generator-input tokens.
2 comments

The endpoint check exposes a real answer-risk failure. We retain that result and have narrowed the theoretical claims to the event representation, computation limits and fixed-candidate coverage bound. Blue text in the revised manuscript marks the main substantive additions and changes relative to the original submission; wording-only edits are not marked individually.

  1. Beyond CRC/LTT. Existing frameworks apply once the all-action loss is supplied; CRC already considers suprema over perturbations. Theorem 1(i) gives the exact event identity for our restoration rule. With scores fixed, the empty proposal attains the worst event, covering all 2K2^K proposals without subset enumeration. Part (iv), c(1−rs)≤1−Pr⁡(B=1)c(1-r_s) \le 1-\Pr(B=1), is a sharp bound for every subset gate sharing those candidates, including an oracle. Neither part provides a new statistical calibration guarantee. Their purpose is to identify the event to calibrate and the coverage limit that gate tuning cannot overcome. Parts (ii)-(iii) characterize minimal harmful witnesses and the difference between monotone and unrestricted exact oracle access. The revision states these roles without presenting the elementary reductions as difficult proofs.

  2. Multiple paths and annotations. Our loss records failure to retain SS. When SS is sufficient, preserving it keeps one valid path, but omitting it may still leave an alternative. When SS is insufficient, preserving the annotation does not ensure sufficient evidence. Actual candidate frames illustrate both issues. For MuSiQue 2hop__61273_20093, the annotated writer-to-death-year route uses 2,11\\{2,11\\}. Non-support paragraph 15 also names Mercury as the writer, so 11,15\\{11,15\\} supplies the recorded answer 1991 without the annotated writer paragraph. A Star Wars example replaces the annotated creator paragraph with a non-support infobox. A further case links Bamberg in Germany to Bamberg, South Carolina, exposing an entity mismatch in the annotated chain. Same-title support/non-support overlap occurs in 421/2,000 questions; this page-level count cannot estimate the still-unknown frequency of alternative reasoning paths. All 98,160 finite event tests pass. Independent programs check case provenance, indices and textual anchors, which verify the records but do not measure semantic prevalence.

  3. Answer risk and attribution. At cap-20, only 756 of 2,000 MuSiQue answers are correct: empirical EM error is 62.20%. A tighter confidence bound would not resolve this failure, which remains reported. This answer loss uses benchmark strings and aliases rather than independent factual adjudication. For support risk, a new replay holds the marginal-quantile convention fixed. Grouping by question lowers learned omission from 10.20% to 4.85%, followed by 4.15% under the stronger high-probability rule. The marginal point's 5.72% test upper endpoint still fails the original 5% high-confidence gate. Compression and EM gains depend on scoring. In the retrieval experiment, the developed systems have different fitted selectors. The oracle ceiling covers all selector capacities on the original BM25 candidates, but its support loss and population remain those of the audit; it supplies no unseen-corpus generalization claim.

  4. Actual MuSiQue cost. We timed 64 fixed questions with three repetitions across five arms, yielding 960 complete runs. The interval includes online embeddings, per-paragraph Qwen2.5-7B necessity inference, lexical/graph features, HGB, authorization and generation. Mean learned latency is 3.228 seconds against 0.718 for full evidence: ratio 4.50, descriptive question-paired 95% interval [4.39,4.60]. Dense and necessity scoring take 1.105 and 1.641 seconds; authorization alone takes about 8 microseconds. Scoring outweighs the generator-token savings. Candidates are preloaded, and loading and fitting are excluded. Prompt and token checks pass for every run, with agreement across repetitions. Online scoring changes one non-support paragraph in each of three learned contexts relative to the frozen quality trace. Replaying the original batches recovers the frozen scores, and substituting necessity features reproduces the context changes. These results require separate online sample-quality reporting; the original 4.96% endpoint cannot be transferred. Section 6.4 and Appendix B give the MuSiQue breakdown and execution conditions separately from the older 2Wiki timings.

Thank you for the clarifications and additional experiments. The new calibration comparison is helpful. With the scores and marginal-quantile convention fixed, the reduction in support omission from 10.20% to 4.85% provides clearer evidence for the effect of question-level calibration. The timing study also answers my cost question.

My main concern is still the extent of the contribution. I understand the revised theoretical positioning, and I find the minimal-witness characterization useful. However, the main reduction for annotated-support omission remains fairly direct, so my assessment of the theoretical novelty has not changed substantially.

The examples of alternative reasoning paths help clarify the guarantee. What remains unclear is how often preserving the annotated support differs from preserving sufficient evidence in practice. The added examples establish that this distinction matters, but do not quantify its impact. Similarly, the 62.20% EM error in the MuSiQue cap-20 setting does not contradict the support guarantee, but shows that controlling this loss alone is far from ensuring reliable answers. I acknowledge that the separate open-retrieval audits provide stronger answer-risk results for their evaluated policies.

The cost measurements also make the practical tradeoff clearer: once scoring is included, the learned pipeline takes 4.50 times as long as full evidence on the measured sample. The shorter context therefore does not give an overall speedup here. In addition, the timed online implementation changes some selected contexts and cannot directly use the frozen-score risk certificate. I appreciate that the revision reports this explicitly, but it leaves the certified quality and measured online cost attached to different execution settings.

The response has addressed several of my questions, and I give the authors credit for the additional checks and clearer claims. Nevertheless, the remaining concerns about novelty and practical scope are substantial enough that I will keep my original recommendation.

Reviewer_XPu9OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents theoretical grounding for evidence pruning. It computes envelope (highest harmful action score) and based on that, tries to prune retrieved evidence.

In my opinion the paper needs better restructuring to avoid confusions. It is a bit hard to read and the flow of the paper is not very smooth. It tries to explain and present both how to handle pruning of required evidence and retrieving missed evidence. As far as I understand, the paper argues and says it should evaluate combination of decisions compared to performing them separately. Combination of actions can be pruning, answer generation, stopping retrieval, etc. For this, it defines envelope (highest harmful admission score). For evidence pruning, paper proposes to train a score function for deletion using labelled data. On a calibration dataset, find the highest deletion score among passages labeled as required support. Then use these to choose a threshold and use this threshold on new questions.

The evaluation shows the selection of threshold using the process is working as desired. However, there exists work that analyzes context compression helps in RAG (e.g., [1]). This paper’s main contribution given these contexts is that, it uses calibration dataset to decide on a threshold to delete passages in retrieved context. In fact, paper presents a formal risk-control approach for evidence pruning. Current comparison is unclear on its practical advantage over established pruning-based methods for example. Further, I am not sure in addition to methodical analysis of the problem, the proposed solution has significant difference to existing related methods. Overall, I feel the process/method proposed by the paper it self is limited. This is in part due to difficult to follow flow of the presentation.

[1] Chirkova, Nadezhda, et al. "Provence: efficient and robust context pruning for retrieval-augmented generation." International Conference on Learning Representations. Vol. 2025. 2025.

Critical strengths

• Paper presents proof for evidence pruning and retrieval. In a sense these facts are known but the paper deeply analyzes these concerns.

Critical weaknesses

• Paper tackles evidence pruning and analyzing the limits caused by missed evidence. Evaluation involves multiple datasets. However, it does not seem to test on selecting a threshold works across datasets, which is more interesting.

• Keeping the evidence does not guarantee correct answer. LLM can still answer incorrectly even though we retain correct passages.

• Gains seem to be relying on how strong the scorer is and also the process needs labeled data. The paper could describe annotation requirements and whether changes to corpus, retrieval pipeline require calibration.

Questions for authors

• What is i.i.d. mentioned in line 55? Abbreviations better be explained when they are introduced first.

• Why don’t authors evaluate context pruning approaches (e.g., [1])? Why they are not comparable? Please explain and provide details.

1 comment

We reorganized the paper around the distinction between pruning within retrieved candidates and changing retrieval itself. The practical comparison now includes native Provence and complete online scoring costs. Blue text in the revised manuscript marks the main substantive additions and changes relative to the original submission; wording-only edits are not marked individually.

  1. Organization and novelty. The introduction and setup now ask which support-omission event covers every authorized subset in a fixed frame, then derive coverage targets that subset gates cannot reach when retrieval misses support. The latter bound identifies when the candidate factory must change; the gate cannot recover the missing passages. We credit standard calibration, including CRC's worst-case-loss formulation, and locate our contribution in the exact evidence-edit representation, its witness and computation limits, and the empirical diagnosis of those limits.

  2. Provence and practical cost. Official native Provence at .1/.5 receives the same candidates, ordering, answer model, prompt and decoding. On the already opened 2,000-question split, EM is 33.30 for full evidence, 37.75 for the frozen learned authorizer and 30.75/22.50 for native Provence. Mean prompt lengths are 2,591.9, 1,507.0 and 305.1/169.8 tokens. Provence compresses much more, so these settings compare quality against compression rather than performance at equal output budgets. Sentence-level fact-retention labels are unavailable, and paragraph identity alone cannot certify support preservation. A separately labelled paragraph-score adaptation retains 17.6355 paragraphs and omits support on 107/2,000 questions (5.35%, 95% upper bound 6.25%). It fails the original 5% gate; we did not retune it. Independent prompt and metric checks pass on all 10,000 records, as do fresh-process replays of 160 answers and 64 pruning calls. For cost, 960 complete online runs on 64 fixed questions give mean latencies of 0.718 seconds for full evidence, 3.228 for learned scoring, 0.490/0.443 for native Provence and 0.933 for the paragraph adaptation. Learned scoring is slower overall despite shorter generator inputs. Online scoring also changes three learned contexts, requiring separate sample-quality results rather than reuse of the frozen-score certificate. Section 6.4 and Appendix B report these secondary comparisons, whose test outcomes were already known before this study.

  3. Transfer and labels. Each experiment calibrates within its specified population; zero-shot threshold transfer across datasets remains unestablished. Changing the corpus, retrieval pipeline, scorer, support definition or deployment distribution generally requires fresh calibration or auditing. Reusing a certificate instead requires establishing its distributional or pointwise-containment conditions for the changed system. Labels enter scorer fitting on proper-training questions and threshold calibration on labelled calibration support sets. Online execution uses no support labels.

  4. Support versus answers and attribution. A generator can answer incorrectly even when all support is retained. The abstract and setup distinguish these losses, and the MuSiQue answer-risk failures remain in the results. Table 3 separates the learned scorer's compression and answer-quality gains from maximum calibration's reduction in support omission through additional retention. We also specify the confidence-convention difference between maximum calibration and the marginal positive-pair baseline.

  5. Notation. At first occurrence, i.i.d. now reads “independent and identically distributed.” The assumption concerns complete query-corpus traces; steps within a trace may remain dependent.

Reviewer_KqrYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  • It studies risk control for RAG systems. It prunes retrieved evidence and decide whether to answer.
  • Calibrates one "envelope" score (the maximum score over harmful actions) so that a single threshold certifies every evidence-pruning action at once.
  • Empirically, MuSiQue evidence is cut 47.4% with a 4.96% omission upper bound and better F1/EM than full evidence.
Critical strengths
  • Clean reduction.
  • Rigorous protocol; policies are frozen before fresh, ID-disjoint audits, with simultaneous bounds and replay verification. The paper also reports gates that fail.
Critical weaknesses

There is a range of nitpicky issues: limited novelty since the conformal techniques used here are from the existing literauture. The novelty lies on the application. I can also take issue with the strength of the gurantees.

Questions for authors

Previous box .

2 comments

The revision separates the evidence-selection result from the established calibration tools and states the scope of each guarantee more precisely. Blue text in the revised manuscript marks the main substantive additions and changes relative to the original submission; wording-only edits are not marked individually.

  1. Contribution. CRC and LTT calibrate a supplied loss, and CRC already includes worst-case perturbation losses. We use that principle. Theorem 1(i) identifies the exact all-proposal support-omission event for our restoration rule and gives its deletion-maximal implementation when scores are fixed. Parts (ii)-(iii) characterize minimal harmful witnesses and the oracle information required when singleton witnesses no longer suffice. Part (iv) bounds every subset selector sharing a fixed candidate factory. The first and fourth reductions are elementary. Their role is to expose the structural limits of evidence selection; the experiments then distinguish scoring, calibration and retrieval effects. The calibration theorem itself is not strengthened.

  2. Guarantee strength. Under the stated sampling assumptions, the fixed-frame certificate controls omission of the registered support set. If that set is sufficient, retaining it preserves a sufficient path; another valid path may make the annotation-based loss conservative. An insufficient annotation can instead pass the retention check without supplying enough evidence. The certificate covers neither annotation completeness nor the generator's answer, and it cannot be carried over to shifted distributions or scores recomputed after editing without further justification. We audit answer error separately for frozen answer-or-abstain policies. The abstract and setup now distinguish these two losses.

  3. Attribution. With learned scores fixed, maximum calibration retains more context and reduces support omission, but loses 2.15 EM points relative to positive-pair calibration (Table 3). Holding maximum calibration fixed reverses the comparison: learned scoring gains 4.70 EM points and reduces generator-input tokens by 37.71%. We now attribute support-risk control to calibration and the efficiency gains to scoring. Because the two calibrators also use different confidence conventions, this experiment alone does not isolate grouping.

To examine grouping separately, we added a support-only replay under the same marginal-quantile convention. Grouping by question lowers learned omission from 10.20% to 4.85%; the stronger high-probability rule then gives 4.15%. This empirical check distinguishes the event and confidence changes without adding a calibration theorem. Failed answer-risk audits remain reported, and the retrieval comparison is confined to the original BM25 candidates and audit population. The separate 64-question online replay takes 4.50 times the full-evidence generation time once learned scoring is included. Shorter generator inputs therefore do not produce an overall speedup here, and recomputed online scores do not inherit the frozen-score quality certificate.

Thanks to the authors for their response. I maintain to have concerns about this work, especially given the remaining concerns raising by other reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.