acceptodds
Under review as a conference paper at ICLR 2027

To Abstain or to Re-Rank? A Regime Law for Risk-Controlled Generation

Abstract

Risk-controlled generation returns a prediction set of candidate answers while controlling the probability that none is admissible. Existing methods calibrate sampling and filtering rules to satisfy this constraint, but the smallest prediction set permitted by the risk budget is not characterized. We study this efficiency problem directly. Classical selective prediction gives the optimal one-answer rule, and its instantiation here yields a simple regime law. Let be the Bayes top-1 error and the target miscoverage risk. If , a valid one-answer predictor has slack and average prediction-set size is reduced by abstaining on low-confidence questions. If , no one-answer predictor is valid and efficiency depends on ranking multiple candidates. We also show that a score carrying only within-question rank cannot selectively abstain because it lacks a confidence scale across questions. These results motivate ACF-LV, a single-stage conformal filter with a calibration-gated verifier. On three QA datasets and five language models, the high-probability single-stage variant gives smaller prediction sets on 13 of 15 settings. Under a unified end-to-end protocol, ACF-LV reduces mean APSS from 4.389 to 1.889, while all eight testable predictions of the regime law agree with the observed effect of re-ranking. Code: https://anonymous.4open.science/r/acflv-A6E8/README.md

Primary area:
foundation or frontier models, including LLMs
Keywords:
Risk-controlled generation, Conformal prediction, Selective generation, Abstention, Answer re-ranking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sXVxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates language models that produce multiple candidate answers per query. It leverages conformal methods to bound the risk of returning an answer set with no valid acceptable answers within a pre-specified budget, while minimizing the set size.

Critical strengths

The paper’s core contribution links set-size minimization in risk-controlled generation to selective prediction and Chow’s optimal rejection rule, yielding a concise regime law. The ACF-LV method achieves notable APSS reduction under a unified protocol, where the majority of improvements come from the straightforward and sensible single-stage calibration design.

Critical weaknesses

The theoretical assumptions are relatively strong, and ablation results demonstrate that the overall improvement mainly stems from single-stage calibration. In addition, several descriptions in the manuscript are unclear, and the writing, mathematical notations and tables require careful revision and verification.

Questions for authors

Beyond the aforementioned weaknesses, I wonder why ACF-LTT increases APSS on ScienceQA with Qwen2.5-7B/14B in the reported results.

Reviewer_KdtnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce a regime law distinguishing abstention from re-ranking in risk-controlled LLM generation and based on this authors introduce ACF-LV, a single-stage conformal filtering framework that reduces prediction set size while maintaining risk control.

Critical strengths
  1. The paper introduces a useful perspective on when abstention or re-ranking is more appropriate for risk-controlled generation.
  2. This method reduces prediction set sizes across multiple datasets and LLMs while maintaining risk control.
Critical weaknesses
  1. In figure1 caption, I can see some typo, please fix that.
  2. Equation (7) uses calibration labels to select between entropy and verifier scores, after which the same calibration data are used to determine the threshold. It is unclear whether the stated guarantees account for this data-dependent score selection.
  3. An issue with Lemma 1 (Section 3.2, Appendix A.1). The stated lower bound on miscoverage does not generally hold under exchangeability alone when coverage scores contain ties and the theorem appears to require an additional assumption or a correction.
  4. Insufficient validation of the regime law (Section 5.4). The Bayes top-1 error is never directly estimated, and only 8 of 15 settings are included in the regime analysis. Classifying regimes using the observed entropy-based set size provides limited evidence for the proposed theoretical boundary.
Questions for authors
  1. It would be helpful for the authors to clarify whether the lower bound in Lemma 1 remains valid when calibration scores contain ties, and to provide the underlying assumptions or a formal proof covering this specific case.
  2. Further explanation is needed regarding how the theoretical guarantee accounts for selecting the verifier using calibration labels and subsequently calibrating the threshold on the same data, including details on whether the Learn-then-Test correction covers both score and threshold selection.
  3. Since the Bayes top-1 error is unobservable, the manuscript should detail how the experiments establish which theoretical regime each setting belongs to, along with the rationale for considering only 8 of the 15 settings testable.
Reviewer_t12rOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper considers efficiency of conformal predictions in LLM question answering. They propose to filter sampled generations so that probability the returned prediction set fails to contains the correct answer is below a specified target risk. using prior rejection rules and minimum set size results, the authors derive a derive regime law which abstains when top 1 error is below target risk and re rank candidates when above. Experiments on SAFER benchmark over 5 open weight LLM are conducted and show significant reduction in set size.

Critical strengths
  • The problem of filtering as selective prediction is clear and natural
  • The proposed method is lightweight and can run on cached candidates
  • They report large improvements on the SAFER benchmark and are transparent about limitations of the verifier /heuristics
Critical weaknesses
  • The method is limited to QA tasks and not free form generation
  • The paper appears rushed and lacks proofreading (broken references, undefined notation, draft artifacts, missing experimental details)
  • The theory seems to follow from previous work and unconnected to empirical results
  • Most of efficiency gain comes from single stage calibration and the experiments do not compare to many other single stage baselines
Questions for authors
  • Does the proposed method also work for long form generation tasks with paraphrases?
  • How does the method compare to existing LLM conformal methods for abstentions? How does it compare to simple baselines such as self consistency with conformal abstention? What is the abstention rate and set size for each method?
  • How is the single stage calibration different from previous work? What is the main technical novelty?
  • Do you test larger models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.