acceptodds
Under review as a conference paper at ICLR 2027

Label-Efficient Group Risk Control for Selective Prediction

Abstract

Certifying selective-prediction risk separately for each group requires labeled evidence from every group. Under a fixed label budget, proportional sampling can leave a small group with an overly conservative certified threshold, forcing abstention even when more of its predictions are safe to accept. We call this coverage loss certification disparity and cast it as a sequential label-allocation problem. Our method, , combines prediction-powered loss corrections with simultaneous betting confidence sequences, allowing labels to follow groups that still need evidence. We prove simultaneous, anytime-valid group-wise risk control under predictable allocation, data-dependent stopping and bounded importance sampling. For a fixed-configuration empirical-Bernstein variant, the certification cost separates into residual-variance and bounded-range terms, and passive and active lower bounds identify the corresponding variance orders. For the EB variant, round-robin stopping certifies targets within the sum of group requirements with high probability under the stated conditions; fixed allocation is bottlenecked by the largest requirement per share. On a controlled four-group benchmark, round-robin allocation raises worst-group coverage from to over proportional allocation while also increasing population coverage, and auxiliary information adds percentage points at matched egalitarian allocation. With stopping, round-robin allocation reduces the median labels needed to reach common coverage targets by relative to fixed uniform allocation.

Primary area:
learning theory
Keywords:
Selective Prediction, Group-Conditional Risk Control, Prediction-Powered Inference, Adaptive Label Allocation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KLKBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the problem of selective predictions across groups and the derivation of uncertainty acceptance thresholds for a fixed predictor to control its risk. The main method, SPARC, uses both auxiliary loss predictions with labeled corrections and anytime-valid confidence bounds to support adaptive sampling and stopping. The authors provide both theoretical results on the certification costs and synthetic experiments to show that reallocating labels can improve worst-group coverage and reduce the extra labels needed for a coverage target.

Critical strengths
  1. The problem is well-motivated and highlights the difference between disparities caused by poor predictions and those caused by insufficient calibration. They make clear why label allocation is a concrete way to improve group coverage.

  2. The theoretical results are interesting as they show simultaneous, anytime valid risk control and relate certification costs to residual variance, acceptance rate and safety margin, with the corresponding lower bounds identifying variance limitations.

Critical weaknesses
  1. The assumption of knowing exactly the importance weights w_t and a known upper bound W is not well justified. In many applications where the group distribution is unknown, these ratios are hard to obtain. Moreover, largew weights can weaken significantly label efficiency.

  2. Can you provide practical sampling protocols where these weigths are exactly computable?

Questions for authors

Please see the weaknesses section.

Reviewer_G36tOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to divide a limited labeling budget among groups to check whether each group's prediction risk is below a target level. The proposed algorithm, SPARC, closely follows the method of Xu et al. (2024). The main additions are group-wise risk certification and an analysis of how many labels are needed under different allocation policies. The method is evaluated through synthetic experiments. In my view, these additions offer limited methodological novelty.

Critical strengths

Clear experimental comparisons. The comparisons in Table 1 of the experimental section are clear and help me understand which factors affect the experimental results.

Critical weaknesses
  1. The paper's presentation is poor. Some notation is introduced without an immediate explanation (e.g., [K][K] and τ^g\hat\tau_g), while many other symbols are introduced with unclear purpose and seem either unused or unnecessary as separate notation. For example, aia_i and bib_i in AppendixB.1 merely rename the existing Zτ,iZ_{\tau,i} and Z~τ,i\widetilde Z_{\tau,i}. Proposition1 also contains the redundant statement let $P_g=P_g$''. The additional shorthand $\widetilde L=\widetilde\ell(X)$ also seems unnecessary; $\widetilde\ell(X)$ could be used directly. In addition, several terms and phrases, such as full abstention'', shared-threshold variant'', imputer'', power tuning'', and certification price'', are unclear to me. ICLR readers are not necessarily familiar with this specific subfield, and I often have to guess what these terms mean. The statements of Theorems 2 and 4 are too long, making it difficult to distinguish the assumptions from the conclusions. Theorem 2 also includes explanatory discussion, such as ``EpoolE_{\mathrm{pool}} alone does not ensure the slack condition,'' within the theorem statement.

  2. Limited novelty. The core methodology already appears in Xu et al. (2024). From a theoretical perspective, much of the additional analysis appears to be a straightforward adaptation of their framework to the group-wise setting. Despite the heavy notation, several key arguments rely on standard proof techniques (e.g., supermartingale constructions, Ville's inequality, and empirical-Bernstein concentration bounds), and I find the additional methodological and theoretical insight relatively limited.

Questions for authors

Table 1 suggests that the choice of certificate has relatively little impact, while the allocation strategy is the primary driver of the observed gains. However, egalitarian and round-robin allocation are well-established strategies. Thus, based on the numerical results, SPARC appears to be only an incremental extension of existing methods. Could the authors address this concern?

Reviewer_Z8BaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper considers a concept terms as "certification disparity" - i.e., when risk must be certified per group on a fixed label budget, small groups get few labels and abstain more than the model needs. The paper proposes a method called SPARC which intends to correct an auxiliary loss estimate with real labels and uses anytime-valid betting confidence bounds, so labels can go to under-certified groups while the guarantee still holds for every group at once.

Critical strengths

I think the problem is interesting and the proposed solution makes sense.

Critical weaknesses

The main weakness that I noticed is insufficient evidence on the framework's success (experimentally): The paper relies on one synthetic benchmark whose auxiliary signal is generated directly from the true error (AUC about 0.99999), with no real data or learned judges, and even that near-perfect signal adds only about 2.5 points. Moreover, it seems to me that under the same proportional allocation, the fixed-sample LTT-CP baseline beats SPARC on worst-group coverage (12.6% vs 1.6%), so every gain comes from reallocating labels, not from the new confidence bounds. Please correct me if I am wrong here.


Update: Thanks for the responses. Please include the new studies in the revised version of the paper.

Questions for authors

See the weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.