- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the problem of selective predictions across groups and the derivation of uncertainty acceptance thresholds for a fixed predictor to control its risk. The main method, SPARC, uses both auxiliary loss predictions with labeled corrections and anytime-valid confidence bounds to support adaptive sampling and stopping. The authors provide both theoretical results on the certification costs and synthetic experiments to show that reallocating labels can improve worst-group coverage and reduce the extra labels needed for a coverage target.
Critical strengths
-
The problem is well-motivated and highlights the difference between disparities caused by poor predictions and those caused by insufficient calibration. They make clear why label allocation is a concrete way to improve group coverage.
-
The theoretical results are interesting as they show simultaneous, anytime valid risk control and relate certification costs to residual variance, acceptance rate and safety margin, with the corresponding lower bounds identifying variance limitations.
Critical weaknesses
-
The assumption of knowing exactly the importance weights w_t and a known upper bound W is not well justified. In many applications where the group distribution is unknown, these ratios are hard to obtain. Moreover, largew weights can weaken significantly label efficiency.
-
Can you provide practical sampling protocols where these weigths are exactly computable?
Questions for authors
Please see the weaknesses section.