acceptodds
Under review as a conference paper at ICLR 2027

Conformal Elicitation: Loss Minimization Controls Set Size Regret

14 views

Abstract

Conformal prediction turns a probabilistic classifier into a prediction set with a prescribed coverage level, but coverage alone does not determine how useful that set is. We view expected set size at fixed marginal coverage as a downstream decision objective and ask how well ordinary probability estimation, via loss minimization, serves this objective. For conformal prediction sets induced by probability conformity scores, we prove that excess expected size over the Bayes-optimal set is at most , where is the number of classes, is the target miscoverage, and Reg_Brier is excess Brier loss. The guarantee requires no margin or density assumption and has sharp dependence on and up to a universal constant. We also prove a more general result that bounds the gap to the minimum set size within a model class using the loss improvement achievable via a family of post-processings corresponding to . These results suggest that standard loss minimization using a proper loss (e.g. Brier loss or log loss) can simultaneously achieve the goal of minimizing conformal set size. We run experiments to compare this method with ConfTr (Stutz et al., 2022), a previous method specifically designed for set size minimization. These experiments show, somewhat surprisingly, that loss minimization consistently achieves smaller set sizes compared to ConfTr at miscoverage level . These results demonstrate the power of standard proper loss minimization in providing high-quality uncertainty quantification.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Conformal Prediction, Loss Minimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vsX1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how to minimize the expected size of prediction sets while maintaining a guaranteed coverage rate in conformal prediction. Theoretically, the authors establish an upper bound on the size of the prediction set in terms of the classifier’s Brier loss, demonstrating that minimizing standard proper scoring rules can effectively approximate an optimal prediction set. Experimental results further show minimizing these standard loss functions outperforms the ConfTr method.

Critical strengths
  1. The author clearly explains the relationship between set size in CP and loss in classification tasks. The motivation and writing are clear.

  2. This approach yields better results than ConfTr, which directly optimizes set size. The author provides an explanation for this, which is interesting.

  3. The authors designed clear synthetic tasks, which is easy to understand.

Critical weaknesses
  1. The real data experiments only report results at 95% and 99% coverage. It is recommended to supplement the common 90% coverage setting.

  2. Limitations and Conclusion were not discussed / included explicitly -- there are no "limitation section" and "Conclusion section". This belongs within the work.

Questions for authors
  1. Could you add real data experiments in 90% coverage setting?

  2. Conformal Prediction (CP) typically consists of three stages: training, calibration, and evaluation. Does changing the proportion of the calibration set affect the conclusions?

  3. As the target coverage level varies, do the theoretical guarantees still hold? How do the validity and practical utility of these results change?

  4. This idea is quite intuitive, but does it remain effective for regression tasks?

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the careful review! We are glad to know that you appreciated our motivation, presentation, and design of experiments. We answer your specific questions below.

Proportion of calibration set. Appendix H (Table 17 and Figure 3) compares 45,000 training / 5,000 calibration with 42,500 training / 7,500 calibration examples, increasing the calibration proportion from 10% to 15% under a fixed total budget. We evaluate CE and plain ConfTr across seven settings with ten paired seeds. Overall, changing the training/calibration allocation from 45,000/5,000 to 42,500/7,500 does not alter our overall observation that CE remains competitive with ConfTr (Appendix H, Table 17 and Figure 3). At 95% target coverage, CE yields smaller mean sets in all seven settings under both allocations, although some comparisons are allocation-sensitive at 99%. A complementary analysis varying calibration size with training fixed is also summarized in Appendix H.2 (Table 18).

Target coverage level. Thank you for this suggestion. We agree that 90% coverage is a practically relevant setting. We have added experiments at 90% nominal coverage (α=0.10\alpha=0.10) on CIFAR-20 and CIFAR-100, reporting prediction-set size and empirical coverage in Table 16 of Appendix G. On ResNet-50, CE produces smaller prediction sets than ConfTr on both datasets at this coverage. Moreover, among the nine training objectives in our expanded comparison, CE achieves the smallest mean set size on CIFAR-20 (1.3221) and the second-smallest on CIFAR-100 (2.2411). The strong performance at 90% coverage is qualitatively consistent with our theoretical bounds, whose 1/α1/\alpha dependence gives tighter control of set-size regret at larger miscoverage levels.

Regression tasks. Please kindly see our response to Reviewer DYho.

Limitations and conclusions. Thank you for the suggestion. We will add a section to discuss limitations and conclusions in our revised version. As you point out, one limitation of our current work is that it does not address regression tasks, which is an interesting question for future research. As our conclusion, we will reiterate our main message that standard loss minimization can be used as a simple method for minimizing conformal set size.

I thank the authors for their thoughtful responses. I have no more question, and I am happy to maintain my positive rating.

Reviewer_DYhoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper brings in a new perspective of loss minimization that leads to set size optimization for conformal prediction sets. Using the proper loss training, the paper establishes that set size regret can be analyzed, bounds derived through minimal assumptions, and interpretable. Extensions to misspecified settings using the local optimality conditions provide further practical explanations of how the proposed strategy could be helpful in such settings. Overall, I enjoyed reading the paper, it is well-written, problem is well-motivated, and does a good job of explaining technical details.

Critical strengths
  1. The paper is well written, problem is well described and motivated. technical details are explained well.

  2. The main contribution of the paper is to bring in a new perspective of achieving set size optimization via loss minimization itself during training, moving away from differential set size optimization proxies, is particularly appealing.

  3. Theoretical results (Theorem 1 and Theorem 2) form the basis of the paper. Their derivations are neat and easy to follow. Connecting regret bounds of set sizes to proper losses and then use them to train the models themselves is the important technical novelty of the paper. I appreciate authors explanation at appropriate places throughout the paper which helped me to understand the ideas much clearly.

  4. Real data experiments also discuss the failure modes (trade-off between model expressibility and very low miscoverage levels) identified by theory.

Critical weaknesses
  1. Empirical study has only one baseline (conformal training). But there are other approaches for set minimization target such as optimized score functions.
Questions for authors
  1. In real data experiments, is there a reason for not considering Brier loss minimized models for comparison?

  2. Is it possible to extend the approach to regression problem? What are the main challenges?

  3. Paper frames the problem of set optimization around proper loss minimization. Once we learn probability conformity scores, I didn’t understand why do we need calibration step (as done in C.3)? Is this calibration step applied to all the experiments?

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the thoughtful review! We are delighted to know that you enjoyed reading our paper and appreciated the paper presentation, the problem we study, and the technical exposition. We answer your specific questions below.

Other baselines. Please kindly see our response to Reviewer YjJ5. We added several other baselines. Among these, C-Adapter directly addresses your suggestion to consider optimized score functions. It learns a transformation of a pretrained classifier’s scores to reduce conformal prediction set size. Appendix G reports the full results.

Brier loss. Brier loss is also a natural choice, and our theoretical results explicitly cover both Brier loss and cross-entropy, as well as a broader class of proper losses. We chose cross-entropy for the real-data experiments because it is the standard classification objective in practice. This lets us examine whether the training procedure practitioners already commonly use can also produce efficient conformal prediction sets. We have now completed additional real-data comparisons using Brier-loss training at 95% nominal coverage (see Appendix G, Table 14). Across the eight dataset–architecture settings, CE yields smaller mean prediction-set sizes in six settings, while Brier yields smaller means in the remaining two.

Regression tasks. Yes, our theory and experiments focus on the multiclass classification task and they do not directly generalize to regression. We will make sure to mention this as a limitation of our work when we revise the paper. Our focus on classification is aligned with previous work on conformal set size minimization that also largely focuses on multiclass classification (e.g., ConfTr, DPSM, RWCE, MB-Fano/DPI and C-Adapter).

It is an interesting question for future research to systematically study conformal set size minimization in regression tasks. The continuous label space in regression tasks presents additional structure that should be accounted for during conformal prediction. For example, there is a natural notion of distance between labels in regression, so we can talk about whether the output label is "close" or "far" from the true label. Our current work does not take this label-space structure into account, and this may be a key challenge for extending our work to regression tasks. Another potential challenge is that, in regression, the conformal sets are usually required to be contiguous, but the conformal sets defined based on an arbitrary predictor may not be contiguous.

That said, it seems possible to generalize our approach to regression tasks with some modifications. First, we need to translate regression models into predictors that produce probability distributions over the continuous space of labels. For example, one idea is to interpret the output of a regression model as the mean of a Gaussian distribution. For the contiguous issue, it is also interesting to find the best way to form and restrict the predictors so that the conformal sets are contiguous while the predictors remain expressive enough to minimize proper losses. We will mention this idea in the revised version as a potential future direction.

Calibration step. After we train a predictor using proper loss minimization, we need the calibration step to produce conformal sets. In the calibration step, we use a separate calibration dataset to calculate the threshold τ\tau and use it to define the conformal sets in the following way. Given a new input xx, we include each label yy in the conformal set if and only if the predicted probability of yy is at least τ\tau. With the standard split-conformal quantile correction, this held-out calibration step provides the finite-sample marginal coverage guarantee under exchangeability. Proper-loss minimization alone does not provide the same guarantee.

The same calibration step is required for all methods considered in our experiments, including ConfTr. These methods differ in how they train the predictor or construct the conformal scores, but the resulting scores still need to be calibrated to produce prediction sets at the target coverage level. We therefore calibrate each fitted model separately using the same held-out calibration examples before comparing their prediction-set sizes.

I thank authors for the response. I have no further concerns and will maintain my already positive assessment of the paper.

Reviewer_YjJ5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper bounds excess prediction-set size by Brier/log-loss regret for randomized probability-threshold sets, proves sharp worst-case dependence on class count and miscoverage, and extends the analysis through post-processing optimality. Synthetic and CIFAR experiments compare proper-loss training with ConfTr; CE generally wins at 95% coverage, but higher-coverage results include reversals.

Critical strengths
  1. Theorem 1 handles predictor-dependent thresholds and yields

SSRα(q)≤K3/2αRegBrier(q).SSR_{\alpha}(q)\le\frac{K^{3/2}}{\alpha}\sqrt{\mathrm{Reg}_{\mathrm{Brier}}(q)}.

The matching K,αK,\alpha dependence makes this a substantive refinement of oracle optimality.

  1. Appendix D shows that a population-global CE minimizer can produce larger sets than another predictor in the same class. This precisely identifies why in-class loss optimality cannot replace Bayes optimality.
Critical weaknesses
  1. Theorem 1 requires population Bayes regret; Section 4 requires optimality against benchmark-dependent post-processings that may leave the trained class. No learning guarantee connects the evaluated training procedure to this stronger requirement; misspecified experiments measure neither post-processing regret nor optimal class-relative set size. Appendix D shows why this gap matters. Separately, Eq. (8) includes all ties, whereas the proof randomizes them: uniform q=p⋆q=p^\star gives deterministic size KK versus randomized optimum K(1−α)K(1-\alpha) despite zero regret. The standard split-conformal extension assumes continuity (Appendix B.4). Novelty also requires a precise boundary: threshold-oracle analysis [1,2], CE-based efficiency motivation [3], and post-processing calibration/omniprediction [4–6] already exist. Omitted works [3,7] require explicit comparison; these precedents do not by themselves make the new bound redundant.

  2. ConfTr dominates the baseline selection; cited DPSM [8] and RWCE [9] are unevaluated, while MB-Fano/DPI [3] and capacity-matched C-Adapter [10] are also relevant. More decisively, Section 5's claim that direct size minimization wins only in restricted classes conflicts with Table 10: on ResNet-50 at 99% coverage, ConfTr+LclassL_{\mathrm{class}} gives size 10.4568 versus CE's 10.9915, with paired difference −0.535±0.215-0.535\pm0.215 and comparable realized coverage (98.935% versus 98.946%). Appendix F.6 acknowledges this reversal. A regret upper bound supplies neither pairwise size ordering nor a capacity-based explanation of the winner. The results establish an advantage over particular ConfTr configurations; they do not establish general superiority of ordinary loss training over specialized objectives.

[1] Sadinle, Lei, and Wasserman. Least Ambiguous Set-Valued Classifiers with Bounded Error Levels. JASA, 2019.

[2] Denis and Hebiri. Confidence Sets with Expected Sizes for Multiclass Classification. JMLR, 2017.

[3] Correia, Massoli, Louizos, and Behboodi. An Information Theoretic Perspective on Conformal Prediction. NeurIPS, 2024.

[4] Hu, Livni Navon, Reingold, and Yang. Omnipredictors for Constrained Optimization. ICML, 2023.

[5] Blasiok, Gopalan, Hu, and Nakkiran. When Does Optimizing a Proper Loss Yield Calibration? NeurIPS, 2023.

[6] Blasiok, Gopalan, Hu, Kalai, and Nakkiran. Loss Minimization Yields Multicalibration for Large Neural Networks. ITCS, 2024.

[7] Casacuberta and Kanade. Selective Omniprediction and Fair Abstention. NeurIPS, 2025.

[8] Shi et al. Direct Prediction Set Minimization via Bilevel Conformal Classifier Training. ICML, 2025.

[9] Jia et al. Cost-Sensitive Conformal Training with Provably Controllable Learning Bounds. AAAI, 2026.

[10] Liu et al. C-Adapter: Adapting Deep Classifiers for Efficient Conformal Prediction Sets. ECAI 2025 proceedings, DOI 10.3233/FAIA251208; publisher online publication in 2026.

Questions for authors
  1. What result connects the actual training procedure to Section 4's comparator-dependent post-processing optimality?

  2. Is randomized inclusion intended throughout? What efficiency claim survives for deterministic scores with atoms?

  3. Which assumptions or conclusions distinguish Theorem 2 from the existing mechanisms in [4–7]?

  4. Which BatchNorm moments are used/updated, and how were real-data batch sizes, learning rates, and smoothing parameters selected?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1/2)

Thank you for your careful reading and detailed comments! We are glad to know that you appreciated our clear presentation and the matching dependency in our upper and lower bounds. Below are our answers to your specific questions. Due to the character limit, we divide our response into two parts.

Other baselines. Thank you for pointing out the limited baseline coverage. We have added DPSM, RWCE (unmasked and released-mask objectives), MB-Fano, DPI, and C-Adapter across CIFAR-20/100 and 99%, 95%, and 90% nominal coverage. The expanded results highlight ordinary proper-loss training as a strong approach to obtaining efficient conformal sets, with particularly strong performance on expressive representations at 90% nominal coverage (consistent with our theory, that set-size regret bounds scale as 1/α1/\alpha). On pretrained ResNet-50, CE achieves the smallest mean prediction set size on CIFAR-20 (1.3221) and the second-smallest on CIFAR-100 (2.2411) among all nine evaluated head-training objectives.

We also evaluate restricted model classes, including affine predictors, to provide a broader comparison and explore the practical scope of this competitiveness. Such classes may have limited ability to approximate conditional class probabilities, so minimizing a proper loss within them need not satisfy the small Bayes-relative regret premise of our guarantee. We interpret these experiments as evidence about performance across model classes, rather than as verification of the theorem’s premises. Full comparison results are shown in Appendix G. Our central empirical message is that ordinary proper-loss training can produce highly competitive conformal sets without a specialized set-size training objective.

We have also revised Section 5 to qualify our earlier statement that specialized objectives improve efficiency only in restricted model classes. The revised wording presents the observed differences across model classes as empirical patterns within the evaluated settings.

Connection between actual training and post-processing optimality. This connection has been well established in previous works that study the relationship between post-processing optimality and calibration (Błasiok et al. 2023; 2024). These works provide both intuitive and theoretical explanations for why actual training of deep neural networks using loss minimization often leads to post-processing optimality.

Intuitively, neural networks in practice are often quite deep, so it is natural to believe that adding a small number of layers to a well-trained neural network would not lead to significant loss improvement. By the universal approximation properties of neural networks, we can expect that these additional layers can express the post-processing functions we use in Section 4.2, giving the post-processing optimality we need. Theoretically, the work of Błasiok et al. (2024) shows that, given a family of post-processing functions that can be expressed as kk-layer neural networks, for all but O(k/ε)O(k/\varepsilon) choices of dd, optimizing a proper loss among the restricted family of dd-layer neural networks within sub-optimality error ε/2\varepsilon/2 provably leads to post-processing optimality within error ε\varepsilon.

We will make sure to add this discussion when we revise the paper. Please don't hesitate to let us know if you have follow-up questions.

Randomized inclusion. Yes, our theoretical analysis uses randomized tie breaking when forming conformal sets based on the threshold τ\tau. Randomization is necessary to guarantee exact 1−α1-\alpha coverage in expectation when the data distribution contains atoms, in order to ensure a fair comparison across methods. At the beginning of Page 19, we show that a deterministic tie-breaking rule also works when the prediction distributions have continuous CDFs, but these distributions do not have atoms.

We will make it clear in the revised version that we focus on the randomized tie breaking rule throughout the paper and that we provide additional results for deterministic tie breaking only when the prediction distributions have continuous CDFs.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2/2)

We continue our response below.

Difference between Theorem 2 and existing mechanisms. Theorem 2 is indeed inspired and closely related to the literature on omniprediction and calibration. In particular, the indistinguishability notions eq(hq)≤ε1e_q(h_q)\le \varepsilon_1 and −eq(hf)≤ε2-e_q(h_f) \le \varepsilon_2 are essentially the same as weight-restricted calibration and multiaccuracy in the literature, and these notions are known as sufficient conditions for omniprediction. We will make sure to discuss relevant prior works clearly in the revised paper.

To the best of our knowledge, our Theorem 2 is the first result that applies the existing framework of indistinguishability notions to solve conformal set size minimization.

Compared to the loss functions used in the omniprediction literature, conformal set size is a more challenging objective to analyze because it depends on the threshold τ\tau that cannot be locally determined using a single data point. Instead, computing the threshold τ\tau requires global information of both the data distribution and the predictor, and we need a separate calibration dataset to obtain this global information. This global dependency requires us to develop new techniques to identify appropriate indistinguishability notions and prove that they are sufficient for conformal set size minimization.

Experiment details. In our original real-data experiments, we used ConfTr’s published best CIFAR-100 configuration, selected by the original authors through grid search: batch size 100, learning rate 0.005, soft-threshold temperature 1, and sorting dispersion 0.1. We transferred this configuration to CIFAR-20 without additional tuning. In our original experiments, we transferred ConfTr’s CIFAR-100 configuration to CIFAR-20 without additional tuning. In the new comparisons in Appendix G, we have already performed dataset- and architecture-specific hyperparameter selection for both ConfTr variants, alongside the other learned objectives.

During pretraining, BatchNorm uses minibatch moments and updates running averages (decay 0.9). During MLP/ResNet head training, it still uses minibatch moments, but running averages and BatchNorm parameters remain frozen. ConfTr computes these moments before splitting the minibatch. Final calibration and testing use the frozen pretraining averages without updates.

Thank you for the expanded comparisons and clarifications. The randomization explanation and revised empirical scope address my corresponding concerns. I will maintain my positive score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

α-Independence: A Practical Advantage of Our Framework

We would like to highlight an additional practical benefit of our framework that was not sufficiently emphasized in the main text. Our proper-loss training objective is independent of α\alpha, the target miscoverage level. However, among the specialized head-training baselines we added in our study, all of them use an α\alpha-dependent training objective, including ConfTr, DPSM, MB-Fano, DPI, and masked RWCE. This provides a clear practical advantage of our method: a single model can be trained before the desired coverage level is specified and subsequently used at multiple coverage levels by changing only the calibration threshold. The same model can thus serve applications with different coverage requirements and accommodate changes in those requirements after deployment. This separation between model training and coverage specification makes proper-loss training particularly attractive in practice.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.