- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates how to minimize the expected size of prediction sets while maintaining a guaranteed coverage rate in conformal prediction. Theoretically, the authors establish an upper bound on the size of the prediction set in terms of the classifier’s Brier loss, demonstrating that minimizing standard proper scoring rules can effectively approximate an optimal prediction set. Experimental results further show minimizing these standard loss functions outperforms the ConfTr method.
Critical strengths
-
The author clearly explains the relationship between set size in CP and loss in classification tasks. The motivation and writing are clear.
-
This approach yields better results than ConfTr, which directly optimizes set size. The author provides an explanation for this, which is interesting.
-
The authors designed clear synthetic tasks, which is easy to understand.
Critical weaknesses
-
The real data experiments only report results at 95% and 99% coverage. It is recommended to supplement the common 90% coverage setting.
-
Limitations and Conclusion were not discussed / included explicitly -- there are no "limitation section" and "Conclusion section". This belongs within the work.
Questions for authors
-
Could you add real data experiments in 90% coverage setting?
-
Conformal Prediction (CP) typically consists of three stages: training, calibration, and evaluation. Does changing the proportion of the calibration set affect the conclusions?
-
As the target coverage level varies, do the theoretical guarantees still hold? How do the validity and practical utility of these results change?
-
This idea is quite intuitive, but does it remain effective for regression tasks?
2 comments
I thank the authors for their thoughtful responses. I have no more question, and I am happy to maintain my positive rating.
Rebuttal by Authors
Thank you for the careful review! We are glad to know that you appreciated our motivation, presentation, and design of experiments. We answer your specific questions below.
Proportion of calibration set. Appendix H (Table 17 and Figure 3) compares 45,000 training / 5,000 calibration with 42,500 training / 7,500 calibration examples, increasing the calibration proportion from 10% to 15% under a fixed total budget. We evaluate CE and plain ConfTr across seven settings with ten paired seeds. Overall, changing the training/calibration allocation from 45,000/5,000 to 42,500/7,500 does not alter our overall observation that CE remains competitive with ConfTr (Appendix H, Table 17 and Figure 3). At 95% target coverage, CE yields smaller mean sets in all seven settings under both allocations, although some comparisons are allocation-sensitive at 99%. A complementary analysis varying calibration size with training fixed is also summarized in Appendix H.2 (Table 18).
Target coverage level. Thank you for this suggestion. We agree that 90% coverage is a practically relevant setting. We have added experiments at 90% nominal coverage (α=0.10) on CIFAR-20 and CIFAR-100, reporting prediction-set size and empirical coverage in Table 16 of Appendix G. On ResNet-50, CE produces smaller prediction sets than ConfTr on both datasets at this coverage. Moreover, among the nine training objectives in our expanded comparison, CE achieves the smallest mean set size on CIFAR-20 (1.3221) and the second-smallest on CIFAR-100 (2.2411). The strong performance at 90% coverage is qualitatively consistent with our theoretical bounds, whose 1/α dependence gives tighter control of set-size regret at larger miscoverage levels.
Regression tasks. Please kindly see our response to Reviewer DYho.
Limitations and conclusions. Thank you for the suggestion. We will add a section to discuss limitations and conclusions in our revised version. As you point out, one limitation of our current work is that it does not address regression tasks, which is an interesting question for future research. As our conclusion, we will reiterate our main message that standard loss minimization can be used as a simple method for minimizing conformal set size.