- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the effect of reusing calibration-set labels on the validity of the prediction sets generated by conformal prediction. The authors show that reusing the count of these labels might harm the validity of the conformal guarantee and provide the criterion under which validity is preserved despite reusing the calibration labels. These findings are demonstrated empirically on real datasets.
Critical strengths
- Theorem 1 is insightful - it provides a sufficient condition for conformal prediction to remain valid despite label count reuse.
- Theorem 2 is also interesting - it demonstrates that the same valid weighting function can either improve or worsen the informativeness of the prediction set size compared to a uniform weighting.
Critical weaknesses
- The paper is very hard to read, and its presentation is unclear. For example, Conf-OT is not fully described; the paper uses notations without defining them (line 043), and the Independent and Augmented prior methods are not sufficiently explained.
- The practical contribution of the valid count-dependent weighting functions is unclear. The experiments do not show that these procedures improve efficiency compared to uniform weighting. The method that uses pre-trained fixed weights constructs smaller prediction sets than the count-reusing methods.
Questions for authors
- The experiments show that the 55 candidates do not achieve improved performance compared to the uniform weighting function. Can the authors provide conditions under which live count reuse improves the efficiency compared to independently fitted weights or uniform weights?
- Since class frequency can be estimated from an independent sample without invalidating the theoretical guarantee of conformal prediction, what is the advantage of using the calibration counts? Can the authors provide a setting in which this approach improves efficiency compared to weights estimated from an independent set?