- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work expands upon the information theoretic perspectives of the set size in conformal prediction through the notions of generalized entropy and information. More specifically, the authors use a decision-theoretic generalization of the entropy that replaces the log probability of the target variable with a loss function for that variable and repurpose it to conformal prediction by using a family of set-valued prediction losses, indexed by . The generalized / conformal entropy is then the expected loss when one selects the optimal set predictor and the generalized / conformal mutual information follows by taking the difference of two conformal entropies. Interestingly, one can also recover the Shannon mutual information by integrating the conformal mutual information over . The authors then employ this theory in two ways. Firstly, they connect set-size reduction in conformal prediction to the classical concept of information gain, showing that the set-size reduction can be sandwiched by conformal mutual information terms, calibration and model error terms. Secondly, they show a data processing inequality property for conformal set-size reduction where, when there is some post processing of the features, the set size reduction with the new features is upper bounded by the set-size reduction of the original features and some additional terms. Finally, the theory is extended from simple probability conformal scores to APS scores. Experimentally, the authors evaluate the validity of their theoretical results on both real and synthetic experiments as well as probe the set-size reduction and Shannon mutual information in a feature selection experiment.
Critical strengths
The main critical strength of the paper is the more direct connection of decision-theoretic quantities to conformal prediction. This can potentially open interesting directions for future works and expansion of domains where conformal prediction can be useful. Both conformal entropy and conformal mutual information are quite intuitive and the fact that a simple integration recovers mutual information is quite interesting.
Critical weaknesses
In my opinion, a critical weakness of this work is the quite limited number of practical experiments; essentially there is only one main experiment on feature selection and the rest are just empirical verification of the theory. This makes it hard to connect the overall theory presented in the paper to something that is practically actionable. For example, the authors showed in Figure 2 an interesting case where Shannon MI cannot distinguish features that reduce the set-size but conformal MI, depending on the value of can. I would have expected more of such applications of the theory in experiments to demonstrate why this new direction is more promising and relevant to pursue than other ones.
Questions for authors
Overall, while I appreciate and find very interesting the decision-theoretic extensions of the information theoretic views of conformal prediction, I am borderline negative for this work, mostly due to the quite limited number of practical experiments that do not make the theory very actionable. If those are sufficiently addressed I would be happy to raise my score. As for the questions
- The conformal information divergence as the authors point out is a pseudo-divergence and thus it can be zero even if the model distribution is not the same as the data distribution . Can the authors expand on when this can and cannot be problematic in practice?
- For the data processing inequality for conformal set-size reduction, is there a reason that the authors consider the quite specific Markov chain where only (and not also ) is modified? Why do you even need to introduce in the first place? By looking at the proof, you only work with and anyway, which hide in their construction.
- Can the authors discuss and motivate their theory with more practical examples where it can have the edge over existing methods?
4 comments
Conformal training results. We report the mean ± 2 standard errors over 10 training seeds, each averaged over 10 calibration/test splits. The best model is bolded, and models that significantly improve on the CE baseline are underlined (one-sided paired t-test across seeds, ).
| Model | Prob. size | APS size | Accuracy (%) | Coverage (Prob. / APS) |
|---|---|---|---|---|
| MNIST | ||||
| CE baseline | ||||
| ConfTr | ||||
| MB Fano | ||||
| DPI | ||||
| Conformal entropy (Ours) | ||||
| Fashion-MNIST | ||||
| CE baseline | ||||
| ConfTr | ||||
| MB Fano | ||||
| DPI | ||||
| Conformal entropy (Ours) | ||||
| EMNIST | ||||
| CE baseline | ||||
| ConfTr | ||||
| MB Fano | ||||
| DPI | ||||
| Conformal entropy (Ours) |
Feature selection reversal results. We report the fraction of feature pairs whose ranking by set-size reduction reverses between coverage levels from 50% to 99%, and the Spearman correlation between the feature rankings at 99% and 50% coverage. Reversals are counted when significant across 50 calibration/test splits.
| Dataset | Reversed feature pairs | Rank corr. @ 99% vs. 50% |
|---|---|---|
| Wine | % | |
| HAR | % | |
| Letter | % | |
| Fashion-MNIST | % | |
| CIFAR-10 | % | |
| CIFAR-100 | % |
[1] Learning Optimal Conformal Classifiers. Stutz et al. ICLR, 2022.
[2] An Information Theoretic Perspective on Conformal Prediction. Correia et al. NeurIPS, 2024.
I would like to thank the authors for their response and new results. These address my concerns and I am happy to increase my score.
We thank the reviewer for their questions and positive feedback, and for taking the time to consider our response and additional results. We will incorporate the new analysis in the revised manuscript where appropriate.
We thank the reviewer for their detailed comments and for noting that our work can open interesting research directions related to conformal prediction. Below, we address the request for additional practical experiments and answer the remaining questions.
Question 1. The conformal information divergence is a pseudo-divergence because it only depends on whether q places each label on the same side of the probability threshold λ as p. Our set-size results operate at a single calibrated threshold, so this poses no issue. However, it may be problematic in applications that require multiple coverage levels. In such cases, one can instead use a mixture of Dλ, which is a (strict) divergence whenever the mixing distribution has full support over (0,1].
Question 2. Our main object of interest is the set-size reduction Δα, which measures the information gained beyond base features Z. The DPI compares this gain when acquiring X versus X~ under the same (fixed) base features. As the reviewer points out, Z only enters the proof through X(i) and X~(i), since ∣Cα(Z)∣ cancels in the difference Δα−Δ~α. Nonetheless, we allow possible Z features for conceptual purposes and to state the result in full generality.
We also note that the Markov chain Y→(X,Z)→(X~,Z) is the standard condition for the conditional DPI, since equivalently, Y⊥X~∣(X,Z). This allows X~ to be any noisy function of X and Z. We will clarify this after Theorem 3.
Question 3 / Weakness. We agree that additional experiments would help demonstrate how our perspective can guide the design of new methods. As an example, we revisit conformal training through the lens of conformal entropy. Then, following the comment on Figure 2, we use our feature selection experiment to explicitly show that the ranking of features by set-size reduction frequently changes with the coverage level on real data.
Conformal training. Our conformal loss admits a corresponding conformal cross-entropy,
Hλ(p,q)=Ep[ℓλ(Y,Γλ∗(X;q))]=Hλ(Y∣X)+Dλ(p∥q).
As in Theorem 1, cross-entropy training uniformly weights Hλ(p,q) over λ∈(0,1]. We instead propose to minimize Eλ∼stopgrad(Wθ)[Hλ(p,qθ)], where Wθ is an estimate of the distribution of the calibrated threshold using held-out data. We note that our objective, which we arrive at from first principles, has a similar form to [1], with some minor differences. Moreover, the objective is smooth in the predicted probabilities, so unlike existing conformal training methods, it does not require soft set assignments or differentiable sorting.
We compare against a cross-entropy baseline, ConfTr [1], and the MB Fano and DPI training objectives in [2], both of which are derived from information-theoretic bounds for conformal prediction. Following prior conformal training work, we use the MNIST, Fashion-MNIST, and EMNIST datasets. At 90% coverage, our objective consistently matches or improves on cross-entropy in set size using probability and APS scores without degrading accuracy. The relative performance of other methods varies across datasets and conformal scores (see the results in the following comment).
Feature selection. To demonstrate that the behavior observed in Figure 2 also arises on real data, we extend our feature selection experiment to coverage levels ranging from 50% to 99%. For each dataset, a substantial fraction of feature pairs reverse their relative ranking in set-size reduction as coverage varies. Shannon mutual information, which induces a single coverage-independent ranking, cannot capture this behavior.
Broader remarks. While our conformal training objective has a slight overall edge over existing baselines, we emphasize that a central contribution of our work is the information-theoretic perspective itself. This allowed us to arrive at an objective similar to existing conformal training methods from first-principles reasoning. Moreover, set size is commonly used to measure uncertainty and information gain, and our work provides a principled explanation for this practice too, which we believe is valuable in its own right.
In terms of algorithm design, our feature selection results demonstrate that Shannon mutual information misses coverage-dependent behavior. This suggests that information-theoretic objectives (e.g., those in active learning and feature acquisition) admit coverage-targeted counterparts. Our conformal training objective is one concrete example, and we believe these connections offer several promising directions for future work. We will include these results and the accompanying discussion in the appendix.