- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper presents a postprocessing mechanism to improve conformal prediction under long-tailed regimes. The authors argue that although existing methods provide class-conditional coverage guarantees, under severe long-tailed distributions (with some classes having as few as 10 examples), some classes may have very few calibration samples, resulting in high variance in the quantile estimates. As a remedy, the authors propose SARC-CP, which postprocesses per-class quantiles by combining them with the global quantile (computed over the entire calibration set) using James-Stein-style weighting. SARC-CP yields smaller average prediction set sizes for tail classes than RC3P across different per-class sample sizes.
Critical strengths
- The paper identifies an important and interesting problem. Current conformal prediction methods guarantee marginal coverage, and some can also provide class-conditional coverage under long-tailed distributions, but none discuss the issue of quantile variance under severe imbalance or propose a way to reduce it.
- The authors provide a theoretical analysis of the conditions under which shrinkage is helpful.
Critical weaknesses
- I have found real difficulties in going through the paper. The paper requires extensive improvements in its presentation. In particular, the authors present their proposed approach in the introduction without introducing the notation, even before the preliminaries section.
- The paper only provides empirical coverage results without proving that shrinkage preserves class-wise validity. In long-tailed settings, when a tail class has , Equation 7 gives , which lowers that class's score threshold. Moreover, RC3P's class-conditional guarantee does not automatically transfer to SARC-CP.
- The results on iNaturalist2018 clearly violate the exchangeability assumption. The authors pool the training and validation images, make a random 80/10/10 split, and at the same time use an official InceptionV3 checkpoint. The public InceptionV3 checkpoint was trained on iNaturalist training images. If the authors used it without retraining after the new split, some calibration and test images were used to train the classifier.
- In Proposition 1, the authors derive a condition for MSE reduction under the assumption that the class-specific quantile and the global quantile are asymptotically uncorrelated. This assumption is generally not true. The global quantile is computed from all calibration scores, including those for class used to estimate . They are therefore dependent by construction, and the covariance term in the MSE does not vanish.
Questions for authors
- You may want to proofread your results. Table 4 shows the exact same values across all metrics for ClasswiseCP and Interp-Q.
- I strongly recommend that the authors rewrite the paper. It contains many overcomplicated words and terms that make it really hard to understand the message. Moreover, the results section, as currently presented, is difficult to follow and does not clearly communicate the main findings.
- The paper only focuses on integrating the proposed method with RC3P. Technically, SARC-CP should work with any nonconformity scoring function. Could the authors report the method's performance with different scoring functions? For example, using HPS would make it easier to understand how SARC-CP improves over ClasswiseCP.
- What dataset was used to produce the results in the right panel of Figure 1? Is it CIFAR-100?
- The whole purpose is to study how a model trained on a long-tailed distribution behaves. However, throughout the experiments, the authors use models trained with loss objectives designed for class-imbalanced settings. How does the proposed method perform with a model trained using the standard cross-entropy loss?
- The authors should discuss the case where the covariance is nonzero (e.g., depending on the sign of the covariance term) or establish conditions under which the correlation vanishes.