acceptodds
Withdrawn from ICLR 2027

Variance-Reduced Conformal Prediction via Shrinkage for Long-Tailed Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Class-conditional conformal prediction is difficult when some classes have few calibration examples. In this regime, class-specific quantile estimates can vary substantially across calibration splits, which leads to large prediction sets. We propose **SARC-CP** (Shrinkage-Adjusted Rank-Calibrated Conformal Prediction). SARC-CP shrinks each class-specific quantile toward a global quantile using a sample-size-dependent weight, . Classes with fewer calibration examples receive more shrinkage, while classes with more calibration data remain closer to their class-specific estimates. We derive a bias-variance condition under which the shrunk estimator has lower mean-squared error than the plug-in quantile. We then combine this estimator with rank-calibrated conformal prediction (RC3P), adding one hyperparameter and no model retraining. Across CIFAR-10/100-LT, ImageNet-LT, and iNaturalist-2018, SARC-CP reduces average prediction set size at comparable class-coverage levels. Relative to RC3P, the reduction reaches 19% on CIFAR-100-LT, 55-65% on ImageNet-LT, and about 80% on iNaturalist-2018. The largest gains occur in the large-class, low-calibration regimes. Code: https://anonymous.4open.science/r/sarc-cp-shrinkage-0CE7/README.md

Primary area:
interpretability and explainable AI
Keywords:
Conformal Prediction, Shrinkage, Long-tailed

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TiMFOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a postprocessing mechanism to improve conformal prediction under long-tailed regimes. The authors argue that although existing methods provide class-conditional coverage guarantees, under severe long-tailed distributions (with some classes having as few as 10 examples), some classes may have very few calibration samples, resulting in high variance in the quantile estimates. As a remedy, the authors propose SARC-CP, which postprocesses per-class quantiles by combining them with the global quantile (computed over the entire calibration set) using James-Stein-style weighting. SARC-CP yields smaller average prediction set sizes for tail classes than RC3P across different per-class sample sizes.

Critical strengths
  1. The paper identifies an important and interesting problem. Current conformal prediction methods guarantee marginal coverage, and some can also provide class-conditional coverage under long-tailed distributions, but none discuss the issue of quantile variance under severe imbalance or propose a way to reduce it.
  2. The authors provide a theoretical analysis of the conditions under which shrinkage is helpful.
Critical weaknesses
  1. I have found real difficulties in going through the paper. The paper requires extensive improvements in its presentation. In particular, the authors present their proposed approach in the introduction without introducing the notation, even before the preliminaries section.
  2. The paper only provides empirical coverage results without proving that shrinkage preserves class-wise validity. In long-tailed settings, when a tail class has qglobal<qyq_{\text{global}} < q_y, Equation 7 gives qy∗<qyq_y^* < q_y, which lowers that class's score threshold. Moreover, RC3P's class-conditional guarantee does not automatically transfer to SARC-CP.
  3. The results on iNaturalist2018 clearly violate the exchangeability assumption. The authors pool the training and validation images, make a random 80/10/10 split, and at the same time use an official InceptionV3 checkpoint. The public InceptionV3 checkpoint was trained on iNaturalist training images. If the authors used it without retraining after the new split, some calibration and test images were used to train the classifier.
  4. In Proposition 1, the authors derive a condition for MSE reduction under the assumption that the class-specific quantile q^y\hat{q}_y and the global quantile q^g\hat{q}_g are asymptotically uncorrelated. This assumption is generally not true. The global quantile is computed from all calibration scores, including those for class yy used to estimate q^y\hat{q}_y. They are therefore dependent by construction, and the covariance term in the MSE does not vanish.
Questions for authors
  1. You may want to proofread your results. Table 4 shows the exact same values across all metrics for ClasswiseCP and Interp-Q.
  2. I strongly recommend that the authors rewrite the paper. It contains many overcomplicated words and terms that make it really hard to understand the message. Moreover, the results section, as currently presented, is difficult to follow and does not clearly communicate the main findings.
  3. The paper only focuses on integrating the proposed method with RC3P. Technically, SARC-CP should work with any nonconformity scoring function. Could the authors report the method's performance with different scoring functions? For example, using HPS would make it easier to understand how SARC-CP improves over ClasswiseCP.
  4. What dataset was used to produce the results in the right panel of Figure 1? Is it CIFAR-100?
  5. The whole purpose is to study how a model trained on a long-tailed distribution behaves. However, throughout the experiments, the authors use models trained with loss objectives designed for class-imbalanced settings. How does the proposed method perform with a model trained using the standard cross-entropy loss?
  6. The authors should discuss the case where the covariance is nonzero (e.g., depending on the sign of the covariance term) or establish conditions under which the correlation vanishes.
Reviewer_MZAQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SARC-CP, which shrinks class-specific conformal thresholds toward a global quantile using calibration-count-dependent weights and combines the resulting thresholds with RC3P-style rank filtering. It derives an MSE improvement condition under unbiasedness and decorrelation assumptions and evaluates the method on CIFAR-10/100-LT, ImageNet-LT, and iNaturalist-2018. The experiments report substantial prediction-set reductions relative to RC3P, including comparisons under common coverage and UCR constraints. Coverage of the complete procedure is assessed empirically.

Critical strengths

Table 5 reports an 18.9% reduction in average set size, from 25.126 to 20.377, alongside improved UCR (.128 to .120) and minimum class coverage (.802 to .820), while both methods exceed the .90 marginal coverage target (.943 for RC3P versus .941 for SARC-CP). These results demonstrate a favorable empirical size–coverage tradeoff for SARC-CP on CIFAR-100-LT.

Critical weaknesses
  1. Interp-Q [1], which the paper cites, already interpolates classwise and global conformal quantiles. The same work introduces Fuzzy CP, whose original implementation adjusts cross-class pooling according to class calibration counts. This directly relevant adaptive pooling method is omitted. Kim’s SCC-CP [2], also omitted, explicitly shrinks cluster-level conformal quantiles toward a global threshold and provides a bias–variance/MSE analysis that includes covariance between the estimators. These works already cover threshold shrinkage, sample-dependent borrowing, and its estimation-risk interpretation and thus narrow the novelty claim. Under the stated equal-count CIFAR and ImageNet protocols, even this weight becomes constant across classes: λy=κN+κ\lambda_y=\frac{\kappa}{N+\kappa}. Consequently, those experiments do not establish an advantage attributable to sample-size adaptation. The paper needs to establish the substantive contribution of its particular count-based weighting rule and its integration with RC3P beyond these existing approaches.

  2. The theoretical analysis does not justify the stated procedure or the claimed automatic benefit for tail classes. Proposition 1 is correct under its assumptions, but it analyzes an unbiased nominal class quantile uncorrelated with the global estimate. The procedure instead uses quantile inflation and computes both estimates from overlapping calibration data. Even interpreting Eq. 11 as an upward inflation of the nominal quantile level, the leading bias is

\qquad \mathbb{E}[\widehat q_y^{,\mathrm{infl}}]-q_y^\star \approx \frac{g}{\sqrt{n_y},f_y(q_y^\star)}.$$ Its squared bias has the same asymptotic order as the variance, so it cannot be discarded in the proposed MSE justification. The shared-data covariance also requires treatment. Appendix C.1 additionally extrapolates a large-sample variance approximation toward zero samples to assert automatic improvement for tail classes. That conclusion does not follow. 3. In Table 3, with 30 calibration examples per class, the ImageNet set size falls from 48.352 to 16.942, but marginal coverage falls from .932 to .901, UCR increases from .184 to .293, and the coverage gap increases from .016 to .037. With 10 calibration examples, SARC-CP achieves only .863 marginal coverage, contradicting Appendix C.3’s statement that every evaluated configuration meets the .90 target. Table 1’s common-constraint comparison provides legitimate evidence of efficiency gains; it does not establish the broader claim of preserved coverage across configurations or identify variance reduction as their cause. 4. For iNaturalist, the paper combines a published checkpoint with a new split of the original train+val pool, without establishing that calibration and test images were excluded from checkpoint training and selection. Table 4 also reports approximately three test examples per class. With three independent examples, even a class whose true coverage is exactly 90% is labeled undercovered with probability \$ 1-0.9^3=0.271. \$ Thus, the reported empirical UCR cannot be interpreted directly as the fraction of truly undercovered classes; averaging random splits does not remove this bias. These limitations materially weaken the evidence for reliable performance in the extreme-tail regime. [1] Tiffany Ding, Jean-Baptiste Fermanian, and Joseph Salmon. Conformal Prediction for Long-Tailed Classification. 2026. [2] Jisu Kim. Shrinkage Class-Clustered Conformal Prediction. M.S. thesis, Seoul National University, February 2026.
Questions for authors
  1. After accounting for Interp-Q, Fuzzy CP, and SCC-CP, what precise methodological or theoretical contribution remains? Which reported result establishes an advantage specific to calibration-count adaptation, given the equal-count CIFAR/ImageNet protocols and the quartile results in Figure 3?

  2. What finite-sample quantile rule is implemented in Eq. 11, including out-of-range handling? Which conclusions of Proposition 1 apply after accounting for inflation bias and shared-data covariance, and what additional condition supports the claimed automatic benefit for tail classes?

  3. Figure 2 reports an average set size of 33.4 without shrinkage, whereas Table 5 reports 25.126 for RC3P. Which inflation, rank-gating, or hyperparameter settings differ between these configurations? Section 5.1 states that hyperparameters are selected on a held-out calibration slice. Was Table 1’s UCR target also determined on that slice, and were the reported results subsequently evaluated on independent test data? Please specify the tuning-slice size and the final per-class calibration counts.

  4. Is the 80% partition used for calibration? Please specify the roles of all three partitions and establish that calibration and test images were excluded from checkpoint training and selection. Please also report the per-class test-count distribution, explain how classes with no test examples are handled, and clarify how finite-test uncertainty affects the interpretation of the reported UCR.

Reviewer_Mw1EOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies and focuses on the variance of coverage in conformal prediction. The main question the paper poses is how to balance class-wise coverage and marginal coverage. In long-tailed scenarios, the cost of class-conditional coverage is high (set sizes are large), but we also want nearly valid predictions for rare and unusual classes.

Critical strengths

The focus on the variance of conformal prediction is quite important and timely. Moreover, class-conditional conformal prediction with many classes and few calibration points per class is a real bottleneck. The experimental result show strong improvement in terms of set-sizes and classical coverage metrics.

Critical weaknesses

Theory: The key property of conformal prediction is its theoretical guarantee. The authors state explicitly that this guarantee is lost. Classwise CP, RC3P, and Interp-Q carry finite-sample class-conditional guarantees, whereas SARC-CP carries none. Moreover, the paper needs to show explicitly why the proposed method is optimal (in whatever sense is intended) within a family of methods.

Practice: The practical results depend on the setup, and I am not entirely sure that all of the baselines were chosen properly, which makes the empirical contribution unclear. For example, Inter-Q was tuned for which \alpha? Probably, if we tune SARC-CP for the same alpha and same theoretical guarantee the result will be opposite.

Questions for authors
  1. Does the method have any valid theoretical guarantee? Perhaps 1 − 2α, as in Interp-Q?
  2. Please confirm that n_y = N for every class in the CIFAR and ImageNet-LT experiments. If so, how does SARC-CP differ from Interp-Q with τ = N/(N + κ) combined with RC3P’s inflation and rank gate, and could you report that combination as a baseline?
  3. How were the hyperparameters for the baselines chosen? Your method is very similar to Interp-Q (one reduces to the other when the classes are equally probable). Both SARC-CP and Interp-Q have hyperparameters that depend on the miscoverage level α. How were these chosen for the baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.