acceptodds
Under review as a conference paper at ICLR 2027

Converting Efficiency to Classwise Coverage: Cluster-Frequency Conformal Prediction

Abstract

Conformal prediction guarantees marginal coverage under exchangeability, but coverage can vary substantially across classes. We propose Cluster-Frequency Conformal Prediction (CFCP), which forms probability vectors from softly pooled cluster-label frequencies, prior smoothing, and a reliability-weighted fallback before standard conformal scoring. Independent final calibration preserves marginal validity. Across seven image and text evaluation settings, including three with long-tailed training and four score families, we examine whether efficient prediction sets provide headroom for classwise coverage. CFCP can achieve smaller prediction sets at matched marginal coverage, motivating comparisons of classwise coverage at equal size budgets. At CFCP's nominal mean set size, paired comparisons favor CFCP in 22 of 28 cells and the selected best baseline in one, with five inconclusive and gains up to 22.47 percentage points. Comparisons averaged over shared size ranges favor CFCP in 24 cells and the baseline in three, with one inconclusive. These results support an empirical classwise coverage-size advantage without establishing classwise coverage guarantees.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Conformal prediction, uncertainty quantification, classwise coverage, prediction-set efficiency, representation-space clustering, local label frequencies

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YeETOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Cluster-Frequency Conformal Prediction (CFCP) to improve the classwise coverage-efficiency trade-off in conformal prediction. CFCP estimates local label distributions from learned representations using soft clustering, prior smoothing, and reliability-weighted interpolation with the base model's probabilities. The resulting probability vectors are then used with standard conformal scores. Experiments across seven image and text settings show favorable classwise coverage-set size trade-offs against existing baselines.

Critical strengths
  1. The paper evaluates CFCP across image datasets with different resolutions, a text classification dataset, multiple model architectures, and four conformal score functions. The inclusion of long-tailed training settings further broadens the evaluation.

  2. CFCP incorporates cluster-level label information into probability estimation while retaining standard conformal calibration. This design is compatible with existing score functions and maintains marginal validity under the stated assumptions. The matched-size and shared-range comparisons also provide a useful perspective on the coverage-efficiency trade-off.

Critical weaknesses
  1. Theorem 1 establishes standard marginal coverage validity, while Theorem 2 characterizes the population optimality of LAC under the true conditional probabilities. Neither result explains why CFCP's cluster-frequency estimator should improve classwise coverage or under what conditions it provides better probability estimates.

The authors explicitly acknowledge that "neither theorem establishes improved local or classwise coverage." It raises questions about the role of Section 3 in supporting CFCP. In particular, Theorem 2 concerns an idealized LAC predictor rather than CFCP's actual estimator, and its relevance to the other score functions remains unclear.

  1. In the long-tailed experiments, CFCP estimates local label frequencies using labels from a class-balanced calibration pool (Protocol B, Appendix C.1.2), rather than the imbalanced training distribution. The improvements may partly arise from access to label-frequency information reflecting the balanced evaluation distribution, rather than the proposed clustering mechanism.

Furthermore, as acknowledged in the paper, adjusting the baseline to CFCP's set-size budget often reduces its marginal coverage. Therefore, improvements in COV may reflect differences in overall coverage as well as differences in how coverage is distributed across classes. The current experiments do not fully disentangle these effects.

  1. Additional comments on presentation.

a. Section 1.1 mixes background concepts, formal definitions, and related work. In particular, placing formal definitions and notation within the related-work subsection interrupts the presentation. Separating these elements could improve the organization.

b. Figure 2 could be substantially improved. Although it illustrates the overall pipeline, its components are not closely connected to the mathematical formulation in Section 2. Explicitly mapping the figure to the soft assignments, cluster-frequency estimation, smoothing, reliability weighting, and final calibration steps would make the method easier to follow.

Questions for authors
  1. What is the intended role of Theorem 2 in supporting CFCP's main contribution? Since it assumes access to the true posterior and only concerns LAC, can the authors clarify how it relates to the proposed cluster-frequency estimator and its observed classwise coverage improvements?

  2. In the long-tailed experiments, can the authors isolate the effect of local clustering from that of using label-frequency information from the balanced calibration pool? For example, how does CFCP compare with a global prior-correction method using the same frequency data but without clustering?

  3. Given that matched-size comparisons can result in different marginal coverage levels, to what extent do the reported COV improvements reflect more balanced classwise coverage rather than higher overall coverage? Additional analysis of classwise coverage dispersion relative to realized marginal coverage would help clarify this.

Reviewer_Tpe3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Although conformal prediction provides marginal coverage guarantees, when applied to distributions with many classes and/or severe long-tailed distributions, it can undercover tail classes (i.e., classes with few examples). Several existing works in the literature provide coverage guarantees across all classes or, more generally, across groups. Following this direction, the authors propose CFCP to improve conditional conformal prediction. Unlike Cluster CP [1], which groups classes into clusters based on their nonconformity scores, the authors argue that the learned model embeddings provide a better semantic representation of the examples. Compared with existing conditional conformal prediction methods, and across datasets with many classes or long-tailed distributions, the results show that CFCP yields smaller average prediction set sizes and a lower proportion of classes that violate the 1−α1-\alpha coverage level.

References

  1. Ding, Tiffany, et al. "Class-conditional conformal prediction with many classes." In NeurIPS. 2023.
Critical strengths
  1. The paper identifies an important and interesting problem.
Critical weaknesses
  1. The paper is really hard to read and requires extensive improvements in its presentation. I recommend that the authors put more effort into improving the paper's clarity and presentation. In particular, I had difficulties understanding Section 2.1.
  2. The whole premise of conformal prediction is its ability to provide uncertainty quantification with provable guarantees. While the authors establish marginal coverage validity for CFCP, the proposed improvements in classwise coverage remain purely empirical. The paper does not provide theoretical guarantees that CFCP improves classwise coverage, and the authors themselves acknowledge this limitation. Given that the method aims to improve classwise coverage, this is an important limitation.
  3. The paper proposes a reliability weight r(⋅)r(\cdot) based on cluster assignment confidence and support, but provides no theoretical justification for this particular weighting function. The authors describe it as a heuristic and provide empirical ablations, but do not establish why this specific form should be effective beyond the reported experiments.
  4. I do not understand the purpose of Theorems 1 and 2. Theorem 1 restates the standard marginal validity result for split conformal prediction, while Theorem 2 recalls a classical population oracle result for LAC. Neither result explains why the proposed method should improve class-wise coverage, justifies the design choices behind CFCP, or establishes its claimed empirical benefits. The authors should better motivate the inclusion of these results and clarify how they contribute to the paper's main claims.
Questions for authors
  1. You may want to proofread your results. Table 1 has inconsistent bold formatting across several rows.
  2. I recommend that the authors add a section, either in the main paper or the supplementary material, briefly explaining what the different baselines are and how they work.
  3. The authors should be wary of the notation used throughout the paper. Usually, qq is used to denote the quantile, but throughout the paper, it is used to denote class probability.
  4. How does performance change for different values of KK and mm?
  5. How does model performance change on datasets with severe long-tail distributions, such as Plant@300K or iNaturalist2018?
  6. How do you compare against Interp-Q [1]?

References.

  1. Ding, Tiffany, Jean-Baptiste Fermanian, and Joseph Salmon. "Conformal prediction for long-tailed classification." In ICLR. 2026.
Reviewer_KXz8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the uneven class-wise coverage of split conformal prediction. Rather than refining the calibration step, as class-conditional, clustered or neighbourhood conformal methods do, the proposed method (CFCP) modifies the probability vector given to a standard conformal score. It clusters the classifier's representations with spherical k-means, estimates a smoothed label distribution for each cluster from a labelled frequency sample, mixes the distributions of the query's nearest centroids, and interpolates the result with the base model's softmax using a heuristic reliability weight. The resulting vector is used with LAC, APS, RAPS or SAPS and calibrated on a separate calibration set, so the standard marginal guarantee is preserved; no class-wise guarantee is claimed. Experiments on CIFAR-100, ImageNet, ImageNet-V2, WOS-46985 and three long-tailed variants, against Split, ICP, CCP, NCP and RC3P, evaluate class coverage (the fraction of classes reaching 0.9 coverage) at matched coverage, at matched mean set size and over shared set-size ranges. At matched set size CFCP is favoured in 22 of 28 dataset–score cells, and its prediction time is lower than that of the evaluated NCP implementation.

Critical strengths

S1. Separating probability estimation from threshold calibration is a clean design: the refined probability vector can be plugged into any standard conformal score, marginal validity is untouched, and no per-class or per-query calibration is needed. Because it queries a compact centroid table rather than the calibration set, it is also considerably faster at prediction time than NCP.

S2. The study covers image and text data, four score families, five baselines and long-tailed training, and goes beyond coverage-matched comparison by evaluating at equal set size and over shared size ranges. The latter two comparisons use paired splits and confidence intervals, and the paper reports cells where the baseline wins or the result is inconclusive. The paper is also explicit that CFCP gives no class-wise or local guarantee and about the limitations of its protocols.

Critical weaknesses

W1. There are many Many hyperparameters, and no sensitivity analysis for the key locality parameters (K, m, τ). CFCP has six hyperparameters (K, m, τ, β, β_sup, γ). Its search spaces contain up to 243 configurations, although not every parameter is tuned independently on every dataset and some settings are transferred between related datasets. The two ablations on CIFAR-100 and WOS-46985 separately test removing cluster-frequency substitution and removing reliability shrinkage, but there is no sensitivity analysis over the number of clusters K, the number of retrieved centroids m, or the temperature τ, which control how local the probability estimate is. It is therefore unclear whether the gains hold for a reasonable default configuration or depend on dataset-specific choices. Cross-method tuning effort is also difficult to compare directly: Split has no hyperparameters, CCP uses automatic settings, RC3P uses its own mixing grid without a hyperparameter split, and NCP is tuned on a 30% hold-out split. One of CFCP's selection rules directly maximises matched-size class coverage, which is close to the reported metric.

W2. The theory does not explain CFCP's class-wise gains. Theorem 1 applies the standard split-conformal argument to the fixed CFCP score and therefore establishes marginal validity, but it does not address class-wise coverage. Theorem 2 is a population-oracle result showing that LAC with the true conditional distribution minimises expected set size among predictors with the required marginal coverage. This provides a relevant efficiency motivation, but the paper does not show (and the appendix explicitly does not claim) that CFCP's vector approaches the true conditional distribution. Consequently, neither theorem explains why cluster-frequency pooling should improve class-wise coverage, nor does it provide a CFCP-specific finite-sample efficiency analysis; those benefits remain empirical.

W3. The writing and organisation are hard to follow. Much of the information needed to interpret the main results (fitting protocols, hyperparameter grids, selection rules) is in an appendix that is hard to navigate; for example, Appendix C.2.1 ("Class coverage") consists of a heading only.

Questions for authors

Table 1 reports means over five splits, whereas Tables 2–3 use ten splits with intervals. Could you add intervals to Table 1?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers. We have uploaded a revised PDF. The method, its marginal guarantee, Theorem 1, the main experiments and the contribution list are those of the submission. The submitted abstract and Limitations already stated that CFCP guarantees marginal coverage only and that its class-level gains are empirical. The revision clarifies wording, reorganizes the presentation, and adds the analyses the reviewers requested:

  • Wording of claims (all). The title drops "Classwise", which read as a guarantee. Section 3 now opens by stating the role of each result, and new Remark 1 relates Theorem 2 to our class-balanced evaluation. The Limitations state that the gains are not evidence of more uniform class coverage.
  • Presentation (all). New Section 1.2 covers setup and notation. Section 2 is built around η^(x)\hat\eta(x), and Figure 2 is redrawn with each step mapped to its equation. Appendices C.1.2–C.1.3 describe the fitting protocols and the baselines.
  • Hyperparameters (KXz8, Tpe3). Appendix C.7 adds a sensitivity analysis over K,m,τ,γK, m, \tau, \gamma, a configuration fixed in advance, and NCP tuned on a larger grid.
  • Reliability weight (Tpe3). Its form is now motivated by the squared-error-optimal mixing weight (Appendix A.2.1).
  • Severe long tail (Tpe3). We added PlantNet-Trunc and a comparison with Interp-Q (Section 4.1, Appendix C.5).
  • Table 1 (KXz8, Tpe3). Rebuilt from a single five-split source, with 95% CIs in Appendix Table 7.

The prior-correction control and the level/balance decomposition requested by YeET are reported in our response to YeET.

Some limitations remain and are stated in the paper: CFCP gives no classwise or local guarantee, and it is not designed for long-tailed calibration data.

We hope the revision addresses the reviewers' main concerns, and we would be grateful if they took it into account in their final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.