- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Cluster-Frequency Conformal Prediction (CFCP) to improve the classwise coverage-efficiency trade-off in conformal prediction. CFCP estimates local label distributions from learned representations using soft clustering, prior smoothing, and reliability-weighted interpolation with the base model's probabilities. The resulting probability vectors are then used with standard conformal scores. Experiments across seven image and text settings show favorable classwise coverage-set size trade-offs against existing baselines.
Critical strengths
-
The paper evaluates CFCP across image datasets with different resolutions, a text classification dataset, multiple model architectures, and four conformal score functions. The inclusion of long-tailed training settings further broadens the evaluation.
-
CFCP incorporates cluster-level label information into probability estimation while retaining standard conformal calibration. This design is compatible with existing score functions and maintains marginal validity under the stated assumptions. The matched-size and shared-range comparisons also provide a useful perspective on the coverage-efficiency trade-off.
Critical weaknesses
- Theorem 1 establishes standard marginal coverage validity, while Theorem 2 characterizes the population optimality of LAC under the true conditional probabilities. Neither result explains why CFCP's cluster-frequency estimator should improve classwise coverage or under what conditions it provides better probability estimates.
The authors explicitly acknowledge that "neither theorem establishes improved local or classwise coverage." It raises questions about the role of Section 3 in supporting CFCP. In particular, Theorem 2 concerns an idealized LAC predictor rather than CFCP's actual estimator, and its relevance to the other score functions remains unclear.
- In the long-tailed experiments, CFCP estimates local label frequencies using labels from a class-balanced calibration pool (Protocol B, Appendix C.1.2), rather than the imbalanced training distribution. The improvements may partly arise from access to label-frequency information reflecting the balanced evaluation distribution, rather than the proposed clustering mechanism.
Furthermore, as acknowledged in the paper, adjusting the baseline to CFCP's set-size budget often reduces its marginal coverage. Therefore, improvements in COV may reflect differences in overall coverage as well as differences in how coverage is distributed across classes. The current experiments do not fully disentangle these effects.
- Additional comments on presentation.
a. Section 1.1 mixes background concepts, formal definitions, and related work. In particular, placing formal definitions and notation within the related-work subsection interrupts the presentation. Separating these elements could improve the organization.
b. Figure 2 could be substantially improved. Although it illustrates the overall pipeline, its components are not closely connected to the mathematical formulation in Section 2. Explicitly mapping the figure to the soft assignments, cluster-frequency estimation, smoothing, reliability weighting, and final calibration steps would make the method easier to follow.
Questions for authors
-
What is the intended role of Theorem 2 in supporting CFCP's main contribution? Since it assumes access to the true posterior and only concerns LAC, can the authors clarify how it relates to the proposed cluster-frequency estimator and its observed classwise coverage improvements?
-
In the long-tailed experiments, can the authors isolate the effect of local clustering from that of using label-frequency information from the balanced calibration pool? For example, how does CFCP compare with a global prior-correction method using the same frequency data but without clustering?
-
Given that matched-size comparisons can result in different marginal coverage levels, to what extent do the reported COV improvements reflect more balanced classwise coverage rather than higher overall coverage? Additional analysis of classwise coverage dispersion relative to realized marginal coverage would help clarify this.
General response to all reviewers
We thank the reviewers. We have uploaded a revised PDF. The method, its marginal guarantee, Theorem 1, the main experiments and the contribution list are those of the submission. The submitted abstract and Limitations already stated that CFCP guarantees marginal coverage only and that its class-level gains are empirical. The revision clarifies wording, reorganizes the presentation, and adds the analyses the reviewers requested:
The prior-correction control and the level/balance decomposition requested by YeET are reported in our response to YeET.
Some limitations remain and are stated in the paper: CFCP gives no classwise or local guarantee, and it is not designed for long-tailed calibration data.
We hope the revision addresses the reviewers' main concerns, and we would be grateful if they took it into account in their final assessment.