acceptodds
Under review as a conference paper at ICLR 2027

Calibration Count Reuse: Validity Does Not Determine Efficiency

Abstract

Calibration count reuse raises separate validity and efficiency questions. We give a validity criterion for general count-dependent nonconformity scores: transferring one count from another class to the scored class must not improve its conformity. A leave-self-out full conformal reference proves the criterion without requiring normalization or preservation of same-class score order. For a common separable transformation, universal exchangeable validity is equivalent to being nondecreasing in the count, provided Kα ≥ 1; normalized multiplicative weights obey the complementary nonincreasing condition. Additive penalties are covered under the stated information restrictions. Efficiency has no parallel ordering: two iid constructions make the same valid rule improve or worsen expected size at unchanged coverage. An expanded 55-rule study finds no resolved advantage from selected live-count rules over uniform weights. Image studies identify undercoverage under iid resampling, including at numerical convergence. Separately, execution of the released Conf-OT pipeline on its DTD and Aircraft benchmark subsets produces near-nominal median coverage under fixed stratified counts. The native results are reported separately from the iid analyses, without treating a benchmark observation as a universal guarantee. The findings separate validity, classifier confidence, numerical convergence, and population-specific efficiency.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, calibration, exchangeability, uncertainty quantification, coverage guarantees, transductive inference, nonconformity scores, prediction set efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_mEoEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the effect of reusing calibration-set labels on the validity of the prediction sets generated by conformal prediction. The authors show that reusing the count of these labels might harm the validity of the conformal guarantee and provide the criterion under which validity is preserved despite reusing the calibration labels. These findings are demonstrated empirically on real datasets.

Critical strengths
  1. Theorem 1 is insightful - it provides a sufficient condition for conformal prediction to remain valid despite label count reuse.
  2. Theorem 2 is also interesting - it demonstrates that the same valid weighting function can either improve or worsen the informativeness of the prediction set size compared to a uniform weighting.
Critical weaknesses
  1. The paper is very hard to read, and its presentation is unclear. For example, Conf-OT is not fully described; the paper uses notations without defining them (line 043), and the Independent and Augmented prior methods are not sufficiently explained.
  2. The practical contribution of the valid count-dependent weighting functions is unclear. The experiments do not show that these procedures improve efficiency compared to uniform weighting. The method that uses pre-trained fixed weights constructs smaller prediction sets than the count-reusing methods.
Questions for authors
  1. The experiments show that the 55 candidates do not achieve improved performance compared to the uniform weighting function. Can the authors provide conditions under which live count reuse improves the efficiency compared to independently fitted weights or uniform weights?
  2. Since class frequency can be estimated from an independent sample without invalidating the theoretical guarantee of conformal prediction, what is the advantage of using the calibration counts? Can the authors provide a setting in which this approach improves efficiency compared to weights estimated from an independent set?
Reviewer_s6knOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how reusing calibration class counts affects the validity and efficiency of conformal prediction. The authors propose a theoretical condition for ensuring valid count reuse and use counterexamples to show that improper reuse can lead to severe undercoverage. They also demonstrate that validity does not necessarily lead to improved efficiency. Through theoretical analysis, image classification experiments, and a reproduction of Conf-OT, this paper further examines the potential risks of count reuse and the uncertainty of its efficiency benefits.

Critical strengths
  • The paper establishes a validity criterion for calibration count reuse, providing a theoretical foundation for understanding when such reuse is safe.
  • The paper distinguishes validity from efficiency and demonstrates that valid count reuse does not necessarily improve prediction efficiency.
  • The paper combines theoretical analysis, counterexamples, and empirical experiments to support its main findings.
Critical weaknesses
  • The paper mainly compares prediction set sizes using the standard calibration procedure, without fully considering other valid calibration methods. This limits the scope of its efficiency conclusions.
  • The statistical tests may not be sensitive enough to detect small efficiency differences, making it difficult to clearly distinguish the performance of different methods.
  • The paper lacks sufficient experiments on long-tailed class distributions. It remains unclear whether the findings generalize to datasets with severe class imbalance.
Questions for authors
  • Why does the efficiency evaluation focus on CEC_E rather than tighter, valid references? Would coverage-matched comparisons change the conclusions?
  • The observed efficiency differences are relatively small. Are the statistical tests sufficiently sensitive to detect meaningful improvements? How reliable is the reported lack of advantage?
  • Does the identified failure mechanism occur in practical methods, particularly native Conf-OT with fixed per-class counts?
  • Would the findings generalize to long-tailed datasets, especially for minority classes?
  • What is the main novelty of Theorem 1 beyond existing validity arguments? What practical guidance does it provide?
  • Do the efficiency reversals in Theorem 2 persist near nominal coverage, rather than under the highly conservative coverage of the constructed examples?
Reviewer_nPgMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the validity and efficiency of reusing calibration-label counts in conformal prediction. It establishes a sufficient condition for marginal coverage, together with necessary and sufficient characterizations for specific score families. The paper also provides theoretical counterexamples showing that valid count-dependent adjustments can either improve or worsen prediction-set efficiency, while invalid reuse can cause severe undercoverage. Experiments with CLIP and SigLIP on image classification datasets further demonstrate these coverage failures, while comparisons of 5-5 valid weighting rules reveal no statistically significant efficiency advantage over uniform weighting.

Critical strengths

Theoretical characterization of validity.

The paper establishes a general sufficient condition for valid calibration-count reuse, along with necessary and sufficient conditions for specific score families. These results provide useful theoretical guidance for designing conformal prediction methods with coverage guarantees.

Theoretical and empirical validation.

The paper provides counterexamples demonstrating severe coverage failures and supports these findings with experiments on real image classification datasets.

Critical weaknesses

Poor clarity, organization, and editorial quality. The paper is unnecessarily difficult to follow, even for readers familiar with conformal prediction. It introduces extensive notation and multiple theoretical constructions without sufficient intuition or a clear explanation of their relationships. Theorem 1 combines several results with different assumptions and scopes, making its precise implications difficult to understand. Moreover, the presentation lacks a coherent narrative connecting the theoretical results to the empirical investigations. These issues substantially undermine the accessibility of the paper and make it difficult to assess the significance of its contributions. Minor presentation issues include the placement of captions below Tables 1 and 3, and the fact that Figure 1 is essentially a table.

More importantly, I have serious concerns about the editorial oversight of the AI-assisted content. The main text contains several unnecessarily detailed implementation-specific references that seem more appropriate for supplementary materials, including:

  • Table 4: "Unadapted is the repository's none branch."
  • Section 5: "The new general_scores/rank_reference.py implements..."
  • Section 7.5: "We separately executed the released extractor and benchmark driver at commit 1a29ced..."

Similar details appear extensively throughout the appendix. While such references do not themselves demonstrate improper AI use, their inclusion in the main text suggests insufficient editorial refinement. I am seriously concerned about the extent to which the manuscript and its underlying implementations were manually reviewed and verified by the authors.

Limited significance and practical impact. While the paper provides a characterization of the validity of calibration-count reuse, the practical significance of the identified failure mechanism remains unclear. In particular, the reproduced Conf-OT benchmark exhibits near-nominal coverage, suggesting that the severe undercoverage demonstrated under alternative sampling settings may not directly translate to the original application. Moreover, the proposed valid weighting rules do not demonstrate statistically significant efficiency improvements over uniform weighting. These observations raise concerns about the broader impact of the theoretical findings.

Questions for authors

Given the near-nominal coverage of Conf-OT in Table 4, how practically significant is the identified count-reuse failure, and under what realistic conditions would severe undercoverage occur?

Reviewer_pwUGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether final calibration labels in split conformal prediction can be reused to adjust class-frequency weights in nonconformity scores Si(h;c)=Ψh(Vi(h),c)S_i(h; c) = \Psi_h(V_i(h), c) where ch=∑i=1n1{Yi=h}c_h = \sum_{i=1}^n \mathbf{1}\{Y_i = h\}. Using a leave-self-out full conformal reference (ClooC_{\text{loo}}), Theorem 1 establishes a general count-transfer validity condition (Ψj(v,c+ej−eh)≥Ψj(v,c)\Psi_j(v, c + e_j - e_h) \ge \Psi_j(v, c) for j≠hj \neq h) and proves sharp iff boundaries for common separable maps (ϕ(v,m)\phi(v, m) nondecreasing in mm) and row-normalized multiplicative weights (f(m)f(m) nonincreasing in mm) when Kα≥1K\alpha \ge 1. Theorem 2 proves that validity does not imply efficiency: the same valid decreasing rule can strictly shrink or enlarge expected set size relative to uniform weights at unchanged coverage. Finally, the paper proves and empirically demonstrates on CLIP/SigLIP features that empirical-prior optimal transport (which uses increasing weights dh=ch+ad_h = c_h + a, violating Theorem 1) suffers severe undercoverage under i.i.d. sampling—dropping to 64.26%64.26\% at Sinkhorn convergence on CIFAR-10 at nominal 90%90\%—whereas Conf-OT's released benchmark avoids this collapse only because its code uses fixed stratified class counts.

Critical strengths

Theorem 1 cleanly settles when calibration label counts can and cannot be reused without breaking exchangeability, unifying both normalized multiplicative weights and additive rank penalties (such as soft TACP) without requiring same-class order preservation.

Critical weaknesses
  • The manuscript is written as if replying to a prior round of hostile reviews or legal audits (e.g., repeatedly referencing "the present revision", "the earlier report", "post-confirmation sensitivity", "historical reconstruction discrepancy", and file paths like reviewer_audit/count_weight_guard.py). Definitions and context for external methods (like Conf-OT, TACP, and Sinkhorn transport cycles) are introduced abruptly without self-contained background equations.
  • Practical Takeaway for Safe Live-Count Reuse is Essentially Negative: To satisfy Theorem 1, multiplicative weights f(ch)f(c_h) must be nonincreasing in the empirical class count chc_h—meaning frequent classes are down-weighted and rare classes are up-weighted in probability, which is the exact opposite of standard prior-adaptation (Bayes' rule p(y∣x)∝p(y)p(x∣y)p(y\mid x) \propto p(y)p(x\mid y)). Consequently, as shown in Section 7.4 (Table 3), across 55 valid nonincreasing candidate rules, live calibration counts never outperform simple frozen weights estimated on a tiny independent split.
Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.