acceptodds
Withdrawn from ICLR 2027

Mahalanobis-Smoothed Trust Conformal Prediction under Limited Calibration Data

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Conformal prediction endows any classifier with a distribution-free coverage guarantee, yet under limited calibration data the guarantee is nearly vacuous: the returned prediction sets routinely grow to almost the size of the full label space, eroding the very utility they are meant to provide. We trace this failure to a structural blind spot shared by every published conformity score: each reads the model solely through the post-softmax probability vector—a bottleneck that discards the class-conditional geometry the encoder has already computed, and which often separates the classes more cleanly than the linear head that projects them. We show that this geometry can be handed back to the calibration pipeline at negligible cost: because every encoder already computes a hidden state alongside its logits, a simple per-class distance to training-set centroids recovers exactly the structure the softmax throws away. \methodlong (\method) builds on this observation—it blends the centroid distance with the familiar rank-regularised score on temperature-scaled logits, and a holdout slice of the calibration data picks the blend for each deployment—so the score leans on geometry where it helps and safely falls back to the softmax ranking where it does not. The guarantee survives this adaptivity on one condition, a data-handling discipline: the geometric statistics must be estimated on training data—estimating them on the calibration set breaks exchangeability and inflates set size by . Across cells of an LLM-style sequence-classification benchmark, \method delivers the smallest prediction set in settings and the smallest average against nine published baselines, reducing average set size by over APS, over RAPS, and over the strongest competing baseline, while preserving the requested marginal coverage to within in every evaluated cell.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Conformal prediction, Prediction set efficiency, Representation geometry

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YZcwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes M-STC, an improved method for conformal prediction designed to produce smaller prediction sets when calibration data is limited. The method combines Softmax-based scores with class-conditional distances derived from the model’s hidden representations, using a separate calibration set to determine the weights for integrating these two components.

Critical strengths
  1. This issue is importance; in constrained calibration scenarios, it presents a very real, practical problem.

  2. The results appear good within the scenario they constructed.

Critical weaknesses
  1. Poor writing. Many strange writings, such as "Across 18 cells against nine baselines (2017–2026),"

  2. The theorem proof is provided only as a Sketch; a formal proof is not included.

  3. This idea is not novel; it appears in a great many papers, such as [1] and [2].

[1] Zeng, Hao, et al. "Parametric scaling law of tuning bias in conformal prediction." arXiv preprint arXiv:2502.03023 (2025).

[2] Teng, Jiaye, et al. "Predictive inference with feature conformal prediction." arXiv preprint arXiv:2210.00173 (2022).

Questions for authors
  1. Can the authors provide a formal proof for the main theoretical claims presented in the paper?

  2. REP-CP is mentioned only briefly, and its methodological formulation is not formally articulated. What is the relationship between REP-CP, the current study, and M-STC? Why are they referred to by different names?

  3. Have the authors explored using representations extracted from other layers of the encoder?

4 comments

Thank you for the careful review. The main changes:

  1. Valid guarantee. The submitted code fitted the temperature T, the median normaliser m̃ and the weight w on the same calibration set that gives the quantile, and used an interpolated quantile with a strict <. Prop. 1 did not cover that procedure. We now use a strict split:

    • T, m̃ and w are fitted on a tuning part D_tune (30% of the labelled budget B);
    • the quantile is the exact ⌈(n+1)(1−α)⌉-th order statistic on the disjoint D_cal;
    • sets use ≤.

    Prop. 1′, with a full proof, covers this. All tuned baselines use the same split, and all methods use the same B.

  2. Prop. 2 is withdrawn as an optimality claim and replaced by a corrected Prop. 2′.

  3. Novelty is stated narrowly. We drop "the first conformal score to use hidden representations". The contribution is a convex blend of a softmax rank score with a training-set class-centroid distance, with the blend weight tuned under a valid split.

  4. New experiments:

    • three fine-tuned encoders: RoBERTa-base, DeBERTa-v3-base, and Qwen2.5-1.5B with LoRA;
    • four datasets: AG News, DBpedia-14, Banking77, 20NG;
    • new baselines: NCP, DkNN, Trust-CP, CSA, atypicality-aware RAPS (AA-RAPS) and Conf-OT;
    • a layer study, and 100 re-splits per trained encoder.

Main results under the strict protocol

  • The paper's synthetic conclusions hold. At B = 1000, the mean size change of M-STC (submitted numbers in brackets) is:

    vsChange
    APS−41.9% [−44.3%]
    TS-RAPS−19.6% [−20.9%]
    RAPS−11.9% [−12.3%]
    LAC−8.9% [−7.0%]
    CSA−7.5% [−4.9%]

    M-STC has the smallest mean in 5 of 6 settings. It is smaller than every external baseline in all 12 settings (B ∈ {500, 1000}); against the best external baseline of each setting it is smaller by 7.5% on average at B = 1000 and 7.2% at B = 500.

  • Coverage is valid (up to Monte-Carlo error) in all 12 synthetic and 48 real groups. Conf-OT, by contrast, under-covers in 15 of the 48 real groups.

  • Real encoders. M-STC is smaller than AA-RAPS, Trust-CP, APS and NCP in 48, 47, 45 and 44 of the 48 groups.

    • On Banking77 (77 classes, about 13 training examples per class), it is the smallest valid method in all 8 RoBERTa and DeBERTa groups: 9.4–51.6% smaller than LAC, with much better worst-class coverage.
    • On the high-accuracy datasets (AG News, DBpedia, 20NG), the softmax scores LAC, CSA and Conf-OT remain very strong, and M-STC is close to them but not ahead.

W1. Writing; "18 cells against nine baselines (2017–2026)". Thank you. "18 cells" meant 6 settings (3 training sizes × 2 α) × 3 seeds. The percentages averaged the seeds within each setting and then averaged the 6 settings; the win counts used the 18 runs. The revision states each aggregation explicitly, and drops "(2017–2026)" and similar phrases. The new tables report 6 settings, each over 5 trained encoders × 100 calibration/test re-splits.

W2 / Q1. Formal proofs. Both proofs are given in full below and will replace the sketches (Appendix B).

Setting. D_tr (training), D_tune (size n_tune) and D_cal = {(X_i, Y_i)}{i=1..n} are disjoint, and (X{n+1}, Y_{n+1}) is the test point. The fitted parameters θ̂ = (μ, σ, T, m̃, w) are any measurable function of (D_tr, D_tune, U), where U is independent auxiliary randomness (e.g. the cross-fitting split). The score is s_θ̂(x, y) = (1−w)·RAPS_T(x, y) + w·‖h(x) − μ_y‖ / (σ_y m̃). Let k = ⌈(n+1)(1−α)⌉. The threshold is q̂ = S_(k), the k-th smallest of S_i = s_θ̂(X_i, Y_i), i ≤ n, with q̂ = +∞ if k > n. The set is C(x) = {y : s_θ̂(x, y) ≤ q̂} ∪ {argmax_y z_y(x)}.

Proposition 1′. If (X_1, Y_1), …, (X_{n+1}, Y_{n+1}) are exchangeable conditional on G = (D_tr, D_tune, U), then P(Y_{n+1} ∈ C(X_{n+1}) | G) ≥ k/(n+1) ≥ 1 − α. If, in addition, the scores are almost surely distinct, then without the top-1 rule the coverage is also < 1 − α + 1/(n+1).

Proof.

  1. Condition on G. Then θ̂ is fixed, and S_1, …, S_{n+1} are one fixed function applied to an exchangeable sequence, so they are exchangeable given G.
  2. If k > n, then q̂ = ∞ and coverage is 1.
  3. Otherwise, suppose S_{n+1} > S_(k). Then at least k of S_1, …, S_n are strictly smaller than S_{n+1}. So, under any tie-breaking, S_{n+1} is not among the k smallest of the n+1 values.
  4. Break ties by an independent uniform permutation. Let R be the rank of S_{n+1} among all n+1 values. By exchangeability, R is uniform on {1, …, n+1}.
  5. Hence P(S_{n+1} ≤ q̂ | G) ≥ P(R ≤ k | G) = k/(n+1).
  6. The event {S_{n+1} ≤ q̂} is contained in {Y_{n+1} ∈ C(X_{n+1})}, because C contains every label with score ≤ q̂, and the top-1 rule only adds labels.
  7. Taking expectations over G gives the marginal statement.
  8. Upper bound: with distinct scores, {S_{n+1} ≤ S_(k)} = {R ≤ k}, so the probability equals k/(n+1) < 1 − α + 1/(n+1). □

The proof places no restriction on how w is chosen. In particular, choosing w to minimise set size on D_tune is allowed. The price is that the n_tune points do not enter the quantile.

Proposition 2′. Assume h | y = c ~ N(μ_c, σ_c² I_D) with prior π_c, and write p_G(c | h) ∝ π_c σ_c^{−D} exp(−‖h − μ_c‖² / 2σ_c²).

  • (a) Among all set-valued rules with marginal coverage ≥ 1 − α, expected set size E|C(h)| is minimised by C*(h) = {y : p_G(y | h) ≥ λ_α}, with randomisation at the boundary. Equivalently, 1 − p_G(y | h) is the oracle-optimal conformity score.
  • (b) If σ_c ≡ σ, a single threshold on s(h, y) = ‖h − μ_y‖ / σ_y is a threshold on the class-conditional density p(h | y). If, in addition, π_c ≡ π, it is a threshold on the joint density p(h, y), and in that case the set minimises the volume Σ_y ∫ 1[y ∈ C(h)] dh at coverage 1 − α, not the expected number of labels.

Proof.

  • (a) Write the set as an indicator φ(h, y) ∈ [0, 1].
    • The problem is: minimise E|C| = ∫ Σ_y φ(h, y) p(h) dh, subject to ∫ Σ_y φ(h, y) p(h, y) dh ≥ 1 − α.
    • For λ ≥ 0, the Lagrangian integrand is Σ_y φ(h, y)[p(h) − λ p(h, y)] = p(h) Σ_y φ(h, y)[1 − λ p(y | h)].
    • It is minimised pointwise by φ = 1 exactly when p(y | h) > 1/λ.
    • Choosing λ so that the constraint is tight (randomising on the boundary) gives a feasible φ*. Its objective is no larger than that of any feasible φ, by the Neyman–Pearson argument.
    • This is Sadinle, Lei and Wasserman (2019, Thm 1) applied to the representation model.
  • (b) With σ_c ≡ σ, p(h | y) = (2πσ²)^{−D/2} exp(−s(h, y)² / 2), which is strictly decreasing in s.
    • So {s ≤ τ} = {p(h | y) ≥ λ(τ)}. With equal priors, p(h, y) = π p(h | y), so it is also a threshold on the joint density.
    • Repeating the argument of (a) with p(h) replaced by Lebesgue measure shows that a joint-density super-level set minimises Σ_y ∫ φ dh subject to the coverage constraint.
    • Since p(y | h) = p(h, y) / p(h) and p(h) varies with h, these sets differ from C* in general. □

Consequence: the Bayes-optimality claim of the submitted Proposition 2 is withdrawn. (b) is what the distance score actually achieves. We also evaluated (a) directly (REP-GDA / M-STC-GDA). On the synthetic benchmark it is 3.1–4.8% larger than the isotropic versions, because σ_c and π_c estimated from small training sets are noisy. On real encoders it is the best method on Qwen / Banking77 (9.78 vs LAC 13.59 at α = .05, B = 1000).

W3. Novelty given Zeng et al. and Teng et al. Both are relevant, and both are now discussed.

  • Zeng et al. analyse the coverage loss from tuning a score on calibration data. That is exactly the issue with our submitted protocol, and their bound (about 0.05 at n = 1000 with an 11-point grid) explains why the reused protocol's empirical coverage was close to nominal. The corrected method avoids the issue by a disjoint split.
  • Teng et al. (feature CP) build the conformal band in feature space via a label pre-image. They do not use class centroids or blend with a softmax score.

We agree that using representations in CP is not new (DkNN, NCP, trust-score CP and Mahalanobis-based scores precede us). We have narrowed the claim to the convex blend of a softmax rank score with a class-centroid distance, tuned under a valid split.

Q2. REP-CP. REP-CP is the endpoint w = 1 of our blend: s(x, y) = ‖h(x) − μ_y‖ / σ_y, with μ_y the training-set class mean and σ_y the mean distance of that class's training points to μ_y, plus standard split-CP calibration. M-STC is the blend (1−w)·RAPS_T + w·s/m̃, with w tuned. The different names follow the usual convention of naming an ablation endpoint.

The revision gives REP-CP a formal definition next to the M-STC score (Section 3.2). It also states clearly that the gain of M-STC over REP-CP is small: −0.9% at B = 1000 on the synthetic benchmark, and +0.3% (i.e., REP-CP is better) at B = 500. The large real-data advantage over REP-CP (−27.0% / −11.8% / −40.8% on RoBERTa / DeBERTa / Qwen) comes from the softmax term rescuing settings where the geometry alone is weak.

Q3. Other layers. Yes. We ran a per-layer study (B = 1000, 3 seeds × 50 re-splits) and a variant that selects the layer on D_tune (3 seeds × 100 re-splits).

  • RoBERTa / AG News, α = .05:
    • Geometry is useful only in the top quarter of the network. REP-CP set size is 3.7–4.0 at layers 0–4, 1.648 at layer 8, 1.185 at layer 11 and 1.250 at layer 12, against TS-RAPS 1.330.
    • M-STC's tuned w correctly goes to 0 at layers 0–4 (it falls back to TS-RAPS, 1.330) and reaches 1.186 at layer 11.
  • RoBERTa / Banking77, α = .05: the same shape, more pronounced. REP-CP is 76.7 (near-trivial) at layer 0, 31.7 at layer 8, 10.4 at layer 10 and 16.9 at layer 12. M-STC is 6.85 at layer 0 (w ≈ 0), 4.44 at layer 10 and 4.61 at layer 12 (TS-RAPS 6.84).
  • Synthetic 3-layer encoder: an intermediate layer is best once the encoder is trained on more data (n_train = 1000: 3.059 at layer 1 vs 3.322 at the last layer). At n_train = 200 the last layer is best.
  • Layer chosen on D_tune together with w (M-STC-LSEL). The quantile is still taken on D_cal only, so Proposition 1′ still applies. Mean set size, last layer → LSEL:
    • AG News α = .05: 1.219 → 1.188 (layers 10–11 chosen in 239 of 300 runs);
    • Banking77 α = .05: 4.679 → 4.550, and α = .10: 2.843 → 2.735 (layer 10 chosen in about 190/300 runs);
    • synthetic n_train = 1000, α = .05: 3.335 → 3.090, and n_train = 500: 4.948 → 4.580 (layer 1 chosen most often);
    • AG News α = .10: 1.015 → 1.017 (no gain).
    • Coverage stays valid (0.9000–0.9543).

The penultimate layers are usually slightly better than the last layer, and choosing the layer on D_tune captures this without breaking the guarantee. We add the study and the LSEL variant as a new appendix section.

I appreciate the author's efforts. In my view, however, this paper does not meet the standard for ICLR. Therefore, I will maintain my score.

Reviewer_x9fdOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
1
Presentation
2(−1)
Soundness
1
Read review
Paper summary

This paper proposes M-STC, which combines temperature-scaled RAPS with a class-normalized distance in the encoder's hidden representation. A validation split is used to select the blending weight, followed by conformal calibration on the full calibration set. The authors claim finite-sample coverage and improved set efficiency, evaluated on a synthetic sequence-classification task.

Critical strengths
  • The method is simple and does not require retraining the encoder.
  • The presentation is good.
Critical weaknesses
  1. Limited novelty. Earlier work already uses learned representations for conformal uncertainty quantification [1,2] and studies weighted combinations of conformity scores [3]. M-STC largely combines these existing ideas.

  2. Unsupported theoretical claims. Proposition 1 applies an exchangeability argument despite fitting score parameters on data reused for final calibration. Proposition 2 also omits class-dependent Gaussian density factors, so its Bayes-optimality claim is not established.

[1] Papernot, N., and McDaniel, P. (2018). Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning.

[2] Ghosh, S., Belkhouja, T., Yan, Y., and Doppa, J. R. (2023). Improving Uncertainty Quantification of Deep Classifiers via Neighborhood Conformal Prediction: Novel Algorithm and Theoretical Analysis. AAAI.

[3] Luo, R., and Zhou, Z. (2025). Conformity Score Averaging for Classification. ICML.

Questions for authors
  1. How does Proposition 1 apply when the temperature, normalization median, and blending weight depend on samples reused to compute the final conformal quantile?
  2. What is the methodological contribution beyond representation-based conformal methods and conformity-score averaging?
6 comments

Thank you for the careful review. The main changes:

  1. Valid guarantee. The submitted code fitted the temperature T, the median normaliser m̃ and the weight w on the same calibration set that gives the quantile, and used an interpolated quantile with a strict <. Prop. 1 did not cover that procedure. We now use a strict split:

    • T, m̃ and w are fitted on a tuning part D_tune (30% of the labelled budget B);
    • the quantile is the exact ⌈(n+1)(1−α)⌉-th order statistic on the disjoint D_cal;
    • sets use ≤.

    Prop. 1′, with a full proof, covers this. All tuned baselines use the same split, and all methods use the same B.

  2. Prop. 2 is withdrawn as an optimality claim and replaced by a corrected Prop. 2′.

  3. Novelty is stated narrowly. We drop "the first conformal score to use hidden representations". The contribution is a convex blend of a softmax rank score with a training-set class-centroid distance, with the blend weight tuned under a valid split.

  4. New experiments:

    • three fine-tuned encoders: RoBERTa-base, DeBERTa-v3-base, and Qwen2.5-1.5B with LoRA;
    • four datasets: AG News, DBpedia-14, Banking77, 20NG;
    • new baselines: NCP, DkNN, Trust-CP, CSA, atypicality-aware RAPS (AA-RAPS) and Conf-OT;
    • a layer study, and 100 re-splits per trained encoder.

Main results under the strict protocol

  • The paper's synthetic conclusions hold. At B = 1000, the mean size change of M-STC (submitted numbers in brackets) is:

    vsChange
    APS−41.9% [−44.3%]
    TS-RAPS−19.6% [−20.9%]
    RAPS−11.9% [−12.3%]
    LAC−8.9% [−7.0%]
    CSA−7.5% [−4.9%]

    M-STC has the smallest mean in 5 of 6 settings. It is smaller than every external baseline in all 12 settings (B ∈ {500, 1000}); against the best external baseline of each setting it is smaller by 7.5% on average at B = 1000 and 7.2% at B = 500.

  • Coverage is valid (up to Monte-Carlo error) in all 12 synthetic and 48 real groups. Conf-OT, by contrast, under-covers in 15 of the 48 real groups.

  • Real encoders. M-STC is smaller than AA-RAPS, Trust-CP, APS and NCP in 48, 47, 45 and 44 of the 48 groups.

    • On Banking77 (77 classes, about 13 training examples per class), it is the smallest valid method in all 8 RoBERTa and DeBERTa groups: 9.4–51.6% smaller than LAC, with much better worst-class coverage.
    • On the high-accuracy datasets (AG News, DBpedia, 20NG), the softmax scores LAC, CSA and Conf-OT remain very strong, and M-STC is close to them but not ahead.

W1 / Q2. Novelty relative to representation-based CP [1, 2] and score averaging [3]. We agree that both ingredients exist. DkNN and NCP use representations, and CSA learns convex weights over conformity scores. CSA, however, averages softmax-based scores (THR, APS, RANK) only. DkNN replaces the score with neighbour-label disagreement, and NCP localises the threshold rather than changing the score. We found no prior method that blends a softmax rank score with a class-centroid distance in hidden space.

This is a modest contribution, and we now state it as such. "First to use hidden representations" is removed.

Empirically, all three methods are now in the comparison under the same budget and split. Synthetic, B = 1000:

MethodM-STC size change
CSA−7.5%
NCP−26.7%
DkNN−38.7%

On the fine-tuned encoders, M-STC is smaller than NCP in 44/48 groups (−25.6% to −43.0% on average), and on Banking77 it is smaller than CSA in all 12 groups (e.g. −23.2% on RoBERTa and −46.1% on DeBERTa at α = .05, B = 1000). On the high-accuracy datasets CSA, like LAC, is very strong. Averaged over all 16 groups, M-STC vs CSA is +2.1% (RoBERTa), −6.5% (DeBERTa) and +11.1% (Qwen).

W2 / Q1. Proposition 1 with reused samples. It does not apply, and we no longer claim it does (see point 1 of the summary above). Under the corrected procedure, T, m̃ and w are fitted only on D_tune, and the quantile is the exact order statistic on D_cal. The full statement and proof (revised Section 3.3 and Appendix B):

Setting. D_tr (training), D_tune (size n_tune) and D_cal = {(X_i, Y_i)}{i=1..n} are disjoint, and (X{n+1}, Y_{n+1}) is the test point. The fitted parameters θ̂ = (μ, σ, T, m̃, w) are any measurable function of (D_tr, D_tune, U), where U is independent auxiliary randomness (e.g. the cross-fitting split). The score is s_θ̂(x, y) = (1−w)·RAPS_T(x, y) + w·‖h(x) − μ_y‖ / (σ_y m̃). Let k = ⌈(n+1)(1−α)⌉. The threshold is q̂ = S_(k), the k-th smallest of S_i = s_θ̂(X_i, Y_i), i ≤ n, with q̂ = +∞ if k > n. The set is C(x) = {y : s_θ̂(x, y) ≤ q̂} ∪ {argmax_y z_y(x)}.

Proposition 1′. If (X_1, Y_1), …, (X_{n+1}, Y_{n+1}) are exchangeable conditional on G = (D_tr, D_tune, U), then P(Y_{n+1} ∈ C(X_{n+1}) | G) ≥ k/(n+1) ≥ 1 − α. If, in addition, the scores are almost surely distinct, then without the top-1 rule the coverage is also < 1 − α + 1/(n+1).

Proof.

  1. Condition on G. Then θ̂ is fixed, and S_1, …, S_{n+1} are one fixed function applied to an exchangeable sequence, so they are exchangeable given G.
  2. If k > n, then q̂ = ∞ and coverage is 1.
  3. Otherwise, suppose S_{n+1} > S_(k). Then at least k of S_1, …, S_n are strictly smaller than S_{n+1}. So, under any tie-breaking, S_{n+1} is not among the k smallest of the n+1 values.
  4. Break ties by an independent uniform permutation. Let R be the rank of S_{n+1} among all n+1 values. By exchangeability, R is uniform on {1, …, n+1}.
  5. Hence P(S_{n+1} ≤ q̂ | G) ≥ P(R ≤ k | G) = k/(n+1).
  6. The event {S_{n+1} ≤ q̂} is contained in {Y_{n+1} ∈ C(X_{n+1})}, because C contains every label with score ≤ q̂, and the top-1 rule only adds labels.
  7. Taking expectations over G gives the marginal statement.
  8. Upper bound: with distinct scores, {S_{n+1} ≤ S_(k)} = {R ≤ k}, so the probability equals k/(n+1) < 1 − α + 1/(n+1). □

The proof places no restriction on how w is chosen. In particular, choosing w to minimise set size on D_tune is allowed. The price is that the n_tune points do not enter the quantile.

All methods are compared at the same total budget. Empirically, over 500 synthetic runs at B = 1000, mean coverage is 0.9501–0.9503 at α = .05 and 0.8995–0.9006 at α = .10.

W2. Proposition 2 omits class-dependent Gaussian factors. Correct. With h | y=c ~ N(μ_c, σ_c² I), the log-posterior contains −D log σ_c + log π_c, so the normalised distance does not order classes by posterior. There is also a second gap: even with equal σ_c and π_c, a single conformal threshold on the distance is a threshold on the class-conditional density, not on p(y | h). By Sadinle et al. (2019), the latter is what minimises expected set size.

We withdraw the optimality claim and replace it with the following (revised Section 3.4 and Appendix B).

Proposition 2′. Assume h | y = c ~ N(μ_c, σ_c² I_D) with prior π_c, and write p_G(c | h) ∝ π_c σ_c^{−D} exp(−‖h − μ_c‖² / 2σ_c²).

  • (a) Among all set-valued rules with marginal coverage ≥ 1 − α, expected set size E|C(h)| is minimised by C*(h) = {y : p_G(y | h) ≥ λ_α}, with randomisation at the boundary. Equivalently, 1 − p_G(y | h) is the oracle-optimal conformity score.
  • (b) If σ_c ≡ σ, a single threshold on s(h, y) = ‖h − μ_y‖ / σ_y is a threshold on the class-conditional density p(h | y). If, in addition, π_c ≡ π, it is a threshold on the joint density p(h, y), and in that case the set minimises the volume Σ_y ∫ 1[y ∈ C(h)] dh at coverage 1 − α, not the expected number of labels.

Proof.

  • (a) Write the set as an indicator φ(h, y) ∈ [0, 1].
    • The problem is: minimise E|C| = ∫ Σ_y φ(h, y) p(h) dh, subject to ∫ Σ_y φ(h, y) p(h, y) dh ≥ 1 − α.
    • For λ ≥ 0, the Lagrangian integrand is Σ_y φ(h, y)[p(h) − λ p(h, y)] = p(h) Σ_y φ(h, y)[1 − λ p(y | h)].
    • It is minimised pointwise by φ = 1 exactly when p(y | h) > 1/λ.
    • Choosing λ so that the constraint is tight (randomising on the boundary) gives a feasible φ*. Its objective is no larger than that of any feasible φ, by the Neyman–Pearson argument.
    • This is Sadinle, Lei and Wasserman (2019, Thm 1) applied to the representation model.
  • (b) With σ_c ≡ σ, p(h | y) = (2πσ²)^{−D/2} exp(−s(h, y)² / 2), which is strictly decreasing in s.
    • So {s ≤ τ} = {p(h | y) ≥ λ(τ)}. With equal priors, p(h, y) = π p(h | y), so it is also a threshold on the joint density.
    • Repeating the argument of (a) with p(h) replaced by Lebesgue measure shows that a joint-density super-level set minimises Σ_y ∫ φ dh subject to the coverage constraint.
    • Since p(y | h) = p(h, y) / p(h) and p(h) varies with h, these sets differ from C* in general. □

We also evaluated (a) directly as REP-GDA / M-STC-GDA. On the synthetic benchmark both are larger than their isotropic counterparts: REP-GDA by 3.1–3.2% vs REP-CP, and M-STC-GDA by 4.4–4.8% vs M-STC (B = 1000 / 500), because σ_c and π_c estimated from small training sets are noisy. On real encoders the picture is mixed. REP-GDA is the best method on Qwen / Banking77 (9.78 vs LAC 13.59 at α = .05, B = 1000), but on average it is 15.0% larger than LAC on RoBERTa and 3.4% larger on Qwen. We therefore do not claim the distance score is preferable in general. We report both, and the revised text motivates the distance by (b), not by optimality.

Reviewer_rdDfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new conformity score for conformal prediction under limited calibration data. It combines the post-softmax probability vector with a geometric signal, i.e., the per-class distance to training-set class centroids in the encoder's hidden representation, via a convex blend. The blend weight is selected on a hold-out slice of the calibration set to minimize prediction-set size subject to the specified coverage. Experiments on an LLM-style sequence-classification benchmark support the method's effectiveness.

Critical strengths
  1. The method is relatively simple and computationally inexpensive. It uses representations already produced by the encoder without additional model training.

  2. The work identifies the blind spot of existing scores (post-softmax only) and augments RAPS with class-conditional geometry to build smaller sets with the same coverage.

Critical weaknesses
  1. The novelty is relatively limited. The paper claims to be the first to exploit hidden representations, yet prior work [1, 2, 3, 4] has used feature-space similarity; direct comparison and discussion are missing.

  2. The stated coverage guarantee does not appear to apply to the algorithm as described. Proposition 1 requires the score parameters to be fixed independently of the final conformal calibration sample. In Algorithm 2, however, the temperature is fitted using the full calibration set before it is split; the representation-score normalization is also based on calibration scores; the blend weight is selected using a calibration holdout. This reuse means the calibration scores and the test score are not independent, violating the exchangeability assumption used in the proof.

  3. The evaluation is narrow. All main results use a synthetic task and a shallow, one-layer Transformer. The benchmark may be useful for controlled analysis, but it does not establish that the gains transfer to standard text-classification datasets, pretrained encoders, or actual LLM deployments.

  4. The class-conditional coverage trade-off is substantial. Although the paper acknowledges this issue, its appendix reports a worst-class coverage of 0.596 for M-STC at a nominal level of 0.95. This is a meaningful limitation for deployments where performance across classes matters.

References:

[1] Ghosh et al. "Improving uncertainty quantification of deep classifiers via neighborhood conformal prediction: Novel algorithm and theoretical analysis." AAAI. 2023.

[2] Yuksekgonul et al. "Beyond confidence: Reliable models should also consider atypicality." NeurIPS. 2023.

[3] Messoudi and Imoussaten. "Hybrid Adaptive Prediction Sets for Conformal Prediction." ICML Workshop. 2026.

[4] Silva-Rodríguez et al. "Conformal prediction for zero-shot models." CVPR. 2025.

Questions for authors

Please refer to the Critical Weaknesses.

5 comments

Thank you for the careful review. The main changes:

  1. Valid guarantee. The submitted code fitted the temperature T, the median normaliser m̃ and the weight w on the same calibration set that gives the quantile, and used an interpolated quantile with a strict <. Prop. 1 did not cover that procedure. We now use a strict split:

    • T, m̃ and w are fitted on a tuning part D_tune (30% of the labelled budget B);
    • the quantile is the exact ⌈(n+1)(1−α)⌉-th order statistic on the disjoint D_cal;
    • sets use ≤.

    Prop. 1′, with a full proof, covers this. All tuned baselines use the same split, and all methods use the same B.

  2. Prop. 2 is withdrawn as an optimality claim and replaced by a corrected Prop. 2′.

  3. Novelty is stated narrowly. We drop "the first conformal score to use hidden representations". The contribution is a convex blend of a softmax rank score with a training-set class-centroid distance, with the blend weight tuned under a valid split.

  4. New experiments:

    • three fine-tuned encoders: RoBERTa-base, DeBERTa-v3-base, and Qwen2.5-1.5B with LoRA;
    • four datasets: AG News, DBpedia-14, Banking77, 20NG;
    • new baselines: NCP, DkNN, Trust-CP, CSA, atypicality-aware RAPS (AA-RAPS) and Conf-OT;
    • a layer study, and 100 re-splits per trained encoder.

Main results under the strict protocol

  • The paper's synthetic conclusions hold. At B = 1000, the mean size change of M-STC (submitted numbers in brackets) is:

    vsChange
    APS−41.9% [−44.3%]
    TS-RAPS−19.6% [−20.9%]
    RAPS−11.9% [−12.3%]
    LAC−8.9% [−7.0%]
    CSA−7.5% [−4.9%]

    M-STC has the smallest mean in 5 of 6 settings. It is smaller than every external baseline in all 12 settings (B ∈ {500, 1000}); against the best external baseline of each setting it is smaller by 7.5% on average at B = 1000 and 7.2% at B = 500.

  • Coverage is valid (up to Monte-Carlo error) in all 12 synthetic and 48 real groups. Conf-OT, by contrast, under-covers in 15 of the 48 real groups.

  • Real encoders. M-STC is smaller than AA-RAPS, Trust-CP, APS and NCP in 48, 47, 45 and 44 of the 48 groups.

    • On Banking77 (77 classes, about 13 training examples per class), it is the smallest valid method in all 8 RoBERTa and DeBERTa groups: 9.4–51.6% smaller than LAC, with much better worst-class coverage.
    • On the high-accuracy datasets (AG News, DBpedia, 20NG), the softmax scores LAC, CSA and Conf-OT remain very strong, and M-STC is close to them but not ahead.

W1. Novelty; comparison with [1–4]. We agree. The "first to exploit hidden representations" claim is removed from the abstract, introduction and contributions. We read [1], [2] and [4] in full (for [3] only the abstract was accessible; see below), together with related papers, and added a comparison table to Related Work (uses h(x)? mechanism; combined with softmax? which parameters are fitted on which split; guarantee). Summary:

  • NCP [1] uses the representation to localise the threshold (kNN-weighted quantile); the score is unchanged.
  • Yuksekgonul et al. [2] use a Gaussian atypicality score for grouping (Mondrian cells over confidence × atypicality), not inside the score.
  • Conf-OT [4] adapts CLIP similarity logits transductively; it does not use encoder hidden states.
  • HAPS [3]: we could access only the abstract, and found no public code, so we could not run it. The abstract describes a multiplicative fusion of probabilities and latent similarities from a frozen foundation model, which is close in spirit to ours. We cite it as concurrent work (July 2026) and do not claim a difference we cannot verify.

M-STC changes the score itself, by a convex blend with a training-centroid distance. That is the remaining, narrower contribution.

On empirical comparison: the new experiments include NCP (our re-implementation: kNN-weighted quantile in the representation space on a TS-APS base score, with λ and the nominal level tuned on D_tune), DkNN, CSA and Trust-CP, and now also [2] and [4], all on the same budget and splits.

  • [2] as a conformal method (AA-RAPS): Mondrian thresholds over confidence × atypicality cells. Atypicality is the tied-covariance Gaussian log-density of h on training data. Cell edges come from D_tune and per-cell thresholds from D_cal. We use the 6 × 6 grid and a coarser 3 × 3 grid, which is fairer at these budgets.
  • [4] (Conf-OT): Sinkhorn adaptation of the softmax over D_cal ∪ test (τ = 1, 3 iterations, as in the paper), label marginal from the D_cal labels, then LAC. It is transductive: it needs the test batch, and the calibration labels shape the test probabilities. The paper gives no formal coverage theorem.

M-STC size change:

Settingvs NCPvs DkNN
Synthetic, B = 1000−26.7%−38.7%
RoBERTa (16 groups)−39.5%−25.8%
DeBERTa (16 groups)−25.6%−24.2%
Qwen (16 groups)−43.0%−3.4%
Settingvs AA-RAPS (6×6)vs AA-RAPS (3×3)vs Conf-OT
Synthetic (12 groups, B ∈ {500, 1000})−34.3%−20.4%−9.7%
RoBERTa (16 groups)−64.1%−30.1%+3.4%
DeBERTa (16 groups)−58.7%−27.4%−1.1%
Qwen (16 groups)−56.9%−19.6%+18.2%
  • AA-RAPS is valid but conservative: with 9–36 cells, each cell has few calibration points, and its mean coverage is above nominal in every group (up to 0.998 at B = 500).
  • Conf-OT is competitive in size, and is the best method in 9 of the 48 real groups (8 on 20NG, 1 on AG News, by small margins). But it under-covers, mainly on Banking77: mean coverage is below 1 − α − 0.005 in 6 / 5 / 4 of the 16 groups (RoBERTa / DeBERTa / Qwen; all 12 Banking77 groups plus three 20NG groups at B = 500), worst 0.831 at α = .10. Calibrating it on all B points instead of D_cal reduces but does not remove this (worst 0.854). Strict M-STC has no group below that line.
  • On Banking77 with RoBERTa (B = 1000, α = .05): M-STC 4.679 (coverage 0.954), Conf-OT 5.836 (0.938), LAC 6.102 (0.951).

W2. Calibration reuse. You are right. See point 1 of the summary above. Under the strict split, Proposition 1′ gives P(Y ∈ C(X)) ≥ ⌈(n+1)(1−α)⌉/(n+1) for any rule that picks w from D_tune. The proof conditions on (D_train, D_tune), so θ̂ is fixed, and uses exchangeability of the D_cal scores and the test score.

Empirically, over 500 synthetic runs (5 encoders × 100 re-splits, 4,000 test points each):

  • At B = 1000, mean coverage is 0.9501–0.9503 at α = .05 and 0.8995–0.9006 at α = .10.
  • The fraction of runs below 1−α−0.01 (12–14% and 19–21%) matches what an exact split-CP procedure with n_cal = 700 gives in simulation (12.9% and 19.7%).

We also report the submitted reuse protocol as "M-STC-ORIG". Its empirical coverage is close to nominal at these sample sizes, which agrees with the tuning-bias bound of Zeng et al. (about 0.05 at n = 1000 and 11 grid values, and smaller in practice). We do not claim a guarantee for it.

W3. Narrow evaluation. We add three fine-tuned encoders on AG News, DBpedia-14, Banking77 and 20NG (Qwen is our closest proxy for an LLM deployment; we did not evaluate generative or multiple-choice QA set-ups): RoBERTa-base and DeBERTa-v3-base (full fine-tuning), and Qwen2.5-1.5B (LoRA, r = 16, last-token state). The setup: n_train = 1000; a held-out pool of 12,000; labelled budget B ∈ {500, 1000}; 4,000 test points; 3 seeds × 100 re-splits.

B = 1000, α = .05, mean set size, M-STC / LAC (best method among all compared in brackets when it is neither; bold = best):

DatasetRoBERTaDeBERTaQwen (LoRA)
AG News1.219 / 1.1821.449 / 1.2571.518 / 1.222
DBpedia1.000 / 1.0001.018 / 1.0001.087 / 1.000
20NG7.417 / 5.5778.625 / 7.271 [Conf-OT 7.222]9.349 / 7.736 [Conf-OT 7.563]
Banking774.679 / 6.10215.007 / 27.62313.230 / 13.588 [REP-GDA 9.782]

Summary over the 48 groups (3 encoders × 4 datasets × 2 α × 2 B):

  • Validity: strict M-STC is valid in every group (0.9010–0.9589; DBpedia 0.978–0.987).
  • Against the representation-based and softmax-rank methods, M-STC is clearly better:
    • smaller than AA-RAPS in 48/48 groups, Trust-CP 47/48, TS-APS 46/48, APS 45/48, NCP 44/48, REP-CP 40/48 and TS-RAPS 39/48;
    • average change −37.9% to −47.5% vs Trust-CP, −25.6% to −43.0% vs NCP, −32.4% to −46.5% vs APS, −3.4% to −25.8% vs DkNN, −7.7% to −16.0% vs TS-RAPS (range over the three encoders).
  • Many-class, low-data regime (Banking77, 77 classes, about 13 training examples per class):
    • M-STC is the smallest valid method in all 8 RoBERTa and DeBERTa groups;
    • vs LAC it is −9.4% to −23.3% (RoBERTa) and −39.8% to −51.6% (DeBERTa);
    • it also has much better worst-class coverage (see W4).
    • On Qwen-LoRA the Gaussian-posterior score REP-GDA (Prop. 2′(a)) is best, and M-STC is still smaller than LAC in 3 of the 4 groups.
  • High-accuracy datasets (AG News, DBpedia, 20NG): the plain softmax scores LAC, CSA and Conf-OT are the strongest methods; LAC has the smallest (or tied) mean in 27 of these 36 groups. Over all 16 groups per encoder, M-STC vs LAC is +3.8% (RoBERTa), −5.1% (DeBERTa) and +13.1% (Qwen).

The reason is visible in the components. M-STC's softmax endpoint (temperature-scaled RAPS) is weaker than LAC on a well-trained fine-tuned head. The distance term improves on that endpoint in 39 of 48 groups, and turns into a clear gain over every softmax score when the softmax is unreliable, as on Banking77.

We checked that the gap is not an implementation artefact. The scores, centroid estimation, median normalisation and blend match the submitted code line by line. Using the classifier-head input instead of the CLS state, or σ_c estimated on held-out data, changes sizes only marginally and in both directions.

Two D_tune-only refinements, both covered by Prop. 1′, close most of the remaining gap:

  • choosing the base score (LAC or RAPS) on D_tune gives −2.4% / −10.0% / +0.8% vs LAC on RoBERTa / DeBERTa / Qwen, instead of +3.8% / −5.1% / +13.1%;
  • choosing the layer on D_tune together with w gives further gains, e.g. Banking77 α = .05: 4.679 → 4.550, AG News α = .05: 1.219 → 1.188, with coverage still valid.

They were designed after seeing the RoBERTa results, so we report them as exploratory.

We will state in the paper that LAC is a strong default on well-trained text classifiers, and that M-STC's geometric term pays off in many-class, low-data regimes and against other representation-based CP methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.