- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes M-STC, an improved method for conformal prediction designed to produce smaller prediction sets when calibration data is limited. The method combines Softmax-based scores with class-conditional distances derived from the model’s hidden representations, using a separate calibration set to determine the weights for integrating these two components.
Critical strengths
-
This issue is importance; in constrained calibration scenarios, it presents a very real, practical problem.
-
The results appear good within the scenario they constructed.
Critical weaknesses
-
Poor writing. Many strange writings, such as "Across 18 cells against nine baselines (2017–2026),"
-
The theorem proof is provided only as a Sketch; a formal proof is not included.
-
This idea is not novel; it appears in a great many papers, such as [1] and [2].
[1] Zeng, Hao, et al. "Parametric scaling law of tuning bias in conformal prediction." arXiv preprint arXiv:2502.03023 (2025).
[2] Teng, Jiaye, et al. "Predictive inference with feature conformal prediction." arXiv preprint arXiv:2210.00173 (2022).
Questions for authors
-
Can the authors provide a formal proof for the main theoretical claims presented in the paper?
-
REP-CP is mentioned only briefly, and its methodological formulation is not formally articulated. What is the relationship between REP-CP, the current study, and M-STC? Why are they referred to by different names?
-
Have the authors explored using representations extracted from other layers of the encoder?
4 comments
W2 / Q1. Formal proofs. Both proofs are given in full below and will replace the sketches (Appendix B).
Setting. D_tr (training), D_tune (size n_tune) and D_cal = {(X_i, Y_i)}{i=1..n} are disjoint, and (X{n+1}, Y_{n+1}) is the test point. The fitted parameters θ̂ = (μ, σ, T, m̃, w) are any measurable function of (D_tr, D_tune, U), where U is independent auxiliary randomness (e.g. the cross-fitting split). The score is s_θ̂(x, y) = (1−w)·RAPS_T(x, y) + w·‖h(x) − μ_y‖ / (σ_y m̃). Let k = ⌈(n+1)(1−α)⌉. The threshold is q̂ = S_(k), the k-th smallest of S_i = s_θ̂(X_i, Y_i), i ≤ n, with q̂ = +∞ if k > n. The set is C(x) = {y : s_θ̂(x, y) ≤ q̂} ∪ {argmax_y z_y(x)}.
Proposition 1′. If (X_1, Y_1), …, (X_{n+1}, Y_{n+1}) are exchangeable conditional on G = (D_tr, D_tune, U), then P(Y_{n+1} ∈ C(X_{n+1}) | G) ≥ k/(n+1) ≥ 1 − α. If, in addition, the scores are almost surely distinct, then without the top-1 rule the coverage is also < 1 − α + 1/(n+1).
Proof.
- Condition on G. Then θ̂ is fixed, and S_1, …, S_{n+1} are one fixed function applied to an exchangeable sequence, so they are exchangeable given G.
- If k > n, then q̂ = ∞ and coverage is 1.
- Otherwise, suppose S_{n+1} > S_(k). Then at least k of S_1, …, S_n are strictly smaller than S_{n+1}. So, under any tie-breaking, S_{n+1} is not among the k smallest of the n+1 values.
- Break ties by an independent uniform permutation. Let R be the rank of S_{n+1} among all n+1 values. By exchangeability, R is uniform on {1, …, n+1}.
- Hence P(S_{n+1} ≤ q̂ | G) ≥ P(R ≤ k | G) = k/(n+1).
- The event {S_{n+1} ≤ q̂} is contained in {Y_{n+1} ∈ C(X_{n+1})}, because C contains every label with score ≤ q̂, and the top-1 rule only adds labels.
- Taking expectations over G gives the marginal statement.
- Upper bound: with distinct scores, {S_{n+1} ≤ S_(k)} = {R ≤ k}, so the probability equals k/(n+1) < 1 − α + 1/(n+1). □
The proof places no restriction on how w is chosen. In particular, choosing w to minimise set size on D_tune is allowed. The price is that the n_tune points do not enter the quantile.
Proposition 2′. Assume h | y = c ~ N(μ_c, σ_c² I_D) with prior π_c, and write p_G(c | h) ∝ π_c σ_c^{−D} exp(−‖h − μ_c‖² / 2σ_c²).
- (a) Among all set-valued rules with marginal coverage ≥ 1 − α, expected set size E|C(h)| is minimised by C*(h) = {y : p_G(y | h) ≥ λ_α}, with randomisation at the boundary. Equivalently, 1 − p_G(y | h) is the oracle-optimal conformity score.
- (b) If σ_c ≡ σ, a single threshold on s(h, y) = ‖h − μ_y‖ / σ_y is a threshold on the class-conditional density p(h | y). If, in addition, π_c ≡ π, it is a threshold on the joint density p(h, y), and in that case the set minimises the volume Σ_y ∫ 1[y ∈ C(h)] dh at coverage 1 − α, not the expected number of labels.
Proof.
- (a) Write the set as an indicator φ(h, y) ∈ [0, 1].
- The problem is: minimise E|C| = ∫ Σ_y φ(h, y) p(h) dh, subject to ∫ Σ_y φ(h, y) p(h, y) dh ≥ 1 − α.
- For λ ≥ 0, the Lagrangian integrand is Σ_y φ(h, y)[p(h) − λ p(h, y)] = p(h) Σ_y φ(h, y)[1 − λ p(y | h)].
- It is minimised pointwise by φ = 1 exactly when p(y | h) > 1/λ.
- Choosing λ so that the constraint is tight (randomising on the boundary) gives a feasible φ*. Its objective is no larger than that of any feasible φ, by the Neyman–Pearson argument.
- This is Sadinle, Lei and Wasserman (2019, Thm 1) applied to the representation model.
- (b) With σ_c ≡ σ, p(h | y) = (2πσ²)^{−D/2} exp(−s(h, y)² / 2), which is strictly decreasing in s.
- So {s ≤ τ} = {p(h | y) ≥ λ(τ)}. With equal priors, p(h, y) = π p(h | y), so it is also a threshold on the joint density.
- Repeating the argument of (a) with p(h) replaced by Lebesgue measure shows that a joint-density super-level set minimises Σ_y ∫ φ dh subject to the coverage constraint.
- Since p(y | h) = p(h, y) / p(h) and p(h) varies with h, these sets differ from C* in general. □
Consequence: the Bayes-optimality claim of the submitted Proposition 2 is withdrawn. (b) is what the distance score actually achieves. We also evaluated (a) directly (REP-GDA / M-STC-GDA). On the synthetic benchmark it is 3.1–4.8% larger than the isotropic versions, because σ_c and π_c estimated from small training sets are noisy. On real encoders it is the best method on Qwen / Banking77 (9.78 vs LAC 13.59 at α = .05, B = 1000).
W3. Novelty given Zeng et al. and Teng et al. Both are relevant, and both are now discussed.
- Zeng et al. analyse the coverage loss from tuning a score on calibration data. That is exactly the issue with our submitted protocol, and their bound (about 0.05 at n = 1000 with an 11-point grid) explains why the reused protocol's empirical coverage was close to nominal. The corrected method avoids the issue by a disjoint split.
- Teng et al. (feature CP) build the conformal band in feature space via a label pre-image. They do not use class centroids or blend with a softmax score.
We agree that using representations in CP is not new (DkNN, NCP, trust-score CP and Mahalanobis-based scores precede us). We have narrowed the claim to the convex blend of a softmax rank score with a class-centroid distance, tuned under a valid split.
Q2. REP-CP. REP-CP is the endpoint w = 1 of our blend: s(x, y) = ‖h(x) − μ_y‖ / σ_y, with μ_y the training-set class mean and σ_y the mean distance of that class's training points to μ_y, plus standard split-CP calibration. M-STC is the blend (1−w)·RAPS_T + w·s/m̃, with w tuned. The different names follow the usual convention of naming an ablation endpoint.
The revision gives REP-CP a formal definition next to the M-STC score (Section 3.2). It also states clearly that the gain of M-STC over REP-CP is small: −0.9% at B = 1000 on the synthetic benchmark, and +0.3% (i.e., REP-CP is better) at B = 500. The large real-data advantage over REP-CP (−27.0% / −11.8% / −40.8% on RoBERTa / DeBERTa / Qwen) comes from the softmax term rescuing settings where the geometry alone is weak.
Q3. Other layers. Yes. We ran a per-layer study (B = 1000, 3 seeds × 50 re-splits) and a variant that selects the layer on D_tune (3 seeds × 100 re-splits).
- RoBERTa / AG News, α = .05:
- Geometry is useful only in the top quarter of the network. REP-CP set size is 3.7–4.0 at layers 0–4, 1.648 at layer 8, 1.185 at layer 11 and 1.250 at layer 12, against TS-RAPS 1.330.
- M-STC's tuned w correctly goes to 0 at layers 0–4 (it falls back to TS-RAPS, 1.330) and reaches 1.186 at layer 11.
- RoBERTa / Banking77, α = .05: the same shape, more pronounced. REP-CP is 76.7 (near-trivial) at layer 0, 31.7 at layer 8, 10.4 at layer 10 and 16.9 at layer 12. M-STC is 6.85 at layer 0 (w ≈ 0), 4.44 at layer 10 and 4.61 at layer 12 (TS-RAPS 6.84).
- Synthetic 3-layer encoder: an intermediate layer is best once the encoder is trained on more data (n_train = 1000: 3.059 at layer 1 vs 3.322 at the last layer). At n_train = 200 the last layer is best.
- Layer chosen on D_tune together with w (M-STC-LSEL). The quantile is still taken on D_cal only, so Proposition 1′ still applies. Mean set size, last layer → LSEL:
- AG News α = .05: 1.219 → 1.188 (layers 10–11 chosen in 239 of 300 runs);
- Banking77 α = .05: 4.679 → 4.550, and α = .10: 2.843 → 2.735 (layer 10 chosen in about 190/300 runs);
- synthetic n_train = 1000, α = .05: 3.335 → 3.090, and n_train = 500: 4.948 → 4.580 (layer 1 chosen most often);
- AG News α = .10: 1.015 → 1.017 (no gain).
- Coverage stays valid (0.9000–0.9543).
The penultimate layers are usually slightly better than the last layer, and choosing the layer on D_tune captures this without breaking the guarantee. We add the study and the LSEL variant as a new appendix section.
I appreciate the author's efforts. In my view, however, this paper does not meet the standard for ICLR. Therefore, I will maintain my score.
Thank you for the careful review. The main changes:
Valid guarantee. The submitted code fitted the temperature T, the median normaliser m̃ and the weight w on the same calibration set that gives the quantile, and used an interpolated quantile with a strict
<. Prop. 1 did not cover that procedure. We now use a strict split:≤.Prop. 1′, with a full proof, covers this. All tuned baselines use the same split, and all methods use the same B.
Prop. 2 is withdrawn as an optimality claim and replaced by a corrected Prop. 2′.
Novelty is stated narrowly. We drop "the first conformal score to use hidden representations". The contribution is a convex blend of a softmax rank score with a training-set class-centroid distance, with the blend weight tuned under a valid split.
New experiments:
Main results under the strict protocol
The paper's synthetic conclusions hold. At B = 1000, the mean size change of M-STC (submitted numbers in brackets) is:
M-STC has the smallest mean in 5 of 6 settings. It is smaller than every external baseline in all 12 settings (B ∈ {500, 1000}); against the best external baseline of each setting it is smaller by 7.5% on average at B = 1000 and 7.2% at B = 500.
Coverage is valid (up to Monte-Carlo error) in all 12 synthetic and 48 real groups. Conf-OT, by contrast, under-covers in 15 of the 48 real groups.
Real encoders. M-STC is smaller than AA-RAPS, Trust-CP, APS and NCP in 48, 47, 45 and 44 of the 48 groups.
W1. Writing; "18 cells against nine baselines (2017–2026)". Thank you. "18 cells" meant 6 settings (3 training sizes × 2 α) × 3 seeds. The percentages averaged the seeds within each setting and then averaged the 6 settings; the win counts used the 18 runs. The revision states each aggregation explicitly, and drops "(2017–2026)" and similar phrases. The new tables report 6 settings, each over 5 trained encoders × 100 calibration/test re-splits.