acceptodds
Under review as a conference paper at ICLR 2027

In-Context Full Conformal Prediction: Validity and Efficiency

Abstract

In-context learning enables pretrained models to adapt to new tasks without parameter updates, but rigorous uncertainty quantification for in-context predictions remains poorly understood. Conformal prediction provides finite-sample distribution-free coverage, yet full conformal prediction can be computationally expensive because it may require candidate-dependent model refitting. We develop an in-context full conformal procedure that replaces these refitting steps with forward evaluations of a fixed pretrained predictor. Under exchangeability and a natural context-order invariance condition, we establish finite-sample marginal coverage. We further characterize the statistical efficiency of the resulting prediction sets, showing that it depends jointly on prediction risk and prompt stability. As a concrete nonparametric instantiation, we show that standard softmax Transformers can approximate Gaussian local polynomial regression and achieve the minimax-optimal prediction rate over Hölder classes. Empirically, we show that replacing candidate-wise refitting with forward evaluations of a frozen in-context predictor substantially reduces online computation while maintaining comparable empirical coverage and set length. Together, these results establish in-context learning as a computationally efficient approach to full conformal prediction with pretrained predictors.

Primary area:
learning theory
Keywords:
in-context learning, conformal prediction, uncertainty quantification, transformers, nonparametric regression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RnqqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses frozen in-context predictors for full conformal prediction, establishing marginal coverage and efficiency bounds based on prediction risk and prompt stability. It also develops a minimax-rate softmax Transformer construction. Experiments on six synthetic regression families show near-nominal coverage, competitive set lengths, and speedups over iterative refitting baselines.

Critical strengths
  1. The paper provides rigorous coverage and oracle-efficiency guarantees, clarifying how prediction accuracy and prompt stability jointly determine prediction-set quality.
  2. The constructive softmax Transformer approximation and minimax-rate analysis provide a concrete theoretical foundation beyond assuming an accurate in-context predictor.
Critical weaknesses
  1. Prior work already studies ICL-based full conformal prediction without repeated training, leaving the methodological novelty insufficiently established.

  2. Minimax-optimal Transformer regression guarantees and generalizations with respect to prompt stability already exist, and the significance of extending these results to softmax attention needs clearer justification.

  3. Comparisons omit existing ICL-conformal methods and split conformal or jackknife+ using the same frozen predictor, leaving the proposed procedure’s added benefit unclear.

  4. Evaluation is restricted to synthetic regression tasks, limiting evidence of practical utility on real datasets or with existing pretrained models.

  5. The largest speedups are against iterative MLP refitting, while structured KRR is slightly faster, indicating that the computational advantage depends substantially on the baseline.

Questions for authors
  1. (W1) Prior work already investigates conformal prediction with in-context learning [1,2,3,4], and the marginal validity guarantee follows from the standard conformal rank argument under permutation symmetry, limiting its novelty.

  2. Conformal validity generally does not require conformal-specific training of the underlying predictor. Thus, presenting frozen forward evaluations with retained finite-sample validity as a standalone contribution appears overstated unless the authors clarify the additional methodological or theoretical challenge addressed.

  3. (W2) [5] already studies prompt stability and generalization, while [6] establishes Transformer approximation of local polynomial regression with excess-risk guarantees. The additional theoretical contribution therefore needs clearer justification. Moreover, the construction alone does not explain when using a pretrained Transformer is preferable to directly applying local polynomial regression under the assumed Hölder model.

  4. The efficiency analysis builds on classical oracle-approximation results, including the second-order effect of prediction error under symmetric noise. The paper should clarify the substantive advance provided by its leave-one-out dependence analysis and stability conditions.

  5. (W3) Comparisons with existing ICL-conformal methods and split conformal or jackknife+ using the same predictor are missing, leaving the trade-off between computational cost and statistical efficiency insufficiently assessed.

  6. (W4) Evaluation covers six synthetic regression families but no real-world datasets, leaving the practical applicability of the method insufficiently demonstrated. Given the broader flexibility of pretrained Transformers, particularly for text data, I am not convinced that these experiments adequately demonstrate the practical benefits of the proposed approach.

  7. The comparisons do not adequately isolate the source of improvement: ICL benefits from extensive auxiliary-task pretraining, whereas the refitting baselines learn from the current task alone. Runtime comparisons also exclude pretraining costs and, for some baselines, use different hardware. Although these choices are disclosed, matched-pretraining and same-predictor comparisons are needed to distinguish the benefits of the conformal procedure from those of the predictor and implementation.

  8. The comparisons use different score constructions: ICL-FCP uses leave-one-out residuals, whereas the refitting baselines use in-sample residuals from the augmented dataset. This confounds the effect of the predictor with that of the conformal score construction.

Please clarify if I have misunderstood the distinctions from prior work or the scope of the results. My current assessment is primarily driven by the limited theoretical novelty and insufficient empirical evidence of the method’s practical advantages. I would be happy to raise my score if these concerns are adequately addressed.

[1] Huang, Zhe, et al. "From predictions to confidence intervals: an empirical study of conformal prediction methods for in-context learning." arXiv preprint arXiv:2504.15722 (2025).
[2] Huang, Xiao Shi, et al. "Unifying Conformal Language Tasks with In-Context Ensembles." arXiv preprint arXiv:2609.03005 (2026).
[3] Deng, Weicao, et al. "Optimizing In-Context Learning for Efficient Full Conformal Prediction." IEEE Signal Processing Letters 33 (2025): 311-315.
[4] Pantelidis, Ippokratis, Korbinian Randl, and Aron Henriksson. "Efficient text classification with conformal in-context learning." arXiv preprint arXiv:2512.05732 (2025).
[5] Li, Yingcong, et al. "Transformers as algorithms: Generalization and stability in in-context learning." International conference on machine learning. PMLR, 2023.
[6] Ching, Michelle, et al. "Efficient and Minimax Optimal In-context Nonparametric Regression with Transformers." arXiv preprint arXiv:2601.15014 (2026).

Reviewer_KTKZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper constructs full conformal prediction sets using leave-one-out predictions of a frozen in-context model. It establishes marginal coverage under context-order invariance and relates set efficiency to prediction risk and two prompt-stability conditions. A constructive softmax-Transformer approximation of Gaussian local polynomial regression supplies nonparametric risk and stability guarantees for a specific comparator. Synthetic regression experiments assess coverage, set length, numerical approximation, and online computation.

Critical strengths
  1. The separation of validity and efficiency is useful. Theorem 1 identifies the symmetry required for the score construction, while Theorem 2 distinguishes prediction error, dependence between leave-one-out scores, and sensitivity to the candidate response. The second-order dependence of interval length on prediction risk under symmetric noise is particularly informative.
  2. The constructive analysis and numerical diagnostics are substantial. The softmax construction includes explicit approximation and complexity bounds, and Appendix C verifies stability for that construction. The experiments distinguish direct-rank coverage from grid coverage and examine resolution, endpoint expansion, optimization budgets, and transfer to a held-out function family.
Critical weaknesses
  1. The efficiency theory remains only partially connected to the trained method. Theorem 4 bounds the prediction risk of an empirical risk minimizer, but Corollary 1 assumes the prompt-stability rates needed for conformal efficiency. Appendix C proves those rates for the explicitly constructed local-polynomial comparator, rather than the learned predictor. The experiments do not measure either stability quantity or convergence to the oracle set. Although this distinction is acknowledged, the central question of when ordinary pretraining produces an efficient full conformal predictor remains insufficiently resolved (Section 4.4; Appendix C.2).
  2. The comparisons do not isolate the benefit of full conformal inference with a pretrained model. A split-conformal or jackknife+-type comparison using the same frozen predictor is absent. Moreover, the strongest comparison on the same device appears in Appendix G.5: RBF-KRR-FCP takes 10.05 ms versus 11.04 ms for ICL-FCP, with mean lengths of 1.843 and 1.624, respectively. This useful tradeoff materially narrows the computational advantage suggested by the main table. A common benchmark including these alternatives is needed to establish the practical contribution.
Questions for authors
  1. Can the candidate-response and noise-replacement sensitivities in Assumption 1 be measured for the trained checkpoints as context size increases? Do they support the rates assumed in Corollary 1 and explain variation in set length?
  2. How does ICL-FCP compare with split conformal and a jackknife+-type procedure using the same frozen checkpoint and total number of observations? Please report coverage, set length, and latency under a common protocol that also includes RBF-KRR.
  3. Can a controlled experiment test Theorem 2 under its bounded, symmetric, homoskedastic noise model, reporting prediction risk and deviation from the known oracle interval? The current mixture with Gaussian and heteroskedastic noise does not directly test those efficiency guarantees.
Reviewer_a31QOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies full conformal prediction with a frozen in-context predictor, replacing candidate-dependent model fitting with forward evaluations. It establishes finite-sample marginal coverage when the predictor is invariant to the ordering of context examples. The paper then constructs softmax Transformers that approximate Gaussian local polynomial regression, establishes a minimax-rate excess-risk bound for an idealized pretrained ERM, and proves stability and conformal efficiency for the explicitly constructed predictor. Experiments on six synthetic regression families are conducted to evaluate the theoretical results.

Critical strengths
  1. The distinction between validity and efficiency is valuable. The analysis explains why accurate prediction alone does not ensure efficient conformal sets and identifies the additional roles of candidate-response sensitivity and dependence among leave-one-out scores.

  2. The softmax construction provides a concrete theoretical example. In particular, Appendix C.2 verifies stability and conformal efficiency for the explicitly constructed predictor, rather than leaving all efficiency conditions unverified.

  3. The experiments are informative, showing that valid coverage can coexist with reduced efficiency under task-prior mismatch.

Critical weaknesses
  1. Some recent closely related advances are not fully discussed. Frozen ICL-based full conformal prediction already appears in [1], while Deng et al. [2] study permutation-invariant Transformers and conformal-aware training. Ching et al. [3] establish Transformer-based local-polynomial approximation and minimax rates. The strongest contributions here appear to be the oracle-efficiency analysis and softmax realization, rather than the basic procedure or rank-based coverage guarantee.

  2. The theory does not fully explain the trained model’s efficiency. Theorem 4 bounds the prediction risk of an idealized ERM, but does not establish its prompt stability. Corollary 1 therefore imposes additional stability conditions, as the paper explicitly acknowledges. The results in Appendix C.2 concern an explicitly constructed predictor, which need not coincide with either that ERM or the AdamW-trained model. This limits how directly the theory explains the empirical set efficiency. Moreover, the efficiency analysis assumes bounded, input-independent symmetric additive noise, whereas the experiments use Gaussian noise and include heteroskedastic tasks.

  3. The comparisons do not isolate full conformalization from pretraining benefits. A pretrained ICL model is compared primarily with predictors fitted separately on each task. This demonstrates an operating-point comparison, but combines the effects of the pretraining prior, predictor architecture, and conformal construction. The held-out-family experiment is useful, but remains limited to synthetic regression and relatively short contexts; real-data evaluation with larger-context measurements would be helpful.

References:

[1] Huang et al. "From predictions to confidence intervals: an empirical study of conformal prediction methods for in-context learning." Proceedings of the 7th Symposium on Advances in Approximate Bayesian Inference, 2025.

[2] Deng et al. "Optimizing In-Context Learning for Efficient Full Conformal Prediction." https://arxiv.org/abs/2509.01840.

[3] Ching et al. "Efficient and Minimax Optimal In-context Nonparametric Regression with Transformers." ICML 2026.

Questions for authors

Please refer to the Critical Weaknesses.

1 comment

Thanks for the authors' clarifications. My concerns have been addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.