- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper uses frozen in-context predictors for full conformal prediction, establishing marginal coverage and efficiency bounds based on prediction risk and prompt stability. It also develops a minimax-rate softmax Transformer construction. Experiments on six synthetic regression families show near-nominal coverage, competitive set lengths, and speedups over iterative refitting baselines.
Critical strengths
- The paper provides rigorous coverage and oracle-efficiency guarantees, clarifying how prediction accuracy and prompt stability jointly determine prediction-set quality.
- The constructive softmax Transformer approximation and minimax-rate analysis provide a concrete theoretical foundation beyond assuming an accurate in-context predictor.
Critical weaknesses
-
Prior work already studies ICL-based full conformal prediction without repeated training, leaving the methodological novelty insufficiently established.
-
Minimax-optimal Transformer regression guarantees and generalizations with respect to prompt stability already exist, and the significance of extending these results to softmax attention needs clearer justification.
-
Comparisons omit existing ICL-conformal methods and split conformal or jackknife+ using the same frozen predictor, leaving the proposed procedure’s added benefit unclear.
-
Evaluation is restricted to synthetic regression tasks, limiting evidence of practical utility on real datasets or with existing pretrained models.
-
The largest speedups are against iterative MLP refitting, while structured KRR is slightly faster, indicating that the computational advantage depends substantially on the baseline.
Questions for authors
-
(W1) Prior work already investigates conformal prediction with in-context learning [1,2,3,4], and the marginal validity guarantee follows from the standard conformal rank argument under permutation symmetry, limiting its novelty.
-
Conformal validity generally does not require conformal-specific training of the underlying predictor. Thus, presenting frozen forward evaluations with retained finite-sample validity as a standalone contribution appears overstated unless the authors clarify the additional methodological or theoretical challenge addressed.
-
(W2) [5] already studies prompt stability and generalization, while [6] establishes Transformer approximation of local polynomial regression with excess-risk guarantees. The additional theoretical contribution therefore needs clearer justification. Moreover, the construction alone does not explain when using a pretrained Transformer is preferable to directly applying local polynomial regression under the assumed Hölder model.
-
The efficiency analysis builds on classical oracle-approximation results, including the second-order effect of prediction error under symmetric noise. The paper should clarify the substantive advance provided by its leave-one-out dependence analysis and stability conditions.
-
(W3) Comparisons with existing ICL-conformal methods and split conformal or jackknife+ using the same predictor are missing, leaving the trade-off between computational cost and statistical efficiency insufficiently assessed.
-
(W4) Evaluation covers six synthetic regression families but no real-world datasets, leaving the practical applicability of the method insufficiently demonstrated. Given the broader flexibility of pretrained Transformers, particularly for text data, I am not convinced that these experiments adequately demonstrate the practical benefits of the proposed approach.
-
The comparisons do not adequately isolate the source of improvement: ICL benefits from extensive auxiliary-task pretraining, whereas the refitting baselines learn from the current task alone. Runtime comparisons also exclude pretraining costs and, for some baselines, use different hardware. Although these choices are disclosed, matched-pretraining and same-predictor comparisons are needed to distinguish the benefits of the conformal procedure from those of the predictor and implementation.
-
The comparisons use different score constructions: ICL-FCP uses leave-one-out residuals, whereas the refitting baselines use in-sample residuals from the augmented dataset. This confounds the effect of the predictor with that of the conformal score construction.
Please clarify if I have misunderstood the distinctions from prior work or the scope of the results. My current assessment is primarily driven by the limited theoretical novelty and insufficient empirical evidence of the method’s practical advantages. I would be happy to raise my score if these concerns are adequately addressed.
[1] Huang, Zhe, et al. "From predictions to confidence intervals: an empirical study of conformal prediction methods for in-context learning." arXiv preprint arXiv:2504.15722 (2025).
[2] Huang, Xiao Shi, et al. "Unifying Conformal Language Tasks with In-Context Ensembles." arXiv preprint arXiv:2609.03005 (2026).
[3] Deng, Weicao, et al. "Optimizing In-Context Learning for Efficient Full Conformal Prediction." IEEE Signal Processing Letters 33 (2025): 311-315.
[4] Pantelidis, Ippokratis, Korbinian Randl, and Aron Henriksson. "Efficient text classification with conformal in-context learning." arXiv preprint arXiv:2512.05732 (2025).
[5] Li, Yingcong, et al. "Transformers as algorithms: Generalization and stability in in-context learning." International conference on machine learning. PMLR, 2023.
[6] Ching, Michelle, et al. "Efficient and Minimax Optimal In-context Nonparametric Regression with Transformers." arXiv preprint arXiv:2601.15014 (2026).
Thanks for the authors' clarifications. My concerns have been addressed.