acceptodds
Withdrawn from ICLR 2027

Continual Calibration: Stale Conformal Thresholds Fail Under Lifelong LLM Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

79 views

Abstract

Continual fine-tuning changes not only a language model's predictions but also the score distribution used by post-hoc uncertainty procedures, and this creates a specific failure mode for conformal prediction: thresholds calibrated before later fine-tuning can lose task-level coverage even when top-1 accuracy changes only modestly. We measure this failure mode on sequentially fine-tuned LLMs across five model configurations and eight task sequences, seven of which are classification-style or multiple-choice settings covered by our formal framework. In the classification-style grid, stale-threshold coverage loss is substantially larger than accuracy loss; by aggregate loss, coverage degrades by roughly as much as accuracy, and the effect is diagnosed by drift in the nonconformity-score CDF rather than by argmax-accuracy drift. We propose task-specific calibration replay, which maintains a small held-out calibration buffer per task and recomputes that task's split-conformal threshold under the current model after each update. Under standard split-conformal exchangeability assumptions, and assuming the calibration buffer is not used for training or model selection, this restores finite-sample marginal coverage for the corresponding task without adding gradient updates. Empirically, using a default calibration-buffer cap of examples per task, with for low-resource tasks, task-specific recalibration (which acts as a continual Mondrian conformal predictor) reduces coverage loss to near nominal while preserving utility, with average set sizes and singleton rates close to a large fresh-calibration reference. The results hold against competitive baselines including online adaptive conformal inference (ACI) and weighted conformal prediction, both of which recover only part of the coverage gap, and a simple zero-shot router retains most of the benefit when task identity is unknown. Our formal guarantees apply to classification-style tasks with known task identity and task-specific buffers; open-ended generation results are visually separated and reported only as exploratory.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, conformal prediction, uncertainty quantification, calibration, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_CQfoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies split-conformal prediction for LLMs that undergo sequential LoRA fine-tuning. It shows that reusing a conformal threshold calibrated on an earlier model causes large drops in task-level coverage, much larger than the accompanying drops in accuracy. Across five models and seven classification sequences, coverage falls by 20.6 points on average while accuracy falls by 5.6. The authors attribute this to drift in the nonconformity-score CDF, which they measure with KS distance and decompose in theory into parameter-induced and task-distribution terms. Their remedy, “calibration replay,” keeps a small held-out labeled buffer per task (up to 200 examples) and recomputes each task’s threshold under the current model after every update. In other words, it is Mondrian split-conformal re-run after each fine-tuning step. With known task identity, standard exchangeability gives finite-sample marginal coverage. Empirically, the coverage gap shrinks to about 1–2 points with compact prediction sets, outperforming pooled recalibration, temperature scaling, ACI, weighted CP, and a max-threshold baseline. A zero-shot task router retains most of the benefit when task identity is unknown.

Critical strengths

Relevant practical warning. The paper emphasizes that continual-learning evaluations should track coverage and uncertainty, not only accuracy. This is a useful message for deployments that rely on confidence for abstention or routing. Broad empirical sweep. The evaluation covers several model families and scales, a range of task sequences, ten accuracy-preserving continual-learning methods, and multiple calibration baselines. It also reports set size and singleton rate alongside coverage. Clear scoping of claims. The claim-support table, the explicit assumptions, and the separation of exploratory generation results make it clear what is proven and what is only observed.

Critical weaknesses
  1. The finding is expected, and the method is standard. Conformal guarantees require exchangeability under the current model, so a threshold calibrated on an older model is not expected to stay valid after fine-tuning. Calibration replay is ordinary per-task (Mondrian) split-conformal recalibration, re-run after each update, and the theory consists of textbook results. The contribution is therefore mainly empirical.
  2. The advanced baselines are unconvincing. ACI has a long-run coverage guarantee, so a 7.2-point gap over thousands of test points needs explanation. Weighted CP should collapse to plain recalibration when calibration and test data share a distribution, and it remains unclear which model was used to compute its scores.
  3. Some results look unusually clean and need verification. Figure 2b reports r2=0.999r^2 = 0.999 between KS distance and coverage drift, although KS distance is only an upper bound on drift at a fixed threshold. Most table entries are integers with very regular standard deviations. It is also unclear whether “coverage loss” counts only under-coverage, since Table 5 contains many over-covered cells.
  4. The main baseline is a straw man, and the hard problem is assumed away. In practice, few would reuse an old threshold after updating the model. The interesting question is how to maintain coverage when fresh labeled calibration data is scarce. The method assumes exactly that resource: a labeled, held-out buffer for every task, plus known task identity.
Questions for authors

See weakness

Reviewer_gGS2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  • This paper studies the degradation of conformal coverage under continual LLM fine-tuning.
  • It is interesting that the coverage can deteriorate significantly even when predictive accuracy remains stable.
  • The authors propose Calibration Replay, which stores task-specific calibration data and recomputes thresholds after each model update.
Critical strengths
  • The paper tackles an important problem concerning the coverage reliability under continual LLM fine-tuning.

  • Extensive experiments demonstrate a clear discrepancy between accuracy retention and conformal coverage.

  • The proposed method shows strong empirical performance, supported by clear theoretical guarantees.

Critical weaknesses
  • The problem setting is overly restrictive. The paper assumes access to separate labeled calibration data whenever a new task is introduced. Under this assumption, coverage can be restored by simply reapplying standard methods (e.g., split conformal prediction) after each model update. Typically, in such situations, recent works try to restore rewards without requiring new ground-truth labels.

  • The proposed method essentially stores task-specific calibration data and recomputes conformal thresholds.

  • Thus, the technical novelty is limited, and the theoretical guarantees largely follow from standard conformal prediction.

  • The method may be difficult to apply in realistic settings where labeled calibration data cannot be continuously obtained or maintained.

Questions for authors
  • Given the availability of task-specific labeled calibration data, what is the key methodological contribution beyond repeatedly applying standard Split Conformal Prediction after each model update?
Reviewer_9HxeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what happens to split-conformal prediction sets when an LLM is sequentially fine-tuned after its threshold was calibrated. The authors found coverage at a stale threshold degrades much more than top-1 accuracy. They propose task-specific calibration replay to serve as a continual Mondrian conformal predictor. The results shows the proposed method outperform the existing baselines. And a zero-shot task router retains most of the benefit when task identity is unknown.

Critical strengths

The paper addresses a practical issue that is easy to overlook: uncertainty thresholds can become unreliable after a model update, even when accuracy stays stable. The experiments cover a good range of models and baselines, and the proposed fix is simple to apply without extra training. I also appreciate that the authors clearly explain their assumptions and the limits of their claims.

Critical weaknesses

My main concern is novelty: the proposed fix is essentially standard per-task conformal recalibration, and most of the theory uses established results. It is also unclear how much of the observed failure comes from using scores that are not normalized over valid labels. An ablation with label-normalized scores would help. The reported under-coverage is hard to reconcile with the stated guarantee. These issues need clarification, ideally with code and per-seed results.

Questions for authors
  1. Under exchangeability, why does calibration replay systematically under-cover by 4.8 points at m = 50, and why does fresh calibration with n = 1,000 under-cover by 1.1 points? From which original dataset splits are the calibration buffers and test sets drawn?
  2. Does the stale-threshold failure persist with label-set-normalized softmax scores, or with APS/RAPS-style scores?
  3. How is conformal coverage computed on SQuAD in the legal, medical, and code sequences?
Reviewer_JrYcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper considered conformal prediction in continual learning with known task boundary. The key idea is to acquire new calibration data whenever new task arrives so that standard conformal prediction technique can be readily applied to the considered continual leaning set-up.

Critical strengths

The idea to (i) acquire new calibration data whenever new task arrives; (ii) update the conformal-prediction threshold for each task (both current and past tasks) using the buffered per-task calibration data, could be useful in practice.

Critical weaknesses

Despite that this work considers conformal prediction for a continual learning set-up, its theoretical guarantee only focuses on a "fixed" setting (i.e., fixed model with a fixed task). That is, given that there is no meaningful guarantees on the continual learning set-up, I think all the theoretical guarantees presented in this work is misleading. For instance, what should be shown in this kind of work is something like: P{for every newly updated model, for all seen tasks, the proposed set contains the true label} \geq 1-alpha ? And I think such result is not trivial given that the same per-task calibration data is reused across different models (see, e.g., [Bates et al., 23] for a similar subtlety where multiple test inputs share the same calibration data set).

[Bates et al., 23] Bates S, Candès E, Lei L, Romano Y, Sesia M. Testing for outliers with conformal p-values. The Annals of Statistics. 2023 Feb;51(1):149-78

(minor comments) The authors are considering calibration-data-conditional conformal prediction and marginal conformal prediction confusingly in the theory section. For instance, Theorem 7 talks about marginal conformal prediction and Theorem 8 talks about cal-data-conditional conformal prediction. Are they comparable? It seems to me that, the authors want to simplify their theory by focusing on the behavior of CDF with fixed calibration data, to ignore the randomness associated with the calibration data per se. I think this is only making things confusing without any meaningful theoretical innovations. Also, most of the results in the theory section is a mere repetition of well-known results in the literature (see, e.g., [Vovk 12] and Pinsker's inequality as well as DKW inequality), and, I think it would be much better to clearly state what is newly shown in this work in terms of theory (which, unfortunately, nothing new at the moment unless the authors can resolve my above concern on "continual-learning-specific coverage guarantees").

[Vovk 12] Vovk V. Conditional validity of inductive conformal predictors. In Asian conference on machine learning 2012 Nov 17 (pp. 475-490). PMLR.

Questions for authors

What would be the "continual-learning-specific coverage guarantees" of this work? Can the authors prove that

P{for every newly updated model, for all seen tasks, the proposed set contains the true label} \geq 1-alpha ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.