acceptodds
Under review as a conference paper at ICLR 2027

The Post-Training Tax: The Moving Target of Calibration

Abstract

Large language models typically achieve higher predictive accuracy after post-training, but a more fundamental question is whether this improvement in accuracy is accompanied by better confidence quality. To investigate this question, we examine the changes induced by post-training from two aspects of confidence: reliability and discrimination. Across 47 model–task cells comprising 16 base/tuned model pairs, 6 post-training settings, and 3 multiple-choice tasks, we find that in the 35-cell alignment cohort, post-training improves average accuracy by 1.1 percentage points while increasing the Brier reliability component by 0.047, with reliability degrading in 32 of 35 cells; discrimination remains largely stable. We refer to this phenomenon as the reliability tax of post-training. To mitigate this reliability loss, we further evaluate several confidence calibration methods and find that simple temperature scaling can substantially reduce calibration error, but its effectiveness is not fixed: when the model, task, and prompt template are held constant while the evaluation pool changes, the median temperature required for harder pools is approximately 5 times that for easier pools, and the excess ECE induced by calibration transfer is strongly correlated with the accuracy gap between pools. Further experiments show that only 20 target labels can substantially improve calibration on a new evaluation pool, reducing held-out ECE by approximately 83.4% relative to the uncalibrated state. Overall, our study reveals the reliability tax introduced by post-training and demonstrates that calibration is strongly dependent on the evaluation pool, while a small amount of target data can support effective recalibration on new evaluation pools.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Confidence Calibration, Uncertainty Estimation, Post-Training, Distribution Shift, Model Reliability

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_1TV6OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how post-training affects model confidence and whether calibration transfers across evaluation pools. Its main findings are that: (1) post-training degrades confidence reliability while leaving discrimination largely unchanged and (2) calibration is strongly dependent on the evaluation pool, but a small number of labeled target-pool examples can substantially mitigate calibration transfer errors.

Critical strengths
  1. The paper provides a well-grounded decomposition of confidence quality, distinguishing reliability from discrimination. This analysis offers a more informative account of post-training effects than aggregate calibration metrics alone.

  2. The experiment covers multiple model sizes, post-training recipes, and tasks, with detailed results and extensive supporting analyses.

  3. The paper clearly discusses important protocol limitations and boundary conditions, including the format mismatch when evaluating reasoning models under plain completion and temperature estimates reaching the search upper bound. This transparency supports reproducibility and helps readers interpret the findings appropriately.

Critical weaknesses
  1. The core analyses rely on softmax confidence over answer options in multiple-choice tasks. Although the paper includes a broader preliminary survey, it remains unclear whether the main findings extend to open-ended generation or other confidence estimation settings. The conclusions should therefore be scoped more explicitly to the evaluated protocols.

  2. Sample difficulty is defined using the average correctness of other checkpoints. This provides a controlled measure of model-relative difficulty, but it may reflect shared biases among the selected checkpoints and does not establish correspondence with human-perceived difficulty. Validation using an external difficulty measure would strengthen this analysis.

  3. The paper does not clearly establish whether moving-target calibration is specific to post-training, rather than being a general consequence of distribution-dependent calibration. The appendix reports a reliability cost of 0.0851 on harder pools versus 0.0019 on easier pools, suggesting a potentially important connection. An explicit analysis of how ΔREL varies with difficulty changes would help determine whether post-training increases sensitivity to difficulty shifts and connect the two core findings of this paper: reliability tax and moving-target calibration.

  4. The broad observations that post-training can degrade calibration and that calibrators may transfer poorly under distribution shifts are already established in prior work. The paper should articulate its novelty more precisely around the systematic separation of reliability and discrimination, and the empirical finding that very small target-label budgets can substantially repair calibration transfer errors. Positioning these as the central contributions would make the motivation more compelling and better aligned with the evidence.

Questions for authors

See Weaknesses.

Reviewer_qXEkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary
  • Post-training raises accuracy but makes confidence less reliable. The paper calls this the "reliability tax". Simple calibration fixes it, but a temperature fitted on one pool fails on a harder pool, and about 20 target labels fix it again. All experiments are multiple-choice tasks (47 model–task cells).
Critical strengths
  • Broad and careful evaluation. The label-budget result (about 20 labels for the typical case, about 200 for the tail) is practical.
Critical weaknesses
  • Incremental. The reliability tax has been reported since the GPT-4 technical report, and the paper itself cites prior work showing it (Ouyang et al., Bai et al., Zhu et al.). Just naming?
  • Limited applicability. Only multiple-choice classification, while most LLM use cases are generative. The pools are built by item difficulty, so the shift is artificial. Natural shifts and free-form generation are left to future work. So it is unclear whether the "moving target" matters in real use.
  • Brittle metric. All calibration results rely on ECE with a single confidence definition (answer-letter probability). ECE is known to change significantly with how confidence is measured ("Does Alignment Tuning Really Break LLMs' Internal Confidence?", BlackboxNLP @ EMNLP 2024), and only binning is checked (App. B.4).
Questions for authors
  • What is the implication of this study for real-world use, where most usages are generative?
Reviewer_DH4LOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how post-training changes the confidence of LLMs on multiple-choice questions. Using 16 paired base/tuned models on three benchmarks, the authors split confidence quality into reliability and discrimination with Murphy and Brier decompositions. They find that post-training makes reliability worse, while accuracy and AUROC stay about the same, and they name this issue as "reliability tax". They further split each benchmark into an easier and a harder pool, and show that the best temperature differs a lot between the two pools, so a temperature fitted on one pool does not transfer to the other. Finally, they show that 20–50 labels from the target pool are enough to bring low ECE.

Critical strengths
  1. The research problem is practically important. Post-trained models are the ones deployed in practice, and their confidence is used for safety-aware decisions.
  2. The motivation is clear. The paper separates two questions that are often mixed together: whether the confidence values are calibrated, and whether they still rank correct answers above wrong ones. The paired base/tuned design makes it possible to link the changes directly to post-training.
  3. The analysis is solid. The decompositions are exact, and the paper shows that recalibration methods that keep the ranking of confidence do not change discrimination (Section 3.2). The statistical tests are also more thorough than in most LLM papers, including paired bootstrap, correction for multiple comparisons, equivalence tests and leave-one-checkpoint-out checks (Tables 11, 15 and 20).
  4. The experiments are broad. The paper compares many calibrators, including TS, Platt, isotonic regression, DACA, SMART, DAC and ATS (Tables 3 and 5), and also tests each method with its original training objective (Table 33). It also studies how many target labels are needed (Sec. 4.5) and when a transferred temperature becomes worse than no calibration, which happens once the accuracy gap between pools exceeds about 0.35 (Sec. 4.4.2).
Critical weaknesses
  1. The novelty of the reliability-tax finding is limited. It is already known that post-training makes LLM confidence less calibrated [1, 2, 3]. It has also been shown that chat models are miscalibrated on multiple-choice questions, but their confidence can still tell correct answers from wrong ones [4]. This is very close to the main finding here, that REL gets worse while AUROC stays about the same. However, the paper does not cite [4], and Section 2.1 says that prior work cannot tell whether post-training hurts the ranking. The new parts are mainly the paired base/tuned design and the exact decompositions.

  2. The two parts of the paper are not well connected. The reliability tax compares base and tuned models (Sec. 4.2). In contrast, the moving-target analysis only uses the tuned models (Table 25), so it does not show whether post-training makes the problem worse. The base models seem to have the same problem. On MMLU-Pro, their ECE rises from 0.066 on the full pool to 0.171 on the harder pool (Tables 3 and 4). So the moving target looks like a general limit of temperature scaling rather than part of the post-training tax.

  3. The moving-target finding is expected, and the suggested fix cannot be used in deployment. Temperature scaling uses one global parameter, so a model that is more overconfident on harder questions needs a different temperature on each pool. The suggested fix is to fit the temperature on 20–50 labeled questions from the target pool (Sec. 4.5). However, in deployment we do not know which pool a new question belongs to, because the pools are defined by how well other models answer it.

  4. The difficulty pools do not reflect a natural shift. Because the pools are defined by other models' correctness, it is unclear whether the results hold when difficulty changes in practice (e.g., GSM8K vs. AIME). The harder pools are also often close to chance, for example, 15.1% on MMLU-Pro where chance is 10% (Table 24), so they may contain more label errors.

  5. The effect of each post-training stage is not analyzed. In the Tulu 3 pipeline, ΔREL on MMLU-Pro is 0.023 after SFT, 0.079 after DPO and 0.084 after RLVR (Table 1), so most of the tax appears at the DPO stage. This is an interesting result, but the paper only says that the tax grows at each stage (Sec. 4.2.2) and does not study why. The comparison group is also unclear. In Table 9, the tuned ECE of Math-1.5B/7B is the same as the base ECE of R1-1.5B/7B, so the "math-RL" group may measure continued pre-training rather than RL.

Questions for authors
  1. For the base models, how large is the temperature ratio between the harder and easier pools? If it is similar to that of the tuned models, how is the moving target related to post-training?
  2. In deployment, the pool of a new question is unknown. How should the target-pool fit be used in practice, and how does it compare with calibrators that adapt to difficulty, such as ACE [5] and few-shot recalibration [6]?

[1] GPT-4 Technical Report, arXiv:2303.08774.

[2] Language Models (Mostly) Know What They Know, arXiv:2207.05221.

[3] Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator, NeurIPS 2025.

[4] Probabilities of Chat LLMs Are Miscalibrated but Still Predict Correctness on Multiple-Choice Q&A, arXiv:2402.13213.

[5] Adaptive Calibrator Ensemble: Navigating Test Set Difficulty in Out-of-Distribution Scenarios, ICCV 2023.

[6] Few-Shot Recalibration of Language Models, arXiv:2403.18286.

Reviewer_hMzcOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how post-training changes confidence reliability and discrimination in large language models. Across 47 model–task cells covering 16 base/tuned pairs and six training settings, the authors report that the 35-cell instruction/SFT/DPO/RLVR cohort gains 1.1 percentage points in mean accuracy while its Brier reliability component increases by 0.047. Estimated discrimination changes are smaller and vary in direction.

The paper then evaluates recalibration and its transfer across constructed difficulty pools. Temperature scaling substantially reduces held-out calibration error, but temperatures fitted on easier pools transfer poorly to harder pools. Across 45 model–task cells, harder pools require approximately five times the temperature at the median. With 20 target labels, the mean of within-cell median held-out ECE values falls to 0.0394. Supporting analyses examine alternative calibrators, confidence signals, composition controls, and weight-space interventions.

Critical strengths
  1. Separating reliability from discrimination makes the post-training effects easier to interpret. The reliability increase appears across several model families and training stages and survives the binning checks.

  2. The held-out label-budget experiments are useful. They show both the benefit of target labels and the remaining instability at small budgets. The comparisons with Platt scaling, isotonic regression, and base-model references strengthen this result.

Critical weaknesses
  1. Most AUROC-change intervals include zero, which does not establish equivalence. Appendix B.5 reports that no individual main-group cell supports equivalence within ±0.02 AUROC. The pooled interval in Appendix B.6 assumes independent cells despite shared checkpoints and evaluation items. These results suggest little systematic average change, but leave uncertainty about meaningful changes in individual settings.

  2. Prior work studies alignment and calibration [1], proposes methods to repair calibration after alignment [3], and links calibration parameters to dataset difficulty [2]. The added value here is the reliability–discrimination analysis and the target-label budget study. However, the core experiments use three-shot plain completion on ARC and MMLU-Pro, including its mathematics subset. It remains unclear whether the conclusions hold under native chat templates or reasoning-enabled inference. This limits the generality of the practical recommendations.

References

[1] Chiwei Zhu, Benfeng Xu, Quan Wang, Yongdong Zhang, and Zhendong Mao. On the Calibration of Large Language Models and Alignment. Findings of EMNLP, 2023.

[2] Yuli Zou, Weijian Deng, and Liang Zheng. Adaptive Calibrator Ensemble: Navigating Test Set Difficulty in Out-of-Distribution Scenarios. ICCV, 2023.

[3] Jiancong Xiao, Bojian Hou, Zhanliang Wang, Ruochen Jin, Qi Long, Weijie J. Su, and Li Shen. Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach. ICML, 2025.

Questions for authors
  1. How do the pooled AUROC intervals change when accounting for shared models and evaluation items? What equivalence margin would you consider practically meaningful?

  2. Does the reliability-versus-discrimination pattern persist under native chat templates for representative instruction-tuned models?

  3. With 20 and 200 target labels, how many individual cells achieve a within-cell 90th-percentile ECE below 0.05? The current average of cell-level summaries makes this difficult to assess.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.