- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates how post-training affects model confidence and whether calibration transfers across evaluation pools. Its main findings are that: (1) post-training degrades confidence reliability while leaving discrimination largely unchanged and (2) calibration is strongly dependent on the evaluation pool, but a small number of labeled target-pool examples can substantially mitigate calibration transfer errors.
Critical strengths
-
The paper provides a well-grounded decomposition of confidence quality, distinguishing reliability from discrimination. This analysis offers a more informative account of post-training effects than aggregate calibration metrics alone.
-
The experiment covers multiple model sizes, post-training recipes, and tasks, with detailed results and extensive supporting analyses.
-
The paper clearly discusses important protocol limitations and boundary conditions, including the format mismatch when evaluating reasoning models under plain completion and temperature estimates reaching the search upper bound. This transparency supports reproducibility and helps readers interpret the findings appropriately.
Critical weaknesses
-
The core analyses rely on softmax confidence over answer options in multiple-choice tasks. Although the paper includes a broader preliminary survey, it remains unclear whether the main findings extend to open-ended generation or other confidence estimation settings. The conclusions should therefore be scoped more explicitly to the evaluated protocols.
-
Sample difficulty is defined using the average correctness of other checkpoints. This provides a controlled measure of model-relative difficulty, but it may reflect shared biases among the selected checkpoints and does not establish correspondence with human-perceived difficulty. Validation using an external difficulty measure would strengthen this analysis.
-
The paper does not clearly establish whether moving-target calibration is specific to post-training, rather than being a general consequence of distribution-dependent calibration. The appendix reports a reliability cost of 0.0851 on harder pools versus 0.0019 on easier pools, suggesting a potentially important connection. An explicit analysis of how ΔREL varies with difficulty changes would help determine whether post-training increases sensitivity to difficulty shifts and connect the two core findings of this paper: reliability tax and moving-target calibration.
-
The broad observations that post-training can degrade calibration and that calibrators may transfer poorly under distribution shifts are already established in prior work. The paper should articulate its novelty more precisely around the systematic separation of reliability and discrimination, and the empirical finding that very small target-label budgets can substantially repair calibration transfer errors. Positioning these as the central contributions would make the motivation more compelling and better aligned with the evidence.
Questions for authors
See Weaknesses.