- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies what calibration training actually teaches LLMs by training to predict accuracy on factual multiplechoice questions across 10 open-weight models and multiple datasets. The authors find that verbalized confidence shifts from nearly uncorrelated with accuracy before training (r=0.04) to strongly correlated after (r=0.73), and that trained confidence carries two distinct signals: true-accuracy tracking on questions near the training distribution and output-consistency tracking on distant ones.
Critical strengths
-
The paper focuses from reporting improved calibration metrics to asking what calibration training actually teaches the model, i.e.whether the signal reflects genuine self-knowledge or an artifact of the answer distribution. This is consequential because it addresses a fundamental gap in the literature and reframes the problem in a way that yields more interpretable insights.
-
Direct and convincing core observations. The U-shaped confidence–accuracy relation and the finding that trained confidence never falls below chance even on confidently incorrect answers are clearly visible in the figures and do not depend on contested statistical machinery. This is consequential because these results stand on their own as robust empirical findings, regardless of how one interprets the deeper mechanistic claims.
Critical weaknesses
-
The main “accuracy tracking generalizes” claim rests on a test split that’s almost a copy of training. The held-out category shares ranges digit counts and prompt templates with the training categories with only the operator symbol changes. So this isn’t a very OOD transfer; it’s interpolation within the same generative process. This is consequential because it is the only setting in which accuracy tracking appears to transfer, yet the abstract and introduction present it as broader category-level generalization; without stronger evidence, the central mechanistic claim is not well supported.
-
The setup doesn’t actually produce a system that both answers and reports confidence. The LoRA makes the network unable to answer questions, while the accuracy consistency targets come from the base model. So there’s no single model doing both things at once, which undercuts the abstention motivation and makes it hard to say what “the LLM learned.” This is consequential because the paper’s framing depends on interpreting the result as something the base model has learned about its own accuracy, but the missing frozen-backbone probe control leaves open the simpler alternative that the LoRA merely reconstructs the base answer distribution internally.
Questions for authors
- If MATH's test category is replaced with a genuinely disjoint one (different magnitude ranges, mixed operators, parenthesized expressions), does accuracy tracking survive?
2 comments
The rebuttal clarifies the intended scope but does not resolve my main concerns. The only evidence of accuracy tracking beyond the training questions comes from a near-in-distribution MATH split, while the cross-dataset results confound distributional distance with changes in task, domain, and difficulty. Moreover, the additional controlled experiments and the frozen-backbone probe needed to clarify what calibration training actually teaches are left to future work. The central claims therefore remain insufficiently supported and are substantially more limited than the presentation suggests. This also weakens the paper’s novelty: the remaining contribution is largely local accuracy interpolation and consistency tracking under changing experimental conditions.
Rebuttal
We thank the reviewer for their careful reading and constructive feedback. We particularly appreciate their recognition that the paper shifts the focus from calibration metrics to what calibration training actually teaches the model. We also appreciate their observation that the core findings are visible directly in the figures: the U-shaped confidence–accuracy relation, and the fact that trained confidence remains high on confidently incorrect answers.
One small clarification: after training, confidence is no longer verbalized but read from a linear probe on the model’s hidden states. The comparison between verbalized confidence before training (r = 0.04) and probe-based confidence after training (r = 0.73) serves as a sanity check that fine-tuning succeeded, not as a comparison between equivalent elicitation formats.
W1. We agree with the reviewer’s description of the MATH test split: it is close to the training categories by design, and accuracy tracking on it could be seen as interpolation within the same generative process. This is precisely our point: across all our experiments the two MATH datasets are the only ones where the model is able to predict its true accuracy consistently beyond the training questions and it is the one closest to the training data. As soon as test questions are more distant, from held-out topics in MMLU-Pro and MedMCQA or from other datasets, trained confidence tracks output consistency instead (Fig. 3) even if the model was trained on MATH. Our paper does not claim that true accuracy tracking generalizes. As stated in the abstract, accuracy tracking “remains local to the training distribution”. The generalization we report concerns output consistency tracking. We recognize, however, that phrases such as “held-out category” and “transfers to unseen domains” (Introduction) may suggest that the MATH test split is more out-of-distribution than intended, and we have clarified them in the revised version.
W2. We agree that our setup does not produce a single model that both answers and predicts its accuracy, and we state this in Section 2.4. This is a deliberate design choice. Training a model to answer and to predict its accuracy at the same time means predicting a moving target: every update changes the model’s accuracy, including on held-out questions. Keeping the base model frozen and training only the adapters to predict its accuracy ensures that true accuracy and output consistency remain fixed throughout training, which allows a clean comparison of which signal the trained confidence tracks. The same frozen weights can still be used in practice, with adapters disabled to answer and enabled to report confidence. We agree that a jointly trained model would match the abstention motivation more closely, and that “predicting one’s own accuracy” should be understood here as an adapted model predicting the accuracy of its frozen base. Our aim is to study what models learn when trained to predict their accuracy, not to build the best-performing confidence estimator. We will make this explicit in the limitations.
We also agree that a probe on the frozen backbone would indicate whether the consistency signal is already linearly available in the base model or computed by the adapters. Our claim concerns what the trained confidence tracks, not how it is computed. The middle-probe experiment (Appendix G) shows that consistency tracking does not simply leak from the output logits which is a necessary control for our claim; localizing the signal more precisely in the network is a question we leave to future work.
Answer to Questions. We thank the reviewer for this suggestion. We designed the MATH dataset to contain single-operation questions that can be answered fairly directly, without requiring complex chain-of-thought, consistent with our focus on knowledge-like recall rather than multi-step reasoning (Section 2). Mixed operators or parenthesized expressions risk introducing such multi-step reasoning to solve the problem. Instead, we propose future work to test the same hypothesis from the other direction. Creating test splits for MMLU-Pro and MedMCQA that are closer to their training data. Combined with the current topic-disjoint test splits, this provides both near and far test sets within each dataset, which is the contrast the reviewer asks for. The extreme version of the question is already answered by our cross-dataset results: MATH-trained models evaluated on MMLU-Pro and MedMCQA track output consistency rather than accuracy (Fig. 3). The new splits would add data at low to moderate distances from the training set, and test whether the MATH results replicate on the other datasets. Based on these results, we predict that on a more distant MATH test set, accuracy tracking would weaken in favor of consistency tracking. Given the short discussion period, we could not implement this new condition yet and leave the question open for future work.