- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper tackles calibration in Large Language Models. They study a notion of calibration termed -calibration, which matches the expectation of a statistic induced by the model with the expectation of the same statistic under a reference distribution. The main problem authors tackle is that the sought calibration procedure, , exponential tilting (ET), requires expensive computations of a partition function. The authors propose an efficient approach to ET based on rejection sampling. Rejection sampling enables to efifciently sample from the tilted distribution and authors additionally propose a methodlogy to estimate the exponential tilt . Authors validate they proposal in terms of entropy calibration with , and with 2 different datasets. The proposal effectively improves entropy calibration of the base models.
Critical strengths
The main empirical strength of the paper is that the proposed method consistently reduces entropy miscalibration relative to the corresponding base models across several model families and datasets.
I also consider the idea behind the methodological contribution appealing and potentially useful. More precisely, the idea to use rejection sampling to enable exact sampling from the exponentially tilted distribution without explicitly computing the partition function is compelling.
Moreover, to the best of my knowledge, the paper is well connected to the relevant literature on ET, -moments matching and entropy calibration, and rejection sampling.
Critical weaknesses
The biggest weakness of the paper is clarity. I found the paper difficult to follow, with several important methodological components presented in a fragmented manner. In particular, the procedure used to estimate the tilt parameter is scattered across different sections, and the connection between the cross-entropy objective, its derivative, the moment-matching condition, and the practical bisection procedure is not explained clearly in one place.
Moreover, the estimation of the entropy difference is entirely detailed in the Appendix. This is a crucial element of the experimental evaluation and the estimator is sufficiently nontrivial that I believe at least its main idea, properties, or practical implementation should be described in the main text. Considering the paper appears to have available space, I do not see why this was not the case in the first place.
The thoeretical contribution of the paper is also of difficult interpretation. Authors state several lemmas and a theorem, but the surrounding discussion provides relatively little intuition about what the bounds imply, when they are informative, and how they connect to the practical algorithm.
I also have concerns about terminology. In particular, the paper uses the term “local calibration” to refer to the teacher-forced setting. I understand the intended distinction, but “local calibration” is already an established term in the calibration literature, typically referring to calibration notions that incorporate locality or similarity between inputs or their representations [1-6]. I therefore consider this terminology potentially confusing and suggest replacing it.
The second major wekaness of the paper is the experimental evaluation. Authors do not include any baselines other than the base model. This surprises me because the introduction and related-work sections explicitly discuss several alternative approaches and motivate the proposal partly through their limitations. Without such baselines, it is difficult to assess whether the proposed method provides a meaningful practical advantage beyond improving the targeted entropy-calibration objective relative to the unmodified model.
This issue is particularly relevant because authors mention existing methods suffering from a quality-diversity trade-off. However, the experiments do not evaluate this trade-off directly nor do they compare the proposal against the alternative methods that should suffer from it. I would have expected an empirical comparison reporting calibration together with quality and diversity metrics.
Finally, the paper makes several claims that appear stronger than what is supported experimentally. For example, the discussion suggests possible applications to phenomena such as over-empathy and sycophancy, but no experiments are provided for these settings. Similarly, the abstract states that the rejection-sampling overhead is expected to improve for larger and more capable models, while the current experiments only cover a limited set of relatively small models. These claims should either be supported experimentally or stated as directions for future work.
REFERENCES
[1.] Barbera, Cesare, et al. "Multiclass local calibration with the Jensen-Shannon distance." AISTATS2026.
[2] Luo, Rachel, et al. "Local calibration: metrics and recalibration." UAI2022.
[3] Ding, Zhipeng, et al. "Local temperature scaling for probability calibration." ICCV2021.
[4] Xiong, Miao, et al. "Proximity-informed calibration for deep neural networks." NeurIPS2023.
[5] Lin, Zhen et al. "Taking a step back with kcal: Multi-class kernel-based calibration for deep neural networks." ICLR2023.
[6] Valk, Kaspar, and Meelis Kull. "Assuming locally equal calibration errors for non-parametric multiclass calibration." TMLR2023.
Questions for authors
I would ask reviewrs to address my concerns in the weaknesses section, especially regarding the experimental evaluation.
One additional question I have is: although the proposed framework is formulated for generic -calibration, what guarantees does improving this moment-matching objective provide with respect to other existing notions of calibration? Does the proposed method improve probabilistic quality beyond the targeted entropy statistic? It would be informative to report additional proper scoring rules, such as NLL and the Brier score, and compare them with those of the base model and the baselines.
4 comments
I appreciate the authors rebuttal. I tried to download the new pdf, but the changes are very hard to track. Before I evaluate their response, I would kindly ask authors to highlight the changes (text of a different color, also for captions of tables and images, is a quite standard pratice)
We posted a new revision with the highlighted text (in case you missed the general comment we posted). Thank you for your consideration, we hope this clarifies our paper.
I thank the authors for their detailed rebuttal and substantial revisions. The presentation has improved, and I appreciate the additional experiments, particularly the NLL and Brier score evaluations. Most of my concerns have been addressed.
However, my main reservation regarding the experimental evaluation remains. The absence of direct comparisons with alternative decoding strategies e.g., temperature or min-p, makes it difficult to assess the practical advantages of the proposed method, particularly concerning the quality-diversity trade-off emphasized in the paper. Given the relatively limited methodological novelty, I consider this comparative validation particularly important.
Overall, I acknowledge the improvements and am willing to increase my score from 1 to 2, but I do not believe the remaining experimental evidence is sufficient to support acceptance.
We thank the reviewer for the careful reading and detailed suggestions. We are glad that the reviewer finds exact sampling without the partition function compelling and the reduction in miscalibration consistent across models and datasets. We addressed each point in the revision, in the order of the review.
1. Estimating α. Section 5.1.1 describes the procedure at a high level, and the new Algorithm 2 (Appendix E) gives the exact algorithm we use. α minimizes the CE ℓ(α) on calibration documents. This objective is convex, with derivative ℓ′(α) = μ_α − μ_P, so minimizing CE and matching the moment are the same problem. We estimate ℓ′(α) from R rejection-sampled tokens per calibration context, without the partition function, and find α by bisection.
2. Entropy estimator. We agree that the estimator is non-trivial, but we kept its derivation in the appendix because it is an evaluation tool rather than part of the method. Neither fitting α nor sampling uses it. It only serves to report the tilted model's entropy in Figures 1–3. We preferred to use the main-text space for the new held-out results (Table 1). To address the concern, Remark 7 now states the estimator's main idea and guarantees in the main text. The derivation and the practical implementation remain in Appendix C.
3. Interpreting the theory. Each result in Section 4 is followed by its interpretation (lines 228–232, 241–247 and 257–269). In short, sampling is exact for any valid bound (Lemma 2), its cost grows with the tilt times the typical value of the statistic (Lemmas 3 and 4), and stopping the CE minimization early gives a smaller, cheaper tilt at a quantified CE loss (Lemma 5, Theorem 6). The revision adds the converse: if the CE can improve only slightly, the fitted tilt must be small and therefore cheap to sample (lines 257–262). Section 4.1 connects the theory to the practical algorithm for entropy, where B = 0 is always a valid bound.
4. Terminology. Thank you for pointing out the clash with the established notion of local calibration in classification, and for the references. We renamed local and global calibration to next-token calibration and full-sequence calibration throughout the paper, where calibration means moment matching calibration (or f-calibration).
5–6. Baselines and the quality–diversity trade-off. We added held-out log-loss (NLL), the diversity measure used by Cao et al., for the base and tilted models (Table 1). For quality, Cao et al. (2025, Appendix E) find that MAUVE is highest when generation entropy roughly matches the model's log-loss, which is the direction in which our tilt moves the model. Temperature and truncation lower entropy at the cost of log-loss, which is well established (Zhang et al. 2020, Cao et al. 2025). Our tilt does not raise log-loss, as Table 1 confirms. The new tilt sweep (Appendix F) traces this trade-off along the tilt itself: held-out log-loss is lowest at the fitted tilt, and larger tilts lower generation entropy further at a growing cost in log-loss.
7. Claims. We removed the sentence on sycophancy and over-empathy, and the statement on larger models is now explicitly a prediction of our bounds, and not an empirical observation only. The prediction is not that larger models are always cheaper. It is a general one: a model that is more capable at a given task leaves less room for CE improvement and has lower average perplexity levels, so it needs a smaller tilt at a lower entropy level, and by Lemma 3 both lower the rejection rate. Table 1 shows this in both families. Within Qwen2.5, the held-out CE gain and the fitted tilt both shrink with model size. Within Qwen3, the 8B model has a smaller CE gain and lower log-loss than the 1.7B model. Qwen3-4B, the one exception to the size trend, has the largest CE gain in its family, hence a larger tilt and more rejections, so it shows the same mechanism at work.
Other notions of calibration, NLL and Brier. We target moment calibration of a chosen statistic, as Braverman et al. and Cao et al. do, and make no claims about notions that condition on the model's confidence. For NLL the method has a guarantee. Since α minimizes CE and α = 0 is a candidate, the tilted model's log-loss on the calibration data cannot exceed the base model's. Table 1 reports held-out NLL, which is lower than the base model's in all eight settings, and, as suggested, the Brier score. The fit does not target the Brier score, which is unchanged within its 95% interval (slightly better for Qwen2.5-7B), so it serves as an independent check.
We believe the revision addresses the presentation and evaluation concerns, and we would be grateful if the reviewer would consider raising their scores.
The review does not point to a specific error in the analysis. If a soundness concern remains, we would be grateful to hear it during the discussion, so that we can address it.