- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
A benchmark of AUROC and ECE across 80 different small (0.6B) to large (671B) models, with the twist being that it measures both classical logit uncertainties but also verbalized uncertainty, either prompting for words or for numbers.
Critical strengths
-
Benchmarks like this, while not introducing new methods, are necessary from time to time. Since nobody compared linguistic to logit uncertainties at this scale yet (to my knowledge), this might become a nice reference paper to point to. I’m happy that someone did the work at this scale.
-
The evaluation is clear and appears fair to me (e.g., Maverick as LLM judge for linguistic statements of all models, with ablations on how good this is in Appendix H). It’s also nice that details like bin density in Fig 3 are reported, which are often overlooked.
-
The paper takes multiple perspectives: Linguistic vs logits, but also Base vs Instruct vs Think models, quantization, MoE vs dense, ...
-
The text is clearly written, everything is defined properly, and I feel I could reproduce the experiments (in parts thanks to the appendix).
Critical weaknesses
-
Everything is evaluated on MMLU-Pro, which is multiple choice. That somewhat limits scope to “classical” tasks / essentially classification. It would be nice to have at least one dataset that goes beyond that, either into open-form mathematical reasoning (probably easier to implement) or agentic tasks (harder to implement, but even more interesting to the community)
-
The models are all over a year / 2-3 generations old at this point. Not a sole reason for rejection, since many paradigms (reasoning etc) stayed the same, but reduces the shelf life / future impact a little.
Questions for authors
(in the order of importance)
-
Given that you have all the raw uncertainty scores, could you report the pairwise correlations TPU vs LVU, TPU vs NVU, and NVU vs LVU? This could improve Section 3.7
-
It would be interesting to look at a dataset that the models have been trained on almost certainty. Because there, at least in theory, proper scoring rules say Base models should already be quite strong. Asking because I’m somewhat surprised by the large ECEs.
-
(If you can’t provide results on an actual reasoning benchmark in the rebuttal time), could you report subsplits of MMLU Pro into more “direct answer” and more “reasoning-like” tasks? I think there could be interesting differences to be seen there between Reasoning and non-reasoning models, if one believes this paper https://openreview.net/forum?id=0sCyk9Tr5J (so basically extend Figure 6 into separating base form Reasoning models)
-
It’s surprising that linguistic uncertainties outperform token-based uncertainties by +0.12 in AUROC across the board. I see two potential confounders that I’d like you to check: 1) Maybe the Maverick judge implicitly rates things it knows is wrong with lower uncertainties. Can you think of an experiment here to check against this? Maybe providing it with answers you know are true or false, plus hedges, and see if there’s a bias? 2) In Fig 2, it looks like think models are better or on par in Token uncertainties. Maybe you could comment on that, or stratify your analysis by especially good models, because Fig 5 (right) currently just averages over everything.
-
RE Section 3.3 and Figure 2: Could it be that Reasoning models are mostly more calibrated because they are simply more accurate? Asking because ECE (which depends on accuracy) falls dramatically while AUROC (which is independent) only increases slightly. Could you do some analysis, maybe stratified by difficulty / accuracy?
-
Especially for the scaling arguments and the MoE / non MoE, it might be nice to use not just parameter count but some sort of loss into account. Because we don’t know how much these models were trained. E.g., loss on [some related train dataset that you believe they might have been trained on] or on MMLU Pro itself. Maybe this shows something interesting, maybe it doesn’t, but it might be worth checking if you have the time.
-
It could be insightful to look at Reasoning models with Reasoning turned off (to seperate out the fact that they were just trained more, and really just look at the test-time scaling bit of Reasoning)
7 comments
Q1. Pairwise correlations between uncertainty scores
We computed per-question Spearman correlations using existing outputs, aligning score directions and retaining common valid questions:
| Configuration | TPU–LVU | TPU–NVU | NVU–LVU |
|---|---|---|---|
| Qwen3-4B | 0.286 | 0.374 | 0.319 |
| Qwen3-4B, reasoning | 0.417 | 0.493 | 0.352 |
| GPT-4.1 | 0.231 | 0.308 | 0.276 |
The positive but moderate correlations suggest that the interfaces share information while assigning meaningfully different example rankings. These results complement Figure 6, which concerns model-level metric rankings rather than individual-question agreement.
Q2. High ECE despite proper-scoring-rule training
Token-level log loss is proper for next-token prediction, whereas we evaluate confidence in answer correctness. Aggregate token likelihoods, generated numerical statements, and externally interpreted linguistic expressions do not automatically inherit calibration for this different target.
For example, a fluent incorrect answer can have high likelihood, and generating “90% confident” does not directly enforce 90% empirical correctness. Familiarity with a dataset does not remove this mismatch. Since exact training membership is also unknown for many models, we will clarify this distinction without assuming verified exposure.
Q3. Separating task type and reasoning mode
Section 3.8 already compares reasoning-oriented and knowledge-oriented subjects. We extended this analysis for the paired Qwen3-4B configurations, retaining the paper’s subject grouping:
| Configuration | Subject group | Accuracy ↑ | LVU ECE ↓ | LVU AUROC ↑ |
|---|---|---|---|---|
| Non-reasoning | Reasoning-oriented | 0.581 | 0.297 | 0.784 |
| Non-reasoning | Knowledge-oriented | 0.463 | 0.369 | 0.674 |
| Reasoning | Reasoning-oriented | 0.824 | 0.113 | 0.823 |
| Reasoning | Knowledge-oriented | 0.687 | 0.191 | 0.719 |
The reasoning-oriented subjects have higher discrimination under both configurations, so the subject-level pattern is not exclusive to reasoning-enabled inference. Enabling reasoning also improves both groups.
These are selected subject-group results, not an exhaustive partition reproducing the overall metrics. Subject labels remain proxies for reasoning demand rather than an item-level separation of reasoning from recall.
Q4. Judge knowledge bias and strong reasoning models
Judge bias. We evaluated 30 matched true/false statement pairs, each expressed with strong and weak hedges, producing 120 inputs for the existing judge. Confidence scores use a 0–100 scale; intervals use paired resampling over the 30 statement sets.
| Controlled comparison | Mean confidence difference | 95% CI |
|---|---|---|
| Strong minus weak hedge, content fixed | +31.46 | [27.18, 35.62] |
| True minus false content, hedge fixed | +8.27 | [3.61, 13.45] |
The judge responds strongly to hedging, but also shows factual-content sensitivity. This supports your concern: LVU is not a pure measurement of linguistic expression. We will state this limitation explicitly. The diagnostic does not quantify how much of the MMLU-Pro advantage is attributable to that effect.
Strong reasoning models. The original results already show that TPU can be competitive:
| Reasoning model | TPU ECE ↓ | LVU ECE ↓ | TPU AUROC ↑ | LVU AUROC ↑ |
|---|---|---|---|---|
| Qwen3-4B | 0.142 | 0.144 | 0.808 | 0.787 |
| Qwen3-14B | 0.090 | 0.084 | 0.809 | 0.752 |
| Qwen3-235B | 0.048 | 0.015 | 0.761 | 0.762 |
We additionally performed 2,000 paired bootstrap resamples for three configurations:
| Configuration | LVU − TPU ECE, 95% CI | LVU − TPU AUROC, 95% CI |
|---|---|---|
| Qwen3-4B | −0.090 [−0.103, −0.077] | +0.089 [0.070, 0.108] |
| Qwen3-4B, reasoning | +0.002 [−0.007, 0.012] | −0.021 [−0.039, −0.004] |
| GPT-4.1 | +0.092 [0.080, 0.105] | +0.064 [0.043, 0.085] |
Negative ECE differences favor LVU; positive AUROC differences favor LVU. These results reinforce a conditional conclusion: LVU’s advantage depends on the model and metric.
Q5. Does higher accuracy explain reasoning-related ECE improvements?
Higher accuracy can explain part of the ECE reduction. However, the results also show discrimination improvements that are not uniformly small. For Qwen3-4B:
| Configuration | Accuracy ↑ | TPU ECE ↓ | TPU AUROC ↑ | LVU ECE ↓ | LVU AUROC ↑ |
|---|---|---|---|---|---|
| Non-reasoning | 0.529 | 0.418 | 0.654 | 0.328 | 0.743 |
| Reasoning | 0.766 | 0.142 | 0.808 | 0.144 | 0.787 |
TPU AUROC increases from 0.654 to 0.808, indicating improved separation of correct and incorrect responses alongside better calibration.
This still does not establish an accuracy-independent causal effect: changing the configuration also changes which questions are answered correctly. We will revise Section 3.3 to describe joint improvements and examine subject-stratified results using the analysis in Q3.
Q6. Accounting for loss and training exposure
Loss could provide a useful complementary measure because parameter count alone does not capture training exposure or predictive quality. However, comparable training losses are unavailable for many models, and benchmark answer likelihood is neither equivalent to training loss nor consistently accessible through closed APIs.
We will therefore interpret scale and architecture comparisons as associations between evaluated configurations. Answer-level negative log likelihood could support a narrower analysis where probabilities are available, but would not fully control for training history.
We will also correct the architectural overstatement: non-reasoning Qwen3-30B-A3B has higher ECE but higher AUROC than Qwen3-32B. The evidence supports a metric-dependent comparison, not a general MoE advantage.
Q7. Switching reasoning off within the same checkpoint
This is a more direct way to examine inference-time reasoning than comparing separately trained checkpoints. We will specify which evaluated pairs share weights and how reasoning is enabled or disabled.
For shared checkpoints, the comparison measures the effect of the inference configuration, including changes in token budget, accuracy, and confidence expression. For different checkpoints, training and inference effects remain combined. We will distinguish these cases rather than attribute all observed differences to inference-time reasoning.
Thank you for the constructive review and for recognizing the value of this comparison. Your questions help sharpen both the analysis and the interpretation of our findings.
W1. Evaluation beyond multiple-choice tasks
MMLU-Pro was chosen for its scale, difficulty, and broad subject coverage. It includes mathematics, physics, engineering, biology, health, law, history, and other disciplines, covering both reasoning-intensive and knowledge-intensive problems. Its multiple-choice answer format therefore should not be equated with an absence of substantive reasoning.
This shared format also provides a consistent correctness criterion for comparing three uncertainty interfaces across 80 model configurations. Section 3.8 and Figure 6 already examine subject-level differences, while Appendix B extends the comparison to multi-generation baselines.
Your suggestion addresses a complementary dimension: generalization beyond constrained answer formats. Open-form mathematics would be a useful next evaluation, whereas agentic tasks introduce additional choices about whether uncertainty concerns an action or an entire trajectory. We will clarify that our findings establish broad comparisons within MMLU-Pro, without claiming transfer to these settings.
W2. Age of the evaluated models
The benchmark is intended to investigate uncertainty interfaces across model scale, reasoning modes, architectures, and quantization settings, rather than provide a permanently current leaderboard. Multiple configurations within a family support comparisons that a collection of only the latest releases would not provide.
We will make the evaluation snapshot explicit and distinguish 80 configurations from 80 independent model families. We will also qualify pooled conclusions where densely sampled families may disproportionately influence the results.