acceptodds
Under review as a conference paper at ICLR 2027

Tokens, Numbers, or Words? Benchmarking How Large Language Models Express Uncertainty

Abstract

As large language models (LLMs) are increasingly deployed in high-stakes applications, robust uncertainty estimation is essential for ensuring the safe and trustworthy deployment of LLMs. We present a large-scale study of uncertainty estimation in LLMs, evaluating 80 models spanning open- and closed-source families, dense and Mixture-of-Experts (MoE) architectures, reasoning and non-reasoning modes, quantization variants and parameter scales from 0.6B to 671B. Focusing on three representative single-pass methods that require no access to hidden states, including token probability-based uncertainty (TPU), numerical verbal uncertainty (NVU), and linguistic verbal uncertainty (LVU), we systematically evaluate uncertainty calibration and selective classification using the challenging MMLU-Pro benchmark, which covers both reasoning-intensive and knowledge-based tasks. Our results show that, on MMLU-Pro, LVU outperforms TPU and NVU on average and for most models, offering stronger calibration and discrimination. We also find that high accuracy does not imply reliable uncertainty, and that model scale, post-training and reasoning ability influence estimation performance, whereas the effects of quantization are small and method-dependent. Notably, LLMs exhibit better uncertainty estimates on reasoning tasks than on knowledge-heavy ones, and calibration and error ranking are only partially aligned. These findings highlight the need for multi-perspective evaluation and position LVU as a promising, though not universally superior, tool for improving the reliability of LLMs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
uncertainty estimation, calibration, LLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NiXROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

A benchmark of AUROC and ECE across 80 different small (0.6B) to large (671B) models, with the twist being that it measures both classical logit uncertainties but also verbalized uncertainty, either prompting for words or for numbers.

Critical strengths
  1. Benchmarks like this, while not introducing new methods, are necessary from time to time. Since nobody compared linguistic to logit uncertainties at this scale yet (to my knowledge), this might become a nice reference paper to point to. I’m happy that someone did the work at this scale.

  2. The evaluation is clear and appears fair to me (e.g., Maverick as LLM judge for linguistic statements of all models, with ablations on how good this is in Appendix H). It’s also nice that details like bin density in Fig 3 are reported, which are often overlooked.

  3. The paper takes multiple perspectives: Linguistic vs logits, but also Base vs Instruct vs Think models, quantization, MoE vs dense, ...

  4. The text is clearly written, everything is defined properly, and I feel I could reproduce the experiments (in parts thanks to the appendix).

Critical weaknesses
  1. Everything is evaluated on MMLU-Pro, which is multiple choice. That somewhat limits scope to “classical” tasks / essentially classification. It would be nice to have at least one dataset that goes beyond that, either into open-form mathematical reasoning (probably easier to implement) or agentic tasks (harder to implement, but even more interesting to the community)

  2. The models are all over a year / 2-3 generations old at this point. Not a sole reason for rejection, since many paradigms (reasoning etc) stayed the same, but reduces the shelf life / future impact a little.

Questions for authors

(in the order of importance)

  1. Given that you have all the raw uncertainty scores, could you report the pairwise correlations TPU vs LVU, TPU vs NVU, and NVU vs LVU? This could improve Section 3.7

  2. It would be interesting to look at a dataset that the models have been trained on almost certainty. Because there, at least in theory, proper scoring rules say Base models should already be quite strong. Asking because I’m somewhat surprised by the large ECEs.

  3. (If you can’t provide results on an actual reasoning benchmark in the rebuttal time), could you report subsplits of MMLU Pro into more “direct answer” and more “reasoning-like” tasks? I think there could be interesting differences to be seen there between Reasoning and non-reasoning models, if one believes this paper https://openreview.net/forum?id=0sCyk9Tr5J (so basically extend Figure 6 into separating base form Reasoning models)

  4. It’s surprising that linguistic uncertainties outperform token-based uncertainties by +0.12 in AUROC across the board. I see two potential confounders that I’d like you to check: 1) Maybe the Maverick judge implicitly rates things it knows is wrong with lower uncertainties. Can you think of an experiment here to check against this? Maybe providing it with answers you know are true or false, plus hedges, and see if there’s a bias? 2) In Fig 2, it looks like think models are better or on par in Token uncertainties. Maybe you could comment on that, or stratify your analysis by especially good models, because Fig 5 (right) currently just averages over everything.

  5. RE Section 3.3 and Figure 2: Could it be that Reasoning models are mostly more calibrated because they are simply more accurate? Asking because ECE (which depends on accuracy) falls dramatically while AUROC (which is independent) only increases slightly. Could you do some analysis, maybe stratified by difficulty / accuracy?

  6. Especially for the scaling arguments and the MoE / non MoE, it might be nice to use not just parameter count but some sort of loss into account. Because we don’t know how much these models were trained. E.g., loss on [some related train dataset that you believe they might have been trained on] or on MMLU Pro itself. Maybe this shows something interesting, maybe it doesn’t, but it might be worth checking if you have the time.

  7. It could be insightful to look at Reasoning models with Reasoning turned off (to seperate out the fact that they were just trained more, and really just look at the test-time scaling bit of Reasoning)

7 comments

Thank you for the constructive review and for recognizing the value of this comparison. Your questions help sharpen both the analysis and the interpretation of our findings.

W1. Evaluation beyond multiple-choice tasks

MMLU-Pro was chosen for its scale, difficulty, and broad subject coverage. It includes mathematics, physics, engineering, biology, health, law, history, and other disciplines, covering both reasoning-intensive and knowledge-intensive problems. Its multiple-choice answer format therefore should not be equated with an absence of substantive reasoning.

This shared format also provides a consistent correctness criterion for comparing three uncertainty interfaces across 80 model configurations. Section 3.8 and Figure 6 already examine subject-level differences, while Appendix B extends the comparison to multi-generation baselines.

Your suggestion addresses a complementary dimension: generalization beyond constrained answer formats. Open-form mathematics would be a useful next evaluation, whereas agentic tasks introduce additional choices about whether uncertainty concerns an action or an entire trajectory. We will clarify that our findings establish broad comparisons within MMLU-Pro, without claiming transfer to these settings.

W2. Age of the evaluated models

The benchmark is intended to investigate uncertainty interfaces across model scale, reasoning modes, architectures, and quantization settings, rather than provide a permanently current leaderboard. Multiple configurations within a family support comparisons that a collection of only the latest releases would not provide.

We will make the evaluation snapshot explicit and distinguish 80 configurations from 80 independent model families. We will also qualify pooled conclusions where densely sampled families may disproportionately influence the results.

Q1. Pairwise correlations between uncertainty scores

We computed per-question Spearman correlations using existing outputs, aligning score directions and retaining common valid questions:

ConfigurationTPU–LVUTPU–NVUNVU–LVU
Qwen3-4B0.2860.3740.319
Qwen3-4B, reasoning0.4170.4930.352
GPT-4.10.2310.3080.276

The positive but moderate correlations suggest that the interfaces share information while assigning meaningfully different example rankings. These results complement Figure 6, which concerns model-level metric rankings rather than individual-question agreement.

Q2. High ECE despite proper-scoring-rule training

Token-level log loss is proper for next-token prediction, whereas we evaluate confidence in answer correctness. Aggregate token likelihoods, generated numerical statements, and externally interpreted linguistic expressions do not automatically inherit calibration for this different target.

For example, a fluent incorrect answer can have high likelihood, and generating “90% confident” does not directly enforce 90% empirical correctness. Familiarity with a dataset does not remove this mismatch. Since exact training membership is also unknown for many models, we will clarify this distinction without assuming verified exposure.

Q3. Separating task type and reasoning mode

Section 3.8 already compares reasoning-oriented and knowledge-oriented subjects. We extended this analysis for the paired Qwen3-4B configurations, retaining the paper’s subject grouping:

ConfigurationSubject groupAccuracy ↑LVU ECE ↓LVU AUROC ↑
Non-reasoningReasoning-oriented0.5810.2970.784
Non-reasoningKnowledge-oriented0.4630.3690.674
ReasoningReasoning-oriented0.8240.1130.823
ReasoningKnowledge-oriented0.6870.1910.719

The reasoning-oriented subjects have higher discrimination under both configurations, so the subject-level pattern is not exclusive to reasoning-enabled inference. Enabling reasoning also improves both groups.

These are selected subject-group results, not an exhaustive partition reproducing the overall metrics. Subject labels remain proxies for reasoning demand rather than an item-level separation of reasoning from recall.

Q4. Judge knowledge bias and strong reasoning models

Judge bias. We evaluated 30 matched true/false statement pairs, each expressed with strong and weak hedges, producing 120 inputs for the existing judge. Confidence scores use a 0–100 scale; intervals use paired resampling over the 30 statement sets.

Controlled comparisonMean confidence difference95% CI
Strong minus weak hedge, content fixed+31.46[27.18, 35.62]
True minus false content, hedge fixed+8.27[3.61, 13.45]

The judge responds strongly to hedging, but also shows factual-content sensitivity. This supports your concern: LVU is not a pure measurement of linguistic expression. We will state this limitation explicitly. The diagnostic does not quantify how much of the MMLU-Pro advantage is attributable to that effect.

Strong reasoning models. The original results already show that TPU can be competitive:

Reasoning modelTPU ECE ↓LVU ECE ↓TPU AUROC ↑LVU AUROC ↑
Qwen3-4B0.1420.1440.8080.787
Qwen3-14B0.0900.0840.8090.752
Qwen3-235B0.0480.0150.7610.762

We additionally performed 2,000 paired bootstrap resamples for three configurations:

ConfigurationLVU − TPU ECE, 95% CILVU − TPU AUROC, 95% CI
Qwen3-4B−0.090 [−0.103, −0.077]+0.089 [0.070, 0.108]
Qwen3-4B, reasoning+0.002 [−0.007, 0.012]−0.021 [−0.039, −0.004]
GPT-4.1+0.092 [0.080, 0.105]+0.064 [0.043, 0.085]

Negative ECE differences favor LVU; positive AUROC differences favor LVU. These results reinforce a conditional conclusion: LVU’s advantage depends on the model and metric.

Q5. Does higher accuracy explain reasoning-related ECE improvements?

Higher accuracy can explain part of the ECE reduction. However, the results also show discrimination improvements that are not uniformly small. For Qwen3-4B:

ConfigurationAccuracy ↑TPU ECE ↓TPU AUROC ↑LVU ECE ↓LVU AUROC ↑
Non-reasoning0.5290.4180.6540.3280.743
Reasoning0.7660.1420.8080.1440.787

TPU AUROC increases from 0.654 to 0.808, indicating improved separation of correct and incorrect responses alongside better calibration.

This still does not establish an accuracy-independent causal effect: changing the configuration also changes which questions are answered correctly. We will revise Section 3.3 to describe joint improvements and examine subject-stratified results using the analysis in Q3.

Q6. Accounting for loss and training exposure

Loss could provide a useful complementary measure because parameter count alone does not capture training exposure or predictive quality. However, comparable training losses are unavailable for many models, and benchmark answer likelihood is neither equivalent to training loss nor consistently accessible through closed APIs.

We will therefore interpret scale and architecture comparisons as associations between evaluated configurations. Answer-level negative log likelihood could support a narrower analysis where probabilities are available, but would not fully control for training history.

We will also correct the architectural overstatement: non-reasoning Qwen3-30B-A3B has higher ECE but higher AUROC than Qwen3-32B. The evidence supports a metric-dependent comparison, not a general MoE advantage.

Q7. Switching reasoning off within the same checkpoint

This is a more direct way to examine inference-time reasoning than comparing separately trained checkpoints. We will specify which evaluated pairs share weights and how reasoning is enabled or disabled.

For shared checkpoints, the comparison measures the effect of the inference configuration, including changes in token budget, accuracy, and confidence expression. For different checkpoints, training and inference effects remain combined. We will distinguish these cases rather than attribute all observed differences to inference-time reasoning.

Reviewer_XKuWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors benchmark three black-box uncertainty estimation techniques across a large collection of open and closed-source models on MMLU-Pro. This evaluation yields a number of observations: LVU appears to be the strongest of the three methods, reasoning models are better at expressing uncertainty, etc.

Critical strengths

The scale of the evaluation here is truly staggering: 80 models are included. Though many of these are just Qwen variants, I don't remember seeing an academic benchmark at this scale. The manuscript is clear and easy to follow.

Critical weaknesses

There are issues with basically every piece of analysis in section 3. Broken down by subsection:

3.3: "REASONING MODE IMPROVES UNCERTAINTY WITHOUT NEEDING HIGHER ACCURACY" -> all reasoning models evaluated have higher accuracy. I'm confused by this claim. Where do you show that the better uncertainty estimates are independent of accuracy?

3.4: "As shown in Figure 4 (right), Qwen3-30B-A3B outperforms its dense counterpart in both reasoning and non-reasoning tasks when evaluated using LVU, achieving better calibration and more reliable selective classification. This result suggests that the conditional computation of MoE may facilitate more nuanced uncertainty estimation." -> no it doesn't. Looking at Figure 4, the non-reasoning MOE has higher ECE than the non-reasoning base model. I don't think there's much evidence at all for the claim in this section that MOEs are better uncertainty quantifiers.

3.5: "Second, higher accuracy does not imply better uncertainty estimates: some high-accuracy models (e.g., GPT-4.1) are poorly calibrated, while others (e.g., Qwen3-235B-A22B) achieve reliable uncertainty estimates despite lower accuracy" -> same issue as in 3.3. This claim, which also appears in the abstract, seems unsupported by the evidence. GPT-4.1 (ECE 0.176/0.074/0.166) is better calibrated than Qwen3-235B-A22B (0.193/0.190/0.188) in Table 3! The reasoning version of the latter model gets higher accuracy and commensurately better calibration.

3.6: LVU is reported as the strongest of the three baselines. This claim is essentially unsupported by the experiments. The LVU judge is supposed to determine a model's uncertainty by interpreting hedging language in the response. In completely "verifiable" settings like this one, however, it can also simply use its own prior knowledge to predict correctness, which is the primary goal here. The authors try to justify their choice of judge with the non-verifiable eval from Belem et al. 2024, but completely ignore the verifiable eval from the same paper, which shows worse agreement with a reference population across the board for precisely this reason. Looking at Table 3, LVU hardly varies at all compared to e.g. NVU, and I think this is why: the judge is probably just paying attention to the answer choices themselves. You wouldn't expect that procedure to vary much with the choice of underlying model. The prompt used for this does not even discourage this kind of degenerate behavior:

"From their perspective, considering the use of hedging words, what is the probability (from 0 to 100) that the statement is true? Concisely provide your response only in the following format:

You only tell the judge to "consider" the use of hedging words. I think the paper desperately needs a) the "verified" eval from Belem and b) some sort of evidence that the judge is using hedging. Maybe ask it to quote precisely the parts of the response it used to arrive at its conclusion and then ablate those words to make sure that they actually affect it. Regardless, I think the method is just fundamentally flawed.

3.7: Same issue as above. LVU is the outlier here, and I suspect that its lack of correlation with ECE says more about LVU than AUROC.

Miscellaneous issues:

TPU is also flawed. The authors use the length-normalized token probability computed across the model's entire response, even though the set of models includes reasoning ones; I don't think this quantity is meaningful over a long reasoning trace. What you'd usually see is the probability on just the answer token, or maybe of the response's answer block.

There are no error bars for any number anywhere in the paper.

This certainly isn't the set of 80 models I would have chosen; there are no models from this year, for example, and a needless number of Qwens.

There's nothing new here; none of the methods or models or benchmarks etc. are novel.

Questions for authors

See above.

4 comments

Thank you for the detailed review and for recognizing the scale and clarity of the evaluation. We address the concerns below, distinguishing corrections to specific statements from limitations on how the empirical findings should be interpreted.

W1. Reasoning improvements may reflect higher accuracy.

The role of accuracy is an important qualification. Our experiments show that reasoning configurations often improve accuracy, calibration, and discrimination together; they do not isolate the causal contribution of each factor.

However, the observations are not limited to an ECE reduction. For Qwen3-4B, TPU AUROC increases from 0.654 to 0.808 and NVU AUROC from 0.691 to 0.783. These results document improved error discrimination alongside improved accuracy.

The heading “without needing higher accuracy” is stronger than these comparisons support and should be narrowed. The empirical finding remains: the evaluated reasoning configurations often provide better uncertainty estimates under the reported metrics. Whether these improvements arise from accuracy, confidence expression, or their interaction is a separate explanatory question.

W2. The MoE comparison does not support a consistent advantage.

The non-reasoning comparison does favor different configurations depending on the metric:

ConfigurationNVU ECE ↓TPU ECE ↓LVU ECE ↓NVU AUROC ↑TPU AUROC ↑LVU AUROC ↑
Qwen3-30B-A3B0.2640.2690.2430.7330.7330.748
Qwen3-32B0.2280.1490.2180.6890.6880.705

The dense model has lower ECE, whereas the MoE model has higher AUROC. Thank you for identifying the inaccurate generalization in the text.

The appropriate conclusion is metric-dependent, rather than a consistent MoE advantage. This correction preserves the comparative result and illustrates a central observation of the study: better calibration and better discrimination need not favor the same configuration. The comparison is descriptive, not a controlled attribution to architecture alone.

W3. The Qwen3-235B/GPT-4.1 comparison contains an error.

The approximately 67% accuracy in the paragraph is incorrect. The original table gives:

ConfigurationAccuracy ↑NVU ECE ↓TPU ECE ↓LVU ECE ↓
Qwen3-235B, non-reasoning0.7280.1930.1900.188
Qwen3-235B, reasoning0.8800.0250.0480.015
GPT-4.10.7910.1760.0740.166

We apologize for this textual inconsistency. GPT-4.1 has lower ECE than non-reasoning Qwen3-235B, while reasoning-enabled Qwen3-235B has both higher accuracy and lower ECE.

This example should not be used to separate accuracy from calibration. Nevertheless, the broader observation that accuracy does not uniquely determine calibration is supported by other reported comparisons. For example, GPT-4.1 has higher accuracy than GPT-4.1-mini, 0.791 versus 0.761, but slightly higher NVU ECE, 0.176 versus 0.168. This is a descriptive counterexample, not a claim of a statistically significant difference.

W4. The linguistic judge may use factual knowledge rather than only hedging.

Your concern identifies a possible confound that the existing validation does not fully resolve. Appendix H provides evidence that the evaluator responds to linguistic confidence expressions, but its non-verifiable statements do not establish that factual knowledge has no influence on verifiable QA.

Two questions should therefore be distinguished: whether the judge is sensitive to hedging, and whether it is exclusively sensitive to hedging. Appendix H supports the former, not the latter. Likewise, a relatively narrow range of scores across models does not by itself establish either validity or contamination.

A matched true/false comparison with fixed hedging would directly investigate the concern. We do not currently have that result and therefore do not claim to have excluded factual-knowledge effects. Supporting phrase quotations would aid inspection, but would not independently establish the absence of such effects.

This limitation qualifies the interpretation of LVU’s performance. It does not, by itself, establish that the observed advantage is caused by factual knowledge or that the full evaluation is invalid. The reported LVU results remain measurements under the specified evaluator; their interpretation as purely linguistic confidence requires additional validation. TPU and NVU are computed independently of that evaluator.

W5. The LVU correlation pattern could reflect evaluator artifacts.

Evaluator effects are a possible explanation for part of the pattern. The empirical observation and its mechanism should nevertheless be separated: Figure 6 reports different model rankings under calibration and discrimination, regardless of why those differences arise.

To complement the model-level analysis, we computed per-question Spearman correlations using common valid questions and aligned score directions:

ConfigurationTPU–LVUTPU–NVUNVU–LVU
Qwen3-4B0.2860.3740.319
Qwen3-4B, reasoning0.4170.4930.352
GPT-4.10.2310.3080.276

These results show partial agreement between the interfaces rather than interchangeable example rankings. They do not identify the cause of the differences or validate the linguistic judge.

The finding is therefore descriptive: the evaluated uncertainty scores and metrics capture different rankings in practice. A mechanistic explanation remains open, and evaluator artifacts are one possibility rather than an established explanation.

W6. Whole-response TPU may be unsuitable for reasoning outputs.

Whole-response TPU and answer-only TPU measure different quantities. The former summarizes likelihood across the generated response; the latter focuses on the final answer. Our comparison evaluates the former, rather than claiming to exhaust all token-based uncertainty estimators.

Your concern is relevant because intermediate reasoning may affect whole-response scores even when the final answer is incorrect. Length normalization does not remove that distinction.

We do not have an answer-only ablation, so the reported results should not be generalized to all answer-focused token-scoring methods. However, the absence of that alternative does not invalidate the comparison with the explicitly evaluated whole-response baseline. It defines the scope of the result and identifies a useful additional baseline.

W7. Results lack uncertainty intervals.

We added 2,000 paired bootstrap resamples over existing question-level outputs for three configurations, using identical resampled questions across methods and recomputing each metric:

ConfigurationLVU − TPU ECE, 95% CILVU − TPU AUROC, 95% CI
Qwen3-4B−0.090 [−0.103, −0.077]+0.089 [0.070, 0.108]
Qwen3-4B, reasoning+0.002 [−0.007, 0.012]−0.021 [−0.039, −0.004]
GPT-4.1+0.092 [0.080, 0.105]+0.064 [0.043, 0.085]

Negative ECE differences favor LVU; positive AUROC differences favor LVU. For reasoning-enabled Qwen3-4B, the ECE interval includes zero, while AUROC favors TPU. For GPT-4.1, the two metrics favor different methods.

These checks support a conditional interpretation rather than universal LVU superiority. They cover three configurations and quantify question-sampling variability, not generation or judge variability; their statistical conclusions should not be extended to every comparison in the paper.

W8. The model collection is dated and overrepresents Qwen variants.

The collection is a documented evaluation snapshot rather than a permanently current leaderboard. Its multiple Qwen configurations serve a specific purpose: they enable within-family comparisons of scale, reasoning, architecture, and quantization.

This design also means that pooled summaries give greater weight to densely sampled families. The 80 configurations should not be interpreted as 80 independent model families, and aggregate results describe the evaluated collection rather than all current or future LLMs.

New releases would extend the coverage, but do not erase the value of the documented comparisons. The contribution includes both the evidence for these configurations and a shared protocol through which subsequent models can be evaluated.

W9. No methods, models, or datasets are novel.

The paper does not introduce a new estimator, foundation model, or QA dataset. Its intended contribution is a systematic empirical study. Although specific statements require correction and the evaluator has limitations, we respectfully believe that this contribution remains valuable:

  1. Comparative evidence: A shared evaluation of token-based, numerical, and linguistic uncertainty across 80 model configurations.
  2. Practical distinctions: Results showing that method rankings depend on the configuration and metric, so better calibration does not necessarily imply better error discrimination.
  3. A reproducible reference: Documented prompts, scoring procedures, and detailed results supporting subsequent comparisons.

Other reviewers explicitly recognize this empirical value. Reviewer Uo8n describes the work as “a useful and overdue empirical study of uncertainty quantification across a large variety of models.” Reviewer NiXR similarly states: “Benchmarks like this, while not introducing new methods, are necessary from time to time.”

These assessments do not substitute for addressing your technical concerns. They clarify why the absence of a new algorithm need not imply the absence of a contribution. We respectfully ask that the systematic comparative evidence be considered alongside the specific corrections and limitations identified in your review.

I appreciate the authors' honesty in conceding most of these points, but ultimately I think this all compromises at least three of the paper's headline findings, and all of the ones that I would have considered non-trivial: that LVU is best, that high accuracy does not imply good uncertainty estimation, and that MoE helps. It evidences serious quality control issues that I think will be hard for the authors to address in the allotted time.

I strongly believe that the manuscript should not be accepted in its current form.

Reviewer_Uo8nOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper does a large-scale evaluation of uncertainty quantification in state-of-the-art models, ablating model size, reasoning status, post-training status, uncertainty quantification method, and more. They find that (1) accuracy doesn’t track calibration (2) increasing model size, doing instruction tuning/RLHF, and using reasoning all improve calibration (3) Linguistic verbal uncertainty (using another model to identify the extent to which the primary model hedges in its response) outperforms other types of uncertainty estimation techniques.

Critical strengths

This is a useful and overdue empirical study of uncertainty quantification across a large variety of models. The methodology seems sound.

Critical weaknesses

Some of the top-level findings feel perhaps a bit obvious, but it's still useful to have a reference to point to that explicitly documents these phenomena.

Questions for authors

N/A

1 comment

Thank you for the positive assessment and for describing this as a “useful and overdue empirical study.” We appreciate your recognition of the evaluation’s breadth and empirical value.

We agree that some observations are intuitively plausible. The contribution is to document where they hold, where they fail, and how they depend on the uncertainty representation. For example, LVU’s aggregate advantage does not extend uniformly to strong reasoning models: Table 4 includes several Qwen3 reasoning configurations where TPU achieves higher AUROC. These exceptions are useful alongside the overall trends.

Following the reviews, we focus the rebuttal additions on one small judge diagnostic and analyses of existing outputs: per-question correlations between uncertainty scores and paired bootstrap intervals for selected method comparisons.

We will also correct the inaccurate Qwen3-235B-A22B accuracy example, qualify the mixed MoE results, and remove the implication that reasoning improvements have been isolated from accuracy changes. Finally, we will make the multiple-choice scope, model-snapshot limitations, and potential judge-knowledge confounding more explicit.

These changes preserve the central empirical contribution while making its interpretation more precise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.