acceptodds
Under review as a conference paper at ICLR 2027

Certainty Is Not Just Correctness: Rethinking Token-Level Certainty in LLM Reasoning

Abstract

Token-level certainty is widely used as a proxy for correctness in LLM training and inference. However, the performance of certainty-based methods depends both on the information in certainty scores and on how those scores are used. We therefore directly assess certainty's predictive ability through controlled empirical evaluations across models and tasks. We distinguish two prediction targets: identifying questions a model is more likely to answer correctly and distinguishing correct from incorrect responses to the same question. In our experiments, certainty is generally better at identifying questions a model is likely to answer correctly than at distinguishing correct from incorrect responses to the same question. Certainty also varies systematically across token types and positions within words, reflecting local properties of words and text form. Information about question difficulty appears early in generation, while the weaker information about answer correctness is more concentrated near the end. These findings show that the information certainty provides for decisions depends on the prediction target, the model, the certainty metric, and which token positions in the response are included in aggregation. We further demonstrate the practical value of these findings for test-time compute. We allocate the number of responses using certainty early in generation and weight answer votes using certainty near the end of each response. Compared with a fixed-sampling majority-voting baseline, this approach increases overall accuracy from 78.71% to 79.54% while reducing the total cost in generated tokens by 82.4%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reasoning, token-level certainty, test-time compute

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fUvTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares 4 token certainty scores across 5 models and 3 benchmarks, separating question difficulty ranking from correctness ranking among answers to the same question. TACS uses early scores to set response counts and late scores to weight answer votes. Average accuracy increases by about 1% with 80%~ fewer generated tokens than majority voting over 256 responses.

Critical strengths

The controlled comparisons on the same responses are useful and comprehensive. I also liked the simple easy and hard question example, it makes the distinction clearer.

the accuracy and token savings are promising. I found the generous generation limits and handling of missing final answers in the diagnostics reassuring (setting these badly may cause reliability issues, especially with AIME).

Critical weaknesses

1.i think the distinction between question difficulty and answer correctness may already have been discussed to some extent in “Confidence improves self consistency in llms”, and the observations about early versus late confidence in “Deep think with confidence”. The systematic evaluation is useful, but i'm not fully convinced of the empirical novelty beyond these earlier papers.

2.IMO the best baseline for TACS is the fixed majority voting baseline the authors evaluated in the appendix. fixed 32 majority voting already reaches 78.3% at 12.5% of the fixed 256 cost, while TACS reaches 79.5% at 17.6%. TACS does better in accuracy, but also spends more. I'm not sure if the benefit of TACS (1% accuracy point) is that significant.

Questions for authors
  1. could these methods simply be better at understanding question difficulty than distinguishing answers because of how they work? Self certainty, negative entropy and maximum log probability all use the next token distribution. Perhaps they reflect familiarity with the question and generated text more than correctness of the specific answer. I'm not saying this invalidates the analysis, but the conclusion concerns these particular measures, not necessarily uncertainty estimation generally. This scope should be clearer in the main paper.

2.How sensitive is TACS to the calibration domain? Calibrating on BRUMO may transfer better to AIME than to other domains. The smaller average token savings on GPQA D in the table might partly reflect this, although I'm not entirely sure. I think this possible limitation should be mentioned. and again, that's okay and doesn't invalidate the method on its own, but requires practitioners to match calibration sets to their test domain, and also be aware of possible failures with OOD.

2 comments
AuthorsOpenReview ↗

Response to Reviewer fUvT

We thank the reviewer for the constructive feedback.

Q.1. What is new beyond prior work on difficulty, correctness, and early/late confidence?

We show how prediction targets affect certainty's usefulness and location. CISC [1] studies within-question discrimination, and DeepConf [2] shows the value of late confidence.

Across five models, three benchmarks, and four measures, we use the same responses to rank questions by empirical accuracy and same-question answers by correctness. Early prefixes rank questions well; short prefixes are near chance for within-question discrimination, where matched-length suffixes do better (Sections 4 and 5). Appendix A tests same- versus different-question correctness pairs and sensitivity to the question set. These findings explain why certainty can rank questions yet poorly select answers, motivating TACS's separate prefix/suffix roles.

We will clarify this contribution and acknowledge precedents for the question/answer distinction and late confidence in Sections 1, 2, and 5.

Q.2. Fixed-32 majority voting is accurate and cheaper. Is TACS's gain significant?

Lower-budget TACS improves overall accuracy and token cost over fixed-32 majority voting in our supplementary evaluation. This aggregate Pareto improvement is a point estimate; post-search statistical superiority remains unconfirmed. The reported 82% saving is relative to fixed-256.

We use the same five models and three benchmarks, with 512 stored responses per model/question pair, separate from the submitted 1,024-response-pool experiment. We tested 272 predefined quota/cap configurations, fixing thresholds, scoring windows, and rank-weighted voting. The most accurate adaptive configuration within the overall fixed-32 token budget has qualifying-response quota 25 and cap 96 across settings.

MethodAccuracy (%)Relative token cost (Maj@32 = 1)
Maj@3278.271.0000
Borda@3279.421.0000
TACS, quota 25 / cap 9679.450.9824

Maj is majority voting; Borda weights votes by certainty rank. Accuracy averages questions, then the 15 settings equally. Cost counts all observed responses, including invalid answers: normalize to fixed-256 per setting, average, then rescale to Maj@32 = 1.

TACS gains 1.17 percentage points (pp) over Maj@32 with 1.76% fewer tokens, though not in every setting. Borda@32 captures most of the gain. TACS adds 0.029 pp over Borda@32 and 0.018 pp over an expected-cost-matched fixed-budget Borda mixture; both paired intervals include zero. This evaluation-set control mixes adjacent budgets per setting, with cost-based probabilities frozen before bootstrapping.

We average 128 shared replays per question, then bootstrap questions within each dataset 2,000 times, sharing weights across methods and models. Replays are not independent questions. Against Maj@32, the nominal paired 95% interval is [0.65, 1.87] pp; the approximate simultaneous interval across all 1,633 predefined search comparisons is [−0.44, 2.79] pp. Selection used the same pool; independent validation is lacking.

With the original quota-32/cap-256 configuration in this pool, Borda gains +1.105 pp [0.568, 1.770] over majority voting on identical adaptive samples, and differs from an expected-cost-matched Borda control by −0.018 pp [−0.083, 0.055]. These nominal intervals condition on stored responses and configurations, without accounting for historical parameter selection. They support a voting benefit; an additional allocation advantage remains unestablished. We will report the intervals alongside accuracy and cost.

Q.3. Could these measures reflect question/text familiarity more than correctness? Should conclusions be limited to them?

Yes, familiarity is plausible. Familiar knowledge and reasoning patterns may make questions easier, consistent with the observed cross-question ranking. Our conclusions apply to the tested measures and settings.

At a fixed context, a gradient-ascent step on target-token log likelihood directly in logit space increases the target's gap over every competitor. This shows how learning can strengthen continuation preferences; it neither equates inference-time certainty with familiarity nor explains all four measures.

We measure model-relative difficulty by empirical accuracy under a specified generation protocol and test same-question correctness separately. Both targets are testable without a unique quantitative definition of familiarity. We will clarify this scope in the main text and discuss the familiarity interpretation, local analysis, and limitations in the appendix.

References

[1] Taubenfeld et al., “Confidence Improves Self-Consistency in LLMs.” 2025. arXiv:2502.06233.

[2] Fu et al., “Deep Think with Confidence.” 2025. arXiv:2508.15260.

Thanks for the additional experiments.

Q.1. i understand that you examine both targets on the same responses, but the response mostly repeats this rather than explaining what is empirically new beyond the two works I mentioned. You acknowledge the precedents, but I don't think this resolves my concern. I'm still not convinced of the empirical novelty.

Q.2. The new configuration gives about 1.17 percentage points over majority voting with 1.7%~ fewer tokens. IMO this is still a small practical improvement. Also, fixed 32 Borda already reaches 79.42% compared to 79.45% for TACS. Borda weights votes by certainty rank and accounts for almost all the gain. As you acknowledge, the additional benefit from adaptive allocation remains not established.

Q.3. I still think the main claims are too broad. The findings about early scores predicting difficulty and late scores predicting correctness concern these particular measures and settings. Familiarity is also a plausible influence on these scores. This limitation should be explicit in the main claims.

*) My question about sensitivity to the calibration domain remains unresolved . I still don't see an analysis of how changing the calibration domain affects accuracy and cost, or how much the results depend on the choice of BRUMO.

Reviewer_QFmCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks what a model's token-level confidence tells us during reasoning. The authors point out that "confidence predicts correctness" can mean two different things, telling easy questions from hard ones, or telling right answers from wrong ones for the same question. Testing five models on AIME and GPQA, they find confidence is much better at the first. They also find that difficulty shows up early in a response, while signs of correctness show up near the end. Based on this, they propose TACS, which uses early confidence to decide how many answers to sample and late confidence to weight the votes. TACS slightly improves accuracy over majority voting while using far fewer tokens.

Critical strengths
  • The main idea is simple and useful. Separating "is this question hard?" from "is this answer right?" is a helpful way to think about confidence, and the paper shows clearly why mixing the two can be somewhat not right.
  • The experiments are good in number, and the analysis includes confidence intervals.
  • TACS is easy to understand and its threshold is tuned on a separate dataset.
Critical weaknesses
  • The method combines ideas that already exist, such as using confidence at the end of a response (DeepConf) and confidence-weighted voting (Self-Certainty). In Table 22, Self-Certainty voting performs almost the same as their voting (79.63% vs 79.71%). Two related methods that also adjust the number of samples (Kim et al., 2026, Wang et al., 2024b) are cited but not compared.

  • The 82% saving is measured against 256 samples, but using just 32 samples already gets very close in accuracy at a fraction of the cost (Table 23). A fairer comparison would use the same number of samples as TACS so it seems the cost savings look bigger than they are.

  • The improvements are small and AIME has only 30 questions per year, so a single question can change the results. The authors already use confidence intervals in their analysis, so it would be easy to add them here.

  • In Appendix A.3, when both comparisons use the same set of questions, the difference is no longer significant and even reverses for one model. This should be discussed in the main paper, as it seems the main finding is weaker than the main text suggests.

  • Response length and how often the sampled answers agree are both easy signals that could predict difficulty and correctness. Without comparing to them, it is hard to know how much confidence adds.

  • All tasks are math or science questions with short final answers, and there are only 258 questions in total. It is not clear if the findings hold for coding, open-ended tasks, or areas like medical QA or other specialized domains. Some design choices for TACS were also picked using the same datasets it is tested on.

  • Even though the paper discusses what its findings mean for training, which it does not test, a limitations section would be helpful.

Comparisons with chance and other simple baselines, proper significance tests, a random-step check in Section 4.3, and results on at least one dataset with human-marked mistakes could make the paper better.

Questions for authors
  • Votes are weighted by each response's confidence rank rather than by the confidence score itself. Why choose ranks, and how do raw scores compare?
  • The threshold is set separately for each model. How sensitive are accuracy and cost to this value, and how would someone set it for a new model?
2 comments
AuthorsOpenReview ↗

Response to Reviewer QFmC (1/2)

Thank you for your feedback.

Q.1. Existing components and similar voting accuracy (79.63% vs 79.71%).

We compare certainty for cross-question ranking and within-question correctness on shared responses to guide TACS. DeepConf [1] filters and weights traces and stops low-confidence traces early; Self-Certainty [2] ranks responses for voting. In the respective settings, cross-question ranking is stronger overall: logit-gap C-Index is 0.740 versus within-question AUROC 0.615. Short prefixes support question ranking but have near-chance within-question discrimination; suffixes better distinguish answers (Section 5.2).

With the suffix voter fixed, allocation cuts cost to 0.176 of fixed-256; accuracy changes from 79.71% to 79.54% (Table 2). Voting adds about one percentage point over majority overall, with model-dependent gains. Table 22's 0.08-point gain over Self-Certainty fits the limited within-question signal; this comparison omits allocation benefits. We will clarify the design and credit reused components in Section 6.

Q.2. Does the 82% saving hold against fixed-32?

The 82% saving uses fixed-256. A lower-budget TACS improves on Maj@32:

MethodAccuracy (%)Cost (Maj@32=1)
Maj@3278.271.0000
Borda@3279.421.0000
TACS, quota 25 / cap 9679.450.9824

Supplementary results use five models, AIME24/AIME25/GPQA-Diamond, 512 stored responses per model-question pair and 128 shared replays, separate from the submitted 1,024-response pool. Accuracy averages replays per question, then questions and 15 settings equally. Cost averages setting-level ratios of all observed tokens to fixed-256; the table rescales to Maj@32. Borda uses suffix-500 logit-gap ranks.

Among 272 predefined quota/cap configurations with thresholds, windows and voting fixed, a shared quota of 25 qualifying responses and cap96 was most accurate within the overall Maj@32 budget: +1.17 percentage points (pp), 1.76% fewer tokens. This aggregate Pareto gain is not universal across settings. Gains over Borda@32 and a cost-matched Borda mixture are only 0.029 and 0.018 pp, both with paired intervals crossing zero.

Same-pool selection gives a nominal 95% interval against Maj@32 of [0.65,1.87] pp; the approximate simultaneous interval for all 1,633 comparisons is [−0.44,2.79]. Search-adjusted superiority is unconfirmed; the default is unchanged.

Q.3. Small gains, 30 AIME questions per year: confidence intervals?

Default quota32/cap256 accuracy is 79.54% [74.68%,84.08%]. Paired differences below are TACS minus comparator:

ComparatorDifference (pp)Paired 95% CI (pp)
Fixed 32, Borda+0.122[+0.036, +0.223]
Fixed 256, Borda-0.174[-0.675, +0.303]
Cost-matched Borda mixture-0.018[-0.083, +0.055]
Same samples, majority+1.105[+0.568, +1.770]

Nominal 95% intervals use 2,000 dataset-stratified question bootstraps shared across models/methods, not independent replays. The gain over fixed-32 Borda costs 40.5% more tokens. The cost-matched control mixes adjacent budgets with cost-only probabilities shared per setting and frozen before bootstrap. It matches evaluation-set expected cost, without independent deployment calibration.

Intervals support voting gains; same-cost allocation gains remain unestablished. They condition on stored responses/configurations without historical-selection correction. Crossing zero proves neither superiority nor equivalence/non-inferiority. We will report intervals with accuracy and cost.

Q.4. The common-support advantage loses significance or reverses.

Both score signal and evaluation matter. Appendix A.2 compares individual correct/incorrect pairs: logit-gap concordance is 0.801 across questions versus 0.583 within questions: difference 0.2180 [0.1584,0.2638]. Scores are unaveraged; pairwise correctness differs from ranking question accuracy.

Averaging may stabilize question scores. Cross-question evaluation includes all-correct/all-incorrect groups; within-question AUROC needs both classes. Originally, cross exceeds within in 60/60 model-dataset-metric combinations; paired 95% intervals are positive in 34/60 and all 20 model-metric averages across datasets.

Common support retains 865/1,290 groups, changing population and accuracy range. Logit-gap C-Index falls from 0.740 to 0.666 versus AUROC 0.615: difference 0.0509 [−0.0066,0.1115]. Self-certainty also crosses zero; negative entropy and maximum log probability retain positive intervals. Gemma4-26B reverses with intervals crossing zero. We will discuss this in Section 4.1. The analyses do not isolate signal/averaging effects or establish an intrinsic information advantage independent of these settings.

[1] Fu, Y., et al. "Deep Think with Confidence." 2025. [2] Kang, Z., Zhao, X., and Song, D. "Scalable Best-of-N Selection for Large Language Models via Self-Certainty." 2025.

AuthorsOpenReview ↗

Response to Reviewer QGmC (2/2)

Q.5. Do the findings extend beyond short-answer math/science to coding or medical/open-ended QA?

HumanEval/HumanEval+ gives partial support for coding: five models, 164 problems, 64 responses per model/problem. Correctness requires both base and plus tests. Logit-gap scoring, prefix/suffix-1,000 windows and thresholds were frozen before evaluation, without HumanEval labels for calibration or operating-point selection.

C denotes question-level C-Index; A denotes within-question AUROC:

ModelPrefix CSuffix CPrefix ASuffix A
Qwen3-8B0.7390.7930.4880.553
Qwen3-32B0.6470.6950.5010.601
GPT-OSS-20B0.6820.6430.5240.486
Gemma4-E4B0.7050.7360.5700.649
Gemma4-26B0.4980.4260.4430.413

Prefix C exceeds 0.5 in four models; suffix A exceeds both 0.5 and prefix A in three. Transfer varies by model.

Selecting the highest-suffix-score valid program versus a random valid program:

BudgetRandom (%)Suffix-best (%)Difference (pp), 95% CI
190.2690.26+0.00 [+0.00, +0.00]
291.7091.88+0.17 [+0.00, +0.35]
491.9292.27+0.34 [+0.04, +0.66]
891.9792.42+0.46 [+0.04, +0.88]
1691.9792.67+0.69 [+0.16, +1.24]
3291.9892.73+0.75 [-0.01, +1.52]
6491.9892.68+0.71 [-0.52, +1.89]

Differences are percentage points (pp). Nominal 95% intervals use 10,000 question-cluster bootstraps, averaging models within each problem. The prespecified primary budget32 interval crosses zero; budget2's lower bound rounds to +0.00.

Adaptive sampling stops at the target count above the frozen prefix threshold or cap64, then selects the highest-suffix-score valid program. This is suffix-1,000 Best-of-N, whereas submitted TACS uses suffix-500 rank-weighted answer voting.

Sampling policyTargetMean samplesAccuracy (%)Mean tokens
Fixed 16—16.0092.6746,614
Fixed 32—32.0092.7393,143
Fixed 64—64.0092.68186,223
Adaptive, cap 6412.5390.7713,557
Adaptive, cap 6424.0892.1019,142
Adaptive, cap 6446.8792.4228,810
Adaptive, cap 64811.8392.5544,690
Adaptive, cap 641620.6692.5770,842
Adaptive, cap 643236.8392.67115,826

Costs include all observed complete responses and average five models and 164 problems. Adaptive sampling does not improve the fixed-budget suffix-selection accuracy-cost frontier. Open-ended QA, medical QA and subjective generation remain untested.

Q.6. Why confidence ranks rather than raw scores?

Ranks use score order without assuming proportional vote weights and are invariant to strictly increasing score transformations. Responses, stopping points and costs are identical within each sampling condition. Raw weights equal suffix-500 mean logit gaps; rank weights use ascending average ranks with exponent one.

SamplingMajority (%)Raw (%)Rank (%)Rank − raw (pp), 95% CI
Fixed-25678.7179.1279.71+0.593 [0.100, 1.272]
TACS (32/256)78.4478.7979.54+0.754 [0.384, 1.188]

Supplementary results use five models, AIME24/AIME25/GPQA-Diamond, 512 stored responses per model-question pair and 128 shared replays, separate from the submitted 1,024-response pool. Accuracy averages replays per question, then questions and all 15 settings equally. Default TACS uses quota32/cap256.

Paired nominal 95% intervals use 2,000 dataset-stratified question bootstraps, shared across models/methods. Both rank-minus-raw intervals are positive; GPT-OSS-20B has a negative mean difference under fixed sampling, with an interval crossing zero. Intervals condition on stored responses/configurations without correcting historical selection.

Q.7. Threshold sensitivity and calibration for a new model?

In this setting, we shift each model's threshold by the same absolute logit-gap offset δ, fixing quota32/cap256 and prefix-1,000/suffix-500 scoring:

Offset δAccuracy (%)Cost (fixed-256=1)
−1.079.4670.133
−0.579.5260.145
079.5410.176
+0.579.5650.219
+1.079.5720.269

Overall accuracy varies by about 0.105 pp; cost changes substantially. Cost averages setting-level ratios of all observed tokens to fixed-256. Higher thresholds require more sampling to reach the quota. These descriptive averages establish neither equivalence nor an optimal threshold.

For a new model, use representative held-out calibration questions and deployment prompts/decoding settings. Test a predeclared threshold grid with other components fixed, select by a prespecified token budget and accuracy requirement, then freeze without consulting test labels. This procedure is unvalidated on additional models. We will describe it and report sensitivity results in Appendix D.1.

Reviewer_ya2GOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how token-level certainty predicts correctness in LLM reasoning tasks. Through within-question and cross-question evaluations, the authors show that certainty is more effective at distinguishing question difficulty than individual response correctness. Additional analysis on how these predictive signals vary across different stages of reasoning is provided.

Critical strengths
  1. The paper studies a more nuanced understanding of token-level certainty.

  2. Detailed analysis focusing on reasoning models.

Critical weaknesses
  1. The paper argues that there is a research gap where prior work prior work mostly evaluate confidence based methods through downstream performance and "they do not directly establish how reliably certainty predicts correctness". However, a substantial body of literature has already investigated this question by directly measuring confidence estimates against correctness labels using discrimination or calibration metrics. (Kadavath et al., 2022; Farquhar et al., 2024). For instance, P(IK )and P(true) in Kadavath et al. seem to be conceptually similar "distinguishing correct from incorrect responses to the same question" and "identifying questions a model is likely to answer correctly " in the paper, respectively. Therefore, the research gap seems to be overstated and there is insufficient comparison with existing confidence estimation methods.

  2. A major difference between the cross-question and within-question evaluations is that the cross question AUROC evaluates individual responses scores, whereas the cross-question evluates mean certainty and aggregated over multiple responses. Averaging can reduce variation and inherently improve ranking stability. Therefore, the stronger cross-question results do not necessarily establish that token-level certainty contains more predictive information about question difficulty.

Kadavath et al., Language Models (Mostly) Know What They Know

Farquhar et al., Detecting Hallucinations in Large Language Models Using Semantic Entropy

Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer ya2G

We thank the reviewer for the thoughtful and constructive feedback. Below, we address the main questions raised in the review and provide relevant clarifications.

Q.1. Prior work has directly measured confidence estimates against correctness labels. Is the research gap overstated?

We agree and will describe these studies more clearly. Kadavath et al. [1] distinguish answer-level P(True) from question-level P(IK); semantic entropy [2] measures uncertainty through semantic variation across answers.

We focus on token-level certainty from native next-token distributions. Self-Certainty [4] uses it for answer selection and weighted voting within a question. DeepConf [5] filters and weights responses and terminates low-confidence traces early. Both also analyze correctness. We ask how reliably these signals distinguish answers within a question compared with ranking questions by difficulty. Voting accuracy and compute savings alone cannot answer this because they also depend on candidate answers and decision rules.

Across five models, three benchmarks, and four metrics, we find generally stronger cross-question ranking under the respective evaluation settings, with limited within-question discrimination. Early prefixes help rank questions, while short suffixes better distinguish answers to one question. This motivates using early certainty to allocate samples across questions and late certainty to weight answers.

We will clarify this contribution in Sections 1 and 2, acknowledging CISC's within-question analysis [3] and TokUR's token-entropy and likelihood baselines [6]. Section 5 will credit DeepConf's late-confidence analyses and NeuralConf's matched head/tail comparisons [7]. We claim neither the two prediction targets nor late confidence as new.

Q.2. Averaging can reduce variation and improve ranking stability. Do stronger cross-question results establish that certainty contains more predictive information about question difficulty?

Thank you for this useful distinction. We also compared individual response scores without averaging in Appendix A.2: certainty distinguishes correct/incorrect response pairs from different questions more reliably than pairs from the same question. This separation is therefore observable without question-level averaging.

We agree that interpreting the cross-question advantage requires considering both the signal and the evaluation procedure. Cross-question evaluation can average response scores and include questions whose sampled answers are all correct or all incorrect. Averaging can stabilize rankings; including these questions broadens the range of empirical accuracies. Within-question AUROC requires both correct and incorrect answers.

The table summarizes overall logit-gap results from the submitted Appendix A, with paired 95% confidence intervals.

ComparisonWithin/same questionCross/different questionDifference [95% CI]
Original evaluation (A.1)0.6150.740+0.1248 [0.0829, 0.1677]
Common question set (A.3)0.6150.666+0.0509 [-0.0066, 0.1115]
Individual response pairs, no averaging (A.2)0.5830.801+0.2180 [0.1584, 0.2638]

The first two rows compare within-question AUROC with cross-question C-Index. Excluding questions with all-correct or all-incorrect sampled answers reduces the advantage, confirming that question support contributes. The difference interval includes zero for logit gap and self-certainty.

The last row compares individual correct/incorrect response pairs, equally weighted within each group. Different-question concordance is higher in all 60 model/dataset/metric combinations, with positive paired intervals in 47/60. However, the different target and pair weighting prevent this analysis from isolating an intrinsic information advantage or quantifying averaging's contribution.

We will clarify in Section 4.1 that the observed advantage depends on the signal, aggregation, and question support. Averaging may contribute; the common-support results demonstrate the contribution of the question set.

References

[1] Kadavath et al. Language Models (Mostly) Know What They Know.

[2] Farquhar et al. Detecting hallucinations in large language models using semantic entropy.

[3] Taubenfeld et al. Confidence Improves Self-Consistency in LLMs.

[4] Kang et al. Scalable Best-of-N Selection for Large Language Models via Self-Certainty.

[5] Fu et al. Deep Think with Confidence.

[6] Token-Level Uncertainty Estimation for Large Language Model Reasoning.

[7] Liu et al. Confidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning.

Thank you for the response. However, my main concern regarding the paper's unclear positioning within the existing literature remains unresolved, and addressing this concern would require substantial revisions to clarify the novelty and contributions relative to the extensive body of work on confidence estimation. In particular, it is still unclear how the paper's objectives of "distinguishing correct from incorrect responses to the same question" and "identifying questions a model is likely to answer correctly " differ from the concept of P(IK )and P(true) in Kadavath et al. Given that this fundamental concern has not been addressed, i have decided to main my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the Area Chair and reviewers for their careful assessment. Our replies clarify the contribution and evaluation controls, with new low-cost comparisons and question-level confidence intervals.

Q.1. What is new beyond prior confidence studies? (ya2G, fUvT, QFmC)

Using the same responses, we compare ranking questions by empirical accuracy with distinguishing correct answers within a question. Early prefixes help rank questions, but short prefixes offer little within-question discrimination; matched-length suffixes do better. This supports TACS's use of prefix certainty to allocate samples and suffix certainty to weight votes. We will clarify this target-dependent contribution in Sections 1, 2, 5, and 6, acknowledging prior work on both targets, late confidence, and weighted voting.

Q.2. Do averaging and question support explain the cross-question advantage? (ya2G, QFmC)

Appendix A shows that individual response scores distinguish correct/incorrect pairs better across questions than within a question, even without averaging. This pairwise task differs from ranking questions by empirical accuracy. Restricting both evaluations to questions with both correct and incorrect responses weakens the advantage, with some paired intervals crossing zero. We will discuss these controls in Section 4.1: the observed advantage depends on aggregation and question support, and does not establish an intrinsic information advantage independent of evaluation conditions.

Q.3. Does TACS improve on fixed-32 voting, and how reliable are the gains? (QFmC, fUvT)

We added low-budget and cost-matched comparisons with question-level bootstrap intervals, using 512 stored responses per model-question pair separately from the submitted 1,024-response-pool experiment. A lower-budget configuration achieves higher aggregate accuracy with fewer tokens than Maj@32. This Pareto improvement is a point estimate: selection used the same pool, and the full-search simultaneous interval crosses zero.

Fixed-32 Borda captures most of the accuracy gain. Paired intervals support certainty-weighted voting on identical samples; an additional allocation advantage over cost-matched Borda remains unestablished. Rank weighting also outperforms raw-score weighting overall, with model-dependent exceptions. These nominal intervals condition on stored responses and configurations without correcting historical parameter selection. We will report them with accuracy and cost, distinguishing voting from allocation gains.

The individual replies give the evidence and limitations. We will clarify the tested measures and settings in the main text and discuss the familiarity interpretation and its limits in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.