- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper compares 4 token certainty scores across 5 models and 3 benchmarks, separating question difficulty ranking from correctness ranking among answers to the same question. TACS uses early scores to set response counts and late scores to weight answer votes. Average accuracy increases by about 1% with 80%~ fewer generated tokens than majority voting over 256 responses.
Critical strengths
The controlled comparisons on the same responses are useful and comprehensive. I also liked the simple easy and hard question example, it makes the distinction clearer.
the accuracy and token savings are promising. I found the generous generation limits and handling of missing final answers in the diagnostics reassuring (setting these badly may cause reliability issues, especially with AIME).
Critical weaknesses
1.i think the distinction between question difficulty and answer correctness may already have been discussed to some extent in “Confidence improves self consistency in llms”, and the observations about early versus late confidence in “Deep think with confidence”. The systematic evaluation is useful, but i'm not fully convinced of the empirical novelty beyond these earlier papers.
2.IMO the best baseline for TACS is the fixed majority voting baseline the authors evaluated in the appendix. fixed 32 majority voting already reaches 78.3% at 12.5% of the fixed 256 cost, while TACS reaches 79.5% at 17.6%. TACS does better in accuracy, but also spends more. I'm not sure if the benefit of TACS (1% accuracy point) is that significant.
Questions for authors
- could these methods simply be better at understanding question difficulty than distinguishing answers because of how they work? Self certainty, negative entropy and maximum log probability all use the next token distribution. Perhaps they reflect familiarity with the question and generated text more than correctness of the specific answer. I'm not saying this invalidates the analysis, but the conclusion concerns these particular measures, not necessarily uncertainty estimation generally. This scope should be clearer in the main paper.
2.How sensitive is TACS to the calibration domain? Calibrating on BRUMO may transfer better to AIME than to other domains. The smaller average token savings on GPQA D in the table might partly reflect this, although I'm not entirely sure. I think this possible limitation should be mentioned. and again, that's okay and doesn't invalidate the method on its own, but requires practitioners to match calibration sets to their test domain, and also be aware of possible failures with OOD.
2 comments
Thanks for the additional experiments.
Q.1. i understand that you examine both targets on the same responses, but the response mostly repeats this rather than explaining what is empirically new beyond the two works I mentioned. You acknowledge the precedents, but I don't think this resolves my concern. I'm still not convinced of the empirical novelty.
Q.2. The new configuration gives about 1.17 percentage points over majority voting with 1.7%~ fewer tokens. IMO this is still a small practical improvement. Also, fixed 32 Borda already reaches 79.42% compared to 79.45% for TACS. Borda weights votes by certainty rank and accounts for almost all the gain. As you acknowledge, the additional benefit from adaptive allocation remains not established.
Q.3. I still think the main claims are too broad. The findings about early scores predicting difficulty and late scores predicting correctness concern these particular measures and settings. Familiarity is also a plausible influence on these scores. This limitation should be explicit in the main claims.
*) My question about sensitivity to the calibration domain remains unresolved . I still don't see an analysis of how changing the calibration domain affects accuracy and cost, or how much the results depend on the choice of BRUMO.
Response to Reviewer fUvT
We thank the reviewer for the constructive feedback.
We show how prediction targets affect certainty's usefulness and location. CISC [1] studies within-question discrimination, and DeepConf [2] shows the value of late confidence.
Across five models, three benchmarks, and four measures, we use the same responses to rank questions by empirical accuracy and same-question answers by correctness. Early prefixes rank questions well; short prefixes are near chance for within-question discrimination, where matched-length suffixes do better (Sections 4 and 5). Appendix A tests same- versus different-question correctness pairs and sensitivity to the question set. These findings explain why certainty can rank questions yet poorly select answers, motivating TACS's separate prefix/suffix roles.
We will clarify this contribution and acknowledge precedents for the question/answer distinction and late confidence in Sections 1, 2, and 5.
Lower-budget TACS improves overall accuracy and token cost over fixed-32 majority voting in our supplementary evaluation. This aggregate Pareto improvement is a point estimate; post-search statistical superiority remains unconfirmed. The reported 82% saving is relative to fixed-256.
We use the same five models and three benchmarks, with 512 stored responses per model/question pair, separate from the submitted 1,024-response-pool experiment. We tested 272 predefined quota/cap configurations, fixing thresholds, scoring windows, and rank-weighted voting. The most accurate adaptive configuration within the overall fixed-32 token budget has qualifying-response quota 25 and cap 96 across settings.
Maj is majority voting; Borda weights votes by certainty rank. Accuracy averages questions, then the 15 settings equally. Cost counts all observed responses, including invalid answers: normalize to fixed-256 per setting, average, then rescale to Maj@32 = 1.
TACS gains 1.17 percentage points (pp) over Maj@32 with 1.76% fewer tokens, though not in every setting. Borda@32 captures most of the gain. TACS adds 0.029 pp over Borda@32 and 0.018 pp over an expected-cost-matched fixed-budget Borda mixture; both paired intervals include zero. This evaluation-set control mixes adjacent budgets per setting, with cost-based probabilities frozen before bootstrapping.
We average 128 shared replays per question, then bootstrap questions within each dataset 2,000 times, sharing weights across methods and models. Replays are not independent questions. Against Maj@32, the nominal paired 95% interval is [0.65, 1.87] pp; the approximate simultaneous interval across all 1,633 predefined search comparisons is [−0.44, 2.79] pp. Selection used the same pool; independent validation is lacking.
With the original quota-32/cap-256 configuration in this pool, Borda gains +1.105 pp [0.568, 1.770] over majority voting on identical adaptive samples, and differs from an expected-cost-matched Borda control by −0.018 pp [−0.083, 0.055]. These nominal intervals condition on stored responses and configurations, without accounting for historical parameter selection. They support a voting benefit; an additional allocation advantage remains unestablished. We will report the intervals alongside accuracy and cost.
Yes, familiarity is plausible. Familiar knowledge and reasoning patterns may make questions easier, consistent with the observed cross-question ranking. Our conclusions apply to the tested measures and settings.
At a fixed context, a gradient-ascent step on target-token log likelihood directly in logit space increases the target's gap over every competitor. This shows how learning can strengthen continuation preferences; it neither equates inference-time certainty with familiarity nor explains all four measures.
We measure model-relative difficulty by empirical accuracy under a specified generation protocol and test same-question correctness separately. Both targets are testable without a unique quantitative definition of familiarity. We will clarify this scope in the main text and discuss the familiarity interpretation, local analysis, and limitations in the appendix.
References
[1] Taubenfeld et al., “Confidence Improves Self-Consistency in LLMs.” 2025. arXiv:2502.06233.
[2] Fu et al., “Deep Think with Confidence.” 2025. arXiv:2508.15260.