- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the uncertainty estimation during LLMs’ reasoning. Specifically, the authors claim the uncertainty should be estimated not only from token probabilities but also from the token counts. The authors use an auxiliary model to identify which tokens are important or uncertain and use token counts to get the model confidence. However, there are some important problems, such as a lack of accurate positioning, the motivation is not grounded, and the assumptions are too strong without sufficient evidence or illustration.
Critical strengths
- Good presentations and paper writing make it easy to follow.
- The experiments have a broad coverage.
Critical weaknesses
-
The positioning of this method is not so clear. For example, in Line 129, the authors state that existing cross-model signal methods primarily focus on response-level uncertainty or ranking performance, implicitly positioning calibration as a gap that this work addresses. However, the use of cross-model signals does not determine whether a method supports or evaluates calibration. Even if some prior studies emphasize ranking, this does not establish a methodological gap in calibration. Moreover, existing cross-model methods already address calibration, including BaseCal (see figure 1 of the BaseCal paper), which the authors cite in the same paragraph and use as a baseline.
-
The motivation in sec 3.3 and 3.4 is insufficiently grounded in prior works. The authors perform an ablation study on UQAC to demonstrate the importance of token count as an uncertainty signal, which is also the source of the core idea and claim of this paper. However, this general observation is already established [1~3].
-
Too strong assumptions without sufficient supportive evidence. See Line 212 and Line 229. Especially for the dual-auxiliary assumption of a closed-source model, which is a very strong assumption and directly relevant to the deployment of this method in real applications, the author didn’t present any supporting experimental or even intuitive evidence.
-
The design choice of two DTC confidence functions lacks justification. For , why does the confidence start from 0.95 and decrease linearly and reach the 0.05 floor finally? Similarly, for , the choice of the offset k=4 is not sufficiently motivated. The authors should clarify how these numbers were determined and discuss whether they vary across different experimental conditions and data settings.
-
Lack of cost analysis and trade-off analysis. The DTC needs 1~2 extra model forwards. But the author didn’t explicitly discuss the relationship between the performance gains and the additional cost.
-
DTC’s confidence estimates depend on the chosen auxiliary model pair, but this dependence is insufficiently justified. Disagreement may reflect differences in training domains and/or knowledge rather than unreliable reasoning.
[1] Trace Length is a Simple Uncertainty Signal in Reasoning Models
[2] Lexical hints of accuracy in LLM reasoning chains
[3] Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens
Questions for authors
-
Does DTC also provide good AUROC ranking signals? Did the author compare the AUROC performance with several baselines, for example, the methods mentioned in related work?
-
DeepSeek v3.2 is still an open-source model. I understand it may not be easy, but it will be better if the authors can test DTC on closed-source models, such as ChatGPT, Claude, or Gemini.
3 comments
Response 2
W1: The positioning of this method is not so clear.
ECE is widely seen as a core metric of calibration in uncertainty quantification [1]. Existing cross-model methods rarely treat it as the goal to improve, and they often do not even report ECE. So it is reasonable to treat calibration as a gap in this line of work. At the same time, we agree that Related Work is not clear enough: it does not separate BaseCal from DTC. We will add this in the revision.
W2: The motivation in Sec. 3.3 and 3.4 is insufficiently grounded in prior works.
We agree that the papers you mention already discuss token counts. The starting point is different. Those papers mainly look at token counts through AUROC / ranking. We start from calibration, and we argue that a count signal that reflects how reliable a reasoning path is can help reduce ECE.
W3: Cost analysis.
DTC does not regenerate traces with the auxiliary model. In the white-box setting, we fix one smaller auxiliary model and run teacher forcing once to compute JSD. This extra cost is small compared with generating the trace. On 1× L40 GPU, using the Transformers package with Qwen2.5-7B-Instruct, the mean time per sample is:
| Mean time per sample (s) | |
|---|---|
| Trace generation | 21.86 |
| Auxiliary teacher-forcing + JSD | 0.16 |
Teacher forcing on the auxiliary model plus JSD is much cheaper than trace generation, and the absolute time is also low.
W4: Disagreement may reflect training or knowledge differences rather than unreliable reasoning.
We agree that looking at the types of divergent tokens helps clarify what the signal is. On Qwen2.5-7B-Instruct and MATH-500, we use the same as in the paper. We then group tokens that mainly reflect writing style or notation — discourse markers, reasoning verbs, task words, LaTeX, and punctuation / space — into one style class. This class is 10.2% of all tokens, but only 5.3% of divergent tokens. Most divergent tokens are content words and numbers (48.4% of the divergent set, vs 34.0% of all tokens). So DTC does not mainly select tokens that differ by model-specific reasoning style.
Refs:
[1] On Calibration of Modern Neural Networks, ICML 2017
Thank you for the response. The rebuttal leaves some of my original concerns unanswered, particularly W4 on the choice of confidence functions. I also find the responses on the paper’s positioning and the interpretation of cross-model disagreement insufficient. I therefore maintain my original rating of 1.
Response 1
Thanks for your constructive comments and the time you dedicated to reviewing our work. Below we answer your questions first, then we reply to the weaknesses.
Q1: Lack of baselines and AUROC of DTC.
The main paper uses ECE as the main metric (calibration is a core measure in uncertainty quantification [1], and it is also the main metric of UQAC). So the experiments use calibration-oriented methods such as UQAC as baselines. We did not include methods that mainly target AUROC / ranking. We agree that those methods should still be compared on AUROC.
We add three baselines that can score existing traces directly: Self-Certainty, DeepConf, and negative generation length. As shown in Figure 14, when n grows, the best ECE stays almost the same, while the best AUROC goes up with n. So this AUROC comparison uses n=20, and we re-select θ on the same validation set. AUROC on four datasets and Qwen2.5-Instruct 7B / 14B / 32B is:
DTC can provide a useful AUROC ranking signal: it is competitive with Cmean and better than the other baselines.
Q2: Test the method on closed-source models.
We are sorry that we cannot access the models you mentioned from our country. DeepSeek is already comparable in ability to many closed-source models, so we believe our results are still reliable.