- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The title, method, and experiments of this paper seem to focus on the uncertainty estimation in LLM, which I think is probably the main scope of this work. However, the whole introduction is all about whether the “reasoning,” a term used in LLMs, is an accurate definition. From the introduction, it should be a paper discussing the philosophy of LLMs. I was trying to understand, but the introduction really confused me. Regarding the method, the authors combined several uncertainty signals to train a binary classifier to assess the correctness of reasoning.
Critical strengths
- Good perspective to model uncertainty from time series.
- This field of uncertainty estimation of LLM is interesting.
Critical weaknesses
- The motivation and problem formulation are not clear. The introduction is very confusing. The authors use a lot of space to discuss the philosophical definition of “reasoning” in LLM. I can not see the necessity to have some discussions. I also don’t know how this discussion motivates the study of uncertainty. Instead of that, I didn’t see the motivation for “tracking uncertainty”.
- Many statements and conclusions are inconsistent or even contradict each other. For example, line 22-25, the abstract states that correct traces exhibit a "steeper and less linear decline in uncertainty." But, in lines 435-439, the authors said that a "steeper and more linear slope" indicates an incorrect trace. Furthermore, lines 359-369 suggest that the relationship between linearity and correctness differs across model families, making the generalized conclusions in the abstract and discussion questionable. These inconsistencies make it difficult to understand what the actual empirical findings are.
- The paper does not establish its contribution relative to previous uncertainty-based correctness prediction methods. The main novelty seems to be temporal features from reasoning traces. However, the paper does not isolate the benefits of temporal modeling from those of supervised learning.
- The proposed method relies heavily on handcrafted features and several seemingly arbitrary design choices, without sufficient theoretical or empirical evidence. Why are these particular uncertainty measures selected? Why are the uncertainty trajectories summarized using these signals? More specifically, Appendix C (Table 3) divides each reasoning trace into 25%–50%–25% segments, but the authors provide no explanation for these particular proportions.
- The presentation and writing really need to be improved.
Questions for authors
See the weaknesses
3 comments
Thanks to the authors for their efforts to address my concerns.
Regarding concerns about motivation and the inconsistent conclusions, the authors acknowledged that the problems exist.
Regarding W3, I'm also "unsure" how reviewers are expected to understand the detailed experimental settings of the baselines when these details are not provided in the paper. The authors didn’t mention that they use DeepConf to train a classifier. As I understand it, DeepConf was originally proposed as a training-free method. The authors should include these details in the paper.
Regarding W4, the authors claim to have conducted additional experiments with more complex features. However, no corresponding results are provided in either the paper or the rebuttal to support this claim. Moreover, the response does not address my specific question regarding the choice of the 25%–50%–25% segmentation in Appendix C.
Regarding W5, I find the authors' response particularly disappointing. Comparing my reading experience with those of other reviewers is neither constructive nor appropriate. I would expect the authors to address concerns about their writing rather than question a reviewer's ability to understand the paper.
Overall, the authors didn’t address most of my concerns, and the paper needs a significant revision. So I keep my original score unchanged.
I also noticed that the submission does not include an AI Use Statement, which is mandatory under the ICLR 2027 Author Guidelines.
We thank the reviewer for the swift reply.
Regarding W3, we realize that we have not adequately communicated the experimental setup and will amend this in future revisions.
Regarding W4, the initial experiments with more complex features were performed as initial experiments and thus not included in the final version of the paper. Below we show the results of correctness prediction on GSM8K and ProntoQA for the five smaller models with all the initial features that we considered included. As the results indicate, using a more complicated feature-set doesn’t provide much in terms of correctness prediction. We realize that this analysis should have been included in the paper.
| Model | GSM8K | ProntoQA |
|---|---|---|
| Llama 3.1 | 0.764 | 0.795 |
| Llama 3.2 | 0.768 | 0.504 |
| Qwen 2.5 | 0.802 | 0.559 |
| DeepSeek R1 | 0.793 | 0.708 |
| Qwen 3 | 0.722 | 0.688 |
Regarding W5, we apologize for our previous response and acknowledge that the comparison with other reviewers was inappropriate. We did not intend to cause offense, but rather hoped to identify specific ways to improve the clarity of the paper. We appreciate the reviewer’s feedback and will take it into account in our revisions.
Based on the reviewers’ comments, we have decided to withdraw the paper to work on a significant revision (see general comment above). We thank the reviewer for the feedback and suggestions.
Thank you for taking the time to review our paper and providing some relevant critiques. We hope our response can alleviate some of your concerns regarding the paper, and that you, with our response in mind, would be willing to provide some concrete suggestions for how to improve the paper.
On your raised points:
First of all we want to acknowledge that the introduction does spend perhaps too long discussing the notion of reasoning in language models. This research started from discussions of reasoning, but in our next revision we will significantly limit this and truncate large parts of the introduction discussing this. With that being said, we believe that stating that the entire introduction concerns reasoning is a slight overstatement. Language model reasoning is primarily discussed from ll. 31 - 36 and ll. 41 - 47 in the introduction which continues to l. 92.
On the first point, this is a mistake on our part, the abstract contains the correct formulation. We will fix the mixup in the discussion section in the next revision. On the second point, as we discuss in ll. 368 - 377, this change in relationship between linearity and correctness is a function of the extent of the reasoning training. We argue that this is not an inconsistency, but a result of the model training regime.
We’re unsure what this refers to with respect to supervised fine-tuning in the context of our paper as all of the correctness prediction methods, including the baselines such as DeepConf, are implemented using a supervised classifier (i.e. the correctness prediction with e.g. DeepConf works by generating a score for each sample and then training supervised classifiers to predict the sample’s correctness using these scores). With respect to the novelty of the contribution we refer to ll. 85-92 (Novelty & Contribution) where we explicitly describe the novelty of the approach with respect to methods such as DeepConf and Self-Certainty. With respect to why the specific uncertainty measures are selected please see Section 2.2. In summary we choose these measures to get a granular picture of the uncertainty dynamics, as only looking at either aleatoric or epistemic will often obfuscate the picture.
The features are selected for the sake of simplicity and interpretability. We have also studied additional features of a much more complex nature (such as skewness, kurtosis, change-point detection, volatility, autocorrelation, cyclicality via FFT among others), but the simple features performed just as well. We therefore made the decision to use simpler features to improve the interpretability of our results.
We are sorry to hear that you found the paper hard to understand and will work on the presentation, including regarding the first point of critique. The other reviewers seemed to have an easier time understanding the paper, but we would welcome any concrete suggestions for improvements.