- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how the relationship between token-level stability and answer correctness varies with a model's verbalized confidence. It proposes VeTo, a training-free method for filtering sampled reasoning paths before majority voting. Stability is measured using jitter, the mean absolute difference between average token-confidence values in consecutive non-overlapping windows. If any paths receive high verbalized confidence, VeTo selects from that pool and favors lower jitter; otherwise, it favors higher-jitter paths from the full pool. Experiments span eight benchmarks across mathematical, multidisciplinary, code, and multimodal reasoning using three model families. The authors report improvements in average accuracy over the compared baselines, together with component ablations and analyses across sampling budgets and model sizes.
Critical strengths
-
Informative controlled observations of confidence dynamics. The misleading-prefix experiment starts from reasoning paths with correct final answers and injects erroneous intermediate steps, reducing ambiguity over whether failures simply reflect an inability to solve the original problems. The resulting observations illustrate that incorrect continuations can remain confident and stable, while correct continuations can exhibit substantial fluctuations. This provides a useful illustration of the limitations of token confidence and stability as proxies for correctness, motivating a more nuanced investigation of these signals.
-
Training-free and transparent path selection. VeTo does not require fine-tuning the generator or training a dedicated verifier or reward model. It operates on sampled reasoning paths using verbalized confidence and token-level scores, with an explicit filtering-and-voting procedure. This avoids additional model-training costs and makes the selection decisions straightforward to inspect.
Critical weaknesses
-
The presentation makes the main argument unnecessarily difficult to follow. Several issues obscure the meaning of the central signals and the interpretation of the supporting evidence. For example, the Introduction and Figure 1 discuss confidence and stability before providing sufficiently clear explanations, and Section 3.1 presents analyses using jitter and confidence groupings before their definitions and operational details appear in Sections 3.2-3.3. Token confidence itself also remains underspecified: Section 3.2 names as a token-confidence value without explicitly stating how it is computed. Readers must therefore infer definitions, consult later sections, and reconstruct the context of key results rather than follow a self-contained account of the motivating observations.
-
The selection rules are insufficiently motivated. It is unclear why verbalized confidence should be trusted enough to discard all low-confidence paths whenever a high-confidence path exists. Likewise, stability becoming less informative under low confidence does not imply that higher jitter should be preferred. Table 7 provides only modest support for this reversal, with a net gain of one correct answer over no filtering in each of the two settings.
-
Unclear hyperparameter selection despite substantial sensitivity. Performance depends strongly on the jitter window and filtering thresholds. For example, changing beta with alpha fixed moves Llama's pooled mathematical accuracy from 70.4% to 80.4% (Table 12). Although sensitivity analyses are provided, the paper does not explain how the main configurations were selected or whether an independent validation set was used. This leaves uncertainty about whether the reported gains transfer beyond the selected configurations.
Questions for authors
-
Clarification: token confidence. How exactly is computed? Is it the probability of the generated token before or after temperature scaling and top-p truncation?
-
Confidence gating. How often does the hard verbalized-confidence gate discard all correct candidate paths? What evidence supports excluding low-confidence paths rather than retaining them?
-
Fallback selection. How robust are the small high-jitter gains over no filtering in Table 7? Please provide paired uncertainty estimates for these comparisons.
-
Hyperparameter selection. How were the window sizes and filtering thresholds selected? Please specify the selection criterion and whether selection used an independent validation set or any of the reported test questions and labels.
Dear Reviewers FqQo and dDuE,
Thank you for your time and effort during the rebuttal phase. As the discussion period is set to close in 24 hours, we would appreciate it if you could let us know whether our rebuttal has addressed your concerns.
If there are any remaining concerns or additional feedback you would like us to consider, please do not hesitate to let us know.
Sincerely,
The authors