acceptodds
Withdrawn from ICLR 2027

Learning When to Sample: Confidence-Aware Selective Sampling for Efficient Chain-of-Thought Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) can achieve strong reasoning performance through chain-of-thought (CoT) reasoning, yet they often generate unnecessarily long reasoning paths that incur high inference cost. Self-consistency-based approaches push accuracy higher, but they require sampling and aggregating multiple reasoning paths, leading to substantial computational overhead. In this paper, we introduce a confidence-aware selective sampling framework that, at inference time, analyzes a reasoning trajectory to adaptively determine whether to trigger multi-path sampling to potentially obtain better performance. The framework uses trajectory-level and sentence-level linguistic features extracted from reasoning states to guide selective multi-path reasoning. We train it on MedQA and evaluate it on MedQA and under mild calibration-only transfer on MathQA, MedMCQA, and MMLU, without further fine-tuning. Experimental results show that the proposed framework maintains comparable performance to full and efficient multi-path reasoning baselines, with insignificant accuracy changes of % and %, respectively, while dramatically reducing token usage by % and %. These findings demonstrate that reasoning trajectories contain rich signals for uncertainty estimation, enabling a simple, transferable mechanism to balance accuracy and efficiency in LLM reasoning.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM efficiency, chain-of-thought, inference methods, calibration/uncertainty, reasoning, question answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_stnvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a lightweight classifier on initial CoT features to decide whether to accept the answer or invoke Dynamic Voting. Each backbone uses a MedQA-trained classifier with task-specific threshold calibration. Experiments across five LLMs and four benchmarks show reduced token usage and average inference time, with some accuracy losses.

Critical strengths
  1. The framework addresses a practical inference-cost problem through a modular design. Assessing an initial completed trajectory before invoking multi-path reasoning can avoid unnecessary sampling, and the approach integrates with an existing training-free aggregation method without modifying the underlying LLM.

  2. The evaluation covers five backbones and four datasets and includes adaptive baselines, wall-clock measurements, and component and feature ablations. Training each decision model on MedQA and transferring it to other tasks with threshold calibration provides useful evidence of applicability beyond the training dataset. These experiments substantiate the practical motivation, although they do not fully establish the value of the proposed modeling choices.

Critical weaknesses
  1. The methodological contribution appears limited. The framework combines a correctness classifier, a calibrated threshold, and an existing DV fallback. The experiments do not establish whether GRU-based temporal modeling outperforms simple confidence or length thresholds or pooled-feature classifiers. This omission is particularly consequential because Table 4 identifies reasoning length as the strongest individual signal.

  2. The accuracy-latency advantage remains insufficiently established. Table 3 does not clarify whether timing includes the additional sentence-prefix answer scoring described in Appendix A and all routing overhead, which matters where speedups are marginal. Moreover, nonsignificant accuracy differences do not establish preservation within a specified tolerance, and some appendix settings show significant losses.

Questions for authors
  1. Additional experimental evidence — incremental value of temporal modeling: Can the proposed decision network outperform simple gates based on final-answer confidence or trajectory length, and a classifier using pooled trajectory features? Please compare these approaches using the same validation calibration budget and DV fallback, include a single-CoT test reference, and report their accuracy–cost trade-offs. Given Table 4, this comparison would materially affect my assessment of the paper’s methodological contribution.

  2. Clarification — end-to-end timing: Does Table 3 include the sentence-prefix answer-option scoring described in Appendix A, feature extraction, and decision-model inference? Please specify inference batching, KV-cache reuse, and whether multi-path generation is sequential or parallel. Do the baseline timings include only the computations required by each method?

  3. Clarification — sampling and baseline configuration: Appendix E describes the initial trajectory as greedy, whereas the multi-path experiments use temperature 1.0. Is the initial greedy trajectory reused in the DV fallback and included in the ten-trajectory budget and voting pool? What stopping criteria and hyperparameters were used for DV and ReASC, and were they selected with a comparable validation budget?

  4. Clarification and additional evidence — accuracy preservation: Could the authors report numerical paired accuracy differences with confidence intervals for each model–dataset pair and clarify the meaning and aggregation of the ± values in the abstract? Given the significant losses in some appendix settings, please distinguish the absence of a statistically significant difference from evidence that degradation remains within an acceptable tolerance.

Reviewer_DqiCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper allocates additional reasoning samples using a completed initial chain of thought. A GRU with feature gating and self-attention processes sentence-level numeric and linguistic features, then accepts the initial answer or triggers dynamic voting. Routers are trained separately for each LLM on MedQA, with target-dataset threshold calibration. Experiments cover five LLMs and four multiple-choice benchmarks.

Critical strengths
  1. The motivation is practical and the intervention is coherent: identifying reliable initial answers can avoid unnecessary multi-path generation. The training and calibration requirements are explicitly described.
  2. The empirical evidence extends beyond token counts. The paper reports substantial token savings, generally small accuracy changes, and wall-clock results across models. Table 2 also supports complementary contributions from the feature families and architectural components.
Critical weaknesses
  1. Value of trajectory modeling. Table 2 establishes useful component contributions within the proposed architecture, but does not compare with a simple length/confidence router. Since length is the strongest individual signal in Table 4, the additional value of temporal modeling remains uncertain.
  2. Correctness versus sampling benefit. The router predicts initial-answer correctness rather than the improvement obtainable through additional sampling. Aggregate results support the heuristic, but do not directly establish that low-scoring examples offer greater recoverable benefit.
  3. Timing coverage. Table 3 reports wall-clock improvements, but its measurement boundary is unclear. Equation 1 and Appendix A require answer-option scoring at each sentence prefix; accounting for this work matters to the claimed efficiency.
  4. Accuracy preservation. Nonsignificant accuracy differences do not establish equivalence. The existing paired bootstrap analysis could more directly quantify the accuracy–cost trade-off. This is a qualification of the claim, rather than evidence that the method is ineffective.
Questions for authors
  1. In one representative setting, how does a simple router using length and final confidence compare under the same calibration protocol? Existing trajectories should permit this comparison.
  2. Could existing trajectories show wrong-to-correct and correct-to-wrong transitions after dynamic voting, grouped by router score, to assess net sampling benefit?
  3. Does Table 3 include sentence-prefix option scoring, feature extraction, and router inference? A representative timing breakdown and caching description would clarify this.
  4. Could the paired accuracy-difference confidence intervals be reported and interpreted against an explicit practical tolerance, using the existing bootstrap results?
Reviewer_SZxnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method for deciding whether an LLM should generate additional reasoning paths after producing an initial answer. It extracts numeric and linguistic features from each sentence of a completed chain of thought and uses a GRU with feature gating and attention to predict whether the answer is correct. Answers above a calibrated threshold are accepted, while the remaining questions are passed to dynamic voting.

A separate decision model is trained on MedQA for each LLM. Evaluation covers five LLMs and four multiple choice benchmarks, with only the decision threshold recalibrated on the target datasets. The paper reports substantial reductions in generated tokens and improvements in runtime in several settings, often with similar accuracy to the sampling baselines. Component ablations and a descriptive feature analysis provide additional evidence about the signals used by the decision model.

Critical strengths
  1. The paper addresses a practical problem with a straightforward approach. Using information from the first completed response to decide whether further sampling is needed could reduce inference cost without fine tuning the underlying LLM. The reported savings are substantial in several settings.

  2. The evaluation covers five LLMs and four datasets and includes both conventional and adaptive sampling baselines. Reporting wall clock runtime alongside generated token counts is particularly useful for understanding practical efficiency. The appendix also provides results for settings where the benefits are smaller, allowing a more complete assessment of the method.

  3. Training the decision model on MedQA and evaluating it on other datasets with only threshold recalibration is a useful test of transfer within each backbone. The component and feature ablations also help explain the design choices. The feature analysis offers useful observations about the relationship between correctness, reasoning length, entropy, and linguistic behavior, while appropriately acknowledging that these associations are not causal.

Critical weaknesses
  1. The evaluation does not clearly establish the added value of the learned routing model. A held out comparison with always accepting the initial CoT is missing, as are simple routing rules based on final answer confidence or reasoning length. The latter is particularly relevant because Table 4 identifies length as the strongest individual signal. These comparisons are needed to determine whether the proposed model offers a meaningful improvement over much simpler ways of deciding when to sample again.

  2. The full computational cost of the method is unclear. Appendix A describes additional answer scoring after each sentence prefix, which requires model computation beyond generating the initial response. Generated token counts do not capture this work, and the runtime section does not explicitly state whether all prefix scoring and feature extraction are included. Although the reported runtime improvements are promising, this clarification is necessary to assess the practical efficiency advantage.

  3. The claim of preserving accuracy needs stronger support and clearer qualification. Nonsignificant differences do not establish that any accuracy loss is sufficiently small. The appendix also shows significant decreases in some settings, including Llama 3.1 8B on MedQA and Qwen 3 32B on MMLU. These results may still represent useful efficiency tradeoffs, but the paper should distinguish them from settings where accuracy is maintained and report paired confidence intervals for the accuracy differences.

Questions for authors
  1. Could you report the held out accuracy and cost of the initial CoT alone, along with routing based on final answer confidence and reasoning length? Using the same fallback sampler and calibration procedure would help isolate the benefit of the learned model.

  2. Do the runtimes in Table 3 include every sentence prefix scoring operation, feature extraction, and decision model inference? How much of the total runtime comes from these additional operations?

  3. Could you provide paired confidence intervals for the accuracy differences against DV and ReASC, and discuss the settings with significant accuracy decreases? This would help establish the magnitude of the tradeoff rather than only whether a difference is statistically significant.

  4. How exactly is the scalar confidence feature p_n computed from the answer choice distribution? Does the scoring procedure use option letters or full answer text? This definition is central to understanding and reproducing the proposed features.

Reviewer_c2qsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a sampling method to reduce repeated reasoning of LLMs, using an auxiliary model: (1) a LLM generates a complete reasoning trajectory + answer (2) a separate classifier predicts whether that answer is correct + (other metrics e.g., confidence, entropy, length, etc.). If its prediction is below a selected threshold, the framework generates additional traces and applies voting. The results show smaller token use, with varying runtime gains and some accuracy losses.

Critical strengths

This paper's motivations are strong, and visited quite often in the existing literature: reducing the cost of reasoning collection (which is very important given the large use of reasoning traces during post-training). I believe the evaluation is thorough e.g., its choice of five model variants and 4 datasets. It reports both generated-token use and elapsed time, so I'm generally happy with its cost analysis.

Critical weaknesses
  1. Limited novelty and weak justification: The paper cites some relevant papers (REASC and Wan et al., (2026)) that propose a similar architecture e.g., Wan et al., (2026) already proposed a learned classifier to control sampling, and REASC shows that confidence can be used to decide whether to sample more trajectories from the first response. There are differences (gating, attention) that make the proposed framework stronger, which the authors aim to support through the ablation in Tab. 2, but Tab. 2 only ablates parts within the proposed classifier. I believe it would have been more sound to compare that classifier with simple sampling rules proposed in earlier works (I acknowledge that ReASC is evaluated elsewhere, but Tab.2 on its own does not really isolate the benefits of the added components. Furthermore, It seems to me that the experiments were performed using a single seed (in line 744), which makes the results less certain. (but in the abstract, a \pm interval is reported. Could you clarify where this comes from?)

  2. Evaluation: I believe the paper lacks justification of its experimental choices. For instance, why was MedQA selected for the training dataset? Several other hyperparameter choices (e.g., accuracy threshold, learning rates, etc.) are not clearly explained.

I have one or two other concerns, but these are the main concerns that would affect my rating the most.

Questions for authors

I've listed some questions I had and would like to ask you about.

  • clarification of contribution: what specific advantage does the method provide (beyond the compared baselines)? Which results demonstrate the strength of the method?
  • cost of the method: regarding the reported wallclock time (Appendix D), does this include the scoring, feature extraction etc.? or just the generation + selection.
  • clarification of experimental choices: Why was MedQA selected? How was the 500 validation set used for calibration (line 355) selected? How sensitive are the reported scores to calibration-set size, accuracy tolerance (current default 0.5 pp), and random seed? Without multiple seeds, it's difficult to assess the results clearly.

Just one minor suggestion (not a weakness!): In lines 70- 72, I believe it would be better to use a \citep command to cite the model names (e.g., gpt-oss~\citep{...}) because it's hard to read.

Reviewer_L3PROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a confidence-aware selective sampling framework that leverages trajectory-level and sentence-level linguistic features extracted from chain-of-thought (CoT) reasoning to adaptively decide whether to trigger multi-path sampling at inference time. The framework is trained on MedQA and evaluated on both MedQA and, under calibration-only transfer, MathQA, MedMCQA, and MMLU. Experiments show that the proposed method maintains accuracy comparable to full and efficient multi-path baselines (with insignificant changes of −0.40±0.57% and −0.30±0.58%, respectively), while reducing token usage by 72.0±4.8% and 37.3±8.8%.

Critical strengths

The proposed method is conceptually simple and clearly motivated, making it easy to follow and potentially adopt in practice.

The efficiency gains are substantial and practically significant, which is a meaningful contribution for deployment scenarios where inference cost is a concern.

Critical weaknesses

Reliance on the model's own linguistic cues. The decision of whether to trigger multi-path sampling depends heavily on linguistic signals from the model's CoT output. If the model is overconfident (i.e., expresses certainty even when its reasoning is incorrect) or simply verbose as a result of training artifacts, these linguistic cues may be unreliable, potentially leading to false negatives (i.e., skipping multi-path sampling when it is actually needed). The paper should discuss this vulnerability and, if possible, provide evidence that the linguistic features are not confounded by such issues.

Causal justification for the numeric features. The trajectory-level and sentence-level features used in the framework are largely statistical in nature. It is not entirely clear why these statistics serve as reliable proxies for reasoning uncertainty. A more explicit argument—perhaps backed by ablation or correlation analysis—would strengthen the motivation.

Confidence calibration. Given that the decision threshold determines when multi-path sampling is triggered, it would be natural to ask whether confidence calibration techniques could improve the precision of this decision. Even a brief discussion of this possibility would be valuable.

Generalization to more complex tasks. The current evaluation is limited to relatively short reasoning tasks. It is unclear whether the method remains effective on tasks that require extended thinking (e.g., step-by-step proofs with many intermediate steps) or tool-augmented inference, where the structure and length of the trajectory differ significantly. The authors should either discuss this limitation or provide additional experiments.

Questions for authors

Please see the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.