acceptodds
Under review as a conference paper at ICLR 2027

Learning to Stop without Learning to Stop: Self-Supervised Confidence Training Improves Reasoning Efficiency

18 views

Abstract

Reasoning models often generate very long reasoning traces, making inference computationally expensive. Existing approaches typically improve efficiency either through inference-time early-stopping mechanisms or by explicitly encouraging shorter reasoning during training, for example through reinforcement learning with length penalties. We show that substantial efficiency gains can instead emerge from a different kind of supervision: confidence. Using a self-supervised procedure, we fine-tune reasoning models to predict their confidence in the answer at intermediate points along their own reasoning trajectories using only 600 training problems. Confidence is used only as a training target: the loss contains no objective for reasoning length, efficiency, or stopping. At inference, the fine-tuned models use the standard generation procedure, with no confidence elicitation or early-stopping mechanism. Despite this, self-supervised confidence fine-tuning makes reasoning more efficient, reducing generated tokens by up to 25% at matched accuracy across Gemma, Qwen, Nemotron, and GPT-OSS models on mathematical, scientific, and coding reasoning benchmarks, with efficiency gains comparable to methods that explicitly optimize for shorter reasoning. Analysis of reasoning episodes further shows that confidence supervision largely preserves the base models' high-level reasoning composition rather than selectively suppressing particular behaviors. Our results suggest that efficient reasoning may emerge as a downstream consequence of learning metacognitive signals, without being directly optimized.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
LLM post-training, Reasoning efficiency, Metacognitive learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XHCCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ConfSFT, which trains reasoning models to predict confidence at intermediate reasoning states using targets derived from their own answer-token probabilities. Training supervises only confidence labels, and inference remains unchanged. Across four model families and five benchmarks, the method reduces average generated tokens by approximately 10–19%, with generally small average accuracy changes. Ablations examine supervision targets and decision-point selection.

Critical strengths
  1. Practical simplicity: Confidence targets require neither gold answers nor external judges, and deployment requires no confidence queries or stopping mechanism. This makes the approach potentially useful with little additional inference complexity.

  2. Broad empirical support: Token reductions occur across multiple model families and task domains. The shuffled-label ablation provides meaningful evidence that the correspondence between reasoning states and supervision targets matters.

Critical weaknesses
  1. Accuracy preservation is overstated: Table 5 reports negative nominal 95% accuracy intervals for several settings, including Qwen on LiveCodeBench (−1.9 ± 1.3 percentage points) and GPT-OSS on LiveCodeBench (−2.9 ± 2.3). These qualify the central “matched accuracy” claim; nonsignificant differences elsewhere do not establish equivalence.

  2. Evaluation and reproducibility remain insufficiently resolved: The confidence-interval construction and exact checkpoint-selection criteria are unclear. Qwen’s identical reported GSM8K pass@1 and pass@8 values warrant verification.

Questions for authors
  1. How were the paired confidence intervals constructed, and how should the accuracy-preservation claim be reconciled with the negative intervals in Table 5?

  2. Are Qwen’s identical GSM8K pass@1 and pass@8 values correct? Please verify the stated independent sampling procedure using per-question correct-completion counts.

  3. What exact checkpoint-selection rules were used? In particular, was A&Z selected on AIME2025, as Appendix F.2 states, despite the earlier statement excluding test-set model selection?

Reviewer_jVeqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ConfSFT, a self-supervised confidence-training procedure for improving reasoning efficiency. The current policy generates reasoning trajectories, and intermediate prefixes are selected for trial-answer probing. The geometric mean of the generated answer-token probabilities is quantized into a textual confidence target. Fine-tuning applies the loss only to the confidence-label tokens, with trajectories and targets refreshed across training rounds.

The objective contains no explicit length penalty or stopping supervision. At inference, the trained model uses ordinary generation without confidence elicitation or an external stopping controller. Across four model families and five mathematical, scientific, and coding benchmarks, the authors report average token reductions of approximately 10.3%–19.2%, with aggregate accuracy remaining close to the corresponding base models. Additional experiments examine confidence prediction, alternative supervision signals, and reasoning-episode composition.

Critical strengths

1. Intermediate answer probing and probability-based confidence have already been used in DEER [1]. Jang et al. [2], which the submission cites, also show that supervising confidence labels alone, without explicit reasoning supervision, can change reasoning length and self-verification behavior. Thus, the observation that confidence-based auxiliary supervision can alter reasoning behavior has a clear precedent. The more specific contribution of this paper is to construct labels from the model’s own answer probabilities without requiring gold answers, iteratively supervise intermediate reasoning states, and observe efficiency improvements during ordinary generation.

This contribution may be valuable, but the authors have not sufficiently isolated it. Figure 2 supports the signal’s predictive informativeness, but does not adequately explain its role as an auxiliary training target. The existing ablations support the importance of correspondence between states and labels, but do not sufficiently distinguish confidence supervision from other informative forms of state supervision. Consequently, these experiments leave unclear whether the efficiency gains arise from properties specific to confidence or from informative auxiliary state supervision more generally.

2. Prior work [2] has already shown that confidence supervision can change reasoning length and self-verification behavior. This paper further demonstrates that self-supervised confidence training at intermediate states can reduce generation length under ordinary decoding, but does not sufficiently clarify what new mechanistic insight this result provides. In particular, the existing experiments do not distinguish an improved ability to stop appropriately from an overall compression of reasoning length: does the model judge the value of continued reasoning more accurately, or does it simply generate shorter reasoning more generally? These explanations have different research implications. Therefore, although the efficiency gains have empirical value, the paper’s distinctive contribution relative to prior work requires further clarification.

References

[1] Yang, C., Si, Q., Duan, Y., Zhu, Z., Zhu, C., Li, Q., Chen, M., Lin, Z., and Wang, W. Dynamic early exit in reasoning models. In ICLR, 2026.

[2] Jang, C., Choi, M., Kim, Y., Lee, H., and Lee, J. . Verbalized confidence triggers self-verification: Emergent behavior without explicit reasoning supervision. CoRR, abs/2506.03723, 2025.

Critical weaknesses
  1. Since the binary-correctness ablation changes both label source and granularity, how can the results distinguish the benefit of probability-based confidence from that of graded, state-dependent supervision more generally?

  2. Do the results show that training selectively reduces reasoning that offers little additional benefit, or only that it shortens generations overall? What evidence distinguishes these explanations?

Questions for authors

N/A

Reviewer_mDdfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ConfSFT, which trains reasoning models to predict intermediate confidence using self-generated targets, without explicitly optimizing reasoning length. At inference, it requires no confidence queries or early-stopping mechanism. Experiments on four model families and five benchmarks show average token reductions of 10.3%–19.2%, with average accuracy close to the base models.

Critical strengths
  1. The paper is clearly structured and easy to follow. Figure 3 and Equations 2–4 explain confidence construction, training examples, and loss masking in a straightforward way.
  2. The proposed approach is simple and practically useful. Confidence targets require neither gold answers nor external judges.
  3. The experiments cover four model families and five benchmarks spanning mathematics, science, and coding. Training only on AIME problems produces token savings across these domains, with comparisons against DEER, On-Policy SFT, and A&Z providing useful context for the results.
Critical weaknesses
  1. The A&Z baseline uses a different training and checkpoint selection protocol. It is trained on DeepMath problems rather than the AIME, and its checkpoint is selected using AIME 2025 test performance. The rationale for these choices is unclear.
  2. The effect of decision point coverage is insufficiently examined. The paper compares Wait with paragraph breaks, but also changes the learning rate and number of training problems between these variants.
Questions for authors
  1. Why is A&Z trained on 10K DeepMath problems instead of the AIME data used by ConfSFT?
  2. Beyond Wait and paragraph breaks, have you tried fixed interval decision points or step detectors that do not rely on the model’s thinking style?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.