acceptodds
Under review as a conference paper at ICLR 2027

Confidence-Gated Step Selection For Long-Cot Supervised Fine-Tuning

Abstract

Supervised fine-tuning (SFT) on long chain-of-thought (CoT) reasoning traces is an effective way to improve reasoning models. Standard SFT applies one-hot supervision to every sampled token, even in steps where current model confidence exceeds the reference score from the trace generator. Using distribution matching as an analytical reference, we show that hard- and soft-target supervision can yield opposite gradient signs for the sampled token’s logit. This observation motivates selecting supervision according to the model’s evolving confidence. We propose Confidence-Gated Step Selection (CGSS), a selective SFT method that dynamically masks reasoning-step losses using the gap between reference scores and current model confidence. Reference scores are precomputed for each step, while current scores reuse the training forward pass. The criterion operates on a fixed corpus of reasoning traces and requires neither token-level alignment nor full- vocabulary teacher distributions during SFT. Experiments across five benchmarks show improvements over vanilla SFT and heuristic selection baselines, achieving an average accuracy gain of up to 6.3 percentage points while reducing step-level confidence discrepancies, with no additional inference-time computatio

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
SFT,reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sQFbOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes Confidence-Gated Step Selection (CGSS), which masks the SFT loss for reasoning steps whose student confidence is not sufficiently below a cached teacher-confidence score. Teacher scores are computed once, student scores reuse the training forward pass, and the full trace remains in context. Across four student models and five reasoning benchmarks, CGSS improves average accuracy over vanilla SFT and several selection heuristics; an alternate teacher experiment and random/static gating ablations provide additional support.

Critical strengths
  1. CGSS adds no inference cost, avoids full-vocabulary teacher distributions during training, and can be implemented as a dynamic loss mask on a fixed trace corpus.

  2. Improvements appear across base and instruction-tuned models and across mathematics and science benchmarks, with sizable average gains for several backbones.

  3. Matched-rate random masking and static selection help show that both the confidence signal and its dynamic recomputation matter. The alternate trace generator also reduces dependence on a single teacher.

Critical weaknesses
  1. The sign-conflict result concerns one sampled token's logit under a shared vocabulary, whereas CGSS compares geometric means over whole steps, potentially across different tokenizers. It neither measures the full parameter-gradient direction nor shows that the masked steps are harmful. The paper acknowledges this gap, but the theoretical framing remains stronger than the evidence.

  2. Geometric-mean token probabilities are not calibrated across models or tokenizations, and the sentence-splitting heuristic can create arbitrary steps. The main paper does not clearly report the chosen margin, retention-rate trajectories, sensitivity to the margin, or robustness to alternative segmentation.

  3. Results are presented as point estimates despite stochastic training and sampling-based evaluation. AIME has very few problems, so four sampled responses do not substitute for independent training seeds. The student models are also concentrated in the Qwen family, limiting evidence for broad generality.

  4. Some missing related work: e.g., Rethinking generalization in reasoning sft: A conditional analysis on optimization, data, and model capability (COLM 2026); On the Role of Reasoning Patterns in the Generalization Discrepancy of Long Chain-of-Thought Supervised Fine-Tuning (COLM 2026);

Questions for authors
  1. Can the authors report multiple training seeds, confidence intervals, the selected margin, retention rates over training, and sensitivity to both the margin and step segmentation?

  2. How well does the step-level confidence gap correlate with actual token-level sign conflicts or full-gradient disagreement, and how does CGSS compare with soft-target distillation using available teacher logits on a controlled subset?

Reviewer_A27wOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies supervision selection for long-chain-of-thought supervised fine-tuning. At a fixed prefix over a common token space, the authors observe that hard-label and teacher soft-target cross-entropy have opposite derivatives for the sampled token's logit when the student assigns that token a higher probability than the teacher. Motivated by this local observation, they propose Confidence-Gated Step Selection (CGSS). Teacher geometric-mean token-probability scores are cached for each reasoning step, and training retains a step's loss only when its teacher score exceeds the current student score by a specified margin. The complete trajectory remains in the conditioning context.

Experiments cover four student models and five reasoning benchmarks, with average accuracy gains of 2.5 to 6.3 percentage points over Vanilla SFT. Random and static selection ablations support the roles of selection content and dynamic updates. The appendix reports smaller step-score discrepancies and positive results with an alternative trace generator.

Critical strengths
  1. Consistent gains supported by relevant ablations. CGSS improves over the corresponding Vanilla SFT model on all five benchmarks for each of four students, with average gains of 2.5 to 6.3 percentage points. Its average accuracy also exceeds that of the other selection methods reported in the appendix. On Qwen3-4B-Base, dynamic CGSS achieves 52.6% average accuracy, compared with 49.8% for static selection and 47.1% for random selection matched to its average step-retention rate. These results support the value of dynamic, content-dependent supervision selection. The experiments include Base and Instruct models at different scales, and positive results with an alternative trace generator provide additional evidence beyond a single setting.
  2. Practical implementation with no additional inference cost. Teacher scores are cached, student scores reuse the training forward pass, and complete trajectories remain in context. This makes the approach practical to incorporate into SFT while preserving access to the full reasoning context and adding no inference-time computation.
Critical weaknesses
  1. Missing experimental specifications limit interpretability and reproducibility. The margin gammagamma directly determines which steps CGSS retains, but its values, selection procedure, and sensitivity are not reported. Selection budgets and tuning settings for EDSP, TDST, and their reverse variants are also unclear. Matching random selection to the overall average step-retention rate is useful, but does not simultaneously control supervised-token counts and the dynamic retention schedule. Together with the absence of independent training repetitions or uncertainty estimates, these gaps leave the robustness of the gains under reasonably configured comparisons insufficiently established. This is the primary concern determining my recommendation.

  2. The connection between the step score and the theoretical motivation remains insufficiently validated. The analysis assumes a common token space, whereas the gate compares per-token geometric-mean probabilities under different tokenizers; tokenization granularity can affect score ordering. The authors acknowledge that this is a proxy, but provide no shared-text-unit normalization or same-tokenizer control. Reducing the same scalar discrepancy in the appendix demonstrates an effect on that metric, but does not independently establish mitigation of the intended gradient conflict. This mainly limits the mechanistic interpretation rather than invalidating the accuracy results.

Questions for authors
  1. Core configuration: What values of gammagamma were used, how were they selected, and was an independent validation set used? Please specify the selection rules, retention budgets, and tuning procedures for all selection baselines. How is Eq. (13) handled when no steps are retained, and is normalization performed per example or per batch?

  2. Robustness of the comparison: Can the authors provide threshold sensitivity and independent training repetitions for at least one representative setting, together with supervised-token retention curves? A random-selection control matched for supervised-token budget and a validation-selected Vanilla SFT checkpoint comparison would help distinguish selection-rule benefits from training-intensity effects.

  3. Validation of the proxy: Do gating decisions and performance gains persist with shared-text-unit normalization or in a same-tokenizer setting? This would affect my assessment of the confidence gap as a supervision-selection signal.

  4. Evaluation protocol: Please specify train-test overlap checks, validation-set construction, the OlympiadBench subset and sample count, the answer-scoring procedure, and the student model used in Table 5. These details affect reproducibility and the interpretation of generalization.

Reviewer_h3v6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies supervision selection for long-chain-of-thought SFT. Using distribution matching as an analytical reference, the authors show that hard-label and soft-target supervision can produce opposite gradient signs for the sampled token's logit when the student assigns it a higher probability than the teacher. Based on this observation, the paper proposes Confidence-Gated Step Selection (CGSS), which dynamically masks step-level losses according to the gap between cached teacher confidence scores and current student confidence scores. The complete reasoning trace remains in the context, while only selected steps contribute direct supervision. Experiments across five reasoning benchmarks show improvements over vanilla SFT and several selection baselines, with no additional inference-time computation.

Critical strengths
  1. The paper combines an analysis of SFT with empirical validation, and the presentation is careful about the scope of the theoretical claims. It provides useful insights into the difference between fitting a sampled reasoning trace and matching the teacher's distribution, and how this difference can motivate supervision selection.
Critical weaknesses
  1. The method relies on a hand-designed rule to split reasoning traces into sentence-like steps. These text segments may not correspond well to actual reasoning steps, especially when the trace contains mathematical expressions or irregular formatting. Since the confidence scores and loss masks depend on this segmentation, the robustness of the method to different segmentation rules is not sufficiently clear.
  2. The evaluation of teacher diversity and strength is still limited. Most experiments use DeepSeek-R1-0528, although the appendix also includes an experiment with gpt-oss-120B. It remains unclear how the benefit of CGSS changes with teacher quality. In particular, would a stronger teacher reduce the need for confidence-based masking, or would similar gains remain?
Questions for authors
  1. The motivation discusses exploratory guesses, self-correction, alternative reasoning branches, and transitional statements. Why should supervision on these parts be reduced? Such thinking patterns may be among the most valuable aspects of long-chain reasoning. I understand that CGSS selects steps based on confidence rather than explicitly removing these categories, but can the authors show that the criterion preserves useful exploration and self-correction rather than suppressing them?
  2. How sensitive is the method to the threshold γ\gamma? For a new teacher-student pair, how should this threshold be selected, and is there a practical selection procedure that does not require extensive tuning?
1 comment
AuthorsOpenReview ↗

Response to Reviewer h3v6 (3/3)

Q1: “Why should supervision on these parts be reduced? … I understand that CGSS selects steps based on confidence rather than explicitly removing these categories, but can the authors show that the criterion preserves useful exploration and self-correction rather than suppressing them?”

Response to Question 1:

Thank you for this thoughtful question. We agree that some exploration and self-correction steps are valuable in long-CoT reasoning. Our goal is not to suppress these behaviors, but to avoid indiscriminately reinforcing every sampled continuation.

1. What supervision selection changes. As you noted, CGSS selects supervision using the teacher–student confidence gap rather than a semantic category. An exploratory or self-corrective step receives direct supervision whenever Δ(sk)>γ\Delta(s_k)>\gamma, just like any other step. Otherwise, its direct hard-label loss is omitted, while its text remains in the conditioning context for subsequent predictions (Eq. 13). The mask itself adds no penalty that explicitly lowers the probability of that content, and a step can re-enter supervision when its updated gap exceeds the margin.

The distinction is between learning useful reasoning behaviors and repeatedly reinforcing a particular sampled realization of those behaviors. Standard SFT applies one-hot supervision regardless of the current teacher–student score relationship. CGSS instead uses this relationship to decide where to apply direct supervision, rather than treating exploration or correction as inherently undesirable.

2. Qualitative examples in Appendix C. The existing qualitative analysis illustrates different confidence relationships within a reasoning trace. Orange highlights steps where the student score exceeds the teacher score, while green marks contrasting regions. Several statements expressing checking, reconsideration, or correction appear in the green regions:

  • Checking an earlier calculation: “Wait, maybe I made a miscalculation.”
  • Reconsidering the approach: “Let's think differently.”
  • Rejecting a preceding inference: “Wait, no.”

The same exploratory cue can also appear in different regions: “Alternatively, consider the Fibonacci recurrence.” is green, whereas “Alternatively, maybe use the property that consecutive Fibonacci numbers are coprime.” is orange. These examples illustrate that exploratory language does not correspond to a single confidence-gap regime; the distinction concerns the particular step and model state, not the presence of a reflection marker.

Appendix C visualizes confidence relationships rather than threshold-specific mask decisions. We will clarify this distinction in the revised appendix and emphasize that exploratory wording is not itself the reason for reducing supervision.

Q2: “How sensitive is the method to the threshold γ? For a new teacher-student pair, … is there a practical selection procedure that does not require extensive tuning?”

Response to Question 2:

Thank you for this practical question. We select γ=0.1\gamma=0.1 once on held-out validation data and then keep it fixed across the original teacher–student settings, rather than tuning a separate threshold for each pair.

1. Threshold selection. We select the margin from {0,0.05,0.10,0.15,0.20}\{0,0.05,0.10,0.15,0.20\} using 500 held-out AceReason-1.1-SFT problems that do not overlap with the training data or test benchmarks. The selected value is then fixed for the four student models and two teachers in the original study. No test benchmark is used for threshold selection.

2. Sensitivity analysis. We evaluate the five candidate margins on Qwen3-4B-Base, keeping the training corpus, optimization settings, and evaluation protocol unchanged. The table reports average accuracy across the five benchmarks.

γ00.050.100.150.20Vanilla SFT
Avg. (%)50.751.852.651.851.048.6

As shown in the table, CGSS outperforms Vanilla SFT at all five tested margins, with gains ranging from 2.1 to 4.0 percentage points. Average accuracy varies by 1.9 points across these settings. These results indicate that the improvement does not depend on one precisely chosen threshold and support using γ=0.1\gamma=0.1 as a practical default.

3. Recommendation for new pairs. For a new teacher–student pair, we recommend starting with γ=0.1\gamma=0.1. If adjustment is needed, a small validation-based comparison of nearby values, such as 0.05 and 0.15, is a practical alternative to a broad parameter search. This approach reuses a shared default while allowing limited adjustment for substantially different model pairs.

We will add the threshold configuration, selection procedure, and sensitivity analysis to Section 3.2 and Appendix F of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.