- Rating
- 2
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes Confidence-Gated Step Selection (CGSS), which masks the SFT loss for reasoning steps whose student confidence is not sufficiently below a cached teacher-confidence score. Teacher scores are computed once, student scores reuse the training forward pass, and the full trace remains in context. Across four student models and five reasoning benchmarks, CGSS improves average accuracy over vanilla SFT and several selection heuristics; an alternate teacher experiment and random/static gating ablations provide additional support.
Critical strengths
-
CGSS adds no inference cost, avoids full-vocabulary teacher distributions during training, and can be implemented as a dynamic loss mask on a fixed trace corpus.
-
Improvements appear across base and instruction-tuned models and across mathematics and science benchmarks, with sizable average gains for several backbones.
-
Matched-rate random masking and static selection help show that both the confidence signal and its dynamic recomputation matter. The alternate trace generator also reduces dependence on a single teacher.
Critical weaknesses
-
The sign-conflict result concerns one sampled token's logit under a shared vocabulary, whereas CGSS compares geometric means over whole steps, potentially across different tokenizers. It neither measures the full parameter-gradient direction nor shows that the masked steps are harmful. The paper acknowledges this gap, but the theoretical framing remains stronger than the evidence.
-
Geometric-mean token probabilities are not calibrated across models or tokenizations, and the sentence-splitting heuristic can create arbitrary steps. The main paper does not clearly report the chosen margin, retention-rate trajectories, sensitivity to the margin, or robustness to alternative segmentation.
-
Results are presented as point estimates despite stochastic training and sampling-based evaluation. AIME has very few problems, so four sampled responses do not substitute for independent training seeds. The student models are also concentrated in the Qwen family, limiting evidence for broad generality.
-
Some missing related work: e.g., Rethinking generalization in reasoning sft: A conditional analysis on optimization, data, and model capability (COLM 2026); On the Role of Reasoning Patterns in the Generalization Discrepancy of Long Chain-of-Thought Supervised Fine-Tuning (COLM 2026);
Questions for authors
-
Can the authors report multiple training seeds, confidence intervals, the selected margin, retention rates over training, and sensitivity to both the margin and step segmentation?
-
How well does the step-level confidence gap correlate with actual token-level sign conflicts or full-gradient disagreement, and how does CGSS compare with soft-target distillation using available teacher logits on a controlled subset?
Response to Reviewer h3v6 (3/3)
Response to Question 1:
Thank you for this thoughtful question. We agree that some exploration and self-correction steps are valuable in long-CoT reasoning. Our goal is not to suppress these behaviors, but to avoid indiscriminately reinforcing every sampled continuation.
1. What supervision selection changes. As you noted, CGSS selects supervision using the teacher–student confidence gap rather than a semantic category. An exploratory or self-corrective step receives direct supervision whenever Δ(sk)>γ, just like any other step. Otherwise, its direct hard-label loss is omitted, while its text remains in the conditioning context for subsequent predictions (Eq. 13). The mask itself adds no penalty that explicitly lowers the probability of that content, and a step can re-enter supervision when its updated gap exceeds the margin.
The distinction is between learning useful reasoning behaviors and repeatedly reinforcing a particular sampled realization of those behaviors. Standard SFT applies one-hot supervision regardless of the current teacher–student score relationship. CGSS instead uses this relationship to decide where to apply direct supervision, rather than treating exploration or correction as inherently undesirable.
2. Qualitative examples in Appendix C. The existing qualitative analysis illustrates different confidence relationships within a reasoning trace. Orange highlights steps where the student score exceeds the teacher score, while green marks contrasting regions. Several statements expressing checking, reconsideration, or correction appear in the green regions:
The same exploratory cue can also appear in different regions: “Alternatively, consider the Fibonacci recurrence.” is green, whereas “Alternatively, maybe use the property that consecutive Fibonacci numbers are coprime.” is orange. These examples illustrate that exploratory language does not correspond to a single confidence-gap regime; the distinction concerns the particular step and model state, not the presence of a reflection marker.
Appendix C visualizes confidence relationships rather than threshold-specific mask decisions. We will clarify this distinction in the revised appendix and emphasize that exploratory wording is not itself the reason for reducing supervision.
Response to Question 2:
Thank you for this practical question. We select γ=0.1 once on held-out validation data and then keep it fixed across the original teacher–student settings, rather than tuning a separate threshold for each pair.
1. Threshold selection. We select the margin from {0,0.05,0.10,0.15,0.20} using 500 held-out AceReason-1.1-SFT problems that do not overlap with the training data or test benchmarks. The selected value is then fixed for the four student models and two teachers in the original study. No test benchmark is used for threshold selection.
2. Sensitivity analysis. We evaluate the five candidate margins on Qwen3-4B-Base, keeping the training corpus, optimization settings, and evaluation protocol unchanged. The table reports average accuracy across the five benchmarks.
As shown in the table, CGSS outperforms Vanilla SFT at all five tested margins, with gains ranging from 2.1 to 4.0 percentage points. Average accuracy varies by 1.9 points across these settings. These results indicate that the improvement does not depend on one precisely chosen threshold and support using γ=0.1 as a practical default.
3. Recommendation for new pairs. For a new teacher–student pair, we recommend starting with γ=0.1. If adjustment is needed, a small validation-based comparison of nearby values, such as 0.05 and 0.15, is a practical alternative to a broad parameter search. This approach reuses a shared default while allowing limited adjustment for substantially different model pairs.
We will add the threshold configuration, selection procedure, and sensitivity analysis to Section 3.2 and Appendix F of the revised manuscript.