- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the issue of how to effectively utilize erroneous reasoning trajectories in On-Policy Distillation (OPD) and proposes a Selective On-Policy Distillation method (SELD). This method leverages a frozen teacher model and reference solution and reference answer to identify the earliest uncorrected error that leads to an incorrect final answer, concentrating distillation supervision on that error step and its predecessor step, while retaining sparse prefix supervision. In addition, the method applies non-uniform weighting to different tokens based on the distribution difference between teacher and student, as well as the overlap in potential next-token predictions, to achieve more targeted distillation supervision. Experimental results based on three Qwen3 teacher–student configurations and four mathematical reasoning benchmarks demonstrate that SELD achieves consistent performance improvements across three different training random seeds. In the main experimental setting, its Macro avg@12 metric improved by 2.54 percentage points compared to standard OPD.
Critical strengths
-
This paper focuses on the problem of supervision signal allocation in erroneous reasoning trajectories. By introducing a semantic error localization mechanism, it guides the distillation process to focus on critical erroneous steps, avoiding uniform supervision over all tokens in failed trajectories. This method is highly targeted and intuitively interpretable, and its design aligns with the step-by-step reasoning characteristics inherent in mathematical reasoning tasks.
-
Based on 159 expert-annotated erroneous reasoning trajectories, this paper evaluates the accuracy of error localization. The results show that semantic diagnosis can identify error locations more accurately compared to methods based on maximum JSD. Moreover, through multiple ablation experiments, the authors analyze the impact of error localization, prefix supervision, and token weight allocation on downstream performance. Experimental results indicate that SELD improves the Macro avg@12 metric under three different teacher–student configurations. The authors also report experimental results under three independently trained random seeds, including mean and standard deviation, and provide performance comparisons under the same computational budget. These analyses and experimental results provide substantial empirical support for the proposed method from multiple perspectives, including methodological effectiveness, training stability, and computational cost.
Critical weaknesses
- SELD completely removes the direct distillation supervision of all tokens following the first uncorrected error, but subsequent reasoning steps may still contain correct local reasoning, valuable intermediate calculations, or potential self-correction information. Directly excluding all tail supervision may result in the loss of some beneficial training signals, and the paper has not adequately analyzed the necessity of this exclusion strategy, nor does it provide a systematic investigation into whether retaining some effective tail supervision could further improve model performance.
- When a student model adopts an incorrect overall problem-solving strategy from the initial stage of reasoning, merely matching the local teacher-student distributions near the first error may not effectively correct such high-level reasoning biases. The paper has not sufficiently analyzed the effectiveness of this method in such situations, so its ability to correct errors in the overall reasoning strategy remains to be further verified.
Questions for authors
-
The suffix-restored ablation experiment reduces the supervision weight for error-transition region while introducing suffix supervision, making it difficult to determine whether the performance drop is due to the suffix supervision itself or the weakened supervision of critical error regions. Could the authors provide additional controlled experiments to distinguish the independent effects of these two factors?
-
SELD requires extra teacher semantic diagnosis for complete erroneous reasoning trajectories, increasing computational overhead. Although the paper provides comparisons under the same computational budget, the cost analysis remains limited to specific model configurations and training budgets. Could the authors further evaluate the trade-off between the additional diagnostic cost and performance gains in scenarios where the student model is weaker and error rates are higher?
3 comments
Reviewer p8rz | Part 2/2: Strategy-level errors and diagnostic costs
W2: How local supervision can address an incorrect initial strategy
Thank you for raising the important case of strategy-level errors. SELD can apply supervision at the beginning of a reasoning trajectory: when the anchor is step 0, the transition consists of step 0, the prefix is empty, and the entire regional budget of the localized loss is allocated to that transition. This directly targets the initial modeling or strategy decision.
In on-policy training, this supervision affects newly sampled rollouts through parameter updates. Learning at early states can change the student's initial strategy choices and thereby improve subsequent reasoning.
The second-round seed-taking example in Appendix G.1 involves incorrect mathematical modeling, while the shoe-pairing example in G.2 involves misunderstanding the objective. These examples illustrate that semantic localization can identify errors in strategy and task interpretation as well as local calculations.
Q2: Diagnostic overhead and gains for weaker students with higher error rates
Thank you for suggesting a closer analysis of costs and benefits in high-error-rate settings. We have added per-seed costs for Qwen3-1.7B ← Qwen3-8B. At the same 50 training steps, OPD, Weighted OPD, and SELD consume 18.57 ± 0.06, 18.94 ± 0.04, and 25.14 ± 0.43 GPU-hours, respectively. SELD's mean overhead relative to OPD is 35.4%; the original 38.3% corresponds to the seed-42 run. Under the shared 25.61 GPU-hour budget, SELD improves Macro avg@12 over OPD and Weighted OPD by 1.61 and 0.91 points.
Actual per-seed costs at the same 50 training steps (GPU-hours).
| Method | Seed 42 | Seed 43 | Seed 44 | Mean ± SD |
|---|---|---|---|---|
| OPD | 18.52 | 18.63 | 18.55 | 18.57 ± 0.06 |
| Full-rollout Weighted OPD | 18.92 | 18.98 | 18.91 | 18.94 ± 0.04 |
| SELD | 25.61 | 25.03 | 24.78 | 25.14 ± 0.43 |
Costs are measured on 8 A100-80GB GPUs and include rollout generation, teacher scoring, semantic diagnosis where applicable, and student updates. Evaluation is excluded, and diagnosis uses no separate GPUs.
Results under the shared 25.61 GPU-hour budget:
three-seed mean ± SD, Table 2.
| Method | Macro avg@12 | Macro pass@12 |
|---|---|---|
| OPD | 34.85 ± 0.35 | 60.00 ± 1.20 |
| Full-rollout Weighted OPD | 35.55 ± 0.30 | 61.00 ± 1.00 |
| SELD | 36.46 ± 0.26 | 63.06 ± 1.04 |
Diagnosis is applied to complete rollouts verified as incorrect. Of the 6,400 rollouts counted in the original submission, 1,710 entered diagnosis and 1,464 received valid anchors. Error rate, generation length, truncation rate, and diagnostic input/output lengths jointly determine the overhead. The proportion and length of complete incorrect trajectories are therefore relevant when assessing high-error-rate settings; the per-seed costs and shared-budget results above quantify the observed cost-benefit trade-off in the current configuration.
Thanks for authors' detailed reply. My concerns have been addressed. So I maintain my initial rating (3: Weak acceptance).
Reviewer p8rz | Part 1/2: Suffix supervision and coefficient controls
Thank you for your careful review and for recognizing the semantic-localization design and multi-seed evidence. Your suggestions on suffix supervision, strategy-level errors, and diagnostic costs helped us refine the analysis and add two region-coefficient controls. We address each point below.
W1, Q1: The value of suffix supervision and separate region-coefficient controls
(a) Why exclude the suffix, and how are useful local reasoning and self-correction handled?
Thank you for drawing attention to useful local reasoning and self-correction in the suffix. The motivation for suffix masking is supervision allocation: autoregressive predictions depend on the preceding history, and an erroneous prefix can lead later reasoning down an incorrect path. We therefore prioritize the neighborhood where the error develops when allocating a limited supervision budget.
SELD defines the anchor as the earliest substantive error that remains uncorrected and leads to the wrong final answer. Exploratory errors explicitly corrected before they are used are excluded from anchor selection, distinguishing persistent errors from self-correction.
Selectively retaining useful suffix segments is a worthwhile direction for further investigation. For the current regional allocation, we have added the Transition-only and Suffix-only controls below.
(b) Separating reduced transition weight from added suffix supervision
The two coefficient changes you identified should indeed be examined separately. The original Suffix-restored control changes the regional coefficients from 15%/85%/0% to 15%/68%/17%, modifying both transition and suffix supervision.
We have added two controls that each change only one regional coefficient, without renormalizing the overall loss:
All results below use seed 42, with each ablation paired against full SELD at the same training step. SELD's step-50 result is the checkpoint recorded in Table 23; its step-100 result is from Table 22.
Here, L_P, L_E, and L_S denote the normalized regional losses; the formulas describe cases with nonempty prefix and suffix regions. Relative to full SELD at the corresponding step, adding suffix supervision reduces avg@12 and pass@12 by 1.79 and 4.21 points, respectively. Reducing only the transition coefficient reduces them by 0.40 and 1.31 points. These paired comparisons show the effects of the two regional coefficient changes separately.
Available benchmark breakdowns for these seed-42 controls;
avg@12 / pass@12 (%).
The step-100 comparison with full SELD is reported at Macro level in the paired table above.
For context, the original Suffix-restored control in Table 3 combines both changes using 0.15 L_P + 0.68 L_E + 0.17 L_S. With validation-selected checkpoints, its three-seed Macro avg@12 / pass@12 is 34.27 ± 0.33 / 56.77 ± 1.20.