- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper addresses the collapse of OPSD on math reasoning tasks, supported by a pilot study showing that the reproduced SDPO and SRPO exhibit rising entropy, decreasing response length, and validation collapse. To address this, the paper proposes STEPS with three components: (1) localizing correctness at the token level — applying forward KL to key spans or reverse KL to error spans, with one of the two branches chosen per run; (2) decaying the KL loss weight and fusing it with the RL loss; (3) switching to pure GRPO after step 40. Experiments are conducted on math-related benchmarks and the out-of-distribution GPQA-D.
Critical strengths
- The method is presented clearly; the figures are well made and easy to read; the writing is internally consistent.
- The theoretical analysis has clearly stated boundaries, which gives the paper a degree of theoretical depth.
- The experiments are extensive. They show that the method is highly training-efficient, with rapid early validation gains.
Critical weaknesses
-
STEPS largely depends on the granularity and annotator reliability, while span-localization quality is unverified on long rollouts and in agentic domains.
-
Among the method's three components, span localization and KL direction have bounded theoretical support, but the weighted loss, the decay schedule, and switching to pure GRPO after step 40 remain heuristic design choices with many hyperparameters.
-
In the cross-family experiments, Llama family falls below SDPO once GPQA-D is excluded. STEPS exhibits a notable ability to retain held-out accuracy on GPQA-D, which is not supported by algorithm design or theory.
-
All experiments are conducted on the OpenThoughts-math training domain, whereas the baselines it compares against, e.g., SDPO, were originally reported with larger gains over GRPO on SciKnowEval, LiveCodeBench, and agentic tool use.
Questions for authors
- What are the results of the decoupled ablation (λ ≡ 0.5 with ρ ≡ 1) and a decay-schedule window (e.g. 20,40,80,120) sweep?
- How should we understand the significant held-out retention that STEPS shows on GPQA-D?
- What are the final response-length and entropy trajectories over training? From Figure 1, the response length appears to grow much more than under GRPO, while the gains reflected on the benchmarks seem modest, is there an overthinking problem?
6 comments
Response to Reviewer oBEt (2/4)
W2 & Q1: Heuristic components; decoupled ablation and decay-window sweep.
Thank you for the suggestion. We ran both experiments on Qwen3-1.7B (RKL on error spans, ). The decoupled run with and keeps the KL term active while fully restoring GRPO on the selected spans:
| Configuration | AVG | Δ vs. GRPO | ||
|---|---|---|---|---|
| GRPO | 0 | 1 | 57.73 | — |
| Constant KL, no span GRPO | 0.5 | 0 | 54.85 | −2.88 |
| Constant KL, full span GRPO (new) | 0.5 | 1 | 56.74 | −0.99 |
| STEPS (decay + GRPO restoration) | 60.11 | +2.38 |
Restoring span GRPO helps (54.85 → 56.74), but at the tested constant weight of 0.5 the model remains below GRPO.
For the window sweep, we keep the decay start at step 10 and end distillation at different steps:
| Hand-off step | 20 | 40 (default) | 80 | 120 |
|---|---|---|---|---|
| AVG | 59.12 | 60.11 | 58.71 | 57.52 |
Among the four hand-off steps tested, step 40 achieved the highest AVG. Steps 20, 40, and 80 all outperformed GRPO, while step 120 scored slightly below it (57.52 vs. 57.73).
We agree that the weighting, decay and hand-off are design choices that do not follow from the theory. They follow one principle: privileged guidance is useful early as a correction signal and should then be withdrawn so that verifier-grounded learning takes over. We fixed the defaults ( and the step-40 hand-off) before running these sweeps, and they are shared across all models. Better ways to time the KL annealing and couple it with GRPO are a direction worth exploring in future work.
W3 & Q2: Llama-3.2-3B without GPQA-D, and GPQA-D retention.
We agree that SDPO is competitive on Llama-3.2-3B-Instruct. Without GPQA-D, the three-benchmark math average is 30.33 for SDPO and 30.10 for STEPS, a gap of 0.23 points that comes mainly from MATH-500 (0.9 points), and both are above GRPO (29.43). SDPO places a full correct sibling rollout in the teacher context and distills over all tokens (Table 8). For a 3B model without native long-form reasoning, imitating a complete correct sibling solution can give a strong surface-level hint on standard problems such as MATH-500, and because Llama responses are short, SDPO does not show the entropy surge and length collapse that it exhibits in our long-reasoning runs. On every model with long-form reasoning, STEPS leads both GRPO and SDPO even when GPQA-D is excluded:
| Math-only average | STEPS | GRPO | SDPO |
|---|---|---|---|
| Qwen3-8B (4 benchmarks) | 87.31 | 84.98 | 70.07 |
| Qwen3-1.7B (4 benchmarks) | 65.47 | 62.87 | 41.68 |
| SmolLM3-3B (3 benchmarks) | 78.15 | 75.97 | 55.54 |
On GPQA-D, the clearest retention effect is at 8B, where GRPO drops from the base model's 58.27 to 53.85 while STEPS stays at 58.33. Our new controls indicate that localization matters here: on Qwen3-8B, all-token FKL with the same schedule lowers GPQA-D to 52.21, below both GRPO and STEPS, and on Qwen3-1.7B a teacher–student-disagreement mask of the same size lowers it to 36.55 (vs. 38.64 for STEPS). This pattern is consistent with prior reports that whole-response self-distillation can raise in-domain math accuracy while hurting other domains: Li et al. (arXiv:2610.02700) find that standard OPSD on Qwen3-8B raises the math average from 61.8 to 64.7 but lowers GPQA-Diamond from 64.1 to 61.7. Conversely, math reasoning gains can transfer to science, as Huan et al. (arXiv:2507.00432) show with math-only RL on Qwen3-14B-Base (GPQA from 42.6 to 57.7).
Two further aspects of the design plausibly contribute. First, the span mask and finite schedule bound cumulative privileged-gradient exposure (Eq. 4; Proposition 5), and in the token-level diagnostic of Figure 6 the mean absolute log-probability shift relative to GRPO on unselected tokens is 0.00052 nats under STEPS versus 0.02425 under all-token guidance. Second, the teacher is conditioned only on a coarse type label such as case_split or missed_case (Appendix C.3), not on a verified solution trace (RLSD) or a correct sibling rollout (SDPO).
Response to Reviewer oBEt (3/4)
W4: Single training domain; SDPO's reported gains in other domains.
Thank you for raising this. We added experiments in two of the domains where SDPO reported gains.
For code, we trained Qwen3-8B (non-thinking) with GRPO, SDPO and STEPS using SDPO's released data pipeline and evaluation (LiveCodeBench v6, 131 problems from 2025.02–2025.05; mean@4 test-case pass rate, SDPO's default metric). In SDPO's released LiveCodeBench scripts, SDPO uses a PPO mini-batch size of 1 while the GRPO baseline uses 8 with the same train batch of 32, so SDPO performs 32 optimizer updates per rollout batch against 4 for GRPO. We used a mini-batch size of 8 for all methods:
| Method | Step 30 | Step 60 | Step 90 | Step 120 |
|---|---|---|---|---|
| GRPO | 57.73 | 62.22 | 63.40 | 64.72 |
| SDPO | 58.83 | 61.97 | 63.84 | 64.72 |
| STEPS-FKL (strong annotator) | 59.76 | 63.27 | 65.19 | 66.14 |
| STEPS-FKL (online-self) | 59.15 | 63.08 | 65.21 | 65.92 |
| SDPO, original mini-batch 1 (reference) | 62.92 | 63.38 | 65.39 | 64.23 |
SDPO performs on par with GRPO, and both STEPS variants are ahead of GRPO and SDPO at all four reported checkpoints under matched optimizer updates (+1.42 and +1.20 over GRPO at step 120). As an additional reference, SDPO with its original mini-batch size of 1 leads at steps 30–90, by 3.16 points over the best STEPS variant at step 30 and by at most 0.18 points at steps 60 and 90, which reproduces its early advantage; at step 120 it reaches 64.23, below both STEPS variants (66.14 and 65.92).
For agentic tool use, we followed SDPO's ToolAlpaca setup (Qwen3-8B, non-thinking, avg@16, best accuracy within 1 h and 5 h of wall-clock training) and ran STEPS with online self-annotation, treating each tool call as an annotation segment. Baseline numbers are taken from SDPO's Table 3:
| Qwen3-8B, ToolAlpaca (avg@16) | Mini-batch | 1 h | 5 h |
|---|---|---|---|
| Base | — | 57.5 | 57.5 |
| GRPO (on-policy) | 32 | 60.2 | 65.7 |
| GRPO (off-policy) | 8 | 64.9 | 67.7 |
| SDPO (on-policy) | 32 | 68.0 | 68.5 |
| STEPS-FKL (online-self) | 32 | 67.8 | 71.2 |
STEPS is comparable to SDPO at 1 h and reaches the highest accuracy at 5 h, 2.7 points above SDPO and 3.5 points above the best GRPO variant.
In mathematical reasoning, our observation that SDPO underperforms GRPO is consistent with independent reports. RLCSD [1] finds SDPO below GRPO on the math average for all four models it tests (e.g., 76.3 vs. 78.6 on Qwen3-8B), with mixed results on logic and agentic tasks. Peng et al. [2] report thinking collapse under standard on-policy self-distillation (Qwen3-1.7B math average 49.8 for the base model vs. 47.7 after OPSD). Kim et al. [3] show that self-distillation can compress reasoning and hurt math OOD performance under certain conditioning, coverage and thinking configurations, while also reporting positive cases, and Zhu et al. [4] find no stable gain for standard OPSD in its math setting.
[1] L. Pan et al. RLCSD: Reinforcement learning with contrastive on-policy self-distillation. arXiv:2606.11709, 2026.
[2] K. Peng et al. Diagnosing and mitigating thinking collapse in on-policy self-distillation. arXiv:2607.10805, 2026.
[3] J. Kim et al. Why does self-distillation (sometimes) degrade the reasoning capability of LLMs? arXiv:2603.24472, 2026.
[4] S. Zhu et al. The many faces of on-policy distillation: Pitfalls, mechanisms, and fixes. arXiv:2605.11182, 2026.
Response to Reviewer oBEt (1/4)
Thank you for the positive assessment, in particular for noting the clear presentation, the explicitly bounded theory, and the training efficiency of STEPS.
Thank you for raising this. Our primary goal is to address the collapse of on-policy self-distillation in mathematical reasoning: STEPS keeps self-distillation from collapsing while preserving its training efficiency. Within this setting, we agree that STEPS relies on the annotator. On Qwen3-8B, all four annotators we tested improve over GRPO (78.75 AVG): the strong API annotator reaches 81.51, online self-annotation 80.65 (80.84 ± 0.27 over two seeds), a mid-size Qwen3-32B annotator 79.84, and the frozen Qwen3-8B base model used as a static annotator 79.67 (Table 10). Weaker annotators give smaller gains, but none falls below GRPO.
Localization quality matters, and simple heuristics do not replace it. In new controls requested by Reviewer N8RE, we kept everything fixed on Qwen3-1.7B except the mask location, and compared annotator-selected spans with masks that select the same number of tokens by student entropy or by teacher–student disagreement:
Annotator-selected spans give the highest score on all five benchmarks. On the same 4,912 annotated incorrect rollouts, the entropy and disagreement masks cover only 24.5% and 29.8% of the annotated error positions, against 18.0% for a random mask of the same size.
The design also bounds the effect of annotation errors. Annotations are used only during the first 40 steps, after which training is pure GRPO, and the mask together with the decay limits cumulative privileged-gradient exposure (Eq. 4), so a wrong span affects a limited number of updates. When the annotator returns no span, which happened for about 14% of incorrect rollouts in the Qwen3-1.7B run, the rollout simply receives GRPO. For granularity, Table 15 varies the coverage cap over 0.10, 0.25, 0.50 and 1.00, and α=0.25 performs best.
All Qwen3 experiments in Table 2 train and evaluate with thinking enabled. Beyond mathematics, new experiments on code generation (LiveCodeBench v6) and agentic tool use (ToolAlpaca, with each tool call as an annotation segment) show that STEPS also improves over GRPO and SDPO (see W4). Extending STEPS to multi-turn interactive agents, with each turn as an annotation segment, is a natural next step.