acceptodds
Under review as a conference paper at ICLR 2027

STEPS: Selective On-Policy Self-Distillation for Reasoning

31 views

Abstract

On-policy self-distillation (OPSD) uses a model as its own teacher under privileged context, providing token-level supervision on the model's own reasoning trajectories. However, persistent all-token guidance can degrade training in our math RL setting. Such guidance may unnecessarily constrain non-critical tokens and reinforce biases induced by privileged information unavailable at inference. Motivated by these risks, we propose STEPS, a selective OPSD framework that controls the location, direction, and duration of distillation. STEPS identifies critical spans in student rollouts, applies forward KL to key spans or reverse KL to error spans, and gradually transitions to pure GRPO after a short distillation phase. Across four models from three families, STEPS improves mean accuracy over GRPO. On Qwen3-8B, it raises average accuracy across four math benchmarks and GPQA-Diamond by an absolute 2.76% with a strong annotator. Using the training policy as its own annotator yields an absolute mean gain of 1.90%, with an estimated 2.7% training-time overhead amortized over 300 steps. Controlled ablations support semantic span selection, class-specific KL directions, and the hand-off to GRPO. Together, these results support brief, targeted self-distillation as an efficient complement to reward-driven reasoning training. Our code is available at [https://anonymous.4open.science/r/trace-749F/](https://anonymous.4open.science/r/trace-749F/).

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy self-distillation, LLM reasoning, Reinforcement learning with verifiable rewards, Training stability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oBEtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper addresses the collapse of OPSD on math reasoning tasks, supported by a pilot study showing that the reproduced SDPO and SRPO exhibit rising entropy, decreasing response length, and validation collapse. To address this, the paper proposes STEPS with three components: (1) localizing correctness at the token level — applying forward KL to key spans or reverse KL to error spans, with one of the two branches chosen per run; (2) decaying the KL loss weight and fusing it with the RL loss; (3) switching to pure GRPO after step 40. Experiments are conducted on math-related benchmarks and the out-of-distribution GPQA-D.

Critical strengths
  1. The method is presented clearly; the figures are well made and easy to read; the writing is internally consistent.
  2. The theoretical analysis has clearly stated boundaries, which gives the paper a degree of theoretical depth.
  3. The experiments are extensive. They show that the method is highly training-efficient, with rapid early validation gains.
Critical weaknesses
  1. STEPS largely depends on the granularity and annotator reliability, while span-localization quality is unverified on long rollouts and in agentic domains.

  2. Among the method's three components, span localization and KL direction have bounded theoretical support, but the weighted loss, the decay schedule, and switching to pure GRPO after step 40 remain heuristic design choices with many hyperparameters.

  3. In the cross-family experiments, Llama family falls below SDPO once GPQA-D is excluded. STEPS exhibits a notable ability to retain held-out accuracy on GPQA-D, which is not supported by algorithm design or theory.

  4. All experiments are conducted on the OpenThoughts-math training domain, whereas the baselines it compares against, e.g., SDPO, were originally reported with larger gains over GRPO on SciKnowEval, LiveCodeBench, and agentic tool use.

Questions for authors
  1. What are the results of the decoupled ablation (λ ≡ 0.5 with ρ ≡ 1) and a decay-schedule window (e.g. 20,40,80,120) sweep?
  2. How should we understand the significant held-out retention that STEPS shows on GPQA-D?
  3. What are the final response-length and entropy trajectories over training? From Figure 1, the response length appears to grow much more than under GRPO, while the gains reflected on the benchmarks seem modest, is there an overthinking problem?
6 comments
AuthorsOpenReview ↗

Response to Reviewer oBEt (1/4)

Thank you for the positive assessment, in particular for noting the clear presentation, the explicitly bounded theory, and the training efficiency of STEPS.

W1: Dependence on annotator reliability and granularity; long rollouts and agentic domains.

Thank you for raising this. Our primary goal is to address the collapse of on-policy self-distillation in mathematical reasoning: STEPS keeps self-distillation from collapsing while preserving its training efficiency. Within this setting, we agree that STEPS relies on the annotator. On Qwen3-8B, all four annotators we tested improve over GRPO (78.75 AVG): the strong API annotator reaches 81.51, online self-annotation 80.65 (80.84 ± 0.27 over two seeds), a mid-size Qwen3-32B annotator 79.84, and the frozen Qwen3-8B base model used as a static annotator 79.67 (Table 10). Weaker annotators give smaller gains, but none falls below GRPO.

Localization quality matters, and simple heuristics do not replace it. In new controls requested by Reviewer N8RE, we kept everything fixed on Qwen3-1.7B except the mask location, and compared annotator-selected spans with masks that select the same number of tokens by student entropy or by teacher–student disagreement:

MaskMATH-500AIME 24AIME 25AMC 23GPQA-DAVG
GRPO (no KL)90.3344.5835.0081.5637.1857.73
All-token89.4543.3334.1780.7835.6756.68
Inverted (25%)90.3043.7535.0081.7237.0657.57
Random (25%)90.7044.1735.4282.6637.3158.05
Teacher–student KL (top-k)90.9544.1736.0482.9736.5558.14
Student entropy (top-k)91.0544.5836.2583.5937.6358.62
Semantic (STEPS)92.0544.7938.3386.7238.6460.11

Annotator-selected spans give the highest score on all five benchmarks. On the same 4,912 annotated incorrect rollouts, the entropy and disagreement masks cover only 24.5% and 29.8% of the annotated error positions, against 18.0% for a random mask of the same size.

The design also bounds the effect of annotation errors. Annotations are used only during the first 40 steps, after which training is pure GRPO, and the mask together with the decay limits cumulative privileged-gradient exposure (Eq. 4), so a wrong span affects a limited number of updates. When the annotator returns no span, which happened for about 14% of incorrect rollouts in the Qwen3-1.7B run, the rollout simply receives GRPO. For granularity, Table 15 varies the coverage cap over 0.10, 0.25, 0.50 and 1.00, and α=0.25\alpha = 0.25 performs best.

All Qwen3 experiments in Table 2 train and evaluate with thinking enabled. Beyond mathematics, new experiments on code generation (LiveCodeBench v6) and agentic tool use (ToolAlpaca, with each tool call as an annotation segment) show that STEPS also improves over GRPO and SDPO (see W4). Extending STEPS to multi-turn interactive agents, with each turn as an annotation segment, is a natural next step.

AuthorsOpenReview ↗

Response to Reviewer oBEt (2/4)

W2 & Q1: Heuristic components; decoupled ablation and decay-window sweep.

Thank you for the suggestion. We ran both experiments on Qwen3-1.7B (RKL on error spans, α=0.25\alpha = 0.25). The decoupled run with λ≡0.5\lambda \equiv 0.5 and ρ≡1\rho \equiv 1 keeps the KL term active while fully restoring GRPO on the selected spans:

Configurationλk\lambda_kρk\rho_kAVGΔ vs. GRPO
GRPO0157.73—
Constant KL, no span GRPO0.5054.85−2.88
Constant KL, full span GRPO (new)0.5156.74−0.99
STEPS (decay + GRPO restoration)0.5→00.5 \to 00→10 \to 160.11+2.38

Restoring span GRPO helps (54.85 → 56.74), but at the tested constant weight of 0.5 the model remains below GRPO.

For the window sweep, we keep the decay start at step 10 and end distillation at different steps:

Hand-off step2040 (default)80120
AVG59.1260.1158.7157.52

Among the four hand-off steps tested, step 40 achieved the highest AVG. Steps 20, 40, and 80 all outperformed GRPO, while step 120 scored slightly below it (57.52 vs. 57.73).

We agree that the weighting, decay and hand-off are design choices that do not follow from the theory. They follow one principle: privileged guidance is useful early as a correction signal and should then be withdrawn so that verifier-grounded learning takes over. We fixed the defaults (α=0.25\alpha = 0.25 and the step-40 hand-off) before running these sweeps, and they are shared across all models. Better ways to time the KL annealing and couple it with GRPO are a direction worth exploring in future work.

W3 & Q2: Llama-3.2-3B without GPQA-D, and GPQA-D retention.

We agree that SDPO is competitive on Llama-3.2-3B-Instruct. Without GPQA-D, the three-benchmark math average is 30.33 for SDPO and 30.10 for STEPS, a gap of 0.23 points that comes mainly from MATH-500 (0.9 points), and both are above GRPO (29.43). SDPO places a full correct sibling rollout in the teacher context and distills over all tokens (Table 8). For a 3B model without native long-form reasoning, imitating a complete correct sibling solution can give a strong surface-level hint on standard problems such as MATH-500, and because Llama responses are short, SDPO does not show the entropy surge and length collapse that it exhibits in our long-reasoning runs. On every model with long-form reasoning, STEPS leads both GRPO and SDPO even when GPQA-D is excluded:

Math-only averageSTEPSGRPOSDPO
Qwen3-8B (4 benchmarks)87.3184.9870.07
Qwen3-1.7B (4 benchmarks)65.4762.8741.68
SmolLM3-3B (3 benchmarks)78.1575.9755.54

On GPQA-D, the clearest retention effect is at 8B, where GRPO drops from the base model's 58.27 to 53.85 while STEPS stays at 58.33. Our new controls indicate that localization matters here: on Qwen3-8B, all-token FKL with the same schedule lowers GPQA-D to 52.21, below both GRPO and STEPS, and on Qwen3-1.7B a teacher–student-disagreement mask of the same size lowers it to 36.55 (vs. 38.64 for STEPS). This pattern is consistent with prior reports that whole-response self-distillation can raise in-domain math accuracy while hurting other domains: Li et al. (arXiv:2610.02700) find that standard OPSD on Qwen3-8B raises the math average from 61.8 to 64.7 but lowers GPQA-Diamond from 64.1 to 61.7. Conversely, math reasoning gains can transfer to science, as Huan et al. (arXiv:2507.00432) show with math-only RL on Qwen3-14B-Base (GPQA from 42.6 to 57.7).

Two further aspects of the design plausibly contribute. First, the span mask and finite schedule bound cumulative privileged-gradient exposure (Eq. 4; Proposition 5), and in the token-level diagnostic of Figure 6 the mean absolute log-probability shift relative to GRPO on unselected tokens is 0.00052 nats under STEPS versus 0.02425 under all-token guidance. Second, the teacher is conditioned only on a coarse type label such as case_split or missed_case (Appendix C.3), not on a verified solution trace (RLSD) or a correct sibling rollout (SDPO).

AuthorsOpenReview ↗

Response to Reviewer oBEt (3/4)

W4: Single training domain; SDPO's reported gains in other domains.

Thank you for raising this. We added experiments in two of the domains where SDPO reported gains.

For code, we trained Qwen3-8B (non-thinking) with GRPO, SDPO and STEPS using SDPO's released data pipeline and evaluation (LiveCodeBench v6, 131 problems from 2025.02–2025.05; mean@4 test-case pass rate, SDPO's default metric). In SDPO's released LiveCodeBench scripts, SDPO uses a PPO mini-batch size of 1 while the GRPO baseline uses 8 with the same train batch of 32, so SDPO performs 32 optimizer updates per rollout batch against 4 for GRPO. We used a mini-batch size of 8 for all methods:

MethodStep 30Step 60Step 90Step 120
GRPO57.7362.2263.4064.72
SDPO58.8361.9763.8464.72
STEPS-FKL (strong annotator)59.7663.2765.1966.14
STEPS-FKL (online-self)59.1563.0865.2165.92
SDPO, original mini-batch 1 (reference)62.9263.3865.3964.23

SDPO performs on par with GRPO, and both STEPS variants are ahead of GRPO and SDPO at all four reported checkpoints under matched optimizer updates (+1.42 and +1.20 over GRPO at step 120). As an additional reference, SDPO with its original mini-batch size of 1 leads at steps 30–90, by 3.16 points over the best STEPS variant at step 30 and by at most 0.18 points at steps 60 and 90, which reproduces its early advantage; at step 120 it reaches 64.23, below both STEPS variants (66.14 and 65.92).

For agentic tool use, we followed SDPO's ToolAlpaca setup (Qwen3-8B, non-thinking, avg@16, best accuracy within 1 h and 5 h of wall-clock training) and ran STEPS with online self-annotation, treating each tool call as an annotation segment. Baseline numbers are taken from SDPO's Table 3:

Qwen3-8B, ToolAlpaca (avg@16)Mini-batch1 h5 h
Base—57.557.5
GRPO (on-policy)3260.265.7
GRPO (off-policy)864.967.7
SDPO (on-policy)3268.068.5
STEPS-FKL (online-self)3267.871.2

STEPS is comparable to SDPO at 1 h and reaches the highest accuracy at 5 h, 2.7 points above SDPO and 3.5 points above the best GRPO variant.

In mathematical reasoning, our observation that SDPO underperforms GRPO is consistent with independent reports. RLCSD [1] finds SDPO below GRPO on the math average for all four models it tests (e.g., 76.3 vs. 78.6 on Qwen3-8B), with mixed results on logic and agentic tasks. Peng et al. [2] report thinking collapse under standard on-policy self-distillation (Qwen3-1.7B math average 49.8 for the base model vs. 47.7 after OPSD). Kim et al. [3] show that self-distillation can compress reasoning and hurt math OOD performance under certain conditioning, coverage and thinking configurations, while also reporting positive cases, and Zhu et al. [4] find no stable gain for standard OPSD in its math setting.

[1] L. Pan et al. RLCSD: Reinforcement learning with contrastive on-policy self-distillation. arXiv:2606.11709, 2026.

[2] K. Peng et al. Diagnosing and mitigating thinking collapse in on-policy self-distillation. arXiv:2607.10805, 2026.

[3] J. Kim et al. Why does self-distillation (sometimes) degrade the reasoning capability of LLMs? arXiv:2603.24472, 2026.

[4] S. Zhu et al. The many faces of on-policy distillation: Pitfalls, mechanisms, and fixes. arXiv:2605.11182, 2026.

Reviewer_he63OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes STEPS, which uses a short period of selective on-policy self-distillation before continuing with GRPO. An annotator identifies key spans in correct responses or error spans in incorrect responses and supplies coarse diagnostic labels. These labels condition a periodically synchronized self-teacher; span masks determine which tokens receive forward- or reverse-KL supervision. Coverage is capped and the distillation coefficient decays to zero while GRPO is restored on annotated positions. The paper provides local KL-gradient identities and conditional bounds on privileged-gradient variation and retained signal, and evaluates four models from three families on mathematical reasoning and GPQA-Diamond. The principal Qwen3-8B result improves the five-benchmark mean from 78.75 to 81.51 with an external annotator; online self-annotation reaches 80.65.

Critical strengths
  1. A coherent and potentially useful training recipe, with encouraging evidence across models. Applying privileged self-distillation briefly at selected spans and then returning to verifier rewards is a sensible design. Four models across three families provide evidence beyond one backbone. The strongest Qwen3-8B result improves on GRPO by 2.76 percentage points, with a paired 95% interval of [0.41, 4.91] conditional on the trained checkpoints (Tables 2 and 16). Online self-annotation offers a practically attractive configuration with an estimated 2.7% training-time overhead, although its gain is less certain. The reported benefits make this a credible empirical contribution.

  2. Meaningful controls and transparent qualifications strengthen the evidence. Random and inverted masks at matched coverage test semantic selection beyond sparsity, while the span-class × KL-direction experiment separates choices that the two main configurations alone would confound (Figure 5 and Appendix E.1). Section 5.1 explicitly selects checkpoints using held-out OpenThoughts validation, not benchmark scores. The authors also acknowledge that the online-self interval includes zero, the handoff ablation is coupled, and the theoretical bounds do not guarantee training stability. These qualifications make the positive findings easier to assess without overstating their scope.

Critical weaknesses
  1. The most attractive practical configuration is not yet statistically robust. Online self-annotation avoids dependence on a stronger external annotator and has low estimated overhead, but its +1.90-point gain over GRPO has a paired 95% interval of [−0.25, 3.75] (Table 16). This does not establish an improvement even conditional on the evaluated checkpoints, and these intervals do not measure training-seed variability. Appendix E.1 also identifies the class/direction and coverage results as single-seed evaluations. The authors correctly acknowledge these limits; the remaining concern is how reliably the inexpensive recipe and its design preferences transfer across training runs.

  2. The combined intervention makes attribution difficult, limiting the conclusions beyond a useful empirical recipe. STEPS changes the support and direction of KL guidance, its duration, and GRPO weighting on annotated positions. In the constant-KL comparison, keeping λ = 0.5 also keeps the selected-span GRPO weight at zero. The drop from 60.16 to 54.85 therefore tests the complete handoff, rather than isolating persistent distillation from GRPO restoration (Figure 5(c), Appendix E.1). The paper discloses this coupling, but the experiment cannot determine which change drives the gain. In addition, Appendix D.8 describes the Qwen3-8B all-token-FKL control qualitatively, while the quantitative coverage controls concern 1.7B. The conditional theory does not resolve this empirical attribution. This limits the mechanistic conclusion and prevents a stronger recommendation, without invalidating the full recipe.

Questions for authors
  1. How many independent training seeds underlie the main strong-annotator and online-self comparisons? If repeat runs already exist, what is the variability of their gains over GRPO? I recognize that checkpoint selection uses held-out training-domain validation; please distinguish that specified rule from how the per-model KL direction, coverage, and schedule were selected. These details would clarify the robustness of the practical recipe beyond the reported checkpoint-conditional intervals.
  2. Could you report the quantitative Qwen3-8B all-token-FKL-plus-decay results described in Appendix D.8, including GPQA and the five-benchmark mean, with the diagnostic context and selected-token GRPO weighting specified? Do any existing controls vary KL duration while holding GRPO weighting fixed, or vice versa? Such evidence would clarify attribution. If those separate controls are unavailable, the handoff result should continue to be interpreted as evidence for the combined intervention, rather than for either change alone.
3 comments
AuthorsOpenReview ↗

Response to Reviewer he63 (1/2)

Thank you for the positive assessment and for the constructive questions on robustness and attribution.

W1 & Q1: Robustness of online-self, number of seeds, and how the configuration was chosen.

Thank you for raising this. Each comparison in the submission used a single training seed. We have since trained both GRPO and online-self from scratch with a second seed under the Table 2 configuration. Online-self improved over GRPO by 1.98 AVG points on the new seed, compared with 1.90 points on the original seed.

Qwen3-8B (2 seeds)MATH-500AIME 24AIME 25AMC 23GPQA-DAVG
GRPO97.20 ± 0.1477.50 ± 0.5969.27 ± 0.4496.41 ± 0.2254.14 ± 0.4078.90 ± 0.21
Online-self97.60 ± 0.0779.17 ± 0.8872.19 ± 0.1597.27 ± 0.3357.99 ± 0.2280.84 ± 0.27

Online-self runs trained independently in other settings also improve over GRPO: by 1.45 AVG on Llama-3.2-3B-Instruct (Table 3), and at every checkpoint in a new LiveCodeBench v6 experiment on Qwen3-8B (65.92 vs. 64.72 at step 120).

Checkpoints are selected by the validation rule in Section 5.1. We use the same coverage cap (α=0.25\alpha = 0.25) and schedule (KL decays from step 10 and is removed at step 40) for every model, without per-model tuning, and we fixed both defaults before running the coverage sweep (Table 15) and the window sweep below. We chose step 40 empirically to implement the design principle of brief guidance followed by a hand-off to GRPO. We report the window sweep under W2. Better ways to time the KL annealing and couple it with GRPO are a direction worth exploring in future work.

For the KL direction, Table 2 reports both configurations at both Qwen3 scales, and both outperform GRPO at each scale:

AVGFKL on Ky\mathcal{K}_yRKL on Ey\mathcal{E}_yGRPO
Qwen3-8B81.5180.1478.75
Qwen3-1.7B58.4160.1157.73

As discussed in Section 4.1, FKL can reinforce teacher-supported continuations on key spans, while RKL can suppress student-favored errors, and Fig. 5(b) supports this pairing. In Table 2, FKL on key spans performs better at 8B and RKL on error spans at 1.7B, possibly because stronger students generate more correct rollouts to reinforce and weaker students more errors to correct. We use this as an empirical rule of thumb, so the 3B cross-family models use RKL.

W2 & Q2: Attributing the gain within the combined intervention.

Thank you for this point. We agree that the original constant-KL comparison tested the complete hand-off. We ran the decoupled control on Qwen3-1.7B (RKL on error spans, α=0.25\alpha = 0.25):

Configurationλk\lambda_kρk\rho_kAVGΔ vs. GRPO
GRPO0157.73—
Constant KL, no span GRPO0.5054.85−2.88
Constant KL, full span GRPO (new)0.5156.74−0.99
STEPS (decay + GRPO restoration)0.5→00.5 \to 00→10 \to 160.11+2.38

With the KL weight held at 0.5, turning on span GRPO raises AVG from 54.85 to 56.74 but still leaves it below GRPO, so persistent privileged guidance is harmful in itself. Moving from this setting to the default schedule, which withdraws the KL term while restoring span GRPO, raises AVG further from 56.74 to 60.11.

We also varied the KL duration under the default restoration rule by ending distillation at different steps:

Hand-off step2040 (default)80120
AVG59.1260.1158.7157.52

Among the four hand-off steps tested, step 40 achieved the highest AVG. Steps 20, 40, and 80 all outperformed GRPO, while step 120 scored slightly below it (57.52 vs. 57.73).

For the Qwen3-8B all-token control in Appendix D.8, the run uses the same diagnostic prefix c, the same decay schedule and the same GRPO restoration (ρk\rho_k from 0 to 1 on all tokens) as STEPS; only the mask covers all response tokens:

Qwen3-8BMATH-500AIME 24AIME 25AMC 23GPQA-DAVG
GRPO97.3077.0868.9696.5653.8578.75
All-token FKL + decay (new)97.1576.6768.3396.4152.2178.15
STEPS (FKL on Ky\mathcal{K}_y + decay)97.8380.2173.5497.6658.3381.51

With the same signal and schedule, all-token guidance falls below GRPO, while span-localized guidance gives the gain. This adds a quantitative coverage control at 8B to the 1.7B coverage sweep in Table 15.

The exposure bound (Eq. 4) predicts that limiting both coverage and duration reduces privileged-gradient exposure, but it does not quantify their effect on accuracy. The controls above are consistent with this prediction: all-token guidance at 8B and longer guidance at 1.7B both reduce the gain.

AuthorsOpenReview ↗

Response to Reviewer he63 (2/2)

Changes to the manuscript.

  • Independent training seeds. §5.2 and Table 22 report the two-seed online-self comparison.
  • Configuration choices. Appendix E.1 states that the 25% coverage cap and the step-40 hand-off were fixed across models before the sweeps; Appendix C.7 explains why the 3B models use RKL on error spans.
  • GRPO restoration and KL duration. Table 20 varies the span GRPO weight at a fixed KL weight of 0.5, Table 21 compares hand-off steps from 20 to 120, and §5.3 summarizes both.
  • All-token control at 8B. Appendix F.1 specifies the shared diagnostic prefix and schedules, and Table 19 reports all five benchmarks.

We sincerely hope that the additional seed, the decoupled hand-off controls and the quantitative all-token control address your concerns, and that you will continue to support the paper during the discussion.

AuthorsOpenReview ↗

A Kind Reminder for Reading the Response

Dear Reviewer he63,

Thank you for your insightful suggestions. We have done our best to address your concerns. Since the discussion period is closing very soon, could you please check our response to see whether it mitigates your concerns? We would greatly appreciate that!

Thank you, The authors of STEPS

Reviewer_N8REOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

STEPS augments GRPO with brief, selective on-policy self-distillation. An annotator identifies key spans in correct rollouts or error spans in incorrect rollouts; separate training variants apply forward KL to key spans or reverse KL to error spans. Distillation decays while GRPO is restored on selected positions. The paper provides local gradient analysis and conditional bounds, and evaluates mathematics-trained models on mathematics and GPQA benchmarks across four models from three families.

Critical strengths
  1. A useful approach to controlling self-distillation. Restricting both the locations and duration of privileged guidance is a sensible way to combine localized supervision with verifier-grounded learning. The option of self-annotation and the removal of auxiliary computation after the initial phase make the approach practically appealing.

  2. Meaningful empirical support. Figure 5 includes matched selection controls and a full span-class × KL-direction comparison on Qwen3-1.7B. On Qwen3-8B, the strong-annotator variant improves AVG over GRPO by 2.76 percentage points, with a paired 95% CI of [0.41, 4.91], and also improves over RLSD with a positive paired interval (Table 16). The cross-family results further support the method's potential usefulness.

Critical weaknesses
  1. The incremental value of semantic annotation remains insufficiently established.

While the matched random and inverted-mask ablations demonstrate the importance of token selection, they do not establish whether semantic annotation provides additional benefits over simpler informed selection strategies, such as student-entropy- or teacher–student-disagreement-based selection. This distinction is consequential because semantic localization is a central component of STEPS and introduces additional annotation machinery. A controlled comparison would help isolate its contribution beyond informed token selection.

  1. The correspondence between the stated and evaluated method needs clarification.

Equation (1) specifies common full-rollout normalization, whereas the released span-loss implementation normalizes by selected-token counts. Its active span path also lacks the reported per-vocabulary KL clipping. These differences affect the relative strength of distillation and GRPO. Although they could reflect an outdated release, they currently prevent a clear assessment of which objective produced the reported results.

Questions for authors
  1. With teacher conditioning, coverage, KL direction, and schedule held fixed, how does semantic selection compare with entropy- or teacher–student-disagreement-based masks? This would directly test the additional value of semantic localization.

  2. For the Table 2 experiments, were the span losses normalized by the full rollout length, as specified in Eq. (1), or by the number of selected tokens, as in the released implementation? Was the reported per-vocabulary KL clipping applied? These choices affect the strength of distillation relative to GRPO, so please clarify which formulation was used.

  3. What is the end-to-end cost of the strong-annotator configuration? For online-self, whose paired improvement interval includes zero, is any independent-training-run evidence available? The reported 2.7% overhead applies to online-self, so the accuracy–cost tradeoff should be assessed separately for these configurations.

  4. C.5 specifies avg@16 for AIME. Under a single evaluation of 30 problems with exactly 16 binary-scored responses each, the Qwen3-1.7B STEPS-RKL AIME24 score of 44.92% cannot arise after ordinary rounding. Were results averaged across repeated evaluations, or were the effective counts different? Please clarify the aggregation and provide per-problem success counts.

7 comments
AuthorsOpenReview ↗

Response to Reviewer N8RE (1/3)

Thank you for the careful review, including the inspection of our code and the per-cell arithmetic.

W1 & Q1: Value of semantic localization beyond informed token selection.

Thank you for this suggestion. We ran the requested controls on Qwen3-1.7B and changed only the mask location. Everything else was fixed: RKL on error spans, the decay and GRPO hand-off schedule, the seed, the checkpoint rule (peak validation mean@16 over 300 steps), and the annotator's diagnostic prefix c for the teacher. For each annotated incorrect rollout, the alternative masks select exactly k=∣Ey∣k = |\mathcal{E}_y| tokens: (i) the top-k tokens by student entropy, and (ii) the top-k tokens by per-token teacher–student RKL, with the teacher conditioned on the same c.

MaskMATH-500AIME 24AIME 25AMC 23GPQA-DAVG
GRPO (no KL)90.3344.5835.0081.5637.1857.73
All-token89.4543.3334.1780.7835.6756.68
Inverted (25%)90.3043.7535.0081.7237.0657.57
Random (25%)90.7044.1735.4282.6637.3158.05
Teacher–student KL (top-k)90.9544.1736.0482.9736.5558.14
Student entropy (top-k)91.0544.5836.2583.5937.6358.62
Semantic (STEPS)92.0544.7938.3386.7238.6460.11

Semantic selection is the highest on all five benchmarks.

On the same 4,912 annotated incorrect rollouts, the entropy and disagreement masks cover 24.5% and 29.8% of the annotated error positions, against 18.0% for a random mask of the same size (averaged over rollouts):

Mask on shared rolloutsOverlap with Ey\mathcal{E}_yIoU with Ey\mathcal{E}_y
Random (top-k)18.0%0.101
Student entropy (top-k)24.5%0.143
Teacher–student KL (top-k)29.8%0.185

W2 & Q2: Correspondence between Eq. (1) and the evaluated objective.

Thank you for inspecting the released code. In the Table 2 runs, each loss component (both KL branches and the GRPO terms on Ny\mathcal{N}_y and Sy\mathcal{S}_y) is summed over its positions. All KL and GRPO components use the total number of valid response tokens in the microbatch as their common denominator. Per-vocabulary KL clipping is active in these runs: each per-vocabulary contribution is clipped to [−τ,τ][-\tau, \tau] with τ=0.05\tau = 0.05 before summation over the vocabulary.

When preparing the release, we inadvertently packaged an early version of the span-loss path, which normalizes by selected-token counts and skips the clipping step. This path did not produce any reported result. We have updated the anonymous repository so that the span-loss path matches Eq. (1) and the training runs, and added a unit test that checks the normalization and the two-sided per-vocabulary clipping.

AuthorsOpenReview ↗

Response to Reviewer N8RE (2/3)

Q3: Cost of the strong annotator and evidence for online-self.

Thanks for asking. We agree that the two configurations should be assessed separately. With FKL on key spans, the strong annotator is called only on correct rollouts during the KL-active phase: 5,069 calls (about 130 per step). It runs with thinking disabled and returns a short JSON list of spans, so the bill is dominated by input tokens (15.1M input vs. 0.24M output) and comes to about $4.08 at list price. The main cost is latency: about 93 s per KL-active step, set by the API rate limit (80 requests per minute). Under the accounting of Appendix E.3 (Eq. 11), the amortized overhead is about 10.4%, against 2.7% for online-self.

Qwen3-8B, FKL on Ky\mathcal{K}_yAnnotation / KL stepOverheadAPI costAVG (Δ vs. GRPO)Paired 95% CI
Online-self17.2 s2.7%—80.65 (+1.90)[−0.25, 3.75]
Strong (qwen3.5-plus)93.3 s10.4%≈ $4.0881.51 (+2.76)[0.41, 4.91]

Both overheads include the teacher forward pass (9.4 s per KL-active step).

We trained both GRPO and online-self from scratch with a second seed under the same configuration. Online-self improved over GRPO by 1.98 AVG points on the new seed, compared with 1.90 points on the original seed.

Qwen3-8B (2 seeds)MATH-500AIME 24AIME 25AMC 23GPQA-DAVG
GRPO97.20 ± 0.1477.50 ± 0.5969.27 ± 0.4496.41 ± 0.2254.14 ± 0.4078.90 ± 0.21
Online-self97.60 ± 0.0779.17 ± 0.8872.19 ± 0.1597.27 ± 0.3357.99 ± 0.2280.84 ± 0.27

Online-self runs trained independently in other settings point the same way: +1.45 AVG over GRPO on Llama-3.2-3B-Instruct (Table 3), and ahead of GRPO at every checkpoint in a new LiveCodeBench v6 experiment on Qwen3-8B (65.92 vs. 64.72 at step 120).

Overall, the strong annotator gives the larger, statistically supported gain at about 10% training-time overhead and a small API bill, while online-self needs no external model and costs 2.7%. In both cases the extra cost is confined to the first 40 steps, and the gains appear early in training: on SmolLM3-3B, STEPS at step 40 already exceeds GRPO's selected step-325 checkpoint (71.78 vs. 69.40 AVG, Table 3). The annotation cost is therefore small relative to the training it saves.

Q4: Aggregation of AIME scores.

Thank you for this careful check. Your arithmetic is correct: 44.92% cannot come from exactly 30 × 16 binary outcomes. Our aggregation averaged correctness over the responses actually recorded for each problem and then averaged over problems. In this cell, one problem (AIME 2024 I, Problem 9) had 15 recorded responses instead of 16 because one generation record was missing. The merge script flags count mismatches only under --strict, which our default launch script did not pass. The per-problem success counts are:

ProblemCorrect / recordedProblemCorrect / recorded
I-116/16II-115/16
I-215/16II-211/16
I-314/16II-312/16
I-415/16II-415/16
I-52/16II-56/16
I-613/16II-613/16
I-714/16II-75/16
I-80/16II-81/16
I-99/15II-91/16
I-102/16II-102/16
I-110/16II-115/16
I-120/16II-124/16
I-134/16II-137/16
I-142/16II-142/16
I-1510/16II-150/16
AIME I total116/239AIME II total99/240

In total, 215 of 479 recorded responses are correct, and (206/16 + 9/15)/30 = 44.92%.

The missing records mainly came from two sources. When Qwen3-1.7B was evaluated in thinking mode, a few responses to hard problems entered long self-correction chains, hit the max_new_tokens limit, and were dropped during merging instead of being scored. On SmolLM3-3B, a few long samples were truncated or returned empty after stop-token parsing and were filtered out. Under our protocol, such cases are failed responses. We therefore count every missing response as incorrect rather than re-sampling it, so every cell is scored over the full number of samples. This cell becomes 215/480 = 44.79%.

We re-checked all evaluation logs and corrected every affected entry under this rule. The corrections are minor and leave all rankings and conclusions unchanged.

AuthorsOpenReview ↗

Response to Reviewer N8RE (3/3)

Changes to the manuscript and code.

  • Semantic localization. §5.3 and Table 18 add the entropy and disagreement controls; Appendix F.1 reports their overlap with the annotated error positions.
  • Training objective. §3.2 specifies the common denominator for all KL and GRPO terms and the two-sided per-vocabulary clipping.
  • Annotation cost and replication. §5.4 and Appendix G report the annotation cost of both configurations, and Table 22 the two-seed online-self comparison.
  • Evaluation counts. §5.1 counts missing generations as incorrect; Tables 2–3 and Fig. 5 use the corrected values, and Table 23 gives the per-problem AIME counts.
  • Released code. The span-loss path now matches the Table 2 runs and includes a unit test.

Thank you again for these detailed questions. We are happy to answer any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.