- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Retrospective On-Policy Self-Distillation (RSD). A teacher conditioned on the model's own previous attempt supervises a student that sees only the question. Before distillation, a brief role-reversed warmup is used to address the teacher's shortened reasoning. RSD uses no answers or rewards during these stages and aims to improve the model's initialization for later training. Experiments on Qwen3-4B and Qwen3-8B report increased pass@k coverage and improved downstream GRPO and RLRT results on mathematical reasoning benchmarks.
Critical strengths
-
The method improves downstream training in several settings without using answers or rewards during RSD. The average pass@1 gain reaches 9.5 percentage points for Qwen3-8B with RLRT. A FLOP-matched GRPO comparison also supports some benefit beyond additional training compute.
-
The paper is well organized and easy to read. The figures clearly explain the training procedure and reported results.
Critical weaknesses
-
The novelty and mechanism are not sufficiently established. RSD adapts existing on-policy self-distillation through prior-attempt conditioning and teacher warmup. Stronger comparisons with related methods and simpler prompting alternatives are needed. Shorter reasoning and lower performance occur together, but this does not establish causality. Likewise, higher pass@512 does not directly explain the benefit to GRPO, which samples eight attempts per prompt.
-
The evidence for broad and stable benefits is limited. Experiments use only Qwen3-4B and Qwen3-8B, mathematical reasoning tasks, and one training seed. Gains vary across settings: 4B with GRPO improves by only 1.6 percentage points, while larger gains occur elsewhere. External-teacher distillation shows almost no benefit. These results leave the method's broader significance and reliability uncertain.
Questions for authors
-
What advantage does role-reversed warmup offer over simply prompting the teacher to produce complete reasoning? How sensitive are the results to stopping warmup at step 20?
-
Can length-controlled or shuffled-prior-attempt comparisons support the proposed mechanism? Does RSD increase training-set pass@8 and the fraction of rollout groups with both correct and incorrect answers?
-
Do the gains persist across training seeds or other model families? Such evidence would help assess whether the benefits are reliable beyond the current settings.
-
Section 6.3 and Table 3 use the 8B baseline at step 180, while Appendix B.4 states step 200. Which checkpoint produced the reported FLOP-matched results?
5 comments
Response to Reviewer Hsqu (Part 2/4)
Q2. Can length-controlled or shuffled-prior-attempt comparisons support the proposed mechanism? Does RSD increase training-set pass@8 and the fraction of rollout groups with both correct and incorrect answers? (Also addresses W1.)
The revision contains each of these comparisons, and we summarize them below.
(1) Shuffled prior attempt
We replaced the prior attempt in the teacher's context with the model's own rollout on a different problem. This keeps the source and the format of the context and removes only its relevance.
With the shuffled attempt, the teacher shows no coverage gain over the base policy, whereas with the relevant attempt it does. The same holds after distillation and RL, when the full RSD procedure is run with the shuffled attempt and followed by GRPO.
| Average pass@1 on AIME 2024–2026 after GRPO (%) | Qwen3-4B | Qwen3-8B |
|---|---|---|
| Base | 25.2 | 33.0 |
| RSD with a shuffled prior attempt | 25.6 | 34.8 |
| RSD (Ours) | 29.7 | 43.5 |
(We have added these results in Section 3, Figure 2, Section 6.2, Figure 3, and Appendix D.5, Table 11.)
This confirms that the gain requires the model's own attempt at the same problem.
(2) Length-controlled comparison
Along the teacher warmup, the reasoning length of the teacher and its coverage change independently (Qwen3-4B, AIME 2024–2026).
| Warmup step | Reasoning length (% of base) | pass@256 (%) |
|---|---|---|
| 0 (teacher of naive RSD) | 36.6 | 70.1 |
| 20 (teacher of RSD) | 109.8 | 71.3 |
| 200 | 98.0 | 67.0 |
| Base policy | 100 | 66.5 |
(We have added these results in Appendix D.8, Figure 7.)
From step 0 to step 20, the length is restored while the coverage stays the same, so the teachers of naive RSD and of RSD differ in their reasoning length and not in their coverage. At step 20, the teacher is as long as the base policy and still 4.8 points above it, so this gain is not an effect of length.
(3) Training-set pass@8
We measured the outcome of the eight rollouts per training prompt during GRPO, at step 0 (the two initializations themselves) and at step 60.
| Training prompts, Base / RSD (%) | pass@8 | All-wrong groups (↓) | All-correct groups (↑) |
|---|---|---|---|
| Qwen3-4B, step 0 | 70.8 / 73.8 | 29.2 / 26.2 | 14.9 / 14.9 |
| Qwen3-4B, step 60 | 76.7 / 80.4 | 23.3 / 19.6 | 24.0 / 30.5 |
| Qwen3-8B, step 0 | 73.4 / 75.2 | 26.6 / 24.8 | 16.1 / 15.2 |
| Qwen3-8B, step 60 | 82.8 / 83.8 | 17.2 / 16.2 | 29.8 / 35.6 |
(We have added these results in Appendix D.2, Table 8.)
RSD raises pass@8 at step 0, so fewer groups have no correct rollout, and these are the groups from which GRPO cannot learn. The fraction of groups whose rollouts are all correct is similar for the two initializations at step 0, and it grows faster from the RSD initialization during training. The higher pass@8 and the lower fraction of all-wrong groups persist at every step.
Response to Reviewer Hsqu (Part 3/4)
Q3. Do the gains persist across training seeds or other model families? Such evidence would help assess whether the benefits are reliable beyond the current settings. (Also addresses W2.)
(1) Other model families
We added Olmo-3-7B-Instruct-DPO, a model outside the Qwen family that has not yet undergone RL post-training. With all settings identical to the Qwen3 experiments, the RSD initialization again improves over the base initialization under both GRPO and RLRT.
(We have added these results in Section 6.2, Table 2.)
(2) Training seeds
Due to the compute cost of each configuration, which consists of the two RSD stages followed by an RL run, we were unable to repeat the runs over several seeds, and each configuration is trained once. However, two properties of the results support their reliability.
First, the evaluation has little sampling noise. We do not estimate pass@k from k generations. Instead, every pass@1 and pass@4 is computed from 64 samples per problem with the unbiased estimator (as described in Appendix B.5). Even in the worst case, the standard error that sampling contributes to the five-benchmark average pass@1 is below 0.5 points, against gains of 2.6 to 9.5 points.
Second, the gain is consistent across independent pairs of runs, where the two runs of each pair share the configuration and differ only in the initialization.
| Average pass@1 over five benchmarks (%) | Base | RSD | Gain |
|---|---|---|---|
| Qwen3-4B, GRPO | 31.2 | 34.2 | +3.0 |
| Qwen3-4B, RLRT | 43.4 | 52.0 | +8.6 |
| Qwen3-8B, GRPO | 36.9 | 45.2 | +8.3 |
| Qwen3-8B, RLRT | 43.7 | 53.2 | +9.5 |
| Olmo-3-7B-Instruct-DPO, GRPO | 34.8 | 37.7 | +2.9 |
| Olmo-3-7B-Instruct-DPO, RLRT | 35.2 | 37.8 | +2.6 |
Across three models and two on-policy algorithms, the RSD initialization is ahead in every pair. The same holds for the two GRPO runs that are trained on logical reasoning instead of mathematics (Section 6.2, Table 3). If the differences came from the variation between training runs, we would expect the sign of the gain to vary between pairs.
Note on the Qwen3-4B GRPO baseline. While running the additional experiments, we found that the Qwen3-4B Base + GRPO run of the submission was run on 8 GPUs, whereas the RSD + GRPO run it is compared with was run on 4. Both use 128 prompts per optimizer step, but in our trainer the scale of the gradient depends on the number of GPUs, so the two runs were not trained under the same conditions. We reran the baseline with the matched configuration and report this run in the revision (Section 6.2, Table 2). With it, the gain of the RSD initialization in this setting is 3.0 / 3.5 points in average pass@1 / pass@4, rather than the 1.6 / 1.2 points of the submission that the reviewer notes in W2.
Response to Reviewer Hsqu (Part 1/4)
We thank the reviewer for the constructive feedback. We respond to each question in order below, and the revised parts of the manuscript are marked in orange.
(1) Prompting versus the role-reversed warmup
Following the reviewer's suggestion, we tested prompting on Qwen3-4B. The teacher without warmup receives one of the following instructions in the system turn of its prompt, with one prior attempt in its context as before:
(We have added these results in
Appendix D.9, Table 13, with a pointer fromSection 4.1.)Both instructions lengthen the reasoning only slightly, and the teacher stays below half the length of the base policy, whereas the warmup restores the full length within 20 steps. This confirms that the length collapse is not removed by an instruction, which is the advantage of the role-reversed warmup. The warmup also requires no labels, since its target is the model's own unconditioned policy.
(2) Sensitivity to the stopping step
To assess the sensitivity, we evaluated the teacher at successive warmup checkpoints (Qwen3-4B, AIME 2024–2026).
(We have added these results in
Appendix D.8, Figure 7, with a pointer fromSection 4.1.)Step 20 is the first checkpoint whose length has recovered. From step 20 to step 50, the teacher keeps both the full length and its coverage gain over the base policy, so any checkpoint in this range is a suitable teacher. Stopping earlier leaves the reasoning short, and continuing to step 200 removes the coverage gain.
The stopping criterion uses only the reasoning length, requiring neither labels nor a verifier.