- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies on-policy self-distillation for continual post-training, focusing primarily on SDPO and comparing it against GRPO. The main empirical finding is that SDPO can achieve substantially faster in-domain specialization, but tends to induce larger parameter/response drift, worse OOD retention, and more severe forgetting under sequential domain shifts. The paper further studies teacher-update stability, CoT distillation, artifact amplification, and proposes restart-and-freeze teachers and post-hoc weight merging as partial mitigations.
The paper also provides a theoretical interpretation based on a KL-minimal "Razor" policy: a teacher-projected policy generally incurs excess KL drift relative to the minimum-change policy achieving the same success probability.
Critical strengths
-
The empirical phenomenon is interesting and practically relevant.
The contrast between strong short-term specialization and poor continual retention is clear. In the sequential MATH → SCIENCE → TOOLUSE → CODING experiment, SDPO initially outperforms GRPO on newly trained domains, but later suffers substantial degradation, while GRPO is much more conservative. -
The teacher stability experiments are useful.
The distinction between teacher freshness and temporal stability is well motivated. The restart-and-freeze experiment is particularly informative: replacing fast EMA with periodic refresh substantially recovers performance, e.g., improving MATH AIME from 34.38 to 55.00 under the matched setting. -
The artifact-amplification failure mode is compelling.
The observed\boxedrepetition collapse provides a concrete example of how token-level self-distillation can create a confirmation loop in which student artifacts are reinforced by the privileged teacher. -
The paper goes beyond benchmark accuracy.
Parameter-space spectral diagnostics and response-space drift provide supporting evidence that SDPO modifies the model more aggressively than GRPO.
Critical weaknesses
-
The central claim is broader than the method actually studied.
The paper frames its conclusions as limits of "on-policy self-distillation", but the instantiated SDPO objective is a direct distribution-matching objective, i.e., essentially a GKD/direct-OPD style objective.
This is importantly different from PG-style on-policy distillation, where teacher information is converted into token-level advantages and applied through a policy-gradient objective rather than explicitly projecting the student distribution toward the full teacher distribution.
Therefore, several conclusions seem better supported as statements about dense direct distribution matching rather than on-policy distillation in general. A PG-style OPD baseline is notably missing.
-
"SDPO drifts more than GRPO" is partly expected from the objective and is not sufficiently controlled.
GRPO optimizes a sequence-level success signal, whereas SDPO explicitly matches a privileged teacher distribution at every token. The latter imposes many more behavioral constraints than simply increasing task success probability. Thus, finding that SDPO moves farther from the base policy is not by itself surprising.
More importantly, the training configurations are not matched in update strength: SDPO uses learning rate , while GRPO uses , and the question batch sizes are 32 versus 8. This makes it difficult to cleanly attribute the larger parameter drift to supervision density/objective structure rather than simply a stronger effective optimization step.
I would find the main claim substantially stronger if the authors compared methods under matched:
- policy KL to the base model,
- parameter-update norm,
- in-domain performance improvement, or
- total effective gradient/update budget.
-
The theoretical result is correct-looking but weaker than the presentation suggests.
The theorem shows that a teacher-projected policy incurs at least as much KL from the base policy as the KL-minimal policy achieving the same success probability.
Conceptually, however, this largely follows from the setup: the teacher specifies additional preferences over successful trajectories (reasoning style, formatting, etc.), whereas the Razor policy is explicitly defined as the minimum-KL solution satisfying only the success constraint.
This does not establish that practical GRPO realizes the Razor policy, nor that PG-style OPD necessarily incurs the same excess drift. I therefore view the theorem primarily as an interpretation of the direct-distillation objective rather than a general explanation of why "on-policy self-distillation" forgets more.
-
The paper does not sufficiently distinguish supervision density from supervision semantics.
The CoT experiments are interesting, but "denser supervision" simultaneously changes what is supervised. Adding CoT tokens does not merely increase the number of supervised tokens; it introduces supervision over a specific reasoning trajectory. The degradation on MATH/SCIENCE may therefore reflect trajectory imitation rather than density itself. The paper partially acknowledges this, but the title and main narrative still emphasize density.
-
Some mitigation results are more diagnostic than methodologically novel.
Restart-and-freeze is useful, but relatively simple. Likewise, linear/spectral weight merging mainly supports the claim that reducing post-training drift improves retention rather than constituting a major algorithmic contribution.
Questions for authors
- How does SDPO compare against a PG-style OPD objective using the same privileged teacher?
- If SDPO and GRPO are matched by final policy KL from the base model or by in-domain improvement, does SDPO still exhibit substantially greater forgetting?
- Why are the learning rates for SDPO and for GRPO? How sensitive are the drift conclusions to this choice?
- Can the authors disentangle "more supervised tokens" from "imitation of a specific reasoning trajectory", e.g., by randomly selecting an equal number of answer/reasoning tokens?
- Does the excess-KL argument extend to sampled-token PG-style distillation, or is it specific to explicit teacher-distribution projection?