acceptodds
Withdrawn from ICLR 2027

Denser Better: Limits of On-Policy Self-Distillation for Continual Post-Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with self-distillation as a particularly attractive approach. We revisit this optimistic claim through self-distillation policy optimization (SDPO). Our experiments show that SDPO accelerates in-domain specialization when teacher signals are stable and well aligned, but struggles to generalize out of distribution. In continual post-training, SDPO exhibits greater forgetting and can even collapse, whereas GRPO, the more established on-policy reinforcement learning method, adapts more conservatively and better preserves prior capabilities. Further analyses link these failures to increased drift in parameter and response space, and to amplification of high-frequency artifacts through a self-reinforcing teacher–student loop. Thus, on-policy data alone is insufficient for continual learning. Self-distillation is effective when teacher targets are stable and token-level supervision is reliable, but should not be treated as a default stabilizer for continual post-training.

Primary area:
reinforcement learning
Keywords:
self-distillation, large language models, post-training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_abpYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies on-policy self-distillation for continual post-training, focusing primarily on SDPO and comparing it against GRPO. The main empirical finding is that SDPO can achieve substantially faster in-domain specialization, but tends to induce larger parameter/response drift, worse OOD retention, and more severe forgetting under sequential domain shifts. The paper further studies teacher-update stability, CoT distillation, artifact amplification, and proposes restart-and-freeze teachers and post-hoc weight merging as partial mitigations.

The paper also provides a theoretical interpretation based on a KL-minimal "Razor" policy: a teacher-projected policy generally incurs excess KL drift relative to the minimum-change policy achieving the same success probability.

Critical strengths
  1. The empirical phenomenon is interesting and practically relevant.
    The contrast between strong short-term specialization and poor continual retention is clear. In the sequential MATH → SCIENCE → TOOLUSE → CODING experiment, SDPO initially outperforms GRPO on newly trained domains, but later suffers substantial degradation, while GRPO is much more conservative.

  2. The teacher stability experiments are useful.
    The distinction between teacher freshness and temporal stability is well motivated. The restart-and-freeze experiment is particularly informative: replacing fast EMA with periodic refresh substantially recovers performance, e.g., improving MATH AIME from 34.38 to 55.00 under the matched setting.

  3. The artifact-amplification failure mode is compelling.
    The observed \boxed repetition collapse provides a concrete example of how token-level self-distillation can create a confirmation loop in which student artifacts are reinforced by the privileged teacher.

  4. The paper goes beyond benchmark accuracy.
    Parameter-space spectral diagnostics and response-space drift provide supporting evidence that SDPO modifies the model more aggressively than GRPO.

Critical weaknesses
  1. The central claim is broader than the method actually studied.

    The paper frames its conclusions as limits of "on-policy self-distillation", but the instantiated SDPO objective is a direct distribution-matching objective, i.e., essentially a GKD/direct-OPD style objective.

    This is importantly different from PG-style on-policy distillation, where teacher information is converted into token-level advantages and applied through a policy-gradient objective rather than explicitly projecting the student distribution toward the full teacher distribution.

    Therefore, several conclusions seem better supported as statements about dense direct distribution matching rather than on-policy distillation in general. A PG-style OPD baseline is notably missing.

  2. "SDPO drifts more than GRPO" is partly expected from the objective and is not sufficiently controlled.

    GRPO optimizes a sequence-level success signal, whereas SDPO explicitly matches a privileged teacher distribution at every token. The latter imposes many more behavioral constraints than simply increasing task success probability. Thus, finding that SDPO moves farther from the base policy is not by itself surprising.

    More importantly, the training configurations are not matched in update strength: SDPO uses learning rate 10−510^{-5}, while GRPO uses 10−610^{-6}, and the question batch sizes are 32 versus 8. This makes it difficult to cleanly attribute the larger parameter drift to supervision density/objective structure rather than simply a stronger effective optimization step.

    I would find the main claim substantially stronger if the authors compared methods under matched:

    • policy KL to the base model,
    • parameter-update norm,
    • in-domain performance improvement, or
    • total effective gradient/update budget.
  3. The theoretical result is correct-looking but weaker than the presentation suggests.

    The theorem shows that a teacher-projected policy incurs at least as much KL from the base policy as the KL-minimal policy achieving the same success probability.

    Conceptually, however, this largely follows from the setup: the teacher specifies additional preferences over successful trajectories (reasoning style, formatting, etc.), whereas the Razor policy is explicitly defined as the minimum-KL solution satisfying only the success constraint.

    This does not establish that practical GRPO realizes the Razor policy, nor that PG-style OPD necessarily incurs the same excess drift. I therefore view the theorem primarily as an interpretation of the direct-distillation objective rather than a general explanation of why "on-policy self-distillation" forgets more.

  4. The paper does not sufficiently distinguish supervision density from supervision semantics.

    The CoT experiments are interesting, but "denser supervision" simultaneously changes what is supervised. Adding CoT tokens does not merely increase the number of supervised tokens; it introduces supervision over a specific reasoning trajectory. The degradation on MATH/SCIENCE may therefore reflect trajectory imitation rather than density itself. The paper partially acknowledges this, but the title and main narrative still emphasize density.

  5. Some mitigation results are more diagnostic than methodologically novel.

    Restart-and-freeze is useful, but relatively simple. Likewise, linear/spectral weight merging mainly supports the claim that reducing post-training drift improves retention rather than constituting a major algorithmic contribution.

Questions for authors
  1. How does SDPO compare against a PG-style OPD objective using the same privileged teacher?
  2. If SDPO and GRPO are matched by final policy KL from the base model or by in-domain improvement, does SDPO still exhibit substantially greater forgetting?
  3. Why are the learning rates 10−510^{-5} for SDPO and 10−610^{-6} for GRPO? How sensitive are the drift conclusions to this choice?
  4. Can the authors disentangle "more supervised tokens" from "imitation of a specific reasoning trajectory", e.g., by randomly selecting an equal number of answer/reasoning tokens?
  5. Does the excess-KL argument extend to sampled-token PG-style distillation, or is it specific to explicit teacher-distribution projection?
Reviewer_a9kXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies SDPO in continual post-training by comparing it with GRPO and examining teacher update rates and CoT distillation. The authors conclude that SDPO is a strong specialization method but a fragile continual-learning signal: it forgets more across domains and may collapse when the teacher is updated too quickly. A rapidly updated teacher can lead to severe collapse, while periodically refreshing and then freezing the teacher improves stability in the authors’ experiments. The authors use these results, together with a KL-based theoretical analysis, to argue that on-policy sampling alone does not protect a model from forgetting

Critical strengths
  1. The research objective is clear and well organized. It first examines the quality of the teacher signal, then compares single-domain and continual-training performance, and finally investigates possible explanations for the observed failures.

  2. The writing is fluent and accessible. The main claims are stated clearly, and the figures generally make the performance trends easy to track.

  3. The evaluation metrics are generally appropriate. The paper uses Acc@8 to track task performance and complementary measures, including JS divergence and parameter and response drift, to examine training stability and behavioral change.

Critical weaknesses
  1. The paper concludes that “SDPO is a strong specialization method but a fragile continual-learning signal.” However, prior work raises more fundamental concerns about SDPO and related privileged-information self-distillation. [1] finds that SDPO’s gains on coding tasks do not transfer to unseen problems and that some in-domain gains come with out-of-domain degradation. [2] finds that privileged-information self-distillation can harm long-form reasoning in thinking models. More directly, [3] reports a decline in SDPO’s training score on mathematics, [4] finds no improvement or degradation on in-domain validation sets across several tasks, and [5] reports substantially worse tool-use performance for SDPO than for GRPO. Even this paper’s Table E.3 shows AIME accuracy falling from 19.17% to 13.75% after math SDPO training on Olmo-3-7B-Instruct-DPO. Given these problems with SDPO itself, how do the authors justify the opposing characterization of SDPO as “a strong specialization method”? Please explain the apparent contradiction.

  2. The hyperparameter comparison between the core methods is unfair. In Appendix D.2, SDPO uses a learning rate of 10^{-5}, while GRPO uses 10^{-6}; their question batch sizes are 32 and 8, respectively. With the same number of training steps and rollouts per question, SDPO also processes more questions and generated responses per step.

  3. SDPO adapts faster to the current domain but forgets more during cross-domain training; excessively rapid teacher updates can even lead to output collapse. This conclusion seems quite natural, as excessive training can itself cause collapse. Does the paper offer a more insightful conclusion?

References [1]. Does self-distillation improve reasoning? [2]. Rethinking On-Policy Self-Distillation for Thinking Models [3]. Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? [4]. Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation [5]. Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents

Questions for authors

Please see Weaknesses.

Reviewer_UvVDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the benefits and limitations of on-policy self-distillation (SDPO), examining teacher stability, CoT distillation, OOD generalization, and continual post-training. The authors find that SDPO enables strong in-domain specialization but can suffer from instability, poor generalization, and forgetting compared to GRPO.

Critical strengths
  • The paper studies interesting and timely questions about on-policy self-distillation.
  • The experiments cover multiple domains and provide useful insights into teacher stability and supervision density.
  • The paper uncovers multiple empirical insights comparing GRPO and SDPO, such as forgetting analysis, generalization and stability.
Critical weaknesses
  • Overclaiming about CoT reliability (Section 2.2): The authors claim that "more supervision is useful only when reliable," but the experiments do not establish that CoT unreliability causes the degradation. CoTs contain backtracking, verification, and redundant steps that may simply be unhelpful distillation targets. The evidence only supports the narrower claim that CoT distillation can hurt performance.

  • Hand wavy hypotheses about Intermediate-distance interference (Section 3.1): The explanation that partially aligned tasks degrade while more distant tasks improve seems speculative. Why would more distant tasks benefit if partially aligned ones are negatively affected? The confidence regions in Figure 5 also raise questions about statistical significance. I understand the empirical results, but the claims about why they might be happening should be acknowledged as purely speculative.

  • Improper experiment control design (Sections 3.2 and 4.1) : SDPO exhibits surprisingly severe degradation during continual training, seemingly contradicting prior SDFT results. Moreover, SDPO uses a 10× larger learning rate and larger batch size than GRPO. It is unclear whether the observed forgetting and parameter drift arise from dense supervision or these hyperparameter differences. Comparing methods at matched in-domain accuracy is needed to make some of the conclusions.

Questions for authors
  • What evidence supports CoT unreliability as the explanation for degraded performance?

  • Why is GRPO excluded from Figure 5? Are the intermediate-distance interference results statistically significant?

  • What explains the discrepancy with SDFT's continual-learning results? Were SDPO hyperparameters adequately tuned? It would be nice to see a matched in-domain accuracy analysis. Using a 10x larger learning rate adds a confound, nothing fundamental can be claimed about the differences in the properties of GRPO and SDPO if the learning rate differences are this large.

  • The authors attribute SDPO's extreme collapse partly to repeated \boxed tokens. Is this actual forgetting of capabilities or degeneration of output formatting? These are different failure modes and should be distinguished.

  • To my understanding, SDPO is a lot more sample efficient than GRPO. However, that does not seem to be discussed anywhere? If SDPO is a lot more sample efficient, doesn't that also mean that it needs to be trained for lesser steps than GRPO, and the current experiment overtrains SDPO which might naturally lead to more forgetting. This is why I believe matched in-domain accuracy might be the right way to compare SDPO to GRPO.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.