- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper tackles the distribution drift issue in on-policy self-distillation where the self-teacher is provided with privileged context that leads to supports rarely visited by the student policy itself. Hellinger loss was employed based on geometric mean of student-teacher overlap to reweigh teacher gradients. Fisher–Rao distance is further adopted to penalize the predictive drift across updated model checkpoints. In-domain and OOD domain performance gain were demonstrated comparing to methods using KL style distillation without reweighing.
Critical strengths
The paper explores an important challenge of covariate drift, where the self-teacher provides dramatically deviant distribution that the student, leading to improved in-distribution performance while degrading generalization.
The experiments cover multiple model families and sizes, with component ablations and analyses of predictive drift and probability concentration.
Critical weaknesses
- Novelty: I don't think the distribution drift issue is new or specific to on-policy self-distillation. Learning on samples from distributions of behavior policy's that differ from the target policy is well-studied in offline or offpolicy RL. The paper frames this problem in self-distillation setting, which is fine. But even within the distillation literature, reweighting teacher signals by student probability mass has been actively studied. Authors should compare with these work in experiments. The reweighting scheme (Eq3 and Eq4) also relies on established density distance measurement. What is the real novelty of this work is worth more elaboration.
- Lacking baselines: Baselines or related work do not include reweighting schemes such as using student probability as dynamic importnace weights https://proceedings.iclr.cc/paper_files/paper/2026/hash/2e53c02ea028cbf603f4b6b47fef3d97-Abstract-Conference.html or its variants https://arxiv.org/pdf/2606.09396 https://arxiv.org/html/2509.23753v1 which are directly relevant to the paper’s core reweighting contribution.
- Generalization: Code transfer is evaluated only on Qwen3-8B, and no evaluation of general language understanding (e.g., MMLU) or instruction-following quality is reported.
Questions for authors
1 How this methods are comparable to work with reweighting schemes?
2 How much impact this method has to model's general utility, compared with other work?
Rebuttal
Thank you for the positive assessment and the pointers to the reweighting literature. We address your questions below. We have added new experiments and clarifications to the revised manuscript.
We agree that importance weighting is classical. The works you point to apply it to supervised fine-tuning: DFT scales each token's loss by the model's own probability, Anchored SFT adds a KL anchor to the base model, and PriFT weights tokens by a frozen pretrained model's probability. Within on-policy distillation, reverse KL is a natural analog, since it also weights by student probability, and it degrades OOD accuracy in all three families (Table 1).
Our contribution is not the reweighting itself, but showing why it is crucial in privileged-context OPSD and what it requires there. Empirically, with the same penalty and optimizer, changing only the per-token weighting moves OOD accuracy from 50.6 (forward KL) to 56.1 (JSD) and 60.5 (Hellinger) (Table 2). Combined with the moving anchor, GeoSD improves OOD accuracy over the base model by 5.7--8.6 points across three families.
Reweighting matters especially here because the teacher can be confident simply because it sees a solution the student never sees: at 27% of high-entropy states, the teacher assigns at least 0.8 to a token the student gives at most 0.1 (Figure 8). Because teacher and student also share parameters, reweighting alone is not sufficient: without the proximal penalty, Hellinger matching reaches only 53.8 OOD (Table 2). This motivated our geometric approach, in which the matching loss and the drift penalty both measure differences between predictive distributions relative to each token's probability, so the two terms are compatible.
Prompted by your remark, we conducted additional experiments on general knowledge and scientific reasoning. We have also extended code-transfer evaluation to all three model families. We added this to the revised manuscript (Appendix C.9, Tables 11 and 12) and we report the results below.
Cross-domain transfer to code holds across all three families: GeoSD achieves the best code accuracy in every family (+1.9, +1.7 and +2.7 over π0), while forward and reverse KL degrade code accuracy in every family.
On two additional benchmarks that cover general knowledge (MMLU-Pro) and scientific reasoning (GPQA-Diamond), GeoSD is the only evaluated method that improves on both datasets. The results in the table below are with Qwen3-8B.