- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates rollout sources for OPSD, where the teacher is the current student conditioned on privileged information. It argues that entropy collapse makes on-policy trajectories progressively less useful and proposes generating trajectories from an EMA student or replaying a fixed offline pool. A high-entropy mask restricts distillation to the most uncertain tokens. Experiments across three Qwen3 sizes on math benchmarks and Knights and Knaves dataset show that offline or EMA trajectories with masking improve performance over standard OPSD. Offline training also reduces training-loop time by removing live rollout generation.
Critical strengths
- The paper challenges the common assumption that self-distillation should rely on on-policy rollouts from the perspective of entropy dynamics. The contrasting entropy dynamics of external-teacher distillation and self-distillation, together with the analysis of high-entropy positions in offline traces, provide valuable insights into how rollout distributions affect self-distillation.
- The proposed EMA and offline self-distillation variants are straightforward to implement and achieve significant improvements across multiple model sizes and two reasoning domains. The factorial ablation of rollout sources and entropy masking is also valuable.
Critical weaknesses
- The methodological novelty of this paper is relatively limited. The main proposed approach essentially shifts the rollout generation strategy from on-policy to more off-policy settings in the context of self-distillation.
- The experimental evaluation is relatively limited, focusing primarily on mathematical reasoning tasks and the Knights and Knaves dataset. The effectiveness of the proposed methods should also be validated on tasks in other domains, such as code generation and scientific knowledge reasoning. The paper addresses a fundamental question in knowledge distillation, namely the relative merits of on-policy and off-policy training. Therefore, a more comprehensive and rigorous empirical evaluation would provide more valuable insights and practical guidance to the broader research community, while also strengthening the paper's overall evaluation and potentially leading to a higher review score.
Questions for authors
- In the leftmost subplot of Figure 4, why does the accuracy of the semi-online + mask configuration drop below that of the online setting during the final few training steps?
- In Table 2, why does the online OPSD + mask configuration exhibit such an extremely low final accuracy of only 1.8%? Given the substantial fluctuations in model accuracy throughout training, why do the authors not conduct experiments with multiple random seeds to reduce the influence of random variations on the experimental conclusions?
- Why are the experiments on the Knights and Knaves dataset conducted only with Qwen3-4B, without including results for Qwen3-1.7B and Qwen3-8B?