acceptodds
Under review as a conference paper at ICLR 2027

Train Where the Model Is Uncertain: Off-Policy Traces for Self-Distillation

Abstract

On-policy self-distillation (OPSD) has emerged as an effective approach for improving large language models. Motivated by the success of on-policy data in reinforcement learning from verifiable rewards (RLVR) and external-teacher distillation, prior work has largely assumed that self-distillation should likewise train on rollouts from the current student policy, leaving off-policy and offline alternatives underexplored. Yet offline training is often more practical. Privileged information such as human feedback may be available only for a fixed corpus of previously collected trajectories and impossible to obtain on live rollouts from the current student policy. Moreover, maintaining a live rollout loop adds substantial computational cost. We study self-distillation when rollouts are generated by policies other than the current student. Surprisingly, offline rollouts outperform on-policy rollouts. We trace this effect to entropy collapse: self-distillation progressively reduces the student's predictive entropy, causing its own rollouts to contain fewer high-entropy tokens—the positions where the privileged-context teacher can provide useful corrections. Offline traces preserve more of these high-entropy positions because they were generated independently of the student's evolving policy. Building on this observation, we introduce two approaches that control rollout staleness to preserve opportunities for the teacher to correct the student. In EMA OPSD, rollouts are generated by an exponential moving average of the student, providing a stale but continually updated rollout policy. In offline OPSD, rollouts come from a fixed corpus generated by other models before training, eliminating rollout generation during training entirely. We additionally introduce high-entropy token masking to focus distillation on uncertain positions. Across mathematical reasoning benchmarks and multi-step Knights and Knaves problems, both approaches substantially outperform standard online OPSD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Off-Policy, Self-Distillation, LLM, Finetuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iZFLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates rollout sources for OPSD, where the teacher is the current student conditioned on privileged information. It argues that entropy collapse makes on-policy trajectories progressively less useful and proposes generating trajectories from an EMA student or replaying a fixed offline pool. A high-entropy mask restricts distillation to the most uncertain tokens. Experiments across three Qwen3 sizes on math benchmarks and Knights and Knaves dataset show that offline or EMA trajectories with masking improve performance over standard OPSD. Offline training also reduces training-loop time by removing live rollout generation.

Critical strengths
  1. The paper challenges the common assumption that self-distillation should rely on on-policy rollouts from the perspective of entropy dynamics. The contrasting entropy dynamics of external-teacher distillation and self-distillation, together with the analysis of high-entropy positions in offline traces, provide valuable insights into how rollout distributions affect self-distillation.
  2. The proposed EMA and offline self-distillation variants are straightforward to implement and achieve significant improvements across multiple model sizes and two reasoning domains. The factorial ablation of rollout sources and entropy masking is also valuable.
Critical weaknesses
  1. The methodological novelty of this paper is relatively limited. The main proposed approach essentially shifts the rollout generation strategy from on-policy to more off-policy settings in the context of self-distillation.
  2. The experimental evaluation is relatively limited, focusing primarily on mathematical reasoning tasks and the Knights and Knaves dataset. The effectiveness of the proposed methods should also be validated on tasks in other domains, such as code generation and scientific knowledge reasoning. The paper addresses a fundamental question in knowledge distillation, namely the relative merits of on-policy and off-policy training. Therefore, a more comprehensive and rigorous empirical evaluation would provide more valuable insights and practical guidance to the broader research community, while also strengthening the paper's overall evaluation and potentially leading to a higher review score.
Questions for authors
  1. In the leftmost subplot of Figure 4, why does the accuracy of the semi-online + mask configuration drop below that of the online setting during the final few training steps?
  2. In Table 2, why does the online OPSD + mask configuration exhibit such an extremely low final accuracy of only 1.8%? Given the substantial fluctuations in model accuracy throughout training, why do the authors not conduct experiments with multiple random seeds to reduce the influence of random variations on the experimental conclusions?
  3. Why are the experiments on the Knights and Knaves dataset conducted only with Qwen3-4B, without including results for Qwen3-1.7B and Qwen3-8B?
Reviewer_4UtgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focus on which rollouts and tokens are actually useful for training for on-policy self-distillation. The authors find that the student’s own rollouts quickly become low-entropy, leaving fewer opportunities for the teacher to provide meaningful corrections. They therefore train on either EMA-generated or fixed offline rollouts and focus on the 10% most uncertain tokens. Offline rollouts work best for math tasks and make training about 4x faster, while EMA rollouts perform best on the harder Knights & Knaves split. Interestingly, rollout diversity seems to matter more than whether the traces are correct.

Critical strengths
  • A pool that is 69% incorrect is nearly as good rollout sources as one that is 88% correct in table 4. Both beat online OPSD by more than 10 points.
  • The authors reframe from which loss to which states and token. The motivation is very clear, privileged information such as human feedback often exists for trajectories that are already collected. And removing such rollout loop has 4x speedup.
Critical weaknesses
  • this work doesn't have an SFT baseline. If you think about offline OPSD, they're using the same prompt, traces and training framework. So there are three different SFT baselines: 1) SFT the golden solutions 2) SFT offline pools, and only keep the correct ones 3) only do the SFT on top-10% of the tokens. If you can compare GRPO or other concurrent OPSD, that would be the best
  • The authors pick 1024 as the training token cap, but 16,384 as the testing token cap? The training-testing gap would leads to worse performance.
  • Are there experiments about the discussion on online rollout has low entropy? Or the explanation is an hypothesis?
Questions for authors
  • How do you pick the checkpoints? Can you provide more seeds and error bars?
  • Is the entropy used for the mask computed at temperature 1.0 or at the loss temperature 1.1?
  • Is Qwen3 thinking mode enabled for rollouts and for evaluation?
  • How are rollouts that hit the 1,024-token cap handled?
Reviewer_UkzFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors focus on the entropy collapse issue in online policy self-distillation (OPSD). To alleviate this problem, they propose using non-on-policy student trajectories, allowing the teacher to conduct OPSD training over diverse student rollouts. Specifically, the authors introduce two off-policy rollout generation approaches: EMA and Offline. The former leverages trajectories collected prior to the current training step for optimization, while the latter utilizes pre-generated trajectories from other models. Furthermore, a strategy targeting only high-entropy tokens is proposed to prevent the model from reducing the probabilities of low-entropy tokens, thereby mitigating entropy collapse. Extensive experiments on multi-step reasoning datasets demonstrate that the proposed methods effectively alleviate entropy collapse and improve accuracy.

Critical strengths

1.The authors' motivation to leverage diverse trajectories for OPSD training is justified. 2.Empirical experiments demonstrate that the proposed method mitigates entropy collapse while simultaneously improving accuracy. 3.High computational efficiency: offline OPSD eliminates rollout generation overhead, achieving about a 4×4\times training speedup.

Critical weaknesses
  1. The paper mentions that OPSD suffers from entropy collapse while standard OPD does not, but it relies mainly on empirical plots and external citations without a concise, direct explanation. A clearer breakdown of the underlying mechanism would help readers better understand how the proposed method tackles the root cause.

  2. The core idea of OPD is for the teacher to correct the student based on the student's own language, rather than having the model learn directly from external teacher trajectories. While EMA fits into this intuition, it is unclear whether using a purely offline method to collect trajectories from other models drifts away from this philosophy. The authors should discuss this conceptual distinction.

  3. The high-entropy token masking strategy fixes the keep fraction at k=0.1 without hyperparameter sensitivity analysis. Providing ablations across different values (e.g., k in {0.05, 0.2, 0.5}) is necessary to evaluate the robustness of this design choice.

  4. The offline trace collection strategy differs between the math reasoning and Knights and Knaves datasets, yet this choice lacks justification. Furthermore, implementation details for offline collection, such as the proportion of trajectories gathered from each model, are missing, which hinders reproducibility.

  5. It would be beneficial to include a dedicated discussion on the limitations of the work.

Questions for authors

1.Did the authors consider mitigating entropy collapse through online approaches, such as sampling multiple student trajectories under different temperatures or using multi-sample rollouts at a single temperature? Were any experiments conducted along this line? 2.Do the authors attribute the main source of the proposed method's effectiveness to the diversity of the trajectories?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.