acceptodds
Under review as a conference paper at ICLR 2027

Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation

Abstract

On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory. However, this interpretation conflates two effects. The reference solution not only reveals the answer to the current instance but also changes the context under which the teacher provides token-level supervision. We investigate the role of target-specific privilege with OPSD (On-Policy Self-Distillation from Other Problems), which replaces the paired reference with a problem and solution from a different example, while preserving the student rollout, teacher, and distillation objective. Across three models and three mathematics benchmarks, OPSD improves over the base model, remains competitive with OPSD. The success of OPSD implies that OPSD gains do not necessarily come from access to the reference solution, and that the teacher's context-induced behavior is an important factor.

Primary area:
foundation or frontier models, including LLMs
Keywords:
knowledge distillation, on-policy distillation, self-distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5dBPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on On-Policy Self Distillation with privilege information. Notably, traditional approach consider privilege information to be the answer to the current question. In this work, this paper consider changing the privilege information to a fully derived problem from a different question. Throughout this process, this paper shows that improvement are still observed despite the absence of the answer in the privileged information given to the teacher during self distillation.

Critical strengths

The problem definition is particularly interesting and relevant. Especially, the authors claim that current evaluation mostly rely on accuracy gain, leaving little to no explanation as to where this gain comes from. Therefore, the motivation is sound and interesting.

Critical weaknesses
  1. Despite criticizing current literature for the lack of interpretation beyond accuracy (l.77-78), I am unsure that this study is much different. Notably:
  • There is a discrepancy of performances gain between reasoning and non reasoning model performances; which is never discussed (see Table 1). Table 11 in the appendix confirms that when the model is reasoning, most conclusions of the paper are weakly supported by the experiments.
  • Trivial context (1+1) makes self distillation decrease performances from around 30 to 5 (see table 3). Why is that? This should be discussed and analyzed.
  • Why would OP2SD work better that OPSD? This should also be discussed.
  1. The breath of experiments seems a bit limited to support all claims. More variety in datasets, distillation policies, thinking and non-thinking models might help support the claims. It feels to me that the paper tells an intuitively convincing story but I am unsure that this generalizes.
Questions for authors
  1. What are the attentions maps for different contexts?
  2. What are the performances of the teacher for different contexts?
  3. What is the impact of using thinking vs non-thinking on the teacher output and why is the proposed method not working as well when the teacher is thinking? Does that mean that the current claim should be narrowed to non-thinking models?
  4. Can you clarify how 'context-induced teacher behavior' (l 166) is not just target-agnostic privilege information? It seems that the current presentation opposes target-specific privilege information to context-induced teacher behavior (see title), but it looks to me that what this paper actually studies is the origin of the privileged information rather than its existence.
Reviewer_N9YROpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper dissects whether the gains of OPSD come from the teacher's privileged access to the target's verified solution or from the worked-solution context itself. The variation, OP2SD, replaces the paired solution in the teacher prompt with a problem–solution pair from a different training example; the student rollout, frozen self-teacher, and objective are unchanged. Experiments across Qwen3 and three math benchmarks show OP2SD matches or exceeds OPSD, by over 9 points at 8B. A control ladder over diversity, correctness, verbosity and domain leads to the conclusion that a substantive mathematical worked-solution context drives the gain.

Critical strengths
  1. The diagnostic question is clear and the control ladder is rich and well designed.
  2. The findings are quite valueable for post-training recipes.
Critical weaknesses
  1. The claimed mechanism (changes in the teacher's token-level behavior) is never directly measured. Besides, it cannot explain the paper's result: standard OPSD is itself a "mathematical context that works," yet OP2SD beats OPSD by over 9 points at 8B, and the paper states these numbers without any explanation or hypothesis.
  2. The main comparison use different template with pairing. From Figure 2, OPSD and OP2SD teacher prompts differ in many ways, not just the unmatched answer. Therefore, "the only difference is the teacher-only context" does not hold at the implementation level.
  3. From Table 6-8, Distillation covers only ≤1,024-token prefixes, yet mean evaluation generations run ~6,979 tokens (Table 3), and the 8B advantage emerges only at large token budgets. Where the observed advantage actually comes from therefore remains unclear.
  4. Checkpoint selection is undisclosed. It is not clear that 1.7B beating 4B and 8B on all three benchmarks.
Questions for authors
  1. Appendix H shows OP2SD degrading from 100 to 400 steps but contains no OPSD arm — with both methods trained to a matched 400 steps, does the OPSD-vs-OP2SD ranking persist?
  2. Could the authors disclose, the checkpoint evaluated and its selection criterion, the full OPSD teacher-prompt template, training steps, and the evaluation mode per row for Table 1?
  3. Could the authors run a (template × pairing) ablation and report token-level divergences between the teacher conditioned on the paired vs. the mismatched solution along the same student prefixes?
Reviewer_jT9pOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether the performance gains of On-Policy Self-Distillation (OPSD) arise from target-specific privileged solutions or from changes in teacher behavior induced by additional context. The authors propose OP²SD, which replaces the reference solution to the current problem with a solution to an unrelated mathematical problem while retaining the distillation framework. Experiments across three Qwen3 model sizes suggest that unrelated mathematical solutions can achieve performance comparable to or better than conventional OPSD. Additional experiments examine fixed, corrupted, and cross-domain contexts to investigate the role of contextual information.

Critical strengths
  • Questioning whether target-specific privileged solutions are necessary for OPSD improvements is an interesting research direction. Using unrelated mathematical solutions provides a simple and informative experimental intervention.

  • The paper evaluates three Qwen3 model sizes and includes additional controls involving fixed, corrupted, and cross-domain contexts. The padding, thinking-mode, and extended-training analyses are also useful for identifying potential limitations of the approach.

Critical weaknesses
  • Incomplete framing of the underlying hypothesis: The paper explains OPSD improvements through two factors: target-specific privileged information and context-induced teacher behavior. However, prior studies [1,2,3] have highlighted another important possibility: privileged-context distillation can primarily affect stylistic or uncertainty-related tokens and compress reasoning trajectories rather than improve reasoning itself. The paper does not adequately distinguish these effects, raising concerns about whether the reported gains reflect meaningful reasoning improvements or merely changes in generation behavior.

  • Inconsistent thinking-mode configurations: The paper uses Student OFF / Teacher ON for Qwen3-1.7B, but Student OFF / Teacher OFF for Qwen3-4B and 8B. Although the 1.7B setting follows the original OPSD implementation, the rationale for using different configurations across model sizes is insufficiently justified. Moreover, Appendix J shows that OPSD's improvement disappears when both student and teacher use thinking mode, raising concerns about the sensitivity of the results to this choice.

  • Personally, I find it somewhat awkward to perform OPSD using models that have a thinking-mode switch, either enabling thinking mode for the teacher while disabling it for the student, or disabling it for both. Wouldn't it more natural to either enable thinking mode for both the student and teacher when we use Qwen thinking models, or just use non-thinking models or use models without a thinking-mode switch, such as DeepSeek or OLMo, to maintain a consistent experimental setting? I understand this paper follows the prior paper's setting, but I would appreciate further author's own clarification on the motivation for using different thinking-mode configurations across experiments.

  • Insufficient training stability analysis: Training rollouts are limited to 1,024 tokens, and Appendix H already reports performance degradation when extending OP²SD training from 100 to 400 updates. However, the paper does not provide full evaluation curves or systematic comparisons between best and final checkpoints. I would like to see evaluation curves for both OPSD and OP²SD up to at least 1,000 steps, along with experiments using longer rollout lengths.

  • Sensitivity to implementation details: Appendix E shows that changing the padding convention substantially affects performance. Although OP²SD still outperforms OPSD under the alternative padding setting, this sensitivity raises concerns about the robustness and reproducibility of the results. Further validation across implementation settings would strengthen the conclusions.

[1] Kim, Jeonghye, et al. "Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?." arXiv preprint arXiv:2603.24472 (2026).

[2] Kaur, Simran, et al. "Rethinking On-Policy Self-Distillation for Thinking Models." arXiv preprint arXiv:2607.05184 (2026).

[3] Pan, Leyi, et al. "RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation." arXiv preprint arXiv:2606.11709 (2026).

Questions for authors
  • Could the authors provide evaluation curves for both OPSD and OP²SD up to at least 1,000 training steps, with evaluations every 100 steps, along with experiments using longer rollout lengths?

  • Could the authors conduct the same OPSD and OP²SD experiments on models without a thinking-mode switch, such as DeepSeek or OLMo, to verify whether the conclusions hold independently of this configuration choice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.