acceptodds
Under review as a conference paper at ICLR 2027

DOPD: Dual On-policy Distillation

13 views

Abstract

On-policy distillation (OPD) offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals. To furnish high-quality supervision sources and thereby elevate the performance frontier of distillation, an intuitive direction is to infuse privileged information to either teacher or student itself. However, this additional input induces a potential failure mode we dub privilege illusion: a pattern that conflates the transferable capability gap that students are meant to close, and the information asymmetry gap that can only be mimicked but never replicated. This issue is further amplified by the inherent non-uniformity of token-level supervision, where only a small subset of tokens carries pivotal capability-bearing signals. To this end, we propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies based on their advantage gap and relative probabilities. Each token receives supervision of different strength, objective, and strategy from either teacher or student itself, which transfers credible capability while simultaneously receiving auxiliary signals. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD and other counterparts. Further results on stability, robustness, continual learning, and out-of-distribution tasks validate its superiority.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, privilege illusion, token-level supervision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YXniOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies on-policy distillation (OPD) with privileged information, such as step-wise hints for LLMs or bounding boxes for VLMs, given to the teacher, the student, or both. The authors argue that this can create a “privilege illusion”. A preliminary experiment (Fig. 2) supports this, showing that adding privileged information does not necessarily improve distillation and can lead to late-stage degradation and entropy collapse. To address this, the authors propose DOPD. Both the teacher and the student receive the privileged input, and each token is routed to one of four loss terms based on the teacher-student log-probability gap and the two models’ predicted probabilities. Experiments are conducted on multiple benchmarks, plus additional analyses on teacher-student size ratios, continual learning, out-of-distribution transfer, and training stability. DOPD achieves substantial gains over vanilla OPD plus some gains over other strong baselines, although all experiments are limited to the Qwen model families.

Critical strengths
  1. the paper studies a relevant and timely problem: what is the best practice for using privileged information to improve supervision in on-policy distillation. The paper provides more insights to the problem and a strong empirical baseline for future research.
  2. The proposed method (DOPD) is simple. Although i have not checked the source code, the method seems easy to implement and relevant details are provided in the paper.
  3. The experiments are multi-faceted, making the main empirical results convincing. However, it does not outweigh the lack of theoretical justifications.
Critical weaknesses
  1. The proposal is purely intuitively driven. There is no theoretical justification for why the routing in the given way works. Besides, although I appreciate that the authors provide the examples in Figure 7, i find the classification of the four types of tokens are superficially random, where some deeper theoretical reasons might be hidden.
  2. The intuitive justification itself seems handwavy. The motivation is to tackle the “privilege illusion” and to separate capability from information for effective distillation. However, the routing strategy essentially just measures teacher-student disagreement and confidence levels. The paper never shows how exactly such strategy separates capability from information and how this resolves the "illusion". It is possible that the teacher and student are subjected to the illusion at the same time while they can agree or disagree on some tokens. Moreover, the concept of "capability" and "information" are vague and never been properly defined in the first place. Therefore, the authors should either reframe the contribution, or provide better groundings for the intuitive claims.
  3. Although the experiments cover a large set of benchmarks, the models are limited to the Qwen3 and Qwen3-VL families, with the teacher-student pairs differing only in size. It is unclear whether the gains hold for other families (e.g., Llama, Gemma) or thinking models. so the claims of “robustness, scalability, and generalization” are stronger than the evidence.
  4. The ablation experiments in Table 6 are insufficient. First, The non-adaptive baseline is only defined in one sentence (Appendix D.3) and it is unclear how each non-adaptive baselines are implemented. Second, adaptive and non-adaptive rows differ in many factors at once, making it hard to make fair comparisons and to isolate the contribution of each type of tokens.
Questions for authors
  1. How does the routing-based token selection criterion of DOPD differ from the criteria of VA-OPD, TIP, and EOPD in principle besides the empirical results?
  2. Why are not other related methods mentioned in related work compared in the experiments? e.g., TA-OPD, REOPOLD, TSD-KD, given that they are all based on some adaptive selection rules?
  3. As mentioned in the weaknesses section, can the author provide evidence of the claimed generality using more model families?
Reviewer_CK36OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DOPD, a token-level on-policy distillation framework that adaptively selects supervision from a privileged teacher or a privileged version of the student. Routing depends on their disagreement over student-generated tokens and their relative probabilities. The method jointly adjusts the supervision source, strength, divergence objective, and vocabulary coverage. Experiments report improvements across language and vision-language benchmarks.

Critical strengths
  1. The paper addresses a meaningful problem: better predictions under privileged contexts do not automatically provide better supervision for a student operating without those contexts.
  2. The privileged student provides a useful intermediate reference for deciding how to apply teacher supervision.
  3. The token-removal experiments and component ablations provide evidence that selective supervision contributes to performance.
Critical weaknesses

Major Concern: The performance gains are not evaluated against their resource costs.

My primary concern is whether DOPD provides a favorable performance–cost trade-off. The reported improvements are encouraging, but comparisons at matched training steps do not establish this, since DOPD introduces an additional privileged-student forward pass, longer conditioning contexts, and full-vocabulary alignment at selected positions.

The additional student branch does not require backpropagation, so its overhead should not be assumed to be prohibitive. However, the actual overhead needs to be measured. Specifically, the paper should report:

  • Memory: Peak GPU memory under comparable effective batch sizes and sequence lengths, with checkpointing, offloading, and gradient accumulation configurations disclosed.
  • Computation: Per-step and cumulative training FLOPs, including rollouts, teacher/student forward passes, and student backpropagation.
  • Wall-clock time: Per-step latency, total training time, and time to reach matched target performance on identical hardware.

The offline costs of generating and filtering privileged information should also be reported separately.

The key comparison is whether DOPD still outperforms strong baselines when those baselines receive the same compute or wall-clock budget, under a comparable memory constraint. Performance-versus-time and performance-versus-cumulative-compute curves would directly address this question.

I do not view additional cost itself as a weakness if it produces a favorable trade-off. Rather, the current evaluation does not provide enough information to assess that trade-off.

Questions for authors
  1. Does the privilege advantage gap predict transfer gains after controlling for gradient magnitude, supervision volume, and compute?
  2. Do the improvements persist under matched memory, cumulative computation, and wall-clock budgets?
  3. Do baselines receive equivalent data filtering and privileged supervision resources?
Reviewer_rbZzOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies on-policy distillation with privileged information and identifies a potential transfer problem termed \emph{privilege illusion}: a teacher's predictive advantage may reflect additional information rather than transferable capability. The proposed DOPD framework generates trajectories from the student without privileged inputs, then uses a privileged teacher and a privileged version of the student to score them. Their token-level log-probability gap and relative probabilities determine four supervision regimes, combining different supervision sources, loss weights, Top-KK reverse KL, and full-vocabulary JS distillation. Experiments cover 16 text and vision-language benchmarks within the Qwen3 and Qwen3-VL families, supplemented by model-size comparisons, continual-learning and cross-domain evaluations, and component ablations.

Critical strengths

(1). DOPD achieves substantial improvements across diverse tasks and model sizes. In Table 1, its average score is 51.4, compared with 47.0 for ExOPD, and it leads among the compared distillation methods on all eight text benchmarks. Its vision-language average is 58.4, compared with 56.3 for VA-OPD in Table 2. DOPD also outperforms Vanilla OPD in all five teacher--student configurations in Table 3. These results provide meaningful evidence of practical value within the tested model families.

(2). The intervention studies provide empirical support for selective supervision. Figure 3 shows that removing high-gap tokens hurts performance more than removing random or low-gap tokens. In Table 8, removing advantage-aware distillation reduces LiveBench performance from 49.8 to 41.3; removing student supervision or adaptive loss designs also lowers performance. These results support the usefulness of the integrated design beyond aggregate benchmark improvements.

Critical weaknesses

(1). The empirical usefulness of the proposed criterion is better established than its mechanistic interpretation. Equation (1), A=∣log⁡qT−log⁡qS∣A=|\log q_T-\log q_S|, measures prediction disagreement under shared privileged inputs, but this disagreement may also reflect calibration, prompt sensitivity, or differences in how the models exploit privileged information. Conversely, low disagreement may indicate knowledge already shared by both models. Figure 3 and Table 6 demonstrate the training value of certain high-gap tokens, but do not distinguish these alternative explanations. Moreover, the strong-teacher branch uses relative probability comparisons without establishing absolute confidence or correctness. The evidence therefore remains insufficient to establish that the criterion separates transferable capability gaps from information gaps.

(2). The main comparisons leave important supervision conditions unspecified. Appendix C.3 states that the baselines are rerun using the same models, but does not clearly identify which methods receive identical filtered examples and privileged inputs, or which divergence objective and supervision granularity each baseline uses. This matters because DOPD combines Top-KK and full-vocabulary supervision. Table 7 provides useful granularity ablations on LiveBench, but does not clarify the configurations underlying Tables 1 and 2. Consequently, the contribution of routing is not fully isolated from differences in information access and supervision density. Furthermore, efficiency is primarily evaluated against training steps, despite an additional student forward pass and privileged-data construction costs. Without measured runtime or comparisons under matched computational budgets, the practical cost--performance benefits remain uncertain.

Questions for authors

(1). With privileged data, supervision sources, loss formulations, and budgets kept as comparable as possible, how does DOPD perform when Equation (1) is replaced with unprivileged teacher--student disagreement, token surprisal, or entropy? What evidence shows that the proposed criterion identifies transferable capability beyond selecting informative or high-disagreement positions?

(2). Please specify the training-data subset, privileged inputs, divergence objective, and supervision granularity used by each method in Tables 1 and 2. Beyond the sensitivity analysis in Figure 10, how were the loss coefficients selected, and were the baselines given comparable tuning opportunities? Do the improvements over strong baselines persist under comparable privileged supervision and computational budgets?

(3). How does privilege illusion differ substantively from the information asymmetry, privileged-information leakage, and late-training degradation analyzed in the already-cited RLSD work? The description as a previously uncharacterized failure mode and the discussion in Appendix A.2 appear too broad. Please clarify the new conceptual contribution separately from the dual-source token-routing algorithm. See Yang et al. (2026a), \href{https://arxiv.org/html/2604.03128v2#S3}{\textit{Self-Distilled RLVR}, Section~3}.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.