acceptodds
Under review as a conference paper at ICLR 2027

Cover the Teacher, Stay Close to the Student: Mode-Covering Trust Regions for On-Policy Distillation

Abstract

On-policy distillation should preserve teacher-supported alternatives at student-generated prefixes, yet its usual reverse-KL signal is mode-seeking: alternatives assigned little probability by a capacity-limited student receive weak supervision and can disappear. Proximal Target Distillation (PTD) makes mode covering explicit. It projects the teacher onto a student-relative KL ball, preserving teacher-supported modes while bounding target movement, then fits that target with forward KL to train underrepresented tokens directly. On the normalized teacher-top- support, the projection follows a closed-form normalized Lambert- path whose radius bounds the initial conditional-logit gradient. In a misspecified Gaussian study, repeated PTD updates recover substantial probability in a teacher region initially nearly absent from the student. Controls identify forward fitting as the recovery mechanism and show that proximal target geometry controls the recovery trajectory. Across math-only and teacher-routed multi-domain language-model experiments, PTD delivers the highest Macro best-of- point estimate among trained student methods. Its advantage over top- reverse KL widens with the sampling budget and overtakes linear interpolation at the larger budget. The controlled density recovery and the growing multi-sample advantage jointly show how a mode-covering objective preserves useful alternatives that a mode-seeking update can lose.

Primary area:
optimization
Keywords:
on policy distillation, KL divergence

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LLMtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Proximal Target Distillation, which aims to incorporate teacher guidance under a KL-based constraint tied to the student model. The method is evaluated on mathematical and scientific reasoning tasks, using DeepSeek-R1-Distill-Qwen-1.5B as the student and Skywork-OR1-Math-7B and Skywork-OR1-7B as the respective teachers. The paper also includes a synthetic Gaussian analysis to illustrate the proposed approach.

Critical strengths

The paper is reasonably organized, with an identifiable progression from the proposed method to the synthetic analysis and LLM experiments. This structure helps readers follow the intended contribution and locate the supporting results.

Critical weaknesses
  1. The mathematical formulation is insufficiently justified in the context of LLM post-training.

The assumptions underlying the mathematical development are not convincingly connected to the practical LLM post-training setting considered in the main experiments. Consequently, it is unclear what the analysis establishes about the implemented method. The authors should explicitly identify which assumptions hold in their training procedure, which require approximations, and how departures from these assumptions affect the conclusions. As presented, the mathematical development does not provide convincing support for the method’s practical use.

  1. The Gaussian synthetic experiment provides limited insight relative to the space it occupies.

The synthetic analysis considers a highly simplified Gaussian setting, but its relevance to the main LLM experiments is insufficiently established. Devoting approximately a full page to this analysis seems disproportionate to the insight it provides. The paper should explain what specific mechanism or prediction the experiment establishes and demonstrate its relevance in the LLM setting. Otherwise, the presentation should be shortened, with additional details moved to the appendix.

  1. The main experiments lack essential baseline comparisons.

The baseline coverage is insufficient to establish the benefit of the proposed distillation method. In particular, the paper should include straightforward alternatives such as supervised fine-tuning on a corpus of teacher-generated responses and rejection sampling. These comparisons are needed to determine whether the proposed formulation offers an advantage over simpler ways of using the same teachers. Data, teacher-generation budgets, and student-training budgets should be clearly reported and made comparable where feasible.

  1. The reported improvements are modest, even against the limited set of included baselines.

The empirical gains are small and do not yet establish a compelling advantage for the proposed method. This concern is particularly relevant given the missing baseline comparisons: the current results leave open whether similar or better performance could be achieved through simpler distillation strategies. The authors should substantiate the consistency and reliability of the improvements and explain the conditions under which the method provides a meaningful benefit. Appropriate uncertainty or variability estimates would help readers interpret these small differences.

  1. The presentation contains excessive numerical precision but insufficient analysis.

The numerical results are reported with an unnecessarily large number of digits, making the presentation harder to read without providing a correspondingly clearer understanding of the findings. The reported precision should be appropriate to the evaluation’s resolution and variability. More substantive analysis would be valuable, particularly regarding sensitivity to the strength of the KL constraint, differences across tasks, and the circumstances in which the method helps or fails to improve performance. Such analysis would better explain the contribution of the proposed formulation.

Questions for authors
  1. Practical relevance to LLM post-training. What practical advantage does the proposed method offer in realistic LLM distillation scenarios? In particular, how do the assumptions underlying the mathematical analysis correspond to the actual LLM training procedure, and under what conditions should the method be preferred over simpler alternatives?

  2. Comparison with stronger baselines. How does the method compare with supervised fine-tuning on teacher-generated responses and rejection sampling under comparable data and compute budgets? These comparisons would help establish whether the proposed formulation provides a meaningful benefit over simpler ways of using the same teachers.

  3. Hyperparameter selection and justification. What is the rationale for the chosen hyperparameter values? Their selection currently lacks clear references or justification. Please clarify which settings follow prior work, which were tuned, and how the tuning was performed. How sensitive are the main results to these choices, particularly the strength of the KL constraint?

Reviewer_1uv7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Proximal Target Distillation (PTD), an approach for on-policy knowledge distillation that aims to mitigate the mode-seeking behavior associated with conventional reverse-KL objectives. The proposed method projects the teacher distribution onto a KL-constrained region around the student distribution and subsequently trains the student to approximate the resulting target using forward KL divergence. The authors derive a closed-form characterization of the projection using the Lambert-W function and present theoretical properties of the resulting optimization framework. The approach is evaluated through a synthetic Gaussian experiment illustrating mode recovery and through language-model distillation experiments on mathematical reasoning and multi-domain tasks. The reported results indicate improvements in best-of-n performance, particularly as the sampling budget increases.

Critical strengths
  • The proposed approach addresses a relevant challenge in on-policy distillation, namely the potential loss of teacher-supported alternatives due to the mode-seeking behavior of reverse-KL optimization. Combining a mode-covering objective with a trust-region constraint is an appealing strategy for balancing knowledge transfer and controlled updates.

  • The experimental results suggest that the proposed method can improve distillation performance, particularly when evaluated using larger sampling budgets. The inclusion of controlled experiments attempting to distinguish the effects of forward-KL fitting and proximal target geometry is also valuable for understanding the behavior of the proposed optimization framework.

Critical weaknesses
  • My primary concern is that much of the proposed methodology and theoretical analysis appears to follow from established concepts in constrained optimization and KL divergence minimization, rather than introducing fundamentally new principles specific to on-policy distillation. While applying these ideas to on-policy distillation may certainly be valuable, the paper should distinguish more clearly between established properties of the underlying optimization framework and genuinely new insights arising from its application to this problem.

  • The experimental study is relatively limited compared with the considerable space devoted to the optimization formulation and its theoretical properties. Over a page is dedicated to a synthetic Gaussian experiment that appears primarily to illustrate generic properties of the optimization framework, rather than its behavior in an actual on-policy distillation setting.

  • I also found the exposition difficult to follow: the paper introduces the proposed optimization machinery rather directly, without first clearly formulating the specific limitations of existing on-policy distillation approaches and explaining why the proposed formulation is particularly suitable for addressing them

Questions for authors
  • Could the authors clarify which theoretical results are genuinely specific to the on-policy distillation setting, as opposed to being direct consequences of established properties of KL divergence and constrained convex optimization? For example, Proposition 1 appears to follow directly from Pinsker's inequality, while the results in Section 3.5 seem to arise from standard properties of the underlying optimization problem. What additional insights do these results provide regarding the behavior of on-policy distillation?

  • Could the authors explain more explicitly why the particular combination of a KL-constrained teacher projection and forward-KL fitting is preferable to simpler alternatives, such as directly minimizing forward KL or combining forward and reverse KL through a weighted objective? In particular, what concrete limitation of these alternatives is addressed by the proposed proximal projection, and to what extent is its advantage established by the experiments?

  • What is the intended connection between the synthetic Gaussian experiment and the actual on-policy distillation setting? The experiment appears to demonstrate properties of the proposed optimization framework independently of on-policy sampling or language-model training. Moreover, the experiment considers continuous-valued distributions, while the proposed formulation operates on discrete distributions over a truncated top-K support. Could the authors clarify which conclusions from this experiment can be expected to carry over to the practical distillation setting?

  • Could the authors provide a more detailed comparison with existing approaches that employ forward-KL or hybrid forward/reverse-KL distillation objectives (e.g., Jin et al. ICML26), preferably under comparable experimental conditions?

  • (Minot) What does the variable (z) represent in the gradient (\nabla_z) in Equation (11)?

Reviewer_q3aoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Proximal Target Distillation (PTD), a variant of on-policy distillation that, instead of directly minimizing the reverse KL divergence, constructs an intermediate reverse-KL constraint and minimizes the forward KL divergence for mode covering. The authors further show its relationship to a normalized Lambert-W solution. Experiments using DeepSeek-Distill-Qwen-1.5B show improvements over naive OPD in both the math and science domains.

Critical strengths
  • The authors provide an interesting and principled target construction that formulates teacher coverage and student proximity as a constrained optimization problem and shows its relationship to a normalized Lambert-W path.
Critical weaknesses
  • Weak Empirical Claims. The empirical claims seem rather marginal and weak. The authors conduct experiments using only a single model, with two naive OPD baselines and a linear-interpolation variant of PTD. Furthermore, the reported numbers in Table 2 show that both top-k OPD and sampled-token OPD, which have themselves been established as valid post-training algorithms, degrade performance relative to the base model, raising questions about the validity and soundness of the training setup.
  • Limited Novelty. Stabilizing OPD has been widely studied in recent works. As also mentioned in the related work, methods such as EOPD, TrOPD, and TRB have all explored similar approaches, some of which already use selective forward-KL-divergence losses. While PTD introduces new formulations, the novelty seems incremental, and it is unclear why PTD should be, or is, better than these methods. The justification for why this OPD variant should stabilize training more effectively than prior work remains unclear. The authors should provide a rigorous theoretical and/or empirical comparison showing how PTD differs from and is superior to established prior methods, especially since the paper does not include these methods as baselines.
Questions for authors
  • Why does switching to forward KL and seeking mode-covering behavior the right question in the first place? While the authors somewhat discussed the pitfalls of reverse-KL loss, and prior studies have shown that naive OPD can be unstable when teacher–student disagreement is large, the mode-seeking nature of reverse KL has been characterized as an important factor in LLM post-training, from RLHF and RLVR to OPD, compared with mode-covering SFT. It seems more natural to regulate only the unstable cases, as done in REOPOLD or EOPD, rather than completely reformulate the loss.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.