acceptodds
Under review as a conference paper at ICLR 2027

OPMD: On-Policy Mechanism Distillation Beyond Output Supervision

15 views

Abstract

On-policy distillation (OPD) improves language-model by supervising a student on prefixes sampled from its own policy, but its supervision remains confined to the output space. Consequently, OPD transfers what a teacher predicts without supervising the intermediate computation that produces those predictions. Simply distilling intermediate representations like representation matching is insufficient since teacher and student representations may differ geometrically, and even well-aligned hidden states need not be usable by the student's subsequent computation. We introduce On-Policy Mechanism Distillation (OPMD), which transfers teacher-derived intermediate computation while explicitly accounting for its compatibility with the student. OPMD first constructs a student-compatible teacher that transforms teacher representations into intermediate states usable by the student's downstream computation, then it internalizes these states from on-policy trajectories. OPMD achieves the highest average accuracy among the evaluated distillation baselines in all three teacher–student settings, and generalizes to multi-teacher distillation and cross-family supervised fine-tuning. Further analyses show that mechanism supervision yields more effective reasoning and meaningfully alters the student's intermediate computation and solution strategies, supporting that it transfers information beyond output supervision alone. Together, these results establish mechanism-level supervision as an effective way to move on-policy distillation beyond output supervision, enabling students to learn inner mechanisms they can use and internalize.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, On-Policy Distillation, Knowledge Distillation, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PDAhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes On-Policy Mechanism Distillation (OPMD), a two-stage framework for transferring teacher-derived intermediate information into a student language model. First, it trains layer-wise projectors to inject teacher representations into a frozen student, optimizing the resulting output behavior to construct a “student-compatible teacher.” Second, the native student learns to reproduce this composite model’s hidden states on student-generated prefixes. The main on-policy experiments train student LoRA parameters using only latent-state matching during this second stage; the teacher and projectors are discarded at inference. Evaluation covers three teacher–student settings on mathematical reasoning benchmarks, alongside multi-teacher math/code distillation and cross-family supervised fine-tuning. OPMD achieves the highest average accuracy among the evaluated baselines in each main setting, although gains vary across tasks.

Critical strengths

(1) The proposed OPMD provides a coherent approach to constructing usable intermediate supervision. Training projectors through downstream output behavior gives a concrete operational meaning to compatibility. This is consequential because it addresses a limitation of geometric alignment and provides a practical alternative for constructing distillation targets.

(2) OPMD achieves the highest average mathematical accuracy in each of the three main teacher–student settings. Including OPRD-Bridge strengthens the comparison beyond output-only methods. These results support the usefulness of the proposed intermediate supervision through task performance rather than representation similarity alone.

(3) The training-time and FLOPs comparisons include both stages and indicate a favorable accuracy–cost trade-off in the examined configuration.

Critical weaknesses

(1) Weight patching establishes that learned parameter updates affect student performance, but does not identify which teacher computations were transferred or distinguish these effects from ordinary distillation. The solution-route analysis examines only two problems selected using observed performance differences, with 16 responses per condition. Its comparisons primarily concern patched and fully distilled students, rather than teacher–student computational correspondence. These analyses are informative exploratory evidence, but do not substantiate the broader claim of transferring how the teacher reasons.

(2) The presented tables and curves lack accompanying uncertainty estimates and clearly specified seed counts, while several gains are small or task-dependent. For example, the 0.61-point HumanEval+ improvement corresponds to approximately one additional solved problem; cross-family supervision also decreases AIME 2024 accuracy.

Questions for authors

(1) For OPRD-Bridge and the projected-MSE controls, which projector architectures, capacities, layer correspondences, training contexts, and optimization budgets differ from OPMD? In particular, does the MSE control use the same residual injection interface as Equation (3), or state replacement? These details determine whether the observed differences isolate functional versus geometric target construction.

(2) How many independent training seeds contribute to each main result and each “available-seed mean,” and are those counts consistent across methods and checkpoints? Can the authors provide uncertainty estimates for the principal comparisons that account for both training variability and shared evaluation problems? This is particularly important for the smaller gains and mixed cross-family results.

Reviewer_eFZKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies intermediate supervision for on-policy distillation, focusing on the distinction between geometric alignment and usability by the student’s downstream computation. OPMD has two stages. First, it freezes the teacher and initial student and trains projectors that inject teacher states as residual updates into the student. An output objective optimizes this composition into a student-compatible teacher. Second, it freezes the composition and trains a native student through LoRA to match its hidden states on student-generated trajectories. The main on-policy experiments use only hidden-state supervision in Stage II, but Stage I still uses an output-space objective. The teacher and projectors are removed at inference. Experiments cover three mathematical teacher–student settings, two-teacher mathematics/code distillation, and cross-family SFT. OPMD achieves the highest three-task average in each main setting, though not the highest score on every metric. The paper also examines response length, training cost, layer-weight patching, and solution-route changes on two selected problems.

Critical strengths
  1. The method demonstrates practical gains across several settings rather than relying on an isolated successful example.
  2. Student-compatible teacher construction provides a concrete and practically useful approach to heterogeneous intermediate supervision.
Critical weaknesses
  1. The additional value of hidden-state internalization over output supervision from the constructed teacher is not isolated, limiting attribution of the central contribution. Stage II changes both the supervision interface and the source of supervision: OPMD uses a student-compatible teacher constructed in Stage I, whereas standard OPD uses the original teacher. Table 1 and Appendix D provide meaningful comparisons with output distillation and geometric representation matching, but do not hold the constructed teacher fixed while comparing its output-distribution supervision against its hidden-state supervision. The results therefore establish pipeline effectiveness without sufficiently distinguishing gains from teacher construction from those requiring the proposed intermediate-state interface. This materially affects the incremental value claimed beyond output supervision.
  2. The evidence more directly supports changes in student computation than the stronger interpretation of teacher-specific mechanism transfer. Weight patching shows that subsets of the learned updates have functional effects, and the two cases illustrate changes in solution routes. However, these analyses primarily compare the full OPMD student with partially patched variants; they do not directly establish correspondence with the teacher’s computation or show, through an analogous OPD control, that the changes are specific to mechanism supervision. The authors appropriately acknowledge case-selection and causal-interpretation limitations, yet the abstract and conclusion retain stronger language about learning and transferring internal mechanisms. The evidence more directly supports teacher-informed intermediate supervision changing and improving student computation, rather than confirming transfer of teacher-specific mechanisms. This distinction does not require identical internal implementations in teacher and student, but affects the strength of the central explanatory contribution.
Questions for authors
  1. Have you compared training the native student through output distributions versus hidden states from the same frozen student-compatible teacher? With shared Stage I construction, student initialization, and comparable on-policy sampling and update budgets, how do the two approaches perform?
  2. Does mechanism transfer operationally mean learning teacher-informed intermediate states usable by the student, or does it additionally mean transferring teacher-specific computations or solution strategies? If the latter is intended, which analyses directly connect the teacher, constructed teacher, and native student, rather than only showing changes before and after student fine-tuning? Please also identify any analogous OPD weight-patching or strategy analyses.
Reviewer_6gnKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes On-Policy Mechanism Distillation (OPMD), which supervises a student’s intermediate hidden states rather than only its outputs. It works in two stages:

  • Stage I (student-compatible teacher). With the teacher and student both frozen, learned projectors inject teacher hidden states into the student as residual updates. The projectors are trained so the combined model’s outputs match the teacher’s, using full-vocabulary reverse KL, or SFT when the tokenizers differ.

  • Stage II (internalization). A LoRA student is trained on its own rollouts to reproduce that combined model’s hidden states via MSE, with no output loss. The teacher and projectors are discarded at inference.

Critical strengths
  1. The paper distinguishes representations that are close to the student’s (low MSE) from representations the student’s later layers can actually use, and Fig. 1b demonstrates the difference convincingly.

  2. The Stage II targets live in the student’s own representation space, so no cross-model mapping is needed during internalization.

  3. The best case is RL-Math → Qwen3-1.7B, with AMC at 40.16 vs. 33.12 and AIME25 at 7.08 vs. 3.13. OPMD also reaches higher accuracy with shorter responses than OPD (Fig. 4).

  4. Fig. 5 includes Stage I in OPMD’s cost, and OPMD still reaches 38.96% at 768 updates against 36.50% for OPD at 1,024.

  5. The annotation protocol uses masked re-labeling and reports agreement statistics, and the limitations section acknowledges how the case-study problems were selected.

Critical weaknesses
  1. OPMD distills from an intermediate model (frozen student plus projectors) that is closer to the student than the raw teacher is. That alone could explain the gains, as in teacher-assistant distillation (Mirzadeh et al., AAAI 2020). Running output-level OPD from this intermediate model would show whether the hidden-state supervision matters, which is the paper’s central claim. The intermediate model’s own accuracy is also never reported.

  2. All results are single training runs; App. B says the curves average repeated evaluations of fixed checkpoints, not independent seeds. The gains are mostly on AMC (40 problems):

  • AIME24 is tied with OPD for Qwen3-4B.
  • On RL-Math → Qwen2.5-3B, OPMD is beaten on both AIMEs (by OPRD-Bridge on AIME24; by SFT and OPD on AIME25).
  • The HumanEval+ gain in Table 2 (0.61 points) is a single problem.
  • In cross-family SFT, AIME24 drops, and both SFT variants fall below the base student.
  1. The Stage I budget may have been tuned on test data. Stage I uses 128, 96 and 128 updates across the three settings (Table 7), and App. C examines this budget using AMC accuracy. Please clarify how the budgets were chosen.

  2. The OPD + projected-MSE variant collapses at all three loss weights, with almost every response hitting the length cap (App. D). The MSE-projector patching in Fig. 1b gives 0%, below the unmodified student. This points to scale or normalization problems rather than a fundamental limit of matching, and the two setups also differ in normalization (OPMD’s loss divides by the hidden dimension; the baseline’s does not). A normalized or cosine-matching baseline would be fairer.

  3. The weight-patching and solution-route analyses (Tables 4–5) have no OPD control, so they don’t show that OPMD transfers anything OPD doesn’t. The route study rests on two problems chosen after the fact and labeled by an LLM annotator.

  4. The main experiments use same-family, largely shared-tokenizer pairs with LoRA students. The only cross-family result (SFT with Llama) is mixed.

  5. The layer pairs and supervised layers used in the main experiments are not specified.

Questions for authors
  1. How does output-level OPD from the student-compatible teacher compare with OPMD?

  2. What accuracy does the student-compatible teacher itself reach?

  3. How were the Stage I budgets chosen?

  4. What is the seed variance for Table 1?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.