acceptodds
Under review as a conference paper at ICLR 2027

Latent-MOPD: Latent Multi-Teacher On-Policy Distillation

11 views

Abstract

On-policy distillation (OPD) trains a student on the responses it generates. Existing LLM multi-teacher OPD transfers what specialists predict through their output distributions. We introduce Latent-MOPD, to our knowledge the first representation-level multi-teacher OPD method for LLMs. It integrates existing specialists through both their predictions and the hidden states used to compute them, without additional teacher training. To coordinate representation supervision from multiple specialists, we select late-layer targets according to the teacher–student relationship, bridge unequal hidden widths with a shared projection, and group updates by domain. Each teacher's supervision gradually shifts from hidden states to token predictions, with both channels using the same routed specialist. In our main same-family setting, Latent-MOPD outperforms the token-only, representation-only and uniform-averaging baselines on all nine benchmarks across math, code and logic. With the same parameter count as each teacher, the student also surpasses the per-benchmark best teacher on a majority of these benchmarks. With larger, separately developed cross-family teachers, Latent-MOPD outperforms both single-channel baselines on all benchmarks. A same-family all-layer representation-only control remains stable with domain-pure updates but collapses when teacher domains are interleaved within an update. Our results show that a single student can integrate capabilities from several specialists through both their output distributions and internal representations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, on-policy distillation, multi-teacher distillation, representation alignment, knowledge transfer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UKnDOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Latent-MOPD consolidates multiple domain specialists into a single student by augmenting domain-routed, on-policy token distillation with late-layer hidden-state supervision from the same teacher. A crossfade schedule gradually shifts supervision from representations to token predictions. The teachers remain frozen, and inference uses only the student. The paper evaluates three settings: same-lineage 1.5B teachers distilled into a 1.5B student, larger Qwen-based 7B teachers distilled into the same student through a learned projector, and distillation from a parameter-merged initialization. The default same-lineage configuration outperforms token-only distillation on all nine reported benchmarks and exceeds the per-benchmark best teacher on five. The paper also presents layer and schedule comparisons and a batching control showing that, in an all-layer representation-only configuration, domain-pure updates are substantially more stable than interleaved updates.

Critical strengths

1.Broad and concrete multi-domain gains. Under the shared 62-step same-lineage protocol, Latent-MOPD outperforms token-only distillation on all nine benchmarks, including AIME24 (46.0→50.8), LiveCodeBench-easy (68.3→73.1), and MuSR (50.7→52.4). The 1.5B student also exceeds the best individual 1.5B teacher on five benchmarks. This is consequential because it directly supports the paper's central goal of consolidating complementary domain capabilities into one deployable student, with no teacher required at inference.

2.Useful evidence beyond a single main-table comparison. The gains extend to larger teachers and a parameter-merged initialization. In addition, the controlled batching experiment shows a large stability difference when only domain grouping within an update is changed: domain-pure training reaches 91.7/56.7 on MATH-500/GYM-200 after 62 steps, versus 43.2/12.8 for interleaved training. This provides a practically useful finding about organizing representation-level multi-teacher supervision.

Critical weaknesses

1.The complete recipe is supported more clearly than its individual design choices. Crossfade simultaneously raises the token-loss weight and lowers the representation-loss weight, so its additional benefit cannot currently be attributed to either change or their interaction. Likewise, the last-1/last-3 comparisons do not distinguish the effect of supervising more layers from the effect of selecting those particular depths. This matters because the paper presents layer selection and supervision scheduling as central elements of the method, but their specific rationale is not yet fully isolated. 2.The incremental value of multiple teachers is not directly established in the larger-teacher setting. AceReason is the strongest teacher on five of the six Table 2 benchmarks, yet there is no matched result using AceReason alone with the same joint-supervision recipe. Therefore, the experiments establish that the proposed recipe improves over the tested multi-teacher baselines more directly than they establish that complementary supervision from multiple teachers is necessary. This limits the interpretation of the paper's central multi-teacher claim.

Questions for authors

1.How were the supervised layers selected? Can the authors clarify whether the last-1/last-3 gains reflect the number of supervised layers or the suitability of those particular depths? 2.Given that constant-weight joint supervision already helps, what explains the additional crossfade gain: gradually removing the representation constraint, increasing token supervision, or their interaction? Can the current results distinguish these possibilities? 3.In Table 2, what additional benefit do the other two 7B teachers provide beyond AceReason? Is a result using AceReason alone under the same prompts, budget, and joint-supervision setup available? 4.During domain rotation, did improvements in one domain coincide with temporary degradation in another? The math and logic curves show overall gains; observations of math, code, and logic at matched checkpoints would help clarify cross-domain interference.

Reviewer_YAVLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Latent-MOPD, a multi-teacher on-policy distillation framework that consolidates multiple domain-specialized RL teachers (math, code, logic) into a single 1.5B student (DeepSeek-R1-Distill-Qwen-1.5B) without retaining teacher models at inference time (Section 1–3). For each domain-labeled prompt, the student generates on-policy rollouts and queries the routed domain teacher πTd\pi _ {T _ d} for both token-distribution supervision over the student's top-kk (k=16k=16) support (Section 3 Eq. (1), Appendix C Eq. (6)–(7)) and ℓ2\ell _ 2-normalized hidden-state alignment on selected late layers (Section 3 Eq. (2), Appendix A Eq. (4)), using a shared bias-free linear or 2-layer MLP projector gψ:RdS→RdTg _ \psi : \mathbb{R}^{d _ S} \to \mathbb{R}^{d _ T} when widths differ. Training is organized into domain-pure updates (rotating math →\to code →\to logic over 62 steps without shuffling) and governed by a per-teacher crossfade schedule (Section 3 Eq. (3)) that linearly ramps the token weight α(u)=min⁡(1,u/Tw)\alpha(u) = \min(1, u/T _ w) from 00 to 11 while decaying the representation weight β(u)=max⁡(0,1−u/Tw)\beta(u) = \max(0, 1 - u/T _ w) from 11 to 00. Experiments evaluate same-lineage 1.5B teachers across 9 benchmarks (Section 4.2 Table 1, normalized headroom score Norm=1.05\mathrm{Norm} = 1.05 vs. 0.900.90 for MOPD-style), 7B Qwen-based teachers across 6 benchmarks (Section 4.3 Table 2, Norm=0.26\mathrm{Norm} = 0.26–0.270.27 vs. 0.160.16), and parameter-merged initialization across 6 benchmarks (Section 4.4 Table 3, Norm=1.03\mathrm{Norm} = 1.03 vs. 0.970.97), supplemented by CKA diagnostics (Section 5 Figure 3a–b, Appendix E Eq. (9)), batch-organization stability checks (Figure 3c, Appendix F Figure 10), and teacher-matching margin analyses (Figure 6, Appendix G Figure 11).

Critical strengths
  1. Practical, inference-free multi-specialist consolidation with thorough implementation documentation. Latent-MOPD tackles a realistic post-training scenario—merging math, code, and logic RL specialists into a single 1.5B student without multi-teacher inference overhead—and provides clear specifications of the routing, late-layer alignment, ridge-initialized linear projection, and evaluation protocol in Appendices A–C, backed by an anonymous code repository.

  2. Informative stability and representation diagnostics. The comparison between domain-pure and interleaved updates (Section 5 Figure 3c, Appendix F Figure 10), the analysis of how top-1% massive activations affect layer-wise CKA profiles (Appendix E Figure 9), and the readout-depth teacher-matching margin analysis (Section 5 Figure 6, Appendix G.1 Figure 11) offer useful empirical insights into multi-teacher representation alignment.

Critical weaknesses
  1. Causal attribution of latent supervision is confounded by the coupled crossfade schedule (Eq. (3)) and mismatched λ\lambda / position settings. In Table 1, constant-weight joint training (Latent-MOPD (constant), Norm=0.94\mathrm{Norm} = 0.94) barely improves over MOPD-style (token-only) (Norm=0.90\mathrm{Norm} = 0.90) and degrades BBH, MuSR, and MBPP+. Nearly the entire reported advantage of Latent-MOPD (last 3) (Norm=1.05\mathrm{Norm} = 1.05) comes from switching to the coupled crossfade in Eq. (3), which simultaneously warms up the token loss from 00 to 11 and decays the representation loss from 11 to 00, leaving pure token distillation for the second half of training. Because the paper omits a token-ramp-only baseline (β≡0\beta \equiv 0 with the same α(u)\alpha(u) ramp) and changes λ\lambda from 11 to 20002000 (plus truncating to 2,000 tokens) for all Rep-only and OPRD-style rows (Appendix A lines 870–880), the effect of representation supervision is not cleanly isolated. Furthermore, the token objective itself (Appendix C Eq. (6)–(7)) truncates to the student's top-1616 support without the teacher-selected support or truncation correction of published MOPD (Ma et al., 2026), potentially weakening the token-only baseline.

  2. Limited generality across model families/scales, lack of teacher complementarity in Table 2, benchmark omissions in Tables 2–3, and absence of multi-seed uncertainty.

    • Single student scale and homogeneous Qwen lineage: Only one student (DeepSeek-R1-Distill-Qwen-1.5B) is tested. In the "cross-family" 7B setting (Section 4.3 Table 2), all three teachers share the Qwen2.5-7B 28-layer architecture and tokenizer, and the Math (Skywork-OR1-Math-7B) and Code (AceReason-Nemotron-7B) teachers are RL-tuned descendants of the "Logic teacher" (DeepSeek-R1-Distill-Qwen-7B). Moreover, AceReason-Nemotron-7B alone achieves the highest teacher score on 5 of the 6 benchmarks in Table 2 (GYM 47.6, BBH 74.3, MBPP 83.7, MBPP+ 70.2, AIME24 64.4), yet no single-teacher AceReason-Nemotron-7B distillation baseline is included.
    • Omitted benchmarks and mixed Table 3 performance: Tables 2 and 3 drop MuSR, LCB-e, and Minerva from the 9-benchmark suite of Table 1 without explanation. In Table 3 (parameter-merge initialization), +Latent-MOPD achieves the highest non-teacher score on only 2 of the 6 benchmarks (BBH and MBPP), trailing +MOPD-style (token-only) on GYM (52.2 vs. 52.3) and AIME25 (36.7 vs. 37.3), trailing +Rep-only (last 3) on AIME24 (53.8 vs. 54.6), and falling below the untrained Parameter merge (start) on MBPP+ (54.9 vs. 55.4).
    • No training-seed variance and 11.1% unshuffled data consumption: All tables report single training runs that consume only 1,984 of the 17,856 prompts (11.1%) without shuffling (Section 4.1, Appendix A line 733), leaving small gains such as +0.6 on GYM and +0.9 on BBH unverified against training and data-ordering variance.
Questions for authors
  1. Decoupling the crossfade schedule (Eq. (3)) and matching λ\lambda across baselines:

    • What is the performance of (a) a token-ramp-only baseline that applies the exact per-teacher ramp α(u)=min⁡(1,u/Tw)\alpha(u) = \min(1, u/T _ w) from Eq. (3) while setting β(u)≡0\beta(u) \equiv 0, and (b) a constant-token + rep-fade variant with α(u)≡1\alpha(u) \equiv 1 and β(u)=max⁡(0,1−u/Tw)\beta(u) = \max(0, 1 - u/T _ w)?
    • Why were Rep-only, OPRD-style, and Latent-MOPD (OPRD-style) in Tables 1 and 3 run with λ=2000\lambda = 2000 on the last 2,000 tokens while default Latent-MOPD used λ=1\lambda = 1 on all tokens (Appendix A, pp. 15–17)? Does interleaved training still collapse if tested on the default λ=1\lambda = 1 last-3-layer Latent-MOPD configuration rather than λ=2000\lambda = 2000 all-layer OPRD-style?
  2. Single-best-teacher baseline in Table 2 and omitted benchmarks (MuSR, LCB-e, Minerva):

    • Since AceReason-Nemotron-7B outperforms both Skywork-OR1-Math-7B and DeepSeek-R1-Distill-Qwen-7B on 5 of the 6 benchmarks in Table 2 (including both logic benchmarks GYM and BBH and AIME24), how does Latent-MOPD compare against distilling solely from AceReason-Nemotron-7B across all three prompt domains?
    • Why were MuSR, LCB-e, and Minerva omitted from Tables 2 and 3, and can you report those scores?
  3. Multi-seed uncertainty, published MOPD estimator, and projector synchronization:

    • Can you report means and standard deviations across at least 3 training seeds (with prompt-pool shuffling) for the main rows of Table 1 (MOPD-style, Latent-MOPD (constant), Latent-MOPD (last 3))?
    • In the 7B setting, are the shared projector parameters gψg _ \psi synchronized across all 8 distributed workers on every step, and how does the 2-layer MLP projector perform when initialized with the same 42-rollout pre-fit used for the linear projector?
Reviewer_mYjdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Latent-MOPD, which distills several domain specialists (math, code and logic) into one student. Each prompt is routed to its domain's teacher, and the student learns from both that teacher's token distribution and its last layer hidden states. The hidden state loss fades out over training while the token loss ramps in, and each update uses prompts from a single domain. With three 1.5B teachers from the same family, the 1.5B student beats token only, representation only and averaging baselines on all nine benchmarks and beats the best teacher on five. With larger 7B teachers it also beats both single channel baselines.

Critical strengths

S1) Using Latent-MOPD the authors produce a single 1.5B student that outperforms the strongest of three 1.5B specialists without adding inference cost. This lets practitioners replace several specialist models with one at a small training cost.

Critical weaknesses

W1) It is unclear that hidden state supervision helps. Most of the gain comes from the crossfade, but the crossfade also ramps up the token loss, and no baseline uses that ramp alone, so the paper does not show the gain is due to the hidden states.

W2) The claim that hidden states transfer how the teachers compute is not well supported. The drop in representation loss (Figure 5c) has no token only comparison and since the same family teachers start from the student's weights, token distillation alone may also align the last layer states.

Questions for authors

Q1) Could you run MOPD style with the same token weight ramp as Latent-MOPD but with the representation term removed, and report the results on the same benchmarks (Table 1). This would help address W1/W2.

Q2) How does the last three layer representation loss against the routed teacher change during MOPD-style training compared to Latent-MOPD?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.