acceptodds
Under review as a conference paper at ICLR 2027

When Does Finer Teacher Routing Help? Per-Token Capability Routing in On-Policy Distillation

Abstract

On-policy distillation from several domain specialists raises a choice a fixed corpus never poses: which teacher supervises each token of the student's own rollout. The standard recipe routes a whole rollout to one teacher by its domain tag, yet when a rollout interleaves capabilities, that one label is wrong on part of the sequence. We propose (Capability-Routed On-Policy Distillation), which assigns a teacher to every token by vocabulary identity: offline, each token id goes to the specialist that most improves its next-token distribution over a shared initialization, and nothing else changes. With granularity isolated, the benefit is conditional. On data that mixes mathematics and code within each sample, per-token routing beats each single-teacher baseline on its weak axis, by points on code and on mathematics. The code gain rises monotonically with within-sample mixing, reverses to a -point loss on single-domain data, and is not detected in a B dense student or a B mixture-of-experts with B active parameters. Per-token routing can help only if the two single-teacher baselines split, a necessary condition that can be checked from runs a practitioner trains anyway, before the per-token router is built.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, large language models, multi-teacher distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bK3AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies teacher assignment in multi-teacher on-policy distillation. The standard recipe sends each student rollout to one domain teacher; Capability-OPD instead builds an offline vocabulary-id table and distills each generated token against the specialist with the larger gold-token log-probability lift over a shared initialization. Teachers grow from the same π0; routing does not look at context. The main runs concatenate a math segment and a code segment in one sample. On Qwen2.5-7B, Table 10 reports PER-TOKEN vs MATH-ONLY +1.67 on the three code sets and +0.43 n.s. on the three math sets; vs CODE-ONLY +1.49 on math and −1.01 on code. The code gap vs MATH-ONLY flips to −1.42 on single-domain data. The same comparison is not detected on Qwen3-4B or a 30B-A3B MoE. The seven-benchmark mean is not significant. Training pays one extra frozen-teacher forward per rollout relative to per-sample routing.

Critical strengths

1.The paper isolates “who to hear at each step” as a granularity variable. Multi-teacher OPD really does have to choose a teacher at each prefix. The authors freeze the teacher pair, data, optimizer, and step count, keep reverse KL, and only replace the sample-level teacher with a vocabulary-id lookup. This is consequential because granularity can be asked about without also changing the teachers or the loss.

2.Finer assignment yields a checkable multi-domain compromise, not a concatenation of two specialists’ best scores. A sample-level teacher hurts the other domain; a finer assignment recovers some of that drop, but still trails the matching specialist on that specialist’s own domain. The code-source swap, multiple seeds, and the mixing-fraction sweep keep the same direction. This is consequential because, for a unified student, the right comparison is the multi-domain compromise rather than stitching together two experts’ peak scores. That comparison still does not identify the table as capability detection.

Critical weaknesses

1.The problem setting and usable range are narrower than the introduction; concatenation shows a controlled domain switch, not natural multi-capability reasoning. Figure 1 shows a domain-B token inside one solution; §3.1 stitches a math item to a code item. Concatenation can be a valid controlled experiment. It shows that a sample-level tag cannot choose a possibly better teacher for each segment. Shared initialization and a same-family student define the present operating range. The seven-benchmark mean vs MATH-ONLY is +0.68 with a CI through 0; every rollout also requires a forward pass from both frozen teachers. This is consequential because the headline 7B result is then a limited empirical edge under constructed switches and extra teacher compute, not evidence that the method identifies capabilities during natural generation. Table 3’s 7B / 4B / MoE comparison also changes model generation and experimental setup together.

2.Routing is a context-independent vocabulary lookup; the present experiments do not show that this lookup identifies the capability a token needs, and they do not rule out simpler assignment. The student writes a vocabulary id and looks up a preassigned teacher, without reading the current prefix. Token identity can serve as a proxy for capability ownership; that proxy does not use context. RANDOM-LABEL is reported only on the seven-benchmark mean (−0.39 n.s.). Figure 4’s spans 1–32 are a plateau; majority-vote budget shifts are associated with the drop; Figure 5’s mass is scaffolding tokens. Recomputation after a teacher-pair or family change is expected for an offline table unless the manuscript claims transfer. Frequent switches do not by themselves make adjacent supervision contradictory. This is consequential because the experiment that would identify the table — matching teacher share and switching frequency, then comparing the static lookup with random token or span assignment on the math and code pools — is not in the present tables.

Questions for authors

1.On mixed reasoning that is not two items concatenated (natural-language reasoning inside contest code, symbolic spans inside a derivation), are Table 10’s pairwise gaps still there: PER-TOKEN vs MATH-ONLY on the three code sets, and PER-TOKEN vs CODE-ONLY on the three math sets? If not, can the headline be stated as a condition on concatenated mixtures, shared-root teachers, and a same-family student?

2.After matching the identity table’s teacher share and switching frequency, does the static vocabulary lookup still beat random token assignment or span assignment? Please report this on the three math sets and the three code sets, not only on the seven-benchmark mean. Is the training table at τ=0 still dominated by function words?

3.Relative to PER-SAMPLE, what is the end-to-end training cost of the extra frozen-teacher forward (wall-clock / teacher FLOPs)? Can the claimed capacity pattern be reproduced inside one model family with the other training and evaluation conditions aligned, rather than Qwen2.5-7B vs Qwen3-4B vs 30B-A3B?

Reviewer_V3xCOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

CAPABILITY-OPD assigns each vocabulary token to a specialist using an offline estimate of its log-probability improvement over a shared initialization. The assigned teacher then supervises that token during student-generated rollouts. The paper studies math/code mixtures and reports gains over each single-teacher baseline on its weaker axis at 7B, alongside aggregate null results, losses on homogeneous data, and weaker results in other student configurations.

Critical strengths
  1. The study makes useful negative findings explicit. The seven-benchmark aggregate does not significantly improve over MATH-ONLY, code performance remains below CODE-ONLY, and adding an instruction-following teacher hurts math and code. Reporting these outcomes helps establish the practical limits of this particular routing rule.

  2. The main dense-model comparisons use six seeds, and the code-source swap reproduces the directional result. The span sweep is especially informative: preserving teacher shares largely preserves performance over spans of 1–32 tokens. The split-half attribution analysis also distinguishes apparent routing headroom from selection bias.

Critical weaknesses
  1. Table 2 compares PER-TOKEN, which uses both specialists, with MATH-ONLY and CODE-ONLY. RANDOM-LABEL matches teacher shares, but its reported comparison is limited to the seven-benchmark aggregate (Section 4.1 and Appendix I). Since this aggregate can hide opposing math and code effects, the corresponding paired differences on those two axes are needed to assess the benefit of token assignment. The span sweep does not settle the comparison with a matched per-rollout mixture.

  2. Proposition 1 states the screening condition too broadly. Equal benchmark averages can hide complementary errors, so the lack of a crossover between single-teacher averages does not establish that no routing scheme can help. Appendix O appropriately calls the size boundary empirical. Since the 7B and 4B comparison also changes model family, the main-text capacity result should likewise be limited to the tested configurations.

Questions for authors
  1. Using the existing RANDOM-LABEL outputs, what are its paired differences from PER-TOKEN on the math and code axes? This would help separate the benefit of teacher mixing from that of token-specific assignment.

  2. Could Proposition 1 be stated as an empirical screening heuristic unless stronger assumptions are supplied? Please also bring the main-text capacity wording into line with the limitation already noted in Appendix O.

Reviewer_WhLFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose CAPABILITY-OPD, a multi-teacher on-policy distillation scheme that assigns a teacher to every token of the student’s rollout. The assignment comes from an offline, context-free lookup table keyed by vocabulary id. This replaces the standard practice of routing the whole rollout to one teacher by its domain tag. Specifically, the paper shows that, when routing granularity is the only changed variable, the benefit of per-token routing is conditional. It grows monotonically with the within-sample mixing fraction of synthetically concatenated math+code sequences, reverses on single-domain data (−1.42), and is not detected on a Qwen3-4B or a Qwen3-30B-A3B MoE student. The authors also propose a “screening” necessary condition to decide in advance whether finer routing can help.

Critical strengths

1. Timely and well-posed research question: How to assign multiple specialist teachers to a single student’s on-policy rollout is a practical question raised by recent multi-teacher OPD recipes, and asking “when” rather than “whether” finer routing helps is a useful framing.

2. Simple, plug-in mechanism: The routing table is static and built once per teacher pair, requiring no learned gate or auxiliary head, which makes the method easy to drop into existing per-sample MOPD pipelines.

Critical weaknesses

1. How reliable is the offline token–teacher table, and what happens when it misroutes? The reference corpus used to build the table is barely specified (source, size, domain ratio), and it is never ablated. Additionally, the table is computed off-policy on teacher-forced gold text but applied context-free to the student’s on-policy tokens. Since all K teachers already score every rollout, an online, context-aware routing would cost nothing extra, but it is neither compared against nor analyzed for misrouting.

2. Are the baselines strong and fair enough to support the claimed gain? On mixed data, MATH-ONLY and CODE-ONLY send 100% of tokens to one teacher, making “winning the weak axis” a low bar. The budget-matched RANDOM-LABEL control does not separate from per-token routing on the aggregate and is not reported per axis. Other many-to-one alternatives are also missing: teacher logit averaging, weight merging run in the same setup, a single mixed teacher at 7B, oracle segment-level routing, and the teachers’ own upper-bound scores.

3. What is the actual cost of per-token routing? Every rollout must be scored by all K teachers instead of one, all K teachers must be kept in memory, and the table must be rebuilt for every new teacher pair. The paper argues the overhead is bounded only in text, with no wall-clock, GPU-hour, memory, or throughput measurement. The “cheap” screening test also requires training two full single-teacher students.

Questions for authors

Please check my Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.