acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability

Abstract

On-Policy Distillation (OPD) trains a student on its own generations using teacher feedback. With different tokenizers, comparing teacher and student predictions requires alignment at both sequence and vocabulary levels. In this paper, we examine whether expanding this alignment coverage improves learning. Across three heterogeneous teacher-student pairs on mathematical reasoning and code generation, strict groups already cover most student-generated tokens despite substantial vocabulary mismatch. On responses sampled from the students before distillation, the shared vocabulary retains nearly all teacher and student probability mass at strictly aligned positions on average. Restricting reverse KL to a student-selected top-16 subset of the shared vocabulary at each strict position achieves accuracy comparable to full shared-vocabulary OPD, outperforming the evaluated cross-tokenizer baselines. Adding mean squared error supervision on span log-probabilities in mismatch groups gives complete supervision coverage, yet reduces accuracy. At checkpoints from training with only the strict loss, the span gradients show weak or negative directional agreement with the strict gradients and grow in magnitude relative to them. These diagnostics may help explain the accuracy drop from adding span supervision. Our findings motivate a shift from maximizing alignment coverage to prioritizing supervision reliability: compact supervision at strict positions can be more effective than broader coverage that introduces weakly aligned or conflicting training signals.

Primary area:
generative models
Keywords:
Large Language Models, On-policy Distillation, Cross-tokenizer Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MCwzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies cross-tokenizer on-policy distillation and asks whether recovering supervision on spans without a 1:1 token alignment actually helps. Across three heterogeneous teacher-student pairs on math and code, strict 1:1 positions cover most student tokens despite low static vocabulary overlap. At these positions, a student-selected top-16 subset of the shared vocabulary already holds most of both models' probability mass. Reverse KL on strict positions beats the evaluated cross-tokenizer baselines (also with a larger teacher), while adding a span MSE on mismatch groups lowers accuracy.

Critical strengths

It establishes useful strong-baseline results: plain reverse KL on strict positions over the shared vocabulary cut down to a student top-16 subset keeps at least 96% of the full-vocabulary gain. The larger-teacher results point the same way. This gives a much simpler default and raises the bar for new cross-tokenizer OPD methods.

Critical weaknesses
  • The coverage-vs-reliability conclusion rests on one mismatch objective. In gradient terms, log-prob MSE on sampled span paths is a single-sample estimate of span-level reverse KL, while the strict term is an exact KL over the shared vocabulary. The drop may therefore come from mixing in a noisy estimator rather than from unreliable mismatch targets. The gradient diagnostics do not separate the two: they are taken on strict-only checkpoints, where a rising span-to-strict ratio is not surprising since only the strict loss is being minimized, and the split-half control uses the same exact estimator on both halves. Because this is the paper’s main message, the evidence currently supports only a narrower claim: this span loss does not help on top of strict OPD.
Questions for authors

Does the same sampled-path MSE also hurt when applied at strict positions in place of the exact KL? Or does a lower-variance mismatch objective (e.g., SimCT-style units) help on top of the same strict loss? If the first hurts or the second helps, the problem is the estimator rather than coverage.

Clarification: is the rising norm ratio mainly the strict gradient shrinking? Absolute norms, and the same diagnostics on a lambda>0 run, would make Section 5 much more convincing; and were the baselines tuned at all, and why does SimCT fall below shared-vocabulary OPD here when its own paper reports the opposite?

Reviewer_r9uvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper empirically re-examines alignment coverage in cross-tokenizer on-policy distillation (OPD), studying three heterogeneous teacher→student pairs (Qwen2.5-7B-Instruct→Llama-3.2-3B-Instruct, Granite-4.1-8B→Phi-4-mini-instruct, Granite-4.1-8B→Qwen2.5-7B-Base) on mathematical reasoning and code generation. The main findings are: (i) strict 1:1 token groups already cover 85.57–96.98% of student tokens during training, despite static vocabulary Jaccard overlap of only 39.49–64.87%; (ii) adding span log-probability MSE supervision on mismatch groups achieves complete supervision coverage, yet all 18 tested positive-weight settings lower full-average accuracy by 0.27–1.20 points relative to strict-only supervision; (iii) on pre-distillation rollouts the shared vocabulary retains nearly all predictive mass at strict positions, and a student-selected top-16 subset retains at least 94.55% (student) and 93.54% (teacher), with training on this subset preserving at least 96% of the full improvement and achieving higher Math and Full averages than ULD, Extended ULD, GOLD, and SimCT; (iv) gradient diagnostics show weak directional agreement between span and strict gradients, with the span-to-strict norm ratio growing during training. The authors conclude that supervision reliability matters more than alignment coverage.

Critical strengths
  1. The combination of dynamic coverage tracking over full training runs, probability-mass probes, and top-k training comparisons gives concrete, quantified evidence that static vocabulary overlap overstates the supervision gap. The resulting recipe — strict reverse KL on a small student-selected shared subset — is simple, practical, and reproducible (code and careful protocols provided, three model pairs plus a 235B-teacher and ALFWorld validation).
  2. Holding the strict objective fixed while sweeping the span-loss weight across 7 values × 3 pairs cleanly isolates the marginal value of mismatch supervision, and the uniform 18/18 directional result is a useful cautionary data point for the growing literature on recovering excluded supervision.
Critical weaknesses
  1. The negative result is established for exactly one mismatch objective (span log-probability MSE), without a scale-controlled control. Since GOLD and SimCT recover mismatch supervision with different objectives and still improve over the undistilled student on the Full average, the experiments do not yet distinguish "coverage is not the bottleneck" from "this loss is the bottleneck" — two readings that lead to rather different research directions.
  2. The improvements over strong baselines (0.51–1.05 full-average points) and the λ-sweep differences (0.27–1.20 points) rest on single-seed runs evaluated at one checkpoint; the compact-variant comparisons are tighter still (top-128 slightly exceeds Strict full on two of three pairs). Confidence intervals or a second seed would substantially strengthen the conclusions.
Questions for authors
  1. Have you evaluated any mismatch-group objective other than span log-probability MSE — e.g., MSE on span probabilities, KL between span-level distributions, or gradient-norm-matched weighting? This seems the most direct way to test whether the negative result reflects coverage itself or the choice of loss.
  2. Could you report variability for the key comparisons (e.g., bootstrap CIs over problems for Table 2, or a second seed for top-16 vs. the strongest baseline and for λ ∈ {0, 1})? Even a rough estimate would help calibrate the margins.
  3. In Table 10, ULD and Extended ULD score 20.87 and 32.00 on Code, well below the undistilled Base (34.51; Tables 2 and 7). Were the baselines validated against their published setups? Clarifying this would help interpret the comparison with SimCT and GOLD.
Reviewer_LeULOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits cross-tokenizer on-policy distillation (OPD), arguing that supervision reliability matters more than alignment coverage. The authors show that strict 1:1 alignment covers most student-generated tokens despite limited vocabulary overlap. They further demonstrate that strict supervision with a student-selected top-16 vocabulary subset achieves competitive performance, while adding span-level MSE supervision on mismatched token groups degrades performance. Gradient analyses are provided to explain the negative effects of mismatch supervision.

Critical strengths
  • Interesting empirical observation: The finding that strict alignment covers 85–97% of student-generated tokens despite substantially lower vocabulary overlap is insightful and challenges the motivation for more complex alignment methods.
  • Consistent empirical results: The experiments across three heterogeneous teacher–student pairs show that relatively simple strict alignment can outperform more elaborate cross-tokenizer distillation baselines.
Critical weaknesses

My understanding is that the paper's main contributions are:

  1. showing that strict alignment coverage is substantially higher in practice than static vocabulary overlap suggests,
  2. empirically demonstrating that strict supervision alone is sufficient to achieve competitive performance, and
  3. showing that expanding supervision to mismatch positions can actually hurt performance.

I find (1) and (2) to be interesting and convincing empirical findings. However, I am not fully convinced that these observations alone provide sufficient novelty for a conference such as ICLR, especially given that top-k supervision has already been explored in several prior OPD literatures [ref1, ref2].

I am less convinced by (3). A tokenizer mismatch does not necessarily imply any semantic mismatch; it simply means that the same text is segmented differently. Intuitively, if the mismatch positions could be properly aligned and supervised, it is not clear why including them should necessarily degrade performance, even if they provide limited additional benefit. More importantly, the experiments change both the alignment setting and the training objective simultaneously. Strict positions are supervised using reverse KL, whereas mismatch positions are supervised using span-level log-probability MSE. Therefore, the observed gradient disagreement could arise from either:

  • The supervision at mismatch positions being inherently unreliable.
  • The use of different objectives (KL vs. MSE) producing conflicting gradients.

The current experiments do not disentangle these possibilities. Additional controlled experiments, such as applying a comparable MSE objective to strict positions or using alternative alignment-based objectives at mismatch positions, would help clarify this issue. Consequently, the results demonstrate that adding span-level MSE supervision at mismatch positions degrades performance, but do not establish that mismatch supervision itself is harmful or unreliable. I therefore find the evidence supporting the paper's third contribution insufficient.

 

[ref1] Fu, Yuqian, et al. "Revisiting on-policy distillation: Empirical failure modes and simple fixes." COLM 2026.

[ref2] Li, Yaxuan, et al. "Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe." arXiv preprint arXiv:2604.13016 (2026).

Questions for authors

How do the authors justify the broader conclusion that mismatch supervision is harmful, rather than simply showing that the particular span-MSE objective is ineffective? Could additional controlled experiments disentangle the effects of tokenizer mismatch and objective choice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.