acceptodds
Under review as a conference paper at ICLR 2027

Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization

Abstract

Open-weight language models from different families exhibit complementary capabilities, motivating their consolidation into a compact student through on-policy distillation (OPD). However, full-vocabulary OPD typically assumes a shared tokenizer, while existing cross-tokenizer methods may discard teacher probability mass or assign it to student tokens with unrelated content. We introduce Byte-Prefix Marginalization (BPM), which re-expresses the teacher's next-token distribution over the student vocabulary in a shared byte space. Specifically, BPM assigns each teacher token's probability to the longest student token whose byte representation is a prefix of the teacher token's bytes, aggregates mass mapped to the same student token, and places otherwise unmatched mass in an explicit residual category. This produces a vocabulary-complete, byte-aligned, and mass-preserving target for dense OPD. The target exactly recovers the teacher-induced byte-prefix marginal when the relevant prefix does not span multiple teacher tokens (a condition satisfied at more than 99% of training positions) and uses a mass-preserving, chain-factorized lower bound otherwise. Across Qwen3-32B, GLM-Z1-9B-0414, and MiniMax-M2.7 as teachers, BPM consistently outperforms current cross-tokenizer methods on six mathematics and programming benchmarks, improving six-benchmark avg@8 by 3.7-6.6 points over the strongest baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, cross-tokenizer distillation, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pZc8OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BPM which converts teacher next token distribution into student distribution through a shared byte space and applied to on policy distillation. BPM assigns teacher probability to the longest student token byte prefix and aggregate them. Individual experiments distill three teachers into qwen 2b using math and coding training data. Results across six benchmarks has shown improvements over the evaluated baselines.

Critical strengths
  1. The paper has evaluated BPM using three model pairs and across 6 math and coding benchmarks. The reported improvement over the other baseline has provided evidence that this method is helpful.
  2. The observations made on whitespace code generation collapse are interesting
Critical weaknesses
  1. A major weakness of the paper is that it overlooks the most directly related work in references [1, 2, 3] provided below --- none of which have been cited or compared to. For example, lemma 1 in [1] expresses a byte prefix probability as the sum of probabilities over its cover set. Proposition 1 in [2] gives a cover expression for character prefix probability. The paper’s Eq 4 groups teacher tokens by their longest matching student token prefix. There appear direct correspondences between Eq 4 at the beginning of generation and results in [1] Furthermore, reference [3] below formulates this conversion in section 4 and uses it for distillation. The lemma 1 extends the marginalization to merge compatible sub vocabulary and in appendix D it uses this token prob with a residual mass that is the same structure of BPM’s forward KL objective. Given that the paper makes. no mention of such directly related prior work, the reviewer believes that the novelty in the paper is not sufficient to recommend acceptance.

  2. The exactness claim is not entirely justified. Eq 4 sums the exact teachers probability through a routing map but this is not necessarily the distribution by retokenizing the teacher generated continuation. For example, a teacher generates [a,b] with 1/2 prob and [c] with 1/2 prob so its first token distribution is ½ for [a] and ½ for [c]. The student has vocabulary of a, b, c, ab so the strings from the teacher are encoded as ab and c in student space. Therefore the teacher induced probability is p(ab) = ½ and p(c) = ½ and p(a) = 0. But the routing uses [a] itself since [ab] doesn’t fit in [a] ie phi(a) = a. So it will produce p(a) = ½, p(c) = ½ and p(ab) = 0 which is different. Spanning does not fix this because the student may roll out [c] in this case which is a non spanning position. But the probability for ab is still wrong. So the observed token non spanning does not mean the distribution is correct. [3] has discussed something like this in section 4.3 which accounts for sequences that merge a short token to a larger token. My concern is that the Eq 4 does not establish the exact teacher induced probability under the student by itself.

[1] Phan et al. Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles. 2025. [2] Vieira et al. From Language Models over Tokens to Language Models over Characters. 2025. [3] Phan, Khisti, and Ullrich. Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillation. 2026.

Questions for authors

Question

  1. What does BPM add on top of the existing probabilistic conversion works? Please explain the differences and the contribution related to the prior work other than applying to on policy distillation.
  2. Please clarify and verify the exactness claim.
Reviewer_YdiAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Byte-Prefix Marginalization (BPM) to address the cross-tokenizer discrepancy in On-Policy Distillation (OPD) by aligning logits at a shared byte level.

Contributions The primary contribution is a deterministic prefix-mapping matrix derived from a longest-prefix matching heuristic to project the teacher's probability distribution onto the student's vocabulary space. To prevent system collapse during alignment, the authors introduce secondary engineering patches, including a spanning-token multiplier for rare boundary errors and a hard-coded whitespace-row mask to block alignment over specific indentation patterns.

Validation The authors validate BPM across multiple open-source model pairs on standard downstream benchmarks, reporting improvements over early cross-tokenizer baselines like ULD and GOLD.

Critical strengths

The paper presents an exceptionally insightful analysis of the "whitespace collapse" phenomenon in Section 4.3, exposing how naive byte-level matching can catastrophically degrade code indentation. Additionally, compared to later derivative works, the core mapping algorithm of BPM is relatively simple, computationally straightforward, and accompanied by transparent reports of runtime latency and matching complexity.

Critical weaknesses
  1. Unjustified Coupling with On-Policy Distillation (OPD) While the paper is heavily framed around "On-Policy Distillation" in its title and introduction, the core contribution—Byte-Prefix Marginalization (BPM)—is fundamentally a static, deterministic vocabulary probability-mapping algorithm. The entire alignment mapping process relies on pre-defined prefix hierarchies and is decoupled from the dynamic exploration or policy-update mechanics of OPD. Coupling this general-purpose alignment method with the OPD paradigm introduces unnecessary engineering friction and runtime latency during online rollout, without a strong theoretical justification for why such online coupling is methodologically essential.

  2. Omission of Closely Related Baselines and Unbalanced Content Prioritization Although the paper explicitly cites span-level alignment methods such as ALM in the related work section, these critical baselines are conspicuously missing from the primary comparative experiments in Table 1. Given that span alignment represents a prominent and highly competitive alternative to byte-level matching, this selective baseline comparison weakens the strength of the empirical claims. Furthermore, the main text allocates substantial space to minor secondary results—such as standard divergence ablations in Table 2—which would be more appropriately placed in the Appendix. Prioritizing these marginal ablations over standard SOTA comparisons (like ALM) affects the overall balance and focus of the evaluation.

  3. Reliance on Brittle Heuristics and Lack of Ablation for Core Corrections The proposed byte-level matching heavily relies on hand-crafted heuristics and ad-hoc engineering "patches" to maintain system stability. The "spanning-token correction" introduced in Section 3.3 is a complex multiplier formulated to patch a rare edge case (<1%) where the core longest-prefix match fails, yet the authors do not provide any empirical ablation study to demonstrate this patch's actual impact on final performance.

Questions for authors

In Section 3.3, the paper introduces a spanning-token correction (Eq. 6) utilizing the chain rule to multiply sequential conditional probabilities (e.g., P(belie) × P(vable ∣ belie)) when a student token spans multiple teacher tokens. Since this multi-step probability multiplication shares a very similar conceptual formulation with span-level alignment methods like ALM, could the authors please clarify the key algorithmic distinctions between the spanning-token correction and standard ALM in these scenarios?

Reviewer_SXBCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BPM addresses on-policy distillation when the teacher and student use different tokenizers. It maps teacher-token probabilities to student tokens through their byte prefixes, with a residual category for unmatched mass and corrections for tokens that cross teacher boundaries. The recipe also handles stop tokens and masks whitespace-only positions. The authors distill three teachers separately into Qwen3.5-2B and evaluate on six math and code benchmarks. Forward KL gives the best results, improving average accuracy over the strongest baseline by 3.7, 6.6, and 5.9 points across the three pairs. The paper also studies why byte-level supervision can cause indentation errors in generated code.

Critical strengths
  1. The basic construction is simple and useful. The token mapping can be precomputed, and retaining a residual category makes it clear where the teacher's probability mass goes. The gains in Table 1 hold across all three teacher pairs in aggregate, which is encouraging, even though BPM does not win on every benchmark. The appendix provides useful implementation details, especially for alignment and stopping.

  2. The whitespace ablation is a strong part of the paper. Masking about 3.5% of aligned code rows prevents the indentation collapse on the GLM pair, while masking the same number of random rows does not (Section 4.3). Together with the code examples and indentation diagnostics, this makes a convincing case that the problem comes from the particular supervision at whitespace positions. This finding should be useful beyond the proposed method.

Critical weaknesses
  1. My main concern is what the exactness claim actually covers. Equation (2) tokenizes a single teacher token's finite byte string. Under greedy matching, Equation (4) computes that distribution exactly. But this is not necessarily the distribution of the first student token in the teacher's continuing output. For example, suppose the student vocabulary contains a, b, ab, and c. The teacher emits a or c with probability 1/2 each, and always follows a with b. The scatter assigns 1/2 to a, whereas greedy tokenization of the continuing stream assigns that mass to ab. An observed c trajectory still counts as a 1:1 case. Thus, counting realized 1:1 and 1:N positions in Table D.2 does not establish that the full target distribution is exact at more than 99% of positions. This is not a contradiction of the finite-string definition in Equation (2), but it is an important gap between that definition and the broader claim. Appendix C also acknowledges the difference between greedy matching and actual BPE; the reported cross-checks need quantitative results to show how much this matters.

  2. The experimental gains are promising, but checkpoint selection makes them harder to interpret. Section 4.1 chooses each method's best checkpoint using the same six benchmarks on which results are reported. BPM's headline result also selects among three divergences. Using this procedure for every method does not eliminate selection bias, especially when some methods collapse during training. The bootstrap analysis is helpful, and Appendix H.4 clearly states its limits: one training run per method, with no uncertainty from training or checkpoint selection. Resampling generations on the same problems does not resolve those limits. Also, the P2 interval is against ULD rather than the stronger SeqKD baseline, and the P3 JSD-versus-ULD interval includes zero. Finally, ALM is described as the closest prior objective but is absent from the comparisons. These points leave me leaning toward weak rejection. A clearer exactness claim and an evaluation with independently selected checkpoints would substantially strengthen the paper.

Questions for authors
  1. Is the intended exact target defined over one finite teacher token or over the teacher's continuing byte stream? Please clarify how Proposition 1 and the >99% claim apply to the a/b/ab example above. A small exact-enumeration experiment, along with a quantitative check against actual BPE tokenization, would help resolve this concern.

  2. Do the gains hold when checkpoints and divergence are chosen on separate validation data, or with a fixed rule chosen before evaluation? Results from multiple training runs on at least one pair, compared with its strongest baseline, would make the size of the improvement easier to assess.

  3. Why is Approximate Likelihood Matching (Minixhofer et al., NeurIPS 2025) [1] not included? A comparable-budget result, or a concrete explanation of why that comparison is not feasible, would help. Please also clarify which stopping and stabilization choices are shared across methods and whether modest validation-based tuning helps the baselines that collapse.

[1] ALM: https://papers.neurips.cc/paper_files/paper/2025/hash/720f9f5dc751eb56952ae4fee2398f73-Abstract-Conference.html

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.