- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes BPM which converts teacher next token distribution into student distribution through a shared byte space and applied to on policy distillation. BPM assigns teacher probability to the longest student token byte prefix and aggregate them. Individual experiments distill three teachers into qwen 2b using math and coding training data. Results across six benchmarks has shown improvements over the evaluated baselines.
Critical strengths
- The paper has evaluated BPM using three model pairs and across 6 math and coding benchmarks. The reported improvement over the other baseline has provided evidence that this method is helpful.
- The observations made on whitespace code generation collapse are interesting
Critical weaknesses
-
A major weakness of the paper is that it overlooks the most directly related work in references [1, 2, 3] provided below --- none of which have been cited or compared to. For example, lemma 1 in [1] expresses a byte prefix probability as the sum of probabilities over its cover set. Proposition 1 in [2] gives a cover expression for character prefix probability. The paper’s Eq 4 groups teacher tokens by their longest matching student token prefix. There appear direct correspondences between Eq 4 at the beginning of generation and results in [1] Furthermore, reference [3] below formulates this conversion in section 4 and uses it for distillation. The lemma 1 extends the marginalization to merge compatible sub vocabulary and in appendix D it uses this token prob with a residual mass that is the same structure of BPM’s forward KL objective. Given that the paper makes. no mention of such directly related prior work, the reviewer believes that the novelty in the paper is not sufficient to recommend acceptance.
-
The exactness claim is not entirely justified. Eq 4 sums the exact teachers probability through a routing map but this is not necessarily the distribution by retokenizing the teacher generated continuation. For example, a teacher generates [a,b] with 1/2 prob and [c] with 1/2 prob so its first token distribution is ½ for [a] and ½ for [c]. The student has vocabulary of a, b, c, ab so the strings from the teacher are encoded as ab and c in student space. Therefore the teacher induced probability is p(ab) = ½ and p(c) = ½ and p(a) = 0. But the routing uses [a] itself since [ab] doesn’t fit in [a] ie phi(a) = a. So it will produce p(a) = ½, p(c) = ½ and p(ab) = 0 which is different. Spanning does not fix this because the student may roll out [c] in this case which is a non spanning position. But the probability for ab is still wrong. So the observed token non spanning does not mean the distribution is correct. [3] has discussed something like this in section 4.3 which accounts for sequences that merge a short token to a larger token. My concern is that the Eq 4 does not establish the exact teacher induced probability under the student by itself.
[1] Phan et al. Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles. 2025. [2] Vieira et al. From Language Models over Tokens to Language Models over Characters. 2025. [3] Phan, Khisti, and Ullrich. Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillation. 2026.
Questions for authors
Question
- What does BPM add on top of the existing probabilistic conversion works? Please explain the differences and the contribution related to the prior work other than applying to on policy distillation.
- Please clarify and verify the exactness claim.