- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Cross-Tokenizer On-Policy Distillation (CTOPD), which enables OPD between models with different tokenizers. The authors do this by applying an algorithm that does a chunk level credit assignment via dual pointer based on semantic-priors. Experiments show effectiveness of the method.
Critical strengths
- Enabling OPD across different kinds of models despite having different tokenizers is an important contribution.
- Intuitive methodology. logic is not really flawed and it make sense
- The paper presents promising experimental results across multiple benchmarks, demonstrating the effectiveness and potential computational efficiency of the proposed method.
Critical weaknesses
-
In the related works section [Cross-Tokenizer Distillation], the authors stated "However, existing methods are mostly designed for offline distillation strategies and cannot effectively solve the token-level signal propagation problem in OPD." However, this remains unclear for me why prior works cannot solve the token-level signal propagation problem. And how does this mean that the paper's Dual-Pointer Chunk Alignment algorithm solves those "limitations" of prior works.
-
Although the experimental results show the effectiveness of the method, I believe more comprehensive experimental validation is needed in order to be accepted to ICLR venue.. For example, despite cross tokenizer is the main keyword and contribution, the authors only show 2 cases of teacher -> student pair. I believe that it would have been better if the paper have shown more experiments on various models and sizes and more.
I have organized my suggestions for additional analyses and experiments in the Questions section below. Please refer to that section for further details.
Questions for authors
-
I have understood that the method's semantic-prior credit assignment allocates larger advantages to tokens with lower probabilities. However, I think low probability tokens may include tokenizer specific fragments / special symbols (e.g., < and >), which are not necessarily semantically important. Could the authors clarify whether their method might disproportionately emphasize such tokens, and how this affects results?
-
Section 5.3's compute cost analysis compares CTOPD against continued SFT, which uses a different training paradigm. While this comparison shows that their method is overall more efficient than SFT, it is unclear how much of the improvement can be attributed specifically to the proposed chunk alignment and semantic-prior credit assignment, rather than to OPD itself. Maybe it would be clearer if the authors can provide a comparison with existing other OPD methods too. Currently, it is unsure whether the method's contribution itself makes the cost more efficient.
-
For Section 5.5, the paper shows the analysis in different student models, which makes it difficult to isolate the effect of tokenizer mismatch. Would it be possible to provide a more controlled comparison using the same student model across both settings, ideally with teachers of comparable capabilities?
-
While reading, I got curious whether the method will be able to show identical or similar performance when the teacher and student models have the same tokenizers. Ideally, and in theory, I think they should be identical or similar to just ordinary OPD with no cross tokenizer algorithm technique. But if there are substantial differences, then I think this should be something worth more investigating. Could the authors verify this?
4 comments
Response to Reviewer 6pbf (Part 2/2)
Question 2: The SFT comparison does not separate the efficiency gains of OPD itself from those of chunk alignment and semantic-prior credit assignment. Can comparisons with existing OPD methods establish the proposed components' contribution to compute efficiency?
In our experiments, all on-policy distillation algorithms we reimplemented in our paper utilize the same amount of computation (the number of training data samples) to ensure a fair comparison. Since CTOPD achieves better performance under fixed budget, it demonstrates higher efficiency. To quantify this efficiency difference, we examined the number of training samples required for CTOPD to match the performance of our strongest baseline GOLD. Compared to 20k samples for GOLD, CTOPD achieves this using approximately 18k training samples. The data efficiency improvement is approximately 1.1x (). Both methods utilize on-policy training, with comparable per-step execution times.
Question 4: Does CTOPD recover ordinary OPD when teacher and student use the same tokenizer? Please verify whether the two methods produce identical or similar performance and explain any substantial differences.
Yes. In same-tokenizer setting, it can be mathematically proved that the alignment results of DPCA is a one-to-one mapping between tokens. Thus, Eq. 9 reduces to the ordinary sampled-token OPD advantage:
For the same-tokenizer Qwen3 teacher–student setting, the sampled-token OPD row is taken from Appendix D (Table 7). The results of CTOPD are comparable to ordinary sampled-token OPD.
| Method | AIME24 | AIME25 | AIME26 | MATH | Avg. |
|---|---|---|---|---|---|
| Sampled-token OPD (same tokenizer) | 65.0 | 64.6 | 63.5 | 89.0 | 70.5 |
| CTOPD | 64.8 | 65.0 | 63.3 | 89.2 | 70.6 |
Thank you very much to the authors for your thoughtful responses and additional experiments. I sincerely appreciate the time and effort you have put into addressing my concerns. I found the additional analyses and clarifications helpful.
However, after carefully considering the responses, I think I still have a few remaining reservations that I would like to share.
-
- I recognize that enabling on-policy distillation across different tokenizers is an important and practically meaningful problem. At the same time, several prior works, including MinED [1], ULD [2], CDM [3], and ALM [4], have already explored different approaches to cross-tokenizer alignment and knowledge transfer. Moreover, GOLD [5] has extended cross-tokenizer distillation to on-policy settings. While I recognize that the paper's methodlogy introduces a distinct approach to token-level credit assignment, my main reservation is whether this specific mechanism constitutes a sufficiently substantial methodological contribution and "novelty", given the range of existing approaches to cross-tokenizer alignment and distillation. I appreciate the proposed approach, but I remain uncertain about the significance of its additional contribution beyond what has already been explored in the literature.
-
- I appreciate the authors' effort to conduct an additional comparison using the same student model across both settings But my original intention in suggesting this experiment was to better isolate the effect of tokenizer compatibility by comparing same-tokenizer and cross-tokenizer distillation under otherwise comparable conditions, particularly with teachers of similar capabilities. In the additional experiment, however, the two teachers differ considerably in model size and potentially in capability and reasoning style. As the authors also suggest, the performance degradation in the same-tokenizer setting may be related to a mismatch in reasoning styles. This makes it difficult to distinguish the effect of tokenizer compatibility from other differences between the teacher models. While I appreciate the additional experiment and the insights it provides, I feel that my original concern remains only partially addressed.
-
- I also appreciate the authors' detailed explanation of the semantic-prior credit assignment mechanism and the supporting ablation results. However, instead of a clarification, I was actaully hoping to see more direct empirical evidence examining the robustness of the proposed mechanism in such cases. In particular, it remains unclear whether semantically unimportant tokens might receive disproportionately large-magnitude advantage signals in practice and, if so, whether this affects the distillation process. I believe a more targeted analysis would have helped address this concern more convincingly.
Overall, I sincerely appreciate the authors' efforts and the additional experiments and clarifications. However, I still have some reservations regarding the significance of the methodological contribution and the remaining limitations in empirical validation.
For these reasons, I have decided to maintain my original score
[1] Wan et al., Knowledge Fusion of Large Language Models, ICLR 2024
[2] Boizard et al., Towards Cross-Tokenizer Distillation: The Universal Logit Distillation Loss for LLMs, TMLR 2025
[3] Chen et al., Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping, Findings of ACL 2025
[4] Minixhofer et al., Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching, NeurIPS 2025
[5] Hugging Face, General Online Logit Distillation (GOLD) Trainer, TRL Documentation, accessed 2026.
Response to Reviewer 6pbf
We thank the reviewer for this insightful comments.
1. Token-level credit on student rollouts. Token-level credit on student rollouts.** MinED, ULD, CDM and ALM align tokens or define distillation losses across tokenizers. When an aligned chunk contains several sampled student tokens, these methods do not specify a signed policy-gradient advantage for each token. This leaves the advantage of the sampled token undefined.
GOLD trains on on-policy student rollouts, but its rank-matching step sorts the teacher and student vocabulary probabilities independently. The loss therefore pairs entries at the same rank even when the underlying tokens encode unrelated continuations.
CTOPD computes the teacher–student log-likelihood difference for the same decoded span and divides that advantage among the sampled student tokens in proportion to their logits under the student. The token advantages sum to the span advantage, and a near-certain continuation receives little credit or blame to preserve semantic prior.
2. Tokenizer compatibility. We fix the R1-Distill-Qwen-7B student and the 20k-prompt budget. To match DeepSeek-R1 as closely as practical in reasoning style and capability, we chose R1-Distill-Qwen-32B, the largest Qwen model distilled on R1 traces in the original release. In this way, teacher and student models share the same reasoning style. Mean is the unweighted average across the four benchmarks. Gain is measured in points relative to the SFT student, using the displayed means.
| Model / training | AIME24 | AIME25 | MATH500 | LCB | Mean | Gain |
|---|---|---|---|---|---|---|
| R1-Distill-Qwen-7B (SFT initialization) | 53.8 | 46.3 | 88.4 | 25.3 | 53.5 | — |
| 7B student, DeepSeek-R1 teacher (cross-tokenizer CTOPD) | 58.5 | 52.7 | 91.8 | 28.0 | 57.8 | +4.3 |
| 7B student, R1-Distill-Qwen-32B teacher (same-tokenizer OPD) | 60.0 | 50.8 | 92.4 | 28.8 | 58.0 | +4.5 |
Both configurations improve on SFT on all four benchmarks. We also discuss the per-benchmark results in point 3 of our response to Reviewer hbYd.
3. Do symbols and fragments receive disproportionate advantage?
Special tokens: We first analyzed the special tokens within the student model. We found that for 99.2% of these special tokens—even though they are split into multiple sub-tokens—the probability of the tokens following the first one is essentially 1. Thus, the advantage of these tokens are zero.
Symbols and fragments: Across 512 student rollouts, we measured the share of total absolute advantage attributable to punctuation and symbols over the full response sequences. These tokens make up 17.56% of all response tokens and account for only 10.40% of the total absolute advantage.
The table focuses on the 181,907 student tokens in aligned chunks containing at least two student tokens, where CTOPD distributes credit within a chunk. Both percentage columns report shares within this subset.
| Token Category | Token Proportion | Proportion of Absolute Advantage (CTOPD) |
|---|---|---|
| Punctuation and Symbols | 5.94% | 2.10% |
| Punctuation and Symbols (including tokens with surrounding spaces) | 7.62% | 3.36% |
| Intra-word Continuations | 0.34% | 0.10% |
| Whitespace-only Tokens (5 tokens) | <0.01% | 0.01% |
Even when tokens with surrounding spaces are included, punctuation and symbols account for only 3.36% of the absolute advantage in these chunks. In the downstream ablation (§5.6), CTOPD averages 53.0 across four math benchmarks, compared with 52.1 when the same chunk advantage is split evenly.
Response to Reviewer 6pbf (Part 1/2)
Thank you for your helpful feedback. We address your comments below.
We thank the reviewer for this insightful comments. Standard policy-gradient OPD requires assigning an "advantage" to each sampled token (Eq. 1). However, existing cross-tokenizer methods (e.g. ALM, GOLD, and ULD) define the SFT-like loss functions at the granularity of "chunks". Consequently, these existing cross-tokenizer methods are unable to provide a token-level advantage that could be incorporated into standard policy-gradient OPD loss function.
CTOPD addresses this through semantic-prior token-level credit assignment. It distributes teacher–student chunk disagreement into signed token advantages while preserving the student's within-chunk log-probability proportions. These advantages integrate directly into the standard OPD loss (Eq. 1). DPCA supplies the aligned chunks by matching NFC-normalized decoded text, with explicit handling of byte-fallback sequences and special tokens (Appendix C).
Broader model-size coverage. The pool of open-source teacher–student pairs suitable for our distillation setup is limited. To extend the evaluation across model sizes, we added DeepSeek-R1 → DeepSeek-R1-Distill-Qwen-14B on AIME24, AIME25, LCB, and MATH-500.
CTOPD achieves a 70.6% average, 1.3 points above GOLD, with gains on all four benchmarks. This extends the evidence for CTOPD to a 14B student.
Same-student comparison. We fix the Llama-3.1-8B-SFT-400K student, initialized with Qwen-style reasoning trajectories, and compare two teachers:
The decline with DeepSeek-R1-Distill-Llama-70B is consistent with a mismatch between its DeepSeek-style reasoning and the student's Qwen-style reasoning. This suggests that reasoning-style compatibility matters alongside tokenizer compatibility. CTOPD expands the choice of teachers across tokenizer boundaries. Teacher capability and reasoning style also vary in this comparison.
(1) Special tokens and their fragments. For a familiar control marker such as
<think>, both the teacher's single token and the student's components (<,think,>) have high conditional probabilities (≈1). The semantic-prior credit assignment assigns small weights to these special tokens.(2) CTOPD allocates larger advantages to lowe-probability tokens to preserve the update direction. A multi-token chunk usually forms a coherent semantic unit (e.g., a compound word or phrase). In subword tokenization, the first token usually carries the key decision/uncertainty for initiating the concept/word, while subsequent tokens represent high-confidence prefix completions (a strong prior acquired during pretraining). For example, consider a word
preprocessing, split by the student into[pre, process, ing]with probabilities [0.6,1,1], while the teacher assigns the whole word probability 0.8. The chunk advantage is positive because 0.8>0.6×12. Uniformly splitting the teacher's chunk log-likelihood sets each target to 0.81/3≈0.928, giving the completion tokens (process,ing) negative advantages, log(0.928/1)<0, opposite to the chunk-level signal. Semantic-prior allocation gives advantages of approximately [0.2,0,0]: all non-negative, with only small corrections to confident completions. This is equivalent to learning from a "teacher" with a probability of [0.8,1,1]. The controlled ablation in Section 5.6 supports our design: semantic-prior credit assignment improves average accuracy by 0.8–0.9 points over the two uniform rules.