acceptodds
Under review as a conference paper at ICLR 2027

Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families

Abstract

On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models. However, existing OPD distillation methods restrict teacher and student models to share the same tokenizer within the model series. Current mainstream practice typically employs Supervised Fine-Tuning (SFT) on teacher-generated responses for cross-tokenizer distillation, which fails to capture the rich knowledge embedded in the teacher's probability distribution. In this work, we propose Cross-Tokenizer On-Policy Distillation (CTOPD) to enable the standard on-policy distillation method to operate across model families, ensuring that high-fidelity token-level signals can propagate across different tokenizers with a precise token-mapping algorithm. Extensive experiments show that CTOPD is significantly more compute-efficient than baselines on various benchmarks. Our results unlock a broader range of teacher–student pairs for OPD, opening up new avenues for adapting and enhancing interactions between LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language models, on-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6pbfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Cross-Tokenizer On-Policy Distillation (CTOPD), which enables OPD between models with different tokenizers. The authors do this by applying an algorithm that does a chunk level credit assignment via dual pointer based on semantic-priors. Experiments show effectiveness of the method.

Critical strengths
  1. Enabling OPD across different kinds of models despite having different tokenizers is an important contribution.
  2. Intuitive methodology. logic is not really flawed and it make sense
  3. The paper presents promising experimental results across multiple benchmarks, demonstrating the effectiveness and potential computational efficiency of the proposed method.
Critical weaknesses
  1. In the related works section [Cross-Tokenizer Distillation], the authors stated "However, existing methods are mostly designed for offline distillation strategies and cannot effectively solve the token-level signal propagation problem in OPD." However, this remains unclear for me why prior works cannot solve the token-level signal propagation problem. And how does this mean that the paper's Dual-Pointer Chunk Alignment algorithm solves those "limitations" of prior works.

  2. Although the experimental results show the effectiveness of the method, I believe more comprehensive experimental validation is needed in order to be accepted to ICLR venue.. For example, despite cross tokenizer is the main keyword and contribution, the authors only show 2 cases of teacher -> student pair. I believe that it would have been better if the paper have shown more experiments on various models and sizes and more.

I have organized my suggestions for additional analyses and experiments in the Questions section below. Please refer to that section for further details.

Questions for authors
  1. I have understood that the method's semantic-prior credit assignment allocates larger advantages to tokens with lower probabilities. However, I think low probability tokens may include tokenizer specific fragments / special symbols (e.g., < and >), which are not necessarily semantically important. Could the authors clarify whether their method might disproportionately emphasize such tokens, and how this affects results?

  2. Section 5.3's compute cost analysis compares CTOPD against continued SFT, which uses a different training paradigm. While this comparison shows that their method is overall more efficient than SFT, it is unclear how much of the improvement can be attributed specifically to the proposed chunk alignment and semantic-prior credit assignment, rather than to OPD itself. Maybe it would be clearer if the authors can provide a comparison with existing other OPD methods too. Currently, it is unsure whether the method's contribution itself makes the cost more efficient.

  3. For Section 5.5, the paper shows the analysis in different student models, which makes it difficult to isolate the effect of tokenizer mismatch. Would it be possible to provide a more controlled comparison using the same student model across both settings, ideally with teachers of comparable capabilities?

  4. While reading, I got curious whether the method will be able to show identical or similar performance when the teacher and student models have the same tokenizers. Ideally, and in theory, I think they should be identical or similar to just ordinary OPD with no cross tokenizer algorithm technique. But if there are substantial differences, then I think this should be something worth more investigating. Could the authors verify this?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6pbf (Part 1/2)

Thank you for your helpful feedback. We address your comments below.

Weakness 1: The claim that prior cross-tokenizer methods cannot effectively propagate token-level supervision in OPD needs explanation. What prevents those methods from doing so, and how does DPCA address these limitations?

We thank the reviewer for this insightful comments. Standard policy-gradient OPD requires assigning an "advantage" to each sampled token (Eq. 1). However, existing cross-tokenizer methods (e.g. ALM, GOLD, and ULD) define the SFT-like loss functions at the granularity of "chunks". Consequently, these existing cross-tokenizer methods are unable to provide a token-level advantage that could be incorporated into standard policy-gradient OPD loss function.

CTOPD addresses this through semantic-prior token-level credit assignment. It distributes teacher–student chunk disagreement into signed token advantages while preserving the student's within-chunk log-probability proportions. These advantages integrate directly into the standard OPD loss (Eq. 1). DPCA supplies the aligned chunks by matching NFC-normalized decoded text, with explicit handling of byte-fallback sequences and special tokens (Appendix C).

Weakness 2 & Question 3: Two teacher–student pairs provide limited coverage of model families and sizes. Please broaden the evaluation and isolate tokenizer mismatch using the same student with teachers of comparable capability.

Broader model-size coverage. The pool of open-source teacher–student pairs suitable for our distillation setup is limited. To extend the evaluation across model sizes, we added DeepSeek-R1 → DeepSeek-R1-Distill-Qwen-14B on AIME24, AIME25, LCB, and MATH-500.

Student / method (teacher: DeepSeek-R1)AIME24AIME25LCBMATH-500Avg.
DeepSeek-R1-Distill-Qwen-14B initialization69.750.053.193.966.7
DeepSeek-R1-Distill-Qwen-14B + GOLD73.553.655.294.969.3
DeepSeek-R1-Distill-Qwen-14B + CTOPD75.055.456.495.470.6

CTOPD achieves a 70.6% average, 1.3 points above GOLD, with gains on all four benchmarks. This extends the evidence for CTOPD to a 14B student.

Same-student comparison. We fix the Llama-3.1-8B-SFT-400K student, initialized with Qwen-style reasoning trajectories, and compare two teachers:

Teacher / methodAIME24AIME25AIME26MATH-500Avg.
SFT initialization38.531.736.580.846.9
Qwen3-8B / cross-tokenizer CTOPD44.443.341.782.653.0 (+6.1)
DeepSeek-R1-Distill-Llama-70B / same-tokenizer OPD30.829.227.577.241.2 (−5.8)

The decline with DeepSeek-R1-Distill-Llama-70B is consistent with a mismatch between its DeepSeek-style reasoning and the student's Qwen-style reasoning. This suggests that reasoning-style compatibility matters alongside tokenizer compatibility. CTOPD expands the choice of teachers across tokenizer boundaries. Teacher capability and reasoning style also vary in this comparison.

Question 1: Low-probability tokens can be tokenizer fragments or symbols rather than semantically important units. Does semantic-prior credit assignment overemphasize these tokens, and how does this affect performance?

(1) Special tokens and their fragments. For a familiar control marker such as <think>, both the teacher's single token and the student's components (<, think, >) have high conditional probabilities (≈1\approx 1). The semantic-prior credit assignment assigns small weights to these special tokens.

(2) CTOPD allocates larger advantages to lowe-probability tokens to preserve the update direction. A multi-token chunk usually forms a coherent semantic unit (e.g., a compound word or phrase). In subword tokenization, the first token usually carries the key decision/uncertainty for initiating the concept/word, while subsequent tokens represent high-confidence prefix completions (a strong prior acquired during pretraining). For example, consider a word preprocessing, split by the student into [pre, process, ing] with probabilities [0.6,1,1][0.6, 1, 1], while the teacher assigns the whole word probability 0.80.8. The chunk advantage is positive because 0.8>0.6×120.8 > 0.6\times1^2. Uniformly splitting the teacher's chunk log-likelihood sets each target to 0.81/3≈0.9280.8^{1/3}\approx0.928, giving the completion tokens (process, ing) negative advantages, log⁡(0.928/1)<0\log(0.928/1)<0, opposite to the chunk-level signal. Semantic-prior allocation gives advantages of approximately [0.2,0,0][0.2, 0, 0]: all non-negative, with only small corrections to confident completions. This is equivalent to learning from a "teacher" with a probability of [0.8,1,1][0.8, 1, 1]. The controlled ablation in Section 5.6 supports our design: semantic-prior credit assignment improves average accuracy by 0.8–0.9 points over the two uniform rules.

AuthorsOpenReview ↗

Response to Reviewer 6pbf (Part 2/2)

Question 2: The SFT comparison does not separate the efficiency gains of OPD itself from those of chunk alignment and semantic-prior credit assignment. Can comparisons with existing OPD methods establish the proposed components' contribution to compute efficiency?

In our experiments, all on-policy distillation algorithms we reimplemented in our paper utilize the same amount of computation (the number of training data samples) to ensure a fair comparison. Since CTOPD achieves better performance under fixed budget, it demonstrates higher efficiency. To quantify this efficiency difference, we examined the number of training samples required for CTOPD to match the performance of our strongest baseline GOLD. Compared to 20k samples for GOLD, CTOPD achieves this using approximately 18k training samples. The data efficiency improvement is approximately 1.1x (20/18≈1.120/18 \approx 1.1). Both methods utilize on-policy training, with comparable per-step execution times.

Question 4: Does CTOPD recover ordinary OPD when teacher and student use the same tokenizer? Please verify whether the two methods produce identical or similar performance and explain any substantial differences.

Yes. In same-tokenizer setting, it can be mathematically proved that the alignment results of DPCA is a one-to-one mapping between tokens. Thus, Eq. 9 reduces to the ordinary sampled-token OPD advantage: A^i=log⁡πT−log⁡πS.\hat{A}_i=\log\pi_T-\log \pi_S.

For the same-tokenizer Qwen3 teacher–student setting, the sampled-token OPD row is taken from Appendix D (Table 7). The results of CTOPD are comparable to ordinary sampled-token OPD.

MethodAIME24AIME25AIME26MATHAvg.
Sampled-token OPD (same tokenizer)65.064.663.589.070.5
CTOPD64.865.063.389.270.6

Thank you very much to the authors for your thoughtful responses and additional experiments. I sincerely appreciate the time and effort you have put into addressing my concerns. I found the additional analyses and clarifications helpful.

However, after carefully considering the responses, I think I still have a few remaining reservations that I would like to share.

    1. I recognize that enabling on-policy distillation across different tokenizers is an important and practically meaningful problem. At the same time, several prior works, including MinED [1], ULD [2], CDM [3], and ALM [4], have already explored different approaches to cross-tokenizer alignment and knowledge transfer. Moreover, GOLD [5] has extended cross-tokenizer distillation to on-policy settings. While I recognize that the paper's methodlogy introduces a distinct approach to token-level credit assignment, my main reservation is whether this specific mechanism constitutes a sufficiently substantial methodological contribution and "novelty", given the range of existing approaches to cross-tokenizer alignment and distillation. I appreciate the proposed approach, but I remain uncertain about the significance of its additional contribution beyond what has already been explored in the literature.
    1. I appreciate the authors' effort to conduct an additional comparison using the same student model across both settings But my original intention in suggesting this experiment was to better isolate the effect of tokenizer compatibility by comparing same-tokenizer and cross-tokenizer distillation under otherwise comparable conditions, particularly with teachers of similar capabilities. In the additional experiment, however, the two teachers differ considerably in model size and potentially in capability and reasoning style. As the authors also suggest, the performance degradation in the same-tokenizer setting may be related to a mismatch in reasoning styles. This makes it difficult to distinguish the effect of tokenizer compatibility from other differences between the teacher models. While I appreciate the additional experiment and the insights it provides, I feel that my original concern remains only partially addressed.
    1. I also appreciate the authors' detailed explanation of the semantic-prior credit assignment mechanism and the supporting ablation results. However, instead of a clarification, I was actaully hoping to see more direct empirical evidence examining the robustness of the proposed mechanism in such cases. In particular, it remains unclear whether semantically unimportant tokens might receive disproportionately large-magnitude advantage signals in practice and, if so, whether this affects the distillation process. I believe a more targeted analysis would have helped address this concern more convincingly.

Overall, I sincerely appreciate the authors' efforts and the additional experiments and clarifications. However, I still have some reservations regarding the significance of the methodological contribution and the remaining limitations in empirical validation.

For these reasons, I have decided to maintain my original score


[1] Wan et al., Knowledge Fusion of Large Language Models, ICLR 2024

[2] Boizard et al., Towards Cross-Tokenizer Distillation: The Universal Logit Distillation Loss for LLMs, TMLR 2025

[3] Chen et al., Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping, Findings of ACL 2025

[4] Minixhofer et al., Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching, NeurIPS 2025

[5] Hugging Face, General Online Logit Distillation (GOLD) Trainer, TRL Documentation, accessed 2026.

AuthorsOpenReview ↗

Response to Reviewer 6pbf

We thank the reviewer for this insightful comments.

1. Token-level credit on student rollouts. Token-level credit on student rollouts.** MinED, ULD, CDM and ALM align tokens or define distillation losses across tokenizers. When an aligned chunk contains several sampled student tokens, these methods do not specify a signed policy-gradient advantage for each token. This leaves the advantage of the sampled token undefined.

GOLD trains on on-policy student rollouts, but its rank-matching step sorts the teacher and student vocabulary probabilities independently. The loss therefore pairs entries at the same rank even when the underlying tokens encode unrelated continuations.

CTOPD computes the teacher–student log-likelihood difference for the same decoded span and divides that advantage among the sampled student tokens in proportion to their logits under the student. The token advantages sum to the span advantage, and a near-certain continuation receives little credit or blame to preserve semantic prior.

2. Tokenizer compatibility. We fix the R1-Distill-Qwen-7B student and the 20k-prompt budget. To match DeepSeek-R1 as closely as practical in reasoning style and capability, we chose R1-Distill-Qwen-32B, the largest Qwen model distilled on R1 traces in the original release. In this way, teacher and student models share the same reasoning style. Mean is the unweighted average across the four benchmarks. Gain is measured in points relative to the SFT student, using the displayed means.

Model / trainingAIME24AIME25MATH500LCBMeanGain
R1-Distill-Qwen-7B (SFT initialization)53.846.388.425.353.5—
7B student, DeepSeek-R1 teacher (cross-tokenizer CTOPD)58.552.791.828.057.8+4.3
7B student, R1-Distill-Qwen-32B teacher (same-tokenizer OPD)60.050.892.428.858.0+4.5

Both configurations improve on SFT on all four benchmarks. We also discuss the per-benchmark results in point 3 of our response to Reviewer hbYd.

3. Do symbols and fragments receive disproportionate advantage?

Special tokens: We first analyzed the special tokens within the student model. We found that for 99.2% of these special tokens—even though they are split into multiple sub-tokens—the probability of the tokens following the first one is essentially 1. Thus, the advantage of these tokens are zero.

Symbols and fragments: Across 512 student rollouts, we measured the share of total absolute advantage attributable to punctuation and symbols over the full response sequences. These tokens make up 17.56% of all response tokens and account for only 10.40% of the total absolute advantage.

The table focuses on the 181,907 student tokens in aligned chunks containing at least two student tokens, where CTOPD distributes credit within a chunk. Both percentage columns report shares within this subset.

Token CategoryToken ProportionProportion of Absolute Advantage (CTOPD)
Punctuation and Symbols5.94%2.10%
Punctuation and Symbols (including tokens with surrounding spaces)7.62%3.36%
Intra-word Continuations0.34%0.10%
Whitespace-only Tokens (5 tokens)<0.01%0.01%

Even when tokens with surrounding spaces are included, punctuation and symbols account for only 3.36% of the absolute advantage in these chunks. In the downstream ablation (§5.6), CTOPD averages 53.0 across four math benchmarks, compared with 52.1 when the same chunk advantage is split evenly.

Reviewer_hbYdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes Cross-Tokenizer On-Policy Distillation (CTOPD), which extends sampled-token on-policy distillation (OPD) to teacher–student pairs with different tokenizers. It first uses a dual-pointer, length-based "catch-up" procedure (DPCA) that splits them into synchronized chunks that decode to identical text. For each pair of chunk, it then projects the teacher's chunk log-likelihood onto the student tokens by providing a closed-form solution for a constrained least-squares problem. The paper shows that their method outperform baselines on two OPD settings (Qwen3-8B → Llama3.1-8B and DeepSeek-R1 → R1-Distill-Qwen-7B).

Critical strengths
  1. This paper studies an important problem in OPD and provide a simple yet effective solution.
  2. Experiments show consistent gains over the closest on-policy baseline, GOLD from huggingface.
Critical weaknesses
  1. The core credit-assignment component is not well-founded

a. The target distribution is mainly built from the student distribution, which seems weird in the OPD setting because it does not reflect student–teacher disagreement.

b. The definition of k in Eq.7 is not clearly stated.

c. The closed-form solution (Eq.8) is independent of k in Eq.7, which appears to be wrong.

  1. The proposed method is not specific to OPD setting (i.e. did not use the properties of OPD). So it is natural to compare with the previous cross-tokenizer method in general, which is missing in the paper. Besides, the proposed method cannot be extended to top-k or full-distribution OPD. The authors' own Appendix D shows top-k OPD is stronger in the same-tokenizer setting. The method's ceiling is sampled-token OPD.

  2. Some experiment settings are not solid.

a. The compute analysis is only compared with the SFT, not other baselines like GOLD

b. It is not convincing to compare the absolute gain in Table 3. It is recommended to use the same student model, and compare the improvement from different teachers with similar capacity (same-tokenizer vs cross-tokenizer)

Questions for authors
  1. How is k defined mathematically in Eq. 7? Could you check if Eq.8 is the right closed-form solution?
5 comments
AuthorsOpenReview ↗

Response to Reviewer hbYd (Part 1/2)

Thank you for taking the time to review our paper and providing valuable feedback. We would like to answer your questions below.

Weakness 1 & Question 1: It is unclear how student-derived targets capture teacher–student disagreement, and kk in Eq. 7 is insufficiently defined. Please define kk mathematically and explain why Eq. 8 is the correct closed-form solution despite not containing kk.

(a) Teacher supervision and student prior play different roles. The teacher-student disagreement determines the total advantage for each chunk, while the student prior handles the distribution within that chunk. Because the tokens within a chunk are generated by the student, their corresponding log-probabilities encapsulate the student’s own semantic priors; preserving these priors allows for the reduction of teacher-student disagreement without compromising the model’s existing token-completion capabilities. In Section 5.6, we provide comparative experimental results that highlight the advantages of this design.

(b&c) Interpretation and closed-form solution of k. The factor k>0k>0 rescales the student's −log⁡pi-\log p_i to match the teacher's chunk likelihood. It is determined separately for each chunk, rather than tuned as a hyperparameter. Let LS(c)=∑ilog⁡pi\mathcal{L}_S^{(c)}=\sum_i\log p_i and LT(c)\mathcal{L}_T^{(c)} denote the student and teacher chunk log-likelihoods. We jointly optimize kk and the target probabilities qiq_i:

\frac12\sum_{i=1}^{m}(\log q_i-k\log p_i)^2 \quad\text{s.t.}\quad \sum_{i=1}^{m}\log q_i=\mathcal{L}_T^{(c)}.$$ The non-negative objective reaches zero when $\log q_i=k\log p_i$. Substituting into the constraint gives $k\mathcal{L}_S^{(c)}=\mathcal{L}_T^{(c)}$, hence $$k^{\ast}=\frac{\mathcal{L}_T^{(c)}}{\mathcal{L}_S^{(c)}}.$$ Eq. 8 follows by substituting $k^*$. > **Weakness 2:** The proposed method is not specific to OPD setting (i.e. did not use the properties of OPD). So it is natural to compare with the previous cross-tokenizer method in general, which is missing in the paper. Besides, the proposed method cannot be extended to top-k or full-distribution OPD. The authors' own Appendix D shows top-k OPD is stronger in the same-tokenizer setting. The method's ceiling is sampled-token OPD. Thanks for the comment. Compared to existing cross-tokenizer method, our core contributions are as follows: * Generalizable Chunk Alignment. Unlike prior methods (e.g. ALM, GOLD, ULD) that rely on heuristic or tokenizer-specific string matching, we introduce an alignment mechanism operating directly on NFC-normalized space and does not need specific replacement rules for a particular tokenizer. Our implementation is more general across tokenizer pairs. * Novel Formulation of Token-Level Credit Assignment via Semantic Prior. To the best of our knowledge, our credit assignment method is a new formulation. It decomposes "chunk-level" reward signals into "token-level" signals and can be directly incorporated into the standard OPD loss function (Eq. 1 \& 2). Meanwhile, it avoids overriding the student's pretrained semantic priors * Existing cross-tokenizer approaches are off-policy distillation methods with a SFT-like loss, while CTOPD makes on-policy cross-tokenizer distillation feasible. CTOPD achieves higher sample efficiency. In our experiments, Section 5.2 compares CTOPD with SFT, CDM, ULD, and GOLD. For fair comparison, we also reimplement ALM to on-policy setting. For example, CTOPD outperforms GOLD by 2.2/2.1 average points in the two model pairs. Section 5.6 further shows a 1.5-point gain over on-policy ALM under matched training settings. Top-k cross-tokenizer OPD remains challenging. For other non-sampled tokens in different vocabularies, establishing a fixed mapping is difficult. This is because, in a cross-tokenizer scenario, the alignment between tokens is entirely context-dependent. Without context, it is impossible to directly map non-sampled tokens, as there is no clear instruction for deciding whether to split a large token from the "teacher" or the "student". Some baselines (CDM, ULD, and GOLD) attempt this transfer, yet CTOPD outperforms all three on both model pairs (Section 5.2). Top-$k$ signal without precisely mapping does not guarantee better performance. Appendix D's Top-32 advantage (71.5% vs. 70.5%) concerns a shared tokenizer and does not establish a performance ceiling for cross-tokenizer CTOPD. Extending CTOPD to faithfully aligned top-k supervision remains future work.
AuthorsOpenReview ↗

Response to Reviewer hbYd (Part 2/2)

Weakness 3: Comparing compute only with SFT and accuracy gains across different students does not isolate CTOPD's contribution. Please compare compute with GOLD and use the same student with similarly capable teachers to test same-tokenizer versus cross-tokenizer distillation.

(a) Compute Analysis

Apart from SFT, all other on-policy distillation algorithms we reimplemented in our paper utilize the same amount of computation (the number of training data samples) to ensure a fair comparison. Since our method achieves superior performance under this identical computational budget, it demonstrates higher efficiency. To quantify this efficiency difference, we examined the number of training samples required for CTOPD to match the performance of our strongest baseline GOLD. Compared to 20k samples for GOLD, CTOPD achieves this using approximately 18k training samples. The data efficiency improvement is approximately 1.1x (20/18≈1.120/18 \approx 1.1). Both methods utilize on-policy training, with comparable per-step execution times.

(b) Same-student comparison. We use the Llama-3.1-8B-SFT-400K as student, with two different teachers to cover both cross and same tokenizer settings:

Teacher / methodAIME24AIME25AIME26MATH-500Avg.
SFT initialization38.531.736.580.846.9
Qwen3-8B / cross-tokenizer CTOPD44.443.341.782.653.0 (+6.1)
DeepSeek-R1-Distill-Llama-70B / same-tokenizer OPD30.829.227.577.241.2 (−5.8)

For DeepSeek-R1-Distill-Llama-70B / same-tokenizer OPD, the student model was initialized using Qwen-style reasoning trajectories, while the teacher model employed a DeepSeek-style reasoning approach. The significant discrepancy between these two reasoning patterns which prevented effective training even under same tokenizer setting. This underscores the critical importance of compatibility in thinking pattern.

Dear Reviewer,

We sincerely appreciate your thoughtful review. We would be truly grateful for any further feedback you may have. If any concerns remain unaddressed, we would be more than happy to provide further clarification.

Reviewer_hC7bOpenReview ↗
Rating
3
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes CTOPD, an on-policy distillation method for teacher–student models with different tokenizers. It aligns token sequences into minimal synchronized chunks and distributes teacher chunk log-likelihoods to student tokens using the student’s probability estimates. Experiments on two teacher–student pairs across six reasoning and coding benchmarks report improvements over existing cross-tokenizer distillation baselines.

Critical strengths
  1. The paper addresses a practically important problem for industrial applications: enabling distillation between models with different tokenizers.
  2. The experimental evaluation is coherent with the proposed method and includes efficiency analysis and case studies, providing relatively comprehensive validation.
  3. The mathematical formulation is clear and easy to follow.
Critical weaknesses
  1. The novelty is limited. Although cross-tokenizer on-policy distillation is practically relevant, the proposed method appears to be an incremental combination of chunk alignment and token-level credit assignment. The paper could better explain the conceptual insights beyond existing alignment-based methods, particularly ALM.
  2. The paper could further analyze the algorithm’s time complexity and clarify whether it is asymptotically optimal under the stated alignment constraints.
  3. Typo: 091 vocabularys 165 An token 253 ovrall
Questions for authors
  1. What are the main limitations of the proposed method?
  2. What broader insights does your method provide for cross-tokenizer distillation?
3 comments
AuthorsOpenReview ↗

Response to Reviewer hC7b

Thank you for your valuable feedbacks on our paper. We appreciate your time and effort in reviewing our work. We would like to address your questions and concerns below.

Weakness 1: The method appears to be an incremental combination of chunk alignment and token-level credit assignment. What conceptual advances over existing alignment methods, particularly ALM.

We thank the reviewer for this insightful comments. We would like to clarify that our method is NOT a straightforward combination of existing techniques. Our core contributions are as follows:

  • Generalizable Chunk Alignment. Unlike prior methods that rely on heuristic or tokenizer-specific string matching, we introduce an alignment mechanism operating directly on NFC-normalized space and does not need specific replacement rules for a particular tokenizer. Our implementation is more general across tokenizer pairs.
  • Novel Formulation of Token-Level Credit Assignment via Semantic Prior. To the best of our knowledge, our credit assignment method is a new formulation. It decomposes "chunk-level" reward signals into "token-level" signals and can be directly incorporated into the standard OPD loss function (Eq. 1). Meanwhile, it avoids overriding the student's pretrained semantic priors

Conceptual advances over ALM. (1) Alignment algorithm. ALM's tokenkit aligns byte-based token strings and relies on a manually maintained replacement table. Our implementation is more general and instead aligns the NFC-normalized decoded text and does not need specific replacement rules for a particular tokenizer. (2) Original ALM is a off-policy distillation method with a SFT-like loss, while CTOPD is a on-policy distillation method. CTOPD achieves higher sample efficiency. (3) The supervision signal for ALM is the coarser chunk level, whereas CTOPD is the finer token level. Chunk-level signals cannot be directly integrated into the standard OPD loss. Our token-level signal is not only compatible with the standard OPD loss but also preserves semantic priors.

Weakness 2: Please analyze DPCA's time complexity and clarify whether it is asymptotically optimal under the stated alignment constraints.

Given cached admissible boundary offsets that are strictly increasing and refer to prefixes of the same fixed reconstructed string, DPCA takes O(m+n) time and O(1) auxiliary space (excluding inputs and outputs). It is worst-case asymptotically optimal in the total token count N=m+n: identical, fully admissible boundary lists require Θ(N) explicit outputs, giving an Ω(N) lower bound. This result concerns the scan and excludes decoding, normalization, and boundary construction. We have clarified the boundary assumptions and algorithm in Section 4.1 and Appendix C, and provide the proofs in Appendix E.

Weakness 3: Please correct the typographical errors “vocabularys,” “An token,” and “ovrall” in the manuscript.

Thanks for the comment. We have corrected the typos in the manuscript.

Question 1: What are the main limitations of the proposed cross-tokenizer on-policy distillation method?

Thanks for the comment. A primary limitation is that our method is built upon sampled-token, policy-gradient OPD, which optimizes using scalar rewards along sampled trajectories rather than full-vocabulary distributions. Consequently, it is not directly compatible with top-kk or full-logit distillation objectives. Extending our credit assignment to top-kk cross-tokenizer distributions remains promising future work.

Question 2: What broader insights does your method provide for cross-tokenizer distillation?

Our work offers three broader takeaways for cross-tokenizer distillation:

  • Enabling Sample-Efficient On-Policy Learning: Prior cross-tokenizer methods rely heavily on off-policy training due to vocabulary mismatch. We make sample-efficient on-policy cross-tokenizer distillation feasible.
  • Necessity of Token-Level Supervision: While cross-tokenizer alignment naturally occurs at the chunk level, decomposing feedback into token-level signals is essential for compatibility with standard OPD losses and yields better performance.
  • Preserving Intra-Chunk Semantic Priors: Different tokenizers induce distinct intra-phrase token transition distributions during pretraining. Effective cross-tokenizer transfer requires aligning high-level semantic intent across chunks while respecting and preserving the student’s native token-level priors rather than overriding them uniformly.

Dear Reviewer,

We sincerely appreciate your thoughtful review. We would be truly grateful for any further feedback you may have. If any concerns remain unaddressed, we would be more than happy to provide further clarification.

Thank you for your response. Your explanations addressed most of my questions and concerns. I’m keeping my overall recommendation positive and raising both my Informativeness and Soundness scores to 3. Good luck to the authors!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.