Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families

On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models. However, existing OPD distillation methods restrict teacher and student models to share the same tokenizer within the model series. Current mainstream practice typically employs Supervised Fine-Tuning (SFT) on teacher-generated responses for cross-tokenizer distillation, which fails to capture the rich knowledge embedded in the teacher's probability distribution. In this work, we propose Cross-Tokenizer On-Policy Distillation (CTOPD) to enable the standard on-policy distillation method to operate across model families, ensuring that high-fidelity token-level signals can propagate across different tokenizers with a precise token-mapping algorithm. Extensive experiments show that CTOPD is significantly more compute-efficient than baselines on various benchmarks. Our results unlock a broader range of teacher–student pairs for OPD, opening up new avenues for adapting and enhancing interactions between LLMs.