- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the challenges of On-Policy Distillation (OPD) when the teacher and student models use different tokenizers, which causes misalignments in both input token sequences and output vocabulary spaces. To resolve this, the authors propose the Dual-Vocabulary Language Model (DVLM), a framework that adapts the teacher's architecture to directly process the student's tokenizations and output predictions within the student's vocabulary space.
Contributions To achieve this cross-vocabulary alignment, the authors introduce a Dual-Vocabulary Teacher that replaces the teacher's original LM head with a newly-trained student-specific projection layer. This structural adaptation is supported by a Parallel-Tokenized Sequence (PTS) input format and a Hybrid-Prefix Attention (HPA) masking mechanism to align input representation spaces during a two-stage training pipeline.
Validation The authors evaluate DVLM by distilling a Qwen-3-4B teacher into Llama-3.2-1B and OLMo-2-1B student models across standard downstream NLP benchmarks.
Critical strengths
The core concept of adapting the teacher model to directly output logits in the student's token space is an elegant and promising intuition that avoids heuristic vocabulary truncation. Additionally, the paper is well-written, with a clear presentation of the methodology and mathematical formulations.
Critical weaknesses
-
High Methodological Overhead for Marginal Performance Gains and Missing Baselines While the goal of bridging the vocabulary gap is commendable, the empirical gains appear marginal compared to the substantial overhead incurred. Adapting the teacher requires modifying the LM head, altering the attention mechanism, and introducing an extra pre-training stage (Stage I). Despite these efforts, the student models in Table 1 only achieve a modest improvement of ~2.8 to 3.1 points, leaving an enormous ~35-point performance gap with the teacher virtually untouched. Furthermore, because the teacher model itself is substantially altered, the authors do not report a self-distillation baseline of the adapted teacher on the student's vocabulary, making it difficult to verify whether the teacher's original generative capabilities were degraded during adaptation.
-
Extreme Sensitivity and Fragility Revealed by the Ablation Study The ablation results in Table 2 raise critical questions regarding the robustness and underlying mechanism of the proposed framework. Specifically, when re-tokenization is removed (w/o Re-token), the performance catastrophically drops to 14.45 and 13.36 on Llama and OLMo, respectively—falling substantially below even the naive base student models (20.43 and 21.85). This dramatic collapse indicates that the adapted teacher backbone cannot reliably project onto the student's vocabulary space on its own, suggesting that the Parallel-Tokenized Sequence acts more as a fragile data-engineering workaround to prevent model collapse rather than facilitating robust, generalizable representation alignment.
-
Invasive Teacher Architecture Modifications and Heuristic Attention Constraints Unlike conventional non-intrusive alignment methods (e.g., span-level or token-free approaches) where the teacher remains fully frozen, DVLM heavily alters the teacher’s core architecture. The Hybrid-Prefix Attention (HPA) relies on a set of hand-crafted, heuristic attention-masking rules to coordinate token visibility between the parallel sequences. Such bespoke masking constraints not only introduce engineering friction when scaling or transferring to different model families and context lengths, but they also compromise the plug-and-play nature expected from modern knowledge distillation paradigms.
Questions for authors
To better evaluate the true effectiveness of the proposed Dual-Vocabulary Teacher, could you provide the evaluation results (e.g., perplexity or zero-shot performance on standard benchmarks) of the adapted teacher model itself after Stage I training?
Could you include stage I when analyzing complexity / runtime?
Rebuttal to Reviewer Xxp7
We are grateful for your careful review, helpful comments and detailed questions that ensued.
For W1&&Q1 'high overhead' and 'the evaluation results of the adapted teacher':
For W2 'sensitivity and fragility':
For W3 'DVLM heavily alters‘ :
For Q2 'analyzing runtime':