- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies alignment-based cross-tokenizer knowledge distillation (CTKD) when tokenizers are not shared across teacher and student models. The authors investigate how constructing distributions from raw logits versus log-probabilities affects the distillation objective, showing that linear candidate scores are invariant to position-wise logit shifts only when all candidates assign equal weight to each token position.
Critical strengths
Cross-tokenizer knowledge distillation is a practical and general issue, as tokenizers are not shared across teacher and student models in most cases.
Critical weaknesses
This paper is still not ready for publication. The presentation is chaotic and provides little concrete, coherent information, while burying me in fragmented information. The core contribution of the main text seems to be merely an evaluation of a single recent method SimCT. I feel the authors are just debugging SimCT without sufficient scientific value.
Questions for authors
Please see Weaknesses.