acceptodds
Under review as a conference paper at ICLR 2027

Before the KL: Auditing Interfaces for Cross-Tokenizer Distillation

Abstract

Cross-tokenizer knowledge distillation (CTKD) applies its loss not directly to native model distributions, but to virtual distributions constructed by an alignment and scoring interface. We show that matching token coordinates is insufficient to reproduce this objective: representations that preserve every native softmax probability can still change candidate scores, virtual distributions, and parameter gradients. For fixed-support linear scoring rules, we derive a necessary and sufficient invariance condition: every candidate must assign the same total weight to each native position. A pinned public SimCT implementation violates this condition. Its published formula aggregates log probabilities, whereas its runtime aggregates raw logits with different position weights for spans and overlaps. In a frozen Falcon3-to-Pythia audit on 224 texts, converting logits to differentiable log probabilities changes native probabilities by at most , yet yields a median complete-gradient cosine of . A matched short-horizon diagnostic further produces distinct parameter paths, while raw-logit and formula-score evaluators prefer different endpoints; native negative log-likelihood favors the raw-trained endpoint. These results show that formula fidelity, evaluator preference, and model quality are distinct quantities. CTKD implementations should therefore report the score construction and its gradient path, rather than alignment alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
cross-tokenizer knowledge distillation, knowledge distillation, tokenizer alignment, language models, interface auditing, model evaluation, reproducibility, trustworthy machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1RRdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies alignment-based cross-tokenizer knowledge distillation (CTKD) when tokenizers are not shared across teacher and student models. The authors investigate how constructing distributions from raw logits versus log-probabilities affects the distillation objective, showing that linear candidate scores are invariant to position-wise logit shifts only when all candidates assign equal weight to each token position.

Critical strengths

Cross-tokenizer knowledge distillation is a practical and general issue, as tokenizers are not shared across teacher and student models in most cases.

Critical weaknesses

This paper is still not ready for publication. The presentation is chaotic and provides little concrete, coherent information, while burying me in fragmented information. The core contribution of the main text seems to be merely an evaluation of a single recent method SimCT. I feel the authors are just debugging SimCT without sufficient scientific value.

Questions for authors

Please see Weaknesses.

Reviewer_un81OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the effect of score construction in cross-tokenizer knowledge distillation. The authors identify a discrepancy between the published formulation of SimCT and its implementation, where raw logits are used instead of log probabilities. They show that this difference can lead to different virtual distributions and training gradients, even when native probabilities remain unchanged. The paper also provides a theoretical condition for invariance to logit shifts and an auditing framework. The experiments show differences in gradients, training trajectories, and evaluator preferences. However, no significant difference in downstream task performance is established.

Critical strengths
  • The paper identifies an interesting discrepancy between the publication and implementation of SimCT. It is a good observation that identical native probabilities do not necessarily result in the same distillation objective.

  • The authors provide controlled experiments to study the effect of this discrepancy on gradients and training. The distinction between objective correctness and model performance is also useful.

Critical weaknesses

Please refer to the questions below.

Questions for authors
  • What would be the main contribution of this paper if the SimCT implementation correctly followed its published formulation? It seems that a considerable part of the contributions and experiments depend on this discrepancy. Without it, the experiments mainly show that two different scoring functions produce different gradients and training trajectories, which is expected.

  • How novel is the theoretical contribution? The invariance condition seems to follow directly from the well-known softmax shift invariance property. Most of the mathematical observations are straightforward. Could the authors clarify what is fundamentally new in the theoretical results?

  • How general is the identified problem beyond SimCT? Most of the main experimental evidence is based on one implementation of SimCT. Is there any evidence that similar issues exist in other CTKD methods? Otherwise, it is not clear whether this is a general problem or mainly an implementation issue in one method.

  • What is the practical significance of the observed differences? The experiments show differences in gradients and optimization trajectories, but no significant downstream performance difference is established. It is not clear how much this discrepancy affects the actual quality of the distilled model.

Reviewer_GSHQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether cross-tokenizer distillation implementations faithfully construct their declared supervision distributions. For fixed-support linear candidate scores, it proves that invariance to position-wise additive logit shifts holds exactly when all candidates assign identical total weights to each native position. It then identifies a discrepancy between SimCT’s published log-probability scoring formula and a pinned implementation that aggregates raw logits.

Critical strengths

The intervention is well controlled

The authors report that native NLL favors the raw-trained student and avoid assuming that matching the published formula guarantees better performance. This separation strengthens the methodological message and prevents an implementation discrepancy from being misinterpreted as evidence of inferior predictive utility.

Critical weaknesses

The invariance proposition is a direct characterization of when candidate-dependent offsets survive softmax, while the executable audit follows standard controlled-comparison principles. Most of the substantive evidence concerns one pinned SimCT implementation. The paper would be stronger with either systematic findings across independently implemented interfaces or deeper evidence about how this discrepancy affects established distillation results.

The main training diagnostic uses 168 texts and 56 updates per fold, with one seed and overlapping training sets. Parameter divergence and evaluator-specific preferences establish different objectives, but provide limited evidence about the significance of the discrepancy in realistic training. The supplementary three-seed task evaluation does not resolve this as checker exceptions or extraction failures affect 8.77% of outputs, and the authors appropriately conclude that the results establish neither an advantage nor equivalence. A reliable evaluation is needed to determine the downstream importance of the finding.

Questions for authors

Does the discrepancy materially change results under the upstream SimCT training protocol, including on-policy rollouts, rather than the fixed-text short-training adapter?

Can the supplementary task outputs be rescored with a verified grader, including checks of extraction failures and mathematically equivalent answers? How do both arms compare with the shared starting checkpoint?

Beyond SimCT, which inspected implementations exhibit an executed score/formula discrepancy, rather than only a potential sensitivity identified by formula fixtures? This distinction would clarify the demonstrated breadth of the audit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.