acceptodds
Under review as a conference paper at ICLR 2027

CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer

Abstract

Cross-tokenizer logits distillation enables knowledge transfer across model families, allowing students to learn from stronger or more specialized teachers. However, differences in vocabulary and segmentation make teacher and student next-token distributions difficult to align. Existing logits-based methods are easy to integrate into standard distillation pipelines, but often rely on restrictive heuristics or tokenizer-pair-specific choices. They may restrict supervision to shared vocabulary subsets, require manually selecting different objectives for different tokenizer pairs, or match only the observed chunk and its complement rather than individual alternative continuations. We introduce Common-Support Distillation (CSD), a unified logits-based framework that constructs a shared support between the teacher and student at each aligned input chunk and matches the induced distributions with a single objective. For one-to-one alignments, CSD uses explicit common-token pairs as the shared support. For other alignments, it constructs a virtual support whose probabilities are computed using the autoregressive chain rule. A residual event captures probability mass outside the explicit support, ensuring that both induced distributions remain normalized. Across four teacher–student settings comprising Llama-3.2-1B and Gemma3-1B-PT students with Qwen and Phi teachers, CSD consistently achieves the highest average performance among the evaluated cross-tokenizer methods, outperforming the strongest alternative by up to points and surpassing both continued pretraining and same-tokenizer distillation. For Llama-3.1-8B, CSD improves average accuracy over the strongest evaluated baseline by up to points for the base student and points for the instruction-tuned student.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Knowledge Distillation, Cross-Tokenizer Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TPBoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Common-Support Distillation (CSD) for knowledge transfer between language models with different tokenizers. It constructs a shared support of token and token-sequence events, using a residual event to capture unmatched probability mass. The resulting distributions are matched with a unified objective. Experiments across several teacher–student settings show improvements over existing cross-tokenizer distillation methods.

Critical strengths
  1. The idea of matching distributions over a common support is well motivated. It provides a unified way to handle different tokenizers while preserving more fine-grained information than matching only the observed sequence.
  2. The experiments cover several teacher–student combinations and model scales, with generally positive results. The top-k experiments are also useful, showing that much of the performance can be retained at a lower computational cost.
Critical weaknesses
  1. The methodological difference from ALM seems relatively modest. Both methods rely on chain-rule probabilities and a residual event to handle unmatched probability mass. CSD mainly extends the binary matching in ALM to multiple candidate events. While this is a reasonable improvement, the underlying formulation remains largely similar to ALM.
  2. Table 9 shows that around 90–98% of aligned chunks are one-to-one, but Table 3 only studies support selection for non-one-to-one chunks. It is therefore unclear how much of the improvement actually comes from the expanded support for one-to-one chunks, which appears to be an important part of the method.
  3. Section 4.1 states that X-Token uses JSD, although its original P-KL and H-KL objectives are based on KL divergence. The paper also adopts a different alignment strategy from the original X-Token and ALM implementations. These changes may be reasonable for a controlled comparison, but their effects on baseline performance should be clarified.
Questions for authors
  1. Could the authors separately ablate the expanded support for one-to-one and non-one-to-one chunks? This would help clarify which part of the support construction contributes most to the performance gains.
  2. Could the authors clarify how X-Token and ALM are implemented, particularly the divergence functions and alignment strategies? Were the original configurations also evaluated?
2 comments
AuthorsOpenReview ↗

Response to Reviewer TPBo (Part 1/2)

We thank the reviewer for the careful and constructive review, and for finding matching over a common support well motivated, as a unified way to handle different tokenizers that preserves more fine-grained information than matching only the observed sequence. Below, we report the separate ablation of 1:1 and non-1:1 support (Comment 1), specify how X-Token and ALM are implemented and report results with their original configurations (Comment 2), and state exactly how CSD differs from ALM and how much each difference contributes (Comment 3). All changes will be incorporated in the revision.

Comment 1: separate ablations of 1:1 and non-1:1 support (W2, Q1)

We thank the reviewer for this suggestion. To separate the two effects, we changed the support of one chunk type at a time on Qwen→Llama and Phi→Llama, starting from binary support (the realized event plus the residual) for both 1:1 and non-1:1 chunks. All rows report Avg-5 at step 30K. On Qwen→Llama:

1:1 supportNon-1:1 supportAvg-5
BinaryBinary32.93
Common pairsBinary33.58
Common pairs1:N and M:1 alternatives (default)33.84
Common pairsAll alternatives, including M:N (full)33.69

On Phi→Llama, the same comparison gives:

1:1 supportNon-1:1 supportAvg-5
BinaryBinary34.82
Common pairsBinary35.22
Common pairs1:N and M:1 alternatives (default)35.84
Common pairsAll alternatives, including M:N (full)35.53
  • 1:1 support. Holding the non-1:1 support at binary, switching 1:1 chunks from binary support to common pairs improves Avg-5 by 0.65 points on Qwen→Llama and by 0.40 on Phi→Llama.
  • Non-1:1 support. Holding the 1:1 support at common pairs, adding the 1:N and M:1 alternatives raises Avg-5 by 0.26 points on Qwen→Llama and by 0.62 on Phi→Llama. Also adding the M:N alternatives (full support) gains less, 0.11 and 0.31 points, so the default scores highest on both pairs.

To answer the reviewer’s question directly: both parts contribute, and which contributes most depends on the pair, the 1:1 support on Qwen→Llama and the non-1:1 alternatives on Phi→Llama. Together, the expanded support adds about one point on both pairs (32.93 → 33.84 and 34.82 → 35.84).

Comment 2: relation to ALM and implementation of the baselines

We thank the reviewer for this question. All cross-tokenizer methods in Tables 1 and 2 share the same offset alignment, frozen teachers, the same data and update budget, and the same detached balancing of cross-entropy and distillation, so that only the comparison of teacher and student predictions within a chunk varies. The revision will state the following details explicitly.

Adapted X-Token loss. We use X-Token’s P-KL projection and H-KL token map exactly as in the original paper. Only two things differ:

  • Different alignment algorithm. X-Token aligns teacher and student sequences with dynamic programming (DP), whereas we use the offset alignment shared by all methods.
  • Different divergence. JSD (β = 0.5) replaces X-Token’s forward KL in P-KL on Qwen→Llama and on the common pairs of H-KL.

Adapted ALM loss. We keep ALM’s binary cross-entropy between teacher and student chunk likelihoods. Some differences are:

  • Chunk temperature. We use a chunk temperature of 10, where ALM’s default is 100.
  • No debiasing. ALM’s aligner places chunk boundaries that match ours on over 99.9% of cuts in all four pairs, then drops boundaries affected by tokenization bias, removing about 9% of them on Qwen→Llama. We skip this step, so that ALM is trained on the same chunks as all other methods.
  • No optional auxiliary losses. We use neither GradMag nor the hidden-state loss, both of which are optional in ALM.

Results with the original configurations. We retrained X-Token in its original configurations and ALM with its original settings (chunk temperature 100 and debiasing). All results are Avg-5 at step 30K with the Llama-3.2-1B student:

Method (pair)Adapted (Table 1 run)Original configuration, retrained
X-Token P-KL (Qwen→Llama)32.4632.18
X-Token H-KL (Phi→Llama)35.1834.28
ALM (Qwen→Llama)31.9231.41

In every case, the original configuration scores below our adaptation, by 0.28, 0.90 and 0.51 points, so our adaptations do not weaken these baselines.

AuthorsOpenReview ↗

Response to Reviewer TPBo (Part 2/2)

Comment 3: relation to ALM

We agree that the two methods share their building blocks: both use chain-rule probabilities over aligned chunks and a residual event for unmatched mass, and CSD with a single explicit event per chunk is ALM-like binary support. CSD differs from ALM in two ways:

  • Distilling dark knowledge through multi-event support. CSD matches a distribution over multiple candidate events for both 1:1 and non-1:1 chunks, with a residual event preserving normalization. This transfers the teacher’s relative preferences among alternatives—the dark knowledge central to knowledge distillation. In contrast, ALM’s binary objective supervises only the teacher’s confidence in the realized chunk (gold-truth) versus its complement, without distinguishing among alternative events
  • Objective and weighting. CSD matches normalized per-chunk distributions with a chosen divergence (JSD in our experiments, without a chunk temperature) and up-weights non-1:1 chunks (β=5), whereas ALM applies binary cross-entropy with a chunk temperature.

How much these differences matter. In Table 1, CSD outperforms adapted ALM by 1.92 points on Qwen→Llama (33.84 vs. 31.92) and by 3.30 points on Phi→Llama (35.84 vs. 32.54). The ablation in Comment 1 splits this margin between the two differences. With binary support for every chunk, CSD keeps ALM’s support but uses its own objective and weighting; this alone scores 1.01 and 2.28 points above ALM. Expanding the support to CSD’s default adds a further 0.91 and 1.02 points. Both differences therefore contribute substantially. The gap also grows with scale: for the 8B base student, CSD improves on ALM by 2.61 and 4.23 points (Table 4).

We thank the reviewer again for the constructive review, which has made both the comparison with ALM and the support ablation more informative.

Reviewer_1qvtOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces how to match the tokenizer in the knowledge distillation scenario. Basically, this algorithm calculates the possible matches and their teacher/student likelihood of matched sequences. After the matching, the typical knowledge distillation divergence/metric is being applied.

Critical strengths

I learned that the tokenizer level matching in the knowledge distillation is being discussed and porposed.

Critical weaknesses

The claim that CSD “surpasses same-tokenizer distillation” is essentially coming from two effects, better teacher or no match error. The same-tokenizer teachers are weaker than the cross-tokenizer teachers, and same-tokenizer KD does not much improve the performance, i.e. CPT. Therefore, we really cannot see the cost of tokenizer mismatch. We need experiments to fixate the teacher and vary only the student tokenizer. Or, we can use a same-family teacher of comparable strength.

The nature of method is marginally relevant to the knowledge distillation. This method does not change any knowledge distillation diversity or metric or learning structure. Rather, this change the matching input to the knowledge distillation. Hence, the matching algorithm is the focus, not the knowledge distillation.

The expanded event space is the paper’s main idea. Table 3 is the only comparisons across the virtual support for non-one-to-one chunks, where the gain is 0.26–0.62 points. The common-token-pair support for one-to-one chunks, which covers 90–98% of all chunks (Table 9), is never ablated. Potentially, where does the gain from Table 1/2 may mainly come from?

In introduction, P-KL and H-KL need to be defined or referenced before you use the term by acronym.

In probability, "||" has a very specific meaning, i.e. in KL divergence notation, so using "||" for the concatenation is very confusing.

Questions for authors

see my critical weakness section. particularly, "The expanded event space is the paper’s main idea. Table 3 is the only comparisons across the virtual support for non-one-to-one chunks, where the gain is 0.26–0.62 points. The common-token-pair support for one-to-one chunks, which covers 90–98% of all chunks (Table 9), is never ablated. Potentially, where does the gain from Table 1/2 may mainly come from? " this is utmost important.

98% are one-to-one matching case. What would be the leverage of the performance improvement from the token sequence matching?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1qvt (Part 1/2)

We thank the reviewer for the careful reading and the precise questions. We address each point below with new experiments, all of which will be incorporated in the revision.

Comment 1: same-tokenizer comparison with teachers of comparable strength

We agree with the reviewer. Tables 1 and 2 mix two opposing effects: the cross-family teachers are much stronger (Table 11), which favors CSD, while same-tokenizer KD incurs no matching error, which favors it. With weak same-family teachers, these results alone cannot show the cost of tokenizer mismatch. Since the reviewer’s first suggestion, varying only the student’s tokenizer, would require re-learning the student’s embeddings and thus change the student itself, we followed the second: to approximately control for teacher strength, we distilled each student from a stronger same-family teacher, using the same forward-KL token-level KD as the same-tokenizer rows (Avg-5 at step 30K):

StudentSame-family teacher (teacher Avg-5)Same-tokenizer KDCSD, Qwen3-4B-Base teacher (64.94)CSD, Phi-4-mini teacher (60.25)
Llama-3.2-1BLlama-3.2-3B (39.59), Table 130.9933.8435.84
Llama-3.2-1BLlama-3.1-8B (51.12)31.8633.8435.84
Gemma3-1BGemma3-4B-PT (45.87), Table 226.5427.7128.98
Gemma3-1BGemma3-12B-PT (62.13)27.3027.7128.98

Gemma3-12B-PT is of comparable strength to the cross-family teachers (62.13, between 60.25 and 64.94), yet same-tokenizer KD from it (27.30) still trails CSD, slightly with Qwen3-4B-Base (by 0.41 points) and clearly with Phi-4-mini (by 1.68), mostly on GSM8K (10.16 vs. 15.62), where both cross-family teachers are much stronger (84.61 and 82.71 vs. 71.27). On Llama, the 8B teacher raises same-tokenizer KD only to 31.86, still 1.98 and 3.98 points below CSD. One possible reason is that the students were already distilled from these or similar models during pretraining: Llama-3.2-1B used logits from Llama-3.1-8B and 70B as token-level targets [1], and Gemma 3 models were also pretrained with knowledge distillation [2].

What this says about the cost of mismatch. We acknowledge that this comparison does not fully separate the two effects, since the teachers still differ in training data and task profile. It does show that, in these settings, the matching cost does not outweigh the advantages of the cross-family teachers, which are stronger on tasks such as GSM8K and may provide knowledge the students did not already receive during pretraining. In CSD, this cost is confined to teacher mass outside the explicit support, which is matched in aggregate through the residual rather than token by token.

Scope of the claim. We do not claim that cross-tokenizer distillation is always better than same-tokenizer distillation. We will revise the abstract and Section 4 to state that CSD from the evaluated cross-family teachers outperforms same-tokenizer KD from the evaluated same-family teachers, including teachers of comparable strength, and add the table above. More generally, cross-tokenizer distillation is useful in two situations. First, a cross-family teacher can be much stronger at the same size and distillation cost: at about 4B parameters, Qwen3-4B-Base and Phi-4-mini score 19 and 14 points above Gemma3-4B-PT, and 25 and 21 points above Llama-3.2-3B (Table 11). Second, for the largest model of a family, or one with its own tokenizer, no larger same-family teacher exists. Our experiments correspond to the first situation, and the comparison above shows that CSD’s advantage persists when the same-family teacher is of comparable strength.

[1] Llama 3.2 Model Card. https://huggingface.co/meta-llama/Llama-3.2-1B, 2024. Released September 25, 2024.

[2] Gemma 3 Technical Report. arXiv preprint arXiv:2503.19786, 2025.

Comment 2: relation to knowledge distillation

We agree that CSD does not propose a new divergence; this is by design and is common in cross-tokenizer distillation. When teacher and student use different tokenizers, they predict over different outcome spaces, so deciding which distributions to compare is the central problem. The literature therefore focuses on matching rather than on new losses: ALM compares the likelihoods of aligned text chunks, X-Token projects between the two vocabularies, and GOLD matches tokens with identical strings, each then applying a standard loss such as KL, JSD or binary cross-entropy. CSD follows the same line: it constructs normalized, directly comparable teacher and student distributions for every chunk shape, to which any standard divergence can be applied. We will state this framing explicitly in the introduction.

Comment 3: presentation

We thank the reviewer for the suggestion. P-KL and H-KL will be defined at their first use in the introduction, and we will write concatenation with ⊕\oplus instead of ∥∥\|\| to avoid confusion with divergence notation in the revision.

AuthorsOpenReview ↗

Response to Reviewer 1qvt (Part 2/2)

Comment 4: where the gain comes from, ablating 1:1 and non-1:1 support

We thank the reviewer for marking this question as the most important; we agree. To answer it, we changed the support of one chunk type at a time on Qwen→Llama and Phi→Llama, starting from binary support (the realized event plus the residual) for both 1:1 and non-1:1 chunks. All rows report Avg-5 at step 30K. On Qwen→Llama:

1:1 supportNon-1:1 supportAvg-5
BinaryBinary32.93
Common pairsBinary33.58
Common pairs1:N and M:1 alternatives (default)33.84
Common pairsAll alternatives, including M:N (full)33.69

On Phi→Llama, the same comparison gives:

1:1 supportNon-1:1 supportAvg-5
BinaryBinary34.82
Common pairsBinary35.22
Common pairs1:N and M:1 alternatives (default)35.84
Common pairsAll alternatives, including M:N (full)35.53
  • 1:1 support, which the reviewer noted had not been ablated Holding the non-1:1 support at binary, switching 1:1 chunks from binary support to common pairs improves Avg-5 by 0.65 points on Qwen→Llama (32.93 → 33.58) and by 0.40 on Phi→Llama (34.82 → 35.22).
  • Non-1:1 support. Holding the 1:1 support at common pairs, adding the 1:N and M:1 alternatives raises Avg-5 by 0.26 points on Qwen→Llama (33.58 → 33.84) and by 0.62 on Phi→Llama (35.22 → 35.84). Also adding the M:N alternatives (full support) gains less, 0.11 and 0.31 points (33.69 and 35.53), so the default scores highest on both pairs.

Where the gains over the baselines in Tables 1 and 2 come from. With binary support everywhere, CSD’s support reduces to ALM’s, yet its objective and chunk weighting alone already score 1.01 and 2.28 points above ALM (Table 1). The expanded support adds a further 0.91 and 1.02 points, making up CSD’s total margin over ALM (1.92 and 3.30). On Phi→Llama, it is also what places CSD ahead of X-Token (35.18) and GOLD (35.00): binary-only CSD scores 34.82 but CSD with expanded support rises to 35.84. The revision will replace Table 3 with this ablation.

Comment 5: leverage of the non-1:1 chunks

We thank the reviewer for this question. In short, the non-1:1 chunks (about 2–3% of chunks for the Llama student and 10% for Gemma; Table 9) have little leverage by count, the weight β sets how much they receive, and they matter most on numerals.

Share of the loss. Using the training loss code itself, we scored the default CSD configuration at its step-30K checkpoint on 2,048 training sequences, weighting the non-1:1 chunks with β = 1 and with the default β = 5:

PairNon-1:1 chunksTheir share of the KD loss, β=1Their share of the KD loss, β=5Digits in non-1:1 chunks
Qwen→Llama1.7%1.1%5.1%51%
Phi→Llama2.8%0.9%4.3%0%
Qwen→Gemma9.4%3.3%14.5%0%
Phi→Gemma10.6%3.8%16.5%52%

With β = 1, the non-1:1 chunks carry less of the distillation loss than their share of chunks; with β = 5, they carry 1.5–3 times their share. Their leverage concentrates on numerals when one tokenizer writes numbers digit by digit and the other groups up to three digits (Qwen→Llama, Phi→Gemma). About half of all digits then fall in non-1:1 chunks, and on GSM8K solutions for Qwen→Llama, the 11% of chunks that are non-1:1 carry 40% of the β-weighted loss and 78% of the digits. When both tokenizers group digits alike (Phi→Llama, Qwen→Gemma), no digits fall in them.

Effect of β on accuracy. Training Qwen→Llama with different values of β shows where this leverage goes (Avg-5 at step 30K, Llama-3.2-1B student; Table 12’s runs plus β = 10, which we add to the revision). “Math” averages GSM8K and MATH; “Other” averages MMLU, WinoGrande and MBPP.

βMMLUWGGSM8KMATHMBPPMathOtherAvg-5
145.2462.3519.717.1230.0013.4245.8632.88
5 (default)45.2261.8021.308.2632.6014.7846.5433.84
1045.4762.9022.528.5031.4015.5146.5934.16
2045.0362.5123.208.8230.0016.0145.8533.91

From β = 1 to β = 20, the math average rises at every step, by 2.59 points in all, while the average of the other three tasks ends where it started (45.86 → 45.85). The same pattern appears across tokenizer pairs: CSD’s GSM8K advantage over the best baseline on that task is largest for the two pairs whose tokenizers split numbers differently, +5.23 points on Qwen→Llama and +3.26 on Phi→Gemma, compared with +0.45 on both other pairs (Tables 1–2). The non-1:1 chunks are therefore few, but they carry most of the distillation signal on numerals where tokenizers disagree.The revision adds the leverage measurements and the β results, and keeps β configurable.

We thank the reviewer again for the critical review, which has made the paper’s claims more precise.

Reviewer_PVrhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CSD constructs corresponding events over aligned text spans for cross-tokenizer distillation. It matches common tokens for one-to-one chunks, expands completions for one-to-many and many-to-one chunks with observed prefixes fixed, and retains the observed event for many-to-many chunks. Chain probabilities and a residual yield normalized distributions for a shared divergence. Experiments cover four 1B teacher-student pairs, larger students, support selection, and logit truncation.

Critical strengths
  1. Explicit paired events and a residual allow the same distribution-matching objective to cover several alignment cases across tokenizer pairs without selecting different loss modes. Chain probabilities preserve supervision beyond the first token, and the support construction makes the computational approximation explicit (§3; Appendix C).

  2. CSD achieves the highest average in all four 1B settings (Tables 1 to 2), with gains also reported for larger students. For the 8B base student, it improves the four-task average over ALM by 2.61 and 4.23 points with two teachers. The instruction-tuned experiment also improves the reported aggregate (Tables 4 to 5). In the evaluated setting, the top-128 configuration retains almost all full-support accuracy and reduces measured training-step time from 7.45 to 3.66 seconds (§4.4; Appendix E.3).

Critical weaknesses
  1. The adapted baseline losses need a precise specification. Section 4.1 states that all cross-tokenizer methods use the same offset alignment and that GOLD, X-Token, and CSD use JSD. Table 10 describes the published formulations: GOLD combines matched-token KL with unmatched-entry L1, while X-Token has projection and hybrid objectives. Which terms does JSD replace, and which unmatched components, normalizations, and coefficients remain? Standardizing alignment and divergence provides a controlled comparison. Its numerical claims apply to the adapted implementations, whose relationship to the published methods should be explicit, especially given the approximately 0.7-point margins in two 1B settings.

  2. Table 3 keeps the one-to-one construction fixed and finds gains of 0.26 and 0.62 average points from expanding non-one-to-one support, without reporting variation across training runs. This ablation uses Llama, whose non-one-to-one chunks are substantially less frequent than Gemma's (Table 9). The modest gains support the chosen policy in the tested settings, but leave its stability and importance in other settings uncertain. This limitation concerns support expansion specifically; the evidence for CSD as a complete method is stronger.

Questions for authors
  1. Please specify the exact adapted GOLD and X-Token losses, including replaced terms and retained coefficients. If original-loss results are already available, they would help contextualize the comparison; otherwise, please make the scope of the numerical claims explicit.
  2. Is uncertainty information or a Gemma support-selection comparison already available to help interpret Table 3? If neither is available, please qualify the attribution to support expansion. Stating this limitation does not require additional large-scale runs.
2 comments
AuthorsOpenReview ↗

Response to Reviewer PVrh (Part 1/2)

We thank the reviewer for the careful and encouraging review, and for highlighting that explicit paired events with a residual let one objective cover several alignment cases without per-pair loss modes. Below, we specify exactly how each adapted baseline differs from its published loss and report results with X-Token’s original configurations (Comment 1), and we report seed variation and a Gemma comparison for the support ablation (Comment 2). All changes will be incorporated in the revision.

Comment 1: exact adapted losses for GOLD and X-Token (W1, Q1)

Shared protocol for adapted losses. All cross-tokenizer methods in Tables 1 and 2 use the same offset alignment, which splits teacher and student sequences into aligned chunks covering the same text. They also share the same data and update budget, and the same detached balancing of cross-entropy and distillation, L=LCE+sg(LCE/LKD)⋅LKDL = L_{CE} + sg(L_{CE} / L_{KD}) · L_{KD}.

Adapted GOLD loss. We use the hybrid objective of GOLD's reference implementation (TRL GOLDTrainer, v1.13.0) with its default loss settings, so no term is replaced. It uses generalized JSD (β=0.5\beta=0.5) on matched tokens and rank-sorted L1 on unmatched tokens, weighted by C/VTC/V_T and 1 − C/VTC/V_T, where CC is the number of matched teacher tokens and VTV_T the teacher vocabulary size. The differences in our adapted version are:

  • Offset units. TRL groups tokens by byte offsets and can split a multi-byte UTF-8 character (e.g., a CJK character or ≠) across groups; we use character offsets, which keep such characters whole. On the Llama pairs, the two groupings agree on 99.9% of groups.
  • Extension to the Gemma student. TRL’s alignment does not support SentencePiece tokenizers such as Gemma’s, and literal string matching finds only 24,520 and 31,384 common pairs because the tokenizers encode spaces differently. We match decoded text instead (App. A.2), giving 94,932 and 130,704 pairs. For the Llama student, string matching reproduces TRL’s vocabulary intersection exactly.

For the Llama student, the only difference from the reference implementation is therefore the offset unit, which affects 0.1% of groups. As a result, we think the GOLD rows of Table 1 could already reflect GOLD’s original loss. Thus we did not retrain it separately but we will clarify these differences in the revision.

Adapted X-Token loss. We use X-Token’s matching exactly as formulated in the original paper: P-KL projects the student distribution onto the teacher vocabulary with the same projection matrix, and H-KL uses the same fixed map of common tokens, extended with first-sub-token matches, and compares the remaining tokens with rank-sorted L1. Only two things differ here:

  • Different alignment algorithm. X-Token aligns teacher and student sequences with dynamic programming (DP), whereas we use the offset alignment shared by all methods.
  • Different divergence. JSD (β=0.5\beta = 0.5) replaces X-Token’s forward KL in P-KL on Qwen→Llama and on the common pairs of H-KL.

Results with the original X-Token losses. To measure what these differences change, we retrained X-Token in its original configurations, keeping our data, update budget and evaluation (Avg-5 at step 30K, Llama-3.2-1B student):

Method (pair)Original configurationTable 1 (adapted)Original configuration, retrained
X-Token P-KL (Qwen→Llama)DP alignment, forward KL32.4632.18
X-Token H-KL (Phi→Llama)DP alignment, forward KL35.1834.28

Both original configurations score below our adapted versions, by 0.28 points for P-KL and 0.90 for H-KL, so our adaptations do not disadvantage X-Token. Against the original configurations, CSD’s margin is 1.66 points on Qwen→Llama and 1.56 on Phi→Llama, where the adapted comparison in Table 1 gives 0.66. Section 4.1 will state that all numerical comparisons refer to these implementations under one shared protocol.

AuthorsOpenReview ↗

Response to Reviewer PVrh (Part 2/2)

Comment 2: uncertainty and a Gemma comparison for Table 3 (W2, Q2)

We thank the reviewer for this suggestion. We now report both: variation across seeds on the Llama pairs, and a comparison on Gemma, where non-1:1 chunks are more frequent.

Variation across seeds. We trained the realized-only and proposed non-1:1 support policies with two additional seeds on both Llama pairs (Avg-5 at step 30K):

Non-1:1 supportQwen→Llama, seeds 1 / 2 / 3Mean ± SDPhi→Llama, seeds 1 / 2 / 3Mean ± SD
Realized-only33.58 / 33.24 / 33.2533.36 ± 0.1935.22 / 35.42 / 35.5835.41 ± 0.18
Proposed33.84 / 33.34 / 33.8833.69 ± 0.3035.84 / 36.15 / 36.0136.00 ± 0.16

Averaged over three seeds, the proposed support outperforms realized-only on both pairs, by 0.33 points on Qwen→Llama and 0.59 points on Phi→Llama, and it improves on realized-only in every seed on both pairs. With seed SDs of 0.16–0.30 points, the Phi→Llama gap is about three seed SDs, whereas the Qwen→Llama gap is within seed variation.

Gemma. Multiple seeds are not yet available for Gemma. For Phi→Gemma, where about 10% of chunks are non-1:1 (Table 9), a single run of realized-only support scores 0.71 points below the proposed policy and lower on all five tasks:

Non-1:1 support (Phi→Gemma)MMLUWinoGrandeGSM8KMATHMBPPAvg-5
Realized-only40.5361.3314.864.8419.8028.27
Proposed (Table 2)41.2861.6415.625.7620.6028.98

In summary, expanding the support for non-1:1 chunks improves on realized-only support across all three pairs: the gain clearly exceeds seed variation on Phi→Llama, is consistent in direction on Qwen→Llama, and appears on all five tasks for Gemma. The revision will report the seed variation in Table 3 and state this evidence per setting in Section 4.3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of reviews and responses

We thank the Program Chairs, Senior Area Chairs, and Area Chairs for their time and effort in coordinating the review process, and all three reviewers for their careful reading and constructive feedback. The reviewers’ comments helped us separate the contributions of the two parts of CSD’s support, specify the baseline implementations precisely and compare them with their original configurations, and test CSD against stronger same-family teachers and across training seeds. We have prepared point-by-point responses with new experiments.

The reviewers agree that CSD gives a unified objective across tokenizer pairs without per-pair loss modes (PVrh, TPBo), that matching over a common support is well motivated (TPBo), that CSD achieves the best average in all four 1B settings with gains for larger students (PVrh), with generally positive results across teacher–student combinations and scales (TPBo), and that top-k truncation retains most of the accuracy at lower cost (PVrh, TPBo). Below we summarize the main concerns and the new evidence.

  1. Where the gains come from, and the relation to ALM (1qvt, TPBo). A new ablation separates the two parts of CSD’s support. Expanding 1:1 chunks to common-token pairs adds 0.65 and 0.40 Avg-5 points on the two Llama pairs, and adding 1:N and M:1 alternatives adds a further 0.26 and 0.62, showing that the default CSD configuration performs best on both pairs. The same ablation splits CSD’s margin over ALM (1.92 and 3.30 points): CSD’s objective and chunk weighting contribute 1.01 and 2.28 points, and its multi-event support, which transfers the teacher’s preferences among alternatives, a further 0.91 and 1.02. On Phi→Llama, the expanded support is also what places CSD ahead of X-Token and GOLD.

  2. Baseline implementations (PVrh, TPBo). We specify exactly how each baseline differs from its published configuration. For the Llama student, GOLD uses its original loss and coefficients. Apart from the cross-entropy balancing shared by all methods, its only difference from the reference implementation is the offset unit (characters vs. bytes), which affects 0.1% of groups. Retrained with their original losses, X-Token and ALM both score below our adaptations (X-Token: 32.18 vs. 32.46 and 34.28 vs. 35.18; ALM: 31.41 vs. 31.92). Against the original X-Token configurations, CSD’s margins widen from 1.38 and 0.66 points to 1.66 and 1.56 points on Qwen→Llama and Phi→Llama.

  3. Robustness of the support ablation (PVrh). Over three seeds, the non-1:1 support improves Avg-5 by 0.33 and 0.59 points, in every seed on both pairs, clearly exceeding seed variation on Phi→Llama. On Phi→Gemma, where such chunks are about 10%, a single run shows a gain of 0.71 points.

  4. Same-tokenizer comparison (1qvt). To approximately control for teacher strength, we distilled each student from a stronger same-family teacher. With Gemma3-12B-PT, of comparable strength to the cross-family teachers, same-tokenizer KD stays 0.41 and 1.68 points below CSD. With Llama-3.1-8B, it stays 1.98 and 3.98 points below. This does not fully isolate the cost of tokenizer mismatch, but it shows that the cost does not outweigh the cross-family teachers’ advantages in these settings. We will scope the claim to the evaluated teachers.

  5. Leverage of non-1:1 chunks (1qvt). These chunks carry 1.5–3 times their share of chunks in the distillation loss, and about half of all digits for tokenizer pairs that split numbers differently. Accordingly, CSD’s GSM8K advantage over the best baseline is largest for exactly these pairs (+5.23 points on Qwen→Llama and +3.26 points on Phi→Gemma, against +0.45 for the other two). On Qwen→Llama, the math average rises steadily with the chunk weight β, by 2.59 points from β = 1 to 20.

We have also clarified how CSD relates to the knowledge distillation literature and fixed the notation (1qvt). Overall, the new results show that both parts of CSD’s support contribute to its gains, that our baseline adaptations do not weaken the baselines, and that CSD’s advantage persists against stronger same-family teachers. All changes will be incorporated in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.