- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Common-Support Distillation (CSD) for knowledge transfer between language models with different tokenizers. It constructs a shared support of token and token-sequence events, using a residual event to capture unmatched probability mass. The resulting distributions are matched with a unified objective. Experiments across several teacher–student settings show improvements over existing cross-tokenizer distillation methods.
Critical strengths
- The idea of matching distributions over a common support is well motivated. It provides a unified way to handle different tokenizers while preserving more fine-grained information than matching only the observed sequence.
- The experiments cover several teacher–student combinations and model scales, with generally positive results. The top-k experiments are also useful, showing that much of the performance can be retained at a lower computational cost.
Critical weaknesses
- The methodological difference from ALM seems relatively modest. Both methods rely on chain-rule probabilities and a residual event to handle unmatched probability mass. CSD mainly extends the binary matching in ALM to multiple candidate events. While this is a reasonable improvement, the underlying formulation remains largely similar to ALM.
- Table 9 shows that around 90–98% of aligned chunks are one-to-one, but Table 3 only studies support selection for non-one-to-one chunks. It is therefore unclear how much of the improvement actually comes from the expanded support for one-to-one chunks, which appears to be an important part of the method.
- Section 4.1 states that X-Token uses JSD, although its original P-KL and H-KL objectives are based on KL divergence. The paper also adopts a different alignment strategy from the original X-Token and ALM implementations. These changes may be reasonable for a controlled comparison, but their effects on baseline performance should be clarified.
Questions for authors
- Could the authors separately ablate the expanded support for one-to-one and non-one-to-one chunks? This would help clarify which part of the support construction contributes most to the performance gains.
- Could the authors clarify how X-Token and ALM are implemented, particularly the divergence functions and alignment strategies? Were the original configurations also evaluated?
2 comments
Response to Reviewer TPBo (Part 2/2)
Comment 3: relation to ALM
We agree that the two methods share their building blocks: both use chain-rule probabilities over aligned chunks and a residual event for unmatched mass, and CSD with a single explicit event per chunk is ALM-like binary support. CSD differs from ALM in two ways:
- Distilling dark knowledge through multi-event support. CSD matches a distribution over multiple candidate events for both 1:1 and non-1:1 chunks, with a residual event preserving normalization. This transfers the teacher’s relative preferences among alternatives—the dark knowledge central to knowledge distillation. In contrast, ALM’s binary objective supervises only the teacher’s confidence in the realized chunk (gold-truth) versus its complement, without distinguishing among alternative events
- Objective and weighting. CSD matches normalized per-chunk distributions with a chosen divergence (JSD in our experiments, without a chunk temperature) and up-weights non-1:1 chunks (β=5), whereas ALM applies binary cross-entropy with a chunk temperature.
How much these differences matter. In Table 1, CSD outperforms adapted ALM by 1.92 points on Qwen→Llama (33.84 vs. 31.92) and by 3.30 points on Phi→Llama (35.84 vs. 32.54). The ablation in Comment 1 splits this margin between the two differences. With binary support for every chunk, CSD keeps ALM’s support but uses its own objective and weighting; this alone scores 1.01 and 2.28 points above ALM. Expanding the support to CSD’s default adds a further 0.91 and 1.02 points. Both differences therefore contribute substantially. The gap also grows with scale: for the 8B base student, CSD improves on ALM by 2.61 and 4.23 points (Table 4).
We thank the reviewer again for the constructive review, which has made both the comparison with ALM and the support ablation more informative.
Response to Reviewer TPBo (Part 1/2)
We thank the reviewer for the careful and constructive review, and for finding matching over a common support well motivated, as a unified way to handle different tokenizers that preserves more fine-grained information than matching only the observed sequence. Below, we report the separate ablation of 1:1 and non-1:1 support (Comment 1), specify how X-Token and ALM are implemented and report results with their original configurations (Comment 2), and state exactly how CSD differs from ALM and how much each difference contributes (Comment 3). All changes will be incorporated in the revision.
Comment 1: separate ablations of 1:1 and non-1:1 support (W2, Q1)
We thank the reviewer for this suggestion. To separate the two effects, we changed the support of one chunk type at a time on Qwen→Llama and Phi→Llama, starting from binary support (the realized event plus the residual) for both 1:1 and non-1:1 chunks. All rows report Avg-5 at step 30K. On Qwen→Llama:
On Phi→Llama, the same comparison gives:
To answer the reviewer’s question directly: both parts contribute, and which contributes most depends on the pair, the 1:1 support on Qwen→Llama and the non-1:1 alternatives on Phi→Llama. Together, the expanded support adds about one point on both pairs (32.93 → 33.84 and 34.82 → 35.84).
Comment 2: relation to ALM and implementation of the baselines
We thank the reviewer for this question. All cross-tokenizer methods in Tables 1 and 2 share the same offset alignment, frozen teachers, the same data and update budget, and the same detached balancing of cross-entropy and distillation, so that only the comparison of teacher and student predictions within a chunk varies. The revision will state the following details explicitly.
Adapted X-Token loss. We use X-Token’s P-KL projection and H-KL token map exactly as in the original paper. Only two things differ:
Adapted ALM loss. We keep ALM’s binary cross-entropy between teacher and student chunk likelihoods. Some differences are:
Results with the original configurations. We retrained X-Token in its original configurations and ALM with its original settings (chunk temperature 100 and debiasing). All results are Avg-5 at step 30K with the Llama-3.2-1B student:
In every case, the original configuration scores below our adaptation, by 0.28, 0.90 and 0.51 points, so our adaptations do not weaken these baselines.