- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a residual combination distillation (RCD) method to address the limitation of conventional token-level knowledge distillation, which matches probability distributions at each position without distinguishing how probability mass is redistributed by the student. RCD introduces a measure of semantic relationships between tokens. Specifically, it uses the teacher model to construct a contextual relationship matrix between tokens and introduces a combination divergence based on the Cauchy–Schwarz inequality. By jointly optimizing the token-level matching loss and the distributional combination loss, RCD preserves the teacher's knowledge while improving the student's prediction quality and generation diversity.
Critical strengths
- The paper points out that conventional token-level distillation focuses on matching the probabilities of corresponding tokens while overlooking their contextual relationships. The authors use an example to illustrate the importance of probability redistribution and clearly motivate the research problem.
- RCD combines the token-level matching loss with a combination divergence based on the Cauchy–Schwarz inequality. While preserving local prediction alignment, it models contextual relationships between tokens. This joint optimization design extends the distillation objective beyond individual token probability matching to capture the overall combinatorial properties of probability distributions.
Critical weaknesses
- In the introduction, the paper uses an example in which two student models assign different probabilities to cat, kitten, and car to illustrate the importance of probability redistribution. However, whether assigning a higher probability to car than to kitten by Student B reflects realistic language modeling distributions remains debatable. For example, car is generally an unlikely continuation in the context “She adopted a.” It is therefore unclear whether this example reflects a common problem in real models or an idealized assumption. In addition, the paper does not sufficiently explain how constructing a token relationship matrix from the teacher's predictions across contexts differs fundamentally from existing relation-aware knowledge distillation methods or what unique advantages it offers.
- The experiments provide insufficient validation of the method's core mechanisms. The paper uses gradient visualization (Fig. 2) and teacher-derived Gram matrix visualization (Fig. 3) to illustrate the effects of RCD on the optimization process and token relationship modeling. However, these results are insufficient to fully validate the proposed mechanisms. Specifically, Fig. 2 shows that the combination divergence changes the gradient direction, but this change alone does not demonstrate that the optimization process is improved, as other regularization terms could also alter the gradients. The key question is whether this change leads the student model to better solutions in actual language modeling tasks. Furthermore, Fig. 3 mainly illustrates the structure and interpretability of the teacher-derived Gram matrix but does not verify whether these relationships actually help the student model assign probability mass to more appropriate tokens.
Questions for authors
-
The authors are encouraged to further discuss whether the idealized token probability distributions reflect the differences between teacher and student models in real language modeling scenarios, and to empirically verify the prevalence of the problem described in the introduction. They should also compare the proposed teacher-conditioned contextual relationship matrix with those constructed by other methods and add a discussion of related approaches, such as CSD, in the introduction to further clarify RCD's unique advantages and improvements over existing methods.
-
The authors are encouraged to conduct targeted ablation studies to further validate the proposed mechanisms, such as replacing the teacher-derived relationship matrix with an identity or uniform matrix to verify the role of actual contextual relationships. In addition, they should analyze gradient changes, training convergence, and changes in actual token probability distributions to determine whether the combination divergence guides the student model toward more reasonable predictions. This would provide stronger evidence linking the proposed mechanism to the final performance improvements.
-
The authors are encouraged to further analyze the trade-off between generation quality and diversity in RCD, with a focus on explaining the decrease in ROUGE-L. They should also clarify the applicable scenarios and limitations of the method to better assess whether RCD can improve prediction performance and generation diversity while keeping the negative impact on generation quality within a reasonable range.
-
The experiments mainly focus on distillation from 1.3B/1.5B teacher models to 100M student models, which differs substantially from current distillation settings involving large language models with 7B or more parameters. Therefore, the current results are insufficient to fully validate the scalability of the method. Moreover, the teacher-derived Gram matrix used by RCD requires full-vocabulary predictions from the teacher model over sampled contexts. Its computational cost, memory requirements, and training stability with large vocabularies and large models remain unverified, leaving the generalizability of the main conclusions uncertain. The authors are encouraged to conduct experiments with larger teacher–student models, such as 7B→1B, or analyze the computational complexity, storage overhead, and practical feasibility of constructing the matrix for large vocabularies to provide stronger evidence for RCD's scalability.
6 comments
Response to reviewer rWWU (2/3)
2. Validation of the Proposed Mechanism.
To assess the role of teacher-derived contextual relationships, we vary while keeping all other training settings and seeds fixed: (i) Identity () removes cross-token relationships; (ii) Permutation () changes their correspondence to token identities while preserving the entries and spectrum of , using one fixed permutation shared across the three training seeds; and (iii) Uniform () makes the composition divergence identically zero, reducing the objective to a scaled SRKL loss. We therefore do not evaluate this case separately.
Compared with the teacher-derived , the identity matrix increases diversity but reduces ROUGE-L, Oracle@5, and both BERTScore-F1 metrics, with comparable NLL and ECE. The permuted matrix reduces diversity and increases NLL and ECE, with comparable generation quality. These results suggest that teacher-derived contextual relationships help balance generation diversity and quality.
| Method | NLL ↓ | ECE ↓ | ROUGE-L ↑ | Oracle@5 ↑ | BERTScore-F1 mean ↑ | BERTScore-F1 min@5 ↑ | S-BLEU ↓ | Dist-2 ↑ |
|---|---|---|---|---|---|---|---|---|
| SRKL | 6.52±0.029 | 0.3240±0.0029 | 19.35±0.49 | 25.40±0.42 | 0.838±0.0008 | 0.822±0.0017 | 0.424±0.014 | 0.555±0.012 |
| RCD (SRKL) | 4.41±0.858 | 0.2445±0.0346 | 18.01±0.29 | 25.22±0.83 | 0.837±0.0011 | 0.818±0.0028 | 0.213±0.021 | 0.755±0.005 |
| RCD (SRKL), | 4.38±0.447 | 0.2352±0.0176 | 16.74±0.09 | 24.12±0.04 | 0.835±0.0002 | 0.815±0.0002 | 0.171±0.007 | 0.807±0.002 |
| RCD (SRKL), | 5.02±0.167 | 0.2783±0.0057 | 18.20±0.26 | 25.14±0.36 | 0.838±0.0006 | 0.819±0.0014 | 0.270±0.019 | 0.695±0.012 |
We further examine lexical coverage using existing sampled generations on the same prompts under identical decoding settings. For each evaluation set, we construct vocabularies from five sampled generations per model. We report (i) the student-to-teacher vocabulary size ratio; (ii) type recall, the fraction of teacher token types produced by the student; (iii) mass recall, the fraction of teacher token occurrences covered by the student vocabulary; (iv) precision, the fraction of student token types observed in teacher generations; and (v) Jensen–Shannon (JS) divergence between their token frequency distributions. Results are averaged equally across four evaluation sets and reported as mean ± standard deviation over three training seeds.
| Method | Student / Teacher | Type recall ↑ | Teacher-mass recall ↑ | Precision ↑ | JS ↓ |
|---|---|---|---|---|---|
| SRKL | 0.633±0.025 | 0.464±0.021 | 0.881±0.008 | 0.725±0.005 | 0.146±0.013 |
| RCD (SRKL) | 0.875±0.016 | 0.582±0.007 | 0.920±0.003 | 0.662±0.005 | 0.113±0.003 |
| RCD (SRKL), | 1.061±0.014 | 0.651±0.009 | 0.936±0.003 | 0.614±0.006 | 0.110±0.004 |
| RCD (SRKL), | 0.794±0.021 | 0.546±0.012 | 0.910±0.004 | 0.684±0.005 | 0.128±0.005 |
The lexical coverage analysis further supports SRKL's mode-seeking tendency. Adding the composition divergence expands the student's vocabulary from 63% to 88% of the teacher's observed vocabulary size, increases teacher-token recall, and reduces unigram JS divergence. While further expands vocabulary coverage, it reduces precision and generation quality.
Together, these results suggest that the composition divergence enables a richer student vocabulary, while teacher-derived contextual relationships help balance generation diversity and quality. We have also included these results in a new subsection in our revised manuscript.
Response to reviewer rWWU (3/3)
3. Trade-off Between Generation Quality and Diversity.
ROUGE-L measures lexical overlap between generated and reference texts. By incorporating contextual relationships, RCD may allow more diverse expressions, which can reduce lexical overlap with the reference and thus lower ROUGE-L scores.
However, BERTScore (mean and min@5) and Oracle@5 remain stable or improve in some settings, suggesting that lower lexical overlap does not necessarily imply a corresponding decline in semantic quality.
This also motivates our use of complementary evaluation metrics beyond ROUGE-L. RCD is therefore particularly suitable for settings where semantic quality and generation diversity are valued over exact lexical agreement with the reference.
We further evaluate generation quality using Qwen2.5-7B-Instruct as an LLM judge. For GPT-2 and OPT, we compare all paired baselines and RCD counterparts. For each dataset, we select up to 100 prompts using a fixed random seed. The judge compares two anonymized responses given the instruction and reference answer(s), assessing correctness, relevance, coherence, and completeness. Each pair is evaluated in both response orders.
| Dataset | RCD Win ↑ | Tie | RCD Loss ↓ | Win − Loss | Consistency |
|---|---|---|---|---|---|
| DollyEval | 3.2 ± 0.5 | 93.2 ± 0.5 | 3.6 ± 0.5 | −0.3 | 62% |
| SelfInst | 2.2 ± 1.0 | 94.2 ± 1.0 | 3.7 ± 1.0 | −1.5 | 61% |
| VicunaEval | 0.3 ± 0.3 | 99.6 ± 0.4 | 0.1 ± 0.2 | +0.1 | 87% |
| S-NI | 4.4 ± 0.9 | 88.9 ± 1.2 | 6.7 ± 0.6 | −2.2 | 47% |
| Overall | 2.5 ± 0.4 | 94.0 ± 0.2 | 3.5 ± 0.2 | −1.0 | 64% |
Win, Tie, and Loss rates (%) are reported from RCD's perspective, averaged across paired comparisons and training seeds. A win or loss requires both response orders to favor the same model; all other outcomes are counted as ties.
Both win and loss rates are low across all four datasets, with 94% of comparisons showing no consistent preference under the two-order evaluation protocol. Overall, the LLM judge shows only a small preference for the baselines under our two-order evaluation protocol.
4. Scalability and Computational Feasibility.
To evaluate scalability, we conduct an additional experiment with OLMo-7B-SFT as the teacher and OLMo-1B as the student (6.9B → 1.2B), trained for 5 epochs using the same objective, data, and evaluation protocol, with three training seeds per method. The results show broadly consistent trends with our smaller-scale experiments: RCD slightly improves NLL, ECE, and generation diversity while maintaining comparable generation quality. However, the magnitude of these differences is considerably smaller at this scale. Notably, the SRKL baseline already exhibits substantially greater diversity than in our GPT-2 and OPT experiments (Self-BLEU: 0.12 vs. 0.42 and 0.29), leaving less room for improvement.
| Method | NLL ↓ | ECE ↓ | ROUGE-L ↑ | Oracle@5 ↑ | BERTScore-F1 mean ↑ | BERTScore-F1 min@5 ↑ | S-BLEU ↓ | Dist-2 ↑ |
|---|---|---|---|---|---|---|---|---|
| SRKL | 3.297±0.019 | 0.2736±0.0018 | 15.24±0.27 | 21.99±0.42 | 0.827±0.0014 | 0.809±0.0026 | 0.120±0.008 | 0.821±0.006 |
| RCD (SRKL) | 3.244±0.103 | 0.2691±0.0143 | 14.89±0.21 | 21.86±0.37 | 0.825±0.0005 | 0.805±0.0021 | 0.112±0.010 | 0.827±0.017 |
Moreover, we report the computational complexity, storage requirements, and training overhead.
We represent each token-conditioned context distribution as a vector in , where is the number of sampled contexts. The resulting Gram matrix , with , is never explicitly constructed or stored. Instead, we compute the required inner products directly through , avoiding the storage cost of the full matrix. Constructing requires teacher predictions over sampled contexts, with storage and preprocessing complexity linear in for fixed vocabulary size .
During training, we restrict the composition divergence to the union of the teacher’s and student’s top-100 tokens at each position and compute the required projections using the corresponding rows of . This avoids full-vocabulary matrix operations. In our GPT-2 experiments on one A100 (40 GB), the factor-based implementation requires approximately 493 seconds per epoch, compared with 461 seconds for the SRKL baseline (wall-clock, including checkpointing and the per-epoch development evaluation), with peak GPU memory increasing from 15.9 GB to 21.4 GB. Importantly, the contextual factor dimensions depend on vocabulary size and the number of sampled contexts, rather than model parameter count, allowing the method to scale to our 7B-to-1B setting without increasing the size of the contextual representation.
Response to reviewer rWWU (1/3)
Weakness 1: Realism of the Motivation and Methodological Novelty.
Realism: We agree that, in language modeling, car is generally a less plausible continuation than kitten in the given context, and our example is illustrative. Nevertheless, the two student predictions in our example yield the same sample-level distillation loss. More generally, multiple distinct categorical predictions can lead to the same positive distillation loss, while the loss value alone cannot distinguish their different contextual implications.
Methodological Novelty: We provide a detailed explanation in our response to Q1.
Weakness 2: Insufficient Validation of the Proposed Mechanisms.
The concerns are addressed in detail in our response to Q2.
Question 1. Motivation and Novelty of the Proposed Formulation.
Motivation: The introductory failure case illustrates a more general limitation of conventional token-level distillation, where the categorical token representation does not explicitly model contextual relationships among vocabulary tokens.
Methodological Novelty:
Our contribution follows five steps:
(i) Phenomenon: Two student predictions can have the same categorical discrepancy from the teacher while assigning probability mass to tokens with different contextual relationships. We illustrate this phenomenon through an extreme case in Figure 1, which serves as our starting motivation.
(ii) Problem: Categorical token representations do not explicitly capture contextual relationships among tokens. Consequently, standard token-based distillation losses do not explicitly account for these relationships when comparing teacher and student predictions.
(iii) Modeling: Inspired by the conditional probabilistic modeling of neighbors in SNE, we use the teacher’s conditional predictions and Bayes’ rule to characterize each token through the contexts that support it. We model tokens through their contextual distributions. This is the central modeling step of our approach.
(iv) Comparison: We construct context mixtures from these conditional distributions, changing the object of comparison from categorical predictions to their induced context mixtures. We then use the Cauchy–Schwarz divergence to compare these mixtures in the L2(PX) space.
(v) Formulation: The comparison in L2(PX) yields a tractable quadratic-form objective involving a Gram matrix K, whose entries arise naturally from inner products of the teacher-conditioned contextual representations. Thus, K is a consequence of our conditional probabilistic modeling and the choice of Cauchy–Schwarz divergence.
Discussion of CSD: Both CSD and RCD involve comparisons across tokens, but their methodological differences can be understood at three levels:
(i) Comparison Target: CSD compares teacher and student predictions through pairwise log-probability ratios within each prediction position. In contrast, RCD changes the object of comparison from categorical predictions to their induced context mixtures.
(ii) Contextual Modeling: For efficient analytic computation, CSD uses pairwise weights that factorize over position-specific token probabilities, without explicitly incorporating relationships estimated across contexts. RCD instead uses the teacher’s conditional predictions and Bayes’ rule to characterize each token through its supporting contexts.
(iii) Composition and Optimization: CSD matches relative logit differences using weighted squared errors. RCD compares the induced mixtures using the Cauchy–Schwarz divergence in L2(PX), allowing the comparison to account for teacher-derived contextual relationships among tokens. This composition term complements sample-level probability matching.
We have also revised the related work in Section 2.1 accordingly.
Limitation and Empirical Prevalence.
Figure 1 presents an extreme case illustrating a more general limitation of sample-level categorical divergences, such as KL. These divergences do not explicitly account for contextual relationships among tokens. When the discrepancy is nonzero, different student predictions can yield the same discrepancy while differing in their contextual relationships to the teacher’s prediction. Our approach introduces teacher-conditioned contextual information to distinguish such cases.