acceptodds
Under review as a conference paper at ICLR 2027

Where the Prediction Goes Matters in Token-Level Knowledge Distillation

Abstract

Token-level knowledge distillation transfers a teacher’s next-token distribution to a smaller student. Common distillation objectives measure how closely the student matches the teacher’s next-token distribution at each prediction position. However, a smaller student cannot in general reproduce the teacher distribution exactly at every position, raising a question: when the student redistributes probability mass relative to the teacher, should the destination of this mass matter? In particular, should equal amounts of this mass be treated the same when they go to different tokens? We argue that their significance should depend on how the teacher relates those tokens across a population of contexts. To model these relations, we represent each token by a teacher-derived distribution over contexts. At each prediction position, we use the teacher and student next-token distributions to define two mixtures of the teacher-derived context distributions and compare them with the Cauchy–Schwarz divergence. We combine this divergence with a standard probability-matching objective to introduce residual composition distillation (RCD). Across two teacher–student model families and multiple evaluation sets, RCD consistently improves prediction quality and generation diversity, with a trade-off in generation quality.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Knowledge Distillation, Token-Level Language Model Distillation, Context Mixtures, Cauchy–Schwarz Divergence

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rWWUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a residual combination distillation (RCD) method to address the limitation of conventional token-level knowledge distillation, which matches probability distributions at each position without distinguishing how probability mass is redistributed by the student. RCD introduces a measure of semantic relationships between tokens. Specifically, it uses the teacher model to construct a contextual relationship matrix between tokens and introduces a combination divergence based on the Cauchy–Schwarz inequality. By jointly optimizing the token-level matching loss and the distributional combination loss, RCD preserves the teacher's knowledge while improving the student's prediction quality and generation diversity.

Critical strengths
  1. The paper points out that conventional token-level distillation focuses on matching the probabilities of corresponding tokens while overlooking their contextual relationships. The authors use an example to illustrate the importance of probability redistribution and clearly motivate the research problem.
  2. RCD combines the token-level matching loss with a combination divergence based on the Cauchy–Schwarz inequality. While preserving local prediction alignment, it models contextual relationships between tokens. This joint optimization design extends the distillation objective beyond individual token probability matching to capture the overall combinatorial properties of probability distributions.
Critical weaknesses
  1. In the introduction, the paper uses an example in which two student models assign different probabilities to cat, kitten, and car to illustrate the importance of probability redistribution. However, whether assigning a higher probability to car than to kitten by Student B reflects realistic language modeling distributions remains debatable. For example, car is generally an unlikely continuation in the context “She adopted a.” It is therefore unclear whether this example reflects a common problem in real models or an idealized assumption. In addition, the paper does not sufficiently explain how constructing a token relationship matrix from the teacher's predictions across contexts differs fundamentally from existing relation-aware knowledge distillation methods or what unique advantages it offers.
  2. The experiments provide insufficient validation of the method's core mechanisms. The paper uses gradient visualization (Fig. 2) and teacher-derived Gram matrix visualization (Fig. 3) to illustrate the effects of RCD on the optimization process and token relationship modeling. However, these results are insufficient to fully validate the proposed mechanisms. Specifically, Fig. 2 shows that the combination divergence changes the gradient direction, but this change alone does not demonstrate that the optimization process is improved, as other regularization terms could also alter the gradients. The key question is whether this change leads the student model to better solutions in actual language modeling tasks. Furthermore, Fig. 3 mainly illustrates the structure and interpretability of the teacher-derived Gram matrix but does not verify whether these relationships actually help the student model assign probability mass to more appropriate tokens.
Questions for authors
  1. The authors are encouraged to further discuss whether the idealized token probability distributions reflect the differences between teacher and student models in real language modeling scenarios, and to empirically verify the prevalence of the problem described in the introduction. They should also compare the proposed teacher-conditioned contextual relationship matrix with those constructed by other methods and add a discussion of related approaches, such as CSD, in the introduction to further clarify RCD's unique advantages and improvements over existing methods.

  2. The authors are encouraged to conduct targeted ablation studies to further validate the proposed mechanisms, such as replacing the teacher-derived relationship matrix with an identity or uniform matrix to verify the role of actual contextual relationships. In addition, they should analyze gradient changes, training convergence, and changes in actual token probability distributions to determine whether the combination divergence guides the student model toward more reasonable predictions. This would provide stronger evidence linking the proposed mechanism to the final performance improvements.

  3. The authors are encouraged to further analyze the trade-off between generation quality and diversity in RCD, with a focus on explaining the decrease in ROUGE-L. They should also clarify the applicable scenarios and limitations of the method to better assess whether RCD can improve prediction performance and generation diversity while keeping the negative impact on generation quality within a reasonable range.

  4. The experiments mainly focus on distillation from 1.3B/1.5B teacher models to 100M student models, which differs substantially from current distillation settings involving large language models with 7B or more parameters. Therefore, the current results are insufficient to fully validate the scalability of the method. Moreover, the teacher-derived Gram matrix K\mathit{K} used by RCD requires full-vocabulary predictions from the teacher model over sampled contexts. Its computational cost, memory requirements, and training stability with large vocabularies and large models remain unverified, leaving the generalizability of the main conclusions uncertain. The authors are encouraged to conduct experiments with larger teacher–student models, such as 7B→1B, or analyze the computational complexity, storage overhead, and practical feasibility of constructing the K\mathit{K} matrix for large vocabularies to provide stronger evidence for RCD's scalability.

6 comments
AuthorsOpenReview ↗

Response to reviewer rWWU (1/3)

Weakness 1: Realism of the Motivation and Methodological Novelty.

Realism: We agree that, in language modeling, car is generally a less plausible continuation than kitten in the given context, and our example is illustrative. Nevertheless, the two student predictions in our example yield the same sample-level distillation loss. More generally, multiple distinct categorical predictions can lead to the same positive distillation loss, while the loss value alone cannot distinguish their different contextual implications.

Methodological Novelty: We provide a detailed explanation in our response to Q1.

Weakness 2: Insufficient Validation of the Proposed Mechanisms.

The concerns are addressed in detail in our response to Q2.

Question 1. Motivation and Novelty of the Proposed Formulation.

Motivation: The introductory failure case illustrates a more general limitation of conventional token-level distillation, where the categorical token representation does not explicitly model contextual relationships among vocabulary tokens.

Methodological Novelty:

Our contribution follows five steps:

(i) Phenomenon: Two student predictions can have the same categorical discrepancy from the teacher while assigning probability mass to tokens with different contextual relationships. We illustrate this phenomenon through an extreme case in Figure 1, which serves as our starting motivation.

(ii) Problem: Categorical token representations do not explicitly capture contextual relationships among tokens. Consequently, standard token-based distillation losses do not explicitly account for these relationships when comparing teacher and student predictions.

(iii) Modeling: Inspired by the conditional probabilistic modeling of neighbors in SNE, we use the teacher’s conditional predictions and Bayes’ rule to characterize each token through the contexts that support it. We model tokens through their contextual distributions. This is the central modeling step of our approach.

(iv) Comparison: We construct context mixtures from these conditional distributions, changing the object of comparison from categorical predictions to their induced context mixtures. We then use the Cauchy–Schwarz divergence to compare these mixtures in the L2(PX)L^2(P_X) space.

(v) Formulation: The comparison in L2(PX)L^2(P_X) yields a tractable quadratic-form objective involving a Gram matrix KK, whose entries arise naturally from inner products of the teacher-conditioned contextual representations. Thus, KK is a consequence of our conditional probabilistic modeling and the choice of Cauchy–Schwarz divergence.

Discussion of CSD: Both CSD and RCD involve comparisons across tokens, but their methodological differences can be understood at three levels:

(i) Comparison Target: CSD compares teacher and student predictions through pairwise log-probability ratios within each prediction position. In contrast, RCD changes the object of comparison from categorical predictions to their induced context mixtures.

(ii) Contextual Modeling: For efficient analytic computation, CSD uses pairwise weights that factorize over position-specific token probabilities, without explicitly incorporating relationships estimated across contexts. RCD instead uses the teacher’s conditional predictions and Bayes’ rule to characterize each token through its supporting contexts.

(iii) Composition and Optimization: CSD matches relative logit differences using weighted squared errors. RCD compares the induced mixtures using the Cauchy–Schwarz divergence in L2(PX)L^2(P_X), allowing the comparison to account for teacher-derived contextual relationships among tokens. This composition term complements sample-level probability matching.

We have also revised the related work in Section 2.1 accordingly.

Limitation and Empirical Prevalence.

Figure 1 presents an extreme case illustrating a more general limitation of sample-level categorical divergences, such as KL. These divergences do not explicitly account for contextual relationships among tokens. When the discrepancy is nonzero, different student predictions can yield the same discrepancy while differing in their contextual relationships to the teacher’s prediction. Our approach introduces teacher-conditioned contextual information to distinguish such cases.

AuthorsOpenReview ↗

Response to reviewer rWWU (2/3)

2. Validation of the Proposed Mechanism.

To assess the role of teacher-derived contextual relationships, we vary KK while keeping all other training settings and seeds fixed: (i) Identity (K=IK=I) removes cross-token relationships; (ii) Permutation (Kperm=PKP⊤K_{\mathrm{perm}}=PKP^\top) changes their correspondence to token identities while preserving the entries and spectrum of KK, using one fixed permutation shared across the three training seeds; and (iii) Uniform (K=11⊤K=\mathbf{1}\mathbf{1}^\top) makes the composition divergence identically zero, reducing the objective to a scaled SRKL loss. We therefore do not evaluate this case separately.

Compared with the teacher-derived KK, the identity matrix increases diversity but reduces ROUGE-L, Oracle@5, and both BERTScore-F1 metrics, with comparable NLL and ECE. The permuted matrix reduces diversity and increases NLL and ECE, with comparable generation quality. These results suggest that teacher-derived contextual relationships help balance generation diversity and quality.

MethodNLL ↓ECE ↓ROUGE-L ↑Oracle@5 ↑BERTScore-F1 mean ↑BERTScore-F1 min@5 ↑S-BLEU ↓Dist-2 ↑
SRKL6.52±0.0290.3240±0.002919.35±0.4925.40±0.420.838±0.00080.822±0.00170.424±0.0140.555±0.012
RCD (SRKL)4.41±0.8580.2445±0.034618.01±0.2925.22±0.830.837±0.00110.818±0.00280.213±0.0210.755±0.005
RCD (SRKL), K=IK=I4.38±0.4470.2352±0.017616.74±0.0924.12±0.040.835±0.00020.815±0.00020.171±0.0070.807±0.002
RCD (SRKL), KpermK_{\mathrm{perm}}5.02±0.1670.2783±0.005718.20±0.2625.14±0.360.838±0.00060.819±0.00140.270±0.0190.695±0.012

We further examine lexical coverage using existing sampled generations on the same prompts under identical decoding settings. For each evaluation set, we construct vocabularies from five sampled generations per model. We report (i) the student-to-teacher vocabulary size ratio; (ii) type recall, the fraction of teacher token types produced by the student; (iii) mass recall, the fraction of teacher token occurrences covered by the student vocabulary; (iv) precision, the fraction of student token types observed in teacher generations; and (v) Jensen–Shannon (JS) divergence between their token frequency distributions. Results are averaged equally across four evaluation sets and reported as mean ± standard deviation over three training seeds.

MethodStudent / TeacherType recall ↑Teacher-mass recall ↑Precision ↑JS ↓
SRKL0.633±0.0250.464±0.0210.881±0.0080.725±0.0050.146±0.013
RCD (SRKL)0.875±0.0160.582±0.0070.920±0.0030.662±0.0050.113±0.003
RCD (SRKL), K=IK=I1.061±0.0140.651±0.0090.936±0.0030.614±0.0060.110±0.004
RCD (SRKL), KpermK_{\mathrm{perm}}0.794±0.0210.546±0.0120.910±0.0040.684±0.0050.128±0.005

The lexical coverage analysis further supports SRKL's mode-seeking tendency. Adding the composition divergence expands the student's vocabulary from 63% to 88% of the teacher's observed vocabulary size, increases teacher-token recall, and reduces unigram JS divergence. While K=IK=I further expands vocabulary coverage, it reduces precision and generation quality.

Together, these results suggest that the composition divergence enables a richer student vocabulary, while teacher-derived contextual relationships help balance generation diversity and quality. We have also included these results in a new subsection in our revised manuscript.

AuthorsOpenReview ↗

Response to reviewer rWWU (3/3)

3. Trade-off Between Generation Quality and Diversity.

ROUGE-L measures lexical overlap between generated and reference texts. By incorporating contextual relationships, RCD may allow more diverse expressions, which can reduce lexical overlap with the reference and thus lower ROUGE-L scores.

However, BERTScore (mean and min@5) and Oracle@5 remain stable or improve in some settings, suggesting that lower lexical overlap does not necessarily imply a corresponding decline in semantic quality.

This also motivates our use of complementary evaluation metrics beyond ROUGE-L. RCD is therefore particularly suitable for settings where semantic quality and generation diversity are valued over exact lexical agreement with the reference.

We further evaluate generation quality using Qwen2.5-7B-Instruct as an LLM judge. For GPT-2 and OPT, we compare all paired baselines and RCD counterparts. For each dataset, we select up to 100 prompts using a fixed random seed. The judge compares two anonymized responses given the instruction and reference answer(s), assessing correctness, relevance, coherence, and completeness. Each pair is evaluated in both response orders.

DatasetRCD Win ↑TieRCD Loss ↓Win − LossConsistency
DollyEval3.2 ± 0.593.2 ± 0.53.6 ± 0.5−0.362%
SelfInst2.2 ± 1.094.2 ± 1.03.7 ± 1.0−1.561%
VicunaEval0.3 ± 0.399.6 ± 0.40.1 ± 0.2+0.187%
S-NI4.4 ± 0.988.9 ± 1.26.7 ± 0.6−2.247%
Overall2.5 ± 0.494.0 ± 0.23.5 ± 0.2−1.064%

Win, Tie, and Loss rates (%) are reported from RCD's perspective, averaged across paired comparisons and training seeds. A win or loss requires both response orders to favor the same model; all other outcomes are counted as ties.

Both win and loss rates are low across all four datasets, with 94% of comparisons showing no consistent preference under the two-order evaluation protocol. Overall, the LLM judge shows only a small preference for the baselines under our two-order evaluation protocol.

4. Scalability and Computational Feasibility.

To evaluate scalability, we conduct an additional experiment with OLMo-7B-SFT as the teacher and OLMo-1B as the student (6.9B → 1.2B), trained for 5 epochs using the same objective, data, and evaluation protocol, with three training seeds per method. The results show broadly consistent trends with our smaller-scale experiments: RCD slightly improves NLL, ECE, and generation diversity while maintaining comparable generation quality. However, the magnitude of these differences is considerably smaller at this scale. Notably, the SRKL baseline already exhibits substantially greater diversity than in our GPT-2 and OPT experiments (Self-BLEU: 0.12 vs. 0.42 and 0.29), leaving less room for improvement.

MethodNLL ↓ECE ↓ROUGE-L ↑Oracle@5 ↑BERTScore-F1 mean ↑BERTScore-F1 min@5 ↑S-BLEU ↓Dist-2 ↑
SRKL3.297±0.0190.2736±0.001815.24±0.2721.99±0.420.827±0.00140.809±0.00260.120±0.0080.821±0.006
RCD (SRKL)3.244±0.1030.2691±0.014314.89±0.2121.86±0.370.825±0.00050.805±0.00210.112±0.0100.827±0.017

Moreover, we report the computational complexity, storage requirements, and training overhead.

We represent each token-conditioned context distribution as a vector in RN\mathbb{R}^N, where NN is the number of sampled contexts. The resulting Gram matrix K=ΦΦ⊤/NK=\Phi\Phi^\top/N, with Φ∈Rm×N\Phi\in\mathbb{R}^{m\times N}, is never explicitly constructed or stored. Instead, we compute the required inner products directly through Φ\Phi, avoiding the O(m2)O(m^2) storage cost of the full matrix. Constructing Φ\Phi requires teacher predictions over NN sampled contexts, with O(Nm)O(Nm) storage and preprocessing complexity linear in NN for fixed vocabulary size mm.

During training, we restrict the composition divergence to the union of the teacher’s and student’s top-100 tokens at each position and compute the required projections using the corresponding rows of Φ\Phi. This avoids full-vocabulary matrix operations. In our GPT-2 experiments on one A100 (40 GB), the factor-based implementation requires approximately 493 seconds per epoch, compared with 461 seconds for the SRKL baseline (wall-clock, including checkpointing and the per-epoch development evaluation), with peak GPU memory increasing from 15.9 GB to 21.4 GB. Importantly, the contextual factor dimensions depend on vocabulary size and the number of sampled contexts, rather than model parameter count, allowing the method to scale to our 7B-to-1B setting without increasing the size of the contextual representation.

Reviewer_ivZ5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is about token-level knowledge distillation. For distilling, it represents each token by a teacher-derived distribution over contexts. and then, at each prediction position, the teacher and student next-token distributions are used to define two mixtures of the teacher-derived context distributions. These distributions are compared with the Cauchy-Schwarz divergence. So the proposed method of Residual Composition Distillation (RCD), considers relationships between tokens across contexts instead of simply matching teacher and student next-token probabilities.

Critical strengths

The paper presents a simple and general framework that can be easily integrated with existing token-level distillation objectives.

The paper demonstrates consistent improvements in prediction quality and generation diversity across multiple teacher-student settings.

Critical weaknesses

I am not sure if the novelty of this work is enough. The primary novelty appears to be the use of a context-based token relation matrix. But it is not sufficient to distinguish the method from existing approaches at a conceptual level. So I think the technical contribution is a bit incremental.

I am not sure if I understand Figure 2 well. The difference between CE and KL is relatively clear, whereas KL and RCD appear highly similar. Would be great if the authors could explain this.

It is unclear why context similarity is chosen over embedding similarity, hidden-state similarity, or logit similarity. I can't find enough justification in the paper for why context similarity should be a more effective way to model token relationships during distillation.

The choice of CSD as the composition loss a bit heuristic to me. The paper does not provide sufficient justification for why CSD is preferable to other divergence measures. I guess including an ablation study to support this design choice would be helpful.

Questions for authors

Please see Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to reviewer ivZ5

1. Clarification of the Novelty and Main Contribution.

We would like to clarify that our main contribution is the conditional probabilistic formulation, not the introduction of a token relation matrix.

I. Motivation.

As illustrated by our failure case, two student predictions can have the same discrepancy from the teacher but differ substantially in their semantic consequences. This motivates distinguishing predictions beyond their discrepancy magnitudes.

II. Conditional modeling: From tokens to contexts.

Our approach is to characterize tokens through teacher-induced conditional context distributions P(C∣I=i)P(C\mid I=i). For example, given “I travel to”, “London” is a more plausible continuation than “elephant”. This motivates representing tokens through the contexts in which they are likely to occur. Our conditional modeling also shares intuition with neighborhood-based approaches such as Stochastic Neighbor Embedding (SNE).

III. A change in the object of comparison.

We introduce context mixtures as a new comparison target for teacher and student predictions:

\qquad Q_q(C)=\sum_i q_i P(C\mid I=i).$$ Comparing these mixtures allows probability redistributions to be evaluated through their induced contextual relationships. *IV. The Gram matrix as a consequence.* Applying the Cauchy–Schwarz divergence to these context mixtures involves inner products in $L^2$. By bilinearity, these inner products can be expressed using a teacher-induced Gram matrix $K$. The Gram matrix thus arises naturally from the teacher-conditioned context distributions and their inner products in $L^2$ when applying the Cauchy–Schwarz divergence. **2. Clarification in Figure 2.** In this illustrative example, only cat and kitten have a high off-diagonal score in $K$, while their scores with car are zero. Comparing Figures 2(b)–(d), we observe three differences in the gradient fields: (i) Along the cat–kitten direction, RCD produces weaker gradients toward cat. This effect becomes more pronounced at ($\eta$=0.9), where the gradient magnitudes from kitten toward cat are further reduced. (ii) In the interior of the simplex, RCD gradients shift toward the cat–kitten region, whereas KL gradients point more directly toward the teacher prediction. (iii) Over much of the interior car-heavy region, RCD produces stronger gradients away from the car vertex, although these gradients weaken again near the vertex. Together, these observations illustrate flatter gradients between contextually related tokens and stronger gradients away from unrelated tokens in this illustrative example. **3. Justification for Contextual Similarity.** Our primary focus is not the choice of a similarity measure, but the conditional probabilistic modeling of contextual distributions for tokens based on the teacher’s next-token predictions. Let $X$ denote a discrete context variable and $Y$ the next-token outcome. Given a context $X=x$, a neural network $f$ parameterizes the conditional distribution $P(Y\mid X=x)$ through its softmax predictions, with $f(x)\in\Delta^{m-1}$, where $\Delta^{m-1}$ is the probability simplex and $m$ is the vocabulary size. Given a context distribution $P(X)$, the joint distribution is $$P(X=x,Y=y)=P(Y=y\mid X=x)P(X=x).$$ By Bayes' rule, we obtain $$P(X=x\mid Y=y)= \frac{P(Y=y\mid X=x)P(X=x)}{P(Y=y)},$$ where $P(Y=y)=\sum_{x\in\mathcal X}P(Y=y\mid X=x)P(X=x)$. Thus, our choice follows from conditional probabilistic modeling of the teacher’s softmax predictions, where token-conditioned context distributions arise naturally through Bayes’ rule. This formulation uses the teacher’s predictions to characterize each token through its conditional distribution over discrete contexts. **4. Motivation for the Cauchy–Schwarz Divergence.** Cauchy–Schwarz Divergence is a choice motivated by a property of $L^2$ inner products: the inner product between two mixtures can be expressed through pairwise inner products of their components. This allows contextual relationships between components to be naturally captured through their inner products in $L^2$. Consider two mixtures over a common domain $\Omega$: $$f(x)=\sum_{i=1}^{m}a_i f_i(x), \qquad g(x)=\sum_{i=1}^{m}b_i f_i(x), \qquad x\in\Omega,$$ where $f_i\in L^2(\Omega)$ are shared components and $a,b$ are mixture weights. Their $L^2$ inner product satisfies $$\langle f,g\rangle_{L^2} = \int_{\Omega} f(x)g(x)\,dx = \int_{\Omega}\left(\sum_{i=1}^{m}a_i f_i(x)\right)\left(\sum_{j=1}^{m}b_j f_j(x)\right)\,dx = \sum_{i=1}^{m}\sum_{j=1}^{m}a_i b_j G_{ij} = a^\top G b,$$ where $G_{ij}=\int_{\Omega}f_i(x)f_j(x)\,dx$. Similarly, $$\lVert f\rVert_{L^2}^2 = a^\top G a, \qquad \lVert g\rVert_{L^2}^2 = b^\top G b.$$ In contrast, commonly used divergences such as KL and general $\alpha$-divergences do not directly admit this bilinear decomposition for mixtures.
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer ivZ5,

Thank you again for your constructive comments. We would appreciate hearing whether our rebuttal and the revised manuscript address your main concerns. If any concerns remain or any of our responses would benefit from further clarification, we’d be happy to discuss them.

Sincerely,

The Authors

AuthorsOpenReview ↗

Kind Reminder on Our Rebuttal

Dear Reviewer ivZ5,

As the discussion period approaches its end, we would like to kindly follow up on our rebuttal and revised manuscript. We remain happy to address any outstanding questions or concerns.

Thank you again for your time and consideration.

Sincerely,

The Authors

Reviewer_qj27OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes RCD, a token-level knowledge distillation objective that accounts for where the student redistributes probability mass relative to the teacher. Each vocabulary token is represented by a teacher-derived distribution over training contexts. At each prediction position, teacher and student next-token probabilities define two mixtures of these context distributions, which are compared using the Cauchy–Schwarz divergence. The resulting composition loss is computed through a teacher-derived Gram matrix and combined with a standard probability-matching objective. Experimental results show improvements in prediction quality and generation diversity, accompanied by a degradation of generation quality

Critical strengths
  1. The motivation is clear that when mismatch exists between teacher and student token-level distributions, the probability should be preferably redistributed to semantically acceptable tokens.
  2. The proposed probabilistic method is principled and sound.
  3. The paper is clear and easy to follow.
Critical weaknesses
  1. The scale and breadth of experiments are limited. Only two small scale teacher-student pairs are tested.
  2. Despite some other benefits, the arguably most important metric — generation quality — degrades consistently.
Questions for authors
  1. When empirically estimating the marginal context distribution, only one prefix is selected from each usable example based on one valid response-token position. Why not selecting multiple positions/prefixes per example? Why not selecting prefixes for all unique response tokens?
  2. Why is it necessary to normalize P(C∣I=i)P(C|I=i) to fix its perplexity over ii?
5 comments
AuthorsOpenReview ↗

Response to reviewer qj27

Weakness 1. The scale and breadth of experiments are limited.

To evaluate scalability, we conduct an experiment with OLMo-7B-SFT as the teacher and OLMo-1B as the student (6.9B → 1.2B).

The results show trends consistent with our smaller-scale experiments. RCD achieves slightly lower NLL and ECE, alongside modest improvements in generation diversity. Generation quality remains comparable, with slightly lower ROUGE-L and BERTScore-F1. The improvements are smaller at this scale, potentially because the SRKL baseline already exhibits substantially greater diversity than in our GPT-2 and OPT experiments.

MethodNLL ↓ECE ↓ROUGE-L ↑Oracle@5 ↑BERTScore-F1 mean ↑BERTScore-F1 min@5 ↑S-BLEU ↓Dist-2 ↑
SRKL3.297±0.0190.2736±0.001815.24±0.2721.99±0.420.827±0.00140.809±0.00260.120±0.0080.821±0.006
RCD (SRKL)3.244±0.1030.2691±0.014314.89±0.2121.86±0.370.825±0.00050.805±0.00210.112±0.0100.827±0.017

Weakness 2. The generation quality degrades consistently.

We acknowledge a trade-off in generation quality. Generation quality does not consistently degrade across the reported metrics. While ROUGE-L decreases across the reported settings, Oracle@5 and BERTScore remain relatively stable or improve in several cases. These metrics capture different aspects of generation quality. ROUGE-L measures lexical and sequential overlap with references. Increased generation diversity can introduce expressions that reduce this overlap.

We further evaluate generation quality using Qwen2.5-7B-Instruct as an LLM judge. For GPT-2 and OPT, we compare all paired baselines and RCD counterparts. For each dataset, we select up to 100 prompts using a fixed random seed. The judge compares two anonymized responses given the instruction and reference answer(s), assessing correctness, relevance, coherence, and completeness. Each pair is evaluated in both response orders.

DatasetRCD Win ↑TieRCD Loss ↓Win − LossConsistency
DollyEval3.2 ± 0.593.2 ± 0.53.6 ± 0.5−0.362%
SelfInst2.2 ± 1.094.2 ± 1.03.7 ± 1.0−1.561%
VicunaEval0.3 ± 0.399.6 ± 0.40.1 ± 0.2+0.187%
S-NI4.4 ± 0.988.9 ± 1.26.7 ± 0.6−2.247%
Overall2.5 ± 0.494.0 ± 0.23.5 ± 0.2−1.064%

Win, Tie, and Loss rates (%) are reported from RCD's perspective, averaged across paired comparisons and training seeds. A win or loss requires both response orders to favor the same model; all other outcomes are counted as ties.

Both win and loss rates are low across all four datasets, with 94% of comparisons showing no consistent preference under the two-order evaluation protocol. Overall, the LLM judge shows only a small preference for the baselines under our two-order evaluation protocol.

Q1. Justification for Sampling One Prefix per Record.

Our sampling operates on the context space XX used to construct Pϕ(X∣Y=y)P_\phi(X\mid Y=y). Each sampled context X=xX=x provides a teacher prediction over the entire vocabulary. Therefore, sampling unique observed tokens is not necessary.

Using all response-token prefixes would substantially increase the dimensionality of the probabilistic representation and computational cost. For example, Dolly-15k contains approximately 1.5×1041.5\times10^4 records. With an average of LL valid positions per record, using all prefixes would require approximately 1.5×104L1.5\times10^4L contexts.

We therefore randomly select one valid response-token position per record, providing equal representation across records while reducing the number of sampled contexts. This is a practical choice balancing context coverage and computational efficiency.

Q2. Why is it necessary to normalize P(C∣I=i)P(C | I=i) to fix its perplexity over i?

Fixing perplexity is not a theoretical requirement of our probabilistic formulation. Instead, fixed perplexity provides an interpretation in terms of effective contextual support.

Our conditional context modeling is inspired by the Stochastic Neighbor Embedding (SNE), and our formulation models a conditional distribution over contexts given each token. In both cases, perplexity characterizes the effective support of the conditional distribution.

In our setting, consider the discrete case where each token is uniformly supported by a subset of contexts. For example, “London” may be supported by “We travel to” and “We live in”, while “movie” may be supported by “We watch a” and “We discuss a”. In this case, perplexity corresponds exactly to the number of supporting contexts. Fixing perplexity therefore provides a common effective support size across tokens, allowing their inner products to reflect contextual overlap on a comparable basis.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer qj27,

Thank you again for your constructive comments. We would appreciate hearing whether our rebuttal and the revised manuscript address your main concerns. If any concerns remain or any of our responses would benefit from further clarification, we’d be happy to discuss them.

Sincerely,

The Authors

AuthorsOpenReview ↗

Kind Reminder on Our Rebuttal

Dear Reviewer qj27,

As the discussion period approaches its end, we would like to kindly follow up on our rebuttal and revised manuscript. We remain happy to address any outstanding questions or concerns.

Thank you again for your time and consideration.

Sincerely,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response: Clarification of Methodological Novelty

We thank the reviewers for their constructive comments. We clarify our methodological novelty through the following steps, from teacher-conditioned probabilistic modeling to context-mixture comparison and its Gram-matrix formulation.

1. Failure Case and Motivation

Our starting observation is that, in the extreme case illustrated in Figure 1, two student predictions can assign the same probability to the correct token and have the same KL divergence from the teacher, while assigning probability to tokens with substantially different contextual relationships to the correct token.

More generally, categorical predictions assign probabilities to discrete token identities without explicitly representing contextual relationships among tokens. Consequently, conventional categorical discrepancies do not explicitly account for these relationships when comparing imperfect student predictions.

This motivates our central question: How can contextual distinctions among tokens be incorporated into the comparison of teacher and student predictions in token-based knowledge distillation?

2. Teacher-Conditioned Probabilistic Modeling

We first represent each token through the contexts in which it is predicted. Given the teacher’s conditional distribution Pϕ(Y∣X)P_\phi(Y\mid X) and the marginal context distribution PXP_X, Bayes’ rule gives

\frac{P_\phi(Y=y\mid X)P_X(X)}{P_\phi(Y=y)},$$ where $$P_\phi(Y=y) = \sum_{x\in\mathcal X} P_\phi(Y=y\mid X=x)P_X(x).$$ The corresponding density ratio with respect to $P_X$ is $$h_y(x)= \frac{dP_\phi(X\mid Y=y)}{dP_X}(x)= \frac{P_\phi(Y=y\mid X=x)} {P_\phi(Y=y)}.$$ Our first contribution is the conditional probabilistic modeling of contextual distributions for discrete tokens, based on the teacher’s predictive behavior. **3. A New Object of Comparison: Context Mixtures** We then change the object of comparison in knowledge distillation. Let $w\in\Delta^{V-1}$ denote a categorical prediction from either the teacher or the student. It induces a context mixture $$P_w(X) = \sum_y w_y P_\phi(X\mid Y=y),$$ with density ratio $$r_w(x)= \sum_y w_y h_y(x).$$ For student and teacher predictions $p$ and $q$, respectively, the corresponding density ratios are $$r_p(x)=\sum_y p_y h_y(x), \qquad r_q(x)=\sum_y q_y h_y(x).$$ Our second contribution is to introduce context mixtures as a new comparison target for teacher and student predictions. These mixtures naturally incorporate teacher-derived contextual relationships among tokens. **4. Cauchy–Schwarz Divergence and the Gram-Matrix Formulation** We compare the induced mixtures using the Cauchy–Schwarz divergence in $L^2(P_X)$: $$D_{\mathrm{CS}}(r_p,r_q) = -\log \frac{\langle r_p,r_q\rangle_{P_X}^{2}}{\lVert r_p\rVert_{P_X}^{2}\lVert r_q\rVert_{P_X}^{2}}.$$ Since the mixtures are linear combinations of conditional density ratios, their inner products take a bilinear form: $$\langle r_p,r_q\rangle_{P_X}= \sum_{i,j}p_iq_j \langle h_i,h_j\rangle_{P_X}= p^\top Kq,$$ where $$K_{ij}= \mathbb{E}_{X\sim P_X} \left[h_i(X)h_j(X)\right].$$ Consequently, $$D_{\mathrm{CS}}(r_p,r_q)= -\log \frac{(p^\top Kq)^2} {(p^\top Kp)(q^\top Kq)}.$$ Applying the Cauchy–Schwarz divergence to the induced context mixtures involves inner products in $L^2(P_X)$. By bilinearity, these inner products take the form of bilinear and quadratic expressions involving the Gram matrix $K$, which arises naturally from the teacher-conditioned contextual distributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.