acceptodds
Under review as a conference paper at ICLR 2027

Whether to Distill Across Modalities: An Exact Gain Identity and a Decision Rule

Abstract

Cross-modal knowledge distillation trains a student in one modality using a teacher in another, but teacher accuracy alone does not determine whether distillation helps. Although mixing teacher outputs with labels can reduce the effect of noise in the training labels, teacher outputs may also introduce variation unexplained by the student's input and alter the student's bias. To characterize this trade-off, we derive an exact gain identity for students with a fixed linear solution map under squared loss. The gain over supervised training on labels alone is quadratic in the mixing weight, and we identify conditions under which six scalars determine the gain curve and optimal weight. We further prove that criteria which ignore the student's labels do not determine whether distillation is beneficial. We therefore develop a gain estimator that reuses the student's existing labeled data together with unlabeled pairs to decide whether and how much to distill, without additional labels or a validation sweep. Experiments on six real-world multi-modal datasets show that existing distillation methods can degrade student performance, whereas our gain-based rule reduces that degradation and, in our main policy comparison, improves on supervised training on average.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
cross-modal knowledge distillation, knowledge distillation, multimodal learning, distillation decision rule, bias–variance decomposition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fVpfOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper looks at cross-modal knowledge distillation, where a student in one modality is trained on a mix of true labels and a teacher's outputs from another modality, and asks whether to distill and with what weight λ\lambda. For a linear student (e.g., ridge regression with a fixed penalty) under squared loss, the authors show that the gain over training on labels alone is exactly G(λ)=2λA−λ2DG(\lambda) = 2\lambda A - \lambda^2 D. Distillation therefore helps only when A>0A > 0, and the best weight is A/DA/D. They also give two toy examples showing that the teacher's accuracy, the label noise, or any criterion that ignores the student's labels cannot tell whether AA is positive. Based on this, they propose a rule that estimates AA and DD from the student's own labeled data plus unlabeled paired data. It does not distill when the estimate of AA is not positive, and otherwise uses a shrunk version of A/DA/D. On six multimodal datasets, mostly with frozen features and ridge or MLP students, a fixed λ=0.5\lambda = 0.5 often hurts, while the rule improves on not distilling by 0.89% of risk on average.

Critical strengths

The gain identity cleanly explains why teacher accuracy alone cannot decide whether cross-modal distillation helps.

Critical weaknesses
  1. Limited novelty. With a linear student, a risk quadratic in the mixing weight follows directly, and closely related analyses already exist. [1] gives a bias–variance view of when a teacher helps. [2] analyzes regularized linear regression with a weight ξ\xi between labels and teacher predictions, and derives the optimal ξ\xi under label noise. [3] gives a closed-form optimal weight between ground-truth targets and teacher outputs in kernel ridge regression, and estimates it without a grid search. [4] tunes a closed-form weight λ\lambda between labels and model predictions. [2] and [3] are not cited. Their setting is self-distillation rather than cross-modal distillation, but the core analysis is very similar.
  2. The theory holds only for a fixed λ\lambda; the deployed rule estimates λ\lambda from training data without guarantees.
  3. Gains are small: +0.89% over not distilling, and level with a simple validation sweep (p=0.554p = 0.554).

[1] A. K. Menon, A. S. Rawat, S. Reddi, S. Kim, and S. Kumar. A Statistical Perspective on Distillation. In ICML, PMLR 139:7632–7642, 2021.

[2] R. Das and S. Sanghavi. Understanding Self-Distillation in the Presence of Label Noise. In ICML, PMLR 202:7102–7140, 2023.

[3] K. Borup and L. N. Andersen. Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation. In NeurIPS, 2021.

[4] A. N. Angelopoulos, J. C. Duchi, and T. Zrnic. PPI++: Efficient Prediction-Powered Inference. arXiv:2311.01453, 2023.

Questions for authors
  1. What is new in Proposition 1 compared with [1] and [2], which also study the optimal label/teacher weight in (kernel) ridge regression?
  2. Why were 238 of the 1,040 candidate cells excluded? Why do the evaluation populations differ across Tables 1, 2, 3 and 15?

[1] R. Das and S. Sanghavi. Understanding Self-Distillation in the Presence of Label Noise. In ICML, PMLR 202:7102–7140, 2023.

[2] K. Borup and L. N. Andersen. Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation. In NeurIPS, 2021.

8 comments

We sincerely thank the reviewer for the careful reading of our manuscript and for the thoughtful and constructive evaluation. We particularly appreciate the reviewer's recognition that our gain identity cleanly explains why teacher accuracy alone cannot decide whether cross-modal knowledge distillation is beneficial. We provide detailed responses to the reviewer's questions and concerns below.


 

W1. Novelty relative to prior analyses (1/4)

Limited novelty. With a linear student, a risk quadratic in the mixing weight follows directly, and closely related analyses already exist. [1] gives a bias–variance view of when a teacher helps. [2] analyzes regularized linear regression with a weight ξ\xi between labels and teacher predictions, and derives the optimal ξ\xi under label noise. [3] gives a closed-form optimal weight between ground-truth targets and teacher outputs in kernel ridge regression, and estimates it without a grid search. [4] tunes a closed-form weight λ\lambda between labels and model predictions. [2] and [3] are not cited. Their setting is self-distillation rather than cross-modal distillation, but the core analysis is very similar.

 

Key differences and our novelty

In [1–3], the teacher shares the student's input, and the teacher of [2, 3] is further fitted to the student's own labels. In contrast, our teacher takes its input from another modality. We analyze what information is needed to decide whether distillation helps, and how this difference changes the risk of the fitted student.

Our analysis yields two results that [1–4] do not provide.

  • First, the teacher's risk and the label noise do not determine whether distillation helps (Theorem 1). As the reviewer noted, our gain identity explains why teacher accuracy alone cannot decide whether cross-modal distillation helps, and Theorem 1 establishes this result formally. Theorem 1 constructs two problems that share these quantities, the risk of the supervised student and the discrepancy ψ\psi, but whose optimal weights are 0 and 1. The two teachers therefore also have the same squared error to the Bayes predictor, the measure of teacher quality in [1]. Any selector that uses only these quantities incurs, in one of the two problems, an expected regret of at least a constant fraction of the gain attainable when distillation helps. Fig. 4 in Appendix G (Fig. 3 in the submission) shows that teachers of nearly the same risk relative to the student can lead to opposite outcomes on real-world datasets.
  • Second, no criterion that ignores the labels can determine whether distillation helps (Theorem 2). Therefore, the weight must be estimated with labels, and our rule estimates it with the student's existing labels. Theorem 2 gives a similar regret bound even when the two problems share the entire joint distribution of the inputs and the teacher outputs. [1–4] do not show that labels are necessary for this decision.

Beyond the quadratic form, the gain identity shows that the sign of AA determines whether a beneficial weight exists and that [A/D][0,1][A/D]_{[0,1]} is the optimal weight (Eqs. 2 and 4). The cross-modal teacher changes both coefficients. Our analysis explicitly accounts for these effects, which [1–4] address only partially or not at all:

  • Teacher variation unexplained by the student's input (σt2\sigma_t^2): zero for the fixed same-input teacher of [1], determined by the student's own labels in [2, 3], and affecting only the asymptotic variance in [4].
  • Covariance between the unexplained teacher variation and the label noise (σyt\sigma_{yt}): likewise zero for the teacher of [1], determined by the student's own labels in [2, 3], and affecting only the asymptotic variance in [4].
  • Changes in the student's bias (δ\delta, β\beta): bounded through the size of the teacher's error rather than its alignment with the student's bias in [1], determined by the true parameter, the inputs and the ridge penalty in [2], described as amplified regularization in [3], and absent from the unbiased objective of [4].

Together, our analysis of the cross-modal teacher and Theorems 1 and 2 characterize whether and how much to distill from a teacher in another modality. We believe that this characterization is a meaningful contribution, since teachers from other modalities are increasingly common as multi-modal data become widely available.

W1. Novelty relative to prior analyses (2/4)


 

Detailed comparison with [1–4]

  • The analysis of [1] considers distillation from the teacher outputs alone, in which the teacher and the student share the same input. Hence, [1] does not analyze the weight between labels and teacher outputs. [1] proposes the teacher's squared error to the Bayes class probabilities as the criterion for a good teacher, but our Theorem 1 shows that two teachers with the same error can still require different weights.
  • [2] analyzes the weight of self-distillation in ridge regression, in which the teacher shares the student's input and is fitted to the student's own labels. Hence, the teacher's outputs contain no information beyond the student's own data. As in our Proposition 1, the parameter error of [2] is quadratic in the weight, and [2] derives its optimal weight in closed form. However, the optimum depends on unknown quantities such as the true parameter and the noise variance, and [2] gives no procedure to estimate the optimum from data.
  • [3] analyzes self-distillation in kernel ridge regression, in which the teacher shares the student's input and is fitted to the student's own labels. The results of [3] are deterministic expressions for the given training and validation data, rather than a risk averaged over label noise. The validation error of [3] is also quadratic in the weight, and [3] estimates its minimizer in closed form without a grid search. However, the estimate requires holding out a labeled validation set, whereas our rule trains the student on all labels.
  • [4] addresses statistical inference with model predictions rather than distillation. [4] tunes a weight to minimize the asymptotic variance of an estimator whose objective remains unbiased for any weight. Since teacher outputs replace part of the labels in distillation, the distillation weight must also account for bias, as our analysis does.

W1. Novelty relative to prior analyses (3/4)


 

Policies adapted from [1–4]

To further address the reviewer's concern, we adapted the weight-selection principles of [1–4] to the cross-modal setting and compared the resulting policies with our rule in the format of Table 3.

Table F. Change in the student's risk under each policy, in a separate run on the sixteen cross-modal directions with nn = 50–300 labels and ten seeds.

PolicyExtra labelsFitsLive: MeanWorst dir.HurtAll: MeanWorst dir.Hurt
always, λ=0.5\lambda = 0.5none1−4.67%−192.21%5/16−17.07%−259.51%7/16
never distillnone1+0.00%+0.00%0/16+0.00%+0.00%0/16
teacher-risk gate, from [1]test labels2+1.16%+0.00%0/16+0.68%−0.49%2/16
self-distillation transfer, from [2]test labels2−37.84%−1088.24%7/16−57.09%−915.89%7/16
validation weight from [3], 20% / 33% splitsplit from nn3−0.91% / −0.63%−14.34% / −8.03%10/16 / 9/16−1.86% / −1.71%−8.98% / −5.94%10/16 / 9/16
clipped and refit, 20% / 33% splitsplit from nn3+2.12% / +2.06%−0.29% / −0.18%4/16 / 3/16+1.11% / +1.30%−1.65% / −0.62%9/16 / 8/16
PPI++ weight, from [4]none1+1.55%+0.00%0/16+0.93%−0.54%5/16
oursnone1+2.00%−0.04%1/16+1.27%−0.75%5/16
best per cell, grid searchtest labels8+3.74%+0.66%0/16+2.83%+0.14%0/16

Since [1] gives no rule for choosing the weight, the teacher-risk gate distills at λ=0.5\lambda = 0.5 when the teacher has lower squared risk than the student. For [2], we compute the optimal self-distillation weight of the same student, clipped to [0,1][0, 1], and apply it to the cross-modal teacher. This policy tests whether the self-distillation weight remains reliable with the cross-modal teacher, rather than the method of [2] in its own setting. For [3], we take the validation split from the student's nn labels. The clipped variant also clips the weight to [0,1][0, 1], as suggested in [3], and refits on all nn labels. For [4], we use its plug-in weight as the distillation weight, clipped to [0,1][0, 1] as suggested in [4]. In our implementation, the first two policies use test labels, which gives them an advantage unavailable in practice.

  • The self-distillation weight from [2] is not reliable with the cross-modal teacher. This weight is derived for a teacher from the same modality as the student, and it degrades the student on average and harms many directions when applied to the cross-modal teacher, even with test labels.
  • The teacher-risk gate from [1] avoids most harm but attains a lower mean gain than our rule, even with test labels.
  • The validation weight from [3] uses part of the label budget. Used directly, it degrades the student on average. With clipping and a refit on all labels, it attains a mean gain comparable to ours but harms more directions in both the Live and All settings.
  • The PPI++ weight and our rule require neither test labels nor additional fits, but the two policies trade off gain and harm differently. The PPI++ weight has a slightly better worst direction, whereas our rule attains a higher mean gain in both the Live and All settings. Both policies avoid most of the harm of the label-blind policy that always distills at λ=0.5\lambda = 0.5, which is consistent with our Theorem 2.

Overall, the policies adapted from [1–3] use test labels or part of the label budget, and in this comparison they are less suited to the cross-modal setting than our rule. The PPI++ weight from [4] trades gain for harm differently, and [4] sets its weight for statistical inference rather than for distillation. Therefore, these comparisons suggest that our analysis of the cross-modal teacher leads to a practical rule that [1–4] do not directly provide.

Reviewer_1ZYHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when cross-modal knowledge distillation (CMKD) should be performed and how strongly the teacher should be weighted. Under a fixed linear solution map with squared loss, it shows that distillation depends not only on noise reduction but also on whether the teacher-induced update aligns with the student’s bias. Based on the theoretical analysis, this paper proposes a gain-based decision criteron to decide whether distillation benefits, and proved effective across several multimodal benchmarks.

Critical strengths
  • The paper theoretically analyze whether and when cross-modal distillation is beneficial.

  • The proposed criterion is theoretically inspired and empirically validated.

Critical weaknesses
  • The theoretical results require the fitted student to be an affine function of the training targets under squared loss, which somewhat limits its application to modern neural networks.

  • The presentation of the theoretical parts is very notation-oriented, which is hard to read for readers without closed background. More intuitional explanations should be added.

  • The theory attributes part of the distillation gain to a noise-reduction term T(σy2−σyt)T(\sigma_y^2-\sigma_{yt}), but the experiments do not appear to directly validate this mechanism by controlling the amount of label noise.

Questions for authors
  • Can the theretical resutls extend to modern neural network settings?

  • Coudld the authors conduct a controlled label-noise study to validate the noise-reduction effect?

  • Will the gain-based criterion help in λ\lambda selection? For example, instead of doing grid search, would directly selecting λ=arg⁡max⁡λ∈ΛG^(λ)\lambda=\arg\max_{\lambda\in\Lambda}\hat G(\lambda) perform well empirically?

6 comments

We sincerely thank the reviewer for the careful reading of our manuscript and for the thoughtful and encouraging evaluation. We particularly appreciate the reviewer's recognition of our theoretical analysis of whether and when cross-modal knowledge distillation is beneficial, and of our criterion as theoretically inspired and empirically validated. We provide detailed responses to the reviewer's questions and concerns below.


 

W1 and Q1. Application of our rule to neural network students

The theoretical results require the fitted student to be an affine function of the training targets under squared loss, which somewhat limits its application to modern neural networks.

Can the theretical resutls extend to modern neural network settings?

 

Although we leave the theoretical extension to nonlinear neural networks for future work, as noted in Section 7, our experiments already apply our rule to MLP students. For these students, we determine the weight from a ridge regression proxy on the same input instead of analyzing the network itself, and then train the network once at this weight. Specifically, the quantities of Eq. 4 are computed from this proxy model. Algorithm 2 in Appendix A describes this procedure.

Table 24 in Appendix H (Table 23 in the submission) now reports MLP students of four sizes together with the MLP student of our main experiments in Table 3. Table C below collects these results and shows that our rule improves every MLP student on average, while distilling at λ=0.5\lambda = 0.5 degrades it. These results indicate that our rule can be extended beyond the linear setting of our theory.

Table C. Our rule on MLP students (mean change in test risk against supervised training). The rows are reproduced from Table 24 in Appendix H, and the first row is the MLP student of Table 3.

MLP studentdistill at λ=0.5\lambda = 0.5oursbest weight per cell
Table 3, main experiments−10.99%+1.27%+5.40%
tiny (23K parameters)−15.48%+2.26%+4.95%
small (307K parameters)−11.79%+1.53%+4.68%
wide (2.8M parameters)−17.12%+1.62%+5.15%
deep (1.0M parameters)−6.61%+1.33%+5.29%

 

Changes in the revision (in blue, with the numbering of the revised paper)

  • Appendix H, Table 24 (Table 23 in the submission): We added the MLP student of Table 3, so that all MLP students in Table C appear in a single table.

W2. Readability of the theory

The presentation of the theoretical parts is very notation-oriented, which is hard to read for readers without closed background. More intuitional explanations should be added.

 

Following the reviewer's suggestion, we have added an intuitive explanation of the theory to Section 3 and a notation table to Appendix I.

Right after Eq. 4 defines the quantities that determine the distillation gain, Section 3 now explains what each quantity means for knowledge transfer, with as few symbols as possible. We quote the added paragraph below.

Interpretation in terms of knowledge transfer. Distillation helps when the teacher's outputs reduce the noise in the student's targets or offset the student's bias, and it hurts when they carry information the student cannot observe or errors the student learns. Eq. 4 separates these effects. The label residual uyu_y is the part of the label that the student's input cannot predict. Distillation reduces this noise and thus raises the gain through σy2\sigma_y^2. The teacher residual utu_t comes from the teacher's modality, which the student does not observe, and acts as noise for the student. As a result, σt2\sigma_t^2 lowers the gain. The covariance σyt\sigma_{yt} lowers it further when the teacher repeats the label residual. The student learns the discrepancy ψ\psi as a systematic error, which lowers the gain through δ\delta. Its alignment β\beta with the student's own bias raises the gain when this error offsets that bias and lowers the gain when it enlarges that bias. Section 6.4 and Appendices H.16 and H.17 test the effects of the label residual, the teacher residual and the systematic error in controlled experiments.

In addition, Appendix I lists the main symbols of Sections 3 to 5, each with its definition and its meaning, so that readers can look up a quantity while reading the theory. We hope that these changes make the theoretical parts easier to follow for readers without a close background.

 

Changes in the revision (in blue, with the numbering of the revised paper)

  • Section 3: We added the paragraph "Interpretation in terms of knowledge transfer" right after Eq. 4.
  • Appendix I: We added a notation table that lists each main symbol with its definition and its meaning.

W3 and Q2. Controlled label-noise study

The theory attributes part of the distillation gain to a noise-reduction term T(σy2−σyt)T(\sigma_y^2 - \sigma_{yt}), but the experiments do not appear to directly validate this mechanism by controlling the amount of label noise.

Coudld the authors conduct a controlled label-noise study to validate the noise-reduction effect?

 

As the reviewer requested, we conducted a controlled label-noise study to validate the noise-reduction effect. Specifically, we added Gaussian noise to the student's labels only and measured how much the distillation gain changes. Table D below shows the results.

According to our theoretical analysis, larger label noise leaves more noise for distillation to reduce. Therefore, both the distillation gain and the optimal weight on the teacher's outputs should rise with the label noise. Table D confirms this prediction, as both quantities increase monotonically with the label noise. Table E shows the same trend for an MLP student with the architecture of Table 3.

In addition, to verify the theoretical prediction quantitatively, we compared the measured gain change with the prediction of our identity, which involves no fitted parameter. For the ridge student, for which our identity is exact, the measured gain changes match the predicted gain changes with a regression slope of 1.002 and an R2R^2 of 0.997. These results directly validate the noise-reduction effect.

Table D. Distillation to the ridge student with Gaussian noise added to its labels only, with cc times the standard deviation of the labels.

Label noise cc00.512
optimal weight λ∗\lambda^* (mean)0.300.370.510.72
ours (mean change)+1.24%+2.47%+6.69%+21.57%
best weight per cell (mean change)+2.79%+4.15%+8.74%+25.47%

Table E. The same study with the MLP architecture of Table 3.

Label noise cc012
optimal weight λ∗\lambda^* (mean)0.340.520.72
ours (mean change)+1.25%+4.41%+8.17%
best weight per cell (mean change)+4.98%+7.41%+13.31%

 

Changes in the revision (in blue, with the numbering of the revised paper)

  • Section 6.4 (new): We added this label-noise study, with the measured gain shift against our prediction (Figure 2) and the results of Tables D and E (Table 4).
  • Appendix H.16 (new): We added the full design and statistics of the study.

Reviewer_1cVGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates when to use teacher supervision in cross-modal knowledge distillation and how to set the distillation weight. The authors argue that a teacher’s predictive accuracy alone is insufficient to determine whether it can improve the student, because the teacher may rely on information unavailable in the student’s modality, and teacher supervision may also alter the student’s bias. For student models with a fixed linear solution map under squared loss, the paper derives an exact identity for the gain from distillation relative to purely supervised learning and, under explicit conditions, characterizes the gain curve and optimal weight using six statistical quantities. Building on this analysis, the authors estimate the distillation gain using the student’s existing labeled data and unlabeled cross-modal paired data, incorporating uncertainty adjustments to determine whether and how much to distill, without requiring additional labels or a validation-based weight search. Experiments on six real-world multimodal datasets show that the proposed rule reduces performance degradation caused by inappropriate distillation and achieves a positive average gain in the main policy comparison.

Critical strengths

(1).The research question is practically relevant. The paper directly addresses the observation that a stronger teacher does not necessarily produce a better student and that inappropriate distillation can harm student performance. It brings the decisions of whether to distill and how much to distill into a unified framework. (2).The theoretical analysis provides an explicit statistical characterization. Under the stated assumptions, the gain identity connects the distillation weight to changes in the student’s risk and identifies cases where teacher-side information is insufficient to determine the benefit of distillation. This helps explain why selecting teachers or distillation weights solely based on teacher accuracy may be unreliable. (3).The experimental evaluation is broad. The paper evaluates the proposed approach across multiple real-world multimodal datasets and transfer directions, including comparisons with related methods, ablation studies, and sensitivity analyses. Its reuse of existing labeled data also offers practical value in settings with limited annotation budgets.

Critical weaknesses

(1).The mechanisms underlying negative transfer remain insufficiently explained. The paper uses statistical quantities, including label residuals, teacher residuals, their covariance, and bias alignment, to characterize when distillation may be beneficial. While this provides a statistical explanation, the connection between these quantities and the specific mechanisms of knowledge transfer remains unclear. For example, which information from the teacher can the student use, and which information causes negative transfer? The current analysis does not sufficiently distinguish between these sources. (2).The practical decision procedure is overly complex. Moving from the theoretical gain expression to the final operational rule involves multiple components, including a bridge model, cross-fitting, bias estimation, proxy substitutions, jackknife-based shrinkage, and projection onto a weight grid. These components complicate the practical procedure and make it difficult to determine whether the performance improvements stem from the core theoretical framework or the empirical adjustments.

Questions for authors

(1).The paper focuses on cross-modal knowledge distillation, but its analysis primarily revolves around statistical quantities such as label residuals, teacher residuals, covariance, and bias alignment. These analyses appear equally applicable to knowledge distillation in general. Could the authors clarify why they frame their study specifically as cross-modal knowledge distillation?

4 comments

We sincerely thank the reviewer for the careful reading of our manuscript and for the thoughtful and constructive evaluation. We particularly appreciate the reviewer's recognition of the practical relevance of deciding whether and how much to distill within a single framework, the explicit statistical characterization provided by the gain identity, and the breadth of our evaluation across real-world multimodal datasets. We provide detailed responses to the reviewer's questions and concerns below.


 

W1. Interpretation of the statistical quantities in cross-modal knowledge transfer (1/2)

The mechanisms underlying negative transfer remain insufficiently explained. The paper uses statistical quantities, including label residuals, teacher residuals, their covariance, and bias alignment, to characterize when distillation may be beneficial. While this provides a statistical explanation, the connection between these quantities and the specific mechanisms of knowledge transfer remains unclear. For example, which information from the teacher can the student use, and which information causes negative transfer? The current analysis does not sufficiently distinguish between these sources.

 

Intuitively, distillation helps the student when the teacher's outputs reduce the noise in the student's training targets or offset the student's bias. Negative transfer arises when the teacher's outputs contain information that the student cannot observe, or errors that the student learns.

 

Meaning of each quantity

Our paper analyzes distillation from a cross-modal teacher. In Section 3, we decompose the teacher's output tt into qq and utu_t, where qq is the part of tt that the student's input can predict. Similarly, we decompose the label yy into mym_y and uyu_y, where mym_y is the part of yy that the student's input can predict. From these parts, we define the quantities that determine the distillation gain. In terms of knowledge transfer, each quantity can be interpreted as follows.

  • Label residual (uyu_y): the part of the label that the student's input cannot predict. A student trained on labels alone fits uyu_y as noise, which adds Tσy2T\sigma_y^2 to the student's risk. Distillation with the teacher's outputs can mitigate the effect of this noise. Therefore, the label residual raises the distillation gain by λ(2−λ)Tσy2\lambda(2-\lambda)T\sigma_y^2 (Eqs. 2 and 4).
  • Teacher residual (utu_t): the part of the teacher's output that the student's input cannot predict. Since utu_t is information specific to the teacher's modality and is not observed by the student, utu_t acts as noise for the student. Therefore, the teacher residual lowers the distillation gain by λ2Tσt2\lambda^2 T\sigma_t^2 only through the student's variance (Eqs. 2 and 4).
  • Their covariance (σyt\sigma_{yt}): whether the teacher's outputs also contain the label residual. If so, distillation mitigates less of the label residual. Therefore, a larger σyt\sigma_{yt} lowers the distillation gain by 2λ(1−λ)Tσyt2\lambda(1-\lambda)T\sigma_{yt} (Eqs. 2 and 4).
  • Systematic error (δ=ψ⊤Σψ\delta = \psi^\top\Sigma\psi): the teacher's error that the student can learn. Since the student can predict both mym_y and qq from its own input, the student learns their difference as ψ=L(my−q)\psi = L(m_y - q) through distillation. Therefore, the systematic error lowers the distillation gain by λ2δ\lambda^2\delta through the student's bias (Eqs. 2 and 4).
  • Bias alignment (β=b⊤Σψ\beta = b^\top\Sigma\psi): whether the teacher offsets or enlarges the student's bias. A student trained on labels alone has its own bias bb, and distillation shifts this bias to b−λψb - \lambda\psi. Therefore, the bias alignment changes the distillation gain by 2λβ2\lambda\beta through the student's bias (Eqs. 2 and 4). Since β\beta depends on the direction of bb, two teachers with the same risk and the same ψ\psi can lead to optimal weights of 0 and 1 (Theorem 1).

These quantities determine the distillation gain G(λ)=2λA−λ2DG(\lambda) = 2\lambda A - \lambda^2 D (Eq. 2) through A=T(σy2−σyt)+βA = T(\sigma_y^2 - \sigma_{yt}) + \beta and D=T(σy2−2σyt+σt2)+δD = T(\sigma_y^2 - 2\sigma_{yt} + \sigma_t^2) + \delta (Eq. 4).

W1. Interpretation of the statistical quantities in cross-modal knowledge transfer (2/2)

 

Controlled experiments

To verify that this interpretation holds in practice, we conducted two controlled experiments.

First, we increased the label residual by adding Gaussian noise to the student's labels only. As shown in Table A below, the distillation gain rises as the label residual grows, consistent with our interpretation of the label residual above. In addition, the gain of our rule, which estimates these quantities from data, also rises with the label noise.

Table A. Distillation to the ridge student with Gaussian noise added to its labels only, with cc times the standard deviation of the labels.

Label noise cc00.512
optimal weight λ∗\lambda^* (mean)0.300.370.510.72
ours (mean change)+1.24%+2.47%+6.69%+21.57%
best weight per cell (mean change)+2.79%+4.15%+8.74%+25.47%

Second, to examine how the teacher harms the student, we added variation of the same variance to the teacher's outputs in two settings. One setting adds noise independent of all inputs, which raises the teacher residual. The other setting adds a random linear function of the student's input, which raises the systematic error. As shown in Table B, adding either the teacher residual or the systematic error lowers the distillation gain for both the ridge and the MLP students, as our interpretation predicts. In particular, the systematic error causes a larger loss than the teacher residual of the same variance.

Table B. Distillation with the same variance added to the teacher's outputs, as a teacher residual or as a systematic error, averaged over both signs of the added variation.

StudentAdded to the teacher's outputsharm rate at λ=0.5\lambda = 0.5λ=0.5\lambda = 0.5 (mean change)best weight per cell (mean change)
ridgenone52%−15.99%+2.32%
ridgeteacher residual54%−16.84%+1.93%
ridgesystematic error91%−28.76%+1.25%
MLPnone52%−12.11%+4.88%
MLPteacher residual62%−15.00%+3.88%
MLPsystematic error76%−26.63%+2.88%

 

Changes in the revision (in blue, with the numbering of the revised paper)

  • Section 3: We added the paragraph "Interpretation in terms of knowledge transfer" right after Eq. 4, which explains each quantity above in terms of knowledge transfer.
  • Section 6.4 (new): We added the controlled label-noise study of Table A, with the measured gain shift against our prediction (Figure 2) and its effect on the weight and the gain (Table 4).
  • Appendices H.16 and H.17 (new): We added the full label-noise study and the teacher-side experiment of Table B (Table 36).

W2. Components of the decision rule and the source of its gain

The practical decision procedure is overly complex. Moving from the theoretical gain expression to the final operational rule involves multiple components, including a bridge model, cross-fitting, bias estimation, proxy substitutions, jackknife-based shrinkage, and projection onto a weight grid. These components complicate the practical procedure and make it difficult to determine whether the performance improvements stem from the core theoretical framework or the empirical adjustments.

 

The ablation in Appendix G indicates that the gain of our rule comes mainly from the estimated quantities of our theory rather than from the empirical adjustment.

Our rule estimates the optimal weight λ∗=[A/D][0,1]\lambda^* = [A/D]_{[0,1]} of Proposition 1 from data. The bridge model, cross-fitting, bias estimation and the proxy substitutions are the steps that estimate AA and DD in Eq. 4. Only the jackknife shrinkage (Proposition 4) adjusts the estimated weight. The projection onto the weight grid is used only for evaluation, and our rule can also return the continuous weight directly. Table 8 in Appendix G (Table 7 in the submission) provides an ablation of the shrinkage, and we report its relevant rows below. Without the shrinkage, our rule still attains 82% of its mean gain.

Table 8 (reproduced from Appendix G, all settings). The row "ours without shrinkage" is "skip if G^(λ^)≤0\hat G(\hat\lambda) \le 0" in Table 8.

PolicyMeanWorst dir.Hurt
always, λ=0.5\lambda = 0.5−26.47%−172.50%11/16
ours without shrinkage+0.73%−1.32%5/16
ours+0.89%−0.47%4/16
best per cell, grid search+2.47%+0.32%0/16

 

Changes in the revision (in blue, with the numbering of the revised paper)

  • Section 5: We added that the projection onto the weight grid serves the evaluation and that our rule can also return the continuous weight directly.

 

Q1. Why cross-modal knowledge distillation

The paper focuses on cross-modal knowledge distillation, but its analysis primarily revolves around statistical quantities such as label residuals, teacher residuals, covariance, and bias alignment. These analyses appear equally applicable to knowledge distillation in general. Could the authors clarify why they frame their study specifically as cross-modal knowledge distillation?

Cross-modal knowledge distillation is an important research topic, since a deployed model often cannot use all the modalities available during training due to cost, latency or privacy. Unlike a unimodal teacher, a cross-modal teacher uses information that the student cannot observe from its own input, and this information can change the outcome of distillation. To analyze this effect, we formulate the problem so that the teacher can take its input from another modality. With this formulation, our analysis incorporates this information through the teacher residual and its covariance with the label residual (W1). This analysis of cross-modal teacher information is a key novelty of our work. In addition, as the reviewer notes, our analysis covers unimodal knowledge distillation as the special case in which the teacher and the student share the same input, where these two quantities vanish (Corollary 2).

We thank the reviewer again for the review. As the discussion period is closing, we would like to ask whether our response has addressed the reviewer's concerns. In particular, we explained when and why distillation hurts in terms of the teacher residual and its covariance with the label residual, added this interpretation to Section 3 with controlled experiments, and showed how much of the gain comes from each component of our rule. If any concern remains, we would be glad to address it.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Gentle Reminder for the Discussion

Dear Reviewers and Area Chair,

As the discussion period is approaching its conclusion, we would like to send a gentle reminder regarding our responses. We express our sincere gratitude to all the reviewers and the Area Chair for the time and effort devoted to carefully evaluating our manuscript. Your constructive feedback has greatly helped us strengthen our work and improve its clarity. To address the reviewers' concerns, we have provided point-by-point responses to all the questions and concerns raised in the initial reviews.

We would be grateful if you could let us know whether our responses have adequately addressed your concerns. We would be happy to answer any further questions and would be delighted to provide additional clarifications.

Best regards, The Authors

Dear Reviewers and Area Chair,

We posted our responses and the revised paper on November 12, and a follow-up on November 17. Reviewer 1ZYH replied that our response addresses most of their concerns. We have not yet received responses from Reviewers 1cVG and fVpf. As the discussion period will close soon, we would appreciate it if they could let us know whether our responses address their concerns, so that we can clarify any remaining points.

Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.