- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper looks at cross-modal knowledge distillation, where a student in one modality is trained on a mix of true labels and a teacher's outputs from another modality, and asks whether to distill and with what weight . For a linear student (e.g., ridge regression with a fixed penalty) under squared loss, the authors show that the gain over training on labels alone is exactly . Distillation therefore helps only when , and the best weight is . They also give two toy examples showing that the teacher's accuracy, the label noise, or any criterion that ignores the student's labels cannot tell whether is positive. Based on this, they propose a rule that estimates and from the student's own labeled data plus unlabeled paired data. It does not distill when the estimate of is not positive, and otherwise uses a shrunk version of . On six multimodal datasets, mostly with frozen features and ridge or MLP students, a fixed often hurts, while the rule improves on not distilling by 0.89% of risk on average.
Critical strengths
The gain identity cleanly explains why teacher accuracy alone cannot decide whether cross-modal distillation helps.
Critical weaknesses
- Limited novelty. With a linear student, a risk quadratic in the mixing weight follows directly, and closely related analyses already exist. [1] gives a bias–variance view of when a teacher helps. [2] analyzes regularized linear regression with a weight between labels and teacher predictions, and derives the optimal under label noise. [3] gives a closed-form optimal weight between ground-truth targets and teacher outputs in kernel ridge regression, and estimates it without a grid search. [4] tunes a closed-form weight between labels and model predictions. [2] and [3] are not cited. Their setting is self-distillation rather than cross-modal distillation, but the core analysis is very similar.
- The theory holds only for a fixed ; the deployed rule estimates from training data without guarantees.
- Gains are small: +0.89% over not distilling, and level with a simple validation sweep ().
[1] A. K. Menon, A. S. Rawat, S. Reddi, S. Kim, and S. Kumar. A Statistical Perspective on Distillation. In ICML, PMLR 139:7632–7642, 2021.
[2] R. Das and S. Sanghavi. Understanding Self-Distillation in the Presence of Label Noise. In ICML, PMLR 202:7102–7140, 2023.
[3] K. Borup and L. N. Andersen. Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation. In NeurIPS, 2021.
[4] A. N. Angelopoulos, J. C. Duchi, and T. Zrnic. PPI++: Efficient Prediction-Powered Inference. arXiv:2311.01453, 2023.
Questions for authors
- What is new in Proposition 1 compared with [1] and [2], which also study the optimal label/teacher weight in (kernel) ridge regression?
- Why were 238 of the 1,040 candidate cells excluded? Why do the evaluation populations differ across Tables 1, 2, 3 and 15?
[1] R. Das and S. Sanghavi. Understanding Self-Distillation in the Presence of Label Noise. In ICML, PMLR 202:7102–7140, 2023.
[2] K. Borup and L. N. Andersen. Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation. In NeurIPS, 2021.
8 comments
W1. Novelty relative to prior analyses (2/4)
Detailed comparison with [1–4]
- The analysis of [1] considers distillation from the teacher outputs alone, in which the teacher and the student share the same input. Hence, [1] does not analyze the weight between labels and teacher outputs. [1] proposes the teacher's squared error to the Bayes class probabilities as the criterion for a good teacher, but our Theorem 1 shows that two teachers with the same error can still require different weights.
- [2] analyzes the weight of self-distillation in ridge regression, in which the teacher shares the student's input and is fitted to the student's own labels. Hence, the teacher's outputs contain no information beyond the student's own data. As in our Proposition 1, the parameter error of [2] is quadratic in the weight, and [2] derives its optimal weight in closed form. However, the optimum depends on unknown quantities such as the true parameter and the noise variance, and [2] gives no procedure to estimate the optimum from data.
- [3] analyzes self-distillation in kernel ridge regression, in which the teacher shares the student's input and is fitted to the student's own labels. The results of [3] are deterministic expressions for the given training and validation data, rather than a risk averaged over label noise. The validation error of [3] is also quadratic in the weight, and [3] estimates its minimizer in closed form without a grid search. However, the estimate requires holding out a labeled validation set, whereas our rule trains the student on all labels.
- [4] addresses statistical inference with model predictions rather than distillation. [4] tunes a weight to minimize the asymptotic variance of an estimator whose objective remains unbiased for any weight. Since teacher outputs replace part of the labels in distillation, the distillation weight must also account for bias, as our analysis does.
W1. Novelty relative to prior analyses (3/4)
Policies adapted from [1–4]
To further address the reviewer's concern, we adapted the weight-selection principles of [1–4] to the cross-modal setting and compared the resulting policies with our rule in the format of Table 3.
Table F. Change in the student's risk under each policy, in a separate run on the sixteen cross-modal directions with = 50–300 labels and ten seeds.
| Policy | Extra labels | Fits | Live: Mean | Worst dir. | Hurt | All: Mean | Worst dir. | Hurt |
|---|---|---|---|---|---|---|---|---|
| always, | none | 1 | −4.67% | −192.21% | 5/16 | −17.07% | −259.51% | 7/16 |
| never distill | none | 1 | +0.00% | +0.00% | 0/16 | +0.00% | +0.00% | 0/16 |
| teacher-risk gate, from [1] | test labels | 2 | +1.16% | +0.00% | 0/16 | +0.68% | −0.49% | 2/16 |
| self-distillation transfer, from [2] | test labels | 2 | −37.84% | −1088.24% | 7/16 | −57.09% | −915.89% | 7/16 |
| validation weight from [3], 20% / 33% split | split from | 3 | −0.91% / −0.63% | −14.34% / −8.03% | 10/16 / 9/16 | −1.86% / −1.71% | −8.98% / −5.94% | 10/16 / 9/16 |
| clipped and refit, 20% / 33% split | split from | 3 | +2.12% / +2.06% | −0.29% / −0.18% | 4/16 / 3/16 | +1.11% / +1.30% | −1.65% / −0.62% | 9/16 / 8/16 |
| PPI++ weight, from [4] | none | 1 | +1.55% | +0.00% | 0/16 | +0.93% | −0.54% | 5/16 |
| ours | none | 1 | +2.00% | −0.04% | 1/16 | +1.27% | −0.75% | 5/16 |
| best per cell, grid search | test labels | 8 | +3.74% | +0.66% | 0/16 | +2.83% | +0.14% | 0/16 |
Since [1] gives no rule for choosing the weight, the teacher-risk gate distills at when the teacher has lower squared risk than the student. For [2], we compute the optimal self-distillation weight of the same student, clipped to , and apply it to the cross-modal teacher. This policy tests whether the self-distillation weight remains reliable with the cross-modal teacher, rather than the method of [2] in its own setting. For [3], we take the validation split from the student's labels. The clipped variant also clips the weight to , as suggested in [3], and refits on all labels. For [4], we use its plug-in weight as the distillation weight, clipped to as suggested in [4]. In our implementation, the first two policies use test labels, which gives them an advantage unavailable in practice.
- The self-distillation weight from [2] is not reliable with the cross-modal teacher. This weight is derived for a teacher from the same modality as the student, and it degrades the student on average and harms many directions when applied to the cross-modal teacher, even with test labels.
- The teacher-risk gate from [1] avoids most harm but attains a lower mean gain than our rule, even with test labels.
- The validation weight from [3] uses part of the label budget. Used directly, it degrades the student on average. With clipping and a refit on all labels, it attains a mean gain comparable to ours but harms more directions in both the Live and All settings.
- The PPI++ weight and our rule require neither test labels nor additional fits, but the two policies trade off gain and harm differently. The PPI++ weight has a slightly better worst direction, whereas our rule attains a higher mean gain in both the Live and All settings. Both policies avoid most of the harm of the label-blind policy that always distills at , which is consistent with our Theorem 2.
Overall, the policies adapted from [1–3] use test labels or part of the label budget, and in this comparison they are less suited to the cross-modal setting than our rule. The PPI++ weight from [4] trades gain for harm differently, and [4] sets its weight for statistical inference rather than for distillation. Therefore, these comparisons suggest that our analysis of the cross-modal teacher leads to a practical rule that [1–4] do not directly provide.
We sincerely thank the reviewer for the careful reading of our manuscript and for the thoughtful and constructive evaluation. We particularly appreciate the reviewer's recognition that our gain identity cleanly explains why teacher accuracy alone cannot decide whether cross-modal knowledge distillation is beneficial. We provide detailed responses to the reviewer's questions and concerns below.
W1. Novelty relative to prior analyses (1/4)
Key differences and our novelty
In [1–3], the teacher shares the student's input, and the teacher of [2, 3] is further fitted to the student's own labels. In contrast, our teacher takes its input from another modality. We analyze what information is needed to decide whether distillation helps, and how this difference changes the risk of the fitted student.
Our analysis yields two results that [1–4] do not provide.
Beyond the quadratic form, the gain identity shows that the sign of A determines whether a beneficial weight exists and that [A/D][0,1] is the optimal weight (Eqs. 2 and 4). The cross-modal teacher changes both coefficients. Our analysis explicitly accounts for these effects, which [1–4] address only partially or not at all:
Together, our analysis of the cross-modal teacher and Theorems 1 and 2 characterize whether and how much to distill from a teacher in another modality. We believe that this characterization is a meaningful contribution, since teachers from other modalities are increasingly common as multi-modal data become widely available.