- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper provides empirical evidence that the average kernel over many potentially weaker models is more similar to the kernel of a strong "oracle" model as the number of models in the average increase. Additionally, the authors also provide evidence that increasing a student's kernel's similarity to the average kernel improves the student model's performance.
Critical strengths
- The authors apply a training method by increasing similarity between average kernel of multiple teachers and the student's kernel. The design is flexible, where the teachers can be specialist/generalist or have different architectures.
- The paper identifies that average kernels improve similarity to the oracle kernel.
- The paper is written without overly dense prose or inscrutable descriptions of settings. The prose flows naturally.
- The experiments are well-designed, easily interpreted, and mostly support the main interpretations.
Critical weaknesses
Major weaknesses
- Limited comparisons with other distillation methods. The paper compares DiffKNN, CKA, UCKA, supervised learning, and ensembling. However, there are no comparisons with established distillation methods, eg Feature Kernel Distillation [1]. Other relevant works include UNIC [2], and [3]. Another reasonable baseline would be logit distillation for balanced/specialist settings, where teachers share the same classification tasks.
- It is unclear whether the student is trained against the averaged kernel or individual kernels from teachers for . Sec 4 (lines 190-198) indicate that student is matched to the average teacher kernel, but lines 227-230 state that similarity loss is averaged across teachers. I am not sure if these are the same since CKA and DiffKNN are generally nonlinear. The authors should clarify this.
- Experiments are limited in modalities. The authors focus on CIFAR10/ImageNet100 image classification experiments. Additionally, there are language-model and cross-modal average kernel alignment experiments in appendix A.9. However, there are no experiments involving models for image generation (eg diffusion). This limits the generality of the paper. Additionally, authors could explore how distilling using average kernels could help in language classification/generation tasks and image generation tasks.
- Observation 3 doesn't appear valid for arbitrary correlated errors (lines 169-171, appendix A.7). Suppose every teacher shares the same random error, , where sometimes changes the nearest-neighbour rankings and sometimes doesn't. Then regardless of how many teachers there are, so the neighbourhood overlap need not converge to a fixed value. The authors should consider including additional assumptions on the errors.
- The DiffKNN loss may be infinite. Eq 4-5 define with 0 probability outside the top-k neighbourhoods. When but , the KL divergence is infinite, and would occur when the neighbourhoods have different supports. The authors should clarify that smoothing or some numerical modification is used in practice for experiments, since the code in the supplemental materials seem to indicate authors defined the loss differently to avoid these numerical issues.
- Over-claim in the conclusion (lines 468-483). The authors indicate that "all teacher kernels reside in the same convex basin", although the empirical evidence for this is weaker in multi-architecture/-task settings (the authors themselves admit to this). Even if the similarity landscapes were empirically convex, this does not establish that the underlying models belong to the same training-loss basin. So I think this interpretation in the conclusion is too strong.
Minor weaknesses
- Having more teachers indirectly increases the amount of training data represented by the teachers, potentially adding a confounder regarding the effects of kernel averaging due to increased data coverage under the specialist setting. This is a reasonable setup but makes it difficult to isolate the benefits of kernel averaging.
- The authors do not fully justify the claim that there is a causal relationship between oracle similarity and performance (eg claimed in lines 92-97). The paper shows that increasing teacher count improves similarity to the oracle kernel and student accuracy (figs 2, 4-5); however, both may improve simply due to more teachers provide additional information. Although the authors investigate weighted kernel mixtures in Figs 3,5c, they don't clearly demonstrate that weighted kernel mixtures with higher oracle similarity lead to better student performance. A direct comparison of similarity between the weighted average kernel & the oracle and student accuracy across different kernel mixtures (fixing other settings) would help support this causal interpretation.
- Observation 1 proof: the minimum gap (lines 778-783) seems to allow negative values, did the authors forget an absolute value?
- Inconsistent definitions of unbiased HSIC: Eq 2 and 19 seem to provide different definitions of the same thing. The authors should clarify which definition is used. Eq 2 seems to have the more common definition.
- DiffKNN limit: lines 709-717 state that the softmax distributions converge to uniform distributions over top-k neighbours as , but softmax should concentrate on the highest-scoring neighbour as rather than becoming uniform.
- Some experimental details need clarification. Fig 4a,4b may be mislabelled, since 4a reaches ~81.3% top-5 accuracy for DiffKNN, corresponding to balanced (and not specialist) ImageNet100 in Table 1 (81.31%). Also, Table 1 caption should be placed above the table.
- Authors give the CKA/MKNN losses (eq 1,4) as the similarity directly, where decreasing numerical value should mean decreasing similarity. However, the authors write the objective in line 229 as CE loss plus similarity, so minimising this loss would lead to lower similarity. Could the authors clarify this?
Although I gave weak accept, the paper is very borderline. I hope the authors can address the weaknesses.
Questions for authors
See weaknesses.
[1] Feature Kernel Distillation. https://openreview.net/forum?id=tBIQEvApZK5
[2] UNIC: Universal Classification Models via Multi-teacher Distillation, https://eccv.ecva.net/virtual/2024/poster/2320
[3] Learning Task-Agnostic Representations through Multi-Teacher Distillation, https://arxiv.org/abs/2510.18680
4 comments
W7: Having more teachers indirectly increases the amount of training data represented by the teachers
Collective data coverage increases in the subset experiments, which are intended to study combining models with complementary data exposure. However, that explanation does not cover all settings. In the multi-architecture experiment, the models are pretrained on the same ImageNet-1K dataset, and Figure 2 still shows improved alignment from averaging. This establishes that expanding the union of teacher-training datasets is not necessary for the observed improvement.
W8: The authors do not fully justify the claim that there is a causal relationship between oracle similarity and performance
The student experiments keep the student’s training data fixed and intervene by adding representation-level teacher guidance. They demonstrate that this guidance can improve the trained student. We will distinguish that result from the stronger claim that oracle similarity itself is an isolated causal mediator of accuracy. Increasing teacher count can change several aspects of the supervision simultaneously, and the present experiments do not isolate those mechanisms. We will revise the causal wording.
W10: Inconsistent definitions of unbiased HSIC:
Apologies, we implement equation 2.
W11: DiffKNN limit: lines 709-717
The stated limit is reversed. For a fixed top- mask, a softmax with similarities divided by becomes uniform over the selected neighbors as , whereas concentrates on the highest-scoring neighbor, or tied maxima. We will correct this limit and distinguish the masked-overlap construction from the teacher-support cross-entropy used for training.
W12: Some experimental details need clarification.
The labels of Figures 4a and 4b are reversed: panel (a) shows balanced ImageNet100, reaching approximately 81.31% top-5 accuracy, and panel (b) shows specialist ImageNet100, reaching approximately 70.81%, consistent with Table 1. We will correct the panel titles and caption associations and move the Table 1 caption above the table.
W13: Similarity loss
Our experiment minimizes a decreasing transform of similarity. In the averaged-kernel implementation, the losses are and , which are added to the task cross-entropy. The individual-teacher CKA implementation uses . Thus, minimizing these objectives favors greater alignment. We will distinguish similarity scores from optimized losses explicitly, correct the missing negative sign in the description, and state the form associated with each reported configuration.
Thanks for the response. I am satisfied with the response to my concern about increasing data coverage, specifically after reviewing the the multi-architecture experiment which uses teachers of different architectures pretrained on the same ImageNet-1K dataset, so increased collective data coverage won't explain this result. I also think this substantially addresses reviewer ejJi's concern that the effect the authors observe is simply a consequence of combining models trained on disjoint subsets.
I am also satisfied with the clarification that the main contribution is the representation-level averaging phenomenon, rather than superiority over existing distillation methods. Given this framing, I think not including additional baselines is ok.
I recommend the authors to do a thorough pass through the manuscript to see if things are consistent/correct. Eg the causal claim still needs to be weakened, lines 229–233 should the correct process for the loss, the statement about logit/multiple distillation (lines 303–307) should be modified etc. I also did not see a response to my W9 about the minimum-gap definition in the proof, but I guess the authors just made a typo in the paper.
Overall, the response resolves many of my substantive concerns, and I'll maintain my weak accept.
Thank you for your thoughtful response. We appreciate that we were able to clarify concerns and will do more passes over the manuscript for consistency and correctness.
Also, re: W9, thank you for catching this. We missed W9 in our rebuttal. The gap in Appendix A.5 can be negative as written. We’ll correct the proof to use the gap between the -th and -st largest similarities, as stated in Observation 1.
We thank the reviewer for their thorough and thoughtful review. We are glad the reviewer found the design of our work flexible, appreciated the prose of our work, and found our experiments well-designed. We hope to address concerns below:
Our contribution concerns the alignment and geometry of averaged representations, measured directly in Figures 2–3. We build on Feature Kernel Distillation and will expand the discussion of UNIC and task-agnostic multi-teacher distillation. Conventional logit distillation is also applicable to our balanced and specialist classification settings, where the label spaces match. We will correct the overly broad statement suggesting otherwise. The current experiments compare representation-guidance objectives and establish useful transfer, but do not establish superiority over all existing distillation algorithms. We will make that distinction explicit.
All experiments average kernels first. We first form the weighted average of the corresponding teacher-layer kernels, and then compute the student’s alignment loss against this averaged kernel. The resulting losses are aggregated across the selected layers. We do not average independently computed teacher losses or assume that these operations are equivalent. The implemented objective is therefore Lsim=∑ℓaℓ,D(KS,ℓ,Kℓ), where aℓ specifies the layer weighting. The phrase “across layers and teachers” at lines 227–230 inaccurately describes this ordering and will be replaced with the explicit objective. The nonlinearity of CKA and DiffKNN does not introduce an inconsistency in the implemented procedure.
As the review notes, Appendix A.9 already extends the representation analysis to language models and to cross-modal comparisons. Figure 7 compares averaged language-model kernels with a stronger language reference, while Figures 8–9 examine language-to-vision and vision-to-language alignment. The evidence for the representation phenomenon therefore extends beyond image classification. Our downstream student-training evidence is limited to vision classification, and we will state that boundary explicitly. Language-generation and diffusion-model experiments would test further applications.
Fair. Observation 3 is a bit broad as stated. Arbitrary dependence does not imply concentration, and a shared random perturbation is a fair counterexample. We will replace it with the conditional statement that, if the averaged kernel converges in probability to a deterministic limiting kernel K† with a positive top-k boundary gap, its neighborhood overlap converges to MKNN(K†,K⋆). This permits systematic bias and dependence satisfying an appropriate law of large numbers, but not unrestricted correlated errors. We can call this the “limiting overlap,” rather than the “best achievable overlap.” The empirical findings do not require assuming that trained models satisfy this condition.
The implemented loss first constructs the averaged teacher kernel (\overline K). For each anchor i, it selects Ti=TopKj=iKij and evaluates both the averaged teacher similarities and the student similarities on this same support. After the configured row standardization and temperature scaling, these define teacher probabilities qi and student probabilities pi. We minimize soft-target cross-entropy, LDiffKNN=−B−1∑i∑j∈Tiqijlogpij, which equals DKL(qi∥pi) plus the fixed teacher entropy, averaged across anchors. Consequently, the independently masked support mismatch described in the review does not arise in the experiments, and the implemented loss is finite for finite logits. Equations 4–5 and the corresponding appendix description will be corrected to match this implementation.
Figure 3 measures oracle alignment over convex combinations of kernels. It does not establish training-loss connectivity between the underlying network parameters or global convexity of the alignment function. The supported conclusion is that kernel mixtures often achieve higher oracle alignment than their constituents. We will replace the stronger basin interpretation in the abstract, introduction, Figure 3 caption, and conclusion with this precise statement. The measured advantages of the evaluated mixtures remain unchanged.