acceptodds
Under review as a conference paper at ICLR 2027

Average Kernels are Better Kernels

Abstract

If multiple artists are asked to draw a circle by hand, each one will produce something slightly imperfect. Yet, the average of their sketches can look strikingly close to ideal. We investigate whether representations produced by separately trained models adhere to a similar principle. Precisely, we study the average model kernel: the Gram matrix computed from all model embeddings. Using metrics of representational similarity, i.e., Centered Kernel Alignment (CKA) and Mutual k-Nearest Neighbors (MKNN), we show that the average kernel increases in similarity with the kernel of a stronger model. We demonstrate this for vision models trained on different datasets, skewed datasets, models of different architectures, and even models trained on different tasks. Empirically, we even find that the similarity landscape with respect to teacher kernels is a convex basin. Finally, we show how averaging can be directly translated into improved model performance by using DiffKNN, a differentiable version of MKNN, to optimize a student network for representational similarity with the average kernel. We observe that the resulting student model increases in task accuracy with the number of models being averaged. Compared to approaches such as weight-averaging or multiple-distillation, which require that models start from the same checkpoint or share the same architecture, this technique is more general. More broadly, our findings hint at a more general framing that can be given for model-merging techniques, in which models can be thought to lie in the same loss basin with respect to their representations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representational alignment, model merging

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6Mr8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides empirical evidence that the average kernel over many potentially weaker models is more similar to the kernel of a strong "oracle" model as the number of models in the average increase. Additionally, the authors also provide evidence that increasing a student's kernel's similarity to the average kernel improves the student model's performance.

Critical strengths
  1. The authors apply a training method by increasing similarity between average kernel of multiple teachers and the student's kernel. The design is flexible, where the teachers can be specialist/generalist or have different architectures.
  2. The paper identifies that average kernels improve similarity to the oracle kernel.
  3. The paper is written without overly dense prose or inscrutable descriptions of settings. The prose flows naturally.
  4. The experiments are well-designed, easily interpreted, and mostly support the main interpretations.
Critical weaknesses

Major weaknesses

  1. Limited comparisons with other distillation methods. The paper compares DiffKNN, CKA, UCKA, supervised learning, and ensembling. However, there are no comparisons with established distillation methods, eg Feature Kernel Distillation [1]. Other relevant works include UNIC [2], and [3]. Another reasonable baseline would be logit distillation for balanced/specialist settings, where teachers share the same classification tasks.
  2. It is unclear whether the student is trained against the averaged kernel or individual kernels from teachers for Lsim\mathcal{L}_{sim}. Sec 4 (lines 190-198) indicate that student is matched to the average teacher kernel, but lines 227-230 state that similarity loss is averaged across teachers. I am not sure if these are the same since CKA and DiffKNN are generally nonlinear. The authors should clarify this.
  3. Experiments are limited in modalities. The authors focus on CIFAR10/ImageNet100 image classification experiments. Additionally, there are language-model and cross-modal average kernel alignment experiments in appendix A.9. However, there are no experiments involving models for image generation (eg diffusion). This limits the generality of the paper. Additionally, authors could explore how distilling using average kernels could help in language classification/generation tasks and image generation tasks.
  4. Observation 3 doesn't appear valid for arbitrary correlated errors (lines 169-171, appendix A.7). Suppose every teacher shares the same random error, Kk=K∗+εK^k = K^* + \varepsilon, where ε\varepsilon sometimes changes the nearest-neighbour rankings and sometimes doesn't. Then Kˉ=K∗+ε\bar K = K^* + \varepsilon regardless of how many teachers there are, so the neighbourhood overlap need not converge to a fixed value. The authors should consider including additional assumptions on the errors.
  5. The DiffKNN loss may be infinite. Eq 4-5 define Pi,QiP_i,Q_i with 0 probability outside the top-k neighbourhoods. When Pi>0P_i > 0 but Qi=0Q_i = 0, the KL divergence is infinite, and would occur when the neighbourhoods have different supports. The authors should clarify that smoothing or some numerical modification is used in practice for experiments, since the code in the supplemental materials seem to indicate authors defined the loss differently to avoid these numerical issues.
  6. Over-claim in the conclusion (lines 468-483). The authors indicate that "all teacher kernels reside in the same convex basin", although the empirical evidence for this is weaker in multi-architecture/-task settings (the authors themselves admit to this). Even if the similarity landscapes were empirically convex, this does not establish that the underlying models belong to the same training-loss basin. So I think this interpretation in the conclusion is too strong.

Minor weaknesses

  1. Having more teachers indirectly increases the amount of training data represented by the teachers, potentially adding a confounder regarding the effects of kernel averaging due to increased data coverage under the specialist setting. This is a reasonable setup but makes it difficult to isolate the benefits of kernel averaging.
  2. The authors do not fully justify the claim that there is a causal relationship between oracle similarity and performance (eg claimed in lines 92-97). The paper shows that increasing teacher count improves similarity to the oracle kernel and student accuracy (figs 2, 4-5); however, both may improve simply due to more teachers provide additional information. Although the authors investigate weighted kernel mixtures in Figs 3,5c, they don't clearly demonstrate that weighted kernel mixtures with higher oracle similarity lead to better student performance. A direct comparison of similarity between the weighted average kernel & the oracle and student accuracy across different kernel mixtures (fixing other settings) would help support this causal interpretation.
  3. Observation 1 proof: the minimum gap Kmin⁡K_{\min} (lines 778-783) seems to allow negative values, did the authors forget an absolute value?
  4. Inconsistent definitions of unbiased HSIC: Eq 2 and 19 seem to provide different definitions of the same thing. The authors should clarify which definition is used. Eq 2 seems to have the more common definition.
  5. DiffKNN limit: lines 709-717 state that the softmax distributions converge to uniform distributions over top-k neighbours as τ→0\tau\to0, but softmax should concentrate on the highest-scoring neighbour as τ→0\tau\to0 rather than becoming uniform.
  6. Some experimental details need clarification. Fig 4a,4b may be mislabelled, since 4a reaches ~81.3% top-5 accuracy for DiffKNN, corresponding to balanced (and not specialist) ImageNet100 in Table 1 (81.31%). Also, Table 1 caption should be placed above the table.
  7. Authors give the CKA/MKNN losses (eq 1,4) as the similarity directly, where decreasing numerical value should mean decreasing similarity. However, the authors write the objective in line 229 as CE loss plus similarity, so minimising this loss would lead to lower similarity. Could the authors clarify this?

Although I gave weak accept, the paper is very borderline. I hope the authors can address the weaknesses.

Questions for authors

See weaknesses.

[1] Feature Kernel Distillation. https://openreview.net/forum?id=tBIQEvApZK5

[2] UNIC: Universal Classification Models via Multi-teacher Distillation, https://eccv.ecva.net/virtual/2024/poster/2320

[3] Learning Task-Agnostic Representations through Multi-Teacher Distillation, https://arxiv.org/abs/2510.18680

4 comments

We thank the reviewer for their thorough and thoughtful review. We are glad the reviewer found the design of our work flexible, appreciated the prose of our work, and found our experiments well-designed. We hope to address concerns below:

W1: Distillation comparison

Our contribution concerns the alignment and geometry of averaged representations, measured directly in Figures 2–3. We build on Feature Kernel Distillation and will expand the discussion of UNIC and task-agnostic multi-teacher distillation. Conventional logit distillation is also applicable to our balanced and specialist classification settings, where the label spaces match. We will correct the overly broad statement suggesting otherwise. The current experiments compare representation-guidance objectives and establish useful transfer, but do not establish superiority over all existing distillation algorithms. We will make that distinction explicit.

W2: the student is trained against the averaged kernel or individual kernels from teachers for

All experiments average kernels first. We first form the weighted average of the corresponding teacher-layer kernels, and then compute the student’s alignment loss against this averaged kernel. The resulting losses are aggregated across the selected layers. We do not average independently computed teacher losses or assume that these operations are equivalent. The implemented objective is therefore Lsim=∑ℓaℓ,D(KS,ℓ,K‾ℓ)\mathcal L_{\mathrm{sim}}=\sum_\ell a_\ell,D(K_{S,\ell},\overline K_\ell), where aℓa_\ell specifies the layer weighting. The phrase “across layers and teachers” at lines 227–230 inaccurately describes this ordering and will be replaced with the explicit objective. The nonlinearity of CKA and DiffKNN does not introduce an inconsistency in the implemented procedure.

W3: Experiments are limited in modalities.

As the review notes, Appendix A.9 already extends the representation analysis to language models and to cross-modal comparisons. Figure 7 compares averaged language-model kernels with a stronger language reference, while Figures 8–9 examine language-to-vision and vision-to-language alignment. The evidence for the representation phenomenon therefore extends beyond image classification. Our downstream student-training evidence is limited to vision classification, and we will state that boundary explicitly. Language-generation and diffusion-model experiments would test further applications.

W4: Observation 3 doesn't appear valid for arbitrary correlated errors

Fair. Observation 3 is a bit broad as stated. Arbitrary dependence does not imply concentration, and a shared random perturbation is a fair counterexample. We will replace it with the conditional statement that, if the averaged kernel converges in probability to a deterministic limiting kernel K†K^\dagger with a positive top-kk boundary gap, its neighborhood overlap converges to MKNN⁡(K†,K⋆)\operatorname{MKNN}(K^\dagger,K^\star). This permits systematic bias and dependence satisfying an appropriate law of large numbers, but not unrestricted correlated errors. We can call this the “limiting overlap,” rather than the “best achievable overlap.” The empirical findings do not require assuming that trained models satisfy this condition.

W5: The DiffKNN loss may be infinite

The implemented loss first constructs the averaged teacher kernel (\overline K). For each anchor ii, it selects Ti=TopK⁡j≠iK‾ijT_i=\operatorname{TopK}{j\ne i}\overline K{ij} and evaluates both the averaged teacher similarities and the student similarities on this same support. After the configured row standardization and temperature scaling, these define teacher probabilities qiq_i and student probabilities pip_i. We minimize soft-target cross-entropy, LDiffKNN=−B−1∑i∑j∈Tiqijlog⁡pij\mathcal L_{\mathrm{DiffKNN}}=-B^{-1}\sum_i\sum_{j\in T_i}q_{ij}\log p_{ij}, which equals DKL(qi∥pi)D_{\mathrm{KL}}(q_i\Vert p_i) plus the fixed teacher entropy, averaged across anchors. Consequently, the independently masked support mismatch described in the review does not arise in the experiments, and the implemented loss is finite for finite logits. Equations 4–5 and the corresponding appendix description will be corrected to match this implementation.

W6 Over-claim in the conclusion (lines 468-483)

Figure 3 measures oracle alignment over convex combinations of kernels. It does not establish training-loss connectivity between the underlying network parameters or global convexity of the alignment function. The supported conclusion is that kernel mixtures often achieve higher oracle alignment than their constituents. We will replace the stronger basin interpretation in the abstract, introduction, Figure 3 caption, and conclusion with this precise statement. The measured advantages of the evaluated mixtures remain unchanged.

W7: Having more teachers indirectly increases the amount of training data represented by the teachers

Collective data coverage increases in the subset experiments, which are intended to study combining models with complementary data exposure. However, that explanation does not cover all settings. In the multi-architecture experiment, the models are pretrained on the same ImageNet-1K dataset, and Figure 2 still shows improved alignment from averaging. This establishes that expanding the union of teacher-training datasets is not necessary for the observed improvement.

W8: The authors do not fully justify the claim that there is a causal relationship between oracle similarity and performance

The student experiments keep the student’s training data fixed and intervene by adding representation-level teacher guidance. They demonstrate that this guidance can improve the trained student. We will distinguish that result from the stronger claim that oracle similarity itself is an isolated causal mediator of accuracy. Increasing teacher count can change several aspects of the supervision simultaneously, and the present experiments do not isolate those mechanisms. We will revise the causal wording.

W10: Inconsistent definitions of unbiased HSIC:

Apologies, we implement equation 2.

W11: DiffKNN limit: lines 709-717

The stated limit is reversed. For a fixed top-kk mask, a softmax with similarities divided by τ\tau becomes uniform over the selected neighbors as τ→∞\tau\to\infty, whereas τ→0\tau\to0 concentrates on the highest-scoring neighbor, or tied maxima. We will correct this limit and distinguish the masked-overlap construction from the teacher-support cross-entropy used for training.

W12: Some experimental details need clarification.

The labels of Figures 4a and 4b are reversed: panel (a) shows balanced ImageNet100, reaching approximately 81.31% top-5 accuracy, and panel (b) shows specialist ImageNet100, reaching approximately 70.81%, consistent with Table 1. We will correct the panel titles and caption associations and move the Table 1 caption above the table.

W13: Similarity loss

Our experiment minimizes a decreasing transform of similarity. In the averaged-kernel implementation, the losses are −log⁡(CKA-\log(\mathrm{CKA} and −log⁡[(1+UCKA)/2]-\log[(1+\mathrm{UCKA})/2], which are added to the task cross-entropy. The individual-teacher CKA implementation uses 1−CKA1-\mathrm{CKA}. Thus, minimizing these objectives favors greater alignment. We will distinguish similarity scores from optimized losses explicitly, correct the missing negative sign in the description, and state the form associated with each reported configuration.

Thanks for the response. I am satisfied with the response to my concern about increasing data coverage, specifically after reviewing the the multi-architecture experiment which uses teachers of different architectures pretrained on the same ImageNet-1K dataset, so increased collective data coverage won't explain this result. I also think this substantially addresses reviewer ejJi's concern that the effect the authors observe is simply a consequence of combining models trained on disjoint subsets.

I am also satisfied with the clarification that the main contribution is the representation-level averaging phenomenon, rather than superiority over existing distillation methods. Given this framing, I think not including additional baselines is ok.

I recommend the authors to do a thorough pass through the manuscript to see if things are consistent/correct. Eg the causal claim still needs to be weakened, lines 229–233 should the correct process for the loss, the statement about logit/multiple distillation (lines 303–307) should be modified etc. I also did not see a response to my W9 about the minimum-gap definition in the proof, but I guess the authors just made a typo in the paper.

Overall, the response resolves many of my substantive concerns, and I'll maintain my weak accept.

Thank you for your thoughtful response. We appreciate that we were able to clarify concerns and will do more passes over the manuscript for consistency and correctness.

Also, re: W9, thank you for catching this. We missed W9 in our rebuttal. The gap in Appendix A.5 can be negative as written. We’ll correct the proof to use the gap between the kk-th and (k+1)(k+1)-st largest similarities, as stated in Observation 1.

Reviewer_ejJiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper finds that the averaged representation of several weaker models is stronger than each individual model, and this representation is more similar to that of a strong model. In addition, internally distilling a student model on this averaged representation improves the accuracy of this student model. The context is using vision models - ResNet models - on CIFAR10 and 100 classes from ImageNet.

Critical strengths

The paper’s motivation is a very interesting problem, about how we think about model representations, and how averaged representation from several models can be different, potentially stronger in certain aspects, and potentially applicable. There is a natural and coherent flow from formulating and finding such averaged representation, to using that as a teacher signal to do distillation and observe student model progressions.

Another strength is that the paper is well-presented, flows easily, and even enjoyable to read.

Critical weaknesses

Unfortunately there are empirical evaluation weaknesses that make it difficult to assess the claim that averaged representations improves in quality.

An experimental design is that student models M_1,..., M_n are trained on distinct small subsets of a larger dataset (CIFAR10, ImageNet). Since they each only see a subset, it is rather unsurprising that their combined representations are stronger than the individual ones. It also becomes natural that this combined representation is closer than the individual ones to the oracle model representation, which is trained on a fuller set.

The way the paper computes kernels and conducts internal distillation is based on previous papers’ methods. This is not by itself an issue at all. However, these aspects make the takeaway and overall contribution of this paper rather limited.

I find the problem that this paper tackles to be interesting. Directions I’m curious about is whether this “averaged representation being higher quality” applies to models with exposure to similar data (as opposed to exposure to different subsets of a larger data). For example, whether the average of several weaker or smaller models with similar data can have higher quality and be closer to a stronger and larger model. Other directions that might help strengthen the paper is, even if we are just given a situation where averaged representations are helpful when individual models see different subsets of data, where can this insight nevertheless be still applied to.

(edit: some concerns are clarified in the rebuttal)

Questions for authors

The main clarifications I would like to see are mentioned in "Weaknesses".

1 comment

We thank the reviewer for their thoughtful review. We appreciate that the reviewer found the paper to study an interesting problem and found the flow of the paper coherent and natural.

W1: Do results extend beyond disjoint subsets?

The setting requested in the review is already included in the submission. In the multi-architecture experiment, the teachers are ResNet, ViT, and ConvNeXt models pretrained on the same ImageNet-1K dataset, with ConvNeXt-Small serving as the stronger reference. Figure 2 evaluates their averaged kernel directly and shows improved alignment with that reference. Thus, expanding the union of disjoint teacher-training datasets cannot explain this result. Figure 4 additionally compares ensemble guidance with guidance from the strongest individual teacher. The subset experiments study complementary data exposure, while this common-dataset experiment shows that the observation also extends to architectural diversity. We will foreground this distinction, because the characterization of our evidence as limited to ResNets trained on distinct subsets omits a central experimental setting.

W2: Contribution of our paper

We explicitly build on existing methods for computing and distilling representations. The contribution is the empirical investigation of the aggregate representation: its alignment with a stronger reference across different sources of teacher diversity, the behavior of kernel mixtures, and the relationship between these observations and useful student supervision. These analyses are not implied merely by observing that an output ensemble outperforms an individual classifier. In particular, Figures 2–3 examine the representation itself before any student is trained.

W3: Practical Relevance

The student experiments demonstrate a concrete use: a model trained on a fixed, limited dataset can benefit from representations supplied by other models without pooling their original training datasets. For example, specialist CIFAR-10 accuracy increases from 55.68% to 65.31% in Table 1 while the student’s training subset remains fixed. The heterogeneous settings also permit representation transfer when feature dimensions or task-specific outputs differ. We present these results as evidence of useful transfer under restricted data access, while distinguishing that evidence from a claim to have benchmarked a complete federated-learning system. We ask that the empirical assessment take the existing common-dataset experiment into account, since the concern that all improvements arise only from increased data coverage does not apply to the full set of results.

Reviewer_2zYWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether averaging the representation kernels of independently trained models produces a stronger representation. The authors report increased alignment with stronger reference models across different training distributions, architectures, and objectives. They also use the aggregated structure to guide student training and show improvements over students trained on individual data subsets. These observations are interesting, and the range of experimental settings is a strength. However, the advantage over existing ensemble-distillation methods remains unclear. The experiments also do not establish whether averaging enough subset-trained models can recover the representation learned through full-data training. Finally, alternative aggregation rules are not explored, leaving open whether the observed convergence reflects the accumulation of complementary knowledge or the recovery of structure already shared across teachers.

Critical strengths
  1. The paper asks a clear and interesting question about representation learning. Studying whether independently trained models can collectively recover a stronger similarity structure is a natural extension of work on representational convergence. The observation that an averaged kernel can align more closely with a stronger model than its individual constituents provides a useful starting point for understanding how knowledge is distributed across models.

  2. The experiments cover several meaningful forms of model diversity. The authors consider balanced and specialist training distributions, different architectures, and different training objectives. This breadth makes the observation more compelling than a result limited to different random seeds of one architecture. Kernel-based aggregation also provides a convenient common interface for models whose feature dimensions and prediction spaces differ.

  3. The paper connects representational observations to downstream learning. The authors go beyond measuring similarity to an oracle by using the aggregated structure to guide student training. Table 1 reports substantial improvements in some settings—for example, specialist CIFAR-10 accuracy increases from 55.68% to 65.31%. These results provide practical evidence that the aggregated relationships contain useful information that a single student can learn.

Critical weaknesses

1.The specific advantage over existing ensemble-distillation methods remains unclear. The averaged-kernel guidance improves the student’s accuracy compared with training on a single subset. However, the student still falls well short of the model trained on the full dataset. The results therefore do not show that combining separately trained models can recover the benefits of training on all the data together. This does not make the approach unhelpful, but it leaves me wondering what advantage kernel averaging offers over other ways of combining teachers.

This is especially relevant given the existing work on ensemble distillation. Feature Kernel Distillation already considers multiple teachers, and FedHeNN uses CKA alignment to an averaged kernel across different architectures. I would like to see direct comparisons with these methods and conventional multi-teacher distillation, using the same student and training data. Those comparisons would help establish whether the gains come specifically from averaging kernels or mainly from giving the student access to additional teachers.

2.The convergence claim needs a more direct test. The current experiments show that averaging more teacher kernels improves alignment with the full-data model, but it remains unclear whether this process can eventually recover the benefits of full-data training. I would like to see an experiment that repeatedly draws random, potentially overlapping subsets of a fixed dataset, trains an independent model on each subset, and averages their kernels as the number of teachers grows. Keeping each subset’s size fixed and continuing after collective data coverage has largely saturated would help distinguish increased coverage from the effect of averaging additional models. Does the average approach the full-data model, or does a substantial gap remain? Repeating this experiment at several subset sizes would also reveal whether each teacher needs sufficient data individually. Evaluating the averaged kernel directly, before student distillation, would make the result easier to interpret.

  1. It is unclear whether the phenomenon depends on additive averaging. Have the authors considered other ways of combining kernels, such as appropriately defined geometric means or product kernels? Arithmetic averaging allows strong support from one teacher to compensate for weak support from others. For nonnegative similarities, multiplicative aggregation instead emphasizes relationships supported consistently across teachers. Comparing these behaviors could clarify what drives the observed convergence: is averaging collecting complementary knowledge from different models, or recovering a structure they already share? This distinction seems important for interpreting the results as evidence for a common representation.
Questions for authors

See weakness.

1 comment

We thank the reviewer for their thoughtful review. We are glad the reviewer found our paper clear and interesting, appreciated the diversity of models we study, and our connections to downstream learning. We address concerns.

W1: Relationship to ensemble distillation and the gap to full-data training.

Our contribution is a finding about representational convergence. Averaging fixed or independently trained models can produce a kernel whose geometry is closer to a stronger reference, before any student is trained. This is a different claim from improving ensemble-distillation accuracy. FKD and FedHeNN use teacher kernels as supervision. Specifically, successful student training alone does not establish that the aggregate has greater oracle alignment than its constituents on average. Figures 2–3 investigate that property directly, across teacher counts, mixture weights, architectures, and training objectives. The shared-data ImageNet-1K setting additionally demonstrates that the effect extends beyond combining disjoint training datasets. These measurements cannot be explained as a consequence of giving a student additional teachers: no student is involved in them. Table 1 then tests whether the aggregate’s structure provides useful supervision, with the student architecture and available training data held fixed. Matching the full-data oracle would establish complete recovery of centralized training. This is a substantially stronger result than either improved representational alignment or useful transfer. The gap in accuracy does not reduce the strength of the previous findings. We will revise the introduction and related-work discussion to distinguish the direct representation-level findings from their downstream distillation application.

W2: Convergence, data coverage, and recovery of the full-data model.

Fair. We distinguish empirical improvement toward a stronger reference from guaranteed recovery of full-data training. The latter does not automatically follow from averaging alone. Under a law of large numbers, an average approaches the mean teacher kernel, which need not equal the kernel learned from the full dataset. Our theoretical recovery statement requires an unbiased-target assumption, while our paper explicitly acknowledges persistent gaps and plateaus. Furthermore, the multi-architecture experiment already provides evidence beyond increasing collective data coverage. The models in Figure 2c are pretrained on the same ImageNet-1K dataset (see Appendix Section A.1). Figure 2 also evaluates averaged kernels directly, independently of student distillation, as requested. The proposed overlapping-subset sweep would further characterize scaling within a fixed architecture, but is not required to establish the finite-ensemble improvements reported here. We will revise the wording so that increased alignment is not conflated with universal recovery of training on the full data.

W3: Alternative aggregation rules and the interpretation of averaging.

We have not evaluated multiplicative aggregation. Arithmetic averaging has a concrete interpretation in our setting i.e. nonnegative weighted average of kernels is the Gram matrix of concatenated, appropriately scaled teacher features. It also corresponds directly to the mean-kernel model. Product kernels are valid alternatives, while geometric constructions require their own definition and assumptions. Our experiments establish that arithmetic averaging improves alignment in the studied settings, without claiming that it is optimal among aggregation rules. Shared structure and complementary teacher information can both contribute to this improvement. We will therefore clarify that the results support the usefulness of arithmetic averaging without uniquely identifying one mechanism or excluding other aggregators.

Comments on the paper

1 comment

We thank the reviewers for their careful reading and discussion. We summarize our responses here.

  • Our central contribution is evidence that averaging independently learned kernels can improve their alignment with a stronger reference before any student is trained.

  • Figures 2–3 study this representation-level phenomenon directly. The student experiments then demonstrate that the aggregate provides useful supervision with the student architecture and available training data held fixed.

    • The evidence extends beyond disjoint-data ensembles. The multi-architecture models share the same ImageNet-1K training dataset, and Appendix A.9 includes language and cross-modal alignment experiments.
  • The remaining gap to full-data accuracy concerns complete recovery of centralized training and does not negate the measured improvements in alignment or transfer.

  • All reported guidance experiments average teacher kernels before applying the alignment loss.

  • Our responses below identify the relevant existing evidence and specify corrections to the mathematical statements and implementation descriptions.

We will make targeted revisions, marked in blue, to state the contribution, assumptions, and supported interpretations precisely.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.