- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The presented work proposes the Centered Kernel Distance (CKD) to evaluate the similarity between neural representations and studies its theoretical properties, showing that CKD can be studied as an unbiased estimator, and providing an error bound on its variance, depending on the number of sampled neurons and experimental conditions. The experimental evaluation supports the proposed theoretical bound and covers a range of examples comparing one layer neural networks with random weights, deep convolutional networks trained on ImageNet, and neural representations from the THINGS Veneral Stream Dataset across brain regions of different monkeys.
Critical strengths
- The scope of the work to study statistical properties and error bounds of the centered kernel distance is well articulated and described in connection to related work. The authors provide a clear description of results, do appropriately acknowledge substantial LLM use, and with the aid of LLMs provide both relevant theoretical and empirical results, which support their main claims. Overall, I believe this is a solid contribution to understanding the estimation error of the proposed Centered Kernel Distance (CKD).
Critical weaknesses
- The statement that CKD² does not suffer from the curse of dimensionality seems overstated. Just because one can find a dimension-free bound, this might not necessarily imply that the empirical estimate might not still practically be impacted by varying dimensionality or varying numbers of samples.
- Figure 1 only validates the results in relatively low dimensions d=1,5,9 rather than providing extensive verification across higher-dimensional settings.
- The experimental evaluation covers a range of examples including convolutional neural networks trained on ImageNet and monkey brain data. Nevertheless, the evaluation could be strengthened through further empirical evaluation, e.g. by verifying the results across a wider range of image datasets, different neural network architectures, and a wider range of varying dimensionalities. Overall, as it stands the empirical evidence is limited to relatively few experimental settings.
- In general, the scope of the work is focused on proposing CKD and evaluating the error bound in Thm 3.1, which is a solid, but perhaps not a very novel or impactful contribution. This can of course be argued about, but for example the results do not demonstrate why CKD would be a more useful representational similarity metrics than existing methods beyond its theoretical study.
- Further discussion and comparison both in terms of the theoretical properties and empirical performance of existing representational similarity metrics, out of which there exist a wide range of alternative methods (see Klabunde et al. (2025)), would strengthen the motivation of proposing and using CKD in the first place.
- The way the theoretical results are structured could benefit from further formalisation and clear separation into theorems and proofs etc., to clarify which of the statements rely on which assumptions, distinguish novel results, and clarify how each statement is derived.
Minor comment:
- CKA and RSA are first mentioned in the abstract but not defined there.
- The quality and resolution of the illustrations is low and could be improved by e.g. using vector graphics.
- The language is difficult to follow at parts.
- Typo in line 471(?) - “providea”
- missing word in line 474 “, bootstrapping would still be constrained ill-parameterized experiments”
Reference: Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of neural network models: A survey of functional and representational measures. ACM Computing Surveys, 57(9):1–52, 2025.
Questions for authors
- How confident are you really in the statement that your estimator does not at all suffer from the curse of dimensionality in practice?
- How would the results from Figure 1 behave in truly higher dimensional settings, for example d=100 and d=1000?
- Could you verify the results on the image datasets for different neural network architectures and different image datasets?
Thank you for your helpful comments.
Our original manuscript did not clearly define what we mean by the “curse of dimensionality”; we rectify this in our revision (Section 3.2). In high-dimensional statistics, the curse refers to cases where the dimensionality D directly decreases the rate at which the data scale (here, M and N) reduces estimation error. In particular, it refers to cases where one needs to exponentially increase the number of samples to compensate for a linear increase in D. Our results rule this out.
We agree that it is important to characterize how the dimension D enters in the constant C out front of our bound. We address this in detail in our common response (A) to reviewers. In short, we have revised the proof to show that C only depends on the maximal fourth moment of the neural response, which — remarkably — is also bounded independent of the dimension under weak assumptions.
Thus, our results show that the hardness of the problem of estimating CKD stems from the maximal fourth moment of the neural responses rather than the dimensionality of the inputs, per se. Moreover, this quantity only impacts the hardness by a multiplicative constant, it does not impact the rate at which collecting more data (more M or N) improves the estimate.
We are extremely confident that the estimator does not suffer from the curse of dimensionality. As detailed above and in our common response, our bound rules out the possibility that the rate of convergence of the variance degrades with dimension.
In terms of numerical validation, we have updated our Figure 1 on random neural networks, and have updated our analysis in Section 4.1 to accommodate new results regarding large-D simulations in random neural networks.
Our results on natural images further verify our claims, as the dimensionality of the space of natural images is quite large, and much more complicated than random Gaussian inputs. We present new results in Appendix K to show that our results are independent of subclass within the space of natural images as well (see response (C) for details).
We address these concerns in point (C) of our global response to reviewers. As mentioned above, our new Appendix K contains relevant results.
We address these concerns in point (B) of our global response to reviewers.
We do not believe there is a single similarity measure that supercedes all others. Different measures quantify different things. For example, CKD is sensitive to the scale of neural responses while CKA is scale-invariant by design. Different applications or hypotheses will call for different measures in this regard.
We disagree that our paper does not highlight cases where CKD may be more useful. In particular, practitioners who are working with noisy, limited data (e.g. experimental neural recordings) and who want to use a method with theoretical guarantees should consider using CKD. For example, if an unbiased estimator is desired, we have given one for CKD while also proving that no such unbiased estimator exists for Procrustes and CKA (see discussion in Section 3 and proof in Appendix F). Furthermore, our main result on the variance of the CKD estimator provides a guarantee undemonstrated in other measures that error scales independently of D.
Overall, we are aware of very few theoretical results that characterize how similarity measures behave when neurons and inputs are simultaneously subsampled. We focused on CKD because it is an exceptionally clean case with clear answers to this question. However, we ultimately hope that this paper will inspire similar investigations into other metrics.
We have revised the appendices for clarity and provided a guide to read the proof (Appendix A). Please let us know if there are particular sections or notation conventions that are confusing to you. We would be very grateful for specific suggestions that give us the opportunity to continue improving the manuscript.