Centered Kernel Distance: Dimension-Free Error Bounds for Comparing Neural Representations

The asymptotic error of representational similarity metrics such as CKA and RSA has been well-characterized, but there are no realistic bounds on how estimates of representational similarity $\text{simultaneously}$ depend on the number of sampled neurons ($N$) and experimental conditions ($M$). We study the natural metric counterpart of CKA and RSA, which we call the Centered Kernel Distance (CKD). We show that CKD can be studied via an unbiased estimator, whereas several popular similarity measures including CKA have no unbiased estimator. Under mild assumptions, we show that estimation error of CKD decays in proportion to $1/M+1/N$. Strikingly, the rate in this bound does not depend on the dimensionality of the space of experimental conditions. Thus, similarity estimation does not suffer from the curse of dimensionality, which plagues similar problems in high-dimensional statistics. We verify our results empirically in random shallow neural networks, trained neural networks, and macaque visual cortex data. Ultimately, this paper advances a framework for studying the error of similarity estimators, and our theoretical and empirical predictions produce specific prescriptions on how to most efficiently estimate the similarity between multiple neural systems.