- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the statistical reliability of representation similarity metrics under finite samples. The authors take population similarity as the target of inference and analyze consistency and concentration guarantees for TSI, QSI, and CKA-RBF under the stated assumptions. They further develop a one-sided hypothesis test for comparing population similarities. The authors also construct a counterexample showing that MutualNN can be highly unreliable under finite samples. The framework is further evaluated on image-text representations from CLIP, SigLIP, and SigLIP2.
Critical strengths
- The paper is clearly written and well organized. It addresses a broadly relevant problem concerning the statistical reliability of representation similarity measures.
- The counterexample constructed for MutualNN is particularly interesting and thought-provoking. It reveals a non-obvious limitation of finite-sample similarity estimation and raises interesting questions for further investigation.
Critical weaknesses
-
Limited theoretical novelty. Statistical testing has already been explored in the context of representation similarity [1]. Standard hypothesis testing alone is insufficient as a contribution to a machine learning conference. Regarding the theoretical results, the concentration bounds for TSI and QSI largely follow from existing U-statistic theory and prior work [2], while concentration results for kernel alignment have also been established [3]. The proposed hypothesis testing framework primarily relies on these existing results and standard probability inequalities, offering limited new theoretical insights.
-
Insufficient coverage for a benchmarking study. A wide range of representation similarity measures have been proposed and systematically reviewed in prior work [4]. Since this paper introduces neither a new similarity measure nor a novel statistical testing technique, I view it primarily as a benchmarking study of the statistical properties of existing measures. From this perspective, analyzing only four metrics is insufficient (especially given that three of them have already been studied theoretically and most of the results follow from standard concentration inequalities). I believe a comprehensive examination of commonly used measures is necessary, distinguishing those whose statistical properties follow directly from established theory from those that present genuine theoretical challenges, such as MutualNN. Without such a systematic investigation, I find the current contribution insufficient for acceptance.
In summary, the two major weaknesses discussed above would require substantial revisions, and I initially leaned toward a clear rejection. However, given the novelty and thought-provoking nature of the MutualNN counterexample, I am inclined to give this paper a weak rejection. I suggest that the authors build upon this counterexample and existing statistical theory to provide a more comprehensive and systematic analysis of representation similarity measures.
References
[1] Grounding representation similarity through statistical testing. 2021
[2] Scalable and interpretable representation alignment with ordinal similarity. 2026
[3] Algorithms for learning kernels based on centered alignment. 2012
[4] ReSi: A Comprehensive Benchmark for Representational Similarity Measures. 2025
Questions for authors
I have no further technical questions regarding the theoretical analysis and experiments for the four selected similarity measures. My main concerns relate to the novelty and scope of the work, as discussed in the Weaknesses section.
4 comments
Response to Reviewer WpX8 (2/2)
Regarding the theoretical results, the concentration bounds for TSI and QSI largely follow from existing U-statistic theory and prior work [2], while concentration results for kernel alignment have also been established [3].
We agree that the / bounds follow from -statistic theory [2] and that the bound builds on [3]. These connections were stated in the proofs, and now we also made them explicit in the main text, with a note on how these distribution-free bounds build on existing work, and minor edits elsewhere in the paper (See L322-325).
3. Further Investigation of MutualNN's Arbitrarily Slow Convergence in Euclidean Spaces
The counterexample constructed for MutualNN is particularly interesting and thought-provoking. It reveals a non-obvious limitation of finite-sample similarity estimation and raises interesting questions for further investigation.
Since the MutualNN counterexample proved so illustrative, we also added one with the Euclidean distance, which is widely used in practice and shows that the slow convergence is not limited to handpicked metric spaces (New Appendix G L1998-2198).
4. Linear CKA Can Converge Arbitrarily Slowly to Its Population Similarity
[...] examination of commonly used measures is necessary, distinguishing those [...] that present genuine theoretical challenges, such as MutualNN.
Following the reviewer's suggestion to examine commonly used metrics that pose genuine theoretical challenges, as MutualNN does, we show that CKA with a linear kernel, as opposed to CKA with a RBF kernel, does not admit non-trivial distribution-free tail bounds. (New Appendix H L2200-2344).
5. Not Only a Benchmarking Study
Insufficient coverage for a benchmarking study. [...] I view it primarily as a benchmarking study of the statistical properties of existing measures.
- Paper Positioning Although, we acknowledge that parts of our work provide a comparative analysis of different similarity metrics, we do not see our work as solely a benchmarking study. Our paper serves other purposes:
- Establishing Population Similarity as the Target of Representation Alignment We define population similarity as the infinite-sample limit of an empirical score and motivate why that limit, rather than a score on a small batch, is the quantity a comparison should target. Empirical rankings at standard sample sizes can reverse the population ranking. We then prove that any bounded metric that converges in probability, including , , and , has a well-defined population similarity and is consistent.
- Definitions and Theorems that Apply Directly to other Metrics in General The paper is organized as a set of reusable conditions rather than a study of four named metrics. The definition of distribution-free tail bounds, the existence result for population similarity, the new existence result for distribution-free bounds, and the -value upper bound are all stated for an arbitrary similarity metric. A new metric is covered once it is checked against these conditions: boundedness and convergence in probability for existence and consistency, outlier robustness and uniform asymptotic unbiasedness for a non-trivial tail bound, and that checked against the tail bound for the population test. No new testing argument is required.
We thank Reviewer WpX8 for feedback that nudged us to derive an important theoretical contribution examining sufficient conditions for the existence of distribution-free bounds. We look forward to the discussion.
[1] Ding et al., Grounding representation similarity through statistical testing, NeurIPS 2021
[2] Soares et al., Scalable and interpretable representation alignment with ordinal similarity, ICML 2026
[3] Cortes et al., Algorithms for learning kernels based on centered alignment, JMLR 2012
[4] Davari et al., Reliability of cka as a similarity measure in deep learning, ICLR 2023
[5] Quadrianto et al., Kernelized sorting, NeurIPS 2008
Thank you for the detailed response. The main additions to the revised manuscript are Theorem 1 and two counterexamples. Theorem 1 follows directly from McDiarmid’s inequality. I therefore believe that the paper’s novelty still lies in these counterexamples.
Given the limited novelty, I believe that a thorough discussion of the statistical properties of commonly used representational similarity measures could itself make a valuable contribution. However, the current revision still falls well short in this respect.
I therefore maintain my original recommendation.
We thank Reviewer WpX8 for their feedback.
Theorem 1 follows directly from McDiarmid’s inequality. I therefore believe that the paper’s novelty still lies in these counterexamples.
While the proof of Theorem 1 builds on McDiarmid’s inequality, the importance and novelty of the theorem lies in its generality and practical implications rather than the proof technique itself. The theorem shows that two simple conditions, robustness to outliers and uniform asymptotic unbiasedness, are sufficient to establish a sub-Gaussian distribution-free tail bound. Importantly, these conditions are not specific to any particular similarity metric or data distribution, but apply to broad classes of metrics. This provides a general criterion for establishing when population-level similarity conclusions can be trusted, and gives practitioners a concrete way to assess whether a similarity metric supports statistically reliable comparisons.
Given the limited novelty, I believe that a thorough discussion of the statistical properties of commonly used representational similarity measures could itself make a valuable contribution. However, the current revision still falls well short in this respect.
The framework already covers the two most commonly used metrics, CKA and MutualNN, as well as recent ones such as TSI and QSI [1,2,3,4]. It provides a way to decide when population-level similarity conclusions can be trusted, which is the question alignment studies need answered.
We thank the reviewer again for their comments, which have helped improve our paper.
[1] Kornblith et al., Similarity of neural network representations revisited, ICML 2019
[2] Huh et al., The platonic representation hypothesis, ICML 2024
[3] Soares et al., Scalable and interpretable representation alignment with ordinal similarity, ICML 2026
[4] Schnaus, et al., Shared geometry as a Rosetta stone: Cross-modal alignment without paired data, Arxiv 2026
Response to Reviewer WpX8 (1/2)
We thank Reviewer WpX8 for their insightful comments. In particular, the suggestion to build on 'statistical theory to provide a more comprehensive and systematic analysis of representation similarity measures' led us to derive sufficient conditions for the existence of distribution-free bounds, a substantial theoretical result that applies to broad classes of metrics.
1. On the Novelty and Importance of Population Level Statistical Testing
2. Extensive and Systematic Analysis of Similarity Metrics' Distribution-Free Bounds
We thank the reviewer for their suggestions that led us to develop a more general theorem of the existence of distribution-free bounds.
(continues ...)