- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this work, the authors argue towards multi-scale similarity analysis. From a theoretical perspective, the authors motivate why certain scales offer limited guidance, e.g. 0, , and provide a useful way to compute multi-scale similarity: by averaging CKA with RBF kernel with varying sigma. The authors validate their framework across controlled synthetic setups, learned representations with gLocal transformations, and a systematic evaluation on the ReSi benchmark spanning vision, language, and graph domains.
Contributions:
- Using CKA with RBF kernel (both normalized and unormalized) with varying sigmas, the authors provide a function that yields a vector representing multi-scale similarity.
- The authors show that this function can be transformed into a traditional similarity metric, i.e. with output between instead of , by simply averaging this vector. The authors establish a connection with the normalized area under the curve.
- Through experiments, the authors are able to motivate that representation alignment can change substantially between scales motivating a multi-scale analysis.
Critical strengths
- Raising awareness for the pitfalls of single-scale similarity analysis.
- Offering an alternative multi-scale metric that builds upon well-known similarity metric like CKA.
- Statistically significant performance increase over other similarity metrics in an externally designed comprehensive benchmark (RESI)
Critical weaknesses
- The paper misses important literature: [1] already analyses the impact of single-scale similarity measurement by selecting a given k-nearest neighborhood to measure similarity with MutualNN. More importantly, other metrics [2] already establish a connection between ordinal similarity and multiple k-nearest neighborhoods (see Section 4.3), directly refuting the claim
Local and global approaches therefore capture complementary aspects of representational alignment, emphasizing neighborhood-level structure and broader organization, [...] yet existing measures typically evaluate similarity at a single predefined scale.
- I would stress that I find this phrasing problematic, since even metrics already analyzed in the paper like CKA, SVCCA, Procrustes, do not ignore information captured at neighborhood-level structure and are truly multi-scale. They however have an implicit weighted aggregation that benefit broader organization scales more heavily, and this is where the metric introduced in this paper differs.
- The computational time complexity associated with computing AUC() can be m times the complexity of computing , which is already a significant bottleneck in many practical applications as displayed by the fact that most practitioners use batches as small as 1000 datapoints, e.g. the Platonic Representation Hypothesis paper.
- The authors introduce a m-dimensional multi-scale similarity function, but its use is not dissimilar to simply performing the CKA computation across m scales and then evaluating each score individually as there is no direct quantitative use of the vector directly. This poses the question of whether such a vector is truly interpretable/useful.
- It is unclear to me if the conceptual leap between this work and previous work is significant, since to a certain extent previous work had already analyzed CKA computations with different sigma selections, e.g. [3]. I suppose the authors would argue that the performance showed in section 4.3 shows a meaningful performance increase in benchmarks that practitioners care about, which is reasonable. Nevertheless, the methodological leap from using CKA to averaging CKA across scales is somewhat direct.
[1] Koepke, A., Zverev, D., Ginosar, S., & Efros, A. A. (2026). Back into Plato's cave: examining cross-modal representational convergence at scale. arXiv preprint arXiv:2604.18572.
[2] Soares, D., Gawade, P., Dittadi, A. & Szczurek, E.. (2026). Scalable and Interpretable Representation Alignment with Ordinal Similarity. Proceedings of the 43rd International Conference on Machine Learning
[3] Davari, M., Horoi, S., Natik, A., Lajoie, G., Wolf, G., & Belilovsky, E. (2022). Reliability of cka as a similarity measure in deep learning. arXiv preprint arXiv:2210.16156.
Questions for authors
Analyze the time complexity of computing AUC(), is there any trick to make this computation comparable to the computation of Or is there always a multiplicative factor associated with computing this m times?
From the current notation, it seems the sigma is fixed for both and , this does not seem ideal as they might have completely different nominal scales. Moreover, it is unclear to me how are selected, I assume they are picked as a function of the median, but this should be made explicit. If this is not the case then there are challenges with invariance to isotropic scaling. (Edit: I now realize that the authors report in the text that the pairwise distances are normalized by the median, this could definitely be improved to make it easier for the reader to understand.)
In the experiments, could you also benchmark the recent multi-scale metrics proposed in [2]?
Could you elaborate on AUC() robustness to outliers? This is a critical requirement of similarity metrics as motivated by [3]
3 comments
The authors address most of my concerns, namely:
- They perform a computational complexity analysis that justifies that the computation of MS-Signal is not asymptotically larger than CKA-RBF (for smaller than and ).
- They show that averaging the CKA-RBF across scales reduces the effect of outliers, a known problem of linear CKA or even CKA-RBF for certain scales.
- They add recently proposed TSI and QSI to the ReSi evaluation to ensure fair comparison across metrics in the literature.
However, some concerns remain:
- I find the main methodological idea to be averaging CKA-RBF across scales, which is incremental from a conceptual perspective. I agree with the authors that looking at different scale grids or potentially performing a weighted average is conceptually interesting, but this is not really explored in this paper.
- I was not convinced by the authors response with respect to this limitation
The authors introduce a m-dimensional multi-scale similarity function, but its use is not dissimilar to simply performing the CKA computation across m scales and then evaluating each score individually as there is no direct quantitative use of the vector directly. This poses the question of whether such a vector is truly interpretable/useful.
Taking this into account I decided to raise my score to Weak Accept
Official Response by Authors
Thank you for taking the time to assess the revision. We are glad the revision addressed most of your concerns, and we appreciate your updated assessment of Weak Accept.
To address your remaining points, we view the MS-Signal itself as our main contribution, providing a general framework for measuring and analyzing representational similarity as a function of scale. Figures 1 and 2 illustrate direct uses of the signal, showing how trends in alignment across scales help interpret the effects of representational transformations. We view AUC aggregation as one example of how the framework can be used, and the ReSi results illustrate the practical value of this summary across domains. We agree that developing further analyses of the full signal and exploring alternative weighting schemes are worthwhile directions for future work.
Rebuttal by Authors
We thank the reviewer for their thoughtful feedback. We have revised the introduction and appendix to clarify our contribution, and expanded the experiments to strengthen its evaluation.
Critical Weaknesses
We thank the reviewer for highlighting these relevant works. We expanded the Introduction to discuss Koepke et al. (2026). We added Appendix F, which discusses existing multiscale approaches, including TSI, QSI, IMD, and RTD, and compares them with our framework. We acknowledge the multiscale information incorporated by these measures while clarifying our complementary approach: explicitly characterizing kernel-based alignment as a function of scale, enabling analysis of scale-dependent similarity patterns and flexible aggregation over specified scale ranges and weightings.
We agree that global measures are not entirely insensitive to local structure. However, their similarity scores primarily reflect broader representational organization, as also illustrated by the nonzero similarity observed between independent high-dimensional representations (Murphy et al., 2024; Gröger et al., 2026). Our distinction is that our MS-Signal directly measures how alignment varies across scales, rather than summarizing these effects into a single score. Moreover, its aggregation allows explicit control over the contribution of different scales. We have clarified this distinction in the introduction and Section 3.3.
We added a computational complexity analysis in Section 3.4. By reusing pairwise distance matrices across bandwidths, the overall complexity is O(n2(pX+pY+m)), rather than m times the cost of RBF-CKA, with O(n2) memory.
We have added a discussion on MS-Signal's interpretability at the end of Section 3.3. Rather than examining individual bandwidths in isolation, MS-Signal allows us to observe the signal dynamics across scales, providing context for interpreting changes in alignment. Moreover, the complete signal enables further analysis beyond its AUC aggregation. In Figure 1c, we use MDS to visualize distances between similarity profiles, revealing distinct patterns associated with local and global transformations that are obscured by scalar measures.
We agree that evaluating and averaging RBF-CKA across bandwidths is mathematically straightforward. However, selecting an appropriate bandwidth remains challenging. Kornblith et al. (2019) reported similar results for RBF-CKA and linear CKA under their chosen bandwidths; this may partly reflect the convergence of RBF-CKA to linear CKA at large bandwidths (Alvarez, 2022). Other studies have demonstrated substantial variation in RBF-CKA results across bandwidths (Nguyen et al., 2022; Gröger et al., 2026), highlighting the difficulty of selecting a single bandwidth in advance. Our multiscale aggregation reduces dependence on individual bandwidth choices, with more consistent performance across domains on the ReSi benchmark, as demonstrated in Appendix D.3, where we compare different single-bandwidth choices. Our meaningful performance on ReSi, which the reviewer kindly acknowledges, demonstrates the practical value of this approach.
Questions for Authors
As detailed in Section 3.4, MS-Signal reuses pairwise distance matrices across all bandwidths. Computing these matrices costs O(n2(pX+pY)), followed by O(mn2) for the m kernel alignments. The total complexity is therefore O(n2(pX+pY+m)), with O(n2) memory when processing bandwidths sequentially.
We have clarified the relative bandwidth normalization in Section 3.4. Each representation uses its own median pairwise distance to define a common relative scale, ensuring invariance to independent isotropic rescaling. We also describe the bandwidth grid selection and examine its sensitivity in Appendix D.
We have added both TSI and QSI to the ReSi evaluation in Section 4.3 (Figure 3). TSI performs particularly well, ranking first in Graphs and Language under both the 90th-percentile and mean-rank summaries. Its strong performance further supports the value of multiscale information in representational similarity.
Our AUC-CKA score achieves more consistent cross-domain performance, ranking first in Vision and second in Graphs and Language by 90th-percentile rank. Appendix D.4 further examines alternative aggregation parameterizations. Together, these results highlight the potential of different approaches to incorporating information across scales and motivate further research in this direction.
We added an outlier sensitivity experiment in Appendix D.1 (Figure 9). Davari et al.'s theoretical results primarily concern linear CKA, and our experiments show that isolated outliers mainly affect the large-scale region of the CKA signal, consistent with its linear CKA limit. Furthermore, UKA exhibits reduced sensitivity to these perturbations. These findings also illustrate how the MS-Signal reveals the scale-dependent effects of outliers.