acceptodds
Under review as a conference paper at ICLR 2027

Similarity as a Signal: Comparing Representations Across Multiple Scales

Abstract

Representational similarity measures provide a quantitative assessment of alignment between representations learned by different models, modalities, and training procedures. Existing measures typically characterize similarity at a single predefined scale, although different aspects of alignment may be reflected in structure ranging from local neighborhoods to broader global organization. Intuitively, local neighborhood structure may be important for retrieval or instance-level matching, whereas broader geometric organization may capture higher-level semantic relationships. We introduce a multiscale framework that treats representational similarity as a function of scale rather than as a single scalar. Motivated by the connection between RBF kernels and heat diffusion, we use the RBF bandwidth as the scale parameter and introduce the multiscale similarity signal to characterize representational alignment across scales. On controlled synthetic data, we show that perturbations at local, intermediate, and global scales produce distinct signatures in the signal. Moreover, on learned neural representations, the signal characterizes more complex transformations spanning a broad range of scales. Finally, we derive scalar scores by aggregating similarity across scales and evaluate them on the ReSi benchmark spanning graph, vision, and language domains, where our primary multiscale score achieves the best 90th-percentile rank in all three domains. Our results show that viewing representational similarity across multiple scales provides both an interpretable tool for analyzing representation geometry and an effective framework for quantifying representational similarity.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representational Similarity, Centered Kernel Alignment, Manifold Learning, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KUu5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors argue towards multi-scale similarity analysis. From a theoretical perspective, the authors motivate why certain scales offer limited guidance, e.g. 0, ∞\infty, and provide a useful way to compute multi-scale similarity: by averaging CKA with RBF kernel with varying sigma. The authors validate their framework across controlled synthetic setups, learned representations with gLocal transformations, and a systematic evaluation on the ReSi benchmark spanning vision, language, and graph domains.

Contributions:

  • Using CKA with RBF kernel (both normalized and unormalized) with varying sigmas, the authors provide a function that yields a vector representing multi-scale similarity.
  • The authors show that this function can be transformed into a traditional similarity metric, i.e. with output between [0,1][0,1] instead of [0,1]m[0,1]^m, by simply averaging this vector. The authors establish a connection with the normalized area under the curve.
  • Through experiments, the authors are able to motivate that representation alignment can change substantially between scales motivating a multi-scale analysis.
Critical strengths
  • Raising awareness for the pitfalls of single-scale similarity analysis.
  • Offering an alternative multi-scale metric that builds upon well-known similarity metric like CKA.
  • Statistically significant performance increase over other similarity metrics in an externally designed comprehensive benchmark (RESI)
Critical weaknesses
  • The paper misses important literature: [1] already analyses the impact of single-scale similarity measurement by selecting a given k-nearest neighborhood to measure similarity with MutualNN. More importantly, other metrics [2] already establish a connection between ordinal similarity and multiple k-nearest neighborhoods (see Section 4.3), directly refuting the claim

Local and global approaches therefore capture complementary aspects of representational alignment, emphasizing neighborhood-level structure and broader organization, [...] yet existing measures typically evaluate similarity at a single predefined scale.

  • I would stress that I find this phrasing problematic, since even metrics already analyzed in the paper like CKA, SVCCA, Procrustes, do not ignore information captured at neighborhood-level structure and are truly multi-scale. They however have an implicit weighted aggregation that benefit broader organization scales more heavily, and this is where the metric introduced in this paper differs.
  • The computational time complexity associated with computing AUC(sJs_J) can be m times the complexity of computing sJs_J, which is already a significant bottleneck in many practical applications as displayed by the fact that most practitioners use batches as small as 1000 datapoints, e.g. the Platonic Representation Hypothesis paper.
  • The authors introduce a m-dimensional multi-scale similarity function, but its use is not dissimilar to simply performing the CKA computation across m scales and then evaluating each score individually as there is no direct quantitative use of the vector directly. This poses the question of whether such a vector is truly interpretable/useful.
  • It is unclear to me if the conceptual leap between this work and previous work is significant, since to a certain extent previous work had already analyzed CKA computations with different sigma selections, e.g. [3]. I suppose the authors would argue that the performance showed in section 4.3 shows a meaningful performance increase in benchmarks that practitioners care about, which is reasonable. Nevertheless, the methodological leap from using CKA to averaging CKA across scales is somewhat direct.

[1] Koepke, A., Zverev, D., Ginosar, S., & Efros, A. A. (2026). Back into Plato's cave: examining cross-modal representational convergence at scale. arXiv preprint arXiv:2604.18572.

[2] Soares, D., Gawade, P., Dittadi, A. & Szczurek, E.. (2026). Scalable and Interpretable Representation Alignment with Ordinal Similarity. Proceedings of the 43rd International Conference on Machine Learning

[3] Davari, M., Horoi, S., Natik, A., Lajoie, G., Wolf, G., & Belilovsky, E. (2022). Reliability of cka as a similarity measure in deep learning. arXiv preprint arXiv:2210.16156.

Questions for authors

Analyze the time complexity of computing AUC(sJs_J), is there any trick to make this computation comparable to the computation of sJ?s_J? Or is there always a multiplicative factor associated with computing this m times?

From the current notation, it seems the sigma is fixed for both XX and YY, this does not seem ideal as they might have completely different nominal scales. Moreover, it is unclear to me how σ1,…,σm\sigma_1, \dots, \sigma_m are selected, I assume they are picked as a function of the median, but this should be made explicit. If this is not the case then there are challenges with invariance to isotropic scaling. (Edit: I now realize that the authors report in the text that the pairwise distances are normalized by the median, this could definitely be improved to make it easier for the reader to understand.)

In the experiments, could you also benchmark the recent multi-scale metrics proposed in [2]?

Could you elaborate on AUC(sCKAs_{CKA}) robustness to outliers? This is a critical requirement of similarity metrics as motivated by [3]

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for their thoughtful feedback. We have revised the introduction and appendix to clarify our contribution, and expanded the experiments to strengthen its evaluation.

Critical Weaknesses

  1. We thank the reviewer for highlighting these relevant works. We expanded the Introduction to discuss Koepke et al. (2026). We added Appendix F, which discusses existing multiscale approaches, including TSI, QSI, IMD, and RTD, and compares them with our framework. We acknowledge the multiscale information incorporated by these measures while clarifying our complementary approach: explicitly characterizing kernel-based alignment as a function of scale, enabling analysis of scale-dependent similarity patterns and flexible aggregation over specified scale ranges and weightings.

  2. We agree that global measures are not entirely insensitive to local structure. However, their similarity scores primarily reflect broader representational organization, as also illustrated by the nonzero similarity observed between independent high-dimensional representations (Murphy et al., 2024; Gröger et al., 2026). Our distinction is that our MS-Signal directly measures how alignment varies across scales, rather than summarizing these effects into a single score. Moreover, its aggregation allows explicit control over the contribution of different scales. We have clarified this distinction in the introduction and Section 3.3.

  3. We added a computational complexity analysis in Section 3.4. By reusing pairwise distance matrices across bandwidths, the overall complexity is O(n2(pX+pY+m))O(n^2(p_X+p_Y+m)), rather than mm times the cost of RBF-CKA, with O(n2)O(n^2) memory.

  4. We have added a discussion on MS-Signal's interpretability at the end of Section 3.3. Rather than examining individual bandwidths in isolation, MS-Signal allows us to observe the signal dynamics across scales, providing context for interpreting changes in alignment. Moreover, the complete signal enables further analysis beyond its AUC aggregation. In Figure 1c, we use MDS to visualize distances between similarity profiles, revealing distinct patterns associated with local and global transformations that are obscured by scalar measures.

  5. We agree that evaluating and averaging RBF-CKA across bandwidths is mathematically straightforward. However, selecting an appropriate bandwidth remains challenging. Kornblith et al. (2019) reported similar results for RBF-CKA and linear CKA under their chosen bandwidths; this may partly reflect the convergence of RBF-CKA to linear CKA at large bandwidths (Alvarez, 2022). Other studies have demonstrated substantial variation in RBF-CKA results across bandwidths (Nguyen et al., 2022; Gröger et al., 2026), highlighting the difficulty of selecting a single bandwidth in advance. Our multiscale aggregation reduces dependence on individual bandwidth choices, with more consistent performance across domains on the ReSi benchmark, as demonstrated in Appendix D.3, where we compare different single-bandwidth choices. Our meaningful performance on ReSi, which the reviewer kindly acknowledges, demonstrates the practical value of this approach.

Questions for Authors

  1. As detailed in Section 3.4, MS-Signal reuses pairwise distance matrices across all bandwidths. Computing these matrices costs O(n2(pX+pY))O(n^2(p_X+p_Y)), followed by O(mn2)O(mn^2) for the mm kernel alignments. The total complexity is therefore O(n2(pX+pY+m))O(n^2(p_X+p_Y+m)), with O(n2)O(n^2) memory when processing bandwidths sequentially.

  2. We have clarified the relative bandwidth normalization in Section 3.4. Each representation uses its own median pairwise distance to define a common relative scale, ensuring invariance to independent isotropic rescaling. We also describe the bandwidth grid selection and examine its sensitivity in Appendix D.

  3. We have added both TSI and QSI to the ReSi evaluation in Section 4.3 (Figure 3). TSI performs particularly well, ranking first in Graphs and Language under both the 9090th-percentile and mean-rank summaries. Its strong performance further supports the value of multiscale information in representational similarity.

    Our AUC-CKA score achieves more consistent cross-domain performance, ranking first in Vision and second in Graphs and Language by 9090th-percentile rank. Appendix D.4 further examines alternative aggregation parameterizations. Together, these results highlight the potential of different approaches to incorporating information across scales and motivate further research in this direction.

  4. We added an outlier sensitivity experiment in Appendix D.1 (Figure 9). Davari et al.'s theoretical results primarily concern linear CKA, and our experiments show that isolated outliers mainly affect the large-scale region of the CKA signal, consistent with its linear CKA limit. Furthermore, UKA exhibits reduced sensitivity to these perturbations. These findings also illustrate how the MS-Signal reveals the scale-dependent effects of outliers.

The authors address most of my concerns, namely:

  • They perform a computational complexity analysis that justifies that the computation of MS-Signal is not asymptotically larger than CKA-RBF (for mm smaller than pXp_X and pYp_Y).
  • They show that averaging the CKA-RBF across scales reduces the effect of outliers, a known problem of linear CKA or even CKA-RBF for certain scales.
  • They add recently proposed TSI and QSI to the ReSi evaluation to ensure fair comparison across metrics in the literature.

However, some concerns remain:

  • I find the main methodological idea to be averaging CKA-RBF across scales, which is incremental from a conceptual perspective. I agree with the authors that looking at different scale grids or potentially performing a weighted average is conceptually interesting, but this is not really explored in this paper.
  • I was not convinced by the authors response with respect to this limitation

The authors introduce a m-dimensional multi-scale similarity function, but its use is not dissimilar to simply performing the CKA computation across m scales and then evaluating each score individually as there is no direct quantitative use of the vector directly. This poses the question of whether such a vector is truly interpretable/useful.

Taking this into account I decided to raise my score to Weak Accept

AuthorsOpenReview ↗

Official Response by Authors

Thank you for taking the time to assess the revision. We are glad the revision addressed most of your concerns, and we appreciate your updated assessment of Weak Accept.

To address your remaining points, we view the MS-Signal itself as our main contribution, providing a general framework for measuring and analyzing representational similarity as a function of scale. Figures 1 and 2 illustrate direct uses of the signal, showing how trends in alignment across scales help interpret the effects of representational transformations. We view AUC aggregation as one example of how the framework can be used, and the ReSi results illustrate the practical value of this summary across domains. We agree that developing further analyses of the full signal and exploring alternative weighting schemes are worthwhile directions for future work.

Reviewer_3drjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a multiscale framework for comparing representations from different models. It evaluates kernel alignment across RBF bandwidths to produce a similarity curve and averages the curve into a scalar score. Experiments evaluate controlled geometric changes, learned representation transformations, and performance on the ReSi benchmark.

Critical strengths
  • The multiscale perspective is intuitive and provides a useful diagnostic for identifying scale-specific differences that a single similarity score may obscure.
  • The evaluation combines controlled experiments with a broad benchmark spanning graph, vision, and language domains, where the proposed scores achieve competitive and consistent aggregate performance.
Critical weaknesses
  • The reported single-scale RBF CKA baseline uses only bandwidth 0.2. Although linear CKA is also included, comparisons with other finite bandwidths or a validation-selected bandwidth would help establish whether the reported advantage comes from multiscale aggregation rather than the particular bandwidth chosen for the RBF baseline.
  • The sensitivity of the proposed aggregate score to its bandwidth range and weighting is not clear. Since these choices determine how different scales contribute to the score, evaluating their influence on the benchmark conclusions would help assess the method’s robustness.
Questions for authors
  1. Are the main benchmark conclusions stable under reasonable changes to the bandwidth limits, number of sampled bandwidths, and weighting across scales?
  2. Does the reported advantage persist against single-scale RBF CKA using other reasonable bandwidths or a bandwidth selected on separate validation cases, beyond the reported fixed bandwidth of 0.2?
  3. Section 3.3 shows that similarity approaches 1 at very small bandwidths regardless of representational agreement. Is the lower bandwidth limit of 0.05 derived from this theoretical analysis or chosen empirically?
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for their constructive feedback and valuable suggestions. In response, we have expanded our experimental evaluation to examine the impact of bandwidth selection and the sensitivity of our method to different scale aggregation choices, strengthening the empirical support for our multiscale approach.

Response to Critical Weaknesses

  1. We added Figure 12 in Appendix D.3, evaluating single-bandwidth RBF-CKA across a range of bandwidths on ReSi. Performance varies substantially across bandwidths and domains, particularly in the 90th-percentile rank, with different bandwidths performing best in different domains. These findings highlight the importance of considering multiple scales and support our multiscale approach, which integrates information across scales to achieve consistent performance across domains.
  2. We added Appendices D.2–D.4 (Figures 10, 11, and 13), examining sensitivity to the number of sampled bandwidths, bandwidth limits, and scale parameterization. We find that the number of sampled bandwidths has little effect on benchmark performance, while the bandwidth range has a greater influence, with wider intervals generally yielding more stable results in our experiments. We also compare logarithmic, square-root, and linearly spaced scale grid sampling, which induce different weightings of the AUC. Although square-root sampling, for example, improves performance on Language, performance varies across these choices, and logarithmic integration provides consistently strong results across domains.

Response to Questions for Authors

  1. The sensitivity analyses described in response to Critical Weakness 2 address the stability of the benchmark conclusions under changes to the bandwidth grid, integration limits, and scale weighting.
  2. Figure 12, discussed in response to Critical Weakness 1, compares RBF-CKA across multiple fixed bandwidths and demonstrates that the preferred bandwidth varies across domains.
  3. The lower bandwidth limit of 0.050.05 was chosen empirically, rather than derived from the theoretical limit in Section 3.3. We selected it by examining the similarity signals and identifying a sufficiently small bandwidth at which most signals approached their limiting behavior. This occurs when the off-diagonal RBF similarities become negligible relative to the diagonal entries. We additionally examine sensitivity to the lower bandwidth limit in Appendix D.2.

I thank the authors for the further clarifications. Most of my concerns has been addressed.

Reviewer_cZSHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed an extension of CKA, which is an integral value of CKA's over various bandwidth of RBF kernel. This integral is a multiscale similarity signal. The proposed similarity measure outperforms traditional CKA and provides good results on ReSi, a benchmark for representational similarity measures.

Critical strengths
  1. Quite simple but good-performing idea.
  2. Strong performance on ReSi benchmark.
  3. The paper is well written and easy to follow.
Critical weaknesses
  1. Barannikov et. al proposed a method for a multi-scale representation comparison. How your AUC-CKA is related to it?

Barannikov, S., Trofimov, I., Balabin, N., & Burnaev, E. (2021). Representation topology divergence: A method for comparing neural network representations.

  1. In paper, authors write "We provide code to reproduce our results on anonymous GitHub" but I can't find a link. Given a reproducibility crisis in our area, this is a critical weakness.
Questions for authors
  1. Why do you need UKA, since AUC further is defined in terms of CKA?
  2. RBF-CKA with RBF kernel is known to have a large computational time. Since your method involves multiple computations of RBF-CKA, what is the computational burden?
  3. While an integral CKA (AUC) works well, is any specific bandwidth also good across many datasets?
  4. The paper lacks a name of a proposed method which makes it harder to comprehend.

Post rebuttal. I am grateful to the authors for answering my questions and conducting additional experiments.

1 comment
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for their constructive feedback, which has helped us improve the clarity and evaluation of our paper.

Response to Critical Weaknesses

  1. We have expanded the related-work discussion in Appendix F to cover existing multiscale approaches, including Representation Topology Divergence (RTD). RTD compares representations through persistent topological features across filtration scales, whereas our framework examines kernel-based alignment between corresponding samples as a function of scale. We believe these approaches offer complementary topological and geometric perspectives on multiscale representational similarity.
  2. We apologize for the unclear presentation of the code link. The words "anonymous GitHub" in the original submission were clickable and linked to our code. We have made the link more visible in the revised paper.

Response to Questions for Authors

  1. We have clarified the motivation for UKA in Section 3.3. Unlike CKA, UKA approaches representation-independent values at both bandwidth limits. Consequently, differences in its aggregated scores are driven primarily by local-to-intermediate scales, without requiring these scales to be identified beforehand. We also added an experiment in Appendix D.1 demonstrating UKA's reduced sensitivity to isolated outliers compared with CKA.
  2. In Section 3.4, we added analysis of the computational complexity. At first glance it might appear intensive but it can be implemented efficiently with complexity comparable to CKA.
  3. We added Figure 12 in Appendix D.3, evaluating single-bandwidth RBF-CKA across a range of bandwidths on ReSi. Performance varies substantially across bandwidths and domains, particularly in the 90th-percentile rank. Different bandwidths perform best in different domains, and the optimal choice is not known in advance. These findings highlight the difficulty of selecting a single bandwidth and motivate our multiscale approach, which avoids this selection while leveraging information across scales to achieve more consistent performance across domains.
  4. We have named the proposed framework MS-Signal (Multiscale Similarity Signal) and updated the paper accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.