Similarity as a Signal: Comparing Representations Across Multiple Scales

Representational similarity measures provide a quantitative assessment of alignment between representations learned by different models, modalities, and training procedures. Existing measures typically characterize similarity at a single predefined scale, although different aspects of alignment may be reflected in structure ranging from local neighborhoods to broader global organization. Intuitively, local neighborhood structure may be important for retrieval or instance-level matching, whereas broader geometric organization may capture higher-level semantic relationships. We introduce a multiscale framework that treats representational similarity as a function of scale rather than as a single scalar. Motivated by the connection between RBF kernels and heat diffusion, we use the RBF bandwidth as the scale parameter and introduce the multiscale similarity signal to characterize representational alignment across scales. On controlled synthetic data, we show that perturbations at local, intermediate, and global scales produce distinct signatures in the signal. Moreover, on learned neural representations, the signal characterizes more complex transformations spanning a broad range of scales. Finally, we derive scalar scores by aggregating similarity across scales and evaluate them on the ReSi benchmark spanning graph, vision, and language domains, where our primary multiscale score achieves the best 90th-percentile rank in all three domains. Our results show that viewing representational similarity across multiple scales provides both an interpretable tool for analyzing representation geometry and an effective framework for quantifying representational similarity.