- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper compares seven representational similarity metrics to evaluate their ability to recover architectural and training-paradigm groupings in artificial neural networks, as well as anatomical groupings in biological neural data. The experiments cover 35 vision models, four image datasets, human NSD fMRI data, and mouse ABO electrophysiological data, with additional analyses of random seeds and network depth. The results show that metrics sensitive to representational geometry and unit tuning generally exhibit stronger grouping performance, although their rankings depend on the dataset and evaluation criterion. The authors further adopt the existing SNF method to integrate multiple metrics, improving most separation scores and exploring the organizational structure of models and brain regions. The primary contribution of this work is a systematic empirical comparison of representational similarity metrics and the methodological insights it provides.
Critical strengths
-
The research question is important and offers a valuable perspective for studying neural networks. The paper emphasizes that different metrics capture different properties of representations and relates metric selection to specific scientific questions. This helps avoid overinterpreting neural systems as equivalent based on a single similarity score.
-
The integration analysis reveals interpretable organizational structures. Without using group labels, SNF identifies clusters organized by training paradigm across architectural boundaries, as well as relationships among brain regions that are consistent with the hierarchical organization of the visual pathway. The authors also explicitly acknowledge that integration does not consistently outperform individual metrics, lending appropriate nuance to their conclusions.
-
The experiments cover a broad range of settings. The authors compare metrics across multiple visual datasets, artificial neural networks, human fMRI data, and mouse electrophysiological data, providing cross-system support for the observed patterns. The variation in metric rankings across datasets also offers useful counterexamples.
Critical weaknesses
-
The main experiments focus on the recovery of predefined groupings, while the supplementary evaluation of brain-region hierarchies also relies on established organizational structures. These results demonstrate the sensitivity of metrics to architectural, training-related, and anatomical differences, but provide more limited evidence regarding behavioral consistency, task-relevant information, or functional equivalence. Consequently, the paper offers limited guidance on metric selection for broader scientific questions.
-
The exclusion of CCA from the NSD analysis due to inflated similarity estimates under high-dimensional, small-sample conditions indicates that dimensionality and sample size can substantially affect metric comparisons. Linear Predictivity is implemented using ordinary least squares, but the paper does not systematically examine how dimensionality matching, sample-size variation, or regularization affects the rankings. Therefore, attributing weaker separation primarily to greater mapping flexibility and reduced preservation of geometric information leaves room for alternative explanations related to estimation conditions.
-
The seed analysis covers only seven CNN configurations, with two seeds per configuration, while the NSD analysis includes only four subjects. The error bars in the figures primarily reflect variation across group pairs, which share models or brain regions. Such error bars are better interpreted as describing heterogeneity across group pairs than as directly estimating precision under independent replication. Appropriate resampling at the stimulus, model-instance, or subject level, together with analyses of ranking and clustering stability, would strengthen the assessment of subtle differences.
Questions for authors
-
In the five-fold cross-validation procedure, are mean centering, dimensionality reduction for SVCCA, and CCA projection all fitted exclusively on the training fold and then applied to the test fold? Could the authors clarify the fitting and evaluation procedures for each metric?
-
For the ABO data, are the cross-validation folds created by randomly splitting individual time points or by splitting contiguous movie segments? Do the authors account for the potential effects of temporal autocorrelation between adjacent time points on test scores?
-
In Section 2.3, how are the similarity-based silhouette score and contrastive ratio handled when similarities are negative or when the denominator is close to zero? Is a consistent handling procedure applied across all metrics?