acceptodds
Under review as a conference paper at ICLR 2027

Comparing and Integrating Different Notions of Representational Correspondence in Neural Systems

Abstract

Representational similarity metrics emphasize different properties, so their agreement with one scientific criterion need not extend to another. We compare seven widely used metrics using hypothesis-driven tests of within-group correspondence and between-group separation in artificial and biological neural systems. The benchmarks include 35 vision models evaluated on ImageNet, Ecoset, CIFAR10, and CIFAR100, human fMRI responses from the Natural Scenes Dataset, and mouse Neuropixels recordings from the Allen Brain Observatory. Geometry- and tuning-sensitive metrics generally recover stronger model-family and anatomical-group separation than more flexible mappings, but the leading metric changes with dataset and separation criterion. In a population of independently trained seeds, the same metrics separate training objectives beyond seed-to-seed variation, while analyses at six network depths show that intermediate-layer rankings differ from final-layer rankings. Integrating the metrics with Similarity Network Fusion improves most group-separation scores and yields data-driven typologies in which the training paradigm can cut across architecture. These results demonstrate metrics' different abilities in separating neural systems, while also showing that two different neural systems could still share great similarity in some representational aspects.

Primary area:
applications to neuroscience & cognitive science
Keywords:
representational similarity, vision models, Transformers, CNNs, visual cortex

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vcKdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares seven representational similarity metrics to evaluate their ability to recover architectural and training-paradigm groupings in artificial neural networks, as well as anatomical groupings in biological neural data. The experiments cover 35 vision models, four image datasets, human NSD fMRI data, and mouse ABO electrophysiological data, with additional analyses of random seeds and network depth. The results show that metrics sensitive to representational geometry and unit tuning generally exhibit stronger grouping performance, although their rankings depend on the dataset and evaluation criterion. The authors further adopt the existing SNF method to integrate multiple metrics, improving most separation scores and exploring the organizational structure of models and brain regions. The primary contribution of this work is a systematic empirical comparison of representational similarity metrics and the methodological insights it provides.

Critical strengths
  1. The research question is important and offers a valuable perspective for studying neural networks. The paper emphasizes that different metrics capture different properties of representations and relates metric selection to specific scientific questions. This helps avoid overinterpreting neural systems as equivalent based on a single similarity score.

  2. The integration analysis reveals interpretable organizational structures. Without using group labels, SNF identifies clusters organized by training paradigm across architectural boundaries, as well as relationships among brain regions that are consistent with the hierarchical organization of the visual pathway. The authors also explicitly acknowledge that integration does not consistently outperform individual metrics, lending appropriate nuance to their conclusions.

  3. The experiments cover a broad range of settings. The authors compare metrics across multiple visual datasets, artificial neural networks, human fMRI data, and mouse electrophysiological data, providing cross-system support for the observed patterns. The variation in metric rankings across datasets also offers useful counterexamples.

Critical weaknesses
  1. The main experiments focus on the recovery of predefined groupings, while the supplementary evaluation of brain-region hierarchies also relies on established organizational structures. These results demonstrate the sensitivity of metrics to architectural, training-related, and anatomical differences, but provide more limited evidence regarding behavioral consistency, task-relevant information, or functional equivalence. Consequently, the paper offers limited guidance on metric selection for broader scientific questions.

  2. The exclusion of CCA from the NSD analysis due to inflated similarity estimates under high-dimensional, small-sample conditions indicates that dimensionality and sample size can substantially affect metric comparisons. Linear Predictivity is implemented using ordinary least squares, but the paper does not systematically examine how dimensionality matching, sample-size variation, or regularization affects the rankings. Therefore, attributing weaker separation primarily to greater mapping flexibility and reduced preservation of geometric information leaves room for alternative explanations related to estimation conditions.

  3. The seed analysis covers only seven CNN configurations, with two seeds per configuration, while the NSD analysis includes only four subjects. The error bars in the figures primarily reflect variation across group pairs, which share models or brain regions. Such error bars are better interpreted as describing heterogeneity across group pairs than as directly estimating precision under independent replication. Appropriate resampling at the stimulus, model-instance, or subject level, together with analyses of ranking and clustering stability, would strengthen the assessment of subtle differences.

Questions for authors
  1. In the five-fold cross-validation procedure, are mean centering, dimensionality reduction for SVCCA, and CCA projection all fitted exclusively on the training fold and then applied to the test fold? Could the authors clarify the fitting and evaluation procedures for each metric?

  2. For the ABO data, are the cross-validation folds created by randomly splitting individual time points or by splitting contiguous movie segments? Do the authors account for the potential effects of temporal autocorrelation between adjacent time points on test scores?

  3. In Section 2.3, how are the similarity-based silhouette score and contrastive ratio handled when similarities are negative or when the denominator is close to zero? Is a consistent handling procedure applied across all metrics?

Reviewer_KARsOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares seven similarity metrics and tests them on 35 vision models, human fMRI (NSD), and mouse Neuropixels data (ABO). It checks which metrics separate model families or brain regions, and it combines all metrics with SNF.

Critical strengths
  • The benchmark is broad: 35 models, four image sets, human fMRI and mouse recordings.
  • The analysis are well controled (comparison of same-training, different architectures/same-architectures, different training/use of seed-to-seed variation)
  • The authors are careful with claims.
Critical weaknesses

Critical Weaknesses

  • Little conceptual novelty. That metrics disagree is already known, and SNF is an existing method. The fused score is also not reliably better.
  • The test of a “good” metric is how well it separates known labels like architecture or brain area. A metric can separate labels well and still miss what matters for function or behavior. The paper has no behavioral or task test.
  • The metrics are not compared on equal footing. CCA and linear predictivity are fitted metrics with many units and few stimuli. Their poor scores may be estimation failure, not a property of the metric. There is no dimensionality reduction or regularization control.
Questions for authors

The take-home message is not clear. Can you give a simple guide on which metric to use for which scientific question, and on when SNF is worth using over a single metric?

The metrics are judged only by how well they separate labels. Can you add a behavioral or task-based test (for example, error patterns or decoding of task variables) to show which metric tracks function?

Can you rerun the analyses with PCA reduction or regularization, matched across metrics, and show whether the ranking still holds?

Reviewer_mUKbOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how different measures of representational alignment recover measured relationships of neural systems. Its premise is that metrics emphasize different geometric properties, so their usefulness should be evaluated against a specific scientific question or measure. The authors analyze seven measures of similarity and evaluate whether similarities are larger within specific groups than between groups constructed using a similarity-based silhouette score and a contrastive ratio. The main findings are that geometry-sensitive metrics recover the specified groups more strongly than the evaluated linear mappings do, and that the results are sensitive to the dataset and network depth. Positive results are also reported with similarity network fusion.

Critical strengths

• Useful scientific question: I think this question is useful and timely. Previous work has found that a high similarity value is probably insufficient to establish that a metric is useful for a particular scientific question. This is comprehensive and useful.

• Broad empirical coverage: A lot of models are tested and many metrics of comparison.

• Inclusion of fusion: I like that similarity includes similarity network fusion. To my knowledge, I haven’t seen this in the literature previously. The paper introduces comparisons with arithmetic averaging, etc. This is very useful.

Critical weaknesses

Some mathematical concerns: My biggest concern is that some mathematical definitions and claims the paper makes are not entirely accurate given how the metrics are defined in their original paper. The paper explains the empirical findings through the invariances of each metric and what transformations they admit. However, certain subtleties make this more unclear. I would appreciate it if the authors clarify:

o RSA: The authors state that RSA is invariant to orthogonal transformations. And usually this is correct. But the authors add an operation of computing dissimilarity by subtracting the stimulus vector mean across neurons. This changes your dot products and I think makes the product sensitive to orthogonal transformations?

o Procrustes: Usually, the optimal Frobenius alignment is invariant to orthogonal changes of basis. But the way the Procrustes score is applied in this work is odd. The paper fits the alignment and reports the mean of unit-wise Pearson correlations. Is this standard? The coordinate wise aggregation shouldn’t inherit the invariance of the fitting objective. Is this how scores are usually reported?

o Soft Matching: I read the cited Soft Matching formulation. This uses a squared-cost transport distance or a correlation score that aggregates matched individual-unit correlations. Does correlating a mixture of units provide a different measurement?

Estimation Difficulty: One difficulty that is hard to overcome is that weak performance could be explained by finite-sample estimation or dimensionality or regularization choices. Working with NSD makes this even more important to think about. The paper ignores CCA metrics between voxel dimensionality, which inflates scores, but estimator behavior is sensitive in this benchmark. This does create concern about metrics, e.g., linear predictivity, which is run unnormalized if I understand correctly.

Novelty: I think some portions of the novelty of this work need to be more properly contextualized. [1] and [2] are cited by this paper, but the strongest differentiating factor seems to be broader empirical coverage. I don’t think this is a bad thing. However, it also makes acceptance depend heavily on correctness since the prior conclusions have already been established.

[1] Ding et. al. Grounding Representation Similarity Through Statistical Testing, NeurIPS 2021

[2] Bo et. al. Evaluating Representational Similarity Measures from the Lens of Functional Correspondence. arXiv 2025

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.