acceptodds
Under review as a conference paper at ICLR 2027

Comparing Neural Network Representations with Bayesian Prior Predictive Distributions

Abstract

How best to compare internal representations of neural networks remains a highly relevant unresolved question in both machine learning and neuroscience, even though numerous similarity measures have been proposed. Here, we propose and extensively evaluate a new measure based on a Bayesian perspective on linear readouts called BayesCompare. BayesCompare compares the prior predictive distributions of linear readouts from the representations using metrics or divergences for probability distributions. We evaluate BayesCompare with a suite of experiments using 6 popular architectures trained on ImageNet-1k, comparing BayesCompare against a wide range of existing methods, including Representational Similarity Analysis (RSA) and Centered Kernel Alignment (CKA). First, we quantify each method's ability to retrieve the corresponding layer from 5 instances of the same architecture. All methods are able to retrieve layers to some extent, but some metrics within BayesCompare improve layer retrieval over existing methods. Second, we confirm that BayesCompare preserves the gradual change between representations throughout the network resulting from incremental operations of layers, to a similar extent as the existing methods. Third, we measure the methods' consistency under different random image sets and set sizes. All BayesCompare metrics provide highly reliable results even with only 50 input images, whereas RSA shows the lowest consistency. We also analyze the effects of rank transformation and assess the agreement between all metrics. BayesCompare metrics are closely aligned with each other, but distinct from existing methods, suggesting that BayesCompare captures a different aspect of representational similarity. Overall, these results establish BayesCompare as an effective way of measuring representational similarity while being easy and fast to compute and give strong guidance on which variants one should use. We provide a Python package to apply BayesCompare.

Primary area:
interpretability and explainable AI
Keywords:
representational similarity, vision, methods

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_tBWoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a way to measure the similarity between the internal representations of neural networks using probability distributions, rather than directly comparing activations or their similarity matrices. They compare the neural network representations through the prior predictive distributions induced by Gaussian linear readouts.

The neural network representations are compared using several divergences and distances between Gaussian distributions, including KL, Jensen–Shannon, Bhattacharyya, Hellinger, total variation, and Wasserstein. I found the idea very interesting.

To evaluate the proposal, the authors used six architectures, with multiple model instances, and three criteria including layer retrieval, layer-order consistency, and stability across image subsets.

Critical strengths

The paper is easy to read and the formulation is mathematically well motivated. The code is available and the implementation details contribute to reproducibility helping the reader.

I found the experimental evaluation to be reasonably broad, including multiple architectures, training configurations, and image subsets.

I liked the way stability is evaluated with respect to the choice and number of input images, particularly relevant for representation similarity methods.

The analysis of correlations between different similarity measures shows a very interesting way to understand redundancy among existing metrics.

Critical weaknesses

To me the novelty of the paper is limited as the central idea of comparing representations through Bayesian predictive distributions was already introduced in the authors' earlier work.

The previous paper also included experiments with ImageNet representations. If I understand correctly, the contribution here is an extension of the evaluation and the family of distances and does not present sufficient novelty to justify the paper.

I think the experimental conclusions are stronger than the reported results support. The results show only two of the six proposed variants outperform RSA in layer retrieval and this should be better evaluated in the paper. Seems that all proposed variants perform worse than several existing methods in the layer-order experiment.

I think the baseline comparison is not fully controlled. For instance, in the GULP case, it appears to use a regularization configuration that can lead to degenerate behavior, while CKA and distance correlation rely on biased estimators. A question for the authors, is the method's hyperparameter selected using the evaluation data?

Questions for authors

Which Gram matrix is actually implemented? the formulation presented in the main text or the centered formulation in the appendix? Will be interesting to understand how this choice affects the invariance properties.

Does the claimed advantage in layer retrieval remain after applying rank transformations? How much of the improvement can be attributed to the nonlinear scaling of the distances?

Please verify whether the statistical results in Figure 10 appear to have incorrectly assigned metric labels. This affects the interpretation of the significance tests supporting the layer-retrieval results.

Please verify the equations because there are mathematical and implementation ambiguities concerning the Gram matrix definition, the claimed invariances, and the expression for Jensen–Shannon divergence.

2 comments

We thank the reviewer for their careful evaluation and constructive feedback.

Responses to Weaknesses

  1. Novelty: While we acknowledge that [44] provides the theoretical foundation for our method, its scope remains purely theoretical, showing only demonstration-style examples without any formal evaluation. In contrast, in our work,

    • We conduct a systematic evaluation of BayesCompare against a diverse set of established similarity measures using a suite of experiments targeting various aspects of representational similarity. Two of these evaluation protocols are inspired by or adapted from existing literature, and we additionally propose a new evaluation focused on the stability of the metrics. Based on the reviews, we added a functional correspondance analysis which helps us to ground the similarity metric with a downstream task performance. This type of functional grounding and systematic evaluation against a cohort of existing methods is missing in [44].
    • We consider a broad range of modern vision architectures trained on a non-trivial dataset. This extends beyond the scope of many prior studies, which often focus on simpler datasets such as MNIST or CIFAR, or less complex architectures. [44] ran a single analysis on 3 pretrained networks, compared to 30 networks compared and trained by us.
  2. Conclusions and Supporting Evidence: We appreciate the chance to further defend our results. We report results for the different metrics one could apply in BayesCompare to provide guidance on which metric one should use within BayesCompare. We do not claim that all metrics work better than existing ones. Practically speaking, we expect that users of BayesCompare will choose their metric wisely, such that the performance of worse metrics does not matter. With the metrics we recommend in our discussion, all qualitative advantages we report can indeed be realized simultaneously.

    We were also careful to point out that no single metric outperforms all others across every task consistent with Klabunde's ReSi [23]. Also, we acknowledge the good performance of the rarely used Jaccard distance. Nonetheless, the layer monotonicity and functional similarity findings varied strongly by architecture (even within vision modality). For instance, CKA yields the best results for monotonicity in EfficientNet but is outperformed in ResNet50. Within this cohort of varyingly successful metrics, BayesCompare seems like a good general purpose solution. It shows the best performance or closely follows the best performer in all of the experiments. Additionally, it comes with properties that are practically relevant, such as having differentiable proper metrics in the space of representations, which are highly stable across image samples. These properties should make BayesCompare particularly well suited for optimizing distances between representations (e.g. stimulus optimizations for explainable AI or teacher student setups for network training).

  3. Baseline Control: For GULP, we included a regularized version because the unregularized variant (λ=0\lambda=0) is equivalent to a linear transformation of ridge-CCA, which is generally excluded from our comparisons (as noted in Section 2) due to its greater flexibility and suboptimality, as established in the literature. The original GULP paper [5] also notes that low regularizations primarily capture matrix rank, while higher regularizations can cluster layers by depth. Thus, the hyperparameter settings for GULP we evaluate correspond to the recommendations of the original authors.

    Thank you for you suggestion of unbiased CKA. We included the unbiased CKA as a control for Exp1 and Exp2, and we are waiting on the Exp3 to finish computing. We, for now, added these results to the Appendix and we plan to include its results in the main text once all experiments are computed. Our results showed that while unbiased CKA shows an intermediate performance for Exp1, layer retrieval, it indeed shows the best performance in Exp2, layer monotonicity, in almost all architectures.

Evaluation data question The aim of our experiments is to evaluate the performance of BayesCompare over the range of possible hyperparameters i.e. metric and noise levels. We do not first optimize the hyperparameter values on one dataset and then evaluate the performance on another. We simply aim to show the users of BayesCompare which of the hyperparameter settings would yield a favorable result in certain use cases, such as layer retrieval, monotonicity or consistency. Therefore, the whole experimental suite can be considered as evaluation.

AuthorsOpenReview ↗

Responses to Questions

  1. We appreciate the reviewer flagging the inconsistency regarding double-centering. We implemented BayesCompare both with a double-centered Gram matrix in the feature dimension (appendix formulation) and without any double-centering (main text formulation). Our intention was to present results without double-centering. However, the initial results in the paper are computed with double-centering in the feature dimension (appendix formulation).We have verified that this preprocessing step does not alter the categorical conclusions, likely because most features have near zero-mean activations. All relevant formulations and figures have now been updated to match the main text (without double-centering). If the reviewer is interested, we can include the double-centered results in the Appendix, though both approaches yield highly similar outcomes.

  2. BayesCompare metrics demonstrate the strongest performance in the layer retrieval task prior to rank transformation. After applying the rank transformation, Jaccard achieves the highest score (mean SS = 0.9797), with BayesCompare’s TVD closely following (mean SS = 0.9762), as reported in Table 1. This indicates that BayesCompare retains strong layer retrieval capability after rank transformation, while Jaccard shows substantial improvement specifically due to the transformation. This non-linear rank transformation appears particularly advantageous for Jaccard, as it moves from being the lowest-performing metric in the original setting to the highest after transformation.

  3. We thank reviewer for their attention and we acknowledge this oversight. There has been a mixup in the names of the x-axis labels which is now fixed. We confirm that the results shown in Fig 10 in Appendix (now Fig. 11 in Appendix) are consistent with the results shown in Figure 3.

  4. We confirm that the definitions regarding Gram matrices have been corrected, and the invariance claims should now hold. We have also revised the definition of Jensen-Shannon Distance for clarity. If there are any further ambiguities regarding Gram matrix implementations, we would be happy to address them specifically.

Reviewer_oKY1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BayesCompare, a Bayesian framework for comparing neural network representations. The method constructs prior predictive distributions of linear readouts from neural representations and measures their differences using several probability distribution distances and divergences, including KL divergence, Jensen-Shannon distance, Wasserstein distance, and Bhattacharyya distance.

The authors evaluate BayesCompare against existing representation similarity measures, including CKA, RSA, GULP, and Jaccard similarity. Experiments across six ImageNet-trained architectures examine three properties: layer retrieval, preservation of layer monotonicity, and consistency across different image sets and sample sizes. The results suggest that BayesCompare achieves competitive layer retrieval performance and relatively high consistency, particularly when using small sets of input images.

Critical strengths
  1. Interesting perspective on representation similarity. The paper introduces a Bayesian interpretation of representation comparison through the prior predictive distributions of linear readouts. This provides an interesting alternative to conventional similarity measures such as CKA and RSA. The framework is relatively simple and supports several probability distribution distances and divergences.

  2. Comprehensive empirical evaluation. The experiments cover six neural network architectures and compare BayesCompare with a broad range of existing approaches. The evaluation considers layer retrieval, layer monotonicity, and consistency across different image sets. The analysis of rank transformations and the sensitivity to the noise parameter provides additional insights into the behavior of different similarity measures.

Critical weaknesses
  1. The definition of the representation kernel and the claimed invariance properties require clarification. Section 3 defines the prior predictive covariance using the Gram matrix (XX⊤XX^\top), and Appendix A.3 argues that BayesCompare is invariant to orthogonal transformations of the representation. However, Appendix A.2 (Eqs. 16–18) constructs the covariance using a feature-centered Gram matrix (X(I−11⊤/k)X⊤X(I-\mathbf{1}\mathbf{1}^\top/k)X^\top). Unlike (XX⊤XX^\top), this matrix is not generally invariant to arbitrary orthogonal transformations of the feature space. Consequently, the stated invariance argument does not appear to apply directly to the implementation described in Appendix A.2. This discrepancy is important because the proposed representation equivalence class is a central property of the method. The authors should clarify which formulation is actually implemented and whether the claimed invariance properties hold for that formulation.

  2. The evaluation does not fully establish the practical advantages of the proposed similarity measure. The experiments primarily assess layer retrieval, layer monotonicity, and consistency across image subsets. While these are useful evaluation criteria, they do not necessarily establish whether BayesCompare better captures functionally meaningful representation similarity. Furthermore, all evaluated methods achieve relatively high layer retrieval scores, and the relative rankings change substantially after rank transformation. In particular, Jaccard similarity achieves the highest layer retrieval score after rank transformation, while several existing methods perform comparably on layer monotonicity. Additional functional evaluations would help establish whether the proposed Bayesian formulation provides meaningful advantages beyond the selected structural metrics.

Questions for authors
  1. Regarding the kernel definition and invariance properties: Section 3 and Appendix A.3 formulate BayesCompare using the Gram matrix (XX^\top), whereas Appendix A.2 uses feature-centered Gram matrices. Could the authors clarify which definition is used in the experiments? Since feature centering generally breaks invariance to arbitrary orthogonal transformations, can the authors verify that the claimed invariance properties hold for the actual implementation? It would also be helpful to evaluate the effect of this centering choice on the reported results.

  2. Regarding the practical significance of BayesCompare: Can the authors provide additional evidence that BayesCompare captures functionally meaningful representation similarity better than existing methods? For example, would the proposed metric better predict linear probe transferability, model stitching performance, or agreement in downstream predictions? Such experiments could help determine whether the improved layer retrieval and consistency translate into practical advantages.

  3. Regarding hyperparameter sensitivity and metric selection: The performance of different BayesCompare variants depends on the noise parameter and the choice of divergence. Moreover, rank transformation substantially changes the relative performance of the methods. Could the authors clarify how these choices should be made in applications where no ground-truth layer correspondence is available? In particular, how robust are the conclusions across different noise levels, sample sizes, and representation preprocessing choices?

2 comments

We thank the reviewer for their valuable feedback and comments.

  1. Clarification on definitions using Gram matrix and centering Thank you for pointing out this inconsistency, we really appreciate your careful reading! This inconsistency arose, because we conducted experiments with and without centering the features over the course of the project. We have corrected this in the revised manuscript, updating both the definitions and the results to using the analyses without feature mean removal everywhere.

    In the interest of transparency: all experiments (except for Exp 3 - consistency) were conducted both with and without double-centering in the feature dimension. By running the experiments both ways, we found that the results do not depend on centering. All our results are virtually identical with or without feature-mean removal. We believe this is due to the layer activations typically having near zero-mean activations anyway. In informal checks across a subset of layers in each network that appears to be the case.

  2. Practical Significance and Comparisons with Other Metrics: We appreciate the reviewer’s suggestion to clarify the practical significance of the proposed metrics and their comparison with other approaches as our results agree with previous papers (like [5]) that no metric dominates across all evaluations.

    The reviewer seems particularly interested in functional relevance of the metrics. To test this, we conducted a fourth experiment: We fitted linear probes from all tested representations to classify bird images from CUB-200-2011 [1] (cited as [54] in the updated manuscript). For each layer, we obtained the test set predictions of these probes and computed Cohen's Kappa between them as a measure of functional similarity. We then computed Spearman's Rho rank correlation between the functional similarities and the representational similarities as an evaluation criterion. For the representational similarity, we consider 3 datasets: (i) MSCOCO unlabeled 2017 set as in the other experiments, (ii) CUB train set and (iii) CUB test set. These sets are ordered by their generalizability relative to the functional similarity task, which is computed on the CUB test set.

    As we report in our updated manuscript, the results are quite similar to the layer monotonicity results. The correspondence between representational and functional similarity varies substantially across models for each metric and using different datasets to compute representations yields even higher variance in metrics' performances (see updated Appendix Fig. 13). Thus, most comparisons between metrics remain uncertain. The only two statements we can make with some certainty are: The Jaccard metric consistently demonstrates the best performance, confirming earlier findings in [5] (in paper citation), whereas GULP performs worst among the metrics evaluated. BayesCompare metrics score similarly well as existing methods with the order depending on the model and dataset used for evaluation.

    Overall, the outcome of our four experiments is consistent with [5]: there is no single metric that performs best across all tasks; rather, different metrics are successful for different purposes. Our results show that all existing metrics from the literature fail in at least one of our experiments, while BayesCompare performs highly across all of them. Thus, BayesCompare seems like a good general purpose solution. Additionally, BayesCompare yields differentiable proper metrics in the space of representations, which are highly stable across image samples. These properties should make BayesCompare particularly well suited for optimizing distances between representations (e.g. stimulus optimizations for explainable AI or teacher student setups for network training). In our opinion these application cases make the BayesCompare metrics clearly practically relevant.

[1] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD birds-200-2011 dataset. (CNS-TR-2011-001), 2011.

  1. Hyperparameter Selection: The hyperparameters have some influence on the numerical performance, but general patterns are highly consitent over hyperparameters. Additioanlly, we believe our experiments give good guidance on what settings one should use to maximise performance. As stated more clearly in the discussion now, we recommend the symmetrized KL-Divergence or Battacharyya distance with b=0.05b=0.05 if a divergence is sufficient or the slightly more computationally expensive JSD with b=0.01b=0.01 if a true metric is required.

    The choice of metric is based primarily on Experiment 1, where BayesCompare's KL Divergence and Bhattacharyya measures performed best among all metrics, independent of noise level. Among the true distance metrics, BayesCompare's JSD with an intermediate noise level (b=0.01b=0.01 corresponding to σϵ2=0.91\sigma_{\epsilon}^2 = 0.91 for 1000 images) performed best leading to our recommendation. In all other experiments the different metrics perform similarly well. The only metric that should really be avoided is the Wasserstein distance, which performs noticeably worse.

    While the noise level can have slight task-dependent effects, it does not change the qualitative results of the experiments. In Experiment 1, the top-performing metrics remain superior regardless of noise level. In Exp 2 (layer monotonicity), the ability of all BayesCompare metrics to capture monotonicity improves with smaller noise levels. We have now included relevant plots for this in the Appendix. For Exp 3, we tested two noise levels b=0.05b=0.05 and b=0.01b=0.01 and the results remain categorically similar, with BayesCompare metrics yielding the most stable outcomes independent of noise level. Thus, the range of noise levels we considered appears to be acceptable in its entirety. We recommend the high noise level for KL-divergence and Bhattacharyya to realize the slight advantage in Exp. 2 and recommend an intermediate noise level for JSD to prevent saturation effects (see Appendix Fig. 9).

    Sample size again alters results very little. We generally find larger gains in stability for BayesCompare for smaller samples of inputs in Exp. 3. Otherwise, we did not observe any coherent effects of sample size.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.