- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes PARIS, a graph metanetwork that learns to predict representational similarity scores across different sites in neural networks. Their method is different from methods such as CKA and CKNNA which require computing and storing model activations on a shared set of probe images. In contrast, PARIS directly performs message massing on model weights to compare them across sites.
Critical strengths
The authors present theoretical justification of their method’s ability to account for parameter symmetries. Empirically, they show that their model beats many other methods on tasks that require accurately measuring representational similarity.
Critical weaknesses
Although the paper provides an interesting method for computing representational similarity, it should be rejected because lacks a few crucial results, and without those results its contribution is minor at best. Specifically:
-
Results are only presented across the same kind of network with different hyperparameters. This limits the usefulness of the authors’ method, especially since previous work (such as the paper that presents the MutualNN metric used in this submission) already compares across different networks and different modalities. It is not clear if PARIS is able to train across different types of networks and/or modalities, and if so, how it performs.
-
In figure 6, it is not clear why the target is an existing metric like a CKA-based dissimilarity. It is also not clear if there is a significant advantage in training a meta-network since the target metric can be directly computed in the situations the authors run experiments on.
-
It seems like one of the paper’s advantages is that they do not need to run inference on samples to compute similarity scores for new networks. However, runtime results are not included for measuring how much compute the meta-network requires throughout training or during inference. Therefore, it is challenging to reason about the usefulness of their method.
Overall, the idea could be promising but the lack of results and analysis make it challenging to reason about the usefulness of their method.
Questions for authors
Is PARIS able to train across different types of networks and/or modalities? And if so, how does it compare to other methods that can?
Is there an advantage in training a meta-network to predict a metric when that metric can be directly computed in the situations the authors run experiments on?
How does inference time compare with existing methods, and if inference time is faster with PARIS, is it fast enough to justify the meta-network training cost?
6 comments
Response to Reviewer 6k9k (Part 2 out of 4)
Extension Beyond Unimodal Models (cont.)
As a first step beyond unimodal setting, we included new results on independently pretrained CLIP models taken from Nezhurina et al. (2025) where image and text encoders were jointly trained on paired image-text data. We compare both PARIS variants against an adapted Graph Metanetwork baseline, illustrating that our PARIS has potential beyond unimodal:
| Modality pair | Graph Metanetwork | PARIS-Ind | PARIS-Joint |
|---|---|---|---|
| Image-Image | 0.553 | 0.702 | 0.745 |
| Text-Text | 0.571 | 0.688 | 0.729 |
| Image-Text | 0.482 | 0.653 | 0.704 |
These results provide initial evidence that PARIS can predict representation relations involving both image and text modalities. We plan to investigate the cross-modal representational alignment (ie from independently trained unimodal vision and language models) directly from trained weights in future work (toward a weight-space Platonic test!).
We have added the pretrained CLIP experiments to and updated the discussion in
W3: It seems like one of the paper’s advantages is that they do not need to run inference on samples to compute similarity scores for new networks. However, runtime results are not included for measuring how much compute the meta-network requires throughout training or during inference. Therefore, it is challenging to reason about the usefulness of their method.
Q3: How does inference time compare with existing methods, and if inference time is faster with PARIS, is it fast enough to justify the meta-network training cost?
Computational Costs and Practical Motivation for Metanetwork-Based Similarity Prediction
We thank the reviewer for raising this question! We agree that it is important to understand the computational costs of our method for assessing the practical usefulness.
Training PARIS-Ind and PARIS-Joint requires approximately 5.56 and 7.64 GPU hours. We additionally benchmark both variants against direct activation-based evaluation with cached descriptors using 200 sites from distinct models:
| Measurement | PARIS-Ind | PARIS-Joint | Direct evaluation |
|---|---|---|---|
| Metanetwork training | ~5.56 GPU h | ~ 7.64 GPU h | — |
| Preparation of 200 sites | ~180 s | ~186 s | ~ 74.2 s |
| Comparison per pair | ~ 1.1 ms | ~265 ms | ~ 40.5 ms |
| Peak memory | ~ 0.34 GB | ~ 0.67 GB | ~ 3.82 GB |
We remark that:
-
Although direct evaluation is approximately 2.4 times faster during preparation, PARIS-Ind achieves an approximately 37-fold speedup in prepared pairwise comparisons and uses approximately 11 times less peak memory. We do not claim that PARIS is computationally preferable for one-off evaluations. Its computational advantage is most relevant when the trained metanetwork and encoded sites are reused across large numbers of comparisons.
-
Beyond computational acceleration, PARIS enables similarity prediction for new models without requiring probe data or running additional base-model inference. This makes it applicable in settings where direct activation-based evaluation is unavailable or impractical. This also avoids introducing variability from newly sampling probe examples at query time, and it allows the learned relationships to be reused across unseen models using only their trained computations.
Our central goal is to study whether a metanetwork can learn to predict internal representation similarities from trained computations and generalize to unseen models without requiring probe data or additional base-model inference. We view this capability as the main methodological motivation for studying metanetwork-based representation similarity, which goes beyond replacing direct similarity computation in settings where it is already inexpensive.
We have expanded to report training costs, inference runtime, and peak memory, and clarified the computational trade-offs and broader motivation for activation-free similarity prediction in
Response to Reviewer 6k9k (Part 3 out of 4)
W2: In figure 6, it is not clear why the target is an existing metric like a CKA-based dissimilarity. It is also not clear if there is a significant advantage in training a meta-network since the target metric can be directly computed in the situations the authors run experiments on.
Q2: Is there an advantage in training a meta-network to predict a metric when that metric can be directly computed in the situations the authors run experiments on?
Why Learn to Predict Existing Representation Similarity Measures?
We thank the reviewer for raising this question! We agree that existing similarity measures can be computed directly when suitable input data (e.g., paired) and activations are available. However, our goal is to study a different question: whether previous activation-based measurements can be used to learn a predictor that subsequently estimates relations between sites in newly encountered models from their trained computations. Beyond this scientific motivation, such an approach also offers practical advantages, as the upfront training cost can be amortized across many comparisons, potentially leading to substantial savings in computation and memory when comparing large numbers of representations.
We compute activation-based reference scores on held-out models to evaluate prediction quality. These activations are not inputs to PARIS. We use established measures as prediction targets because they capture different properties of internal representations. By training PARIS on similarity scores computed from activations, we investigate whether a metanetwork can learn these relationships from model parameters (i.e., those we called neural artifacts) and generalize to unseen models. Once trained, PARIS can predict these scores without requiring probe data or collecting new activations.
Prior metanetwork work has demonstrated that the parameters and computational structure of trained neural networks contain useful information about their properties. PARIS builds on this insight but addresses a different prediction target. Instead of predicting model-level properties (eg performance scores or characteristics of the training data), we aim to predict how similar two specific internal representations are under a given activation-based similarity measure. These similarities provide a more fine-grained view of how representations relate to one another, capturing aspects of their underlying geometry that go beyond individual model properties.
We also investigate whether this approach offers practical benefits:
-
PARIS-Ind produces compact site embeddings that can be reused across different comparisons and similarity measures. In our additional runtime experiments, prepared PARIS-Ind comparisons are approximately 37 times faster per pair of sites than direct activation-based comparison with cached descriptors, while using approximately 11 times less peak memory in the evaluated workload. Although training the metanetwork and encoding the graphs introduce upfront costs, these costs can be amortized across subsequent comparisons. Specifically, when performing many comparisons with large probe sets, the substantial gains in computational efficiency and memory usage can readily outweigh the initial investment, making PARIS-Ind advantageous at scale (Appendix D.3).
-
We evaluated whether predicted similarity scores can support downstream tasks (Sections 4.3 and 4.4), including model retrieval and stitching-candidate selection. We have further extended our evaluation to pretrained CLIP models, including representation comparisons across different modalities (updated in ), and investigate transfer with adaptation to a new population of pretrained vision encoders on Flowers-102 (updated ). We observe that both PARIS variants outperform the adapted Graph Metanetwork baseline in cross-modal image-text similarity prediction (0.653/0.704 vs. 0.482) and Flowers-102 transfer (0.64/0.73 vs. 0.58). These experiments provide additional evidence for the utilities and broader applicability of learning representation relations from trained computations.
Response to Reviewer 6k9k (Part 1 out of 4)
We thank the reviewer for their thoughtful comments and constructive suggestions! We appreciate their recognition of PARIS's ability to compare representations directly from model weights without relying on shared probe images and the empirical results.
Following the reviewer’s feedback, we have conducted additional experiments and revised the manuscript to clarify and strengthen several aspects of PARIS.
For convenience, we group related comments in our responses below and point to the corresponding revisions, which are highlighted in blue in the revised paper. In terms of weaknesses and questions:
Architecture Coverage and Generalization
We thank the reviewer for raising this important question about the generality of PARIS!
Yes, PARIS can indeed be trained on heterogeneous architecture families -- Appendix E.5 evaluates a metanetwork trained on a population containing both ResNets and ViTs. We have expanded this experiment with Deep Meta Classifier (DMC) and Graph Metanetwork baselines, and report the results:
We also introduce an out-of-distribution architecture evaluation, where the metanetwork is trained on architectures with smaller hidden dimensions and evaluated on held-out architectures with larger hidden dimensions. PARIS-Ind and PARIS-Joint achieve correlations of 0.715 and 0.732, respectively, compared with 0.375 for DMC and 0.654 for Graph Metanetwork. These experiments show that PARIS can learn from heterogeneous architecture populations and generalize across the tested architectural configurations.
We have expanded Appendix E.5 to include the additional metanetwork comparisons and OOD evaluation. Thanks so much for this comment!
We could also revise the discussion in the main text, but we wonder whether it would be better to keep these in the appendix to maintain the flow of the paper (which is what we have done for now). From a reader’s perspective, does the reviewer have a preference for one of these two options over the other?
Extension Beyond Unimodal Models
We thank the reviewer for this helpful comment and agree that extending our framework beyond unimodal setting is a valuable direction.
Existing approaches, including the MutualNN-based analysis, can compare representations across modalities using activations computed from paired observations. Our method addresses a complementary question: whether such representation relations can be predicted from trained computations without requiring input data or activations at inference. So these two are not competing estimators -- activation-based methods provide the similarity scores that PARIS learns to predict, while PARIS aims to predict these scores for unseen models without collecting new activations.
Extending the metanetwork framework to independently trained unimodal vision and language models or more structured outputs is not trivial, since it would require constructing suitable cross-modal reference relations (i.e., paired, partial paired, unpaired data) and adapting the computation-graph representation to different model architectures (eg different positional encoding), which we view as important and promising future work. We are actively working on this extension and plan to address it in a follow-up work toward a weight-space analogue of representational-convergence tests.