acceptodds
Under review as a conference paper at ICLR 2027

A Data-Free Metric for Estimating the Similarity of Neural Networks

Abstract

The information learned by a neural network is encoded entirely in its weights and computation graph, but this information is viewed as practically inaccessible without probing the model's functionality with sample data. For this reason, existing methods for characterizing model similarity draw comparisons between outputs or internal representations, using a shared set of inputs. A weakness of such methods, however, is that evaluating model similarity based on a finite probing set does not guarantee similarity across all possible inputs. Here, we provide evidence that training-relevant information can, in fact, be gleaned from model weights directly and used to compare what models have learned. We present and validate the weight-based Ahmad RV coefficient (wARV): a novel, data-independent metric for quantifying neural network model similarity. We show that wARV captures training-relevant variation across different architectures, weight initializations, and batch ordering. Our results demonstrate the utility of wARV as a general-purpose similarity metric and provide a starting point for further applications leveraging what models learn in a data-agnostic manner.

Primary area:
interpretability and explainable AI
Keywords:
deep learning, model similarity, parameter space symmetry, model interpretation, computer vision

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zGjAOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
3
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper introduces wARV, a weight-space metric measuring neural network similarity without using data probes. wARV is based on the Ahmad RV coefficient, computed on the chain weight matrix of the neural network. Some layer types require adaptations or approximations.

The authors perform multiple experiments to demonstrate the capabilities of wARV. They show that their metric is very capable when trying to identify the task a model is trained on, including cross-architectures. They further note that their metric evolves throughout the training of the model, and can also capture differences such as adversarial training. They finally demonstrate that wARV evolves gradually with incremental mixes of training tasks.

Critical strengths

I find this paper very interesting, and like the idea behind wARV. This paper has, in my opinion, strong potential.

  1. S1 - Significance: wARV overcomes several limitations of existing methods to compare neural networks. First and foremost, it is not limited by the need to choose and use probes. Then, compared with other weight-space metrics, it is invariant to the intertwiner group, e.g. permutations. It therefore has the potential to become a standard distance metric for weight space learning.
  2. S2 - Results: Experimental results look promising. There is a relatively strong correlation and predictive power between wARV and training task. Cross-architectural compatibility of wARV is another strong advantage of the metric.
Critical weaknesses

In its current state, I still think, however, that this paper has key weaknesses. I look forward to the discussion period to see the authors’ opinion on these.

  1. W1 - Experiment setup: I find the experiment setup quality uneven throughout the paper. While Sections 4.1 and 4.4 are mostly clear, I have more difficulty with Sections 4.2 and 4.3. In 4.2, the authors use the AUROC metric, including for individual models, but do not detail how the predictions are performed, or what splits are used. It is also unclear what probes are being used for the cosine similarity between logits. In 4.3, the authors show a difference between normal and adversarial training using visual inspection of distribution curves: why not re-use the same metrics as in Section 4.2 to see whether wARV effectively classifies those two training schemes?
  2. W2 - Limitations: I think that the authors insufficiently explore and discuss the limitations of wARV. It is, however, crucial to do so: if one wants to use a metric, one should know to what it applies, and to what it does not. First, wARV omits nonlinear activations (except after batch normalisation layers) and bias terms: is that critical in some cases? What happens when activations are not in the extended ReLU family? What happens when the neural networks use biases? Furthermore, wARV uses several approximations, for example to represent attention or residual connections, and we see in the experiments that performance in larger BMLP and ViT drops, probably because of this. This raises questions: what would the performance of wARV be on even larger networks? For example, ViT-Small is twice as large as the larger ViT seen in the paper. Under what conditions would wARV fail because of these approximations? Finally, we see that wARV shows strong performance mostly for identifying the training tasks, and authors note that it works less well to identify training hyperparameters such as initialisation or batch ordering: it would be helpful for the conclusion to address what wARV can help measure, and what it cannot.
Questions for authors

Please refer to the Weaknesses Section.

6 comments
AuthorsOpenReview ↗

Response to Reviewer: Experiment setup and Limitation

We thank the reviewer for identifying unclear experimental details and for encouraging a more explicit discussion of the limitations of wARV. We will revise Sections 4.2 and 4.3 to increase clarity and add a separate Limitations section. We would appreciate feedback from the reviewer on whether these proposed changes fully address their concerns.

W1. Experimental setup and logic

"...how the predictions are performed, or what splits are used."

Whereas Section 4.1 reveals that some comparisons discriminate models by training task, it does not guarantee meaningful cross-architectural comparisons. Section 4.2 examines this through binary classification: AUROC quantifies how well wARV detects whether cross-architectural model pairs were trained on similar tasks, treating MNIST and EMNIST digits as the same task.

We use a simple prediction rule: "two models were trained on the same task if their wARV value exceeds a threshold." We took 1000 evenly spaced thresholds and computed the true/false positive rates for each. Since the goal is to quantify discrimination, no classifier is fitted and no separate train/test split is used. Fig. 2b reports AUROCs for comparisons involving the indicated architecture and other architectures, not individual models' predictive accuracies.

For cosine similarity of logits, we use 1000 training images, with 200 selected from each of the five task datasets.

"...why not re-use the same metrics as in Section 4.2...?"

Section 4.3 examines additional sources of variation in controlled settings. For discriminating model pairs trained using the same versus different methods (standard/adversarial), AUROC is 0.597 (95% CI [0.582, 0.610]), consistent with a smaller contribution of training method to variation in wARV.

W2. Limitations and interpretation

"...wARV omits nonlinear activations ... and bias terms..."

Omitting nonlinear activations results in loss of information when constructing the chain weight matrix. We suggest approximating activations where practical (Appendix B.6), particularly in pre-normalized skip connection blocks (B.7). Our invariance proof covers ReLU, leaky ReLU, and GELU. For other activations, invariance depends on the corresponding intertwiner group and is not guaranteed in general.

Regarding bias terms, the methods detailed in Appendix B can be extended to compute an analogous "chain bias vector," retaining invariance under the same symmetries. We explored combining cosine similarity of chain biases with wARV using a weighted sum, but we elected to focus on the weight component only for this paper. Models differing only in omitted bias terms can have identical wARV despite having different functions.

"...what would the performance of wARV be on even larger networks ?..what wARV can help measure, and what it cannot."

We agree that the chain weight matrix may lose critical information, especially for larger and deeper models. For example, our attention approximation omits the query and key matrices and cannot detect differences confined to those parameters. We have not established reliability for ViT-Small or a model-size threshold for approximation failure. Stress testing, customizing, and scaling wARV to larger models are important future directions.

However, a lower AUROC does not necessarily imply less accurate measurement. It can imply weaker discrimination of the specified factor, which may reflect other sources of learned variation as well as approximation error. AUROC should not be treated as the sole measure of wARV's validity.

We expect lower discrimination for subtler factors such as adversarial training. Trained models differing only in initialization or batch ordering are considered highly similar by wARV. This is consistent with the conjecture that SGD solutions can lie in a common basin after accounting for hidden-neuron permutations (Entezari et al., 2022), although wARV alone does not establish mode connectivity. When these random factors do not meaningfully alter what models learn, chance-level discrimination is desirable.

Our goal is to detect whether models learn similar "knowledge," rather than maximize discrimination of every training factor. The weights are updated through training while activation functions remain fixed. Our results show that these weights alone preserve substantial information about what models have learned: wARV identifies task as the dominant source of variation, detects subtler changes associated with training method, and recapitulates expected training dynamics.

We view wARV as a general framework, not a fixed method, and welcome extensions incorporating bias terms and alternate nonlinearities. As discussed in the paper, potential applications include studying linear mode connectivity and constraining unintended model changes during machine unlearning or interpretability interventions. High wARV, however, does not guarantee preservation of all functional behavior.

Reviewer_zGjAOpenReview ↗

Response to the authors' rebuttal

Dear authors,

I have read through your rebuttal and the other reviews, and I appreciate your on-point responses.

W1: Experiment setup is now much clearer to me. On Section 4.3, the quantitative result is appreciated. As discussed in your rebuttal, I find it perfectly acceptable that wARV carries less discriminative power regarding adversarial training. As long as the elements in the rebuttal are all written into the revision, I consider this weakness addressed, thank you.

W2: I remain convinced that a proposed metric such as wARV should be tested and proven on a wide variety of cases to be useful as a metric, and therefore for acceptance at a conference such as ICLR. Testing wARV on common designs like bias terms or realistic-size ViT is, in my opinion, needed. I unfortunately cannot see W2 as completely addressed without these additional experiments, since the limited scope in which wARV is tested would in turn limit the significance of the metric. I mostly agree with the rest of your remarks in your response.

Related to the other reviews. Both other reviewers highlighted existing works in the same area as wARV, which I also did not know about. Do you think it would be relevant to compare wARV to at least Chen et al. (2023) - which I am not familiar with, in a similar way as the authors compare wARV with cosine similarity on logits? It would be helpful in understanding the differences between both metrics.

In general, I would reiterate that I see strong potential in this paper. However, with the proposed method being a metric, I think the scope in which it is tested and proven remains at this stage too narrow and therefore negatively affects the significance of wARV.

Reviewer zGjA

AuthorsOpenReview ↗

Larger ViT experiments, bias clarification, and similarity benchmark

We thank the reviewer for encouraging us to test wARV in additional settings. Below, we provide additional evidence and clarify the remaining limitations. We would appreciate feedback on whether these additions address the reviewer's concerns.

1. wARV on a realistic-size ViT

We computed wARV across publicly available OpenCLIP ViT-B/32 training checkpoints, similar to the analysis of BMLP training dynamics in Fig. 3a. This model has ~150M parameters. Here we report wARV between checkpoints at different epochs:

epoch b \ a124812162428
20.22
40.110.55
80.070.400.60
120.070.400.580.72
160.100.360.540.670.72
240.050.170.290.330.380.39
280.050.110.190.240.280.250.91
320.050.110.170.210.250.230.890.99

From epochs 1 to 16, wARV recapitulates expected training dynamics: nearby checkpoints generally have higher similarity than more distant checkpoints, and similarity between nearby checkpoints increases as training converges. Notably, epochs 24–32 form a highly similar group, with only moderate similarity to earlier checkpoints. This suggests a change followed by stabilization in the weight structure captured by wARV, may indicate a transition between local basins with prolonged training. We will include this result (as a heatmap similar to Fig. 3a) in revision.

2. wARV on models with bias terms

We apologize that this was unclear: every model evaluated in our experiments, including MLP, BMLP, LeNet, ResNet, and ViT, contains bias terms. Thus, our experiments already evaluated wARV on models with biases, although the chain weight approximation omits these terms. Our exploratory attempts to combine chain weight and bias information did not consistently improve the results and sometimes degraded them, we thus focused on weights alone. Our findings suggest that weights alone capture significant aspects of model variation, and that bias and nonlinearity may not play as an important role in learnt variation as weights.

3. Comparison with other similarity measures, particularly Chen et al.

We agree that comparing wARV with measures beyond logit cosine similarity is important. In our exploratory work, we compared wARV with functional similarities based on accuracy, prediction, Jensen–Shannon divergence of probabilities, besides logit cosine similarity, as well as with representational similarities Procrustes and Centered Kernel Alignment (CKA). Notably, Chen et al. (2023) report a correlation of 0.955 between CKA and their method in Fig. 2.

We will include the following benchmarking result in revised SI:

We considered 50 pairs of independently initialized, untrained networks and 50 pairs of consecutive checkpoints near the end of training, serving as low- and high-similarity reference anchors, respectively. We report here similarity values (mean ± SD). The separation gap Δ = min(consecutive-checkpoints similarity) − max(random-network similarity); larger positive values indicate better separation between anchors.

MeasureRandom networksConsecutive checkpointsSeparation gap Δ
Accuracy similarity0.98 ± 0.021.00 ± 0.00−0.01
Prediction agreement0.10 ± 0.090.99 ± 0.01+0.63
Logit cosine0.09 ± 0.120.98 ± 0.02+0.55
1 − JSD(probability)0.98 ± 0.000.99 ± 0.010.00
Procrustes0.88 ± 0.020.99 ± 0.01+0.03
CKA (0.955 correlation with Chen et al.)0.71 ± 0.100.98 ± 0.02+0.06
wARV0.00 ± 0.000.98 ± 0.02+0.93

Among the measures evaluated, wARV is the best calibrated: it has a tightly concentrated baseline near zero for independently initialized networks, versus similarity near one for consecutive late-training checkpoints.

We prioritize these widely used measures because they apply to independently trained models across architectures. Functional similarity has been used as near groundtruth in previous studies evaluating representational similarity, our reply to Reviewer k1Zy explains our choice of logit cosine as a representative functional measure. While we are impressed by Chen et al.'s method, it assumes filter-atom decomposition with layerwise shared atom coefficients, and we did not find publicly available checkpoints in such setting.

Reviewer_5DKcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes wARV, a framework for comparing neural networks without probing data. It maps a network to a chain weight matrix by composing linear approximations of its components, then compares these matrices using Ahmad's bias-corrected RV coefficient. The main experiments cover five image classification tasks with different architecture families and training setups. The paper reports an AUROC of 0.83 for identifying models trained on the same task across architectures. Additional experiments examine gradual task changes, residual connections, and pretrained ImageNet models.

Critical strengths
  1. Assessing model similarity without access to probing data is an interesting and critical problem. Extending chain weight representations to convolutional, residual, and attention-based architectures provides a concrete framework for studying it.

  2. The factorial experiment covers several sources of model variation. The gradual class-replacement experiment and supplementary pretrained ImageNet models provide useful evidence beyond binary task identification.

Critical weaknesses
  1. The novelty relative to existing data-free similarity methods is unclear. Chen et al. [1] already compare networks without probing data and study task relationships and training dynamics. Their method requires shared atom coefficients. The paper neither discusses this work nor compares against it in a compatible setting.

  2. The experiments mainly show that wARV can identify models trained on the same task. This does not show that those models make similar predictions. Among models trained on the same task, wARV has a Spearman correlation of only 0.18 with logit similarity. The evidence for the claimed functional similarity is therefore limited.

Reference

[1] Wei Chen, Zichen Miao, and Qiang Qiu. Inner Product-based Neural Network Similarity. NeurIPS, 2023.

Questions for authors
  1. Does the claimed invariance cover independently permuting one model's output classes? Please explain how this follows from ARV's orthogonal invariance and how classes are matched across unrelated tasks.

  2. What sampling model justifies the significance test for jointly trained class rows? The main experiments have only ten classes. Please validate false-positive rates on networks, beyond the Gaussian-matrix simulations in Figure 5.

  3. How were PERMANOVA permutations and AUROC confidence intervals computed? The 1,200 checkpoints share 600 training runs, and pairwise comparisons reuse models. Please account for these dependencies in resampling.

4 comments
AuthorsOpenReview ↗

Response to Reviewer: Novelty, Functional similarity, Invariance, and Statistical validation

We thank the reviewer for these constructive comments. We would appreciate feedback on whether the proposed changes address their concerns.

W1. Novelty relative to Chen et al.

We agree that Chen et al. (2023) is important and relevant, and apologize for overlooking it. We will incorporate the following into the "Weight-Based Similarity" subsection:

"Building on filter atom decomposition for continual learning [Miao et al., 2022], Chen et al. (2023) introduced an efficient, probing-free similarity measure based on cosine similarity between corresponding filter atoms. Their formulation assumes layerwise shared atom coefficients and analyzes task relationships and training dynamics in federated and continual learning paradigms."

wARV constructs chain weight matrices from independently trained neural networks and does not require shared filter atom coefficients, corresponding hidden layers, or the same architecture. Beyond data independence, our contribution lies in constructing a comparable weight space across architectures.

W2. Relationship to functional similarity

Identifying reliable ground truth for model similarity remains an open challenge. In Table 3 of Klabunde et al. (2025), none of the 24 representational distance measures correlates significantly with both accuracy and hard-prediction differences at the 0.01 level; many correlations are significantly negative. This broad variability illustrates how similarities measured from different perspectives can disagree while still providing meaningful insights, and how astonishingly difficult it is to reconcile different similarity measures.

We view as cosine similarity is a ``reasonable" measure of model similarity, see our reply to reviewer k1Zy for reasons we chose it as the "representative". Its utility , however, is limited by its dependence on the set of inputs on which it is calculated (Fig. 6). Logit cosine similarity is therefore an inherently biased metric. The low correlation we report between wARV and cosine similarity is at least in part due to our inability to better probe the possible space of inputs to the models, that is, to reduce the bias in its estimation.

The validity of wARV does not depend solely on high correlation with logit cosine similarity: it recovers known sources of model variation and recapitulates training dynamics. We do not claim wARV as the single "best" metric and advocate interpreting it together with other measures. Nevertheless, it captures learned information in weights, enabling data-free, cross-architecture comparison.

Q1. Output-class permutations

In the output-by-input orientation used in the paper, wARV is generally not invariant to independently permuting one model's output classes. We will correct the corresponding statements and provide proofs in the revision.

ARV's orthogonal invariance applies to its variable, or column, axis; output classes index rows in our formulation. Applying the same output permutation to both models preserves the score. Our implementation pairs identical classes where possible, but falls back to numerical class indices for unrelated tasks. The latter does not establish semantic correspondence, and scores can depend on the chosen class ordering.

When output correspondence is unknown, computing wARV in the input-by-output orientation instead provides invariance to independent output-class permutations, while requiring corresponding input coordinates. More broadly, the desired invariance properties for comparing neural networks remain an open question.

Q2. Significance-test calibration

We acknowledge that joint training can violate the sampling assumptions underlying the ARV significance test.

We are currently examining two surrogate nulls: randomly permuted row correspondences in trained chain weight matrices and chain matrices constructed from products of random weight matrices. We will specify these null models and report their empirical rejection rates in Appendix A.

Q3. Dependent resampling

Our original PERMANOVA calculations used unrestricted permutations and did not explicitly account for dependence between checkpoints from the same training run. We acknowledge that this may affect calibration of the reported p-values. We will recompute significance using factor-specific restricted permutations that preserve the relevant dependence structure and revise the associated significance claims accordingly.

We compute AUROC confidence intervals by bootstrapping. Although cross-architecture comparisons exclude pairs from the same training run, comparisons still reuse models, which may affect CI calibration. We will therefore recompute AUROC and its confidence interval using non-overlapping cross-architecture model pairs, with each training run contributing to only one pair, and bootstrap these pairs. We will provide the resampling details and code with the revision.

AuthorsOpenReview ↗

Updated PERMANOVA results with restricted permutations

We thank the reviewer for guiding us towards more appropriate statistical analyses. Following our previous response, we repeated PERMANOVA for wARV and logit cosine similarity using 9,999 restricted permutations. For run-level factors, we permuted training runs as units, keeping their checkpoints together; for epoch, we swapped labels only within runs.

SourcewARV R²wARV pLogit cosine R²Logit cosine p
Task0.37820.00010.58780.0001
Architecture0.08160.00010.12940.0001
Training Method0.00460.00020.00270.0006
Epoch0.00300.00010.00470.0001
Batch Ordering0.00100.250.00020.86
Weight Initialization0.00150.490.00140.14
Residual0.52990.2738

Our conclusions in the paper remain unchanged: wARV identifies task as the dominant source of variation, followed by architecture; it captures subtler but significant variations from training method and epoch; and suggests random factors — batch ordering and weight initialization — account for little variation. These patterns are consistent with cosine similarity of logits. Restricted permutation increases p values for weight initialization and batch ordering, which are now clearly nonsignificant.

We also clarify that model reuse in pairwise comparison does not invalidate PERMANOVA. We do not permute individual distances as independent observations. Instead, we permute model labels, equivalently permuting rows and columns of the distance matrix together.

We will update PERMANOVA results and resampling details in the revision.

AuthorsOpenReview ↗

Updates on Significance Test Calibration and AUROC Confidence Interval

Q2. We have performed additional experiments to confirm the calibration of the significance test in more realistic null settings than the weak- and strong-spike random matrices we reported originally:

  1. We have generated row-correlated random matrices by multiplying together two random matrices. We find that calibration remains identical to the other null models reported in figure 5.
  2. We have performed comparisons between the chain weight matrix generated from a trained model and one random row permutation of that matrix (N=600 total). We find that across all significance levels tested, [0.001, 0.002, ... 0.050], the fraction of comparisons having p values below a particular significance level very closely matches that significance level. This linear trend supports the calibration of the significance test.

Q3. We have performed a Monte Carlo subsampling procedure to eliminate the dependencies present in our original bootstrapping computation of AUROC confidence intervals. Briefly, a sample is drawn by randomly picking model pairs without including any individual model more than once in the sample. The procedure is repeated 10,000 times, with AUROC computed for each sample.

The mean AUROC value reported in Section 4.2 is unaffected, but the 95% confidence interval is expanded to [0.760, 0.852]. We note that this is a conservative confidence interval generated from small samples, and we emphasize that this does not change any of our core conclusions. Nevertheless, we thank the reviewer for this interesting statistical suggestion.

Thank authors for the detailed response. Most of my concerns have been alleviated. Hope the changes will be taken in the revision. I'd raise my score accordingly.

Reviewer_k1ZyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a common comparison space for neural networks of distinct architectures, constructed from a model’s parameters and computation graph, and invariant to key weight-space reparameterizations. In particular, this paper tries to map heterogeneous NNs into a shared input-to-output operator space derived from their trained states.

Critical strengths
  • Introducing technical challenges clearly:

    • First, many model architectures and components are in common use, including multi-layer perceptrons (MLPs), convolutional neural networks CNNs), and transformers, with or without skip connections, batch and layer normalization, and max pooling. There is no obvious correspondence between weights across different architectures
    • Second, even for identical architectures, it is not straightforward to obtain a 1:1 weight mapping between models, as neurons in a hidden layer can be reordered without changing the network outputs
  • Considering realistic assumptions: differ in classification task, architecture, training method, and initialization

  • Trying to show that collapsing a model’s weights and computation graph into a permutation-invariant input-to-output comparison space preserves substantial information about what a model has learned.

Critical weaknesses
  • The logic and flow is broken. For example, I struggle to understand the connection between challeneg introduced in the first sentence and similarity in the second sentence: "A fundamental challenge in modern deep learning is understanding how artificial neural networks (NNs) behave when faced with unpredictable inputs that may fall outside the training distribution. Analyzing the similarity of NN functionality is typically accomplished by comparing output logits or internal representations across a shared set of inputs"

  • Partial cover of the literature. This paper only discuses existing methods that characterize model similarity based on comparisons between outputs or internal representations, using a shared set of inputs. This paper misses some part of literature such as similarity based on local approximation: A zest of lime: Towards architecture-independent model distances ICLR 2022.

  • The report of accuracy and the impact of that on the sensitivity of the proposed solutions are missing. This paper claims " We train models from scratch on one of five tasks: MNIST, EMNIST Digits, CIFAR10, and two ImageNet tasks corresponding to the CIFAR10 classes (C10-ImgNet, a subset of CINIC10 (Darlow et al., 2018)) or to distinct categories (Not-C10)." and then "We consider five architecture families – MLP, Bottleneck MLP (BMLP) (Bachmann et al., 2023), LeNet (LeCun et al., 1998), ResNet (He et al., 2016), and Vision Transformer (ViT) (Dosovitskiy et al., 2021) – each with approximately one-million- and eleven-million-parameter variants." Given this varieties of dataste and architecutres, it is important to measure and report the accuracy and analyze its impact to the findings.

  • Unsupported and vague choices. For example, could you elaborate on "We train the NNs with either clean samples or a 50/50 mixture of clean and FGSM-perturbed adversarial samples Goodfellow et al. (2015)."?

  • Lack of appropriate comparisons with existing methods. Section 4.1 and Table 1 only compares with cosine similarity of logits without clarifying if this is one specific work in the literature or an arbitrary choice.

  • Problematic mathematical notations.

    • undefined and unnecessary notations: θi∈Rνiθ_i \in R^{ν_i} in line 99, what is viv_i?
    • lost transition from θθ to WW from 3.1 to 3.2
    • pip_i and qq lost and not used anymore, instead dL×d0d_L \times d_0 used
  • This paper misses articulating the purpose of similarity they are interested in. For example, if the use of similarity is for catching model stealing, does the proposed solution can support?

Questions for authors

see above

4 comments
AuthorsOpenReview ↗

Response to Reviewer: Motivation, Related Work, Experiments, and Notation

We will incorporate the changes described below into a consolidated revised manuscript, which we will upload once our other updates are complete. Before then, we would appreciate feedback from the reviewer on whether these proposed changes fully address their concerns:

1. Motivation and logical flow

We are sorry that our language was insufficiently clear.

What we were trying to communicate is that when model similarity is functional or representational in nature then it can only be calculated on concrete sets of inputs. Those sets are, in practice, finite and thus cannot probe the similarity of the responses of the models to un-sampled inputs.

In contrast, by construction, a similarity metric that is parametric in nature will not depend on particular inputs and will provide a more complete metric of similarity insensitive to input sampling strategies.

2. Coverage of the literature

We thank the reviewer for bringing up this weakness in our literature review.

We do agree that the type of approaches to model similarity that include LIME are important, and we are sorry for having overlooked them.

The revised manuscript will include a new paragraph in the related work section discussing this line of work:

Similarity via local approximations. Another line of work compares models by approximating their local behaviors. Utilizing Local Interpretable Model-Agnostic Explanations (LIME) [Ribeiro et al. (2016)], Zest [Jia et al. (2022)] fits local linear surrogate models around reference data points, then compares concatenated surrogate coefficients via cosine similarity. Zest enables architecture-independent comparison without accessing model weights, and has been applied to catch model stealing and verify machine unlearning. Although Zest operates on surrogate weights, surrogate models are fitted on model predictions of reference data points and their perturbations. wARV provides a complementary perspective by constructing a chain weight matrix directly from original model weights.

3. Accuracy and its impact

We thank the reviewer for bringing up this matter. The revised SI will include a table with detailed accuracy information so that readers can check the results.

One of our main results is the fact that different architectures, which achieve very different levels of accuracy on the more challenging tasks, are still amenable to comparison of their similarity using wARV. In our view, this speaks to its power to capture more task-relevant variation than is possible with accuracy measurements on a narrow range of labeled data.

4. Clean and adversarial training samples

We are sorry that we failed to make the reasons for this choice clearer.

A 50/50 mixture of clean and adversarially attacked images for adversarial training is a standard practice. In fact, it is the single protocol used by Goodfellow et al. (2015) in their foundational study.

Further speaking to how standard this choice is: it is the default choice in adversarial training packages such as Adversarial Robustness Toolbox: AdversarialTrainer(classifier, attacks, ratio=0.5).

5. Choice of cosine similarity of logits

The revised SI will include a new section, "Functional similarities and choice of cosine similarity of logits," to elaborate on this matter.

The figure references below refer to the forthcoming revised SI.

Briefly, we chose the cosine similarity of logits to compare with wARV in the main paper because it is relatively robust to out-of-distribution evaluation sets (Fig. 6), highly correlated with prediction agreement and probability similarity (Fig. 7), and shows near zero similarity for independently initialized, untrained NNs (Fig. 8).

6. Mathematical notation

What is νᵢ in the definition of θᵢ?

νᵢ is the number of weights in the model.

Transition from θ to W from 3.1 to 3.2.

θ is the multidimensional vector with the values of all the weights. When calculating M it is more natural to express it in terms of the individual matrices W.

pᵢ and q are replaced by d_L × d_0.

We use a general formulation in 3.1 to describe our generic goal with respect to the input and output dimensions of the models, while emphasizing which of these dimensions must be compatible. We then use a more specific formulation in 3.2 when detailing our approach for generating a mapping for a single model, which involves considering the inner dimensions throughout that model. The shift in notation was intentional to make the distinction from general to concrete clear.

7. Purpose and applications

If the reviewer is suggesting that our similarity could be used for detecting model stealing, we are not sure whether this would be a straightforward application. This could be an exciting matter to investigate using our metric.

Nevertheless, we have already explained in detail three possible areas of application of our metric in the discussion section of the manuscript.

Reviewer_k1ZyOpenReview ↗

Thanks for your response, a couple of follow-up questions

In contrast, by construction, a similarity metric that is parametric in nature will not depend on particular inputs

Why this is the case? a metric can still be data dependant or needs data to tune its parameters.

We do agree that the type of approaches to model similarity that include LIME are important, and we are sorry for having overlooked them.

Any empirical comparison against those missing work? If not, why?

The revised SI will include a table with detailed accuracy information so that readers can check the results.

How those results impact your findings?

AuthorsOpenReview ↗

Data Independence, Empirical Comparison with Zest, and Accuracy-Matched Analysis

We thank the reviewer for these follow-up questions and would appreciate feedback on whether the clarifications and additional analyses below address their concerns.

1. Language revision

We apologize for incorrectly implying that all parametric measures are data-free. Indeed, Zest of lime (Jia et al., 2022) operates on surrogate weights fitted using model predictions on reference inputs and their perturbations. Thus, Zest is not data-free. wARV, specifically, is data independent, as it operates directly on original weights and requires neither reference inputs nor surrogate fitting.

We are revising the opening sentences of the Introduction to:

Understanding whether artificial neural networks (NNs) have learnt similar “knowledge” is a fundamental challenge in modern deep learning. Analyzing the similarity of NNs is typically accomplished by comparing outputs or internal representations across a shared set of inputs (Klabunde et al., 2025a).

2. Empirical comparison with Zest

We have now compared wARV and Zest using publicly available ViT-B/32 and ViT-L/14 checkpoints fine-tuned on MNIST (M), SVHN (S), and EuroSAT (E). MNIST and SVHN are similar tasks recognizing digits 0–9, whereas EuroSAT classifies land-cover, unrelated to digits.

wARV (higher = more similar)

L14-ML14-SL14-E
B32-M0.4320.3960.073
B32-S0.4700.4450.046
B32-E−0.033−0.0390.679

Zest cosine distance (lower = more similar)

L14-ML14-SL14-E
B32-M0.0820.1290.143
B32-S0.0650.0510.084
B32-E0.1120.0720.079

In these comparisons, wARV consistently groups the digit-recognition tasks together and separates them from EuroSAT. Zest instead identifies ViT-B/32 fine-tuned on EuroSAT as slightly more similar to ViT-L/14 fine-tuned on SVHN (0.072) than on EuroSAT (0.079). We argue that wARV better captures task-relevant learned "knowledge", while Zest may be particularly suitable for catching model stealing.

wARV is also computationally more efficient -- it operates directly on original weights, whereas Zest requires model evaluations on perturbed reference inputs and LIME fitting. Using 128 reference images and 1,000 perturbations per image, as in the Zest paper, we obtained the following empirical runtimes:

MethodHardwareViT-B/32ViT-L/14
wARV8 CPU cores8.1 s23.2 s
ZestA100 GPU2.6 min45.8 min

To summarize our novelty:

  • wARV is fully data-free, requiring neither reference inputs nor surrogate models;
  • wARV better captures task-relevant variation;
  • wARV is computationally efficient.

3. Accuracy-matching sensitivity analysis

We understand the concern that accuracy could confound comparisons between models trained on tasks of different difficulty. We therefore conducted a sensitivity analysis, matching accuracy for different-task pairs vs. same-task pairs with a 5% tolerance. After matching, wARV's AUROC is 0.81 [0.76, 0.86], closely aligning with the original analysis, 0.81 [0.76, 0.85], and accuracy similarity's AUROC is near chance, 0.51 [0.44, 0.59].

AuthorsOpenReview ↗

Short summary of our answers to your follow-up questions

Dear Reviewer k1Zy,

Thank you for your follow-up questions of 16 Nov. To make them quick to check, here is a one-line summary of how we addressed each one (details are in our response above):

  1. Data dependence. We agree that a parametric metric can still be data-dependent, and we have revised the Introduction accordingly. Our claim is now limited to wARV itself, which uses only the trained weights and no reference inputs.

  2. Empirical comparison with Zest. We added a comparison on ViT-B/32 and ViT-L/14 checkpoints fine-tuned on MNIST, SVHN and EuroSAT, including runtimes for both methods.

  3. Impact of accuracy. Using accuracy-matched pairs (5% tolerance), the AUROC of wARV is unchanged (0.81 [0.76, 0.86]), while accuracy similarity alone is at chance (0.51 [0.44, 0.59]).

Could you let us know whether these answer your three questions, and which of the concerns in your original review, if any, you still consider open? If something remains unresolved, we would be glad to run a further analysis in the time remaining.

Best regards,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated PDF Has Been Uploaded

We thank the reviewers for your thoughtful comments and considerate engagement throughout the review period. We greatly appreciate your time and feedback. The manuscript has been updated to reflect the changes discussed.

Summary of the primary changes:

  1. wARV on larger ViT models: Lines 369-371, Appendix G
  2. Comparison with other similarity metrics: Appendix F, G
  3. Justified choice of cosine similarity as comparison: Lines 224-226, Appendix D
  4. Extended Literature Review: Section 2
  5. Restricted-Factor PERMANOVA: Lines 237-240, updated Table 1
  6. wARV identifies training dataset using reference models: Lines 313-316, Appendix E
  7. Confidence Interval Estimation for AUROC: Lines 303-310, updated Figure 2a
  8. Significance Test Calibration: Appendix A (Lines 801-806 and added Figure 6)
  9. Accuracy-Matching Sensitivity Analysis: New experiment in Lines 316-321, raw accuracy tables in Appendix J
  10. Limitations Section: Added Section 6
  11. Clarification that wARV is not invariant to row permutations: Lines 185-190, 200-201, Appendix C.2

Section G.2.2: In the course of performing experiments on OpenCLIP models (per reviewers k1Zy’s and zGjA’s suggestions), we wondered to what extent observed variation could be attributed to fine-tuning of the vision encoder vs. to the classification head frozen during fine-tuning. Compared with the zero-shot baseline, wARV identifies additional similarities between models fine-tuned for similar tasks, while further differentiating models fine-tuned on different tasks. These results are in Appendix G.2.2.

For the reviewers' reference, we provide a list of specific modifications we made in response to their reviews and subsequent comments. Note that # refers to a primary change listed above.

Reviewer k1Zy:

  • W1: We updated language in the abstract and lines 29-30
  • W2: #4
  • W3: #9
  • W4: We clarified language in lines 208-209, emphasizing standard practice for adversarial training
  • W5: #2, #3
  • W6: We clarified the meaning of our notation in lines 114 and 121-122
  • W7: We clarified lines 21-23. We reiterate that the purpose of wARV is to measure model similarity in a data-free manner (see specific contributions lines 58-59, as well as the Conclusion). We emphasize lines 482-500 in the Discussion, which provide three suggested applications of wARV. See also #6.
  • Response Q1: We updated language in lines 29-30
  • Response Q2: #2
  • Response Q3: #9

Reviewer 5DKc:

  • W1: #2, #4
  • W2: We clarify that reconciling similarity metrics from across the field is an important open question in lines 355-357, and discuss this in more depth in the Limitations section (lines 518-523).
  • Q1: #11
  • Q2: #8
  • Q3: #5, #7

Reviewer zGjA:

  • W1: We clarified the experimental setup in lines 298-302. We quantified the training method result in line 404.
  • W2: We clarify that the trained models have bias terms that are not considered in wARV in lines 209-210. See also #10 for limitations and #1 for larger ViTs.
  • Response 1: #2
  • Response 2: For wARV on multiple ViT model sizes - see #1. For discussion on bias terms - see lines 505-511.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.