A Data-Free Metric for Estimating the Similarity of Neural Networks

The information learned by a neural network is encoded entirely in its weights and computation graph, but this information is viewed as practically inaccessible without probing the model's functionality with sample data. For this reason, existing methods for characterizing model similarity draw comparisons between outputs or internal representations, using a shared set of inputs. A weakness of such methods, however, is that evaluating model similarity based on a finite probing set does not guarantee similarity across all possible inputs. Here, we provide evidence that training-relevant information can, in fact, be gleaned from model weights directly and used to compare what models have learned. We present and validate the weight-based Ahmad RV coefficient (wARV): a novel, data-independent metric for quantifying neural network model similarity. We show that wARV captures training-relevant variation across different architectures, weight initializations, and batch ordering. Our results demonstrate the utility of wARV as a general-purpose similarity metric and provide a starting point for further applications leveraging what models learn in a data-agnostic manner.