- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper is an experimental analysis paper which compares different weight generation methods.
The paper makes two contributions:
- An evaluation framework called WeightAudit to evaluate neural weight generation algorithms. It consists of 5 different metrics to measure novelty, functional competence, and behavioral diversity.
- A novel weight generation method called WINS, which trains a diffusion model to model the parameter distribution of high-performing network checkpoints. Training WINS involves a denoising loss which is computed over a set of model checkpoints (3000 in the experiments) and a supervised task loss which is computed over a subset of the training set of a specific task (2000 samples in the experiments).
The paper provides many experiments on small scale datasets, such as MNIST, KMNIST, EMNIST, SVHN and CIFAR10.
Critical strengths
The paper is in general well written, easy to follow and understand. The experimental protocols are documented well, which is especially important for an experimental analysis paper.
Critical weaknesses
There are several weaknesses to this paper:
Motivation of amortized DNN training The paper motivates neural network weight generation with amortized DNN training. More specifically, that the millions of checkpoints available should be used to learn good (conditional priors) rather than running gradient based training from scratch. Indeed, this is an interesting direction to either speed up training or save energy.
However, this motivation is not reflected in the proposed methods and experimental setup. In fact, WINS proposes an unconditional generator which is dataset and architecture dependent. In practice, WINS requires 3000 trained networks for the exact downstream task and target architecture to train the diffusion model that captures the model weight distribution. This diffusion model is then used to sample new model parameters for exactly the same task and same network architecture. The cost to train this population is excluded from the comparisons (See question 1).
To be of practical relevance, weight generation methods must allow either for cross-dataset or cross-architecture transfer.
Proposed metrics The five proposed metrics include a weight space based metric to measure the similarity of DNNs which is based on the Euclidean distance (Weight Dependence Gap). In experiments, this metric is used to measure novelty, i.e. if the generator memorized the training set. However, this is problematic because:
- Heterogenous layer scales Classification layers of CNNs are huge in comparison to the convolutional layers and might dominate the metric (See question 2).
- Scaling: Although the authors mentioned to use Git Re-Basin to standardize the checkpoints, this only helps against permutation of the network parameters. However, layers might also be subject to scaling, wich does not effect network behaviour if the network contains layer-norm.
- There are further problems for some DNN layers that are not discussed. E.g. the attention mechanism in transformers is invariant to transformations like and for invertible .
- Parameter space is not isotropic: Behavioral changes of a DNN when changing the model parameters are typically direction dependent. Small changes in some directions can cause severe performance degradation, while large changes in others can be ignored. This fact is for example used by pruning algorithms that set large parts of a DNN weight vector to zero without degrading its performance (See question 3)
-
Fairness of experiments* WINS leverages 2000 supervised samples to improve parameter generation quality. While the authors state that the other algorithms can have access to these samples as well, it is not obvious if they really do use these samples to compute a task-loss (see question 4).
-
Small scale experiments* Experiments are conducted on very simple datasets that seem inappropriate for an experimental analysis paper in 2026. Even limited compute resources should allow for more realistic setups.
Questions for authors
-
What is the end-to-end cost of WINS? This means constructing the 3000-checkpoint pool (7.64 GPU-hours according to Appendix G.2) plus generator training, compared with directly training the required number of networks (256). Under what scenario does generation pay off, given that the pool already contains 3000 networks of the target kind? Can the authors show any downstream use of the generated population? Can WINS be conditioned so it generates weights for a dataset or architecture it was not trained on? How would Differentiable Behavioral Supervision work in that case, since it requires labeled data from the target task?
-
Git Re-Basin removes only permutation symmetry. ReLU rescaling yields WDG = 1.15 for functionally identical networks (Table 9). How do the authors make sure WDG > 1 reflects real novelty rather than unremoved symmetry?
-
Euclidean distance treats all parameter directions equally, but function changes depend strongly on direction. Did the authors consider function-space or curvature-aware distances?
-
Is it correct, that baselines are trained on weights only, while WINS also uses labeled task data through the CE term. From the experiments it seems that most of the improvements come from task supervision. Can the authors add the same CE term to the strongest baselines (HyperDiffusion, DeepWeightFlow)?