- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The work presents a Graph Meta Network which learns the Flow Matching objective on MLP graphs, where nodes are biases and edges are weights, and positions this as a solution to the practical problems posed to weight space generative models by the permutation symmetries of modern neural network architectures. Their GMN solves this problem by virtue of being a message passing graph neural network and is therefore similarly permutation equivariant. The authors compare against established weight space generative models, namely P.diff [4], DeepWeightFlow (DWF) [2], and SANE [5] on a large dataset of small MLP and CNN models to demonstrate that their GMN flows survive larger training samples, while other methods do not. They preset a Joint Wasserstein Similarity (JWS) score and a permutation corrected cosine similarity as additional metrics for their experiments. Their experiments also investigate extrapolation to unseen MLP widths and applying the GMN flows in data/class conditioned settings.
Updated score from clear reject to weak reject on 11/18.
Sources:
[1] Geometric flow models over neural network weights, Ege Erdogan https://arxiv.org/pdf/2504.03710 and https://github.com/ege-erdogan/weightflow
[2] DeepWeightFlow: Re-Basined Flow Matching for Generating Neural Network Weights, Saumya Gupta and Scott Biggs and Moritz Laber and Zohair Shafi and Robin Walters and Ayan Paul, https://openreview.net/forum?id=fOwsr1VTi8
[3] Git Re-Basin: Merging Models modulo Permutation Symmetries, Samuel K. Ainsworth and Jonathan Hayase and Siddhartha Srinivasa, https://arxiv.org/abs/2209.04836
[4] Neural Network Diffusion, Kai Wang and Dongwen Tang and Boya Zeng and Yida Yin and Zhaopan Xu and Yukun Zhou and Zelin Zang and Trevor Darrell and Zhuang Liu and Yang You, https://arxiv.org/abs/2402.13144
[5] Towards Scalable and Versatile Weight Space Learning, Konstantin Schürholt and Michael W. Mahoney and Damian Borth, https://arxiv.org/abs/2406.09997
[6] Generative Modeling of Weights: Generalization or Memorization? Boya Zeng, Yida Yin, Zhiqiu Xu, Zhuang Liu, https://openreview.net/forum?id=QbSeApdgfi
[7] Graph Neural Networks for Learning Equivariant Representations of Neural Networks, Miltiadis Kofinas and Boris Knyazev and Yan Zhang and Yunlu Chen and Gertjan J. Burghouts and Efstratios Gavves and Cees G. M. Snoek and David W. Zhang, https://arxiv.org/abs/2403.12143
[8] Graph Metanetworks for Processing Diverse Neural Architectures, Derek Lim and Haggai Maron and Marc T. Law and Jonathan Lorraine and James Lucas, https://arxiv.org/abs/2312.04501
Critical strengths
Strengths:
(1) The authors show that their GMN appears to be able to generalize to new unseen MLP layer widths [Fig. 6]. While this is so far only demonstrated on small MLP models, it remains an exciting sign.
(2) In view of Zeng et. Al. [6], the proposed JWS metric is valuable, and their method shows signs of significant progress towards desirable scores by this metric.
(3) The work is well written and formatted, and the method is well described (despite my concerns regarding its novelty, see Weaknesses).
(4) The domain-conditioned transfer experiments are well thought out and genuinely insightful.
Critical weaknesses
Weaknesses:
(1) The presented method, a Graph Meta Network, appears to be a limited extension of the method of Erdogan [1], which applies a message passing graph neural network to learn the flow matching objective on MLP graphs (using the standard setup of Kofinas et al. [7]). The authors cite Erdogan, but only very briefly mention their work in the ‘Post-Hoc Alignment’ section [line 148], where they acknowledge that Erdogan’s method does permutation-equivariant flow matching on MLPs, but fail to mention that this is done with a similar message passing graph neural net and evaluated on similarly sized MLPs on similar/overlapping tasks (for example, MLPs on MNIST classification, see Erdogan sec 7 table 7.2). The identifiable difference between the presented method and that of Erdogan is simply the choice of AdaLN conditioning. Erdogan does not appear to use AdaLN conditioning in favor of a Relational Transformer setup (Erdogan Sec 6.3 [1]). The authors’ message passing neural net (MPNN) layers, which appear to be simply MLPs (Eqs 7-10) are similar to Erdogan’s (see Erdogan 6.3 [1] and ‘nn/gnn.py’ in the implementation), as Erdogan uses MLPs for edge updates (‘nn/gnn.py’ lines 846-878). Both these authors and Erdogan correctly credit Lim and Kofinas throughout their work, including for this edge updating MLPs [7,8]. Given the similarity of their GMN and Erdogan’s implementation and experimental settings, a discussion and comparison of their methods in context is necessary. The Abstract’s claims that “We instead parameterize a flow-matching velocity field with a permutation-equivariant Graph Meta Network … that our method closely …”, and the Conclusion’s opening line, “We proposed a permutation-equivariant GMN velocity field for learning weight distributions directly from independently trained networks…”, appear to be false in light of this prior work: the motivation for the GNN flow approach and the GNN flow matching itself do not appear to be uniquely attributable or novel contributions of these authors.
The concept of applying Graph Networks (and even the term Graph Meta Network and its acronym, GMN) are correctly attributed to Lim and Kofinas in the authors’ Section 4.2 [7, 8]. However, the contribution of Erdogan is precisely the application of these methods to the task of weight space generation, which the authors fail to identify [1].
In short, the method these authors present as a novel and significant contribution is in fact of limited novelty. Clarifying the true substance and novelty of their method in the context of Erdogan, Lim, and Kofinas is crucial [1,7,8]. Without access to the authors’ implementation these questions cannot be investigated further.
(2) The method is only demonstrated on tiny neural networks, without extension to ResNets or Transformers. The authors rightly identify that the cost of running GNNs on large and/or dense graphs balloons, but a discussion of this complexity and its evolution over model type, width, or depth does not appear to be present or clearly cited. The authors’ reasoning that models should work at small scales before they work at large scales is only defensible if the model could eventually work at said larger scale: however, this underlying question does not appear to be explored.
(3) The authors claimed results against aligned DWF are confounded by opaque hyperparameter selection, as well as scaling along a memorization prone axis during dataset scaling experiments [2, 3, 6]. When comparing against an alignment-based method, which uses the Git Re-Basin method to iteratively align the model dataset into a common loss basin, the step number/budget is a critical hyperparameter, and it is not discussed [2, 3]. It seems reasonable to expect larger and larger datasets of independently initialized and trained models to explore more and more functionally equivalent loss basins, leading to the intuition that a fair comparison of their GMN with an alignment-based method should hold the alignment step count or otherwise measured alignment budget in proportion to dataset size. If this is not the case, or if there is a tradeoff here, then it should be identified and explored rather than left as a source of confounding.
(4) The authors do not explore if/how the competitive SANE_KDE method performs in the same dataset-scaling setting as Weakness 3, despite SANE_KDE showing the next most desirable JWS scores in Table 1. By the JWS and IoU metrics presented there, SANE_KDE appears to be the more closely matched baseline, but they do not compare their GMN method against it in the data-scaling setting.
Questions for authors
Questions:
(1) The comparison against DWF with alignment is sensitive to the number of Git Re-Basin alignment steps used – the computational cost each step is tied to model size, not dataset size [2, 3]. What was the value used in these experiments, and how was it determined? Experiments which illuminate the tradeoffs between aligned and unaligned methods would clarify the papers position. Similarly, Figure 5 shows baseline methods at 2M and 10M parameter generator scales: however, these are obviously prone to memorization considering the small models they are tasked with generating [6]. Perhaps alignment budget or some aggregate compute metric would be a more balanced comparison [2, 3]?
(2) Table 1 would indicate that the method most competitive with GMN in the large corpus setting is SANE with the Kernel Density Estimator method (SANE_KDE) [5]. However, Figure 6 only shows GMN (potentially mislabeled as GNN?) vs unaligned and aligned DWF, and a DiT flow matching model which appears to be simply the unaligned DWF case. How does SANE_KDE’s performance evolve with dataset size, if at all? In the same setting, what is the difference between unaligned DWF and the MLP/DiT methods?
(3) The GMN method shows signs of successful extension to unseen MLP widths – can it also extend to unseen depths? If not, why not? Further, SANE can generate for unseen architectures as well – how do SANE and GMN compare in this setting [5]?
(4) What are the population and sample sizes in Table 1? The dataset sizes are presented in lines 346-348, but the number of models generated by SANE, P.diff, DWF, and GMN is not clear. Table 2 does not to report this data either.
(5) Can the domain transfer results of Figure 7 be compared against DWF in the same setting? They are clear and intriguing. DWF appendix K does briefly discuss this case but does not appear to be a mature method [2].
My score could change if:
• The contribution is clarified and compared against the method of Erdogan [1]. An anonymized implementation would provide invaluable context in continuing this portion of the review.
• Findings are more cleanly extracted from the present confounding hyperparameter and scaling choices with regard to comparisons against alignment-based methods like DeepWeightFlow.
• Comparisons against the competitive SANE_KDE method are provided or clearly linked in the main body.
• A discussion of scaling dynamics is provided or clearly cited within the main body.
9 comments
Response 2/5
Response to Weakness 1 (Part 2/2).
We did not claim to invent the GNN in the this paper. And still, the architectures differ.
Architectural differences beyond AdaLN with Erdogan (2025). As observed by the reviewer, our use of AdaLN is one of the differences with the work of Erdogan (2025). However, our implementation still differs from it as described in the thesis and in the implementation provided by the reviewer. Specifically, there are differences with respect to: (1) Aggregation functions, (2) Seperate MLPs for directions, (3) Global feature vector, (4) Residual Connections and more.
Architecture clarification and implementation access. We amended the architecture section of the submission to better reflect that (lines 249–269). At the request of the reviewer and in an act of good faith, we also provide an anonymized implementation of our GMN in a supplementary zip file. This should allow the reviewer to inspect these differences, and compare in detail with Erdogan (2025). We commit to releasing our full code, not just for the GMN, upon acceptance. If specific additional implementation components would help clarify the comparison, we would be happy to consider providing them.
Direct comparison. We also performed a direct comparison with Erdogan’s code. Please see our global message and the new Appendix G.
Response to Weakness 2
Evaluation scale and scalability. We acknowledge the limited network scale, which is due to technical limitations. We restricted ourselves to small, not tiny, neural networks. Yet, even at this scale, the evaluated baselines already exhibit clear memorization or poor accuracy, despite these methods supporting larger networks. In fact, if they already memorize at small scales, they will most likely memorize at larger scales: unreliability at smaller scales puts into question effectiveness on larger networks. This echoes recent findings, particularly on memorization (Zeng et al., 2026; Wang et al., 2026; Han et al., 2026), and makes reliable distribution learning an important challenge even at these small scales. We have expanded the main-text discussion of scalability and suggest concrete scaling directions in the limitations and future works discussion in Section 6 [lines 523–533]. Large-scale generation remains challenging future work.
Can our method scale? A GMN perpsective. In general, GMNs can already process substantially larger networks and Transformers than those we studied. Most notably, NiNo uses graph neural networks to predict weights during optimization, including for a 111.5M-parameter Llama3-style Transformer (Knyazev et al., 2025). Furthermore, GMN graph constructions already support residual connections, normalization, and multihead attention (Lim et al., 2024). The machinery of GMNs is thus ready to scale, and we have used in practice a GMN to compute the flow field, giving us hope that we might be able to scale our method when using a GMN in the future.
Can our method scale? Beyond GMNs’ perspective. Our alignment-free formulation is not restricted to GMNs, thus other architectures could be considered to the flow matching model. We only require a velocity field equivariant to the network’s permutation symmetries. If we would choose to specialize a generator to a particular finite family of architectures, permutation-equivariant Transformers could provide a scalable alternative (Zhou et al., 2023). Indeed, there are especially optimized libraries for Transformers.
Can our method scale? An equivariant compression perspective. Rather than generating the raw weights, the amount of which scales poorly, we could generate compressed versions of them, e.g. latent representations of them, as is done in regular applications of generative models. However, it is important to preserve the equivariance in the compression in order to preserve the equivariance of our approach. We have already performed and implemented in the original submission some levels of equivariant compression by using our input-layer folding and vector-valued kernel edges (Appendix C.2). Beyond these, symmetry-aware compression and hierarchical processing could offer further scaling directions, e.g. Section 7.2 of Han et al. (2026) and Section 5 of Wang et al. (2026). A compact latent representation could further reduce costs, provided the encoder, latent flow, and decoder remain equivariant under compatible symmetry actions.
In summary, while our method currently does not scale beyond small settings, we believe there are reasonable existing encouraging directions to perhaps allow our framework to scale in the future.
Response 3/5
Response to Weakness 3
Hyperparameters and Alignment. We have provided in the revised manuscript the requested clarifications regarding the used hyperparameters. We now make baseline selection and alignment explicit in Section 5.1 and Appendix C.7, and expand the scaling protocol and capacity-sensitivity analysis in Section 5.1.1 and Appendix D.
Regarding “opaque hyperparameter selection”. Table 1 reports results selected from 110 baseline configurations, including 81 DWF configurations. Appendix C.7 documents the searches and their results. For each dataset and setting, we report the configuration with the highest mean JWS. For CIFAR-10, DWF’s capacity approximately matches and slightly exceeds ours: 80M versus 77M parameters. We address the concern about generator capacity and memorization in our response to Question 1.
Regarding “alignment budget in proportion to dataset size”. Our protocol in the original submission already provides this: each network receives its own allowance of 100 matching iterations, so the total available budget grows linearly with collection size. Across all DWF experiments, each network is matched to the collection’s first network using canonicalization.py from the authors’ official repository. Matching stops after a complete sweep in which no layer’s assignment score improves by more than . For scaling, we align all 50K networks once and reuse prefixes. Each network is matched independetly to the reference network.
We also checked stopping behavior across the full collections: matching terminated after 2–8 iterations on mfeat-karhunen, 3–10 on MNIST, and 2–15 on CIFAR-10. Every network reached the early-stopping criterion before the cap, ruling out premature termination.
Response to Weakness 4
SANE_KDE scaling comparison. We added a requested comparison with SANE_KDE, which is the method providing the best results among baselines in Table 1 (but with JWS scores well below ours revealing quite some memorization issues). In the added experiment, we used 2M and 10M SANE_KDE models across all ten collection sizes to Section 5.1.1 and Figure 5, with additional training schedules and per-sample analysis in Appendix D.2–D.3.
SANE_KDE scaling results and memorization diagnostics. Regarding SANE_KDE scaling, the results reveal two distinct issues. The 10M SANE_KDE combines high accuracy with excessive weight similarity (WCS), clearly revealing memorization. For the 2M model, separate averages obscure a strong accuracy-WCS correlation (Pearson –): accurate outputs closely resemble training checkpoints, while less similar outputs perform poorly (Appendix D.3, Figure 13). Thus, again, revealing important memorization issues with SANE_KDE.
SANE correction. While carrying out these additional experiments, we actually found an evaluation error on our side, that we corrected (See Table 1). SANE_KDE used 30% randomly selected checkpoints instead of the top-performing 30% of the training collection, and SANE_Gauss omitted bootstrapping. Now SANE_KDE and SANE_Gauss report slightly higher values than originally reported, however, these updated results do not change the result landscape and conclusions drawn from them. Following our fix, we re-evaluated every results for these two specific baselines across the entire paper. In the revised version, all results for these methods use the corrected evaluation.
Response 1/5
We thank the reviewer for the feedback. We enjoy reading that they found our method and experiments "well thought out and genuenly insightful" and "clear and easy to follow". Before addressing the concerns and questions raised by the reviewer, we first clarify several aspects of our scope, representation, and theoretical contributions that do not seem to be accurately represented in the reviewer’s summary:
“a solution to the practical problems posed to weight space generative models” (Summary)
Our central focus is specifically to avoid permutation alignment when learning from independently trained networks.
“learns the Flow Matching objective on MLP graphs” (Summary)
Our formulation and experiments cover both MLP and CNN graphs.
“where nodes are biases and edges are weights” (Summary)
This is not our methodology, and it corresponds instead to that of a different work Erdogan (2025), where biases become bias features and nodes become node features, see Figure 1 of Erdogan (2025).
Instead, we follow the methodology of Lim et al. (2024). Quoting them: "Bias parameters could be encoded as node features as done by Zhang et al. (2023) or as self-loop edges on each neuron. Instead, we include a bias node for each layer and encode the bias parameters as edges from that node to the corresponding neurons in the layer." This is different from what the reviewer reported.
Furthermore, we did not only consider MLPs, but also CNNs. In CNNs, simply looking at nodes and biases is oversimplifying the networks. In particular, the convolutional kernel is encoded as a vector-valued edge feature, following Kofinas et al. (2024).
Additionally, on MNIST MLPs, naively applying Lim et al. (2024) makes training painfully slow. To overcome this issue, we proposed a trick that folds the input layer of MLPs into features of the first hidden layer (Appendix C.2; Figures 8–9). To the best of our knowledge, this has not been done before. In this trick, nodes are can actually end up containing weights.
“Their GMN solves this problem by virtue of being a message passing graph neural network” (Summary)
We provide a theoretical analysis of why alignment can be avoided, with proofs in Appendix A. This applies to any permutation-equivariant velocity parameterization, in particular, but not only to, a GMN (Secs. 4.1–4.2).
We now address the reviewer’s concerns and questions.
Response to Weakness 1 (Part 1/2).
Our novelty compared to Erdogan (2025). We appreciate the request for a clearer comparison with the master thesis of Erdogan (2025).
Although Erdogan (2025), which we referenced perhaps too much in passing, indeed proposed a similar construction, ours is different and novel. Note that the reviewer already recognized some distinctions between our work and Erdogan (2025), but more are present. We address concerns regarding our positioning with respect to this prior work in the general response above. We have revised our manuscript to better clarify our distinctions with prior work, especially in the Abstract, Introduction (lines 76-84), and Related Works (lines 162–167), Conclusion (lines 516-518) and clarify credit attribution to Erdogan’s (2025) prior work. We also included in Appendix G a more detailed methodological comparison with Erdogan (2025) than in the main manuscript, along with direct experimental evaluation. Beyond the many conceptual, technical, and methodological differences between our approach and prior work, the difference also clearly appears in terms of performance: our approach surpasses Erdogan (2025) empirically by significant gaps in terms of JWS (but also accuracy), see our new Fig 16, and our method does not run out of memory at moderately sized batch sizes of 256 (our default setting) on our compute, unlike Erdogan (2025).
We thank the reviewer for providing us with the opportunity to clarify our novelty stance.
We re-iterate our main contributions:
We are the first to propose equivariant flow models, specifically the GMN, to avoid alignment in weight space generation.
We back this up theoretically.
We show that we can train on multiple different architectures simultaneously, generate unseen widths, condition on tasks and generate unseen conditions. We consider this a remarkable achievement in the field.
We do a comprehensive memorization-aware evaluation, which is an issue recently observed in weight space generation. We propose a new metric to evaluate memorization.