acceptodds
Under review as a conference paper at ICLR 2027

Permutation-Equivariant Flow Matching for Alignment-Free Neural Weight Generation

123 views

Abstract

A trained neural network can be represented by a parameter vector in high dimensions. Learning distributions over these vectors enables the generation of new models across various tasks and architectures. A central challenge is permutation symmetry: permuting hidden neurons can produce distant parameter vectors representing the same function. This introduces variations that a generative model must account for when learning from trained networks. Existing methods typically address this using networks derived from a common base model or costly approximate neuron alignment. We instead parameterize a flow-matching velocity field with a permutation-equivariant Graph Meta Network, enabling direct learning from independently trained networks without alignment. Extensive experiments show that our method reproduces the joint statistics of accuracy, functional similarity, and weight similarity of independently trained collections, providing evidence of generation beyond checkpoint memorization. A single conditional model also generates task-specific networks on heterogeneous architectures and generalizes to unseen hidden-width configurations. On a tabular domain-shift task, intermediate conditioning produces individual networks with performance comparable to logit ensembles across both domains. Taken together, our results show how permutation equivariance enables learning from diverse collections of independently trained networks without permutation alignment.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
graph neural networks, weight space, generative models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dNR9OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work presents a Graph Meta Network which learns the Flow Matching objective on MLP graphs, where nodes are biases and edges are weights, and positions this as a solution to the practical problems posed to weight space generative models by the permutation symmetries of modern neural network architectures. Their GMN solves this problem by virtue of being a message passing graph neural network and is therefore similarly permutation equivariant. The authors compare against established weight space generative models, namely P.diff [4], DeepWeightFlow (DWF) [2], and SANE [5] on a large dataset of small MLP and CNN models to demonstrate that their GMN flows survive larger training samples, while other methods do not. They preset a Joint Wasserstein Similarity (JWS) score and a permutation corrected cosine similarity as additional metrics for their experiments. Their experiments also investigate extrapolation to unseen MLP widths and applying the GMN flows in data/class conditioned settings.

Updated score from clear reject to weak reject on 11/18.

Sources:

[1] Geometric flow models over neural network weights, Ege Erdogan https://arxiv.org/pdf/2504.03710 and https://github.com/ege-erdogan/weightflow

[2] DeepWeightFlow: Re-Basined Flow Matching for Generating Neural Network Weights, Saumya Gupta and Scott Biggs and Moritz Laber and Zohair Shafi and Robin Walters and Ayan Paul, https://openreview.net/forum?id=fOwsr1VTi8

[3] Git Re-Basin: Merging Models modulo Permutation Symmetries, Samuel K. Ainsworth and Jonathan Hayase and Siddhartha Srinivasa, https://arxiv.org/abs/2209.04836

[4] Neural Network Diffusion, Kai Wang and Dongwen Tang and Boya Zeng and Yida Yin and Zhaopan Xu and Yukun Zhou and Zelin Zang and Trevor Darrell and Zhuang Liu and Yang You, https://arxiv.org/abs/2402.13144

[5] Towards Scalable and Versatile Weight Space Learning, Konstantin Schürholt and Michael W. Mahoney and Damian Borth, https://arxiv.org/abs/2406.09997

[6] Generative Modeling of Weights: Generalization or Memorization? Boya Zeng, Yida Yin, Zhiqiu Xu, Zhuang Liu, https://openreview.net/forum?id=QbSeApdgfi

[7] Graph Neural Networks for Learning Equivariant Representations of Neural Networks, Miltiadis Kofinas and Boris Knyazev and Yan Zhang and Yunlu Chen and Gertjan J. Burghouts and Efstratios Gavves and Cees G. M. Snoek and David W. Zhang, https://arxiv.org/abs/2403.12143

[8] Graph Metanetworks for Processing Diverse Neural Architectures, Derek Lim and Haggai Maron and Marc T. Law and Jonathan Lorraine and James Lucas, https://arxiv.org/abs/2312.04501

Critical strengths

Strengths:

(1) The authors show that their GMN appears to be able to generalize to new unseen MLP layer widths [Fig. 6]. While this is so far only demonstrated on small MLP models, it remains an exciting sign.

(2) In view of Zeng et. Al. [6], the proposed JWS metric is valuable, and their method shows signs of significant progress towards desirable scores by this metric.

(3) The work is well written and formatted, and the method is well described (despite my concerns regarding its novelty, see Weaknesses).

(4) The domain-conditioned transfer experiments are well thought out and genuinely insightful.

Critical weaknesses

Weaknesses:

(1) The presented method, a Graph Meta Network, appears to be a limited extension of the method of Erdogan [1], which applies a message passing graph neural network to learn the flow matching objective on MLP graphs (using the standard setup of Kofinas et al. [7]). The authors cite Erdogan, but only very briefly mention their work in the ‘Post-Hoc Alignment’ section [line 148], where they acknowledge that Erdogan’s method does permutation-equivariant flow matching on MLPs, but fail to mention that this is done with a similar message passing graph neural net and evaluated on similarly sized MLPs on similar/overlapping tasks (for example, MLPs on MNIST classification, see Erdogan sec 7 table 7.2). The identifiable difference between the presented method and that of Erdogan is simply the choice of AdaLN conditioning. Erdogan does not appear to use AdaLN conditioning in favor of a Relational Transformer setup (Erdogan Sec 6.3 [1]). The authors’ message passing neural net (MPNN) layers, which appear to be simply MLPs (Eqs 7-10) are similar to Erdogan’s (see Erdogan 6.3 [1] and ‘nn/gnn.py’ in the implementation), as Erdogan uses MLPs for edge updates (‘nn/gnn.py’ lines 846-878). Both these authors and Erdogan correctly credit Lim and Kofinas throughout their work, including for this edge updating MLPs [7,8]. Given the similarity of their GMN and Erdogan’s implementation and experimental settings, a discussion and comparison of their methods in context is necessary. The Abstract’s claims that “We instead parameterize a flow-matching velocity field with a permutation-equivariant Graph Meta Network … that our method closely …”, and the Conclusion’s opening line, “We proposed a permutation-equivariant GMN velocity field for learning weight distributions directly from independently trained networks…”, appear to be false in light of this prior work: the motivation for the GNN flow approach and the GNN flow matching itself do not appear to be uniquely attributable or novel contributions of these authors.

The concept of applying Graph Networks (and even the term Graph Meta Network and its acronym, GMN) are correctly attributed to Lim and Kofinas in the authors’ Section 4.2 [7, 8]. However, the contribution of Erdogan is precisely the application of these methods to the task of weight space generation, which the authors fail to identify [1].

In short, the method these authors present as a novel and significant contribution is in fact of limited novelty. Clarifying the true substance and novelty of their method in the context of Erdogan, Lim, and Kofinas is crucial [1,7,8]. Without access to the authors’ implementation these questions cannot be investigated further.

(2) The method is only demonstrated on tiny neural networks, without extension to ResNets or Transformers. The authors rightly identify that the cost of running GNNs on large and/or dense graphs balloons, but a discussion of this complexity and its evolution over model type, width, or depth does not appear to be present or clearly cited. The authors’ reasoning that models should work at small scales before they work at large scales is only defensible if the model could eventually work at said larger scale: however, this underlying question does not appear to be explored.

(3) The authors claimed results against aligned DWF are confounded by opaque hyperparameter selection, as well as scaling along a memorization prone axis during dataset scaling experiments [2, 3, 6]. When comparing against an alignment-based method, which uses the Git Re-Basin method to iteratively align the model dataset into a common loss basin, the step number/budget is a critical hyperparameter, and it is not discussed [2, 3]. It seems reasonable to expect larger and larger datasets of independently initialized and trained models to explore more and more functionally equivalent loss basins, leading to the intuition that a fair comparison of their GMN with an alignment-based method should hold the alignment step count or otherwise measured alignment budget in proportion to dataset size. If this is not the case, or if there is a tradeoff here, then it should be identified and explored rather than left as a source of confounding.

(4) The authors do not explore if/how the competitive SANE_KDE method performs in the same dataset-scaling setting as Weakness 3, despite SANE_KDE showing the next most desirable JWS scores in Table 1. By the JWS and IoU metrics presented there, SANE_KDE appears to be the more closely matched baseline, but they do not compare their GMN method against it in the data-scaling setting.

Questions for authors

Questions:

(1) The comparison against DWF with alignment is sensitive to the number of Git Re-Basin alignment steps used – the computational cost each step is tied to model size, not dataset size [2, 3]. What was the value used in these experiments, and how was it determined? Experiments which illuminate the tradeoffs between aligned and unaligned methods would clarify the papers position. Similarly, Figure 5 shows baseline methods at 2M and 10M parameter generator scales: however, these are obviously prone to memorization considering the small models they are tasked with generating [6]. Perhaps alignment budget or some aggregate compute metric would be a more balanced comparison [2, 3]?

(2) Table 1 would indicate that the method most competitive with GMN in the large corpus setting is SANE with the Kernel Density Estimator method (SANE_KDE) [5]. However, Figure 6 only shows GMN (potentially mislabeled as GNN?) vs unaligned and aligned DWF, and a DiT flow matching model which appears to be simply the unaligned DWF case. How does SANE_KDE’s performance evolve with dataset size, if at all? In the same setting, what is the difference between unaligned DWF and the MLP/DiT methods?

(3) The GMN method shows signs of successful extension to unseen MLP widths – can it also extend to unseen depths? If not, why not? Further, SANE can generate for unseen architectures as well – how do SANE and GMN compare in this setting [5]?

(4) What are the population and sample sizes in Table 1? The dataset sizes are presented in lines 346-348, but the number of models generated by SANE, P.diff, DWF, and GMN is not clear. Table 2 does not to report this data either.

(5) Can the domain transfer results of Figure 7 be compared against DWF in the same setting? They are clear and intriguing. DWF appendix K does briefly discuss this case but does not appear to be a mature method [2].

My score could change if:

• The contribution is clarified and compared against the method of Erdogan [1]. An anonymized implementation would provide invaluable context in continuing this portion of the review.

• Findings are more cleanly extracted from the present confounding hyperparameter and scaling choices with regard to comparisons against alignment-based methods like DeepWeightFlow.

• Comparisons against the competitive SANE_KDE method are provided or clearly linked in the main body.

• A discussion of scaling dynamics is provided or clearly cited within the main body.

9 comments
AuthorsOpenReview ↗

Response 1/5

We thank the reviewer for the feedback. We enjoy reading that they found our method and experiments "well thought out and genuenly insightful" and "clear and easy to follow". Before addressing the concerns and questions raised by the reviewer, we first clarify several aspects of our scope, representation, and theoretical contributions that do not seem to be accurately represented in the reviewer’s summary:

  • “a solution to the practical problems posed to weight space generative models” (Summary)

    Our central focus is specifically to avoid permutation alignment when learning from independently trained networks.

  • “learns the Flow Matching objective on MLP graphs” (Summary)

    Our formulation and experiments cover both MLP and CNN graphs.

  • “where nodes are biases and edges are weights” (Summary)

    This is not our methodology, and it corresponds instead to that of a different work Erdogan (2025), where biases become bias features and nodes become node features, see Figure 1 of Erdogan (2025).

    Instead, we follow the methodology of Lim et al. (2024). Quoting them: "Bias parameters could be encoded as node features as done by Zhang et al. (2023) or as self-loop edges on each neuron. Instead, we include a bias node for each layer and encode the bias parameters as edges from that node to the corresponding neurons in the layer." This is different from what the reviewer reported.

    Furthermore, we did not only consider MLPs, but also CNNs. In CNNs, simply looking at nodes and biases is oversimplifying the networks. In particular, the convolutional kernel is encoded as a vector-valued edge feature, following Kofinas et al. (2024).

    Additionally, on MNIST MLPs, naively applying Lim et al. (2024) makes training painfully slow. To overcome this issue, we proposed a trick that folds the input layer of MLPs into features of the first hidden layer (Appendix C.2; Figures 8–9). To the best of our knowledge, this has not been done before. In this trick, nodes are can actually end up containing weights.

  • “Their GMN solves this problem by virtue of being a message passing graph neural network” (Summary)

    We provide a theoretical analysis of why alignment can be avoided, with proofs in Appendix A. This applies to any permutation-equivariant velocity parameterization, in particular, but not only to, a GMN (Secs. 4.1–4.2).

We now address the reviewer’s concerns and questions.

Response to Weakness 1 (Part 1/2).

Our novelty compared to Erdogan (2025). We appreciate the request for a clearer comparison with the master thesis of Erdogan (2025).

Although Erdogan (2025), which we referenced perhaps too much in passing, indeed proposed a similar construction, ours is different and novel. Note that the reviewer already recognized some distinctions between our work and Erdogan (2025), but more are present. We address concerns regarding our positioning with respect to this prior work in the general response above. We have revised our manuscript to better clarify our distinctions with prior work, especially in the Abstract, Introduction (lines 76-84), and Related Works (lines 162–167), Conclusion (lines 516-518) and clarify credit attribution to Erdogan’s (2025) prior work. We also included in Appendix G a more detailed methodological comparison with Erdogan (2025) than in the main manuscript, along with direct experimental evaluation. Beyond the many conceptual, technical, and methodological differences between our approach and prior work, the difference also clearly appears in terms of performance: our approach surpasses Erdogan (2025) empirically by significant gaps in terms of JWS (but also accuracy), see our new Fig 16, and our method does not run out of memory at moderately sized batch sizes of 256 (our default setting) on our compute, unlike Erdogan (2025).

We thank the reviewer for providing us with the opportunity to clarify our novelty stance.

We re-iterate our main contributions:

  1. We are the first to propose equivariant flow models, specifically the GMN, to avoid alignment in weight space generation.

  2. We back this up theoretically.

  3. We show that we can train on multiple different architectures simultaneously, generate unseen widths, condition on tasks and generate unseen conditions. We consider this a remarkable achievement in the field.

  4. We do a comprehensive memorization-aware evaluation, which is an issue recently observed in weight space generation. We propose a new metric to evaluate memorization.


AuthorsOpenReview ↗

Response 2/5

Response to Weakness 1 (Part 2/2).

We did not claim to invent the GNN in the this paper. And still, the architectures differ.

Architectural differences beyond AdaLN with Erdogan (2025). As observed by the reviewer, our use of AdaLN is one of the differences with the work of Erdogan (2025). However, our implementation still differs from it as described in the thesis and in the implementation provided by the reviewer. Specifically, there are differences with respect to: (1) Aggregation functions, (2) Seperate MLPs for directions, (3) Global feature vector, (4) Residual Connections and more.

Architecture clarification and implementation access. We amended the architecture section of the submission to better reflect that (lines 249–269). At the request of the reviewer and in an act of good faith, we also provide an anonymized implementation of our GMN in a supplementary zip file. This should allow the reviewer to inspect these differences, and compare in detail with Erdogan (2025). We commit to releasing our full code, not just for the GMN, upon acceptance. If specific additional implementation components would help clarify the comparison, we would be happy to consider providing them.

Direct comparison. We also performed a direct comparison with Erdogan’s code. Please see our global message and the new Appendix G.

Response to Weakness 2

Evaluation scale and scalability. We acknowledge the limited network scale, which is due to technical limitations. We restricted ourselves to small, not tiny, neural networks. Yet, even at this scale, the evaluated baselines already exhibit clear memorization or poor accuracy, despite these methods supporting larger networks. In fact, if they already memorize at small scales, they will most likely memorize at larger scales: unreliability at smaller scales puts into question effectiveness on larger networks. This echoes recent findings, particularly on memorization (Zeng et al., 2026; Wang et al., 2026; Han et al., 2026), and makes reliable distribution learning an important challenge even at these small scales. We have expanded the main-text discussion of scalability and suggest concrete scaling directions in the limitations and future works discussion in Section 6 [lines 523–533]. Large-scale generation remains challenging future work.

Can our method scale? A GMN perpsective. In general, GMNs can already process substantially larger networks and Transformers than those we studied. Most notably, NiNo uses graph neural networks to predict weights during optimization, including for a 111.5M-parameter Llama3-style Transformer (Knyazev et al., 2025). Furthermore, GMN graph constructions already support residual connections, normalization, and multihead attention (Lim et al., 2024). The machinery of GMNs is thus ready to scale, and we have used in practice a GMN to compute the flow field, giving us hope that we might be able to scale our method when using a GMN in the future.

Can our method scale? Beyond GMNs’ perspective. Our alignment-free formulation is not restricted to GMNs, thus other architectures could be considered to the flow matching model. We only require a velocity field equivariant to the network’s permutation symmetries. If we would choose to specialize a generator to a particular finite family of architectures, permutation-equivariant Transformers could provide a scalable alternative (Zhou et al., 2023). Indeed, there are especially optimized libraries for Transformers.

Can our method scale? An equivariant compression perspective. Rather than generating the raw weights, the amount of which scales poorly, we could generate compressed versions of them, e.g. latent representations of them, as is done in regular applications of generative models. However, it is important to preserve the equivariance in the compression in order to preserve the equivariance of our approach. We have already performed and implemented in the original submission some levels of equivariant compression by using our input-layer folding and vector-valued kernel edges (Appendix C.2). Beyond these, symmetry-aware compression and hierarchical processing could offer further scaling directions, e.g. Section 7.2 of Han et al. (2026) and Section 5 of Wang et al. (2026). A compact latent representation could further reduce costs, provided the encoder, latent flow, and decoder remain equivariant under compatible symmetry actions.

In summary, while our method currently does not scale beyond small settings, we believe there are reasonable existing encouraging directions to perhaps allow our framework to scale in the future.


AuthorsOpenReview ↗

Response 3/5

Response to Weakness 3

Hyperparameters and Alignment. We have provided in the revised manuscript the requested clarifications regarding the used hyperparameters. We now make baseline selection and alignment explicit in Section 5.1 and Appendix C.7, and expand the scaling protocol and capacity-sensitivity analysis in Section 5.1.1 and Appendix D.

Regarding “opaque hyperparameter selection”. Table 1 reports results selected from 110 baseline configurations, including 81 DWF configurations. Appendix C.7 documents the searches and their results. For each dataset and setting, we report the configuration with the highest mean JWS. For CIFAR-10, DWF’s capacity approximately matches and slightly exceeds ours: 80M versus 77M parameters. We address the concern about generator capacity and memorization in our response to Question 1.

Regarding “alignment budget in proportion to dataset size”. Our protocol in the original submission already provides this: each network receives its own allowance of 100 matching iterations, so the total available budget grows linearly with collection size. Across all DWF experiments, each network is matched to the collection’s first network using canonicalization.py from the authors’ official repository. Matching stops after a complete sweep in which no layer’s assignment score improves by more than 10−1210^{-12}. For scaling, we align all 50K networks once and reuse prefixes. Each network is matched independetly to the reference network.

We also checked stopping behavior across the full collections: matching terminated after 2–8 iterations on mfeat-karhunen, 3–10 on MNIST, and 2–15 on CIFAR-10. Every network reached the early-stopping criterion before the cap, ruling out premature termination.

Response to Weakness 4

SANE_KDE scaling comparison. We added a requested comparison with SANE_KDE, which is the method providing the best results among baselines in Table 1 (but with JWS scores well below ours revealing quite some memorization issues). In the added experiment, we used 2M and 10M SANE_KDE models across all ten collection sizes to Section 5.1.1 and Figure 5, with additional training schedules and per-sample analysis in Appendix D.2–D.3.

SANE_KDE scaling results and memorization diagnostics. Regarding SANE_KDE scaling, the results reveal two distinct issues. The 10M SANE_KDE combines high accuracy with excessive weight similarity (WCS), clearly revealing memorization. For the 2M model, separate averages obscure a strong accuracy-WCS correlation (Pearson r=0.935r = 0.935–0.9610.961): accurate outputs closely resemble training checkpoints, while less similar outputs perform poorly (Appendix D.3, Figure 13). Thus, again, revealing important memorization issues with SANE_KDE.

SANE correction. While carrying out these additional experiments, we actually found an evaluation error on our side, that we corrected (See Table 1). SANE_KDE used 30% randomly selected checkpoints instead of the top-performing 30% of the training collection, and SANE_Gauss omitted bootstrapping. Now SANE_KDE and SANE_Gauss report slightly higher values than originally reported, however, these updated results do not change the result landscape and conclusions drawn from them. Following our fix, we re-evaluated every results for these two specific baselines across the entire paper. In the revised version, all results for these methods use the corrected evaluation.


Reviewer_9utmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an alignment-free neural weight generation framework using a permutation-equivariant Graph Meta Network as the flow-matching velocity field. The authors provide theoretical justification for alignment-invariant training and introduce Joint Wasserstein Similarity (JWS) to evaluate generation quality. Experiments demonstrate improved distributional matching, heterogeneous-architecture generation, unseen-width transfer, and domain-conditioned synthesis.

Critical strengths

The paper provides a theoretically grounded approach to alignment-free weight generation, supported by consistent aligned/unaligned results. The proposed method achieves substantially higher JWS than evaluated baselines on MNIST and CIFAR-10. Extensive baseline searches, held-out-network validation, and refinement ablations strengthen the empirical evidence. Furthermore, a single conditional generator handles heterogeneous architectures across 20 tasks and transfers to unseen hidden widths.

Critical weaknesses

W1. Although the authors establish alignment-invariance theoretically, the experiments do not sufficiently demonstrate that permutation equivariance itself is the primary cause of the substantial generation-quality improvements.

W2. A matched GMN baseline without strict permutation equivariance, under identical model capacity, training, and sampling settings, is missing, making it difficult to isolate the contribution of the equivariant inductive bias.

W3. JWS matches only three summary statistics; even with the reported validation, high JWS does not establish full weight/function distribution matching or conclusively exclude memorization.

W4. Limited evaluation scale and baseline coverage. Unconditional generation is evaluated only on small MNIST and CIFAR-10 networks, without ImageNet-scale experiments or direct comparisons with graph-hypernetwork-based weight generation approaches discussed in the paper.

W5. Unclear novelty relative to prior equivariant flow matching. Erdogan (2025) already explores permutation-equivariant flow matching, yet no direct comparison is provided, leaving the methodological novelty beyond alignment-free independent-checkpoint generation insufficiently distinguished.

W6. Table 15 reveals substantial task-dependent failures: on letter, generated models underperform scratch-trained networks by 25.16/13.81/16.49 percentage points at 0.5×/1×/1.5× widths, respectively, questioning the robustness of width generalization.

Questions for authors

Refer to Weakness, please.

6 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for their comments. In particular, we are grateful that the reviewer recognized our theoretically grounded approach, empirical validation and extensive analysis. We address below the concerns raised by the reviewer.

W1–W2. Equivariance inductive bias for performance. We thank the reviewer for pointing out the need to better isolate the contribution of equivariance for GMNs. We carried out the requested ablation, comparing a baseline GMN without strict equivariance built-in to one with it (ours), and revised our submission accordingly by adding Appendix H (Table 23), explicitly linked from Section 5.1 (p. 8, line 406). Keeping the GMN backbone and training budget fixed, we compare encodings that break strict equivariance, at approximately matched capacity. The strictly equivariant GMN achieves the highest observed accuracy and JWS. Both distinct-encoding variants learn approximately equivariant fields but achieve lower accuracy and JWS. Removing positional and layer encodings also severely degrades generation. Together, these results support explicitly enforcing hidden-neuron permutation equivariance while retaining informative structural encodings for high-fidelity generation.

W3. JWS guarantees against memorization. JWS indeed incorporates three complementary statistics, providing a diagnostic regarding memeorization. In particular, low JWS may indicate that either the networks perform badly relative to the training collection, meaning the generator has failed to learn anything meaningful, or the generator has memorized and cannot generalize. However, as observed by the reviewer, high JWS does not guarantee full weight or functional distribution matching nor the complete absence of memorization. Thus, we have expanded the motivation for our evaluation in Section 4.3 [lines 289–298] and explicitly stated in Section 6 [lines 536–538], that high JWS neither establishes full-distribution matching nor conclusively excludes memorization.

JWS nevertheless provides a useful diagnostic. Table 1 already illustrates this with SANE_KDE: its 94.2% MNIST accuracy accompanies WCS of 0.941, far above the training reference of 0.557. Its JWS of 0.59 exposes an important mismatch with training statistics that raw accuracy alone conceals.

Additionally, we now provide SANE_KDE scaling experiments and per-sample analysis, revealing an even more subtle issue: for its 2M model, accuracy and WCS are strongly correlated (Pearson r=0.935r = 0.935–0.9610.961). Lower weight similarity accompanies poorer accuracy, a dependence obscured by separate averages and motivating joint evaluation. Together, these findings support JWS as a practical diagnostic of generation quality. These new results appear in Section 5.1.1 [lines 422–427] and Appendix D.3, Figure 13.

W4. Limited scale and hypernetworks. We acknowledge the limited network scale, but the evaluated baselines already exhibit memorization or poor accuracy in this setting, making reliable distribution learning an important challenge even at these scales. We have expanded the main-text discussion of scalability and suggest concrete scaling directions in the limitations and future works discussion in Section 6 [lines 523–539]. We also clarified the distinction from hypernetworks in Section 3 [lines 142–146]. Large-scale generation remains challenging future work.

Regarding hypernetworks, our learning objectives differ. Established hypernetworks produce weights through task objectives or posterior inference (Ha et al., 2016; Krueger et al., 2017), including architecture-conditioned graph hypernetworks (Zhang et al., 2019; Knyazev et al., 2023). We instead start with an existing collection of independently trained checkpoints and learn to sample their weight distribution without alignment. The checkpoints themselves are our training data, and we evaluate both performance and distributional agreement beyond checkpoint memorization. We therefore do not include these hypernetworks as baselines: they solve a different learning problem from modeling the weight distribution of an existing checkpoint collection.


AuthorsOpenReview ↗

Response 2/2

W5. Our novelty compared to Erdogan (2025). Although Erdogan (2025), which we referenced, indeed proposed a similar construction, ours is different and novel. Note that the reviewer already points out significant distinctions between our work and Erdogan (2025), as indeed our work is both "alignment-free" and with "independent-checkpoint" training collections. We address concerns regarding our positioning with respect to this prior work in the general response above. We have revised our manuscript to better clarify our distinctions with prior work, especially in the Abstract, Introduction (lines 76–101), and Related Works (lines 162–167), and clarify credit attribution to Erdogan’s (2025) prior work. We also included in Appendix G a more detailed methodological comparison with Erdogan (2025) than in the main manuscript, along with direct experimental evaluation. Beyond the many conceptual, technical, and methodological differences between our approach and prior work, the difference also clearly appears in terms of performance: our approach surpasses Erdogan (2025) empirically by significant gaps in terms of JWS (but also accuracy), and our method does not run out of memory at moderately sized batch sizes of 256 (our default setting) on our compute, unlike Erdogan (2025).

W6. Failure case. We thank the reviewer for pointing out an interesting reported failure case of our method. In an experiment where we train a GMN to cover simultaneously 20 datasets, there is only one dataset, Letter, for which the generated networks have substantially lower performance (-14% to -25%), even when generating on seen architectures. On all other 19 architectures, we provide successful generations (with worse accuracy drop of -2% on seen architectures for the mfeat-karhunen dataset, and mean accuracy drop accross all 20 datasets of -1% on seen architectures). Thus, failures can indeed occur, but they seem to be fairly rare. Letter is a challenging dataset with 26 classes to classify, compared to at most 10 (and mostly 2-3) classes for all other datasets. A natural question thus arose, given the increased complexity of the Letter task, did we provide sufficient capacity to successfully generate those complex networks? As such, we asked ourselves wether the failure on the Letter dataset is inherently due to some core aspect of our pipeline, or wether it is the result of ill-tuned hyperparameters.

To answer this question, we added a Letter-specific follow-up experiment in Appendix F.5, linked from Section 5.2.1, where we explicitly state this specific failure. The Letter-only GMN improves accuracy by 3.42/4.69/5.01 percentage points at 0.5/1/1.5× widths. The new results show that the deficit can be reduced at both native and unseen widths, although the remaining gaps, particularly at 0.5×, remain a limitation. We thus believe that the problem arising from the Letter dataset is that its challenging 26-class setting places capacity greater demands on the generator, rather than a core limitation of our pipeline.


We thank the reviewer for the constructive comments, which helped to improve our paper, and hope to discuss any ongoing suggestions further.

References

Ha et al. (2016). David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016.

Krueger et al. (2017). David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint arXiv:1710.04759, 2017.

Zhang et al. (2019). Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rkgW0oA9FX.

Knyazev et al. (2023). Boris Knyazev, Doha Hwang, and Simon Lacoste-Julien. Can we scale transformers to predict parameters of diverse ImageNet models? In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 17243–17259. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/knyazev23a.html.

Erdogan (2025). Ege Erdogan. Geometric flow models over neural network weights. arXiv preprint arXiv:2504.03710, 2025.

Thank you for your response. I appreciate the authors' clarifications, but I believe additional empirical evidence and a more substantive comparison with recent graph-hypernetwork-based approaches are still needed to adequately address W4.

First, my original concern specifically referred to graph-hypernetwork-based weight generation methods, rather than hypernetworks in general. Recent works have demonstrated substantial progress in generating parameters for larger networks and more challenging datasets. I would appreciate a meaningful comparison under a shared evaluation setting. Representative examples include:

  • LoGAH (Zhou et al., 2024): Predicting 774-Million-Parameter Transformers using Graph HyperNetworks with 1/100 Parameters.
  • TAL (Xu et al., ICML 2025): Learngene Tells You How to Customize: Task-Aware Parameter Initialization at Flexible Scales.
  • PITH (Xu et al., CVPR 2026): Unlocking Pre-trained Weights: Parameter Inheritance for Zero-Shot Initialization.

Second, these studies have demonstrated the feasibility of parameter prediction for substantially larger networks, including evaluations on CIFAR-100 and ImageNet. Given these advances, evaluating the proposed approach only on small networks trained on MNIST and CIFAR-10 provides limited evidence of its scalability.

I would therefore encourage the authors to provide at least a preliminary experiment on a more challenging setting, such as ImageNet-1K or a larger network architecture, together with an analysis of computational and memory requirements as model size increases.

Reviewer_Pdd7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a permutation-equivariant flow-matching model for weight generation using graph meta-network (GMN) architecture. The paper puts substantial emphasis on achieving generalizable weight generation instead of weight-space memorization. In particular, the paper introduces several metrics to distinguish memorization from generalization. In particular, Joint Wasserstein Similarity (JWS) measures the similarity between the joint distributions of three statistics (task performance, max error IoU, and WCS). It captures functional similarity, such as error-pattern similarity, that cannot be assessed by accuracy alone. Experimental results confirm that the proposed equivariant GMN outperforms baselines both in terms of accuracy and JWS.

Critical strengths

The proposed metric, JWS, is novel. It attempts to tackle a challenging problem of distinguishing checkpoint memorization and generalization. As memorizing the training checkpoint would also yield high accuracy, distinguishing memorization and generalization is difficult.

Critical weaknesses

[Concern 1] Using permutation-equivariant flow matching for neural weight generation is already explored by Erdogan, 2025, "Geometric Flow Models over Neural Network Weights," which limits the methodological novelty. However, combining GMN with equivariant flow-matching to avoid permutation alignment is novel.

[Concern 2] The paper lacks a theoretical justification for why permutation equivariant FM mitigaets checkpoint memorization. The authors prove permutation equivariance and alignment invariance, but their connection with memorization is unclear. The evidence against memorization is purely empirical.

[Concern 3] The choice of statistics for JWS appears somewhat arbitrary. Furthermore, the paper lacks evidence that higher JWS = better generalization.

Questions for authors

The operational necessity of JWS is unclear to me. From a practical perspective, achieving high accuracy in unseen settings (e.g., unseen width or task) should be the ultimate goal. JWS attempts to capture distributional similarity that accuracy alone cannot reveal. This also implies that what JWS captures can be largely irrelevant to the accuracy. But then why do we have to care about this?

5 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for their feedback. We enjoy reading that they recognizing the novelty of JWS and alignment-free equivariant flow matching. We address all 3 concerns raised by the reviewer and respond to their question.

Concern 1

Our novelty compared to Erdogan (2025). Although Erdogan (2025), which we referenced, indeed proposed a similar construction, ours is different and novel. Note that the reviewer already recognized a significant distinction between our work and Erdogan (2025), as indeed "combining GMN with equivariant flow-matching to avoid permutation alignment is novel". We address concerns regarding our positioning with respect to this prior work in the general response above. We have revised our manuscript to better clarify our distinctions with prior work, especially in the Abstract, Introduction (lines 76-84), and Related Works (lines 162–167), and clarify credit attribution to Erdogan’s (2025) prior work. We also included in Appendix G a more detailed methodological comparison with Erdogan (2025) than in the main manuscript, along with direct experimental evaluation. Beyond the many conceptual, technical, and methodological differences between our approach and prior work, the difference also clearly appears in terms of performance: our approach surpasses Erdogan (2025) empirically by significant gaps in terms of JWS (but also accuracy), and our method does not run out of memory at moderately sized batch sizes of 256 (our default setting) on our compute, unlike Erdogan (2025).

Concern 2

Equivariance against memorization. We agree that our evidence for equivariance to help against memorization is empirical rather than theoretical. As acknowledge by the reviewer our "evidence against memorization is purely empirical", whereas our equivariance theory (Sections 4.1-4.2) tackles instead the alignment invariance aspect. We included an acknowledgment of this lack of theoretical guarantee of equivariance avoiding memorization in the limitations discussion (Section 6).

Concern 3

Why JWS and link to generalization. We clarified the motivation for constructing our JWS score based on the chosen statistics in Section 4.3 [lines 289–298]. Essentially, each of our chosen statistic addresses independently one of three complementary questions:

  1. Performance: Do generated networks perform the task well? This can be measured in accuracy.

  2. Functional similarity: Are they functionally similar to training networks? This can be measured by seeing if the generated networks do the make the same mistakes as the training ones (error-IoU).

  3. Weight similarity: Do they reproduce weights already present in the training collection? This can be measured by permutation-matched weight cosine similarity (WCS).

The first statistic measures a prerequisite for any successful generator: if generated networks have low task performance, then the generator is bad. The other two scores measure memorization from complementary perspectives, namely functional (black box input-output behavior) and in weight space (actual weights). By combining all three statistics into a single number, and normalizing it with respect to the behaviour of the training collection, our novel proposed score JWS better quantitatively evaluates the quality of the weight generator.

Higher JWS points towards higher generalization. Or rather, low JWS reveals poor generalization capability. Indeed, the JWS is low if either statistics are poor. If the accuracy is low, the generator is bad, regardless if the networks seem different from the training collection. If the error-IoU or the WCS are high, then the generated networks are much too similar, either functionally or directly in weight space, to the networks in the training collection. This means that the generator is strongly memorizing, and thus that it does not generalize well. Thus, JWS is by design a diagnostic evaluator of the generalization capability. We clarified this property of the JWS (Section 4.3, lines 349–351).

While lower JWS reveals poor generalization, it might be possible to still have some memorization with higher JWS scores, should our statistics not cover every possible memorization aspect. We clarified this issue in the limitations (Section 6), specifically that high JWS does not certify full-distribution matching or conclusively exclude memorization.

Note that the JWS score only applies in the same-task, same-architecture setting (Table 1). It does not apply when generating networks on unseen architectures, widths, and conditions (Section 5.2) as by definition, there is no training collection corresponding to these unseen cases. As such, JWS cannot be used to measure the generalization capability to these unseen setting.


AuthorsOpenReview ↗

Response 2/2

Question 1: Why care about JWS?

JWS vs. Accuracy in the unseen setting. As stated above, JWS does not apply in the unseen setting (architecture or task) as there is no training collection to compare with. In this setting, downstream performance, measured in accuracy, is indeed the goal we care about and reveals directly the strong generalization capability of the model. This explains why in our evaluations of the unseen settings, we only focused on accuracy (Figure 6) and AUC (Figure 7).

Nevertheless, solely focusing on accuracy can be misleading. In the weight generation community, unseen settings are not usually possible and thus seen settings are the norm. In seen settings, which are different from unseen settings, solely reporting high accuracy is not a satisfactory methodology, even though it still remains the most common benchmark (Gupta et al., 2026; Schürholt et al., 2024; Wang et al., 2024). Indeed, in seen settings, the generator can produce high performing networks by simplying copying networks from the training collection. This means the generator is memorizing rather than functioning as a good generator capable of generating new networks different from the training ones. That is why we proposed the JWS score for the seen settings to better evaluate the quality of the generator in this case. We expanded Section 4.3 [lines 289–321] to clarify why joint evaluation matters. The JWS is irrelevant for the unseen case and we do not claim its relevance there.


We thank the reviewer for the feedback, improving our paper, and hope to discuss any ongoing suggestions further.

References

Gupta et al. (2026). Saumya Gupta, Scott Biggs, Moritz Laber, Zohair Shafi, Robin Walters, and Ayan Paul. Deep-weightflow: Re-basined flow matching for generating neural network weights. arXiv preprint arXiv:2601.05052, 2026.

Schürholt et al. (2024). Konstantin Schürholt, Michael W Mahoney, and Damian Borth. Towards scalable and versatile weight space learning. In Proceedings of the 41st International Conference on Machine Learning, pages 43947–43966, 2024.

Wang et al. (2024). Kai Wang, Dongwen Tang, Boya Zeng, Yida Yin, Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural network diffusion. arXiv preprint arXiv:2402.13144, 2024.

Erdogan (2025). Ege Erdogan. Geometric flow models over neural network weights. arXiv preprint arXiv:2504.03710, 2025.

Thank you for the detailed response and revisions. The contribution relative to prior work is now clearer. However, my concerns are only partially resolved. The connection between permutation equivariance and avoiding memorization remains largely empirical, which the authors also acknowledge. In addition, while low JWS can indicate failure, higher JWS does not necessarily imply better generalization. I will therefore maintain my score.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response: Comparison to Prior Work Part 1/3

We thank the reviewers for their insightful feedback and comments. We thank the reviewers for recognizing the novelty of JWS and avoiding alignment through equivariant FM (Pdd7), our “theoretically grounded approach” (9utm), and our “well thought out and genuinely insightful” domain-conditioned experiments (dNR9). Below, we clarify our contributions relative to Erdogan (2025). However, all three share the same important concern regarding how our work relates to Erdogan (2025). We have thus decided to address this issue in the main response rather than individual responses to each reviewer. We addressed every single other concern, providing many additional experiments when requested, and we have revised our submission with changes marked in blue. We now focus on the novelty concern regarding Erdogan (2025)

While we did reference and credit Erdogan (2025) in our original submission, perhaps too much in passing, we have discussied it in more detail in the revised version to clarifiy our contribution with respect to existing work. In our revised manuscript, we have added a dedicated discussion in the main text, along with a detailed comparison discussion in the appendix. We have included empirical comparisons showing a clear difference in performance of our method over it. In light of the new comparison, we maintain that our method is novel.

In the rest of this discussion, [E] refers to the relevant master thesis Erdogan (2025, arXiv v1), whereas [Ours] refers to our submission.

In short. Our most important contribution is showing that an equivariant velocity field makes alignment unnecessary for weight generation. This lifts what we see as the central limitation of prior work, which motivates our paper: the restriction to aligned collections, either aligned after training ([E], DWF (Gupta et al., 2026), SANE (Schürholt et al., 2024)) or built from dependent networks ([E], P-diff (Wang et al., 2024), D2NWG (Soro et al., 2025), RPG (Wang et al., 2025)). The generator can therefore learn from independently trained networks without requiring alignment or a shared training history.

Our main contributions are the following:

  • (1) Alignment-free generation, proved and shown on large training collections.
  • (2) One generator across architectures, which is not tied to a single type of architecture as we have learned to generate both MLPs and CNNs. We can also generate networks of unseen architecture, specifically of different widths from networks in the training collections.
  • (3) Conditional generation across tasks and domains.
  • (4) Memorization-aware evaluation, including a novel metric to diagnose memorization.

We had explicitly, albeit briefly, credited Erdogan’s earlier exploration of GMN flow models for MLP weight generation. However, our GMN architectures, training objective, sampler, and capabilities differ. Most importantly, we did not and do not claim to be the first to use GNNs for weight generation. Importantly, the thesis [E] did not identify that equivariant modeling already makes alignment unnecessary: in addition to working on non-independently trained checkpoints (a priori alignment in our terminology) it explicitly aligns every network (post hoc alignment in our terminology) and treats the GMN as a complement to approximate alignment [E, Sec. 4.4].

We also included in the revised manuscript a direct empirical comparison. It clearly shows that our method substantially outperforms that of [E], revealing that our differences are of importance in practice.

To make our positioning crystal clear with respect to [E], we included in the revised manuscript a point-by-point comparison, which we provide below.


AuthorsOpenReview ↗

General Response: Comparison to Prior Work: Part 2/3

Point-by-point comparison.

1. Alignment-free generation. In his thesis, Erdogan trains on collections of dependent checkpoints: many checkpoints share the same optimization history within each of a small pool independent training runs [E, Sec. 6.1; App. A]. He further prescribes aligning these checkpoints to a common reference [E, Secs. 1.2, 6.2.1]. Unaligned training appears in a small ablation [E, Table 7.1], which retains the same dependent-checkpoint construction. He motivates the GMN as a complement to alignment, to handle permutation symmetries left unresolved by approximate matching [E, Sec. 4.4].

In contrast, we instead establish the equivariant generator as a means of avoiding alignment altogether. We prove that the marginal FM velocity is permutation-equivariant and that, for an equivariant model, alignment leaves the expected FM objective unchanged [Ours, Secs. 4.1–4.2; App. A.2–A.4]. We therefore train, without alignment, on independent final checkpoints: 10K MNIST MLPs, 50K CIFAR-10 CNNs, 100K tabular MLPs [Ours, Sec. 5; App. C.1]. Aligning MNIST changes nothing (JWS 0.91 vs. 0.90; [Ours, Table 2]). Reviewers call this “novel” (Pdd7) and “theoretically grounded” (9utm).

2. One generator across architectures. Erdogan trains one flow per architecture and dataset and leaves a single flow across them as future work [E, Sec. 8.1.2]. In contrast, our model reads the architecture as its input graph [Ours, Eq. 11]: trained on 100K networks from 16 architectures and 20 tasks, it generates networks for all of them (82.2% mean accuracy; collection: 83.2%) and generalizes to 0.5× and 1.5× hidden widths without alignment, retraining, fine-tuning, target-width examples, or new generator parameters [Ours, Sec. 5.2.1]. We therefore establish a reusable generator across architectures, which dNR9 credits as an “exciting sign“.

3. Conditional generation. Erdogan provides no classifier-free conditioning, and it is left as future work [E, Sec. 8.1.3]. In contrast, our model takes the task or domain as a condition, with classifier-free guidance [Ours, Eq. 11]. With domain conditioning, unseen values yield single networks comparable to two-network logit ensembles on both domains [Ours, Sec. 5.2.2], which dNR9 credits as “genuinely insightful” and ”clear and intriguing”.

4. Memorization-aware evaluation. Erdogan measures diversity among samples [E, Sec. 7.3.1] but does not compare them with training checkpoints. This is an issue as weight generators have been found to actually memorize rather than properly generate new networks, thus comparisons with the training checkpoints are necessary to know if the method actually works.

In Erdogan’s defence, the memorization problem in the field was observed after Erdogan wrote his thesis. Following Zeng et al. (2026), we carefully study memorization and introduce another statistic, weight cosine similarity (WCS), and a new score JWS combining three important statistics to be compared with that on the training collection [Ours, Sec. 4.3]. We show that focusing solely on accuracy can hide memorization: SANE-KDE reaches 94.2% on MNIST (ours: 94.4%) while nearly replicating training networks (WCS 0.94, way above the reference 0.56 within the training collection; [Ours, Table 1]).

5. Loss, sampler, and theoretical justification. Erdogan minimizes endpoint-prediction error and subtracts the initial noise to reconstruct the velocity, claiming equivalence to flow matching [E, Secs. 3.2.2, 6.4; App. A]:

LE=E∥fθ(wt,t)−w1∥22,w˙t=fθ(wt,t)−w0.\mathcal{L}_{\mathrm{E}} = \mathbb{E}\lVert f_\theta(w_t,t)-w_1\rVert_2^2, \qquad \dot{w}_t=f_\theta(w_t,t)-w_0.

But this reconstruction does not generally recover the FM velocity field E[W1−W0∣Wt=w]\mathbb{E}[W_1-W_0\mid W_t=w]: even with the optimal endpoint predictor, it instead gives E[W1∣Wt=w]−w0\mathbb{E}[W_1\mid W_t=w]-w_0. We instead use standard flow matching, directly predicting and integrating the velocity [Ours, Eq. 11]:

LFM=E∥vθ(wt,t)−(w1−w0)∥22,w˙t=vθ(wt,t).\mathcal{L}_{\mathrm{FM}} = \mathbb{E}\lVert v_\theta(w_t,t)-(w_1-w_0)\rVert_2^2, \qquad \dot{w}_t=v_\theta(w_t,t).

Its optimal predictor is the marginal FM velocity. We build our equivariance and alignment-invariance theory on this formulation [Ours, Secs. 4.1–4.2; App. A].

AuthorsOpenReview ↗

General Response: Comparison to Prior Work 3/3

6. Direct comparison (new). We ran Erdogan’s official implementation. On 50K independently trained mfeat-karhunen MLPs (64 → 16 → 8 → 10; 1,266 parameters; our Appendix D). We matched both generators to approximately 2M parameters and tested batch sizes {8, 64}, integration steps {5, 10, 20}, and aligned/unaligned collections, using his Re-Basin implementation for alignment, providing 75K training steps for each configuration.

Our method achieves significantly higher accuracy and JWS in all 12 completed matched settings. At both matched batch sizes, our unaligned model with only 5 integration steps already outperforms Erdogan’s 20-step model in both its aligned and unaligned settings, on both accuracy and JWS. The table shows our results at all three step counts against his prescribed 20-step setting. We provide these results in the revised manuscript both as a table (Table 20) and a figure (Figure 16), which visually shows even better the large gap between our method and that of [E].

BatchAlignmentOurs: 5 steps — Accuracy (%) / JWSOurs: 10 steps — Accuracy (%) / JWSOurs: 20 steps — Accuracy (%) / JWSErdogan: 20 steps — Accuracy (%) / JWSOurs — Peak training memory (GiB)Erdogan — Peak training memory (GiB)
8Unaligned79.44 / 0.870181.49 / 0.883582.13 / 0.889972.06 / 0.83910.954.40
Aligned79.00 / 0.874381.13 / 0.885981.80 / 0.886774.29 / 0.8372
64Unaligned83.27 / 0.904585.43 / 0.928985.95 / 0.937881.76 / 0.89717.2430.96
Aligned83.52 / 0.907585.42 / 0.930186.05 / 0.935581.40 / 0.8983
256Unaligned83.95 / 0.910985.95 / 0.936186.64 / 0.9434OOM28.63OOM
Aligned84.02 / 0.923985.84 / 0.943086.52 / 0.9495OOM

We ran this on an NVIDIA L40 (48 GiB). At batch size 256, our GMN completes a full update, using 28.63 GiB; Erdogan’s implementation runs out of memory during training. Trying to run Erdogan’s code on MNIST also ran out of GPU memory: these MLPs have architecture 784 → 32 → 32 → 10 and 26,506 parameters (our Appendix C.1).

Erdogan’s method does not support CNN generation or joint heterogeneous-architecture experiments, so we did not compare with it in these settings.

We also tried Erdogan’s other flow variants (Normalized and Geometric), which produced even poorer results. This makes us conclude that this method was never designed to work on independently, unaligned networks, unlike ours. Additional analyses are found in the new Appendix G.

Revisions. The abstract (lines 18–20), introduction (lines 76–84), related work (lines 162–167), and conclusion (lines 516–518) now credit more and further discuss this prior work [E] and state clearly our contributions as above. A new appendix G (lines 2182–2336) compares the methods side by side, including the sampler; Appendix G (lines 2222–2282) reports the direct comparison.


We thank all reviewers for their detailed comments, which truly helped to improve our paper. We are looking forward to discuss any ongoing suggestions further.

References

[E] Erdogan (2025). Ege Erdogan. Geometric flow models over neural network weights. arXiv preprint arXiv:2504.03710, 2025.

Gupta et al. (2026). Saumya Gupta, Scott Biggs, Moritz Laber, Zohair Shafi, Robin Walters, and Ayan Paul. Deep-weightflow: Re-basined flow matching for generating neural network weights. arXiv preprint arXiv:2601.05052, 2026.

Schürholt et al. (2024). Konstantin Schürholt, Michael W Mahoney, and Damian Borth. Towards scalable and versatile weight space learning. In Proceedings of the 41st International Conference on Machine Learning, pages 43947–43966, 2024.

Wang et al. (2024). Kai Wang, Dongwen Tang, Boya Zeng, Yida Yin, Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural network diffusion. arXiv preprint arXiv:2402.13144, 2024.

Soro et al. (2025). Bedionita Soro, Bruno Andreis, Hayeon Lee, Wonyong Jeong, Song Chong, Frank Hutter, and Sung Ju Hwang. Diffusion-based neural network weights generation. In International Conference on Learning Representations, volume 2025, pages 99644–99675, 2025.

Wang et al. (2025). Kai Wang, Dongwen Tang, Wangbo Zhao, Konstantin Schürholt, Zhangyang Wang, and Yang You. Scaling up parameter generation: A recurrent diffusion approach. Advances in Neural Information Processing Systems, 38:93014–93046, 2025.

Zeng et al. (2026). Boya Zeng, Yida Yin, Zhiqiu Xu, and Zhuang Liu. Generative modeling of weights: Generalization or memorization? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 41974–41984, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.