acceptodds
Under review as a conference paper at ICLR 2027

Transferable Graph Metanetworks

Abstract

A *weight space network* (or *metanetwork*) takes the weights of another neural network as input and predicts properties of it. Most prior work trains such models on input networks of one or a few fixed sizes and evaluates them in-distribution. The few attempts at out-of-distribution size generalization remain limited in scope and have achieved only modest success. Consequently, the potential efficiency gains of training on small networks and evaluating on much larger ones remain largely unrealized. We propose *Transferable Graph Metanetworks*, which extend the graph metanetwork paradigm with a set of modifications that make performance transferable across input networks of different *widths*. The modifications follow two principles: invariance to the ways in which networks of different widths represent the same function, and continuity, such that weights representing similar functions receive similar predictions. We further study whether size generalization is possible for input networks trained independently from random initialization. Empirically, our modifications significantly improve size generalization on every task we consider. Performance is strongest on input networks trained under the maximal-update parameterization (P), where it remains robust up to the training width. Theoretically, we explain these observations with infinite-width limit theory: we prove size-generalization guarantees for our model on P-trained inputs, and explain why it can fail under other parameterizations.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
weight space, metanetwork, graph neural networks, transferability, size generalization, any-dimensional learning, muP, infinite-width limit, tensor program

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.20 over 5 reviews, above 96% of the others.
Reviewer_nrQYOpenReview ↗
Rating
4
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes size generalization for Graph Metanetworks (GMNs): the GMN is trained on small input networks and evaluated on much wider ones, with theoretical guarantees. The approach consists of two levels of modification. The first makes any GMN invariant to duplication widening; the second drops the learned edge features and lets the weights act on node features only through matrix multiplication, which makes the GMN invariant to the broader class of block-wise function-preserving widenings and Lipschitz continuous in a normalized spectral norm. The method is evaluated on MNIST/FMNIST (INR classification) and CIFAR-10/SVHN (accuracy prediction), showing that both modifications improve size generalization and that µP-trained input networks are much easier to generalize over than SP-trained ones.

Critical strengths
  1. The problem is well presented, and the introduction and related work make clear that existing approaches generalize across sizes only to a limited extent.
  2. The design is clean, well motivated, and architecture-agnostic.
  3. The invariance and continuity properties of the proposed modifications are all supported by theoretical proofs (Propositions 1–3).
Critical weaknesses
  1. Although the paper is well written, the main text lacks a summary of the empirical results. A compact table is needed to understand the gains clearly without having to dig through the appendix.
  2. Theorem 4 covers 1-layer input MLPs only, and the obstruction at depth ≥ 3 is proven at initialization; Theorem 5 requires the target to be exactly representable by a matrix-product GMN and gives no rate. The experiments use 2-hidden-layer INRs and CNNs, for which neither theorem applies
Questions for authors
  1. Could you add a main-text table reporting, for each dataset, parameterization, and model, the score at the training width, the score at the largest test width, and the resulting size-generalization gap, with error bars over GMN training seeds?
  2. Theorem 5 requires the target to be exactly representable by a matrix-product GMN, and the experiments use 2-hidden-layer INRs and CNNs that neither theorem covers. Can you argue that the INR-classification and accuracy-prediction targets fall in that class, and is there any empirical evidence on whether the depth-≥3 obstruction persists after training?
  3. Can you report training-width performance for all three models in a table, to support the claim at line 371 that the matrix-product restriction costs no performance?
2 comments
AuthorsOpenReview ↗

Response to reviewer nrQY

We thank the reviewer for the constructive suggestions.

W1, Q1, Q3: experiment results table

We appreciate the suggestion and have now added the requested table to Sect. 4 (Fig. 2(b)), reporting the numerical values as mean ±\pm std over three GMN training seeds (Fig. 2 now also shows a min–max band). As the "train" columns show, the matrix-product GMN's in-distribution test score is close to those of the other models, so in our experiments the restriction causes little performance loss. We also added a proof of universality of the matrix-product GMN on fixed width in App. B.5, theoretically showing that the restriction loses no expressivity at a fixed width.

W2, Q2: theory-experiment gap

We agree that, strictly speaking, none of the experimental settings in Sect. 4 is exactly covered by Theorems 4 and 5. Nonetheless, the assumptions of Theorem 5 are fairly general. We discuss the gaps one by one, and we believe most are small and technical rather than conceptual.

Target (Q2). Exact representability by a matrix-product GMN is sufficient but not necessary for Theorem 5. We have edited the paper to make clear that the target assumption is computability by a tensor program. For accuracy prediction, the target is a function of finitely many evaluations of the input network on a fixed, width-independent test set. This is exactly the example in our remark on the target assumption (App. E.3): any Lipschitz function of such evaluations, e.g., the test loss or a smoothed accuracy, is computable by a tensor program. Hence the accuracy prediction task is covered by our target assumption. The only discrepancy is that the exact 00--11 accuracy is discontinuous, but any smooth surrogate of accuracy satisfies the target assumption. For INR classification, whether the target can be uniformly expressed by matrix-product GMNs is indeed an open question. As we mention in the paper, this concerns the expressivity of GMNs uniformly across widths, an important future direction. Moreover, for any target that is not uniformly approximable by matrix-product GMNs, one should not expect robust size generalization anyway. In this sense, the guarantee given by Theorem 5 is fairly general.

Architecture. Theorem 5 is stated for MLP inputs. We expect the proof to extend directly to CNNs (and possibly transformers), but we have not checked the details. Intuitively, the training of CNNs and transformers is expressible as a tensor program ([1], App. A). Hence the evaluation of the GMN on CNN and transformer weights can also be appended to the tensor program, as in App. E.3.

Depth. Theorem 4 indeed covers only one-hidden-layer inputs, but its main purpose is not generality; it is to provide geometric intuition. The restriction comes from its proof route. Theorem 5 is the more general result and applies to MLP inputs of general depth.

We have revised Sect. 5 to clarify the target assumption and to note that the assumptions are fairly general.

Q2: Does the depth-≥3\ge 3 obstruction persist after training?

Yes, we checked this numerically, though these checks are not in the paper. The direct check uses trained 2-hidden-layer MNIST INRs (500 per width), for which we compute the mean normalized spectral norm of each layer, under both SP and μ\muP.

Param.Layerw=24w=24w=64w=64w=128w=128w=256w=256∣Δ∣\lvert\Delta\rvert (24→25624\to256)
SPfirst2.472.471.921.921.691.691.561.560.910.91
SPhidden-to-hidden13.6213.6214.7714.7714.7814.7814.2714.270.650.65
SPlast27.1727.1738.9138.9147.6547.6556.3556.3529.1829.18
μ\muPfirst2.472.472.502.502.532.532.542.540.070.07
μ\muPhidden-to-hidden13.6913.6913.1513.1512.6112.6111.8911.891.801.80
μ\muPlast27.2027.2026.4626.4626.6326.6326.8326.830.370.37

Under SP, we prove that ∥θn∥Vn,op=Θ(n)\lVert\theta_n\rVert_{V_n,\mathrm{op}}=\Theta(\sqrt n) at initialization for every depth, due to the last layer. This growth persists after training: the last-layer statistic increases monotonically from 27.1727.17 to 56.3556.35.

Under μ\muP, we prove that lim inf⁡W1(μn,μ2n;d∞op)≥c>0\liminf W_1(\mu_n,\mu_{2n};d_\infty^{\mathrm{op}})\ge c>0 at initialization, due to the hidden-to-hidden layer. This obstruction also persists after training: the first and last layers are nearly constant, but the hidden-to-hidden layer drifts, decreasing monotonically from 13.6913.69 to 11.8911.89.

References

[1] Greg Yang. Wide feedforward or recurrent neural networks of any architecture are Gaussian processes. NeurIPS 2019.

Thank you very much for your detailed responses which addressed my concerns and also helped me in better understanding. I will maintain my positive score and raise my confidence.

Reviewer_bnsFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies width generalization for metanetworks. A graph metanetwork (GMN) is trained on the weights of narrow input networks and evaluated on the weights of much wider ones. The design follows the any-dimensional learning framework of Levin et al. (2026). In that framework, a model should be compatible (equal outputs on equivalent inputs of different sizes) and continuous with respect to a norm that compares inputs across sizes. The paper proposes two levels of modification to a standard message-passing GMN (Eqs. 2–3).

  1. Duplication-compatible GMN: This level has three changes: fan-in rescaled edge features Wˉ(ℓ)=nℓ−1W(ℓ)W̄^{(ℓ)} = n_{ℓ−1}W^{(ℓ)}, mean instead of sum aggregation, and a layer-wise mean readout. Proposition 1 states that the result is exactly invariant to the duplication widening (Eq. 4). That widening copies hidden neurons and rescales their weights, so the represented function is unchanged.
  2. Matrix-product GMN: The paper defines two norms on weight space (Eq. 5): a normalized spectral norm and a normalized ℓ1ℓ_1 norm. Proposition 2 shows a "misalignment" between them. The realization map (weights → function) is Lipschitz in the spectral norm but can be discontinuous in the ℓ1ℓ_1 norm. The duplication-compatible GMN is the reverse: it is Lipschitz in ℓ1ℓ_1, and for input MLPs with more than one layer it can be discontinuous in the spectral norm. The fix is to drop edge features and entrywise edge processing, so that messages become WˉijW̄_{ij} A hjh_{j} (Eq. 7) and the weights enter only through products WHAW H A. Proposition 3 states that this GMN is Lipschitz in the spectral norm on bounded balls. The forward variant is also invariant to all block-wise widenings (Eq. 6). The bidirectional variant is invariant only to those block-wise widenings whose blocks have equal column sums.

Experiments

  • Tasks: INR classification (MNIST, FMNIST) and CNN generalization prediction (CIFAR-10, SVHN).
  • Input networks: re-trained at many widths under standard parameterization (SP) and maximal-update parameterization (µµP), calibrated to coincide at the training width.
  • Protocol: the GMN is trained and tuned at the smallest width and tested at larger widths.
  • Baseline: the bidirectional GMN implementation of Kalogeropoulos et al. (2024).
Critical strengths

Level 1 modification consists of three one-line changes (footnote 3 gives the widening itself as a PyTorch one-liner). Proposition 1 makes it exactly invariant, and the argument that it loses no expressivity at a fixed width is correct for the complete layered computation graph. Proposition 2 gives a principled reason why compatibility alone is insufficient, and Eq. (7) is a minimal fix.

Critical weaknesses

Fig. 2 is the only quantitative evidence in the main text. It shows one curve per model, with no error bars, and Sec. 4 does not state how many runs underlie each point. The large INR effects (e.g., ≈98% vs. ≈19% at MNIST width 1024) are unlikely to be seed artifacts. However, several smaller differences that the paper interprets are of the size that run-to-run variance could produce:

  • At the FMNIST training width (32), the baseline GMN reaches ≈77% on µP inputs and ≈82% on SP inputs. Yet Sec. 4 (Obs. 2, l. 427) says that SP and µP "share the same training distribution" at that width.
  • On SVHN at width 512: matrix-product on SP inputs (≈0.51) vs. duplication-compatible on µP inputs (≈0.55).
  • On MNIST up to width 256: matrix-product on SP inputs vs. duplication-compatible on µP inputs.
Questions for authors
  1. How many GMN seeds underlie Fig. 2? Can you add mean ± std, and explain the ≈5-point gap between the GMN on SP and on µP inputs at the FMNIST training width?
  2. Can you report a probing baseline, a weight-statistics baseline, an augmentation baseline and an in-distribution oracle at a large width, on at least one INR and one CNN dataset?
  3. Theorem 4 predicts that even the duplication-compatible GMN generalizes on µP-trained one-hidden-layer MLPs. Does a one-hidden-layer INR experiment confirm this, with the gap from the matrix-product GMN appearing only at depth ≥ 2? If such an experiment already exists in the appendix, please move it into the main text. It is the cleanest validation of the theory.
  4. Do the modifications extend to transformer-style GMNs such as the attention variant in Kofinas et al. (2024)? Softmax attention is already a weighted mean, but attention logits computed from entrywise edge features would seem to reintroduce Problem 1.
1 comment
AuthorsOpenReview ↗

Response to reviewer bnsF

We thank the reviewer for the constructive suggestions.

Weakness and Q1: experiment results table

We appreciate the suggestion and have now revised Fig. 2 and added a table. Fig. 2 now averages three GMN seeds per point with a min–max band, and the new table (Fig. 2(b)) reports mean ±\pm std numerical values. These confirm that seed variation is small relative to the differences we interpret.

Regarding the 5-point gap between SP and μ\muP at the FMNIST training width: we have checked that SP and μ\muP data are identically distributed as stated. The gap comes from randomness: the SP and μ\muP training, validation, and test sets are independent fits of the same images, and each parameterization trains its own GMN, with its own seed, checkpoint selection, and a learning rate chosen on a noisy sweep.

Regarding the SVHN width 512 and MNIST width 256 comparisons: we do not compare matrix-product GMN on SP inputs with duplication-compatible GMN on μ\muP inputs. Each observation in Sect. 4 compares either different models on the same inputs or the same model on SP versus μ\muP inputs, never different models on different inputs.

Q2: probing, weight-statistics, and augmentation baselines

Our experiments are designed to answer a conceptual question: whether the proposed modifications improve GMN size generalization on independently trained weights, and how this depends on the parameterization. They are not meant to establish our method as the state of the art for size generalization. We therefore believe the paper is self-contained without these baselines, and that adding them would dilute the main message. Instead, we compare with each baseline conceptually below.

  1. Probing. We agree that probing-based methods are a natural approach to size generalization. However, the forward matrix-product GMN can itself emulate probing: its message passing can simulate evaluating the input network on finitely many probes, and the probes can be learned through the GMN's learnable maps (the proof of the newly added Proposition 11 makes this explicit).
  2. Weight statistics. Standard entrywise weight statistics are in general not invariant under the equivalence studied here: a block-wise widening leaves the represented function unchanged, yet changes these statistics. We therefore do not expect methods built on these statistics to size-generalize robustly. See also our response to Reviewer Hnjz (W4/Q6).
  3. Augmentation. Augmenting with finitely many sampled widenings may encourage invariance to those particular widenings, but it does not help size generalization beyond the range of augmented widths. For example, if the training set is augmented with widenings up to width mm, we expect performance to degrade beyond width mm, just as it degrades beyond the training width without augmentation.

In-distribution oracle. An oracle trained at a large width would be a useful diagnostic, but training it is extremely expensive, and we believe the existing results already support our claims without it.

Q3: one-hidden-layer experiment

Thank you for the suggestion; we ran this experiment and added it to Sect. 5 (Fig. 3), next to Theorem 4. The setting is identical to the MNIST INR size-generalization experiment, except that the INRs have one hidden layer. The results match the theory: on μ\muP-trained inputs, both the duplication-compatible and the matrix-product GMN size-generalize equally well.

Q4: attention-based GMNs as in Kofinas et al. (2024)

We partially agree with the reviewer's intuition. But there is a caveat: Kofinas et al.'s [1] NG-T uses global relational self-attention: that is, it operates on a fully connected attention graph whose nodes are neurons, and for each ordered pair, the edge feature encodes the corresponding input-network weight if one exists; otherwise, it represents a "no-edge" relation. Hence, for each query neuron, the softmax acts globally on all neurons rather than locally on its neighbours. So the NG-T is not duplication compatible in general. It could be made duplication compatible, for example, by using fan-in-rescaled edge features, applying softmax separately over the key/value neurons from each input-network layer, and combining the layer-wise outputs in a width-independent way. But this replaces NG-T's global transformer attention with relation-wise graph attention, which makes it exactly a GMN. Therefore, we do not cover NG-T, as the required modifications are so substantial that they essentially replace the transformer structure with a GMN-like one.

References

[1] Miltiadis Kofinas et al. Graph neural networks for learning equivariant representations of neural networks. ICLR 2024.

Reviewer_HnjzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies size generalization of graph metanetworks (GMNs) across input-network widths. It treats weights of different widths that represent the same function via neuron duplication as equivalent, and proposes two modifications: (i) a duplication-compatible GMN, exactly invariant to duplication widening; and (ii) a matrix-product GMN, in which the weights enter only through matrix products with node features. The latter enjoys desirable properties, such as Lipschitz continuity and invariance to the broader class of block-wise widenings. Experiments on INR classification and CNN accuracy prediction, training at the smallest width and testing up to 32–42× wider, show that the matrix-product variant surpasses the duplication-compatible one, while both outperform the baseline GMNs as width scales. The design is supported by formal compatibility and continuity results, and the authors further prove size-generalization guarantees.

Critical strengths
  1. Well-motivated paper with a well-written storytelling.

  2. The design is derived, not ad hoc: the diagnosis that the target is continuous in the normalized spectral norm while entrywise GMNs are continuous in ℓ1\ell_1 directly motivates the matrix-product architecture.

  3. The experiments are to-the-point, each testing a specific theoretical prediction: numerical checks of compatibility and continuity (Tables 2–4), hand-crafted widenings, and a controlled SP/μP\mu P comparison in which both parameterizations coincide at the training width.

  4. The sanity check on hand-crafted widenings is convincing: the models are invariant by design, and the results show no degradation exactly where the theory predicts it.

  5. The results are promising: the ordering baseline < duplication-compatible < matrix-product is consistent across tasks, datasets, and GMN variants, and on μP\mu P-trained inputs the matrix-product GMN remains nearly flat.

Critical weaknesses
  1. Writing: The manuscript is dense and hard to follow for a non-expert. Theorems appear only informally in the main text; most of the substance is in a ~55-page appendix. Minor: Figure 2 overlays six curves per panel with a dashed/solid encoding that is hard to read.

  2. Reliance on μP\mu P / practical relevance: Under SP inputs, all models degrade with width. The proposed modifications delay the degradation but do not prevent it, since flat curves appear only with μP\mu P-trained inputs. Public model populations are likely SP-trained with per-width retuned hyperparameters, which is precisely the regime excluded by the theory and where empirical results are weakest. This limits the method's use as a general-purpose metanetwork.

  3. Theory–experiment gap: Theorem 4 covers only 1-hidden-layer MLPs, while the INRs have two hidden layers. Theorem 5 assumes the target is expressible as a tensor program (e.g., representable by the matrix-product GMN itself), which does not obviously cover the classification or test-accuracy targets used. Assumption 2 requires a pseudo-Lipschitz σ′\sigma^\prime, which ReLU (used for the INRs) violates. The CNN experiments have no theoretical counterpart. Finally, the guarantees are asymptotic in the training width and come without rates, so they say nothing quantitative about the experimental setting (e.g., training at width 24 and testing at 1024).

  4. Expressivity of the matrix-product GMN: Weight edges carry only the fixed scalar Wˉij\bar{W}_{ij} and are never updated. Thus the weights act only as linear maps on node features. As a consequence of invariance to block-wise widenings, the model cannot distinguish weights with different entrywise statistics, which have been found predictive of generalization in prior work [1]. Expressivity is not analyzed beyond the remark that no loss was observed, and it is unclear whether action-only access to the weights suffices for richer targets on larger, real-world models.

  5. Missing baselines: Probing-based methods (e.g., [4]) are naturally width-agnostic and continuous in the represented function, making them natural competitors for size generalization. Although the authors discuss this line of work and it is not commonly used as a baseline in the GMN literature, the comparison is particularly relevant here.

  6. Experiments: In-distribution (same-width) results are not reported as numbers; they can only be read off the leftmost point of each plot. The benchmarks are regenerated (necessary for multi-width test data, but also at the training width), so results are not comparable to published GMN [2] / ScaleGMN [3] numbers.

  7. Scope of tasks: Although the introduction motivates metanetworks that output new weights or weight updates, all experiments and theory concern invariant (graph-level) predictions; no editing task (e.g., INR editing as in [3]) is included. Extending the framework to such tasks is non-trivial.

  8. Format (minor): There is no dedicated Limitations or Conclusion section, although limitations are discussed inline.

[1] Unterthiner, Thomas, et al. "Predicting neural network accuracy from weights." arXiv preprint arXiv:2002.11448 (2020).

[2] Lim, Derek, et al. "Graph metanetworks for processing diverse neural architectures." International Conference on Learning Representations, 2024.

[3] Kalogeropoulos, Ioannis, Giorgos Bouritsas, and Yannis Panagakis. "Scale equivariant graph metanetworks." Advances in Neural Information Processing Systems 37 (2024): 106800–106840.

[4] Kahana, Jonathan, et al. "Deep linear probe generators for weight space learning." International Conference on Learning Representations.

Questions for authors
  1. The paper notes that duplication widening extends to most standard architectures. For networks with LayerNorm/BatchNorm, residual connections, etc., does the matrix-product form survive?

  2. If the method applies to such networks, could the authors evaluate on a zoo containing them?

  3. Could the authors provide a table with the in-distribution (same-width) results for all variants, and ideally results on at least one standard benchmark, so they can be compared with published GMN/ScaleGMN numbers? This would also test the claim that the matrix-product restriction costs no performance.

  4. Which GMN direction is shown in Figure 2 ? Given that only the forward matrix-product GMN is invariant to all block-wise widenings (Prop. 3(ii)), and that the ScaleGMN paper reports the forward variant suffices for property prediction tasks on shallow networks, why use the bidirectional variant as the main baseline? In Figure 4, the Matrix-product bidirectional GMN also shows performance degradation.

  5. Do the theoretical results extend to training on inputs of multiple widths? An experiment on a zoo with varying widths (depth is explicitly out of scope) would be informative.

  6. Can the matrix-product GMN capture entrywise weight statistics known to predict generalization? How do the authors expect its restricted, action-only access to the weights to affect richer tasks?

  7. How do the proposed models compare to probe-based metanetworks in size generalization?

  8. Can the theoretical results extend to CNN input networks?

  9. How would the framework extend to equivariant tasks such as weight editing? In particular, what is the appropriate notion of widening-equivariance for the outputs, and how could the matrix-product GMN, which has no edge states, produce per-weight outputs?

5 comments
AuthorsOpenReview ↗

Response to reviewer Hnjz (1/3)

We thank the reviewer for the careful reading and the many constructive suggestions.

W2: reliance on μ\muP assumption

The dependence on μ\muP limits the scope of our strongest transfer guarantee. However, we believe this does not limit the practical use of our method, for two reasons.

First, μ\muP is not a requirement of our method; it is needed only for the strongest form of our size-generalization guarantee. All results in Sect. 3 (exact invariance to widenings and width-independent continuity) are properties of the GMN architecture and hold for any input weights, regardless of the parameterization they were trained with. μ\muP enters only in Sect. 5, where it is the condition under which we prove this strongest form: generalization uniformly over all larger widths, even as the width goes to ∞\infty. This condition is not there for technical convenience; some assumption relating the training of wide networks to that of narrow ones is unavoidable. Without such an assumption, wide networks could follow a distribution unrelated to that of the narrow ones and behave completely differently, so no metanetwork could be guaranteed to generalize.

Second, we believe that the improvement of size generalization performance is strong even on SP-trained input networks. Our test range is deliberately extreme, up to about 42×42\times the training width, whereas in practice a metanetwork only needs to stay flat over the widths it actually encounters. For example, on SP-trained MNIST INRs, the matrix-product GMN stays flat up to about 5×5\times the training width, whereas the baseline GMN already deteriorates at 1.3×1.3\times (Fig. 2). We believe this is already relevant in practice.

W3, Q8: theory-experiment gap

We agree that, strictly speaking, none of the experimental settings in Sect. 4 is exactly covered by Theorems 4 and 5. Nonetheless, the assumptions of Theorem 5 are fairly general. We discuss the gaps one by one, and we believe most are small and technical rather than conceptual.

Target. For accuracy prediction, the target is a function of finitely many evaluations of the input network on a fixed, width-independent test set. This is exactly the example in our remark on the target assumption (App. E.3): any Lipschitz function of such evaluations, e.g., the test loss or a smoothed accuracy, is computable by a tensor program. Hence the accuracy prediction task is covered by our target assumption. The only discrepancy is that the exact 00--11 accuracy is discontinuous, but any smooth surrogate of accuracy satisfies the assumption. For INR classification, whether the target can be uniformly expressed by matrix-product GMNs is indeed an open question. As we mention in the paper, this concerns the expressivity of GMNs uniformly across widths, an important future direction. Moreover, for any target that is not uniformly approximable by matrix-product GMNs, one should not expect robust size generalization anyway. In this sense, the guarantee given by Theorem 5 is fairly general.

Activation. We believe the regularity assumption (Assumption 2) that excludes ReLU is purely technical. The regularity assumptions are inherited from Tensor Programs theory for infinite-width scaling limits (Assumption 2.3.2 of [5]). They hold for GELU, tanh, and sigmoid. They fail for ReLU because its derivative is discontinuous, but they hold for any smooth approximation of ReLU. [5] call this "a common technicality not treated in the theoretical literature" and conjecture that their results also hold for ReLU.

Architecture (Q8). Theorem 5 is stated for MLP inputs. We expect the proof to extend directly to CNNs (and possibly transformers), but we have not checked the details. Intuitively, the training of CNNs and transformers is expressible as a tensor program ( [6], App. A). Hence the evaluation of the GMN on CNN and transformer weights can also be appended to the tensor program, as in App. E.3.

Depth. Theorem 4 indeed covers only one-hidden-layer inputs, but its main purpose is not generality; it is to provide geometric intuition. The restriction comes from its proof route. Theorem 5 is the more general result and applies to MLP inputs of general depth.

We have revised Sect. 5 to clarify the target assumption and to note that the assumptions are fairly general.

References

[5] Etai Littwin and Greg Yang. Tensor programs IVb: Adaptive optimization in the infinite-width limit. ICLR 2023.

[6] Greg Yang. Wide feedforward or recurrent neural networks of any architecture are Gaussian processes. NeurIPS 2019.

AuthorsOpenReview ↗

Response to reviewer Hnjz (2/3)

W4, Q6: expressing entrywise weight statistics; expressivity is not analyzed beyond the remark that no loss was observed

That entrywise statistics are predictive of generalization in [1] does not mean that sensitivity to them is necessary for the task. In this paper, we focus on tasks whose target depends only on the function that the input network computes, which includes the generalization-prediction task of [1]. Many weights with different entrywise statistics compute exactly the same function, so a model can predict such a target without being sensitive to these statistics. We have added App. B.5, which discusses expressivity in more detail. There we extend [7] to prove that, at every fixed width, the matrix-product GMN is universal for continuous functionals of the represented function on compact weight sets. Its restriction therefore loses no expressivity at a fixed width for any function-level target. The empirical results support this as well (see our response to W6/Q3 below).

Across varying widths, sensitivity to entrywise statistics is not merely unnecessary but harmful to size generalization. Block-wise widenings preserve the represented function, but they change entrywise weight statistics. The matrix-product GMN is designed to be invariant to these widenings, so it cannot express any entrywise weight statistic that they change. This is an intended trade-off: what the model gives up is exactly the distinguishing power that obstructs generalization across widths.

Finally, the weight statistics that [1] finds most informative can be expressed by the matrix-product GMN uniformly across widths. [1] finds that the first and last layers are the most useful single layers, and that biases stand out among the most important features (see their Sect. 4.5, Table 2, App. B). Biases are node features, and the first-layer (and, in the bidirectional model, last-layer) weights attach to input (and output) neurons, which carry distinct positional features. The matrix-product GMN therefore reads these entries individually, followed by a layer-wise mean readout that preserves such information. Hence, it can express these statistics uniformly across widths.

W5, Q7: comparison with probe-based metanetworks

We agree that probing methods [4] are naturally width-agnostic and invariant under transformations that preserve the represented function. However, our experiments are designed to answer a conceptual question: whether the proposed modifications improve GMN size generalization on independently trained weights, and how this depends on the parameterization. They are not meant to establish our method as the state of the art for size generalization. We therefore believe the paper is self-contained without further adding probing methods as baselines, and that adding them would dilute the main message.

Additionally, we note that the forward matrix-product GMN can itself emulate probing: its message passing can simulate evaluating the input network on finitely many probes, and the probes can be learned through the GMN's learnable maps (the proof of the newly added Proposition 11 makes this explicit).

W6, Q3: experiment results table

We appreciate the suggestion and have now added the requested table to Sect. 4 (Fig. 2(b)). As the "train" columns show, the matrix-product GMN's in-distribution test score is close to those of the other models, so in our experiments the restriction causes little performance loss. We also added a proof of universality of the matrix-product GMN on fixed width in App. B.5, theoretically showing that the restriction loses no expressivity at a fixed width.

Regarding why we regenerate the datasets also at the training width, which makes them not directly comparable with existing benchmarks: a controlled size-generalization test requires networks at all widths to come from the same generation procedure. The published benchmarks contain a single width, and their pipelines cannot be rerun exactly at other widths, so we generate all widths, including the training width, with one pipeline. We believe that the lack of comparability is less of a concern here, since we study size-generalization behavior rather than state-of-the-art accuracy.

References

[7] Dayan, Adir, Yam Eitan, and Haggai Maron. "On the Expressive Power of Permutation-Equivariant Weight-Space Networks." ICML 2026.

AuthorsOpenReview ↗

Response to reviewer Hnjz (3/3)

W7, Q9: equivariant task and INR editing

Our method can be directly extended to equivariant tasks. We added a discussion in App. F.2, with a theoretical discussion and an INR-editing experiment. In brief:

  • Widening-equivariance. The appropriate notion is that, for every duplication (block-wise) widening θ↑\theta^\uparrow of θ\theta, the output fΘ(θ↑)f^\Theta(\theta^\uparrow) should represent the same function as fΘ(θ)f^\Theta(\theta).
  • Output heads. The matrix-product GMN has no edge features, so each weight output is instead a learned bilinear form W^ij(ℓ)=1nℓ−1hi⊤Bℓhj\widehat W^{(\ell)}_{ij}=\frac{1}{n_{\ell-1}}h_i^\top B_\ell h_j in the node features of the two neurons it connects.
  • Guarantees. The guarantees of our Propositions 1 and 3 can be extended directly to the equivariant setting.
  • Experiment. MNIST INR editing as in ScaleGMN [3] (target: 3×33\times3 grey-scale dilation of the digit). The results are consistent with the invariant-task experiments of Sect. 4: observations there still hold.

W8: no dedicated Limitations or Conclusion section

We have added a Conclusion and limitations section (App. G), which summarizes our findings and collects the limitations and future work: generalization across depth, theory beyond MLP input networks, and an expressivity theory for metanetworks of arbitrary width.

Q1, Q2: extension of the method to architectures with LayerNorm/BatchNorm, residual connections, etc

We believe the same modifications apply to all architectures the original GMN was designed for. To illustrate the generality of our work, we added a transformer experiment (Sect. 4, the ViT column of Fig. 2a and the ViT rows of Fig. 2b), and this transformer has LayerNorm and residual connections. We will try to add the experiment details in the Appendix before the end of the discussion period. (Update: it's now in Appendix D) The task is to predict the test accuracy of small vision transformers (two blocks, two attention heads, LayerNorm and residual connections) trained on CIFAR-10 and SVHN. The GMN acts on a parameter graph similar to that of [2], and the modifications are the same as for MLPs/CNNs. We did not prove exact invariance to widenings or continuity with respect to the spectral distance for the transformer GMN during the rebuttal period, but we verified both numerically. The size-generalization empirical results are very similar to those on MLPs and CNNs.

Q4: forward vs bidirectional GMN

Fig. 2 shows the bidirectional variant, one of four baselines (forward/bidirectional ×\times GMN/ScaleGMN). Since all variants show the same trend, we show one in the main text and the rest are in Fig. 6 (App. D.3). We chose to show the bidirectional one because it is the more general message-passing scheme, and Theorems 4 and 5 cover it. To clarify, invariance to block-wise widening is a design motivation, not a claimed necessary condition for size generalization on independently trained weights. So the bidirectional matrix-product GMN's lack of invariance to some widenings need not hurt it there. Indeed, Theorems 4 and 5 cover both forward and bidirectional variants (we highlight this in the revision), and empirically, on trained weights the forward (Fig. 6a) and bidirectional (Fig. 6b) variants perform similarly, unlike on the hand-crafted block-wise widenings (now Fig. 5).

Q5: training on multiple widths

The theory extends directly to training on several widths. Theorems 4 and 5 bound the gap between the GMN's expected loss at two widths for every fixed choice of GMN parameters, regardless of how those parameters were learned.

We added a multi-width experiment in the appendix (Fig. 10, App. F.1). We use the same experimental setup on MNIST INR classification, except that we now train on three widths (24, 32, 48) instead of one (24). Generalization is good at the training widths, and at larger widths (>48>48) the overall trend is the same as in the single-width case. The main observations are unchanged: our modifications improve size generalization, and only the matrix-product GMN on μ\muP-trained inputs stays flat as the width grows.

Reviewer_8UFGOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes transferable graph metanetworks for cross-width generalization through widening invariance and continuity, supported by experiments on INR classification and accuracy prediction, and asymptotic guarantees under μP.

Critical strengths
  1. Principled design: Widening invariance and continuity provide a theoretically grounded framework for cross-width generalization.

  2. Practical relevance: Transfer to independently trained networks up to 42.7× wider supports the value of training on small networks.

Critical weaknesses
  1. Theory–experiment gap: The guarantees rely on regularity and target assumptions whose applicability to the ReLU-based classification experiments is not established. They are also asymptotic, leaving their connection to transfer from the small training widths indirect. This limits the theory’s support for the main empirical claims.

  2. Unclear model-selection protocol: The main text describes validation-based model selection, whereas Appendix D.3.2 mentions selection using test performance at the training width. This discrepancy requires clarification: if test data influenced model selection, the reported results could be optimistically biased, weakening confidence in the comparisons.

Questions for authors
  1. Clarification—evaluation protocol: Were GMN hyperparameters and checkpoints selected exclusively using validation data? How should the validation-based selection described in Section 4 be reconciled with the reference to test-based selection in Appendix D.3.2?

  2. Clarification—theoretical scope: Which experimental settings are formally covered by Theorem 5? In particular, how do the ReLU training dynamics and classification targets satisfy the required regularity and target assumptions?

1 comment
AuthorsOpenReview ↗

Response to reviewer 8UFG

We thank the reviewer for the constructive suggestions.

W1, Q2: theory-experiment gap

We agree that, strictly speaking, none of the experimental settings in Sect. 4 is exactly covered by Theorem 5. Nonetheless, the assumptions of Theorem 5 are fairly general, and the experiments test the qualitative predictions of the theory. We discuss the remaining gaps one by one.

Target. Exact representability by a matrix-product GMN is sufficient but not necessary for the target assumption of Theorem 5. We have edited the paper to make clear that the target assumption is computability by a tensor program. For generalization prediction, the target is a function of finitely many evaluations of the input network on a fixed, width-independent test set. This is exactly the example in our remark on the target assumption (App. E.3): any Lipschitz function of such evaluations, e.g., the test loss or a smoothed accuracy, is computable by a tensor program. Hence the generalization prediction task is covered by our target assumption. The only discrepancy is that the exact 00--11 accuracy is discontinuous, but any smooth surrogate of accuracy satisfies the assumption. For INR classification, whether the target can be uniformly expressed by matrix-product GMNs is indeed an open question. As we mention in the paper, this concerns the expressivity of GMNs uniformly across widths, an important future direction. Moreover, for any target that is not uniformly approximable by matrix-product GMNs, one should not expect robust size generalization anyway. In this sense, the guarantee given by Theorem 5 is fairly general.

Activation. We believe the regularity assumption that excludes ReLU is purely technical. The regularity assumptions are inherited from Tensor Programs theory for infinite-width scaling limits (Assumption 2.3.2 of [1]). They hold for GELU, tanh, and sigmoid. They fail for ReLU because its derivative is discontinuous, but they hold for any smooth approximation of ReLU. [1] call this "a common technicality not treated in the theoretical literature" and conjecture that their results also hold for ReLU.

Architecture. Theorem 5 is stated for MLP inputs. We expect the proof to extend directly to CNNs (and possibly transformers), but we have not checked the details. Intuitively, the training of CNNs and transformers is expressible as a tensor program ([2], App. A). Hence the evaluation of the GMN on CNN and transformer weights can also be appended to the tensor program, as in App. E.3.

We have revised Sect. 5 to clarify the target assumption and to note that the assumptions are fairly general.

W2, Q1: model-selection protocol

All selection of GMN hyperparameters and checkpoints uses the validation performance only, where the validation data comprises networks at the training width. The test networks (at all widths) are never used for selection. We apologize for the confusion caused by our imprecise phrasing in App. D.3.2, and we have now corrected it.

References

[1] Etai Littwin and Greg Yang. Tensor programs IVb: Adaptive optimization in the infinite-width limit. ICLR 2023.

[2] Greg Yang. Wide feedforward or recurrent neural networks of any architecture are Gaussian processes. NeurIPS 2019.

Reviewer_NRBHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents the construction of a "transferable graph metanetwork." The basic idea is to extend graph metanetworks --- or models that make predictions about the weights of a neural network (e.g. whether it generalizes or performs well on a given task). These models need to be invariant to a number of edits that don't affect the behavior of networks, like permutations of neurons or certain rescalings. In contrast to past work, this paper shows how to incorporate invariances when the width of a layer is modified, using a clever construction. The situation is also analyzed theoretically.

Critical strengths

The paper is fairly well written, and the problem is interesting. I've never worked on metanetworks before (sorry, the ICLR process matches papers with pretty random reviewers...) but found the problem interesting. The exposition of the main body is fairly clear, at least when it comes to the basics of the problem and of the approach, and the construction of an architecture invariant to layer width is clever. This was a problem I haven't thought about before, and I enjoyed learning about it by reading the paper.

Critical weaknesses

My main complaint is that the PDF here is 67 pages, so the body of the paper is really just an advertisement for the longer treatment. In the time allotted for ICLR reviewing, I am unable to go through all 67 pages in detail. So in essence, publishing this work is like publishing a paper without rigorous review of every claim. On the other hand, I understand this is a common practice among ICLR submissions.

The main technical difficulty left to future work is invariance to adding layers instead of making layers wider. I don't consider this a weakness of the paper (the contribution presented as-is is already enough) but would appreciate if the authors comment a bit on what it might take to extend to this case, as a way to motivate future work.

Notation in eq (2) and some other similar constructions was quite hard to read.

Section 4 could use revision, especially if the paper is intended to be read/understood just from the main body rather than reading the appendices. Only two experiments are presented involving INRs/CNNs, and the details don't appear in sec4. I'd appreciate more thoughtful summaries of the experiments in the appendix. What are the main take aways? How large are the networks your meta-networks were able to process successfully (number/width of layers)? And how many parameters are in these meta-networks? What is the metanetwork training/inference time? I'd be curious to see more of these kinds of numbers to understand if the proposed method is practical and extends to large models.

At least for my taste, I didn't find the theory laid out in sec5 to be particularly illuminating. This section needs more thoughtful discussion of how to interpret the theorems, and what they tell us about the proposed method (or the field itself). Are these results surprising/interesting, or just standard applications of generalization bounds? Are the additional assumptions (e.g. Theorem 4 only applies to one-layer models) so restricting that we can't learn much from the statement?

Questions for authors

The paper mentions that the method is "architecture agnostic," but I'm wondering how far this claim goes. Can attention mechanisms or sequence models be compared using the construction here? Or is the "agnosticism" just across variants of MLPs that use specialized linear layers (e.g., CNNs, GNNs)?

2 comments
AuthorsOpenReview ↗

Response to reviewer NRBH (1/2)

We thank the reviewer for the positive feedback and constructive suggestions.

Question: how far does "architecture agnostic" go

The construction also applies to transformers. We added a transformer experiment (Sect. 4, the ViT column of Fig. 2a and the ViT rows of Fig. 2b). We will try to add the experiment details in the Appendix before the end of the discussion period. (Update: it's now in Appendix D) The task is to predict the test accuracy of small vision transformers (two blocks, two attention heads, LayerNorm and residual connections) trained on CIFAR-10 and SVHN. The GMN acts on a parameter graph similar to that of [1] and the modifications are the same as for MLPs/CNNs. We did not prove exact invariance to widenings or continuity with respect to the spectral distance for the transformer GMN during the rebuttal period, but we verified them numerically. The size-generalization empirical results are very similar to those on MLPs and CNNs.

Future work on size generalization across depth

We have added the following discussion of future work to the new Conclusion and limitations section (App. G). Extending the results to depth is interesting but challenging, as none of the three ingredients of our construction has a depth counterpart. First, the natural function-preserving deepening (e.g., inserting a residual block with a zero branch) breaks the symmetry. Copies of an exchangeable neuron are indistinguishable from the original, whereas an inserted layer is not exchangeable with the others. Second, in the width case we are able to define norms and metrics on the space of weights of all widths and consider continuity there. It is unclear whether this is possible on the space of weights of all depths. Third, Sect. 5 relies on the infinite-width limit theory, a law of large numbers over exchangeable neurons, whereas the infinite-depth limits of ResNets are ODE/SDE limits. Therefore, extending Sect. 5 to depth would require a very different set of tools.

Details of experiments

We have revised the experiment section. It now includes experiments on three kinds of input weights (MLP, CNN, ViT), each on two different datasets. We added a table showing means and stds. We include some details such as the number of layers of the input networks and of the GMNs. The widths of the input networks are given on the x-axis of the plots. We stress that the key takeaway is conceptual, summarized as three observations just before Sect. 5, which motivate the theory in Sect. 5. We do not propose our method as the state of the art for size generalization, so further experiment details may dilute the main messages.

Notation hard to read

The notation is somewhat unavoidable, as it is needed to make the discussion of how the GMN modifications are implemented, and the statements of their properties in later parts of the paper, precise. If the reviewer prefers, we are happy to add an illustration of the GMN message passing mechanism to the appendix to make it clearer.

References

[1] Derek Lim et al. Graph metanetworks for processing diverse neural architectures. ICLR 2024.

AuthorsOpenReview ↗

Response to reviewer NRBH (2/2)

Theory in Sect. 5: Is this a standard generalization bound?

No, Sect. 5 is not a standard generalization bound. A standard generalization bound controls sup⁡Θ∣R^n(fΘ)−Rn(fΘ)∣\sup_{\Theta}\lvert\widehat{\mathcal R}_n(f^\Theta)-\mathcal R_n(f^\Theta)\rvert, where R^n\widehat{\mathcal R}_n is the empirical loss on a training sample drawn from μn\mu_n. This is the gap between the training loss and the in-distribution test loss. Our theorems instead bound sup⁡m≥n∣Rn(fΘ)−Rm(fΘ)∣\sup_{m\ge n}\lvert\mathcal R_n(f^\Theta)-\mathcal R_m(f^\Theta)\rvert, the gap between the in-distribution test loss at the training width nn and the out-of-distribution test loss at every larger width mm. This gap is specific to size generalization, and it is exactly what the flatness of the curves in Fig. 2 measures.

The proof technique is different from standard generalization bounds. Our theory compares μn\mu_n and μm\mu_m, the laws of trained weights on spaces of different dimensions, which capture the training and test distributions. The content of the theorem is about understanding these two distributions using the theory of infinite-width training dynamics of neural networks, showing that the training and test distributions are similar enough for size generalization to be possible.

Theory in Sect. 5: What does the theory tell us?

We agree that Sect. 5 was presented in a way that hid the essential theoretical takeaways. We've now made substantial edits to improve the presentation in Sect. 5 to better explain the mechanism behind each guarantee and relate Theorems 4 and 5 to the distance-based and distance-free approaches to GNN transferability. In brief:

The theory in Sect. 5 establishes a connection between two areas that, to our knowledge, have not been connected before: size transferability of GNNs, and infinite-width scaling limits of neural networks. In the GNN transferability literature, input graphs of growing size converge to a common limiting object, such as a graphon. The GNN outputs are then proved to converge as well, so a GNN trained on small graphs transfers to large ones. Our setting is very similar in spirit, with input networks of growing width in place of graphs of growing size. Under μ\muP, trained input networks of growing width converge to a common limiting object, the infinite-width limit of the training dynamics described by Tensor Programs theory. The matrix-product GMN respects this convergence, so its outputs converge too.

This extends transferability theory to the weight-space setting in two ways. Theoretically, it brings in a new limiting object, the infinite-width limit of trained networks, together with new tools to analyze it. Practically, the convergence of the inputs is not a modeling assumption on the data (as in graphon theory), but holds whenever the input networks are independently trained with μ\muP. As such, the theoretical guarantee obtained here is directly related to practical applications in weight-space learning.

Theory in Sect. 5: Are the assumptions too restrictive?

Theorem 4 covers only 1-hidden-layer input MLPs, but provides a geometric intuition: under μ\muP, the trained weights themselves become close in the distance in which the GMN is continuous. It also explains why the duplication-compatible GMN already suffices at depth 1 (Fig. 3), whereas deeper inputs require the matrix-product GMN.

Theorem 5 is the general result. It holds for input MLPs of any fixed depth and for both forward and bidirectional matrix-product GMNs. The assumptions are fairly mild. We have now added a comment in the main paper to highlight this, and it's discussed in more detail in App. E.3, and in our responses to Reviewers 8UFG and nrQY. In brief:

  • Target. The target only needs to be computable by a tensor program. Exact representability by a matrix-product GMN is sufficient but not necessary. This includes any Lipschitz function of finitely many evaluations of the input network on a fixed input set, e.g., the test loss or a smoothed accuracy on a fixed test set. So it does cover the generalization prediction task.
  • Activation. The regularity assumption is a technicality inherited from Tensor Programs theory [2]. It holds for GELU, tanh, and sigmoid, and excludes ReLU only because of the discontinuity of its derivative, a technicality that the authors of [2] conjecture can be removed.
  • μ\muP. This assumption is not a technical convenience: some assumption relating the training of wide networks to that of narrow ones is unavoidable, since otherwise wide networks could follow a distribution unrelated to that of narrow ones (see also our response to Reviewer Hnjz, W2).

References

[2] Etai Littwin and Greg Yang. Tensor programs IVb: Adaptive optimization in the infinite-width limit. ICLR 2023.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. We have uploaded a revised version, with major changes highlighted in blue. Apart from fixing some small typos and notation, the main additions are:

  • To better illustrate that our method is architecture-agnostic on the input network, we added experiments for transformer input networks (Sect. 4, Fig. 2). The experiment is on generalization prediction for Vision Transformers (two blocks, with LayerNorm and residual connections) on CIFAR-10 and SVHN. The findings are consistent with those for MLPs and CNNs. We will try to add the experiment details in the Appendix before the end of the discussion period. (Update: it's now in Appendix D) [NRBH, Hnjz]
  • For all the experimental results in Sect. 4, we repeat the experiments with three GMN training seeds, show the min--max bands in Fig. 2a, and add a results table (Fig. 2b) with the mean ±\pm standard deviation as detailed numerical values. [nrQY, Hnjz, bnsF]
  • We add an experiment on one-hidden-layer INRs (Sect. 5, Fig. 3) for direct verification of theory: on μ\muP inputs, the duplication-compatible GMN size-generalizes as well as the matrix-product GMN, as Theorem 4 predicts. [bnsF]
  • We add an experiment for training on multiple widths (App. F.1, Fig. 10). [Hnjz]
  • We show that our method directly extends to equivariant tasks. Propositions 1 and 3 extend to this setting, and we add an INR editing experiment (App. F.2, Fig. 11). [Hnjz]
  • To show that the restriction to matrix-product GMNs does not cost expressivity at a fixed width, we prove that the matrix-product GMN remains universal there. (App. B.5, Prop. 11). This result directly extends Theorem 6.1 of Dayan et al. [Hnjz, nrQY]
  • We make substantial edits to improve the presentation in Sect. 5 to (1) better explain the mechanism behind each guarantee and relate Theorems 4 and 5 to the distance-based and distance-free approaches to GNN transferability (now reviewed in Sect. 2.2), and (2) highlight that the assumptions of these guarantees are fairly general. The edits concern presentation only; the theorem statements and proofs are unchanged. [NRBH, 8UFG, nrQY, Hnjz]
  • We add the discussion of conclusion and limitations, including discussion of future work (App. G). [NRBH, Hnjz]

References. Dayan, Adir, Yam Eitan, and Haggai Maron. "On the Expressive Power of Permutation-Equivariant Weight-Space Networks." ICML 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.