acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Graph Metanetworks

Abstract

The emerging field of weight space learning has demonstrated the potential to enable metanetworks, i.e. models that process neural network (NN) data, to support several steps of the machine learning lifecycle, from model optimisation to evaluation and adaptation. However, this potential has yet to materialise beyond small-scale MLPs or CNNs: most current models, including message-passing based graph metanetworks (GMNs), operate exclusively at the finest level of granularity, processing every neuron and weight of the input model. Their cost therefore scales with the number of parameters. Yet NNs are inherently organised hierarchically, with neurons forming layers and layers forming blocks and modules. Additionally, several of their properties, such as Lipschitz upper bounds and generalisation measures, are compositions of layer- and block-level quantities. To incorporate this inductive bias, we introduce Hierarchical Graph Metanetworks (H-GMNs), which leverage the input architecture’s hierarchy to coarsen and refine the graph using permutation-symmetry-preserving pooling and un-pooling layers. Further, we extend H-GMNs to Transformers by introducing a graph construction that preserves permutation symmetries and enables multi-level pooling. Our evaluation includes new ResNet-18 and ViT model zoos with up to 21M parameters per model, on tasks spanning model evaluation (an analytic Lipschitz bound proxy, generalisation and robustness prediction), testing (backdoor detection) and editing (pruning). Experimental results show that, compared to non-equivariant alternatives and flat GMNs, H-GMNs are markedly more sample-efficient, and their cost remains nearly constant as metanetwork depth grows, allowing deeper and more expressive processing where flat GMNs run out of memory, without compromising accuracy, which in certain cases considerably improves.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
weight space learning, graph neural networks, hierarchical pooling, symmetries, metanetworks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Q1LuOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors propose H-GMN, a hierarchical message passing based graph metanetwork. This method leverages the Neural Network’s inherent hierarchical architecture to coarsen and refine the graph. This is achieved using permutation-symmetry-preserving pooling and un-pooling layers. The same ideas are then extended to Transformer architectures. The proposed methods are validated on benchmark model zoos on tasks including Lipschitz bound proxy prediction, backdoor detection and pruning.

Critical strengths
  • The hierarchical method seems novel, well motivated, and is relevant to the field of weight space learning.
  • The paper is well written (clear and easy to follow).
  • The paper presents interesting efficiency results (sample efficiency and cost under increasing network depth).
  • The experimental setup seems comprehensive with varied tasks.
Critical weaknesses
  1. The methodology brings together several ingredients: multiscale representations, architecture-derived grouping, permutation-invariant pooling, additional message-passing depth, and, in some experiments, GRIT with relative random-walk positional encodings. The scalability results are encouraging and motivate the use of such a complex architecture but given that the paper also advocates that the hierarchical nature of the model is essential it begs the question of the real contribution of the hierarchy as an inductive bias (see question1 below). For instance, an ablation comparing the architecture-derived pooling with simpler graph-level or layer-level aggregation followed by the same coarse-level GRIT architecture would be interesting to see.
  2. Given that the experiments are not all the result of several runs, it is hard to assess the statistical significance and robustness of the reported results.
Questions for authors
  1. Contribution of the hierarchy: Can the authors disentangle the effect of architecture-derived pooling from the effect of using deeper or more expressive coarse-level GNNs, particularly GRIT?
  2. Statistical reliability: Are the reported generalisation and robustness results averaged over multiple seeds or splits? Can uncertainty estimates be provided, especially where the differences are small?
3 comments

We thank the reviewer for the positive assessment and the thoughtful questions, which led to new experiments in this rebuttal.

W1: The real contribution of the hierarchy as an inductive bias. Q1: Can the authors disentangle the effect of architecture-derived pooling from the effect of using deeper or more expressive coarse-level GNNs, particularly GRIT?

We thank the reviewer for their question. We ran several controls (Global Response, Section 1), which we group by the confound they address.

Is it GRIT or coarse-level depth?

  • Parameter-matched H-GMN, one layer per level (new). On the CIFAR-100 ViT zoo (generalisation prediction), we restricted H-GMN to the parameter budget of flat GMN (~664K parameters) and to a single layer at each level, in two variants: one with GMN layers at every level, and one with GRIT on the coarsest level. Both clearly outperform flat GMN (τ\tau 0.804): τ\tau 0.936 with GMN layers only and 0.924 with GRIT (Table 5, Section A.8.1). The gains therefore require neither GRIT, nor deeper coarse-level processing, nor a larger model.
  • Lipschitz-bound prediction, ResNet-18 (original submission). With GMN layers only and all models under the same parameter budget, 3-lvl-H-GMN matches flat GMN using one-fifth of the training data, and 2-lvl-H-GMN is better at every data size (Fig. 2).

Can the same layers succeed without architecture-derived pooling?

  • Flat GRIT (new). Applying GRIT directly on the full Parameter Graph (GMN GRIT) falls below every H-GMN variant on MNIST ViT generalisation prediction (τ\tau 0.863 vs. 0.919) and on ResNet-18 at all three training-set sizes, and fails on Blended and WaNet backdoor detection (AUROC 0.430 / 0.578; Tables 2 and 4).

  • Deeper flat GMN (new). On the MNIST ViT zoo, the smallest one, a flat GMN approaches H-GMN only with 8 or more layers (τ\tau 0.917 vs. 0.919), at more than twice the training time (Section A.8.2, Fig. 14). As shown in Figure 6, increasing the depth of a flat GMN quickly becomes prohibitive on larger ViT models.

The reviewer's suggested ablation (new). The pooling operations of H-GMN are already simple: each one summarises the vertices and edges of an architectural component, such as a layer or a residual block, through permutation-invariant statistics, with the grouping determined by the architecture rather than learned. Motivated by the reviewer's suggestion, we evaluated the simplest instance of this design (Section A.8.5, Table 8): a parameter-matched 3-lvl-H-GMN on ResNet-18 generalisation prediction in which the embedded weights are pooled immediately, with no message passing on the Parameter Graph (k1=0k_1 = 0), and the coarse graphs are processed by GMN layers only. This variant matches the full 3-lvl-H-GMN at every training-set size (τ\tau 0.851 / 0.882 / 0.898 vs. 0.850 / 0.881 / 0.893 at 10% / 50% / 100% of the data), and clearly outperforms flat GMN when training data is scarce (τ\tau 0.851 vs. 0.744 at 10%). It also outperforms StatNN, which summarises each layer with fixed statistics but performs no subsequent message passing (τ\tau 0.851 vs. 0.677 at 10%).

Taken together, summarising each architectural component and processing the resulting coarse graphs accounts for most of the improvement, without GRIT, without deeper coarse-level processing, and even without message passing on the Parameter Graph. The hierarchy also provides an affordable way to obtain a large receptive field, which a flat GMN can reach only through much greater depth, and only on small zoos.

W2: Not all experiments are the result of several runs. Q2: Are the reported generalisation and robustness results averaged over multiple seeds or splits? Can uncertainty estimates be provided, especially where the differences are small?

We thank the reviewer for raising this point. The backdoor detection results are averaged over three seeds, and we now also report the standard deviations in the revised manuscript (Section A.8.6, Table 9).

Given the large number of zoos, tasks and baselines covered by our evaluation, the remaining experiments were each run with a single seed. We therefore base our claims on differences that are large relative to the variation we observe between closely related variants. On ResNet-18 generalisation prediction, for example, the full 3-lvl-H-GMN, its RC variant and its variant without message passing on the Parameter Graph lie within ~0.01 in τ\tau of each other at every training-set size (Table 8), and swapping the fine-level module changes τ\tau by at most ~0.03 across all generalisation and robustness experiments. The gaps we highlight are considerably larger: τ\tau 0.927 vs. 0.804 on CIFAR-100 ViT, 0.850 vs. 0.744 on ResNet18-0.1, and 0.849 vs. 0.577 on the Varying ViT zoo. Where differences are small, as on ResNet-18 with the full training set, we only claim that H-GMN matches flat GMN. Finally, the Lipschitz-bound experiment shows that the advantage of H-GMN is consistent across five training-set fractions (Fig. 2).

I thank the authors for addressing my comments and providing additional ablations. I have no further questions and have raised my score accordingly.

Reviewer_aja8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Hierarchical Graph Metanetworks (H-GMNs), a weight-space learning framework that processes neural network parameters as a graph at multiple resolutions instead of performing all message passing on the full parameter graph. The hierarchy is derived from the architecture of the input network: individual neurons and weights are progressively consolidated into layers, attention heads, and residual blocks using pooling functions intended to preserve the relevant permutation symmetries. For editing tasks, the authors introduce unpooling and U-Net-style skip connections so that coarse representations can be combined with fine-grained information before producing parameter-level modifications. They also propose a per-head parameter-graph construction for multi-head self-attention, with the goal of representing the symmetries within individual heads and between interchangeable heads. The evaluation covers a compositional Lipschitz upper-bound proxy, generalization and robustness prediction, backdoor detection, and weight pruning, using ResNet-18 and ViT model populations, including a new ViT-S/16 zoo with approximately 21 million parameters per model. On generalization prediction, H-GMNs are particularly competitive in low-data regimes and perform substantially better than the flat GMN baseline on the large CIFAR-100 ViT zoo. The backdoor-detection and pruning experiments also report favorable results, while the runtime and memory experiments show that additional processing at coarser resolutions is considerably cheaper than stacking the same number of message-passing layers on the full graph; however, the evidence does not establish that H-GMNs outperform every baseline on every task or that the approach already scales to genuinely large language models.

Critical strengths
  • H-GMN addresses the high cost of flat graph metanetworks by moving some message passing to smaller graphs that represent layers, attention heads, and residual blocks. This makes deeper processing more practical.

  • The pooling and unpooling operations account for neuron and attention-head permutations. This is important because equivalent neural networks can have different parameter arrangements.

  • Unlike methods that only predict properties of a model, H-GMN uses an encoder-decoder with skip connections to retain detailed weight information for tasks such as pruning.

  • On the CIFAR-100 ViT task, three-level H-GMN achieves a Kendall correlation of 0.927 and (R^2=0.979), compared with 0.804 and 0.673 for flat GMN. This supports the value of hierarchical processing for this setting.

Critical weaknesses

(W1) Limited methodological novelty. H-GMN combines established ideas from graph metanetworks, hierarchical graph pooling, and U-Net-style architectures. Its main contribution is how these components are combined with the structure and symmetries of neural networks, rather than a completely new learning principle.

(W2) The source of performance gains is unclear. Some H-GMN experiments use GRIT at the coarsened level, while the flat GMN baseline does not use an equivalent graph transformer. Without matched comparisons, it is difficult to determine how much improvement comes from hierarchical pooling rather than GRIT's greater expressive power.

(W3) Limited evidence of generalization. The ResNet-18 and ViT experiments use fixed architecture families, and the ViT splits do not hold out entire pretrained backbones. The results therefore do not establish that H-GMN generalizes to unseen architectures, backbones, or model sizes.

(W4) The attention-head hierarchy is not isolated. The paper introduces a head-level graph for ViTs but does not clearly compare two-level and three-level H-GMN on the same ViT tasks. It is therefore unclear how much this specific design improves performance.

Questions for authors

(Q1) Can the authors isolate the contribution of hierarchy from the use of GRIT? The appendix states that some experiments search over both ordinary GNNs and GRIT for the coarse-level module. In the backdoor-detection and pruning experiments, however, the strongest H-GMN configuration uses GRIT while the flat GMN baseline does not.

Could the authors report the following controls, where computationally feasible?

  • H-GMN with ordinary GMN layers at the coarse resolution.
  • A flat GRIT baseline for at least one of the tasks.
  • A comparison matched by parameter count or compute budget.

These experiments would help establish whether the improvements come from architecture-derived pooling, the larger effective receptive field of GRIT, or their combination.

(Q2) Does H-GMN generalize to an unseen pretrained backbone or model width?

(Q3) How much does the explicit attention-head level contribute?

(Q4) Does H-GMN provide a practical pruning benefit beyond conventional recovery methods? The pruning experiment compares H-GMN and GMN with magnitude pruning without editing. Could the authors include a standard post-pruning fine-tuning baseline under a comparable compute budget? The goal would be to determine whether the metanetwork learns a useful model-level editing strategy beyond a conventional recovery procedure, rather than only showing that one metanetwork is better than another.

5 comments

We thank the reviewer for the detailed and constructive review. The requested controls led to several new experiments, summarised in the Global Response and detailed below.

W1: Limited methodological novelty.

We agree that H-GMN is built from ingredients that are well established in graph machine learning. Its contribution, however, does not lie in any single component, but in identifying a property of the weight-space modality that previous metanetworks have neglected, its hierarchical structure, and in discovering how to exploit it. This was neither obvious nor expected to work, as we detail below.

1. An architecture-derived, symmetry-respecting hierarchy. Generic hierarchical pooling methods learn which nodes to group. On parameter graphs with millions of edges, this is expensive and offers no guarantee that the learned clusters align with the components whose internal permutations are symmetries of the network. H-GMN instead derives the coarsening from the architecture itself: neurons are pooled into layers, attention heads and residual blocks, invariantly to permutations within each component and equivariantly to permutations of interchangeable components, such as heads. This coarsening is therefore not tailored to a specific zoo: virtually all neural networks, whether MLPs, CNNs or Transformers, are organised hierarchically, with neurons forming layers and layers forming blocks and modules. Our experiments cover both CNNs and Transformers, and in the new Varying ViT zoo a single H-GMN processes architectures of different widths, depths and MLP ratios with the same coarsening, generalising to architectures unseen during training (Global Response, Section 2).

2. Multi-resolution processing helps structure-aware metanetworks scale. For a flat GMN, every layer operates on the full parameter graph, so its cost grows with both metanetwork depth and input size. As input networks grow, a flat GMN must therefore become shallower and/or narrower to fit in memory (Figs. 5–6). H-GMN moves most of the processing to coarse graphs whose size scales with the number of architectural components, not with the parameter count. Its cost thus remains nearly constant as metanetwork depth grows, and costlier, more expressive layers such as graph transformers become affordable at the coarse levels.

3. U-Net-style architecture. For editing tasks, which need equivariant outputs at the neuron level, the fine-grained information aggregated by pooling must be recovered. This is what led us to equivariant un-pooling layers within a U-Net-style encoder–decoder with skip connections. While U-Nets are well known, their equivariant adaptation to weight space is, to our knowledge, new.

4. A per-head graph construction for multi-head attention, which exposes both within-head and head-relabelling symmetries to pooling and enables a head-level resolution. While not the core contribution of the paper, we believe it is of independent interest for any metanetwork processing Transformers. Its head level substantially improves generalisation prediction on CIFAR-100 ViT (τ\tau 0.862 → 0.927; Q3).

5. Non-obvious empirical findings. Our results also challenge two common assumptions in graph metanetworks:

  • Early invariant pooling does not hurt. For prediction tasks, H-GMN aggregates neuron-level information into coarse nodes after a single fine-level layer. One might expect this to discard essential information, yet H-GMN matches or outperforms flat GMNs, even under a matched parameter budget (Table 5, Fig. 2).

  • Neuron-level message passing is not essential. During the rebuttal, we replaced the fine-level message-passing layer with a lightweight row–column layer (H-GMN-RC). It performs on par with the original (differences in τ\tau between −0.026 and +0.022), while using about 60% less GPU memory and running up to 2.2× faster (Global Response, Section 4). This suggests that, once the hierarchy is in place, the established message-passing paradigm of GMNs is not necessary at the finest level.

Overall, H-GMN offers a structure-aware alternative to flat GMNs that scales far better, by exploiting a previously neglected property of the weight-space modality.

W2: The source of performance gains is unclear.

Please see our answer to Q1, where we report all three requested controls.

W3: Limited evidence of generalisation.

Please see our answer to Q2.

W4: The attention-head hierarchy is not isolated.

Please see our answer to Q3.

Q1: Can the authors isolate the contribution of hierarchy from the use of GRIT?

We report each of the three requested controls.

1. H-GMN with ordinary GMN layers at the coarse resolution.

  • Generalisation prediction, CIFAR-100 ViT (new). H-GMN with GMN layers at every level reaches τ\tau 0.936, against 0.804 for flat GMN (Table 5).

  • Backdoor detection, ResNet-18. The GMN-only 2-lvl-H-GMN already outperforms flat GMN on Blended, WaNet and the multi-class setting (Table 4).

  • Lipschitz-bound prediction, ResNet-18 (original submission). With GMN layers only, 3-lvl-H-GMN matches flat GMN using one-fifth of the training data, and 2-lvl-H-GMN is also better at every data size (Fig. 2).

2. A flat GRIT baseline (new). We apply GRIT directly on the full Parameter Graph, without RRWP, which is unaffordable at this resolution.

  • Generalisation prediction, MNIST ViT. GRIT falls below every H-GMN variant, and also below plain GMN: τ\tau 0.863, against 0.916 for flat GMN and 0.919 for H-GMN (Table 2).
  • Generalisation prediction, ResNet-18. GRIT falls below every H-GMN variant at all three training-set sizes, with no consistent gain over plain GMN (e.g. 10% of the data: 0.764, against 0.744 for flat GMN and 0.846–0.860 for H-GMN; Table 2).
  • Backdoor detection, ResNet-18. GRIT fails on Blended and WaNet (AUROC 0.430 / 0.578, against 1.000 / 0.970 for the GMN-only H-GMN; Table 4).

A more expressive layer alone therefore does not explain the gap.

3. A comparison matched by parameter count.

  • Generalisation prediction, CIFAR-100 ViT (new). We restricted H-GMN to the parameter budget of flat GMN (~664K parameters), in two variants: one with GMN layers at every level, and one with GRIT on the coarsest level, as in our full model. Both clearly outperform flat GMN (τ\tau 0.804): τ\tau 0.936 with GMN layers only and 0.924 with GRIT (Table 5, Section A.8.1).
  • Lipschitz-bound prediction, ResNet-18 (original submission). All models in Fig. 2 already share the same parameter budget.

Together, all these controls show that the improvements come mainly from the architecture-derived hierarchy, not from GRIT or model size. GRIT helps mainly at the coarse levels, where the hierarchy makes it (and RRWP) affordable.

Q2: Does H-GMN generalise to an unseen pretrained backbone or model width?

We thank the reviewer for this question, which prompted two new experiments:

Unseen widths and architectures We constructed a new Varying ViT zoo (Section A.3.3) of 1,015 ViTs from 2.0M to 85.9M parameters, varying width d∈{192,…,768}d \in \{192, \dots, 768\}, depth L∈{6,…,12}L \in \{6, \dots, 12\} and MLP ratio r∈{2,3,4}r \in \{2, 3, 4\}, each derived from one of eight public ViT-B/16 checkpoints. On test architectures never seen during training, H-GMN-RC reaches τ\tau 0.849 / R2R^2 0.941, against 0.577 / 0.491 for flat GMN and 0.795 / 0.554 for StatNN (Table 2). Going beyond the claims of the paper, we further stress-tested all methods on a Beyond split of architectures larger than every training model, which also requires extrapolating to considerably higher accuracies. All methods degrade, and H-GMN is the only one with a positive R2R^2 (Table 6, Section A.8.3). We include this as an additional stress test, not as a claim of size extrapolation (Global Response, Section 2).

Unseen pretrained backbones. In the Varying ViT zoo, backbones are balanced across splits, so it tests unseen architectures rather than unseen backbones. We therefore also ran an experiment on the CIFAR-100 ViT zoo, where we leave one backbone out: we train on four of the five backbones, holding out all models fine-tuned from the supervised (SL) backbone, and evaluate on both the in-distribution (ID) test split of the four training backbones and the held-out SL backbone (Section A.8.4, Table 7).

MethodID τ\tauID R2R^2ID within-backbone τ\tauHeld-out SL τ\tauHeld-out SL R2R^2
StatNN0.8550.7900.6000.003−0.048
GMN0.8210.7830.6270.109−0.384
H-GMN0.8850.9080.8020.203−0.109

On the ID split, H-GMN clearly outperforms both baselines, especially when ranking models fine-tuned from the same backbone (within-backbone τ\tau), and it is the only method that ranks well the models of the contrastive (CL) backbone, on which both baselines fail (τ\tau −0.092 / 0.160 vs. 0.777; Table 7). On the held-out SL backbone, performance drops sharply for all methods and R2R^2 is negative throughout. H-GMN retains the strongest ranking, but evidently generalisation across pretraining backbones remains an open challenge. While this setting lies beyond the scope of the present work, which targets scalable, structure-aware processing rather than robustness to distribution shifts, we hope these results provide a useful reference point for future work and have included them in the revised manuscript (Section A.8.4).

Q3: How much does the explicit attention-head level contribute?

We thank the reviewer for proposing the experiment. We compared 2-lvl-H-GMN, which pools the Parameter Graph directly to the Layer Graph, with 3-lvl-H-GMN, which additionally processes the Head Graph. The two models are otherwise identical. On CIFAR-100 ViT generalisation prediction, the head level raises τ\tau from 0.862 to 0.927 and R2R^2 from 0.872 to 0.979 (Table 2). On mCE prediction the gain is smaller (τ\tau 0.925 vs. 0.932; Table 3). The head level therefore contributes substantially, especially for generalisation prediction.

Q4: Does H-GMN provide a practical pruning benefit beyond conventional recovery methods?

We thank the reviewer for suggesting this comparison. As we detail in Global Response, Section 3, we fine-tune all 1,200 pruned ResNet-18 test networks for up to 1,500 steps, from two starting points under one identical protocol: the magnitude-pruned weights (Magnitude + FT) and the H-GMN edit of the same network (H-GMN + FT). Both use the same fixed magnitude mask, update exactly the parameters the metanetwork edits, and minimise the metanetwork's own objective, with a learning rate tuned for Magnitude + FT (Section A.7). Results are shown in Figure 4 of the updated manuscript.

  • Cost of matching the edit. Fine-tuning needs about 10, 50 and 200 SGD steps (batches of 128 images, i.e. up to ~25.6K training images) at 80%, 50% and 20% remaining weights to match the accuracy of H-GMN's one-shot edit, which takes a single forward pass of the metanetwork and no data at edit time.

  • Warm start. At 20% remaining weights, starting fine-tuning from the H-GMN edit still halves the accuracy loss after 1,500 steps, and gives a lower JSD on 93% of the networks. At milder pruning, both starting points converge to the same accuracy, as expected.

The metanetwork therefore learns a useful editing strategy that conventional recovery does not reproduce for free: it replaces tens to hundreds of fine-tuning steps with a single forward pass, and under heavy pruning it remains a better starting point even after long fine-tuning.

Reviewer_RL9zOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Hierarchical Graph Metanetworks (H-GMNs), a framework for scaling weight-space learning by exploiting the natural hierarchical structure of neural networks. It extends existing graph metanetworks through symmetry-preserving pooling and unpooling across multiple architectural levels, enabling more efficient processing of larger neural networks.

Critical strengths
  1. The paper addresses a well-defined scalability limitation of existing graph metanetworks, with a clear and practically relevant objective.

  2. The hierarchical approach is intuitive, technically sound, and demonstrates substantial computational benefits while preserving important permutation symmetries.

  3. Evaluation covers model property prediction, backdoor detection, and pruning, demonstrating improved computational and sample efficiency and competitive or improved performance.

  4. The paper is well written, with a clear explanation of the methodology and its advantages.

Critical weaknesses
  1. Although scalability is central, experiments are limited to networks up to 21 million parameters. The authors acknowledge that in the limitations.

  2. While experiments demonstrate improved sample efficiency, it would be useful to disentangle benefits of hierarchical inductive bias from reduced computational cost and more efficient information propagation.

Questions for authors

Could you please conduct an experiment to address weakness 2?

2 comments

We thank the reviewer for the positive assessment and the constructive comments.

W1: Scalability limited to networks up to 21 million parameters.

We thank the reviewer for raising this point. To go beyond the ~21M-parameter regime and fixed architectures, we constructed a new Varying ViT zoo of 1,015 models from 2.0M to 85.9M parameters, varying in width, depth and MLP ratio (Section A.3.3). We summarise the main findings here; full details are in Section 2 of the Global Response.

  • Generalisation to unseen architectures. On test architectures never seen during training, H-GMN-RC reaches τ\tau 0.849 / R2R^2 0.941, against 0.577 / 0.491 for flat GMN and 0.795 / 0.554 for StatNN (Table 2).
  • Scalability. With metanetworks of matched size, flat GMN runs out of memory on a 64 GB GPU beyond 38M input parameters at inference and 22M in training, whereas 3-lvl-H-GMN-RC processes the full 86M-parameter ViT-B, both at inference and in training (Fig. 7). In the Varying ViT experiment, flat GMN could only be trained on models up to ~48M parameters after reducing its hidden size.
  • Stress test. On the 9 largest architectures (60–86M parameters, all significantly larger than every training model), all methods degrade, and H-GMN is the only one with a positive R2R^2 (Table 6). We report this as a stress test rather than a claim of size extrapolation.

W2: Disentangle the benefits of the hierarchical inductive bias from reduced computational cost and more efficient information propagation. Q1: Could you please conduct an experiment to address weakness 2?

We agree this is an important question, and we ran several new controls (Section 1 of the Global Response). We address the two confounds separately.

Computational cost and model size.

  • Parameter-matched H-GMN (new). On the CIFAR-100 ViT zoo (generalisation prediction), we restricted H-GMN to the parameter budget of flat GMN (~664K parameters), in two variants: one with GMN layers at every level, and one with GRIT on the coarsest level, as in our full model. Both clearly outperform flat GMN (τ\tau 0.804): τ\tau 0.936 with GMN layers only and 0.924 with GRIT (Table 5, Section A.8.1). The gains therefore come neither from model size nor from the graph transformer.

  • Flat GRIT (new). Conversely, giving flat GMN a more expressive layer, by applying GRIT on the full Parameter Graph, does not close the gap. On generalisation prediction, it falls below every H-GMN variant on MNIST ViT (τ\tau 0.863, against 0.916 for flat GMN and 0.919 for H-GMN) and on ResNet-18 at all three training-set sizes, with no consistent gain over plain GMN (e.g. 10% of the data: 0.764, against 0.744 for flat GMN and 0.846–0.860 for H-GMN; Table 2). On ResNet-18 backdoor detection, it fails on Blended and WaNet (AUROC 0.430 / 0.578, against 1.000 / 0.970 for a GMN-only H-GMN; Table 4).

  • Lipschitz-bound prediction (original submission). As already shown in the original manuscript, on ResNet-18 models, with all metanetworks under the same parameter budget and with GMN layers only, 3-lvl-H-GMN matches flat GMN using one-fifth of the training data, and 2-lvl-H-GMN is better at every data size (Fig. 2). We designed this task to require hierarchical reasoning, since the target is computed compositionally from layer- and block-level quantities.

Information propagation.

A flat GMN needs one message-passing layer per hop, whereas H-GMN propagates information between distant layers through its coarse graphs. To test this directly, we trained flat GMNs of 2, 8 and 16 layers on the MNIST ViT zoo, the smallest one, where deep flat GMNs still fit in memory (Section A.8.2, Fig. 14). With enough depth, a flat GMN does approach H-GMN (8 GMN layers: τ\tau 0.917 vs. 0.919), confirming that part of the benefit comes from a larger receptive field. However, this comes at more than twice the training time (8.1 h for 8 layers and 15.9 h for 16, against 3.5 h for H-GMN), and, as shown in Figure 6, increasing the depth of a flat GMN quickly becomes prohibitive on larger ViT models.

Taken together, the hierarchy accounts for a large share of the improvement, while also offering an affordable way to obtain a large receptive field, which a flat GMN can reach only through much greater depth, and only on small zoos.

I thank the authors for their rebuttal, i am maintaining my positive score.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Comment and Rebuttal Summary

We would like to thank all reviewers for their careful reading and constructive feedback. Their comments led us to run additional experiments that further support our claims. Below, we first summarise the strengths highlighted by the reviewers, and then outline our responses and the changes to be included in the revised manuscript.

  • Significance (RL9z, aja8, Q1Lu): the paper tackles a "well-defined scalability limitation of existing graph metanetworks" (RL9z), "the high cost of flat graph metanetworks" (aja8), and is "relevant to the field of weight space learning" (Q1Lu).

  • Well-motivated, symmetry-aware method (Q1Lu, RL9z, aja8): "novel, well motivated" (Q1Lu); "intuitive, technically sound [...] preserving important permutation symmetries" (RL9z); "pooling and unpooling operations account for neuron and attention-head permutations" (aja8); "Unlike methods that only predict properties of a model, H-GMN uses an encoder-decoder with skip connections" (aja8).

  • Efficiency (RL9z, aja8, Q1Lu): "substantial computational benefits" and "improved computational and sample efficiency" (RL9z); "makes deeper processing more practical" (aja8); "interesting efficiency results" (Q1Lu).

  • Comprehensive evaluation (RL9z, Q1Lu, aja8): "model property prediction, backdoor detection, and pruning" with "competitive or improved performance" (RL9z); "comprehensive with varied tasks" (Q1Lu); aja8 noted the gains of 3-level H-GMN over flat GMN on CIFAR-100 ViTs.

  • Clarity (RL9z, Q1Lu): "well written, with a clear explanation of the methodology and its advantages" (RL9z); "clear and easy to follow" (Q1Lu).

Rebuttal Summary

1. Disentangling the hierarchy from expressive layers, depth and compute (RL9z W2, aja8 W2/Q1, Q1Lu W1/Q1)

All three reviewers asked whether the gains come from the hierarchy itself, or from using more expressive layers ( e.g. GRIT [1]), extra depth, or a larger compute budget. We ran the following controls (task: generalisation prediction, metric: Kendall's τ\tau unless stated):

ControlSettingResultRef.
Parameter-matched, GMN layers at every levelCIFAR-100 ViTH-GMN 0.936 > GMN 0.804Tab. 5
Parameter-matched, GRIT at the coarsest levelCIFAR-100 ViTH-GMN 0.924 > GMN 0.804Tab. 5
Parameter-matched, no message passing on the Parameter Graph (k1=0k_1 = 0), GMN layers onlyResNet-18, 10% dataH-GMN 0.851 > GMN 0.744Tab. 8
Parameter-matched, GMN layers onlyResNet-18, Lipschitz bound (R2R^2)H-GMN with 1/5 of the data ≈ GMN with all of itFig. 2
Flat GRIT on the full Parameter GraphMNIST ViTH-GMN 0.919 > GRIT 0.863Tab. 2
Flat GRIT on the full Parameter GraphResNet-18, 10% dataH-GMN 0.850 > GRIT 0.764Tab. 2
Flat GRIT on the full Parameter GraphResNet-18 backdoor (AUROC, Blended / WaNet)H-GMN 1.00 / 0.97 > GRIT 0.43 / 0.58Tab. 4
Head levelCIFAR-100 ViT3-lvl 0.927 > 2-lvl 0.862Tab. 2
Deeper flat GMN (8 layers)MNIST ViTH-GMN 0.919 ≈ GMN 0.917, at 3.5 h vs. 8.1 h trainingFig. 14

Takeaway. At the same parameter budget, and without GRIT, H-GMN still clearly outperforms flat GMN. GRIT on the full parameter graph does not close the gap, and a much deeper flat GMN approaches H-GMN on the smallest zoo, at a far higher cost. The hierarchy itself therefore accounts for a large share of the improvement, while also presenting an affordable way to achieve a larger receptive field. GRIT helps mainly at the coarse levels, where the hierarchy makes it (plus RRWP) affordable.

2. Scalability and generalisation to unseen architectures (RL9z W1, aja8 W3/Q2)

To move beyond fixed-architecture zoos and the ~21M-parameter regime, we constructed a new Varying ViT zoo (Section A.3.3): 1,015 ViTs fine-tuned on CIFAR-100 (5,075 checkpoints), ranging from 2.0M to 85.9M parameters. Architectures vary in width d∈{192,…,768}d \in \{192, \dots, 768\}, depth L∈{6,…,12}L \in \{6, \dots, 12\} and MLP ratio r∈{2,3,4}r \in \{2, 3, 4\}. Each model is derived by structured weight selection from one of eight public ViT-B/16 checkpoints (four AugReg, the original ImageNet-21k ViT, SAM, DINO and DeiT), and fine-tuned with varied optimisers and hyperparameters. We train on models up to ~48M parameters and test on 6 held-out architectures never seen during training.

Results (generalisation prediction, Table 2):

Methodτ\tauR2R^2
StatNN0.7950.554
GMN0.5770.491
H-GMN-RC (Ours)0.8490.941

The test architectures are never seen during training: they differ from all training architectures in width, depth and/or MLP ratio, and therefore in parameter count, while lying within the training size range (2.4M–33.5M; Fig. 8). This split thus measures generalisation to unseen architectures of familiar size. H-GMN generalises to unseen widths, depths and MLP ratios, outperforming both baselines by a large margin, while flat GMN degrades sharply.

Scalability. With metanetworks of matched size, flat GMN runs out of memory on a single 64 GB GPU beyond 38M input parameters at inference and 22M in training, whereas 3-lvl-H-GMN-RC is the only model that processes the full 86M-parameter ViT-B, both at inference and in training (Fig. 7). In the Varying ViT experiment, training flat GMN on models of up to ~48M parameters was only possible after reducing its hidden size, while H-GMN required no such compromise.

Stress test. We additionally evaluate on a Beyond split of the 9 largest architectures (60–86M parameters, all larger than every training model, and also considerably more accurate). All methods degrade under this joint extrapolation in size and label space, and H-GMN is the only method with a positive R2R^2 (Table 6, Section A.8.3). We report this as a stress test, not as a claim of size extrapolation.

3. Post-pruning fine-tuning (aja8 Q4)

We thank Reviewer aja8 for suggesting this comparison. We fine-tune all 1,200 pruned test networks for up to 1,500 steps, from two starting points under one identical protocol: the magnitude-pruned weights (Magnitude + FT) and the H-GMN edit of the same network (H-GMN + FT). Both use the same fixed magnitude mask (pruned weights stay at zero), update exactly the parameters the metanetwork edits, and minimise the metanetwork's own objective (the JSD between dense and pruned outputs), with a learning rate tuned for Magnitude + FT. Details are in Section A.7.1 and results in Figure 4.

Results.

  • Cost of matching the edit. Fine-tuning needs about 10, 50 and 200 SGD steps (batches of 128 images, i.e. up to ~25.6K training images) at 80%, 50% and 20% remaining weights to match the accuracy of H-GMN's one-shot edit, which takes a single forward pass of the metanetwork and no data at edit time.
  • Warm start. At 20% remaining weights, starting fine-tuning from the H-GMN edit still halves the accuracy loss after 1,500 steps, and gives a lower JSD on 93% of the networks. At milder pruning, where fewer weights are removed, both starting points converge to the same accuracy, as expected.

The edit is therefore not something fine-tuning reproduces for free: it replaces tens to hundreds of fine-tuning steps with a single forward pass, and under heavy pruning it remains a better starting point even after long fine-tuning.

4. A lighter fine-level module (Q1Lu W1, RL9z W1)

We also tested whether message passing on the full Parameter Graph is necessary before pooling. In H-GMN-RC, we replace the fine-level GMN layer with a row–column (RC) layer adapted from [2]: each weight is updated from its own embedding and the mean embeddings of its row, its column and its whole weight matrix, rather than through neuron-to-neuron message passing (Section A.4).

Results. Comparing each H-GMN variant with its RC counterpart (2-lvl with 2-lvl, 3-lvl with 3-lvl), the two perform on par. On generalisation prediction (Table 2), the differences in τ\tau range from −0.009 to +0.010, both on ResNet18-0.1. On mCE prediction (Table 3), they range from −0.026 to +0.022, both on ResNet18-0.5. Meanwhile, the RC variants use about 60% less GPU memory and run up to 2.2× faster (Section A.1).

Since two fine-level modules with quite different inductive biases lead to similar results, the gains do not rely on the specific fine-level module but on the hierarchy built on top of it.

AuthorsOpenReview ↗

References

[1] Ma, Liheng, et al. "Graph inductive biases in transformers without message passing." International Conference on Machine Learning. PMLR, 2023.

[2] Hartford, Jason, et al. "Deep models of interactions across sets." International Conference on Machine Learning. PMLR, 2018.

Dear Reviewers,

The authors have responded to your comments. Could you please review their responses and update your reviews?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.