- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In this paper, the authors propose H-GMN, a hierarchical message passing based graph metanetwork. This method leverages the Neural Network’s inherent hierarchical architecture to coarsen and refine the graph. This is achieved using permutation-symmetry-preserving pooling and un-pooling layers. The same ideas are then extended to Transformer architectures. The proposed methods are validated on benchmark model zoos on tasks including Lipschitz bound proxy prediction, backdoor detection and pruning.
Critical strengths
- The hierarchical method seems novel, well motivated, and is relevant to the field of weight space learning.
- The paper is well written (clear and easy to follow).
- The paper presents interesting efficiency results (sample efficiency and cost under increasing network depth).
- The experimental setup seems comprehensive with varied tasks.
Critical weaknesses
- The methodology brings together several ingredients: multiscale representations, architecture-derived grouping, permutation-invariant pooling, additional message-passing depth, and, in some experiments, GRIT with relative random-walk positional encodings. The scalability results are encouraging and motivate the use of such a complex architecture but given that the paper also advocates that the hierarchical nature of the model is essential it begs the question of the real contribution of the hierarchy as an inductive bias (see question1 below). For instance, an ablation comparing the architecture-derived pooling with simpler graph-level or layer-level aggregation followed by the same coarse-level GRIT architecture would be interesting to see.
- Given that the experiments are not all the result of several runs, it is hard to assess the statistical significance and robustness of the reported results.
Questions for authors
- Contribution of the hierarchy: Can the authors disentangle the effect of architecture-derived pooling from the effect of using deeper or more expressive coarse-level GNNs, particularly GRIT?
- Statistical reliability: Are the reported generalisation and robustness results averaged over multiple seeds or splits? Can uncertainty estimates be provided, especially where the differences are small?
3 comments
W2: Not all experiments are the result of several runs. Q2: Are the reported generalisation and robustness results averaged over multiple seeds or splits? Can uncertainty estimates be provided, especially where the differences are small?
We thank the reviewer for raising this point. The backdoor detection results are averaged over three seeds, and we now also report the standard deviations in the revised manuscript (Section A.8.6, Table 9).
Given the large number of zoos, tasks and baselines covered by our evaluation, the remaining experiments were each run with a single seed. We therefore base our claims on differences that are large relative to the variation we observe between closely related variants. On ResNet-18 generalisation prediction, for example, the full 3-lvl-H-GMN, its RC variant and its variant without message passing on the Parameter Graph lie within ~0.01 in of each other at every training-set size (Table 8), and swapping the fine-level module changes by at most ~0.03 across all generalisation and robustness experiments. The gaps we highlight are considerably larger: 0.927 vs. 0.804 on CIFAR-100 ViT, 0.850 vs. 0.744 on ResNet18-0.1, and 0.849 vs. 0.577 on the Varying ViT zoo. Where differences are small, as on ResNet-18 with the full training set, we only claim that H-GMN matches flat GMN. Finally, the Lipschitz-bound experiment shows that the advantage of H-GMN is consistent across five training-set fractions (Fig. 2).
I thank the authors for addressing my comments and providing additional ablations. I have no further questions and have raised my score accordingly.
We thank the reviewer for the positive assessment and the thoughtful questions, which led to new experiments in this rebuttal.
We thank the reviewer for their question. We ran several controls (Global Response, Section 1), which we group by the confound they address.
Is it GRIT or coarse-level depth?
Can the same layers succeed without architecture-derived pooling?
Flat GRIT (new). Applying GRIT directly on the full Parameter Graph (GMN GRIT) falls below every H-GMN variant on MNIST ViT generalisation prediction (τ 0.863 vs. 0.919) and on ResNet-18 at all three training-set sizes, and fails on Blended and WaNet backdoor detection (AUROC 0.430 / 0.578; Tables 2 and 4).
Deeper flat GMN (new). On the MNIST ViT zoo, the smallest one, a flat GMN approaches H-GMN only with 8 or more layers (τ 0.917 vs. 0.919), at more than twice the training time (Section A.8.2, Fig. 14). As shown in Figure 6, increasing the depth of a flat GMN quickly becomes prohibitive on larger ViT models.
The reviewer's suggested ablation (new). The pooling operations of H-GMN are already simple: each one summarises the vertices and edges of an architectural component, such as a layer or a residual block, through permutation-invariant statistics, with the grouping determined by the architecture rather than learned. Motivated by the reviewer's suggestion, we evaluated the simplest instance of this design (Section A.8.5, Table 8): a parameter-matched 3-lvl-H-GMN on ResNet-18 generalisation prediction in which the embedded weights are pooled immediately, with no message passing on the Parameter Graph (k1=0), and the coarse graphs are processed by GMN layers only. This variant matches the full 3-lvl-H-GMN at every training-set size (τ 0.851 / 0.882 / 0.898 vs. 0.850 / 0.881 / 0.893 at 10% / 50% / 100% of the data), and clearly outperforms flat GMN when training data is scarce (τ 0.851 vs. 0.744 at 10%). It also outperforms StatNN, which summarises each layer with fixed statistics but performs no subsequent message passing (τ 0.851 vs. 0.677 at 10%).
Taken together, summarising each architectural component and processing the resulting coarse graphs accounts for most of the improvement, without GRIT, without deeper coarse-level processing, and even without message passing on the Parameter Graph. The hierarchy also provides an affordable way to obtain a large receptive field, which a flat GMN can reach only through much greater depth, and only on small zoos.