- Rating
- 4
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes size generalization for Graph Metanetworks (GMNs): the GMN is trained on small input networks and evaluated on much wider ones, with theoretical guarantees. The approach consists of two levels of modification. The first makes any GMN invariant to duplication widening; the second drops the learned edge features and lets the weights act on node features only through matrix multiplication, which makes the GMN invariant to the broader class of block-wise function-preserving widenings and Lipschitz continuous in a normalized spectral norm. The method is evaluated on MNIST/FMNIST (INR classification) and CIFAR-10/SVHN (accuracy prediction), showing that both modifications improve size generalization and that µP-trained input networks are much easier to generalize over than SP-trained ones.
Critical strengths
- The problem is well presented, and the introduction and related work make clear that existing approaches generalize across sizes only to a limited extent.
- The design is clean, well motivated, and architecture-agnostic.
- The invariance and continuity properties of the proposed modifications are all supported by theoretical proofs (Propositions 1–3).
Critical weaknesses
- Although the paper is well written, the main text lacks a summary of the empirical results. A compact table is needed to understand the gains clearly without having to dig through the appendix.
- Theorem 4 covers 1-layer input MLPs only, and the obstruction at depth ≥ 3 is proven at initialization; Theorem 5 requires the target to be exactly representable by a matrix-product GMN and gives no rate. The experiments use 2-hidden-layer INRs and CNNs, for which neither theorem applies
Questions for authors
- Could you add a main-text table reporting, for each dataset, parameterization, and model, the score at the training width, the score at the largest test width, and the resulting size-generalization gap, with error bars over GMN training seeds?
- Theorem 5 requires the target to be exactly representable by a matrix-product GMN, and the experiments use 2-hidden-layer INRs and CNNs that neither theorem covers. Can you argue that the INR-classification and accuracy-prediction targets fall in that class, and is there any empirical evidence on whether the depth-≥3 obstruction persists after training?
- Can you report training-width performance for all three models in a table, to support the claim at line 371 that the matrix-product restriction costs no performance?
2 comments
Thank you very much for your detailed responses which addressed my concerns and also helped me in better understanding. I will maintain my positive score and raise my confidence.
Response to reviewer nrQY
We thank the reviewer for the constructive suggestions.
We appreciate the suggestion and have now added the requested table to Sect. 4 (Fig. 2(b)), reporting the numerical values as mean ± std over three GMN training seeds (Fig. 2 now also shows a min–max band). As the "train" columns show, the matrix-product GMN's in-distribution test score is close to those of the other models, so in our experiments the restriction causes little performance loss. We also added a proof of universality of the matrix-product GMN on fixed width in App. B.5, theoretically showing that the restriction loses no expressivity at a fixed width.
We agree that, strictly speaking, none of the experimental settings in Sect. 4 is exactly covered by Theorems 4 and 5. Nonetheless, the assumptions of Theorem 5 are fairly general. We discuss the gaps one by one, and we believe most are small and technical rather than conceptual.
Target (Q2). Exact representability by a matrix-product GMN is sufficient but not necessary for Theorem 5. We have edited the paper to make clear that the target assumption is computability by a tensor program. For accuracy prediction, the target is a function of finitely many evaluations of the input network on a fixed, width-independent test set. This is exactly the example in our remark on the target assumption (App. E.3): any Lipschitz function of such evaluations, e.g., the test loss or a smoothed accuracy, is computable by a tensor program. Hence the accuracy prediction task is covered by our target assumption. The only discrepancy is that the exact 0--1 accuracy is discontinuous, but any smooth surrogate of accuracy satisfies the target assumption. For INR classification, whether the target can be uniformly expressed by matrix-product GMNs is indeed an open question. As we mention in the paper, this concerns the expressivity of GMNs uniformly across widths, an important future direction. Moreover, for any target that is not uniformly approximable by matrix-product GMNs, one should not expect robust size generalization anyway. In this sense, the guarantee given by Theorem 5 is fairly general.
Architecture. Theorem 5 is stated for MLP inputs. We expect the proof to extend directly to CNNs (and possibly transformers), but we have not checked the details. Intuitively, the training of CNNs and transformers is expressible as a tensor program ([1], App. A). Hence the evaluation of the GMN on CNN and transformer weights can also be appended to the tensor program, as in App. E.3.
Depth. Theorem 4 indeed covers only one-hidden-layer inputs, but its main purpose is not generality; it is to provide geometric intuition. The restriction comes from its proof route. Theorem 5 is the more general result and applies to MLP inputs of general depth.
We have revised Sect. 5 to clarify the target assumption and to note that the assumptions are fairly general.
Yes, we checked this numerically, though these checks are not in the paper. The direct check uses trained 2-hidden-layer MNIST INRs (500 per width), for which we compute the mean normalized spectral norm of each layer, under both SP and μP.
Under SP, we prove that ∥θn∥Vn,op=Θ(n) at initialization for every depth, due to the last layer. This growth persists after training: the last-layer statistic increases monotonically from 27.17 to 56.35.
Under μP, we prove that liminfW1(μn,μ2n;d∞op)≥c>0 at initialization, due to the hidden-to-hidden layer. This obstruction also persists after training: the first and last layers are nearly constant, but the hidden-to-hidden layer drifts, decreasing monotonically from 13.69 to 11.89.
References
[1] Greg Yang. Wide feedforward or recurrent neural networks of any architecture are Gaussian processes. NeurIPS 2019.