- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper CrossGMN looks at learning transformations between weight spaces of networks with different widths and depths. The work builds on graph metanetworks (Lim et al., 2024), which represent neural networks as permutation-symmetry preserving graphs, and extends them to weight-space learning across different architectures. Instead of mapping the source weights directly to target weights, the authors define an operator that takes a trained source network and an initialization of the target network and refines the initialization. This makes the symmetry well defined as the operator is invariant to source permutations and equivariant to target permutations. CrossGMN joins the two parameter graphs with cross edges and runs a graph meta network over the joint graph, and they prove it is universal for such operators for MLPs. They test it on model compression, where CrossGMN refines a structured pruning initialization before knowledge distillation. They show speedups for using this work for different architectures like MLPs, CNNs, ViTs. They also show transfer from MNIST to FashionMNIST INRs and one operator trained on several MLP source architectures.
Critical strengths
- Authors address the problem of having permutation invariant representation of neural networks in a mathematically empirically sound approach. The work shows a principled way of representation neural networks as permutation invariant graphs for cross architecture weight space transformation.
- Their formulation of these networks is the main contribution. Taking the target initialization as a second input solves the problem that source and target have different symmetry groups. It is well motivated by showing that a direct source to optimal target map can be discontinuous, while a fixed number of distillation steps is a continuous operator of this form.
- The evaluation is careful and honest. They report per-teacher speedups with IQR and failure rates, wall-clock time including the forward pass, and reach-budget plots against several pruning baselines.
Critical weaknesses
- Models are small, up to about 210k parameters, and the architecture scales badly. Each layerwise join has d_src × d_tgt edges. The strided policy reduces the number of layer pairs but not this product, CNNs add k² parallel edges per channel pair, and for transformers every attention and residual coordinate is a node. So the scaling problem comes from the design, not just from compute budget.
- The method needs thousands of trained teachers of the same architecture. Their own ablation shows that with 1,000 teachers the speedup drops to 1.67×, and with 100 it is 0× with 51% failures. Practical compression usually targets a single large model. The wall-clock speedup also does not include the cost of training the operator, so it is not clear when this pays off.
- Classifier compression ratios are modest (about 1.04–2.4×) and so are the gains. The CNN result is a single width setting on a 2.4k-parameter model, and two of the four ViT settings are slower than the anchor (0.74× and 0.86×).
- All results are single runs without seeds so no error bars have been reported.
- It is not clear what the operator learns. The output is anchor + s·Δθ, results depend heavily on a good pruning initialization and there is no analysis of Δθ. The cross edges carry no correspondence information. All cross edges in a join have the same features, and the value is 0 for CNNs and ViTs. Any neuron alignment has to be learned through the messages. With pruning anchors the kept student neurons are exact copies of teacher neurons, so the gains may come from that rather than a real cross-architecture mapping. 6."Cross-architecture" here means different widths and depths within one model family. All experiments are within-family (MLP->MLP, CNN->CNN, ViT->ViT), while works cited as motivation, such as Liu et al. 2024, use the term for different model families. Cross-family transfer is listed as future work. Source and target must also share input and output dimensions.
- The theory covers MLPs only, and the universality proof is for plain GMN with position wise boundary joins. The experiments use ScaleGMN, and CNNs and ViTs use dense boundary joins. The formal results also cover only permutation symmetry, even though the backbone is scale or sign equivariant.
- Very few experimentation results on CNNs and ViTs.
- The work cites and talks about old weight space generation methods, there is no mention of latest flow matching , diffusion etc. based weight space learning methods and no comparison shown against any such methods.
Questions for authors
- Please report absolute test accuracy (classifiers) and PSNR to ground truth (INRs) for the teacher, the anchor and the CrossGMN student, both at step 0 (directly after the forward pass) and at the distillation budget. This would show whether CrossGMN produces useful students on its own, and whether faster convergence also gives equal or better final task performance, including for the teachers counted as failures.
- How large is the update s·Δθ relative to the anchor? How does one CrossGMN pass compare to taking k distillation steps from the anchor at equal compute? The forward pass takes about 18–20 ms, roughly 8–10 distillation steps. Does Δθ resemble a few distillation steps or something qualitatively different?
- The cross edges have no correspondence information, so how does the operator transfer information from specific teacher neurons to specific student neurons? Does performance hold up if the anchor is perturbed or not a direct copy of teacher neurons? In the random initialization ablation, what is the speedup measured against?
- Recent work (e.g. DeepWeightFlow, ICLR 2026) finds that non-equivariant models with canonicalization perform well on weight-space tasks at moderate scale. What is the authors’ take on its findings on permutation symmetry? Can they compare CrossGMN against a non-equivariant operator on re-basined teacher and anchor weights, to show that equivariance is what drives the gains?
- Nonlinearities enter the parameter graph only as optional node labels, and normalization parameters are auxiliary nodes with no cross edges, so they only get teacher information indirectly through the residual nodes. Could this explain the weaker ViT results? The depth anchors (layer fold) also remove intermediate nonlinearities. How does CrossGMN compensate for this, and is there any analysis of what it learns there? How can the network handle diverse normalization schemes and non linear activation functions?
- How do scale, sign and normalization symmetries, and teacher and student pairs with different activations, fit into the framework? What is the intended use case, given that the method needs thousands of teachers of the same architecture? How much does training the operator cost, and how many teachers need to be compressed before it pays off? Is it feasible for realistic model sizes such as ResNet-18 or a small language model?
- Please either clarify the scope in the title and abstract (e.g. “cross-width/depth” or “within-family”), or include at least one cross-family experiment such as MLP→CNN or CNN→ViT. How would cross edges be defined between different layer types?
12 comments
Authors’ Response to Reviewer KCk4 - Part 2
Training cost and break-even. We conduct a break-even analysis to determine how many model compressions are needed to pay off the training cost.
For each teacher, we compare the GPU time required by CrossGMN (including its forward pass and distillation) and the pruning-derived anchor to reach a teacher-relative performance target, independent of the anchor. Both methods use the same distillation budget (B) (Sec. 5.1). We average the time saved across test teachers and divide CrossGMN's training cost by this average to obtain the break-even number of compressions. If this number is below the zoo size, CrossGMN provides net GPU-time savings.
The table reports training cost, mean time saved per teacher, the break-even count, and net savings at the listed zoo size. Targets are within 10 dB of teacher PSNR (INRs) or at least 95% of teacher accuracy (classifiers). Results are aggregated over three seeds as mean ± SD where applicable.
| Setting | Training GPU-h | Saving sec/teacher | Break-even teachers | Zoo size | Net zoo savings (GPU-h) |
|---|---|---|---|---|---|
| MNIST INR width, [2,32,32,1]→[2,4,4,1] | 3.1 | 0.70 ± 0.00 | 15,962 ± 67 | 70,000 | +10.5 ± 0.1 |
| MNIST INR depth, [2,32,32,1]→[2,32,1] | 2.6 | 0.81 ± 0.05 | 11,679 ± 668 | 70,000 | +13.1 ± 0.9 |
| CIFAR-10 MLP width, [3072,64,64,64,64,10]→[3072,32,32,32,32,10] | 80.3 | 1.18 ± 0.10 | 245,191 ± 21,357 | 6,100 | −78.3 ± 0.2 |
| CIFAR-10 ViT depth, blocks 4→2 | 5.2 | 3.77 ± 0.18 | 4,934 ± 239 | 6,100 | +1.2 ± 0.3 |
The results demonstrate that, in several compression settings, the total computational savings from faster compression exceed the cost of training CrossGMN, yielding net savings of up to 13.1 GPU-h. CrossGMN achieves positive average per-teacher savings in all four settings. Despite faster compression on average, the CIFAR-10 MLP setting does not break even due to its higher training cost (80.3 GPU-h). Notably, our training configurations were not optimized to minimize training GPU time, leaving room for further improvements. If requested, we would include the full break-even analysis, in the manuscript appendix.
W3, W4: Modest classifier gains and independent runs.
Even a 5% acceleration is practically meaningful, particularly at scale, where small relative gains can translate into substantial computational savings. We therefore view these positive results as a promising first step. The three-seed results below show that the positive classifier gains are consistent: their geometric SD is at most 1.05×.
An important factor that helps explain the difference between classifiers and INRs is the strength of their pruning-derived anchors. Their task quality also varies across settings: at the distillation budget B, classifier anchors approach teacher accuracy (MLP: 47.4–50.8% vs. 53.1%; ViT: 71–73% vs. 73.3%; CNN: 95.2% vs. 96.0%), whereas MNIST INR anchors achieve 16.2–21.5 dB PSNR versus 25.5 dB for the teacher. The latter gap corresponds to approximately 2.5–8.5× higher reconstruction MSE, leaving more room to improve the initialization.
We repeated the compression experiments across three seeds. The table below reports representative settings; classifier speedups vary little across seeds. Speedups below are the geometric mean of the per-run median speedups, with geometric SD shown as ×/÷. Acceleration and failure rates are mean ± SD across three runs.
| Setting | Speedup | >1x (%) | Failed (%) |
|---|---|---|---|
| MNIST INR, [2,32,32,1]→[2,8,8,1] | 5.51 ×/÷ 1.30 | 78.5 ± 5.0 | 16.3 ± 4.1 |
| MNIST INR, [2,32,32,1]→[2,16,1] | 7.95 ×/÷ 1.18 | 85.5 ± 1.5 | 12.9 ± 1.9 |
| ModelNet40 INR, [3,48,48,48,48,1]→[3,48,48,1] | 6.82 ×/÷ 1.75 | 75 ± 10.1 | 12.6 ± 5.2 |
| CIFAR-10 MLP, [3072,64,64,64,64,10]→[3072,32,32,32,32,10] | 1.06 ×/÷ 1.01 | 54.6 ± 1.0 | 4.9 ± 0.6 |
| CIFAR-10 MLP, [3072,64,64,64,64,10]→[3072,64,64,10] | 1.29 ×/÷ 1.05 | 62.6 ± 2.5 | 2.1 ± 0.4 |
| CIFAR-10 ViT, blocks 4→2 | 1.20 ×/÷ 1.01 | 94.7 ± 0.4 | 0.0 ± 0.0 |
| CIFAR-10 ViT, heads 2→1 | 0.88 ×/÷ 1.02 | 20.1 ± 4.2 | 0.3 ± 0.1 |
We will incorporate the complete seed results and these clarifications into the revised manuscript.
Authors’ Response to Reviewer KCk4 - Part 3
W7: Experimental coverage across architectures. CrossGMN is evaluated on MLPs, CNNs, and ViTs across multiple tasks and compression settings, offering broader architectural coverage than is typical in weight-space learning papers. For example, DWS (Navon et al. (2023)) and NNiT (Kim et al. (2026)) primarily evaluate MLPs, while Transformer-NFN (Tran et al. (2025)) focuses on Transformers. Although expanding our CNN and ViT evaluations would strengthen the evidence, the current results demonstrate applicability across three prominent architecture families.
Q1: Absolute task quality. We agree that absolute task-quality metrics provide important complementary evidence. As requested, we report test accuracy for CIFAR-10 classifiers and ground-truth image PSNR for MNIST INRs.
The table below compares task quality before distillation (step 0) and at the setting-specific distillation budget B, determined by anchor convergence (Appendix D.2). A and C denote the anchor and CrossGMN initializations. Where applicable, values are reported as mean ± SD across three seeds. CrossGMN improves initial task quality, including a 7.7 dB PSNR gain on MNIST INRs and a 12.4 percentage-point accuracy gain on CIFAR-10 MLPs. After distillation, the gaps are smaller and classifier accuracies approach those of the teachers. These results demonstrate that CrossGMN provides a substantially better initialization, with faster convergence as the primary benefit.
| Setting and metric | Teacher | A at 0 | C at 0 | A at B | C at B |
|---|---|---|---|---|---|
| MNIST INR, [2,32,32,1]→[2,8,8,1], PSNR dB | 25.5 | 6.0 | 13.7 ± 0.4 | 21.5 | 22.8 ± 0.2 |
| CIFAR-10 MLP, [3072,64,64,64,64,10]→[3072,32,32,32,32,10], accuracy % | 53.1 | 18.5 | 30.9 ± 0.6 | 49.0 | 49.3 ± 0.1 |
| CIFAR-10 ViT, blocks 4→2, accuracy % | 73.3 | 24.1 | 28.3 ± 0.6 | 71.4 | 71.7 ± 0.0 |
| CIFAR-10 ViT, attention dimension 48→24, accuracy % | 73.3 | 45.6 | 52.2 ± 0.2 | 73.0 | 73.0 ± 0.0 |
The full table of absolute task quality results is available here.
Authors’ Response to Reviewer KCk4 - Part 1
We thank the reviewer for recognizing the principled, symmetry-aware formulation of CrossGMN. We also appreciate their acknowledgment of the theoretical motivation, as well as their assessment of our evaluation as careful and honest. We address their comments below.
W1: Scalability. Our evaluation reaches models with 210k parameters, substantially larger than those in several prominent weight-space benchmarks, including the CNN Zoo (~5k parameters) used in NFN (Zhou et al. (2023)) and the diverse-architecture model zoo (up to ~88k) evaluated in GMN (Lim et al. (2023)).
We agree that each layerwise join introduces dsrc×dtgt cross edges to preserve equivariance. However, this overhead remains modest at our evaluated scales. For our largest CIFAR-10 MLP teachers (~210k parameters), cross edges account for only 7–9% of the joint graph's edges. For CNNs, the k2 parallel edges between channels represent the original convolutional weights, whereas cross-network connections introduce only one edge per joined channel pair, independent of kernel size. For Transformers, we follow the parameter-graph construction of the original GMN work (Lim et al. (2023)), where each residual-stream, attention and FFN coordinate is one node. Their number scales linearly with width, although the number of parameter edges scales quadratically. Our ViT teacher contains 1,447 nodes and 115,930 parameters.
Crucially, CrossGMN is applied only once per compression task, making its overhead small relative to distillation. Across our evaluated settings, the forward pass costs the equivalent of just 7–134 distillation steps, compared with distillation budgets of 1,750–9,900 steps, amounting to only 0.13–4.06% of distillation runtime. Thus, the one-time inference overhead remains modest across our evaluated architectures, despite the additional graph connectivity.
W2, Q6: Intended use case, training cost, and break-even.
Intended use case. We agree that compressing a single large model is an important practical setting. However, CrossGMN targets recurring compression across model collections. For example:
Personalized on-device models: Applications such as speech recognition may fine-tune a large model for individual users, requiring each personalized model to be compressed for deployment on resource-constrained mobile devices.
Distributed RF sensor networks: A large model for signal monitoring and emitter identification may be fine-tuned for each sensor based on its local signal environment, requiring each adapted model to be compressed for deployment on resource-constrained sensors. As new data arrives, the models may be updated and require repeated compression.
Growing model repositories: Platforms such as Hugging Face continuously receive independently trained models, which may need to be compressed for deployment on edge devices.
Related work on amortized distillation across post-trained LLMs (Zhou et al. (2026)) further motivates this recurring-compression setting.
In these settings, CrossGMN could learn a reusable compression operator across a model population, accelerating compression of new or updated models. Our heterogeneous-MLP experiments demonstrate that source architectures need not be identical, although extending this capability to broader model families remains future work.