acceptodds
Under review as a conference paper at ICLR 2027

CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformations

36 views

Abstract

Weight-space networks operate directly on parameters of other neural networks, enabling tasks such as predicting model properties, editing trained models, and generating weights. Equivariance has emerged as a key design principle for these networks, owing to weight-space symmetries such as neuron permutations. However, existing equivariant weight-space architectures have primarily been studied for transformations that preserve the network architecture. In contrast, many practical transformations, including model compression and upscaling, map a trained source network into a target network with a different architecture. In this setting, the source and target permutation symmetries act on different parameter spaces, making equivariance less straightforward to formulate. Our key idea for addressing this mismatch is to reformulate cross-architecture operators with two inputs: a trained source network and an initialization of the target network. This lets us define equivariant cross-architecture operators that refine the initialization of the target network using information from the source network, while being invariant to source-network permutations and equivariant to target-network permutations. Based on this formulation, we introduce CrossGMN, a graph metanetwork that jointly processes both networks through symmetry-preserving cross-network message passing. We prove CrossGMN is universal for continuous cross-architecture operators on compact sets under a general-position assumption. We further evaluate CrossGMN for model compression, where it predicts the parameters of a smaller network to accelerate subsequent knowledge distillation. Across 2-D and 3-D INRs and image classification with MLPs, CNNs, and Vision Transformers, CrossGMN speeds up distillation by up to 8.89×, transfers across datasets without retraining (3.78×), and a single model can accelerate compression from heterogeneous source architectures into a common target architecture.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
weight-space learning, equivariant neural networks, graph metanetworks, model compression, graph neural networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KCk4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper CrossGMN looks at learning transformations between weight spaces of networks with different widths and depths. The work builds on graph metanetworks (Lim et al., 2024), which represent neural networks as permutation-symmetry preserving graphs, and extends them to weight-space learning across different architectures. Instead of mapping the source weights directly to target weights, the authors define an operator that takes a trained source network and an initialization of the target network and refines the initialization. This makes the symmetry well defined as the operator is invariant to source permutations and equivariant to target permutations. CrossGMN joins the two parameter graphs with cross edges and runs a graph meta network over the joint graph, and they prove it is universal for such operators for MLPs. They test it on model compression, where CrossGMN refines a structured pruning initialization before knowledge distillation. They show speedups for using this work for different architectures like MLPs, CNNs, ViTs. They also show transfer from MNIST to FashionMNIST INRs and one operator trained on several MLP source architectures.

Critical strengths
  1. Authors address the problem of having permutation invariant representation of neural networks in a mathematically empirically sound approach. The work shows a principled way of representation neural networks as permutation invariant graphs for cross architecture weight space transformation.
  2. Their formulation of these networks is the main contribution. Taking the target initialization as a second input solves the problem that source and target have different symmetry groups. It is well motivated by showing that a direct source to optimal target map can be discontinuous, while a fixed number of distillation steps is a continuous operator of this form.
  3. The evaluation is careful and honest. They report per-teacher speedups with IQR and failure rates, wall-clock time including the forward pass, and reach-budget plots against several pruning baselines.
Critical weaknesses
  1. Models are small, up to about 210k parameters, and the architecture scales badly. Each layerwise join has d_src × d_tgt edges. The strided policy reduces the number of layer pairs but not this product, CNNs add k² parallel edges per channel pair, and for transformers every attention and residual coordinate is a node. So the scaling problem comes from the design, not just from compute budget.
  2. The method needs thousands of trained teachers of the same architecture. Their own ablation shows that with 1,000 teachers the speedup drops to 1.67×, and with 100 it is 0× with 51% failures. Practical compression usually targets a single large model. The wall-clock speedup also does not include the cost of training the operator, so it is not clear when this pays off.
  3. Classifier compression ratios are modest (about 1.04–2.4×) and so are the gains. The CNN result is a single width setting on a 2.4k-parameter model, and two of the four ViT settings are slower than the anchor (0.74× and 0.86×).
  4. All results are single runs without seeds so no error bars have been reported.
  5. It is not clear what the operator learns. The output is anchor + s·Δθ, results depend heavily on a good pruning initialization and there is no analysis of Δθ. The cross edges carry no correspondence information. All cross edges in a join have the same features, and the value is 0 for CNNs and ViTs. Any neuron alignment has to be learned through the messages. With pruning anchors the kept student neurons are exact copies of teacher neurons, so the gains may come from that rather than a real cross-architecture mapping. 6."Cross-architecture" here means different widths and depths within one model family. All experiments are within-family (MLP->MLP, CNN->CNN, ViT->ViT), while works cited as motivation, such as Liu et al. 2024, use the term for different model families. Cross-family transfer is listed as future work. Source and target must also share input and output dimensions.
  6. The theory covers MLPs only, and the universality proof is for plain GMN with position wise boundary joins. The experiments use ScaleGMN, and CNNs and ViTs use dense boundary joins. The formal results also cover only permutation symmetry, even though the backbone is scale or sign equivariant.
  7. Very few experimentation results on CNNs and ViTs.
  8. The work cites and talks about old weight space generation methods, there is no mention of latest flow matching , diffusion etc. based weight space learning methods and no comparison shown against any such methods.
Questions for authors
  1. Please report absolute test accuracy (classifiers) and PSNR to ground truth (INRs) for the teacher, the anchor and the CrossGMN student, both at step 0 (directly after the forward pass) and at the distillation budget. This would show whether CrossGMN produces useful students on its own, and whether faster convergence also gives equal or better final task performance, including for the teachers counted as failures.
  2. How large is the update s·Δθ relative to the anchor? How does one CrossGMN pass compare to taking k distillation steps from the anchor at equal compute? The forward pass takes about 18–20 ms, roughly 8–10 distillation steps. Does Δθ resemble a few distillation steps or something qualitatively different?
  3. The cross edges have no correspondence information, so how does the operator transfer information from specific teacher neurons to specific student neurons? Does performance hold up if the anchor is perturbed or not a direct copy of teacher neurons? In the random initialization ablation, what is the speedup measured against?
  4. Recent work (e.g. DeepWeightFlow, ICLR 2026) finds that non-equivariant models with canonicalization perform well on weight-space tasks at moderate scale. What is the authors’ take on its findings on permutation symmetry? Can they compare CrossGMN against a non-equivariant operator on re-basined teacher and anchor weights, to show that equivariance is what drives the gains?
  5. Nonlinearities enter the parameter graph only as optional node labels, and normalization parameters are auxiliary nodes with no cross edges, so they only get teacher information indirectly through the residual nodes. Could this explain the weaker ViT results? The depth anchors (layer fold) also remove intermediate nonlinearities. How does CrossGMN compensate for this, and is there any analysis of what it learns there? How can the network handle diverse normalization schemes and non linear activation functions?
  6. How do scale, sign and normalization symmetries, and teacher and student pairs with different activations, fit into the framework? What is the intended use case, given that the method needs thousands of teachers of the same architecture? How much does training the operator cost, and how many teachers need to be compressed before it pays off? Is it feasible for realistic model sizes such as ResNet-18 or a small language model?
  7. Please either clarify the scope in the title and abstract (e.g. “cross-width/depth” or “within-family”), or include at least one cross-family experiment such as MLP→CNN or CNN→ViT. How would cross edges be defined between different layer types?
12 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer KCk4 - Part 1

We thank the reviewer for recognizing the principled, symmetry-aware formulation of CrossGMN. We also appreciate their acknowledgment of the theoretical motivation, as well as their assessment of our evaluation as careful and honest. We address their comments below.

W1: Scalability. Our evaluation reaches models with 210k parameters, substantially larger than those in several prominent weight-space benchmarks, including the CNN Zoo (~5k parameters) used in NFN (Zhou et al. (2023)) and the diverse-architecture model zoo (up to ~88k) evaluated in GMN (Lim et al. (2023)).

We agree that each layerwise join introduces dsrc×dtgtd_{\mathrm{src}}\times d_{\mathrm{tgt}} cross edges to preserve equivariance. However, this overhead remains modest at our evaluated scales. For our largest CIFAR-10 MLP teachers (~210k parameters), cross edges account for only 7–9% of the joint graph's edges. For CNNs, the k2k^2 parallel edges between channels represent the original convolutional weights, whereas cross-network connections introduce only one edge per joined channel pair, independent of kernel size. For Transformers, we follow the parameter-graph construction of the original GMN work (Lim et al. (2023)), where each residual-stream, attention and FFN coordinate is one node. Their number scales linearly with width, although the number of parameter edges scales quadratically. Our ViT teacher contains 1,447 nodes and 115,930 parameters.

Crucially, CrossGMN is applied only once per compression task, making its overhead small relative to distillation. Across our evaluated settings, the forward pass costs the equivalent of just 7–134 distillation steps, compared with distillation budgets of 1,750–9,900 steps, amounting to only 0.13–4.06% of distillation runtime. Thus, the one-time inference overhead remains modest across our evaluated architectures, despite the additional graph connectivity.

W2, Q6: Intended use case, training cost, and break-even.

Intended use case. We agree that compressing a single large model is an important practical setting. However, CrossGMN targets recurring compression across model collections. For example:

  • Personalized on-device models: Applications such as speech recognition may fine-tune a large model for individual users, requiring each personalized model to be compressed for deployment on resource-constrained mobile devices.

  • Distributed RF sensor networks: A large model for signal monitoring and emitter identification may be fine-tuned for each sensor based on its local signal environment, requiring each adapted model to be compressed for deployment on resource-constrained sensors. As new data arrives, the models may be updated and require repeated compression.

  • Growing model repositories: Platforms such as Hugging Face continuously receive independently trained models, which may need to be compressed for deployment on edge devices.

  • Related work on amortized distillation across post-trained LLMs (Zhou et al. (2026)) further motivates this recurring-compression setting.

In these settings, CrossGMN could learn a reusable compression operator across a model population, accelerating compression of new or updated models. Our heterogeneous-MLP experiments demonstrate that source architectures need not be identical, although extending this capability to broader model families remains future work.

AuthorsOpenReview ↗

Authors’ Response to Reviewer KCk4 - Part 2

Training cost and break-even. We conduct a break-even analysis to determine how many model compressions are needed to pay off the training cost.

For each teacher, we compare the GPU time required by CrossGMN (including its forward pass and distillation) and the pruning-derived anchor to reach a teacher-relative performance target, independent of the anchor. Both methods use the same distillation budget (B) (Sec. 5.1). We average the time saved across test teachers and divide CrossGMN's training cost by this average to obtain the break-even number of compressions. If this number is below the zoo size, CrossGMN provides net GPU-time savings.

The table reports training cost, mean time saved per teacher, the break-even count, and net savings at the listed zoo size. Targets are within 10 dB of teacher PSNR (INRs) or at least 95% of teacher accuracy (classifiers). Results are aggregated over three seeds as mean ± SD where applicable.

SettingTraining GPU-hSaving sec/teacherBreak-even teachersZoo sizeNet zoo savings (GPU-h)
MNIST INR width, [2,32,32,1]→[2,4,4,1]3.10.70 ± 0.0015,962 ± 6770,000+10.5 ± 0.1
MNIST INR depth, [2,32,32,1]→[2,32,1]2.60.81 ± 0.0511,679 ± 66870,000+13.1 ± 0.9
CIFAR-10 MLP width, [3072,64,64,64,64,10]→[3072,32,32,32,32,10]80.31.18 ± 0.10245,191 ± 21,3576,100−78.3 ± 0.2
CIFAR-10 ViT depth, blocks 4→25.23.77 ± 0.184,934 ± 2396,100+1.2 ± 0.3

The results demonstrate that, in several compression settings, the total computational savings from faster compression exceed the cost of training CrossGMN, yielding net savings of up to 13.1 GPU-h. CrossGMN achieves positive average per-teacher savings in all four settings. Despite faster compression on average, the CIFAR-10 MLP setting does not break even due to its higher training cost (80.3 GPU-h). Notably, our training configurations were not optimized to minimize training GPU time, leaving room for further improvements. If requested, we would include the full break-even analysis, in the manuscript appendix.

W3, W4: Modest classifier gains and independent runs.

Even a 5% acceleration is practically meaningful, particularly at scale, where small relative gains can translate into substantial computational savings. We therefore view these positive results as a promising first step. The three-seed results below show that the positive classifier gains are consistent: their geometric SD is at most 1.05×.

An important factor that helps explain the difference between classifiers and INRs is the strength of their pruning-derived anchors. Their task quality also varies across settings: at the distillation budget B, classifier anchors approach teacher accuracy (MLP: 47.4–50.8% vs. 53.1%; ViT: 71–73% vs. 73.3%; CNN: 95.2% vs. 96.0%), whereas MNIST INR anchors achieve 16.2–21.5 dB PSNR versus 25.5 dB for the teacher. The latter gap corresponds to approximately 2.5–8.5× higher reconstruction MSE, leaving more room to improve the initialization.

We repeated the compression experiments across three seeds. The table below reports representative settings; classifier speedups vary little across seeds. Speedups below are the geometric mean of the per-run median speedups, with geometric SD shown as ×/÷. Acceleration and failure rates are mean ± SD across three runs.

SettingSpeedup>1x (%)Failed (%)
MNIST INR, [2,32,32,1]→[2,8,8,1]5.51 ×/÷ 1.3078.5 ± 5.016.3 ± 4.1
MNIST INR, [2,32,32,1]→[2,16,1]7.95 ×/÷ 1.1885.5 ± 1.512.9 ± 1.9
ModelNet40 INR, [3,48,48,48,48,1]→[3,48,48,1]6.82 ×/÷ 1.7575 ± 10.112.6 ± 5.2
CIFAR-10 MLP, [3072,64,64,64,64,10]→[3072,32,32,32,32,10]1.06 ×/÷ 1.0154.6 ± 1.04.9 ± 0.6
CIFAR-10 MLP, [3072,64,64,64,64,10]→[3072,64,64,10]1.29 ×/÷ 1.0562.6 ± 2.52.1 ± 0.4
CIFAR-10 ViT, blocks 4→21.20 ×/÷ 1.0194.7 ± 0.40.0 ± 0.0
CIFAR-10 ViT, heads 2→10.88 ×/÷ 1.0220.1 ± 4.20.3 ± 0.1

We will incorporate the complete seed results and these clarifications into the revised manuscript.

AuthorsOpenReview ↗

Authors’ Response to Reviewer KCk4 - Part 3

W7: Experimental coverage across architectures. CrossGMN is evaluated on MLPs, CNNs, and ViTs across multiple tasks and compression settings, offering broader architectural coverage than is typical in weight-space learning papers. For example, DWS (Navon et al. (2023)) and NNiT (Kim et al. (2026)) primarily evaluate MLPs, while Transformer-NFN (Tran et al. (2025)) focuses on Transformers. Although expanding our CNN and ViT evaluations would strengthen the evidence, the current results demonstrate applicability across three prominent architecture families.

Q1: Absolute task quality. We agree that absolute task-quality metrics provide important complementary evidence. As requested, we report test accuracy for CIFAR-10 classifiers and ground-truth image PSNR for MNIST INRs.

The table below compares task quality before distillation (step 0) and at the setting-specific distillation budget B, determined by anchor convergence (Appendix D.2). A and C denote the anchor and CrossGMN initializations. Where applicable, values are reported as mean ± SD across three seeds. CrossGMN improves initial task quality, including a 7.7 dB PSNR gain on MNIST INRs and a 12.4 percentage-point accuracy gain on CIFAR-10 MLPs. After distillation, the gaps are smaller and classifier accuracies approach those of the teachers. These results demonstrate that CrossGMN provides a substantially better initialization, with faster convergence as the primary benefit.

Setting and metricTeacherA at 0C at 0A at BC at B
MNIST INR, [2,32,32,1]→[2,8,8,1], PSNR dB25.56.013.7 ± 0.421.522.8 ± 0.2
CIFAR-10 MLP, [3072,64,64,64,64,10]→[3072,32,32,32,32,10], accuracy %53.118.530.9 ± 0.649.049.3 ± 0.1
CIFAR-10 ViT, blocks 4→2, accuracy %73.324.128.3 ± 0.671.471.7 ± 0.0
CIFAR-10 ViT, attention dimension 48→24, accuracy %73.345.652.2 ± 0.273.073.0 ± 0.0

The full table of absolute task quality results is available here.

Reviewer_H2khOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to learn transformations between neural networks with different architectures while respecting the permutation symmetries of their respective parameter spaces. Its central idea is to formulate the transformation as a two-input operator that receives both a trained source network and an initialization of the target network, thereby requiring invariance to source-network permutations and equivariance to target-network permutations. Based on this formulation, the authors introduce CrossGMN, which represents the source and target networks as parameter graphs, connects them through symmetry-preserving cross edges, and uses graph message passing to predict a residual update to the target initialization. The paper establishes a universality result for continuous cross-architecture operators on compact sets of weights in general position, with a corresponding extension to finite collections of source architectures sharing a common target architecture. The method is evaluated on width, depth, and combined compression using 2-D SIREN implicit neural representations (INRs) of MNIST images, 3-D INRs of ModelNet40 shapes, CIFAR-10 MLPs and Vision Transformers, and MNIST CNNs; CrossGMN is trained to match teacher outputs and its predictions are subsequently refined through knowledge distillation. The authors report median distillation-step speedups of up to 8.89× on MNIST INRs and 6.85× on ModelNet40 INRs, along with transfer to FashionMNIST and compression across heterogeneous MLP teacher architectures. However, the gains are considerably less consistent for image classifiers, and some Vision Transformer compression settings become slower rather than faster when the cost of CrossGMN inference is included.

Critical strengths

The paper studies how to transfer knowledge from a trained network to a target network with a different architecture. Its key idea is to use both the source network and the target initialization as inputs, so the method can respect the different permutation symmetries of the two networks. CrossGMN implements this idea through a joint graph with cross-network message passing. This gives the method a clear motivation and a sound design.

The authors prove that CrossGMN can approximate a broad class of continuous cross-architecture transformations under specific assumptions, including a general-position condition on the network weights. They also extend the result to multiple source architectures that share a common target architecture. This provides a theoretical basis for the method beyond its empirical results, although the guarantees have a limited scope.

The method is tested on 2D and 3D implicit neural representations (INRs), as well as MLPs, CNNs, and Vision Transformers. The reported speedup reaches 8.89× on MNIST INRs and 6.85× on ModelNet40 INRs. A model trained on MNIST INRs also transfers to FashionMNIST without retraining, achieving up to 3.78× speedup. These results show that a learned weight-space model can provide useful initializations for faster knowledge distillation in some settings.

Critical weaknesses

(W1) Performance is uneven across tasks. Although CrossGMN performs well on several INR tasks, the gains are much smaller for some classification tasks. For example, the reported speedup is 1.04× for CIFAR-10 MLP width reduction, while some Vision Transformer settings become slower after accounting for CrossGMN inference time. This limits the claim that the method is broadly useful for accelerating compression. More analysis of why it works better for some architectures than others would strengthen the paper.

(W2) Limited comparison with closely related methods. The experiments mainly compare CrossGMN with pruning-based initializations followed by distillation. However, the paper does not directly compare against closely related learned methods for weight transfer or model editing, including the cited Meta Pruning via Graph Metanetworks. As a result, it remains unclear how much of the improvement comes from CrossGMN's symmetry-aware design rather than from learning a better initialization in general.

(W3) Generalization is promising but not yet strong enough to support broad claims. The transfer experiments cover MNIST to FashionMNIST and a limited set of unseen source architectures. In the pooled unseen-architecture setting, the median speedup is only 1.06×, and just 51% of test teachers achieve a speedup greater than 1×. Moreover, these experiments retain a common target architecture. The results therefore provide some evidence of transfer, but do not show that CrossGMN works reliably across arbitrary architectures or model families.

(W4) The practical cost and benefit need clearer analysis. CrossGMN adds the cost of training and running a separate metanetwork. Although the paper accounts for inference time in some wall-clock measurements, it does not fully establish how many compression tasks are needed to recover the training cost. This matters especially when the measured gains are small or negative. A break-even analysis would help readers decide when the method is actually useful.

(W5) Some experimental and theoretical details need clarification. The paper should explain more clearly how the pruning baseline is selected and confirm that held-out test results do not influence this choice. It would also benefit from results across independent training runs to establish the reliability of small gains. Finally, the main universality theorem focuses on MLPs under specific assumptions; the paper should distinguish this guarantee from the broader experiments on CNNs and Vision Transformers.

Questions for authors

(Q1) How does CrossGMN compare with recent learned weight-transfer and model-editing methods? Could the authors provide a concise comparison with WARM (ICASSP 2026) and clarify the relationship to Meta Pruning via Graph Metanetworks (Liu et al., 2025)? A small experiment on one shared architecture family would be sufficient; a large new benchmark is not necessary. If a direct comparison is infeasible, a precise explanation of the incompatible assumptions and objectives would still help.

(Q2) How exactly is the pruning anchor selected? The paper states that the anchor is the structured-pruning method with the lowest initialization loss among the evaluated methods. Please clarify whether this choice is made using training/validation teachers or held-out evaluation teachers, and whether the selected method is fixed for an entire setting or selected separately for each teacher.

(Q3) Which theoretical guarantees apply to the actual experimental architectures? The main universality theorem is stated for fixed MLP source and target architectures under a general-position assumption. The experiments use additional symmetry-aware backbones and also include CNNs and Transformers. Could the authors clarify which guarantees hold for these instantiated models, and which are limited to the plain-GMN MLP setting? It would also help to state explicitly that the heterogeneous and unseen-architecture experiments retain a common target architecture.

(Q4) Are the small classifier gains robust across operator-training seeds? Could the authors report paired uncertainty estimates for one representative INR setting and one representative classifier setting, ideally including at least a small number of independent CrossGMN training runs?

5 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer H2kh - Part 1

We thank the reviewer for recognizing the motivation and theoretical foundations of our symmetry formulation, as well as the promising INR and transfer results demonstrating that learned weight-space models can provide effective initializations that accelerate knowledge distillation. We address their comments below.

W2, Q1: Learned baselines and the contribution of symmetry-aware modeling.

We thank the reviewer for this suggestion. To distinguish the benefits of learning an initialization from those of symmetry-aware modeling, we added three baselines: MetaPrune-GMN (Liu et al. (2025)), WARM (Gong & Liu (2026)), and a non-equivariant MLP.

  • MetaPrune-GMN. We use the released code to transform each teacher into a more prunable network using task loss and sparsity regularization, then prune by group-norm importance and distill under the same protocol as CrossGMN. We match operator size, training compute, and seeds, and select hyperparameters and checkpoints on validation teachers. For width+depth compression, we add the same layer folding used by our anchor.

  • WARM. Although WARM reports pruning experiments, its construction is formulated for widening, without specifying a corresponding compression procedure. We retain WARM’s layer alignment and compare four compression adaptations: permutation-based truncation, Hadamard projection, and Hadamard projection with either independent per-tensor permutations or fan-in rescaling. We select the best adaptation based on validation teachers.

  • Non-equivariant MLP. To more directly assess whether the gains arise from learning an initialization alone, we trained an MLP on re-basined teacher and anchor weights, using canonicalization. We closely match its parameter count to CrossGMN (2.648M vs. 2.647M).

The table below reports MNIST INR speedups relative to the same pruning anchor, evaluated on 1,000 test teachers with matched distillation budgets. Values are the geometric mean ×/÷ geometric SD of per-run median speedups across the same three seeds.

Teacher → StudentCrossGMNMetaPrune-GMNWARM (adapted)Non-equivariant MLP
[2,32,32,1] → [2,8,8,1]5.51 ×/÷ 1.300.71 ×/÷ 1.160.37 ×/÷ 1.082.88 ×/÷ 1.11
[2,32,32,1] → [2,16,1]7.95 ×/÷ 1.182.19 ×/÷ 1.020.00x5.01 ×/÷ 1.13

Overall, CrossGMN substantially outperforms the tested baselines under matched evaluation conditions. Crucially, its advantage over the canonicalized non-equivariant MLP indicates that learning an initialization alone does not account for the full observed gain. Together with the MetaPrune and WARM comparisons, these results support the benefit of CrossGMN's symmetry-aware design.

W1: Uneven performance across tasks.

Smaller classifier gains. Even a 5% acceleration is practically meaningful, particularly at scale, where small relative gains can translate into substantial computational savings. We therefore view these positive results as a promising first step. The three-seed results below show that the positive classifier gains are consistent: their geometric SD is at most 1.05×.

Why performance differs across tasks and architectures.

  • An important factor that helps explain the difference between classifiers and INRs is the strength of their pruning-derived anchors. Their task quality also varies across settings: at the distillation budget B, classifier anchors approach teacher accuracy (MLP: 47.4–50.8% vs. 53.1%; ViT: 71–73% vs. 73.3%; CNN: 95.2% vs. 96.0%), whereas MNIST INR anchors achieve 16.2–21.5 dB PSNR versus 25.5 dB for the teacher. The latter gap corresponds to approximately 2.5–8.5× higher reconstruction MSE, leaving more room to improve the initialization.

  • Weight-space learning is considerably more mature for MLPs than for architectures such as ViTs, with more established methods and simpler symmetry structures. In particular, existing ViT weight-space representations do not fully capture richer symmetries, including attention-head permutations and general linear (GL) reparameterizations. We hypothesize that more complete symmetry modeling, together with improved optimization and hyperparameter tuning, could strengthen performance on these architectures. We therefore view the current classifier results as an initial demonstration rather than a performance ceiling, while explicitly acknowledging the modest gains and unsuccessful settings.

W5, Q2: Anchor selection clarification. We agree that the selection protocol should be more explicit. We select one pruning method per compression setting using initialization loss on validation teachers, then keep it fixed for test teachers. We do not select the best method separately for each test teacher.

AuthorsOpenReview ↗

Authors’ Response to Reviewer H2kh - Part 2

W5, Q4: Independent training runs. We repeated the compression experiments across three seeds. The table below reports representative settings; classifier speedups vary little across seeds. Speedups below are the geometric mean of the per-run median speedups, with geometric SD shown as ×/÷. Acceleration and failure rates are mean ± SD across three runs.

SettingSpeedup>1x (%)Failed (%)
MNIST INR, [2,32,32,1]→[2,8,8,1]5.51 ×/÷ 1.3078.5 ± 5.016.3 ± 4.1
MNIST INR, [2,32,32,1]→[2,16,1]7.95 ×/÷ 1.1885.5 ± 1.512.9 ± 1.9
ModelNet40 INR, [3,48,48,48,48,1]→[3,48,48,1]6.82 ×/÷ 1.7575 ± 10.112.6 ± 5.2
CIFAR-10 MLP, [3072,64,64,64,64,10]→[3072,32,32,32,32,10]1.06 ×/÷ 1.0154.6 ± 1.04.9 ± 0.6
CIFAR-10 MLP, [3072,64,64,64,64,10]→[3072,64,64,10]1.29 ×/÷ 1.0562.6 ± 2.52.1 ± 0.4
CIFAR-10 ViT, blocks 4→21.20 ×/÷ 1.0194.7 ± 0.40.0 ± 0.0
CIFAR-10 ViT, heads 2→10.88 ×/÷ 1.0220.1 ± 4.20.3 ± 0.1

We will incorporate the complete seed results and these clarifications into the revised manuscript.

W3: Generalization to unseen architectures. To strengthen the evidence for generalization to unseen architectures, we evaluated the same trained heterogeneous operator, without retraining, on four additional unseen CIFAR-10 MLP architectures spanning different widths and depths, as presented in the table below. Hidden widths specify the source architecture between input dimension 3072 and output dimension 10; the target architecture is fixed. Speedup is the geometric mean of per-run median speedups across three seeds, with geometric SD shown as ×/÷.

Unseen source hidden widthsSpeedup (×/÷ geometric SD)
[64,64]1.49 ×/÷ 1.03
[32,32,32,32]1.13 ×/÷ 1.04
[128]1.11 ×/÷ 1.09
[48,48]1.60 ×/÷ 1.09
All four (pooled)1.31 ×/÷ 1.02

All four unseen architectures achieve median acceleration, with a pooled speedup of 1.31×, compared to 1.06× in the original evaluation (Table 5). Moreover, 59.4 ± 1.3% of teachers accelerate, compared to 51% in the original evaluation (Table 5).

These results provide further evidence that one trained CrossGMN operator can transfer across unseen source widths and depths. The target architecture is fixed in these experiments; extending generalization to new target architectures is an interesting direction for future work. We will clarify this distinction accordingly.

AuthorsOpenReview ↗

Authors’ Response to Reviewer H2kh - Part 3 (Final)

W4: Training cost and break-even. We agree that CrossGMN's practical benefit must account for its training cost. We therefore conduct a break-even analysis to determine how many model compressions are needed to recover CrossGMN's training cost.

For each teacher, we compare the GPU time required by CrossGMN (including its forward pass and distillation) and the pruning-derived anchor to reach a teacher-relative performance target, independent of the anchor. Both methods use the same distillation budget (B) (Sec. 5.1). We average the time saved across test teachers and divide CrossGMN's training cost by this average to obtain the break-even number of compressions. If this number is below the zoo size, CrossGMN provides net GPU-time savings.

The table reports training cost, mean time saved per teacher, the break-even count, and net savings at the listed zoo size. Targets are within 10 dB of teacher PSNR (INRs) or at least 95% of teacher accuracy (classifiers). Results are aggregated over three seeds as mean ± SD where applicable.

SettingTraining GPU-hSaving sec/teacherBreak-even teachersZoo sizeNet zoo savings (GPU-h)
MNIST INR width, [2,32,32,1]→[2,4,4,1]3.10.70 ± 0.0015,962 ± 6770,000+10.5 ± 0.1
MNIST INR depth, [2,32,32,1]→[2,32,1]2.60.81 ± 0.0511,679 ± 66870,000+13.1 ± 0.9
CIFAR-10 MLP width, [3072,64,64,64,64,10]→[3072,32,32,32,32,10]80.31.18 ± 0.10245,191 ± 21,3576,100−78.3 ± 0.2
CIFAR-10 ViT depth, blocks 4→25.23.77 ± 0.184,934 ± 2396,100+1.2 ± 0.3

The results demonstrate that, in several compression settings, the total computational savings from faster compression exceed the cost of training CrossGMN, yielding net savings of up to 13.1 GPU-h. CrossGMN achieves positive average per-teacher savings in all four settings. Despite faster compression on average, the CIFAR-10 MLP setting does not break even due to its higher training cost (80.3 GPU-h). Notably, our training configurations were not optimized to minimize training GPU time, leaving room for further improvements. If requested, we would include the full break-even analysis, in the manuscript appendix.

W5, Q3: Theoretical guarantees for the experimental architectures. We thank the reviewer for highlighting the need to distinguish the theoretical guarantees from the experimental architectures. We clarify their scope below.

Universality and experimental backbones. Theorem 1 establishes universal approximation for plain-GMN CrossGMN with fixed MLP source and target architectures on compact general-position sets. Our experiments primarily use ScaleGMN, which performed better empirically but is not covered by these universality guarantees. We also evaluated plain GMN, obtaining somewhat weaker but still competitive results.

CNNs and Transformers. Appendix C.5 discusses how the universality guarantees established for MLPs can be extended to Transformers and CNNs, provided corresponding fixed-architecture universality results for invariant/equivariant networks hold. Such results for Transformers appear in Dayan et al. (2026). We expect analogous results for CNNs to follow from a similar approach.

Common target architecture. Both the heterogeneous-source and unseen-source-architecture experiments use a fixed target architecture.

We will explicitly incorporate all the above clarifications into the manuscript.

We hope that our answers and additional experiments addressed your concerns. If so, please consider raising our score.

Reviewer_eDD4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper approaches to learn equivariant mappings between networks of different architectures. The authors define a cross-architecture operator that takes as input a trained source network and an initialization of the target. This operator is invariant to the permutation symmetries of the source network and equivariant to the permutation symemtries of target one. The proposed CrossGMN builds the parameter graphs of both networks and connects them with bidirectional cross edges. It runs a graph metanetwork on the joint graph and outputs v_init + sΔ from the target-side readout. Empirically, CrossGMN is applied on the knowledge distillation task. The target initialization is a structured pruning of the teacher, the operator is trained to match teacher outputs directly, while its prediction is used to warm-start standard distillation. The evaluation covers SIREN INRs (MNIST, transfer to FashionMNIST, ModelNet40), CIFAR-10 MLPs and ViTs, MNIST CNNs, and heterogeneous and unseen MLP source architectures.

Critical strengths
  1. Problem formulation. Taking both the trained source and a target initialization as inputs gives the operator a well-defined symmetry structure (source-invariant, target-equivariant). This is also motivated by the discontinuity counterexample (Prop. 4) and tied to standard distillation.

  2. Principled architecture design. The layer-level cross-edge structure follows from the invariance requirement.

  3. Expressivity guarantee. Universality is established for continuous cross-architecture operators on MLPs, including a single operator serving finitely many source architectures.

  4. Breadth of evaluation. The experiments span various model families and including also heterogeneous datasets, transfer across datasets and OOD.

  5. Transparent reporting. The authors propose a few ways of reporting their results (e.g. per-teacher medians, wall-clock speedups, reach–budget plots and the Δ@Budget). However, some important metrics should also be included (see Weaknesses). Importantly, authors are also upfront regarding negative results, which are reported rather than omitted (e.g. ViT).

Critical weaknesses
  1. Modest, high-variance gains.

    1. Outside INRs, median speedups range in [1.04x, 1.25x] (wall-clock in [1.01x, 1.20x], while the ViT attention-dimension and head-count reductions slow distillation (0.74x, 0.86x).
    2. Even on INRs, IQRs frequently span from 0 to over 10x, and failure rates reach 18–35% (MNIST width 11.3x and 32x, ModelNet width and width+depth).
  2. No direct report of student quality. Speedup targets are defined by fidelity to the teacher, relative to the anchor’s own plateau; task performance is never reported.

  3. Missing numbers: the performance of teacher, the step-0 of the anchor and of v^S\hat{v}_S, and the model's after distillation (e.g. accuracy or any other metric thereof)

  4. The two-input formulation is degenerate in all main experiments. According tosection 5.1, the anchor v0v_0 is a deterministic function of the teacher (structured pruning), so the learned map is effectively vT→F(vT)v_T \rightarrow F(v_T). The initialization is never an independent input, except in the random-init ablation. Is that true?

  5. Cross-edge design: In their effort to fuse the two graphs, the authors construct the joint graph which consists of the two graphs and a set of edges connecting T to S (ExE_x: cross edges). Besides symmetries, the authors don't clearly motivate this fusion, both in terms of adding edges from T to S (and not in another invariant fashion) and in terms of which teacher nodes to connect with which student nodes. If not for message passing, such an approach assumes that only a few teacher layers affect each student layer. However, if we were to rely on MPNN for propagating information between remote layers, then this would introduce long-range dependencies in slightly deeper networks. Moreover, thinking of BERT -> DistilBERT, the standard method does use specific teacher layers to initialize student layers, implying that such an assumption could stand. However, this is not a general property of model families.

  6. Pairing policy ablation (input models). The pairing policy is ablated only on [2,32,32,1]→[2,8,8,1], where strided and all-to-all differ by two joins. If I am not mistaken, the two policies coincide whenever the student has a single hidden layer. It is never ablated on deep depth reductions, where pairing should matter most.

  7. Pairing policy ablation (alternative policies) The FiLM approach is underspecified, the "no cross edges" is important, yet naive. I believe more results here would be more convincing regarding the choice of the strided policy.

  8. Baselines. Besides ablations, the authors don't provide any other baselines.

  9. Training/evaluation mismatch. CrossGMN is trained to output an edited student but evaluated only as an initialization for the distillation process that will happen later.

Questions for authors
  1. Absolute student quality. Could the authors report task performance directly: teacher performance, the anchor and v~S\tilde{v}_S at step 0, and both at convergence?

  2. For the failure cases (S = 0), does the CrossGMN-initialized student converge more slowly, or does it converge to a worse solution than the anchor?

  3. Role of the two inputs. As stated in W4, it needs a clarification whether CrossGMN does take as input the random initalization of the student or not.

  4. Do the authors view the strided policy as an intended locality prior, analogous to every-other-layer initialization in DistilBERT, or only as an efficiency device?

  5. Could the policy be ablated in a more informative setting? In the current ablation, teacher and student share the same depth.

  6. Have the authors compared lighter fusion schemes, such as virtual nodes ?

  7. Could accounting for gradients help the knowledge distillation task? For example GradMetanet [1] might be an interesting baseline.

  8. Other modules and scope. What happens with modules like LayerNorm, BatchNorm (including its running statistics)? Do the studied NN Zoos contain models that have such modules?

  9. Why is the ViT residual width never compressed? Have the authors tried it?

[1] Gelberg, Yoav, et al. "Gradmetanet: An equivariant architecture for learning on gradients." Advances in Neural Information Processing Systems 38 (2026): 154115-154158.

8 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer eDD4 - Part 1

We thank the reviewer for recognizing the motivation of our two-input formulation, the principled architecture design, the universality guarantee, and the breadth and transparency of the evaluation. We address their comments below.

W8: Additional baselines. We added MetaPrune-GMN (Liu et al. (2025)) and a non-equivariant MLP, all evaluated under the same distillation protocol as CrossGMN.

  • MetaPrune-GMN. We use the released code to transform each teacher into a more prunable network using task loss and sparsity regularization, then prune by group-norm importance and distill under the same protocol as CrossGMN. We match operator size, training compute, and seeds, and select hyperparameters and checkpoints on validation teachers. For width+depth compression, we add the same layer folding used by our anchor.

  • Non-equivariant MLP. To assess the contribution of symmetry-aware modeling, we also train an MLP on re-basined teacher and anchor weights, using canonicalization. We closely match its parameter count to CrossGMN (2.648M vs. 2.647M).

The table below reports MNIST INR speedups relative to the same pruning anchor, evaluated on 1,000 test teachers with matched distillation budgets. Values are the geometric mean ×/÷ geometric SD of per-run median speedups across the same three seeds.

Teacher → StudentCrossGMNMetaPrune-GMNNon-equivariant MLP
[2,32,32,1] → [2,8,8,1]5.51 ×/÷ 1.300.71 ×/÷ 1.162.88 ×/÷ 1.11
[2,32,32,1] → [2,16,1]7.95 ×/÷ 1.182.19 ×/÷ 1.025.01 ×/÷ 1.13

Overall, CrossGMN substantially outperforms all tested baselines, supporting the benefit of learning a structured, symmetry-aware cross-architecture initialization over pruning-oriented transformations, and the tested canonicalized non-equivariant alternative.

W1: Modest classifier gains. Even a 5% acceleration is practically meaningful, particularly at scale, where small relative gains can translate into substantial computational savings. We therefore view these positive results as a promising first step. The three-seed results below show that the positive classifier gains are consistent: their geometric SD is at most 1.05×.

An important factor that helps explain the difference between classifiers and INRs is the strength of their pruning-derived anchors. Their task quality also varies across settings: at the distillation budget B, classifier anchors approach teacher accuracy (MLP: 47.4–50.8% vs. 53.1%; ViT: 71–73% vs. 73.3%; CNN: 95.2% vs. 96.0%), whereas MNIST INR anchors achieve 16.2–21.5 dB PSNR versus 25.5 dB for the teacher. The latter gap corresponds to approximately 2.5–8.5× higher reconstruction MSE, leaving more room to improve the initialization.

We repeated the compression experiments across three seeds. The table below reports representative settings; classifier speedups vary little across seeds. Speedups below are the geometric mean of the per-run median speedups, with geometric SD shown as ×/÷. Acceleration and failure rates are mean ± SD across three runs.

SettingSpeedup>1x (%)Failed (%)
MNIST INR, [2,32,32,1]→[2,8,8,1]5.51 ×/÷ 1.3078.5 ± 5.016.3 ± 4.1
MNIST INR, [2,32,32,1]→[2,16,1]7.95 ×/÷ 1.1885.5 ± 1.512.9 ± 1.9
ModelNet40 INR, [3,48,48,48,48,1]→[3,48,48,1]6.82 ×/÷ 1.7575 ± 10.112.6 ± 5.2
CIFAR-10 MLP, [3072,64,64,64,64,10]→[3072,32,32,32,32,10]1.06 ×/÷ 1.0154.6 ± 1.04.9 ± 0.6
CIFAR-10 MLP, [3072,64,64,64,64,10]→[3072,64,64,10]1.29 ×/÷ 1.0562.6 ± 2.52.1 ± 0.4
CIFAR-10 ViT, blocks 4→21.20 ×/÷ 1.0194.7 ± 0.40.0 ± 0.0
CIFAR-10 ViT, heads 2→10.88 ×/÷ 1.0220.1 ± 4.20.3 ± 0.1

The full table of seed results is available here.

Q2: Failure cases (S=0S = 0). Our evaluation assigns S=0S = 0 whenever the target is not reached within H=1.5BH = 1.5B distillation steps, where B is the setting-specific anchor-convergence budget (Appendix D.2). To distinguish slower convergence from a worse final solution, we continued failed runs to convergence. We separate runs that reach the target later, remain within 1% of it without reaching it, or converge farther from it. For CIFAR-10 MLP depth compression, 46.3 ± 8.6% of failures eventually reach the target, 46.8 ± 16.5% converge within 1%, and 6.8 ± 8.7% remain farther away. Thus, while these runs remain failures under our protocol, the vast majority eventually converge near or beyond the target. The full analysis is available here.

AuthorsOpenReview ↗

Authors’ Response to Reviewer eDD4 - Part 2

W2–W3, Q1: Absolute task quality. We agree that absolute task-quality metrics provide important complementary evidence. In addition to the performance-retention curves in Figure 2 and Appendix D.9, we therefore report ground-truth image PSNR for MNIST INRs and test accuracy for CIFAR-10 classifiers.

The table below compares task quality before distillation (step 0) and at the setting-specific distillation budget B, determined by anchor convergence (Appendix D.2). A and C denote the anchor and CrossGMN initializations. Where applicable, values are reported as mean ± SD across three seeds. CrossGMN improves initial task quality, including a 7.7 dB PSNR gain on MNIST INRs and a 12.4 percentage-point accuracy gain on CIFAR-10 MLPs. After distillation, the gaps are smaller and classifier accuracies approach those of the teachers. These results demonstrate that CrossGMN provides a substantially better initialization, with faster convergence as the primary benefit.

Setting and metricTeacherA at 0C at 0A at BC at B
MNIST INR, [2,32,32,1]→[2,8,8,1], PSNR dB25.56.013.7 ± 0.421.522.8 ± 0.2
CIFAR-10 MLP, [3072,64,64,64,64,10]→[3072,32,32,32,32,10], accuracy %53.118.530.9 ± 0.649.049.3 ± 0.1
CIFAR-10 ViT, blocks 4→2, accuracy %73.324.128.3 ± 0.671.471.7 ± 0.0
CIFAR-10 ViT, attention dimension 48→24, accuracy %73.345.652.2 ± 0.273.073.0 ± 0.0

The full table of absolute task quality results is available here.

W9: Training objective versus downstream evaluation. We train the predicted student directly using teacher-output MSE/KL losses (Appendix D.1), without unrolling subsequent distillation steps. The current objective optimizes step-0 fidelity as a surrogate for reducing downstream distillation time. We therefore evaluate both the immediate performance of the predicted student (reported above) and its subsequent distillation trajectory.

W4, Q3: Dependence between the two inputs. We agree that in our main experiments, the student initialization is obtained by deterministic pruning of the teacher. However, this is not a limitation of our two-input formulation. The most naive alternative would be to first prune the teacher and then apply a weight-space operator to edit the resulting student (vT→vSpruned→vSeditedv_T \to v_S^{\mathrm{pruned}} \to v_S^{\mathrm{edited}}). However, this discards information from the full teacher and has less flexibility in general with respect to the compression method.

Our two-input formulation offers both empirical and theoretical advantages. Empirically, Table 16 (row 2, No cross edges) shows that removing cross-network connections and relying only on the pruned student reduces speedup from 5.20× to 3.18×. Theoretically, our universality results establish that CrossGMN can approximate gradient descent and the full distillation process with respect to any student initialization, independently of the compression method.

We use pruning-derived initializations because they perform better in practice, not because our formulation requires them. Indeed, Table 16 (row 6, Random initialization) shows with a randomly initialized student, CrossGMN achieves a 3.76× speedup (measured against the same pruning-based anchor), compared to 5.20× with a pruning-derived anchor. we will make this point clearer in the next revision.

AuthorsOpenReview ↗

Authors’ Response to Reviewer eDD4 — Part 3 (Final)

W5–W7, Q4-Q6: Cross-edge design, pairing policies, and lighter fusion schemes.

Our strided pairing policy is an intentional locality prior. Beyond enabling greater computational efficiency by reducing the number of layerwise joins, locality is also motivated by prior work. As noted by the reviewer, DistilBERT (Sanh et al. (2019)) provides a precedent for layerwise transfer. More recently, MATCHA (Sudakov et al. (2026)) reports broadly depth-monotone activation alignments across 1B–8B LLMs. These findings provide empirical motivation for locality-based cross-model interactions, including at larger scales.

To test pairing where depth differs, as requested, we added a ModelNet40 ablation from four to two hidden layers ([3,48,48,48,48,1]→[3,48,48,1]). This makes layer selection more informative than in the original equal-depth ablation. Strided pairing achieves a 6.85× speedup, compared with 6.67× for all-to-all pairing and 1.10× for FiLM conditioning. Thus, the strided policy is competitive with all-to-all pairing while using fewer layerwise joins.

We do not claim that strided pairing is optimal, and investigating more sophisticated pairing strategies remains a promising direction for future work. Nevertheless, our ablation demonstrates that this simple, efficient heuristic already performs competitively with all-to-all pairing.

Virtual-node fusion. We additionally evaluated virtual-node fusion, in which teacher and student graphs communicate through a single learned virtual node rather than direct cross-edges. We compared this baseline with CrossGMN on MNIST INR compression ([2,32,32,1] → [2,8,8,1], 11.3× compression), using 3 seeds. CrossGMN achieves a 5.51× median speedup (geometric SD ×/÷1.30), compared with 4.43× (×/÷1.10) for virtual-node fusion. CrossGMN also accelerates a larger fraction of teachers (78.5% vs. 72.6%) and has a lower failure rate (16.3% vs. 19.5%). These results suggest that our cross-edge design provides an advantage over global virtual-node fusion, while confirming that the lighter alternative also achieves significant acceleration.

Q7: GradMetaNet and gradients. Gradient information could be useful, and GradMetaNet (Gelberg et al. (2025)) is a relevant candidate for an adapted comparison. Our current operator takes weights as inputs; a gradient-informed variant would additionally require evaluating teacher–student losses and obtaining student gradients, whose cost should be included.

Q8: LayerNorm, BatchNorm, and running statistics. Our ViT zoo includes LayerNorm, whose learned scale and shift parameters are represented through auxiliary nodes and parameter edges. The functional loss uses actual teacher and student forward passes, thereby accounting for normalization effects. While we have not yet evaluated BatchNorm in CrossGMN, the original GMN work (Lim et al. (2023)) successfully handled architectures with BatchNorm, supporting the feasibility of extending our approach to this setting, including appropriate treatment of running statistics.

Q9: ViT residual-width reduction. We explored reducing the residual width from 48 to 24, but the resulting initializations achieved only 10–14% initial accuracy (close to random in CIFAR10 task) and did not yield successful compression. Unlike MLP-width reduction, residual-width reduction is more complex as it requires coordinated changes across attention projections, normalization layers, the FFN, and the readout. While these preliminary results highlight the difficulty of this setting, they do not isolate the source of failure.

We hope that our answers and additional experiments addressed your concerns. If so, please consider raising our score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Author Summary: Main Contributions, Reviewer Concerns, and New Experimental Results

We sincerely thank all three reviewers for their thoughtful and constructive feedback. Our responses include new experiments and analyses addressing the main concerns. We summarize the recognized strengths, concerns, and new evidence below.

Strengths recognized by reviewers

Reviewers highlighted our novel two-input formulation for resolving symmetry mismatches between source and target architectures, alongside our symmetry-preserving design, universality guarantees, and experimental evaluation. They also recognized the breadth of experiments across INRs, MLPs, CNNs, and ViTs, including heterogeneous architectures and cross-dataset transfer, and our transparent reporting of variability and failures.

Main concerns and new evidence

The primary concerns involved learned baselines, scalability, cost-effectiveness, generalization and teacher-to-student information transfer. We addressed them as follows:

  1. Learned baselines: We compared against MetaPrune-GMN (Liu et al., 2025), WARM (Gong & Liu, 2026), and a canonicalized non-equivariant MLP. On MNIST INR width compression, CrossGMN achieves 5.51× speedup versus 0.71×, 0.37×, and 2.88×, respectively, supporting the value of our symmetry-aware formulation.
  2. Scalability to larger models: We evaluated CrossGMN on 1.25M-parameter ViTs, approximately 10× the previously evaluated ViT scale, retaining positive speedups of 1.20× and 1.11× for two depth-compression settings.
  3. Cost-effectiveness (break-even analysis): Accounting for the one-time operator training cost, compression savings exceed training costs in several settings, yielding up to 49.4% net savings over the evaluated zoo.
  4. Generalization to Unseen architectures: The same trained operator achieves median acceleration on four additional unseen MLP architectures, with 1.31× pooled speedup, versus 1.06× in our original unseen-architecture evaluation.
  5. Independent training runs and task quality: Three independent seeds confirm the consistency of speedup gains. We also report absolute student quality, including step-0 improvements of up to 7.7 dB PSNR on MNIST INRs and 12.4 percentage points in CIFAR-10 MLP accuracy.
  6. Virtual-node ablation: A shared virtual node for teacher–student communication achieves 4.43×, versus 5.51× with direct cross edges.
  7. Direct evidence of teacher-to-student transfer: We introduced a wrong-teacher experiment: replacing the teacher input while keeping the student anchor fixed reduces speedup from 5.51× to 3.27× and approximately doubles the initial loss. Moreover, with a randomly initialized student containing no copied teacher weights, adding cross edges increases speedup from 1.38× to 3.67×. These results demonstrate that CrossGMN uses information from the teacher graph beyond what is already contained in the anchor.
  8. Understanding the learned operator: We analyzed the magnitude and direction of learned weight updates, compared them with ordinary distillation steps and investigated failed compression cases. These analyses further clarify what CrossGMN learns and distinguish its updates from learning a small number of distillation steps.

Clarifications and revision

We also clarified the scope of our theoretical guarantees, within-family cross-architecture versus cross-family transfer, and the role of the two inputs.

We believe these results materially strengthen the evidence for CrossGMN while clarifying its limitations. Beyond compression, our central contribution is a principled, symmetry-aware framework for weight-space transformations between different neural architectures.

We are incorporating these results and clarifications into the manuscript and plan to upload a revised version within the next week.

We thank the reviewers and Area Chair for their time and consideration.

AuthorsOpenReview ↗

Closing Rebuttal Summary and Key Developments

Dear Area Chair and Reviewers,

As the rebuttal period approaches its conclusion, we sincerely thank all three reviewers for their thoughtful feedback and continued engagement. We are particularly grateful to reviewers KCk4 and H2kh, who acknowledged that our additional experiments addressed most or all of their concerns and indicated their intention to increase their scores from 2 to at least 3 (these changes are not yet reflected in the official ratings).

The discussion has led to substantial new experiments and analyses that strengthen the paper and clarify its contributions and limitations. Building on our earlier author summary, we would like to highlight three particularly important developments:

  • Scalability: We demonstrated CrossGMN on 1.25M-parameter ViTs (~10× larger than our original models), retaining a 1.20× speedup and reaching the scale of recent weight-space works.
  • Practical utility: Accounting for the one-time operator training cost, compression savings exceed training costs in several settings, yielding up to 49.4% net savings over the evaluated zoo. Notably, optimized training reduces the ViT break-even point to as few as 150 compressions, with up to 20.7% net computational savings.
  • Methodological evidence: New controlled experiments, including wrong-teacher tests, random-initialization and cross-edge ablations, and comparisons with learned and non-equivariant baselines, provide stronger evidence for teacher-to-student information transfer and the benefits of the two-input, symmetry-aware formulation.

We believe the rebuttal has materially strengthened both the empirical support and the practical motivation for CrossGMN, alongside its original theoretical and methodological contributions.

Thank you again for your time, constructive feedback, and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.