acceptodds
Under review as a conference paper at ICLR 2027

Interfaces and Modules: A Scalable Representation for Weight-Space Learning across Architectures

Abstract

Weight-space learning across architectures requires representations that respect parameter symmetries and scale to large networks. Existing equivariant tensor encoders require architecture-specific symmetry specifications, while neuron graphs make global attention costly. We introduce a typed graph organized around shared coordinate-permutation actions. Its nodes, called interfaces, represent coordinate spaces whose bound tensor axes must be permuted jointly; operation occurrences, called modules, form typed hyperedges. A compiler derives this graph from supported PyTorch models, with exact recovery of the permutation factors defined by its constraints. The Universal Weight Encoder (UWE) combines local equivariant updates to parameter features with global attention over invariant interface summaries. The small number of interfaces makes global attention practical, and a single encoder processes different supported architectures. UWE is competitive with architecture-specific state-of-the-art methods on established benchmarks. On a new zoo of 1,200 transformers spanning 100 architectures, it outperforms the evaluated baselines at ranking networks within unseen architectures, including architectures larger than those used for training, while using substantially less memory.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
weight-space learning, graph metanetworks, permutation equivariance, neural network symmetries, model zoos, geometric deep learning, transformers

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9YkcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses weight-space learning across neural network architectures. It introduces a typed interface–module graph, an automatic compiler for supported PyTorch models, and the Universal Weight Encoder (UWE), which combines local equivariant updates with global attention over interface summaries. Experiments evaluate predictive performance, transfer to unseen transformer architectures, and computational scalability.

Critical strengths
  1. The interface-module representation is a valuable contribution. It supports a broad range of common operations while explicitly capturing dependencies between parameter axes that must be permuted jointly. The compiler can automatically derive interfaces and parameter bindings from supported model implementations, eliminating the need for manually specified symmetries for each combination of operations. This offers a promising foundation for processing diverse architectures with a shared encoder.

  2. Computational scalability is demonstrated through substantial memory savings against the evaluated baselines (Tables 5 and 12). The paper clearly distinguishes computational feasibility on large networks from predictive performance at that scale and acknowledges the associated runtime tradeoff.

Critical weaknesses
  1. The transformer-zoo results in Table 3 use a single training seed, and Table 4 provides no uncertainty estimates. This limits confidence in the stability of both the comparative improvements and the ablation conclusions. Repeated runs for small UWE and the strongest NG-GNN configuration, together with per-architecture results, would help establish robustness.

  2. Appendix D.2 uses the same learning rate and training schedule across methods and acknowledges limited method-specific hyperparameter searches. Although the common training budget and alternative NG-GNN readout are useful controls, they do not establish that each method is comparably well optimized. A modest validation-only learning-rate sweep and convergence curves for the strongest NG-GNN would strengthen the comparison.

Questions for authors
  1. Residual additions can cause many operation occurrences to share an interface. Could aggregation into shared interface states become a bottleneck as residual depth increases? A breakdown of the existing held-out-architecture results by depth would help assess this concern.

  2. Appendices D.1 and H.2 describe learned probes that provide activation features for INR classification, following prior work. How much does UWE’s INR performance depend on these observations? An ablation with and without probe features would help distinguish the contribution of the weight representation from that of the probes.

  3. Appendix G.1, Equation (18), defines a parameter-valued decoder, but the experiments focus on prediction tasks. Has this decoder been validated for weight-to-weight tasks such as weight editing? Additionally, what extensions would be needed to generate weights for a specified target architecture different from the input architecture? A clarification of the supported and empirically validated scope would be helpful.

3 comments

W1. Result stability. We repeated small UWE and the all-node NG-GNN (the strongest baseline) on the original 1,200-model zoo (three seeds, original splits, 100 epochs). Validation selected each checkpoint. We report mean test MSE for the 60-output profile ± sample SD across seeds.

On the full split (70/15/15 train/validation/test architectures), small UWE scored 0.318±0.012 and NG-GNN 0.384±0.013. Table 3 reported 0.3355 and 0.3637 for these models. UWE had lower error on 14/15 architectures (Q1). On the common split (43/7/7), the means were 0.270±0.044 and 0.385±0.055, against the reported 0.2790 and 0.4411.

We rerun the ablation that removes the global attention and report confidence intervals (three seeds on the common split). This scores 0.781±0.086 as compared to 0.84 reported in the submission.

W2. Baseline tuning. On the common validation split we tried learning rates 0.0001, 0.0003, 0.0006 and 0.001 for both models, then varied weight decay at the two best rates. Below is validation MSE for the selected recipes (seed 1701).

Epoch20406080100
Small UWE0.3730.3220.3020.2790.259
All-node NG-GNN0.6620.4970.4460.3630.358

Best validation MSE was 0.248 at epoch 78 for UWE (rate 0.0006, decay 0.001) and 0.349 at epoch 89 for NG-GNN (0.001, decay 0). Those checkpoints scored 0.237 and 0.366 on the test networks. Both methods report gains from finetuning.

Q1. Can shared residual interfaces become a bottleneck? Each operation retains its parameter states, but averaging messages at a shared interfaces can lose information. At 2, 3, 4 and 6 blocks, mean test MSE for small UWE / NG-GNN was 0.196/0.363, 0.313/0.362, 0.288/0.344 and 0.439/0.457. The advantage was smallest at six blocks.

Each architecture (blocks/width/heads/feed-forward ratio) has 12 test networks. The UWE pair uses seed 715. The last two columns give three-seed means ± SD.

ArchitectureSmall UWEAll-node NG-GNN
2/96/3/20.199±0.0590.339±0.026
2/128/2/30.171±0.0230.414±0.012
2/128/2/40.217±0.0080.335±0.016
3/96/3/30.166±0.0420.289±0.060
3/160/2/40.318±0.0120.342±0.014
3/288/6/40.340±0.0180.366±0.016
3/352/4/40.341±0.0280.374±0.063
3/384/4/20.401±0.0180.440±0.016
4/128/4/20.189±0.0100.241±0.012
4/288/6/20.307±0.0130.381±0.044
4/384/4/30.369±0.0160.412±0.018
6/96/3/20.248±0.0350.321±0.041
6/288/4/40.445±0.0180.516±0.028
6/384/4/30.630±0.0460.550±0.016
6/384/8/30.432±0.0290.439±0.026

At six blocks, large UWE improved on two of four architectures. Width, heads and feed-forward ratio also vary, so depth alone cannot explain this pattern.

UWE can widen the vector at each interface coordinate (dHd_H) independently of the parameter-cell width (dEd_E). This could ease the bottleneck without causing memory or runtime degradation. We have not tested the effect of varying dHd_H alone.

Q2. Effect of probes. We varied only the probe count at seed 715 (300 epochs on MNIST, 100 on Fashion-MNIST). Each validation-selected checkpoint was evaluated on 10,000 test INRs.

Test accuracy, 0 → 64 probes (gain in points)MNISTFashion-MNIST
UWE (one seed)89.25% → 97.98% (+8.73)68.76% → 84.27% (+15.51)
NG-GNN91.4% → 94.7% (+3.3)68.0% → 74.2% (+6.2)
NG-T92.4% → 97.3% (+4.9)72.7% → 74.8% (+2.1)

NG-GNN/T, ScaleGMN, and UWE all use 64 probes in Table 2.

Adding 64 probes gave UWE a larger accuracy gain on both datasets in our single-seed comparison. The experiment leaves open why the gain was larger. We suspect this might be due to the way each methods uses the probes. Neural graph methods project the probes and add them to the representations of nodes. In UWE, probes are used to intialise interface states, which might give a more direct way to combine what a neuron does with how it connects to the rest of the network.

Q3. Can the decoder edit weights or change architectures? Eq. (18) predicts an additive update per INR weight or bias. On 10,000 held-out MNIST INRs, zero-probe dilation gave pixel MSE 0.02243±0.00014 on the three seeds. Unchanged weights gave 0.08717. Published zero-probe MSE was 0.0238 for NG-GNN and 0.0196 for NG-T. [Kofinas et al., Table 5]. On 10,000 Fashion-MNIST test INRs, zero-probe UWE reduced dilation MSE from 0.07301 to 0.02018 (one seed, no sign augmentation).

For a specified target architecture that differs from the source, we could use UWE to encode the source network. A UWE-style decoder on the target graph would then use the resulting embedding to predict the target weights. Developing and testing such a decoder would require substantial further work, including the choice of suitable training datasets. We will make clear in the revision that our experiments cover only editing within the source architecture.

I thank the authors for the detailed response, informative new experiments, and transparency regarding the limitations.

W1 (Result stability): Resolved. The three-seed experiments confirm the main improvements, and the per-architecture results provide stronger evidence of robustness.

W2 (Baseline tuning): Resolved. The learning-rate and weight-decay sweeps, convergence results, and validation-selected checkpoints substantially strengthen the fairness of the comparison.

Q1 (Shared residual interfaces): Remains a concern. The depth-wise results suggest that UWE's advantage diminishes for deeper architectures, potentially due to information loss at shared interfaces. The authors propose increasing interface width as a mitigation, but this remains untested. Moreover, increasing representation dimensionality does not necessarily recover information lost through averaging, and wider interfaces should generally incur additional computational costs. An experiment varying interface width, reporting both predictive performance and memory/runtime costs, would help address this concern. This is particularly important given the prevalence of deep residual connections in modern architectures, including Transformers.

Q2 (Probe dependence): Resolved, with a remaining observation. The results suggest that UWE benefits substantially more from activation probes than the baselines, rather than consistently learning a superior weight-only representation. Since the methods incorporate probes differently, it remains unclear whether this difference stems from UWE's architecture or its probe-integration mechanism. Nevertheless, this is still a meaningful contribution. A small ablation isolating the probe-integration mechanism would be valuable if feasible, but I do not consider it a blocker.

Q3 (Weight editing): Resolved. Although UWE does not outperform the strongest baseline on weight editing, the results demonstrate a meaningful ability to perform weight transformations. It would be interesting to evaluate this capability on larger models, where UWE's scalability advantages may become particularly relevant.

Overall, the rebuttal substantially strengthens the paper and resolves most of my concerns. However, I am keeping my score unchanged for now, primarily because Q1 raises an important question about the method's applicability to deeper architectures with extensive residual connections. I would appreciate further experimental evidence addressing this potential bottleneck and the computational trade-offs of the proposed mitigation.

Thank you for the follow-up and for recognising the additional evidence on stability, tuning and weight editing.

We tested wider interface states, increasing dHd_H from 32 to 64 and 128. These single-seed experiments did not show a consistent improvement on deeper architectures, so they do not confirm our initial hypothesis that increasing interface width would address the observed weakness.

We would like to clarify that, even when residual blocks share an interface, each module and its weights remain separately represented. The encoder also maintains separate learned states for their parameters.

A more specific hypothesis therefore concerns how the current encoder aggregates and reads out this information. UWE's current invariant network-level readout pools only interface states, so information held in module states must first pass through interface aggregation to influence the prediction. When many modules share an interface, this aggregation may make their individual contributions less accessible to the readout. This is a plausible mechanism, rather than an established explanation of the depth-dependent results.

A concrete direction for the encoder is to use an invariant readout head that combines module and interface states, giving predictions direct access to both while preserving the representation’s symmetries. The representation already supports this design choice. We will report the negative width result and make clear that the observed depth-dependent performance remains a potential limitation of the current UWE configuration.

Reviewer_QiDeOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a unified way of representing a much greater variety of architectures in a unified framework for weight space learning than before. This is achieved through a compiler working with the PyTorch computation graph that analyzes supported operations, yielding a new graph that represents the permutation symmetries applicable to the weights. This unified representation can be processed by encoders, which allows them to work over many different architecture types.

Critical strengths

Unified framework

From my reading, the overall goal of the paper is reasonable in being able to unify different architectures in weight space correctly. The approach proposed by the paper roughly makes sense to me in encoding permutation symmetries correctly, including more complex structures that allow encoding of architectures such as modern pre-norm MHA architectures. I believe that this is a great strength, as previous work required significant amount of error-prone hand design of specifying the precise equivariance needed. The compiler looks to be a significant aid in being able to extend this representation to future architectures.

Critical weaknesses

Insufficient clarity on explaining the idea and representation

Section 2 describes some overall ideas, but it is very difficult for me to follow what is what. After careful reading, I am still uncertain about what exactly a module is. To me, section 2 lacks precision: statements are made vaguely and details are hand-waved away. As it stands, while I believe that the fundamental ideas are probably sound (I can kind of see how things are supposed to work), it is difficult for me to verify any of it. Please see my questions for some fundamental aspects that I did not find clear enough, such as basic concepts of what actually constitutes a node and an edge in the proposed bipartite graph. There is a significant amount of backsolving I tried to do from Figure 2 to figure out the text, indicating that the text simply is not clear enough for a human reader.

The same applies to section 4, introducing the concept of glue but specifying in very little detail of what happens with it. I understand that there are more details in the appendix, but as it stands, the main text alone is insufficient to get sufficient understanding of what is happening under the hood.

In conclusion, this is the main reason why I lean towards a reject for now. The paper does not clear the bar for me in terms of being understandable, relying too much on rough overviews of ideas without any precision and insufficient exposition. In my view, making the paper understandable would require a substantial rewrite, so in the current form I cannot recommend it for acceptance.

Questions for authors
  1. Why do you claim scalable in the title, when I don't see any concrete experiment on scalability? I agree that it is unified, but in which sense is it scalable? If anything, limitation iv in the paper sounds like it is actually not scalable ("substantially slower per step than neural graph baselines") [clarification]
  2. Please clarify several aspects of representation. I figured certain parts out from the figures and text, but it is best if you can state these concretely: what is a node, what is an edge, what data is carried with each, and a conceptual overview of how the graph is built from an architecture. What makes the graph bipartite? I understand that aspects of this may exist in the appendix already, but I believe that the current main text does not sufficiently give a clear picture of what happens [clarification]
3 comments

We agree that the main text leaves too much for readers to infer from Figure 2 and the appendix. We are rewriting Section 2 to define modules and interfaces more precisely. Section 4 will include the worked example from the appendix. We will also drop some of the new terminology where it does not improve readability and bring equations governing the encoder into the main text.

We have posted a two-part clarity note in the general comments on the submission, which we are incorporating into the revised main text.

1. Scalability

We primarily mean the source network size that fits in a fixed GPU memory budget. Limitation (iv) reports the cost in time. To provide additional evidence, we compared four-layer UWE (72,636 parameters) and NG-GNN (71,768) on the same 80 GB H100 within each size group. Both used FP32 and a 70 GiB allocation cap. Each pilot epoch used 12 randomly initialised transformers, one per forward pass and 12 per optimiser step. Each cell gives UWE / NG-GNN. Memory figures are peak allocations. Parentheses give the ratio of the unrounded values.

Source parametersMedian forward (ms)Median epoch (s)Peak inference (GiB)Peak training (GiB)
3–10M425 / 108 (3.94×)20.1 / 5.1 (3.91×)1.10 / 11.01 (10.0%)3.19 / 11.78 (27.1%)
14.7M776 / 257 (3.02×)41.1 / 13.3 (3.09×)1.92 / 19.79 (9.7%)5.48 / 32.73 (16.8%)
19.8M935 / 348 (2.69×)52.8 / 16.9 (3.13×)2.57 / 26.62 (9.6%)7.04 / 44.03 (16.0%)
30M1,014 / 521 (1.94×)62.5 / 25.1 (2.49×)3.86 / 40.25 (9.6%)10.34 / 66.58 (15.5%)
40M1,200 / OOM85.2 / OOM5.08 / OOM13.44 / OOM
200M2,871 / –285.4 / –25.32 / –63.73 / –

OOM means the run exceeded the cap. A dash marks an unmeasured run. Forward times exclude graph construction and transfer.

The UWE/NG-GNN inference-time ratio fell with source size across the four paired groups shown, from 3.94× at 3–10M to 1.94× at 30M. The epoch-time ratio rose slightly at 19.8M before falling to 2.49× at 30M. At 40M, UWE completed both inference and training, while even the 71,768-parameter NG-GNN ran out of memory in both. Under the same cap, UWE completed pilot epochs through 200M.

We profiled three forward passes on one source at each of 14.7M and 30M. UWE issued 47,541 and 58,059 kernel launches per forward, against 326 and 404 for NG-GNN. At both sizes, over 93% of UWE kernels ran for less than 10 µs. In one 30M forward pass, communication between modules and interfaces accounted for 52% of launches. Exchange among interfaces, including global attention, accounted for 1.8%. These short kernels suggest two directions for future encoders on interface–module graphs. We could batch local updates across modules or design a different permutation-equivariant local update that needs fewer launches.

2. The representation

A module represents one occurrence of an operation, such as a linear layer, a convolution, a normalisation layer or an activation. It holds the operation’s parameter tensors, if any. A parameter-free activation carries a vector initialised from its operation type. An interface represents a shared coordinate space whose ordering every connected module must respect. It carries one feature vector per coordinate.

A binding specifies which axis of a module uses an interface and in what role, such as input or output. If the interface coordinates are permuted, the bound axis must be reordered in the same way. For example, the weight matrix WW and bias vector bb are two parameter units of one linear layer. The input interface indexes the columns of WW. The output interface indexes both the rows of WW and the entries of bb, so those rows and bias entries must be reordered together. This gives a general recipe for incorporating modern DL components. In contrast, in neural graphs, individual weights are edge features and biases are node features and representation of nromalisation, skip conenction, attention heads is often arbitrary.

Figure 2 shows the representation as a hypergraph. Interfaces are vertices, and each module is a hyperedge joining the interfaces it uses. To obtain the equivalent bipartite graph, draw each module as a node in its own right. In this view, interfaces and modules are the two types of nodes. Each binding gives one edge joining a module to an interface, labelled by its role.

The compiler traces the network on example inputs. The coordinate axes of tensors between operations are candidates for interfaces. Each supported operation has a rule describing how its coordinates can be renumbered. A glue requires two aligned coordinate intervals to be renumbered together. Pointwise activations and normalisation layers glue their inputs to their outputs. An ordinary linear layer imposes no such requirement.

I appreciate the authors for their detailed response and the extra effort that went into providing some new experimental results. The answers do clear up some aspects that I had not understood until now.

However, I am hesitant to give a more positive score at this moment. Since a major rewrite is needed (and promised by the authors), I cannot give a higher score until I see a version that sufficiently improves the writing.

Thank you! We have revised Sections 1–4 to address the presentation concerns, adding clearer definitions, worked examples, and explicit equations for the encoder. Our general comment under the submission summarises the changes.

Reviewer_Xy8uOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses scaling weight space methods to larger input data. It aims for a single representation that respects the symmetries of network parameters, works across different architectures, and stays compact enough to scale to large models. Its central idea is to describe a network by its shared coordinate spaces, called interfaces, which group together the parameter axes that must be relabelled jointly. The operations connecting these spaces become typed hyperedges, called modules. Because a network has far fewer interfaces than neurons, global attention over them becomes practical. To process this new graph representation, the authors design the Universal Weight Encoder, which combines local symmetry-respecting updates to parameters with global attention across interfaces, so one model can process many architectures. They also introduce a new benchmark of vision transformers spanning many architectures to test transfer to unseen and larger designs. Based on the evaluation, UWE is competitive with baselines on standard benchmarks, transfers better than existing approaches on the new zoo while using less memory, and can process networks far larger than prior graph-based methods can handle.

Critical strengths
  1. The paper attempts to address the scalability issues of neural graphs/graph metanetworks, which is one of the biggest impediments in the weight space learning field. Innovative works towards this direction will unlock further progress in the whole field.

  2. Principled, symmetry-aware representation. The authors provide a much more compact graph representation, while provably preserving the targeted permutation symmetries.

  3. The authors construct and evaluate on a Zoo of varying Transformer Architectures (1,200 CIFAR-100 vision transformers), a model family under-represented in the WSL field, especially in a varying architectures setup.

  4. The proposed UWE outperforms the baselines by a large margin in Table 3, while also requiring much less memory (Table 5).

  5. Table 4 presents an insightful ablation. Global attention seems to be responsible for the performance improvements, an observation which supports the need for more compact graph representations that make attention affordable.

Critical weaknesses
  1. Writing. The paper is dense and hard to follow:

    1. The whole section 2 could be better (especially 2.1).
    2. The interface–module graph construction procedure should be written much better. At its current state, it is difficult to follow.
    3. Table 1 lists large neural network architectures alongside the sizes of their instantiated interface–module graphs, yet none of these networks are evaluated in the experiments. Similarly, Table 5 reports the peak allocated memory of UWE and of the baselines for models that also do not appear in the empirical evaluation. While these metrics provide useful context, they would be better suited for the Appendix alongside related supplementary material (e.g., Table 12). In its current form, highlighting unexamined models in the main text may mislead readers regarding the experimental scope.
    4. The paper introduces a heavy vocabulary of new terminology (e.g., interface, module, binding, interface/module pass, delivery, emission, codes, lift, glue, stick). While these abstractions are likely intended to formalize the framework, they feel counterintuitive and hinder rather than facilitate understanding. Grounding these concepts in standard, established literature or simplifying the taxonomy would significantly improve clarity.
    5. The actual encoder is only detailed in the Appendix G.1, not in the main paper.
  2. Evaluation (Datasets). The proposed method is evaluated only on one model family of larger networks, i.e. on Transformers. They also evaluate on the standard benchmarking datasets of the initial works in WSL, however these Zoos consist of models of only a few thousand parameters.

  3. Evaluation (Tasks). The main empirical results of the paper are concentrated in Table 3. Although the performance of UWE is promising, it is only evaluated on one task, rendering the evaluation poor.

  4. Evaluation (Size of NNs). The main contribution of the proposed method is the scalability. However, to my understanding, the biggest model of the constructed Zoo has ~9.1M parameters.

  5. Efficiency. Despite being much more memory efficient, in Table 3 we see that UWE needed substantially more hours to train, when compared to the baselines (with the only exception of NG-T). To have a better understanding of the efficiency, the memory and time complexities should be reported for both UWE and the baselines.

Questions for authors
  1. How do the authors interpret that in INR datasets their method surprasses the permutation equivariant alternatives and the scale equivariant one, but in the generalisation prediction task, it performs on par with the permutation equivariant baselines?

  2. Have the authors evaluated their method on additional tasks, datasets or model families?

  3. Is the proposed method suitable for editing tasks? Although less frequently evaluated, such tasks stand as the biggest benefit of parameter-processing meta-models against probing methods.

  4. Can the authors show predictive results for at least one trained model on >10M parameter NNs, to support the scalability claims beyond feasibility?

  5. Have the authors tried plain GNN models on top of their new graph representation?

  6. Conversely, have the authors tried using the neural-graphs/GMN graph representation and then apply a UWE-like variant? If I am not mistaken, the proposed method applies a pool-attention-broadcast-update mechanism. What stops us from applying something similar to the baselines' graphs representation?

  7. Previous works motivate the graph construction keeping in mind the property of expressing the forward/backward passes of the input NN [1,2] (also proved in [3]). Does the proposed method enjoy such a property?

  8. As written in Appendix D.2 (should be briefly mentioned in the main body), in Table 3 both baselines and UWE are trained for a fixed budget of epochs. The same batch size and epochs guarantees same amount of steps, but have all considered models converged?

  9. If I understand correctly, in Table 3 the output of each metamodel is a 60-dimensional vector. Why did the authors decide to evaluate on this new task rather than on plain accuracy as in almost every other paper in the literature? It is indeed an interesting property to predict, but having also the plain accuracy would help the reader understand whether this property is actually more difficult to predict or not.

[1] Lim, Derek, et al. "Graph metanetworks for processing diverse neural architectures." International Conference on Learning Representations, 2024.

[2] Kalogeropoulos, Ioannis, Giorgos Bouritsas, and Yannis Panagakis. "Scale equivariant graph metanetworks." Advances in neural information processing systems 37 (2024): 106800-106840.

[3] Navon, Aviv, et al. "Equivariant architectures for learning in deep weight spaces." International Conference on Machine Learning. PMLR, 2023.

4 comments
AuthorsOpenReview ↗

Comment 1/2

W1. Clarity. We agree that the main text leaves too much for readers to infer from Figure 2 and the appendix. We are rewriting Section 2 to define modules and interfaces more precisely. Section 4 will include the worked example from the appendix. We will also drop some of the new terminology where it does not improve readability and bring equations governing the encoder into the main text. We have posted a two-part clarity note in the general comments on the submission, which we are incorporating into the revised main text. We’ll also further clarify which results in the main text refer to computation feasibility and which evaluate predictive performance.

W2 & W3 & W4 Addressed in responses to questions below.

W5. Scalability. In addition to experiments reported in the submission, we compared four-round UWE (72,636 parameters) and NG-GNN (71,768) in FP32 on an 80 GB H100 with a 70 GiB cap. Each epoch used 12 random-weight transformers. Entries give UWE / NG-GNN (ratios before rounding).

Source parametersMedian forward (ms)Median epoch (s)Peak inference (GiB)Peak training (GiB)
3–10M425/108 (3.94×)20.1/5.1 (3.91×)1.10/11.01 (10.0%)3.19/11.78 (27.1%)
14.7M776/257 (3.02×)41.1/13.3 (3.09×)1.92/19.79 (9.7%)5.48/32.73 (16.8%)
19.8M935/348 (2.69×)52.8/16.9 (3.13×)2.57/26.62 (9.6%)7.04/44.03 (16.0%)
30M1,014/521 (1.94×)62.5/25.1 (2.49×)3.86/40.25 (9.6%)10.34/66.58 (15.5%)
40M1,200/OOM85.2/OOM5.08/OOM13.44/OOM
200M2,871/–285.4/–25.32/–63.73/–

OOM means the cap was exceeded. A dash means unmeasured.

The UWE/NG-GNN inference-time ratio fell with source size across the four paired groups shown, from 3.94× at 3–10M to 1.94× at 30M. The epoch-time ratio rose slightly at 19.8M before falling to 2.49× at 30M. At 40M, UWE completed both inference and training, while even the 71,768-parameter NG-GNN ran out of memory in both. Under the same cap, UWE completed pilot epochs through 200M.

We profiled three forward passes on one source at each of 14.7M and 30M. UWE issued 47,541 and 58,059 kernel launches per forward, against 326 and 404 for NG-GNN. At both sizes, over 93% of UWE kernels ran for less than 10 µs. In one 30M forward pass, communication between modules and interfaces accounted for 52% of launches. Exchange among interfaces, including global attention, accounted for 1.8%. These short kernels suggest two directions for future encoders on interface–module graphs. We could batch local updates across modules or design a different permutation-equivariant local update that needs fewer launches.

We’ll also add O-notation scalability information to the revised version. In short, at fixed encoder width, NG-GNN and GMN take O(V+E)O(V+E) work and feature storage per round for VV nodes and EE edges. NG-T's dense attention takes O(V2)O(V^2) work and stores O(V2)O(V^2) scores if materialised. UWE attends over AA interfaces in O(A2)O(A^2) work. Widening at fixed depth leaves AA unchanged. Both UWE and NG-GNN then do work linear in source parameter count. UWE's main states occupy O(CdE+SdH)O(Cd_E+Sd_H) memory for CC parameter cells and SS interface coordinates. Since cells far outnumber coordinates, this allows for favorable storage use. Transformer-NFN's cost depends on padded tensor dimensions (1.18B parameters at width 288 in Table 3).

Q1. Why does UWE lead on INR classification but perform similarly to other methods on CNN accuracy prediction? Our interpretation is that the CNN accuracy benchmark is close to saturation. UWE quickly reached near-zero training and validation error. Several methods already rank these CNNs accurately, leaving little room for the gains we see on INR classification. We report the CNN benchmark because it allows a direct comparison with prior work and serves as a “sanity check”. Its limited headroom was one reason we built the transformer zoo in Table 3, where we can test how well an encoder predicts performance on unseen architectures.

Q2. Have you evaluated additional tasks, datasets or model families? We added dilation on both INR datasets (Q3) and clean accuracy prediction on CIFAR-100 (Q9). The larger transfer test uses two new architectures in the same transformer family (Q4). We have not added a new dataset or model family.

Q3. Can UWE edit weights? Yes. Eq. (18) predicts an additive change to each INR weight and bias. We render the edited INR and score it against the ground-truth image after a 3×3 dilation. On 10,000 MNIST test INRs, unedited pixel MSE was 0.08717. Zero-probe UWE scored 0.02243 ± 0.00014 on the three seeds (mean ± sample SD). Published zero-probe means are 0.0238 for NG-GNN and 0.0196 for NG-T. [Kofinas et al., Table 5]. On 10,000 Fashion-MNIST test INRs, UWE reduced pixel MSE from 0.07301 to 0.02018 (one seed, zero probes, no sign augmentation). We reused hyperparameters from INR prediction experiments (Table 1); no finetuning was performed.

AuthorsOpenReview ↗

Comment 2/2

Q4. How well do the trained metamodels predict on networks above 10M parameters?

We trained 12 CIFAR-100 transformers at each new width using the original recipes. All 24 were held out from metamodel training, normalisation and selection. We applied the small UWE and all-node NG-GNN checkpoints from Section 5.2 . Entries are means ± sample SD over three seeds. Kendall's τb\tau_b ranks the 12 networks at each width by mean profile loss.

Source parametersProfile MSE ↓, UWEProfile MSE ↓, NG-GNNKendall's τb\tau_b ↑, UWEKendall's τb\tau_b ↑, NG-GNN
14.7M0.402 ± 0.0260.640 ± 0.0700.828 ± 0.0170.657 ± 0.076
30.0M0.666 ± 0.0861.413 ± 0.1410.646 ± 0.0170.596 ± 0.097

UWE had lower mean error at both widths. Both methods made larger errors and ranked the networks less accurately at 30M.

Q5. Have you tried a plain GNN on the interface–module graph?

We kept NG-GNN's four message-passing layers and all-node readout. To give every node a vector of the same width, we pooled each module's parameter cells with a learned DeepSets map and used one summary per interface. On 84 held-out transformers (seed 715), profile MSE was 0.923 for this model, 0.428 for flat-graph NG-GNN and 0.280 for small UWE.

Once a weight matrix is pooled into one vector, the GNN can no longer address its rows or columns individually. The ablation changes both the graph and its node features, so it cannot isolate the contribution of the representation. Because of the bipartite/hypergraph nature of the representation, the use of a plan GNN is difficult without breaking the representation in some way. An alternative would be to use a bipartite GNN that keeps coordinate states and passes messages along their bindings, but that would be close in spirit to UWE and fail to isolate the effect of the representation either.

Q6. Could a flat neural graph also pool, attend and broadcast?

Yes, in principle. However, on a flat neural graph, one still has to decide which features to pool together and where to broadcast each update. These choices become harder to specify as architectures grow larger and more complex. Our construction removes that ambiguity.

Q7. Can UWE express a forward and backward pass of the source network?

We have no general proof that UWE can reproduce both passes, and we have not tested this empirically. We will make that clear in the revision. In future work, we plan to investigate whether encoders on interface–module graphs can learn weight updates for neural network optimisation and fine-tuning.

Q8. Had all the methods converged by the end of training?

All models selected best validation checkpoint that was before 100th epoch, which suggests but does not establish convergence.

Additionally, we acknowledge that neither baselines nor UWE underwent meaningful baseline finetuning. As requested by another reviewer, we further tuned UWE and NG-GNN with all node readout (best baseline), detail in response to review 9Ykc.

UWE's test MSE fell from 0.270±0.044 to 0.226±0.034 after tuning (mean ± sample SD). Checkpoints used here were selected by best validation loss (epoch 78 for UWE and 89 for NG-GNN). Visual inspection of training curves strongly suggests convergence. We did not have sufficient compute budget to finetune all baselines over the discussion period.

Q9. Why predict 60 values, and how difficult is plain accuracy prediction? We wanted a harder standardised task less prone to ceiling effects than the CNN zoo (Table 1). Additionally, the split between the common and beyond cohort helps test transfer tests transfer to larger models.

To address the clean error question, we trained small UWE and all-node NG-GNN (our strongest baseline) from scratch on clean error. Both trained for 100 epochs (seed 1701). We chose learning rates and checkpoints on 84 validation networks and tested on 84 from seven held-out architectures.

ModelNormalised validation MSENormalised test MSE
Small UWE0.05610.0541
All-node NG-GNN0.09860.1344

Both models have considerably lower normalised test MSE on the scalar target than on the full profile, confirming our suspicion about potential ceiling effect.

I thank the authors for their detailed rebuttal and the additional experiments.

My main concern was the writing and presentation of the paper, a concern that other reviewers share. I appreciate the revised Sections 1–4. However, a rewrite of the full method description is a substantial change that needs to be checked carefully against the rest of the paper.

Some of my other concerns remain only partly resolved. The evaluation is still limited, while the larger-scale experiments show feasibility rather than predictive performance. UWE also seems substantially slower than the baselines.

For these reasons, I will keep my initial score.

Thank you for considering the revised manuscript and additional experiments. We would like to clarify four points.

1. Presentation and consistency of the revision.

Our general comment gives a section-by-section breakdown: Section 2 introduces the representation through a worked example and explicit definitions; Section 3 brings the state shapes and encoder equations into the main text; and Section 4 defines the compiler constraints and demonstrates their interaction. These changes address concerns raised here and in the QiDe discussion. The methods themselves are unchanged: we have revised their presentation, including moving the encoder equations from the appendix into Section 3.

2. Predictive performance on larger networks.

As reported in Q4 of Comment 2/2, we extended predictive evaluation from the original zoo's maximum of 9.1M parameters to trained source networks with up to 30M parameters.

Although modest by modern deep-learning standards, these networks are larger than those in several established weight-space benchmarks: Small CNN Zoo uses 4,970-parameter networks, the released MNIST-Transformer configuration of Transformer-NFN has approximately 19k trainable parameters, and SANE's ResNet-18 prediction benchmarks use approximately 12M-parameter networks. This comparison concerns these benchmarks, rather than claiming the largest models studied in weight-space learning.

3. Additional tasks and robustness checks.

As reported in Q3 of Comment 1/2 and Q9 of Comment 2/2, we also extended evaluation to zero-probe INR weight editing and scalar clean-error prediction. On 3×3 dilation of 10,000 held-out INRs per dataset, UWE achieved pixel MSE 0.02243 ± 0.00014 on MNIST (three seeds, sign-flip augmentation) and 0.02018 on Fashion-MNIST (one seed, no augmentation), versus 0.08717 and 0.07301 without editing.

Three-seed replications, per-architecture results and validation-only hyperparameter tuning are reported in W1–W2 and Q1 of our response to Reviewer 9Ykc. Predictive evaluation on larger networks remains confined to the transformer family.

4. Memory savings and runtime.

The transformer timings in W5 of Comment 1/2 and the QiDe thread show a runtime cost that narrows with width. We have now added a CIFAR-style ResNet/WRN sweep on one A100 SXM4 80 GB, in FP32 with a 70 GiB cap for both methods. Below are median forward times and peak inference allocations; entries are UWE / NG-GNN. Width multipliers are relative to ResNet-18, ResNet-34 and WRN-40-4.

Source (parameters)Forward (ms)Peak memory (GiB)
ResNet-18 ×1 (11.2M)164 / 1120.28 / 2.15
ResNet-18 ×2 (44.8M)253 / 3800.92 / 8.40
ResNet-18 ×3 (100.6M)413 / 4612.04 / 18.78
ResNet-18 ×4 (178.8M)644 / 7893.62 / 33.28
ResNet-34 ×1 (21.3M)291 / 2030.42 / 3.81
ResNet-34 ×2 (85.2M)458 / 4061.61 / 15.05
ResNet-34 ×3 (191.5M)771 / 8373.60 / 33.73
ResNet-34 ×4 (340.4M)1,212 / OOM6.38 / OOM
WRN-40-4 ×1 (9.0M)270 / 930.26 / 1.58
WRN-40-4 ×2 (35.8M)339 / 3180.67 / 6.19
WRN-40-4 ×3 (80.5M)470 / 3931.47 / 13.84
WRN-40-4 ×4 (143.0M)622 / 6342.60 / 24.53

Completed pairs use the same two randomly initialised sources and GPU. Both encoders use four rounds, with 68,801 / 98,273 parameters. OOM denotes a CUDA allocation failure.

Runtime varies with architecture and width, and UWE becomes faster than NG-GNN as source networks widen in this sweep. It overtakes NG-GNN for ResNet-18 at ×2–×4 width and ResNet-34 at ×3, and approximately matches it for the widest WRN, while using only about 11–16% of NG-GNN’s inference memory across completed comparisons. The relative speed of the two encoders is therefore architecture-dependent, rather than determined by source parameter count alone. Processing larger networks within a fixed memory budget is also a scalability advantage in its own right, as demonstrated by the feasibility experiments in the paper and here. At 340.4M parameters, UWE completes inference while NG-GNN exceeds the cap. Even where UWE incurs a runtime cost, its lower memory requirements extend the range of networks that can be processed within the available budget.

Comments on the paper

3 comments

We agree that the paper is dense and that the main text leaves too much for the reader to infer from Figure 2 and the appendix. We are revising the main text and aim to post it before the discussion period ends. Below we provide a shorter clarification note as a response to presentation issues raised across the reviews.

Motivation. Three questions motivate our work on generalisable weight-space learning. (1) Can a single representation and encoder accept a broad class of network architectures? (2) As these networks grow, how much memory and time does the encoder need to process their weights? (3) Once trained, can it make accurate predictions on unseen networks and architectures? Our construction addresses the first two tests. Tensor-based methods often need a new symmetry specification when the architecture changes. Neural graphs can accommodate different architectures, but the construction often requires custom modifications for more complex architectures and the size of the graph becomes difficult to manage for larger networks. The transformer zoo allows for testing predictive transfer between different architectures within a family. Transfer between families remains open.

Representation. A module is one occurrence of an operation (named after nn.Module). Consider a fully connected MLP with h=ReLU(Wx+b)h=\mathrm{ReLU}(Wx+b) and y=Vhy=Vh. The linear layer, the activation and the output map are three modules. A module's parameter blocks are its units. The first linear layer contains two units, the weight matrix WW and the bias vector bb. ReLU is a module with no parameter units.

Now consider what happens when we renumber the hidden neurons. Hidden coordinate jj indexes row jj of WW, entry jj of bb and column jj of VV. These axes share a coordinate space and must use the same ordering. We call this shared coordinate space an interface. If the hidden layer has width nn, all n!n! orderings preserve the network's function when applied consistently to these axes. In a standard fully connected MLP, each hidden layer can be renumbered independently, so a layer of width nn contributes one factor SnS_n to the group of hidden-neuron permutations.

A skip connection shows why an interface can span several hidden spaces. In a residual block s=h+f(h)s=h+f(h), the addition pairs coordinate jj of hh with coordinate jj of f(h)f(h). Renumbering one requires renumbering the other in the same way. The two branches and their sum therefore share one interface.

We can now say how a module connects to its interfaces. In Wx+bWx+b, the columns of WW use the input interface. The rows of WW and the entries of bb use the output interface. We call each such assignment of a tensor axis to an interface a \emph{binding}. In the MLP graphs of Kofinas et al., weights are edge features and biases are node features. Here both belong to the module, and their bindings specify how each is processed.

This gives two equivalent ways to draw Figure 2. In the hypergraph, interfaces are vertices and each module is a hyperedge joining the interfaces it uses. Drawing each module as a vertex in its own right gives the bipartite graph. Each edge joins a module to an interface and carries the role of that connection, such as input or output. ReLU's apparent self-loop in Figure 2 corresponds to two such edges: its input and output use the same interface.

Compiler. We trace the model in evaluation mode using example inputs. The trace records the operations, and the coordinate axes of the tensors between them are candidates for interfaces. Each supported operation has a rule describing how its coordinates can be renumbered. A glue requires two aligned coordinate intervals to be renumbered together. Pointwise activations and normalisation layers glue their input coordinates to their output coordinates. In the residual block s=h+f(h)s = h + f(h), addition glues both inputs, hh and f(h)f(h), to the output ss. All three therefore use the same coordinate ordering. A linear layer or an ungrouped convolution imposes no such requirement.

A split marks a boundary that permutations must preserve, for instance where two branches are concatenated. If hh is formed by concatenating branches of widths two and three, the split falls after its second coordinate. The glue from addition carries that boundary to f(h)f(h) and ss. The compiler propagates boundaries until no new ones appear, then groups linked intervals into interfaces. Here the first two coordinates of all three tensors share one interface and the remaining three share another. The operation occurrences and their connections to these interfaces give the final graph.

Figure 3 blurs the distinction between modules and units. Its caption calls the rectangular and diagonal parameter blocks modules, although Section 2.2 defines them as units. The encoder updates each unit separately (Appendix G.1). We will correct the terminology in the figure and caption.

This is a continuation of the note addressing presentation issues posted in the comment below.

States. For a linear layer with bias, UWE encodes each scalar weight or bias as a vector of width dEd_E. This initial encoding is what the paper calls the lift. The resulting tensors have shapes [nout,nin,dE][n_{\rm out},n_{\rm in},d_E] and [nout,dE][n_{\rm out},d_E]. Each occurrence of a parameter-free operation, such as ReLU, carries one vector of width dEd_E. An interface of width nIn_I carries nIn_I coordinate vectors, giving a tensor of shape [nI,dH][n_I,d_H]. It contributes just one pooled vector to global attention. We use narrower features for parameter cells (dE<dHd_E<d_H) because weights/parameters far outnumber interface coordinates. We additionally introduce vectors that identify operation types, parameter kinds and binding roles embeddings, replacing the term codes, to make the intend and use more intuitive to understand (by analogy to positional embedding of the transformer).

Encoder. For modelling, we find the bipartite view more natural because interfaces and modules play different roles in the encoder. Like a transformer layer, each UWE layer combines global attention with a local update analogous to the per-token feed-forward network. Each interface pools its coordinate states into one summary, and all summaries attend to one another. The pairwise cost of attention is therefore quadratic in the number of interfaces.

The local updates take place within modules. For example, for a linear layer with two units, the weight matrix and the bias vector, and UWE updates each unit separately. Within the matrix, each cell combines its own features with summaries of its row and column. Within the bias vector, each entry combines its own features with a summary of the whole vector. Other local updates can serve the same purpose provided they respect the coordinate permutations.

Since attention and local updates act on different sides of the graph, two steps connect them. First, interfaces send information to the modules that use them. Each module uses this information to update its units, then sends information from the updated units back to the interfaces. In the revision we will drop the names for the four passes and bring the equations for one round from Appendix G.1 into the main text.

We will answer the remaining questions in the individual review threads and report the requested experimental results there as they become available.

We have revised Sections 1–4 to address the presentation concerns raised in the reviews and incorporated the explanations from our clarity note into the main text. Thank you for the detailed feedback, which helped guide these changes.

  • Section 1 — Introduction. We expanded the motivation around three requirements for generalisable weight-space learning: accepting different architectures, controlling memory and runtime as source networks grow, and predicting accurately on unseen networks and architectures. We also removed unnecessary terminology and deferred definitions to the sections where they are needed.

  • Section 2 — Representation. We rewrote this section around a worked MLP example, extended with a residual connection. Modules, units, interfaces and bindings are now introduced and defined alongside the example, showing explicitly which parameter axes must be permuted together. We streamlined the terminology and clarified the distinction between operation occurrences and their parameter blocks. We also removed the bipartite graph view and now present the representation only as a hypergraph.

  • Section 3 — Encoder. We rewrote this section to explain the connection to transformer architectures, define the encoder’s states and their shapes, and give equations for one encoder layer. These cover attention, local updates, and communication between interfaces and modules. We replaced “codes” with “embeddings” and “lift” with “initialisation”, and removed the names previously assigned to the individual steps of an encoder layer.

  • Section 4 — Compiler. We now explicitly define the constraints used by the compiler—glue and split—and work through a residual network with concatenated branches. The example shows how the rules interact, why boundaries must propagate through linked axes, and how the resulting intervals become interfaces. We also explain how this reduces to the simpler residual MLP from Section 2.

The experimental results, discussion and appendix are unchanged in this update.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.