- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper addresses weight-space learning across neural network architectures. It introduces a typed interface–module graph, an automatic compiler for supported PyTorch models, and the Universal Weight Encoder (UWE), which combines local equivariant updates with global attention over interface summaries. Experiments evaluate predictive performance, transfer to unseen transformer architectures, and computational scalability.
Critical strengths
-
The interface-module representation is a valuable contribution. It supports a broad range of common operations while explicitly capturing dependencies between parameter axes that must be permuted jointly. The compiler can automatically derive interfaces and parameter bindings from supported model implementations, eliminating the need for manually specified symmetries for each combination of operations. This offers a promising foundation for processing diverse architectures with a shared encoder.
-
Computational scalability is demonstrated through substantial memory savings against the evaluated baselines (Tables 5 and 12). The paper clearly distinguishes computational feasibility on large networks from predictive performance at that scale and acknowledges the associated runtime tradeoff.
Critical weaknesses
-
The transformer-zoo results in Table 3 use a single training seed, and Table 4 provides no uncertainty estimates. This limits confidence in the stability of both the comparative improvements and the ablation conclusions. Repeated runs for small UWE and the strongest NG-GNN configuration, together with per-architecture results, would help establish robustness.
-
Appendix D.2 uses the same learning rate and training schedule across methods and acknowledges limited method-specific hyperparameter searches. Although the common training budget and alternative NG-GNN readout are useful controls, they do not establish that each method is comparably well optimized. A modest validation-only learning-rate sweep and convergence curves for the strongest NG-GNN would strengthen the comparison.
Questions for authors
-
Residual additions can cause many operation occurrences to share an interface. Could aggregation into shared interface states become a bottleneck as residual depth increases? A breakdown of the existing held-out-architecture results by depth would help assess this concern.
-
Appendices D.1 and H.2 describe learned probes that provide activation features for INR classification, following prior work. How much does UWE’s INR performance depend on these observations? An ablation with and without probe features would help distinguish the contribution of the weight representation from that of the probes.
-
Appendix G.1, Equation (18), defines a parameter-valued decoder, but the experiments focus on prediction tasks. Has this decoder been validated for weight-to-weight tasks such as weight editing? Additionally, what extensions would be needed to generate weights for a specified target architecture different from the input architecture? A clarification of the supported and empirically validated scope would be helpful.
3 comments
I thank the authors for the detailed response, informative new experiments, and transparency regarding the limitations.
W1 (Result stability): Resolved. The three-seed experiments confirm the main improvements, and the per-architecture results provide stronger evidence of robustness.
W2 (Baseline tuning): Resolved. The learning-rate and weight-decay sweeps, convergence results, and validation-selected checkpoints substantially strengthen the fairness of the comparison.
Q1 (Shared residual interfaces): Remains a concern. The depth-wise results suggest that UWE's advantage diminishes for deeper architectures, potentially due to information loss at shared interfaces. The authors propose increasing interface width as a mitigation, but this remains untested. Moreover, increasing representation dimensionality does not necessarily recover information lost through averaging, and wider interfaces should generally incur additional computational costs. An experiment varying interface width, reporting both predictive performance and memory/runtime costs, would help address this concern. This is particularly important given the prevalence of deep residual connections in modern architectures, including Transformers.
Q2 (Probe dependence): Resolved, with a remaining observation. The results suggest that UWE benefits substantially more from activation probes than the baselines, rather than consistently learning a superior weight-only representation. Since the methods incorporate probes differently, it remains unclear whether this difference stems from UWE's architecture or its probe-integration mechanism. Nevertheless, this is still a meaningful contribution. A small ablation isolating the probe-integration mechanism would be valuable if feasible, but I do not consider it a blocker.
Q3 (Weight editing): Resolved. Although UWE does not outperform the strongest baseline on weight editing, the results demonstrate a meaningful ability to perform weight transformations. It would be interesting to evaluate this capability on larger models, where UWE's scalability advantages may become particularly relevant.
Overall, the rebuttal substantially strengthens the paper and resolves most of my concerns. However, I am keeping my score unchanged for now, primarily because Q1 raises an important question about the method's applicability to deeper architectures with extensive residual connections. I would appreciate further experimental evidence addressing this potential bottleneck and the computational trade-offs of the proposed mitigation.
Thank you for the follow-up and for recognising the additional evidence on stability, tuning and weight editing.
We tested wider interface states, increasing from 32 to 64 and 128. These single-seed experiments did not show a consistent improvement on deeper architectures, so they do not confirm our initial hypothesis that increasing interface width would address the observed weakness.
We would like to clarify that, even when residual blocks share an interface, each module and its weights remain separately represented. The encoder also maintains separate learned states for their parameters.
A more specific hypothesis therefore concerns how the current encoder aggregates and reads out this information. UWE's current invariant network-level readout pools only interface states, so information held in module states must first pass through interface aggregation to influence the prediction. When many modules share an interface, this aggregation may make their individual contributions less accessible to the readout. This is a plausible mechanism, rather than an established explanation of the depth-dependent results.
A concrete direction for the encoder is to use an invariant readout head that combines module and interface states, giving predictions direct access to both while preserving the representation’s symmetries. The representation already supports this design choice. We will report the negative width result and make clear that the observed depth-dependent performance remains a potential limitation of the current UWE configuration.
W1. Result stability. We repeated small UWE and the all-node NG-GNN (the strongest baseline) on the original 1,200-model zoo (three seeds, original splits, 100 epochs). Validation selected each checkpoint. We report mean test MSE for the 60-output profile ± sample SD across seeds.
On the full split (70/15/15 train/validation/test architectures), small UWE scored 0.318±0.012 and NG-GNN 0.384±0.013. Table 3 reported 0.3355 and 0.3637 for these models. UWE had lower error on 14/15 architectures (Q1). On the common split (43/7/7), the means were 0.270±0.044 and 0.385±0.055, against the reported 0.2790 and 0.4411.
We rerun the ablation that removes the global attention and report confidence intervals (three seeds on the common split). This scores 0.781±0.086 as compared to 0.84 reported in the submission.
W2. Baseline tuning. On the common validation split we tried learning rates 0.0001, 0.0003, 0.0006 and 0.001 for both models, then varied weight decay at the two best rates. Below is validation MSE for the selected recipes (seed 1701).
Best validation MSE was 0.248 at epoch 78 for UWE (rate 0.0006, decay 0.001) and 0.349 at epoch 89 for NG-GNN (0.001, decay 0). Those checkpoints scored 0.237 and 0.366 on the test networks. Both methods report gains from finetuning.
Q1. Can shared residual interfaces become a bottleneck? Each operation retains its parameter states, but averaging messages at a shared interfaces can lose information. At 2, 3, 4 and 6 blocks, mean test MSE for small UWE / NG-GNN was 0.196/0.363, 0.313/0.362, 0.288/0.344 and 0.439/0.457. The advantage was smallest at six blocks.
Each architecture (blocks/width/heads/feed-forward ratio) has 12 test networks. The UWE pair uses seed 715. The last two columns give three-seed means ± SD.
At six blocks, large UWE improved on two of four architectures. Width, heads and feed-forward ratio also vary, so depth alone cannot explain this pattern.
UWE can widen the vector at each interface coordinate (dH) independently of the parameter-cell width (dE). This could ease the bottleneck without causing memory or runtime degradation. We have not tested the effect of varying dH alone.
Q2. Effect of probes. We varied only the probe count at seed 715 (300 epochs on MNIST, 100 on Fashion-MNIST). Each validation-selected checkpoint was evaluated on 10,000 test INRs.
NG-GNN/T, ScaleGMN, and UWE all use 64 probes in Table 2.
Adding 64 probes gave UWE a larger accuracy gain on both datasets in our single-seed comparison. The experiment leaves open why the gain was larger. We suspect this might be due to the way each methods uses the probes. Neural graph methods project the probes and add them to the representations of nodes. In UWE, probes are used to intialise interface states, which might give a more direct way to combine what a neuron does with how it connects to the rest of the network.
Q3. Can the decoder edit weights or change architectures? Eq. (18) predicts an additive update per INR weight or bias. On 10,000 held-out MNIST INRs, zero-probe dilation gave pixel MSE 0.02243±0.00014 on the three seeds. Unchanged weights gave 0.08717. Published zero-probe MSE was 0.0238 for NG-GNN and 0.0196 for NG-T. [Kofinas et al., Table 5]. On 10,000 Fashion-MNIST test INRs, zero-probe UWE reduced dilation MSE from 0.07301 to 0.02018 (one seed, no sign augmentation).
For a specified target architecture that differs from the source, we could use UWE to encode the source network. A UWE-style decoder on the target graph would then use the resulting embedding to predict the target weights. Developing and testing such a decoder would require substantial further work, including the choice of suitable training datasets. We will make clear in the revision that our experiments cover only editing within the source architecture.