- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Authors propose a Universal HyperNetwork (UHN), which leverages deterministic encodings or target architectures and tasks to demonstrate both multi architecture and multitask generation, including recursive generation of subsequent UHN generations. Authors explore applications to a wide range of tasks and target architectures, and generally find that UHN generated models perform well, including signs of performance on previously unseen architectures.
Sources:
[1] HyperNetworks, David Ha and Andrew Dai and Quoc V. Le, https://arxiv.org/abs/1609.09106
[2] Generative Modeling of Weights: Generalization or Memorization? Boya Zeng, Yida Yin, Zhiqiu Xu, Zhuang Liu, https://openreview.net/forum?id=QbSeApdgfi
[3] Towards Scalable and Versatile Weight Space Learning, Konstantin Schürholt and Michael W. Mahoney and Damian Borth, https://arxiv.org/abs/2406.09997
Critical strengths
Strengths:
(1) The capability to generate across multiple tasks and architectures with a single Hypernetwork seems to be a genuinely novel contribution and is significant in its own right.
(2) The application of UHN to many types of objectives and model architectures is thorough, with many potentially promising results.
(3) Despite my concerns (see Weaknesses), the initialization training phase is an clear bridge between the previously disparate weight space generator and hypernetwork model families and a tidy contribution.
(4) Despite my concerns (see Weaknesses), producing a capable and fixed size hypernetwork method is a good contribution.
Critical weaknesses
Weaknesses:
(1) Comparisons against prior HyperNetwork (eg, Ha et. al. [1] ) methods are totally absent from the main body text, outside of Figure 3b’s parameter count comparison. As such it is quite challenging to discern if the presented UHN method is actually an advancement in terms of performance, and no tradeoffs between the universal or fixed architecture methods can be explored.
(2) In training, the UHN ‘sees’ weight indices and position embeddings, deterministic encodings, and is sometimes even initialized by matching the target parameter distributions. This strikes me as a recipe for memorization, a regime where the UHN becomes a very fancy lookup table. The authors do not investigate this possibility; there are no experiments done to measure and compare original or produced model functional or weight level diversity [2]. If this failure mode is in play, it explains the success of the UHN, the “outlier” in Figure 4a, and precisely why the initialization warmup helps. The discussion of task encodings in appendix E does not refute this concern, but potentially reinforces it.
(3) Figure 4a and 4b both show that the model is only asked to generalize within or only just slightly outside the training example range. However, it is possible that simply copying weights and padding with noise or the same distribution could achieve this same result. Perhaps SANE or some model soup method could be a basis of comparison [3]? A generalization signal would be more robust if there was a true hold-out regime, but in the current state the train and test distributions appear to heavily overlap, casting doubt on generalization claims.
Questions for authors
Questions:
(1) Table 1 reports error bars, but the sample size is not provided. What is ?
(2) The apparent step change in test accuracy shown is Figure 3a seems surprising. Perhaps having real models or some other reference would clarify the message. This might also be an opportune place to compare against prior methods.
(3) The outlier in Figure 4a seems to be at most 5% outside the training range, yet indicates an almost 80% drop in absolute performance. Is this outlier persistent to multiple sampling attempts, or just an extremely unlucky result? Perhaps adding a max and min score to Table 2 would illuminate this.
(4) What is the motivation for recursive UHN generation? It is a neat capability, but it does not seem well motivated in the text.
(5) The formula regression RMSE in Table 3 (also mentioned around line 431) is a +70% increase from single to multitask UHNs, with indicated multitask error variance of more than the RMSE of the entire single-task UHN. There is little context for this outcome. What’s happening here?
(6) How does the UHN method compare against other Hypernetwork and weight-generator methods? If there are tradeoffs, what are they?
My score could change if:
• The possibility of UHN memorizing training data is robustly investigated and refuted
• Comparisons against prior weight generative or HyperNetwork technique baselines are provided where possible
• More robust testing of UHN generalization to unseen architectures is demonstrated
2 comments
3. Specific questions
Q1. Sample sizes. Appendix C.3 specifies 3 random seeds for image/text tasks and 10 seeds for graph tasks. We will also clarify this in the relevant captions.
Q2. Figure 3a. The apparent step change occurs because the figure combines two scaling series: CNN-20/32/44/56 (increasing depth), followed by CNN-20-2/4/6 (increasing width). It is not a discontinuity within one scaling sequence. This is described in Appendix C.2 and Table 15; we will make the distinction explicit in the figure caption.
Q3. Figure 4a outlier. Additional three-seed experiments confirm that certain CNN architectures consistently exhibit severe native-generation failures. For example, achieves 7.17±1.15%, whereas classifier transfer achieves 85.23±2.12%. These results support a recurring classifier bottleneck, although they do not directly replicate every architecture in Figure 4a.
Q4. Recursive generation. Our motivation is a potential hierarchical, tree-like generation system, where each UHN generates a limited number of child UHNs, which in turn generate further UHNs or subsets of target networks. This could distribute generation responsibilities across a hierarchy rather than requiring one UHN to generate every target directly. Our experiments establish feasibility, not yet efficiency or scalability benefits.
Q5. Multi-task formula regression. We agree that the increased RMSE and variance indicate negative transfer. Sharing one generator across heterogeneous tasks introduces competing optimization demands that the current method does not explicitly resolve. The experiment demonstrates shared-generation feasibility, not positive transfer.
Q6. Hypernetwork trade-offs. As detailed in Appendix C.6 and Table 15, specialized hypernetworks can achieve better accuracy and computational efficiency. UHN deliberately predicts weights at scalar granularity, conditioning each prediction on its parameter index, local architecture, and task descriptors. This provides flexibility across target shapes without redesigning the generator, at the cost of greater generation and training overhead. Accelerating generation while preserving this flexibility is an important direction for future work.
We appreciate the reviewer's suggestions and will clarify the distinction between demonstrated architectural generalization, reusable generation mechanisms, and remaining limitations.
We thank the reviewer for recognizing the contributions of heterogeneous generation, initialization, and recursive generation. We address the concerns regarding memorization, extrapolation, baselines, and the specific questions below.
1. Memorization and architectural generalization
We agree that deterministic descriptors alone do not rule out memorization. We therefore conducted additional experiments on the CIFAR-10 mixed-depth CNN family described in the paper. Let (d1,d2,d3) denote the three variable stage depths, with training depths 6<=di<=10.
Some unseen architectures fail primarily because of their generated output classifiers rather than their entire backbones. Replacing the classifier with one generated for (10,9,10) by the same frozen UHN substantially restores accuracy without fine-tuning.
Results are mean CIFAR-10 test accuracy (%) over 3 seeds, without parameter updates during transfer.
Architectures (10,10,11) through (10,10,50) exceed the training depth range. Their retained backbone functionality cannot be explained by simply retrieving complete training architectures, although reuse of learned layer patterns remains possible.
Motivated by the classifier failure, we hypothesized that positional encoding may dominate output-layer generation, making it difficult to maintain a consistent output-layer role across varying depths. We therefore tested a role-partition variant, assigning the classifier a fixed layer index of −1 while preserving other descriptors and normalization bounds.
This improves native accuracy to 85.63% on (10,10,10) and 84.99% on (10,10,11). Mean unseen accuracy across the original mixed-depth evaluation improves from 84.30±0.23% to 88.19±0.03%, with comparable seen performance.
Role partitioning is a new variant; classifier transfer demonstrates a capability of the originally submitted UHN. Extrapolation remains stage-dependent: extending earlier stages, e.g., (10,11,10), can still fail even with classifier transfer. We will clarify this limitation rather than claim unrestricted architectural generalization.
2. Comparisons with prior methods
Comparisons with Transformer-HN, HA-HN, and Chunked-HN appear in Appendix C.6 and Table 15. Figure 3 uses Transformer-HN as a representative baseline for readability, but we agree this makes the broader comparisons less visible.
UHN's distinguishing contribution is a fixed generator formulation supporting heterogeneous architectures and tasks through parameter-wise structured conditioning, rather than superior accuracy or efficiency on every benchmark.
Unlike approaches tied to particular output shapes or chunks, UHN predicts weights at scalar granularity. This allows each weight to be generated from its own structural and task descriptors, avoiding architecture-specific generation heads, but introduces substantial computational overhead.
Regarding SANE and related weight-space methods, these primarily study representations or generation of pretrained weight distributions, whereas UHN directly generates weights through downstream task-loss optimization. Such approaches offer complementary perspectives, and we agree that weight-diversity comparisons would be valuable.