acceptodds
Under review as a conference paper at ICLR 2027

Universal Hypernetworks via Structured Conditioning Across Architectures and Tasks

Abstract

Conventional hypernetworks are typically designed around a specific target-network parameterization, so changing the target architecture often requires redesigning and retraining the hypernetwork. We introduce the Universal Hypernetwork (UHN), a fixed-architecture generator that predicts weights from deterministic parameter, architecture, and task descriptors. By moving target specificity into these descriptors, UHN decouples the generator design from the structure of the target model. We evaluate UHN across vision, graph, text, and formula-regression tasks, showing that a fixed generator architecture can be applied across different target structures. In addition, a single UHN supports multi-model generalization within a model family and multi-task learning across heterogeneous models. Finally, by treating a UHN itself as a target network, the generation process can be applied recursively.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Hypernetworks, Implicit Neural Representations, Neural Network Parameterization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SBHHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors propose a Universal HyperNetwork (UHN), which leverages deterministic encodings or target architectures and tasks to demonstrate both multi architecture and multitask generation, including recursive generation of subsequent UHN generations. Authors explore applications to a wide range of tasks and target architectures, and generally find that UHN generated models perform well, including signs of performance on previously unseen architectures.

Sources:

[1] HyperNetworks, David Ha and Andrew Dai and Quoc V. Le, https://arxiv.org/abs/1609.09106

[2] Generative Modeling of Weights: Generalization or Memorization? Boya Zeng, Yida Yin, Zhiqiu Xu, Zhuang Liu, https://openreview.net/forum?id=QbSeApdgfi

[3] Towards Scalable and Versatile Weight Space Learning, Konstantin Schürholt and Michael W. Mahoney and Damian Borth, https://arxiv.org/abs/2406.09997

Critical strengths

Strengths:

(1) The capability to generate across multiple tasks and architectures with a single Hypernetwork seems to be a genuinely novel contribution and is significant in its own right.

(2) The application of UHN to many types of objectives and model architectures is thorough, with many potentially promising results.

(3) Despite my concerns (see Weaknesses), the initialization training phase is an clear bridge between the previously disparate weight space generator and hypernetwork model families and a tidy contribution.

(4) Despite my concerns (see Weaknesses), producing a capable and fixed size hypernetwork method is a good contribution.

Critical weaknesses

Weaknesses:

(1) Comparisons against prior HyperNetwork (eg, Ha et. al. [1] ) methods are totally absent from the main body text, outside of Figure 3b’s parameter count comparison. As such it is quite challenging to discern if the presented UHN method is actually an advancement in terms of performance, and no tradeoffs between the universal or fixed architecture methods can be explored.

(2) In training, the UHN ‘sees’ weight indices and position embeddings, deterministic encodings, and is sometimes even initialized by matching the target parameter distributions. This strikes me as a recipe for memorization, a regime where the UHN becomes a very fancy lookup table. The authors do not investigate this possibility; there are no experiments done to measure and compare original or produced model functional or weight level diversity [2]. If this failure mode is in play, it explains the success of the UHN, the “outlier” in Figure 4a, and precisely why the initialization warmup helps. The discussion of task encodings in appendix E does not refute this concern, but potentially reinforces it.

(3) Figure 4a and 4b both show that the model is only asked to generalize within or only just slightly outside the training example range. However, it is possible that simply copying weights and padding with noise or the same distribution could achieve this same result. Perhaps SANE or some model soup method could be a basis of comparison [3]? A generalization signal would be more robust if there was a true hold-out regime, but in the current state the train and test distributions appear to heavily overlap, casting doubt on generalization claims.

Questions for authors

Questions:

(1) Table 1 reports error bars, but the sample size is not provided. What is nn?

(2) The apparent step change in test accuracy shown is Figure 3a seems surprising. Perhaps having real models or some other reference would clarify the message. This might also be an opportune place to compare against prior methods.

(3) The outlier in Figure 4a seems to be at most 5% outside the training range, yet indicates an almost 80% drop in absolute performance. Is this outlier persistent to multiple sampling attempts, or just an extremely unlucky n=1n=1 result? Perhaps adding a max and min score to Table 2 would illuminate this.

(4) What is the motivation for recursive UHN generation? It is a neat capability, but it does not seem well motivated in the text.

(5) The formula regression RMSE in Table 3 (also mentioned around line 431) is a +70% increase from single to multitask UHNs, with indicated multitask error variance of more than the RMSE of the entire single-task UHN. There is little context for this outcome. What’s happening here?

(6) How does the UHN method compare against other Hypernetwork and weight-generator methods? If there are tradeoffs, what are they?

My score could change if:

• The possibility of UHN memorizing training data is robustly investigated and refuted

• Comparisons against prior weight generative or HyperNetwork technique baselines are provided where possible

• More robust testing of UHN generalization to unseen architectures is demonstrated

2 comments

We thank the reviewer for recognizing the contributions of heterogeneous generation, initialization, and recursive generation. We address the concerns regarding memorization, extrapolation, baselines, and the specific questions below.

1. Memorization and architectural generalization

We agree that deterministic descriptors alone do not rule out memorization. We therefore conducted additional experiments on the CIFAR-10 mixed-depth CNN family described in the paper. Let (d1,d2,d3)(d_1,d_2,d_3) denote the three variable stage depths, with training depths 6<=di<=106<=d_i<=10.

Some unseen architectures fail primarily because of their generated output classifiers rather than their entire backbones. Replacing the classifier with one generated for (10,9,10)(10,9,10) by the same frozen UHN substantially restores accuracy without fine-tuning.

Stage depthsNativeClassifier transfer
(10,10,10)7.1785.23
(10,10,11)13.2384.96
(10,10,20)9.6883.23
(10,10,30)9.4278.91
(10,10,50)12.4471.43

Results are mean CIFAR-10 test accuracy (%) over 3 seeds, without parameter updates during transfer.

Architectures (10,10,11)(10,10,11) through (10,10,50)(10,10,50) exceed the training depth range. Their retained backbone functionality cannot be explained by simply retrieving complete training architectures, although reuse of learned layer patterns remains possible.

Motivated by the classifier failure, we hypothesized that positional encoding may dominate output-layer generation, making it difficult to maintain a consistent output-layer role across varying depths. We therefore tested a role-partition variant, assigning the classifier a fixed layer index of −1 while preserving other descriptors and normalization bounds.

This improves native accuracy to 85.63% on (10,10,10)(10,10,10) and 84.99% on (10,10,11)(10,10,11). Mean unseen accuracy across the original mixed-depth evaluation improves from 84.30±0.23% to 88.19±0.03%, with comparable seen performance.

Role partitioning is a new variant; classifier transfer demonstrates a capability of the originally submitted UHN. Extrapolation remains stage-dependent: extending earlier stages, e.g., (10,11,10)(10,11,10), can still fail even with classifier transfer. We will clarify this limitation rather than claim unrestricted architectural generalization.

2. Comparisons with prior methods

Comparisons with Transformer-HN, HA-HN, and Chunked-HN appear in Appendix C.6 and Table 15. Figure 3 uses Transformer-HN as a representative baseline for readability, but we agree this makes the broader comparisons less visible.

UHN's distinguishing contribution is a fixed generator formulation supporting heterogeneous architectures and tasks through parameter-wise structured conditioning, rather than superior accuracy or efficiency on every benchmark.

Unlike approaches tied to particular output shapes or chunks, UHN predicts weights at scalar granularity. This allows each weight to be generated from its own structural and task descriptors, avoiding architecture-specific generation heads, but introduces substantial computational overhead.

Regarding SANE and related weight-space methods, these primarily study representations or generation of pretrained weight distributions, whereas UHN directly generates weights through downstream task-loss optimization. Such approaches offer complementary perspectives, and we agree that weight-diversity comparisons would be valuable.

3. Specific questions

Q1. Sample sizes. Appendix C.3 specifies 3 random seeds for image/text tasks and 10 seeds for graph tasks. We will also clarify this in the relevant captions.

Q2. Figure 3a. The apparent step change occurs because the figure combines two scaling series: CNN-20/32/44/56 (increasing depth), followed by CNN-20-2/4/6 (increasing width). It is not a discontinuity within one scaling sequence. This is described in Appendix C.2 and Table 15; we will make the distinction explicit in the figure caption.

Q3. Figure 4a outlier. Additional three-seed experiments confirm that certain CNN architectures consistently exhibit severe native-generation failures. For example, (10,10,10)(10,10,10) achieves 7.17±1.15%, whereas classifier transfer achieves 85.23±2.12%. These results support a recurring classifier bottleneck, although they do not directly replicate every architecture in Figure 4a.

Q4. Recursive generation. Our motivation is a potential hierarchical, tree-like generation system, where each UHN generates a limited number of child UHNs, which in turn generate further UHNs or subsets of target networks. This could distribute generation responsibilities across a hierarchy rather than requiring one UHN to generate every target directly. Our experiments establish feasibility, not yet efficiency or scalability benefits.

Q5. Multi-task formula regression. We agree that the increased RMSE and variance indicate negative transfer. Sharing one generator across heterogeneous tasks introduces competing optimization demands that the current method does not explicitly resolve. The experiment demonstrates shared-generation feasibility, not positive transfer.

Q6. Hypernetwork trade-offs. As detailed in Appendix C.6 and Table 15, specialized hypernetworks can achieve better accuracy and computational efficiency. UHN deliberately predicts weights at scalar granularity, conditioning each prediction on its parameter index, local architecture, and task descriptors. This provides flexibility across target shapes without redesigning the generator, at the cost of greater generation and training overhead. Accelerating generation while preserving this flexibility is an important direction for future work.

We appreciate the reviewer's suggestions and will clarify the distinction between demonstrated architectural generalization, reusable generation mechanisms, and remaining limitations.

Reviewer_up6BOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a Universal Hypernetwork (UHN) framework that generates individual scalar weights of target networks using structured descriptors of parameter indices, model architectures, and tasks. This design enables a fixed-architecture hypernetwork to generate parameters for target models of different sizes, architectures, and tasks. The paper evaluates the feasibility of this approach through single-model generation, shared generation across architectures within a model family, multi-task learning across heterogeneous models, and recursive hypernetwork generation. The experiments cover image classification, graph node classification, text classification, and formula regression. The results demonstrate that UHN can provide a unified weight-generation mechanism across the evaluated models and tasks. However, the evaluation is largely limited to small-scale models and relatively simple tasks, and the current evidence is insufficient to fully support the paper's broad claims of universality.

Critical strengths

The paper presents a unified weight-generation framework that enables a fixed-architecture hypernetwork to generate parameters for target models of different sizes, architectures, and tasks. This is a potentially valuable research direction.

Critical weaknesses
  1. Insufficient experimental scale. The paper emphasizes universality across architectures, model scales, and tasks, but the experiments are largely limited to small-scale models (up to approximately 1.37M parameters in the multi-model setting) and relatively simple datasets such as MNIST and CIFAR-10. This creates a substantial gap between the experimental evidence and the broad claims made in the Introduction. Given the paper's objectives, further evaluation on representative architectures such as ResNet-34 and ViT-Tiny/Small/Base using ImageNet-1K is necessary. Although these experiments are considerably more challenging than the current settings, they are reasonable and important for assessing the framework's scalability, generalization, and claimed universality.

  2. Insufficient comparisons with existing methods. Previous hypernetwork methods, such as GHN and LoGAH, have explored parameter generation across different architectures. Although their generation mechanisms and training settings differ from UHN, comparisons under appropriately aligned settings would help establish the advantages of the proposed framework. The paper includes some hypernetwork baselines in the single-model scaling experiments, but lacks comprehensive comparisons with representative methods in its key multi-model and multi-task generation settings. Consequently, the practical advantages of UHN remain unclear.

  3. Insufficient analysis and presentation quality. The paper lacks depth analysis of the effectiveness of its core design choices. In addition, the overall writing, method illustrations, and presentation of experimental results need substantial improvement, which weakens the clarity and overall quality of the paper.

Questions for authors
  1. Experimental scale and universality: The current experiments are largely limited to small-scale models and relatively simple datasets, leaving a substantial gap between the experimental evidence and the broad claims made in the Introduction. Could the authors further evaluate UHN on representative architectures such as ResNet-34 and ViT-Tiny/Small/Base using ImageNet-1K to demonstrate its effectiveness on larger models and more challenging tasks?

  2. Comparisons with existing methods: Previous hypernetwork methods, such as GHN, have already explored parameter prediction across different architectures. Why does the paper lack sufficient comparisons with such relevant methods? Although their objectives and generation mechanisms differ from UHN, the authors should provide comparisons under appropriate experimental settings or clearly explain the distinctive advantages of the proposed framework.

1 comment

We thank the reviewer for recognizing the potential value of a unified weight-generation framework. We address the concerns regarding experimental scale, comparisons with prior methods, and design analysis below.

1. Experimental scale and universality

We agree that evaluations on larger architectures, such as ResNet-34 and ViT models on ImageNet-1K, would provide valuable additional evidence. However, Section 4.3 already demonstrates UHN's parameter-count scalability: a fixed 612,117-parameter UHN generates a 9.6M-parameter target model, approximately 14.7× larger than the generator.

We clarify that "universal" refers to a fixed generator architecture that accommodates heterogeneous target structures and tasks through structured parameter descriptors, rather than demonstrated effectiveness on arbitrary model scales. Our experiments cover CNNs, GNNs, Transformers, and KANs across image, graph, text, and regression tasks, including multi-architecture, multi-task, and recursive generation.

We acknowledge that parameter-count scalability does not establish competitive performance on ImageNet-scale architectures. We will clarify the scope of our claims and identify larger-scale evaluation as an important direction for future work.

2. Comparisons with prior hypernetworks

We agree that GHN and LoGAH are relevant approaches to architecture-conditioned weight generation. Our contribution is not the first demonstration of cross-architecture weight prediction, but a unified scalar-wise parameter-generation formulation that conditions each weight on its index, architectural structure, and task descriptors.

Unlike generators requiring predetermined output structures, UHN predicts individual weights without architecture-specific output heads or fixed output tensor shapes. This enables the same generator design to accommodate heterogeneous target architectures and tasks, although scalar-wise generation introduces substantial computational overhead.

We already compare UHN with Transformer-HN, HA-HN, and Chunked-HN in Appendix C.6 and Table 15. We acknowledge that UHN does not consistently outperform specialized hypernetworks in accuracy or efficiency.

GHN and LoGAH employ different parameter-generation and training mechanisms. We agree that controlled comparisons under aligned settings would further clarify their relative advantages. We will strengthen their discussion and make the existing baseline results more prominent.

3. Design ablations and architectural generalization

Existing ablations. We investigate Task-structure-encoder (TSE), Fourier index encoding, initialization, and generator capacity in the appendix. These experiments demonstrate the importance of structured parameter descriptors, the task-dependent contribution of TSE, and the optimization benefits of initialization.

Additional generalization analysis. We conducted further experiments on the CIFAR-10 mixed-depth CNN family described in the paper. Let (d_1,d_2,d_3) denote its three variable stage depths, with (6<=d_i<=10) during training.

We found that certain unseen architectures fail primarily because of their generated classifiers. For example, (10,10,10) achieves only 7.17±1.15% native accuracy, but replacing its classifier with one generated for (10,9,10) by the same frozen UHN restores accuracy to 85.23±2.12%, without fine-tuning.

This suggests that positional index encoding may exert a stronger influence on generation than parameter roles. We therefore introduced a role-partition variant assigning the output classifier a fixed positional index of −1, while preserving other descriptors and normalization bounds.

The modification improves native accuracy to 85.63±1.13% on (10,10,10) and 84.99±0.88% on the extrapolated (10,10,11), across three seeds. These results highlight the importance of explicitly representing parameter roles for architectural generalization.

Reviewer_HpMZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes UHN, a hypernetwork that predicts every weight of the target network one scalar at a time. Each query is built from descriptors: which weight it is (layer, weight/bias, coordinate), what architecture it belongs to (depth, widths, layer types) and what task it’s for. These get Fourier-feature encoded and go through a small residual MLP. There’s also an optional transformer encoder (the task-structure encoder) that pools the architecture/task descriptors into a conditioning vector that gets added in. Training just backprops the task loss through the generated weights, with an optional init stage first that matches the mean/std of each layer to standard init.

Critical strengths

The method is simple and well explained, and the same generator design works on a lot of different target types, including GAT and KAN.

The appendix is really thorough. All the sweeps, all the baselines, compute costs and ablations are there, and the limitations section is honest about compute and the gap to direct training.

Code and model-set seeds are included.

Critical weaknesses

In most single model experiments, the hyper network is larger than the networks its estimating weights for. These results show a small network can be reparametarized by a larger one trained ent to end, which is expected to work. The cases where UHN is smaller than the target it never beats the best chunked hypernetwork.

The claim that it generalizes to unseen architectures within a family isn't established. This tests Interpolation within the training range. The one real extrapolation where the model is asked to output a model thats deeper than anything in training is below random accuracy.

Another claim is that one uhn handles several different tasks but this shows no benefit. As shown in table 3, when a single UHN is shared across tasks, every task gets worse. The paper gives no offsetting benefit such as positive transfer, storage or compute.

Table 15 shows UHN is never the best hypernetwork and ranks last on 4 of 7 targets. Table 18 shows it costs up to 38× Direct’s and 17× Transformer-HN’s training time. The main-text Figure 3 shows only Transformer-HN, which hides this.

The title attributes UHN’s shared generation to structured conditioning on architecture and task descriptors, but the paper’s own ablation in appendix G3 shows that removing this conditioning changes little. Results stay the same on Mixed Width and AG News and improve on Transformer Mixed and KAN and only multi-task CIFAR-10 gets slightly worse. The authors themselves say that “the index-based generator already provides most of the information.”

Questions for authors

These things could improve my rating during the rebuttal phase:

Since the hypernetwork is often larger than the task model (table 1) I would like to see a setup where the hypernetwork is smaller. The appendix section G2 does this for CIFAR10, but not for the graph and text tasks where the gap to direct is already biggest. Does reducing the capacity of the hypernetwork break convergence?

If you were to take the currently trained UHN and evaluate it on architectures outside of the training range. For example 1 or 2 layers longer, or widths past the current max. Even a small performance drop as opposed to the model completely collapsing would be acceptable. Figure 4a has one architecture deeper than anything in training, and its quality collapses to below random.

1 comment

We thank the reviewer for the constructive feedback. We conducted additional experiments addressing both questions and clarify the scope of our contributions below.

Q1. Can UHN learn when smaller than its target?

We extended the capacity experiments (Appendix G.2) to CiteSeer GCN and AG News Transformer:

  • CiteSeer: Direct 69.96±0.51%, original UHN 65.29±1.90%, reduced UHN 62.95±3.32% (10 runs).
  • AG News: Direct 91.86±0.08%, original UHN 90.99±0.05%, reduced UHN 90.41±0.12% (3 runs).

The reduced UHN has 158,613 parameters, fewer than both targets (237,446 for CiteSeer; 378,372 for AG News), corresponding to 66.8% and 41.9% of their sizes.

Both models learn successfully without training collapse. The accuracy reduction is modest, particularly on AG News. Importantly, we retained the original training schedules, which were limited to mitigate overfitting. Smaller generators may require more optimization steps; thus, these results establish successful learning under a fixed budget rather than full convergence.

Q2. Can UHN extrapolate beyond its training range?

We conducted additional experiments on the CIFAR-10 mixed-depth CNN family described in the paper. We denote the three variable stage depths by (d1,d2,d3)(d_1,d_2,d_3), where training depths satisfy 6<=di<=106<=d_i <=10.

Diagnosing the failure. We found that certain unseen architectures fail primarily because of their generated classifiers. For example, (10,10,10)(10,10,10) achieves only 7.17% native accuracy. Replacing its classifier with one generated for (10,9,10)(10,9,10) by the same frozen UHN restores accuracy to 85.23% without fine-tuning.

This suggests that the positional index encoding may dominate certain aspects of generation, making it difficult to consistently distinguish output classifiers from intermediate layers as architecture depth changes.

We therefore tested a role-partition variant, assigning the final classifier a fixed layer index of −1 while preserving other descriptors and normalization bounds.

Additional results (3 seeds, test accuracy %):

Stage depthsOriginalTransferRole-partition
(10,10,10)7.1785.2385.63
(10,10,11)13.2384.9684.99
(10,10,12)9.3584.5984.42
(10,10,20)9.6883.2375.25
(10,10,50)12.4471.4348.68

"Transfer" replaces the classifier with that generated for (10,9,10)(10,9,10) by the same checkpoint, without parameter updates.

Two findings emerge. First, the original UHN generates transferable backbones far beyond the training range: even (10,10,50)(10,10,50) achieves 71.43% with classifier transfer. Second, role partitioning substantially improves native extrapolation: (10,10,11)(10,10,11) reaches 84.99% without swapping classifiers.

Across the original mixed-depth evaluation, role partitioning improves mean unseen accuracy from 84.30±0.23% to 88.19±0.03%, while preserving seen accuracy (88.42±0.31% vs. 88.37±0.22%).

Role partitioning is a new variant; the transfer results demonstrate capabilities of the originally submitted UHN. Extrapolation remains stage-dependent: (10,11,10)(10,11,10) and (11,10,10)(11,10,10) still fail even with classifier transfer. We acknowledge this limitation.

Multi-task sharing

We agree that joint training exhibits negative transfer. However, the experiment aims to demonstrate that one fixed UHN can generate heterogeneous architectures and tasks, not necessarily positive transfer.

It also provides a concrete storage advantage: the shared UHN contains 663,151 parameters, compared with 1,279,920 across the separate target models—a 48.2% reduction in learned parameters. This benefit comes with an accuracy trade-off; we do not claim reduced training computation.

Structured conditioning and Task-structure-encoder

We clarify that structured conditioning refers to the overall descriptor-based framework, not solely to the optional Task-structure-encoder (TSE). Even without TSE, parameter queries contain structured information about indices, parameter types, architecture, and task.

We agree that TSE provides limited and task-dependent additional benefits. The ablation indicates that the descriptors themselves already carry substantial structural information. Our role-partition experiments further demonstrate the importance of how these descriptors represent parameter roles.

Accuracy and efficiency

We acknowledge that UHN does not consistently outperform specialized hypernetworks in accuracy or efficiency. Its contribution is a unified generation formulation across heterogeneous target architectures and tasks, rather than superior performance on every benchmark.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.