acceptodds
Under review as a conference paper at ICLR 2027

SAHyper: Breaking the Rank Bottleneck and Rethinking Pretrained Weight Inheritance in Graph Hypernetworks

Abstract

Graph HyperNetworks (GHNs) generate parameters from computational graphs, enabling a single generator to serve models across scales and tasks. Recent methods incorporate pretrained knowledge to improve the quality of generated parameters. However, existing decoders limit the rank of generated weights, and increasing their rank is costly. Existing weight inheritance also treats related pretrained attention matrices independently. We introduce SAHyper, a Structure-Aware Hypernetwork that improves both direct generation and pretrained weight inheritance. Its Multi Component Decoder sums low-rank components to enable higher-rank weight generation with a compact hypernetwork. Structure-Aware Spectral Inheritance combines pretrained singular values with generated orthogonal bases and coordinates related bases within attention blocks. On ImageNet-1K, SAHyper generates a ViT-Base model that achieves 63.86% Top-1 accuracy without further training, surpassing the SOTA method by 10.5 points. It also outperforms the SOTA method on Visual Domain Decathlon and sets of 14 and 20 image classification tasks, with gains maintained after further training and on unseen tasks. SAHyper also outperforms the SOTA method in ResNet and LLaMA experiments.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
hypernetwork, model initialization, parameter prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_v69kOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SAHyper is a graph hypernetwork that generates model weights from computational graphs.

Specifically, it introduces two components: (1) a multi component decoder, which sums low-rank factors to increase the rank of generated weights efficiently, and (2) structure-aware spectral inheritance, which combines pre-trained singular values with generated orthonormal bases and shares selected bases across related attention matrices.

Experiments cover several downstream tasks, including image classification, multitask learning, and unseen tasks transferability.

Critical strengths
  • S1. The paper clearly identify the limitations of existing methods.

  • S2. Experiments are comprehensive that they cover diverse downstream tasks.

  • S3. Reduction in parameter cost is promising in the use cases.

Critical weaknesses
  • W1. Missing end-to-end training-time comparison. Reducing training cost is a key motivation for this line of research, yet the paper does not establish this benefit through end-to-end runtime comparisons. Although the authors analyze decoder parameter complexity and report implementation-level efficiency improvements, they should also compare wall-clock training time against competing methods, ideally reporting the time required to reach comparable accuracy.

  • W2. Insufficient motivation for basis sharing. The observed overlap between attention subspaces is intriguing, but it does not directly justify enforcing shared bases across attention matrices. Could the authors provide clearer intuition connecting the empirical observations to this design choice, particularly why exact basis sharing is appropriate rather than a weaker alignment constraint?

  • W3. Unclear high-level presentation of the method. The main method figure focuses heavily on implementation details, making it difficult to grasp the central idea and how the proposed components work together. The authors should revise the figure to provide a clearer high-level overview, highlighting the roles of the two pathways and their key differences from prior methods.

Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer v69k

Thank you for your questions and suggestions. We address them individually below.

Q1. Reducing training cost is a key motivation for this line of research, yet the paper does not establish this benefit through end-to-end runtime comparisons. Although the authors analyze decoder parameter complexity and report implementation-level efficiency improvements, they should also compare wall-clock training time against competing methods, ideally reporting the time required to reach comparable accuracy. (W1)

A1. Table 1 below provides the performance and training costs of the complete PITH and SAHyper hypernetworks. With similar hypernetwork parameter counts, SAHyper achieves higher average accuracy and a lower calculated training cost.

MethodGHN params (M)GPU-hoursAverage Acc. (%)
PITH50.5289550.99
SAHyper50.1525559.80

Table 1. Performance and cost comparison of PITH and SAHyper trained on ImageNet-1K for 200 epochs, with costs calculated for a single H100.

In addition, our training pipeline optimizations reduce CPU memory usage from approximately 200–294+ GB to 58–63 GB and model library disk storage from 96.4 GB to 222 MB (Section 5.4, Table 7, and Appendix A.10 in the paper).

Q2. The observed overlap between attention subspaces is intriguing, but it does not directly justify enforcing shared bases across attention matrices. Could the authors provide clearer intuition connecting the empirical observations to this design choice, particularly why exact basis sharing is appropriate rather than a weaker alignment constraint? (W2)

A2. SAHyper follows PITH's two-path framework: one path directly generates target model parameters, while the other inherits pretrained parameters. Shared bases constrain only the inheritance path, leaving the direct generation path unrestricted in this respect. SAHyper adaptively balances the two paths through learnable fusion. Thus, shared bases provide structural guidance for the overall model, rather than imposing a hard constraint that the final generated parameters must share bases exactly (Section 3 of the paper).

We directly examine the ViT-Base attention weights generated by the trained hypernetworks. Compared with PITH, SAHyper's Q/K output subspaces are more aligned within each layer, and the gap in Subspace Affinity between within-layer and cross-layer comparisons is closer to the pretrained observation (Figure 2(b) in the paper and Table 2 below).

Weight SourceSubspace AffinityAffinity Gap
Pretrained ViT-Base (Fig. 2b)0.30890.3093
PITH0.06200.0341
SAHyper0.42200.2659

Table 2. Comparison of Q/K output subspace relationships in ViT-Base parameters from different sources. Subspace Affinity reports the mean within-layer affinity across 12 layers, and Affinity Gap is the within-layer mean minus the cross-layer mean.

These results show that the expected within-layer relationships are reflected in the final generated parameters, consistent with the pretrained observations. The generated models also achieve higher accuracy (Table 5 in the paper).

Q3. The main method figure focuses heavily on implementation details, making it difficult to grasp the central idea and how the proposed components work together. The authors should revise the figure to provide a clearer high-level overview, highlighting the roles of the two pathways and their key differences from prior methods. (W3)

A3. The original paper provides an overall SAHyper workflow in Figure 7 of the appendix, referenced in Section 3. We agree that a clearer overview would help, and have added Figure 8 in Appendix B. The new figure compares PITH and SAHyper side by side, showing the roles of the two paths and the improvements introduced by the MC decoder and SA Inherit.

Thank you for your responses. I maintain my score from the positive side.

Reviewer_uJ91OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SAHyper, which changes both generation paths of PITH, a graph hypernetwork that fuses directly generated weights with weights inherited from a pretrained model. The Multi Component decoder forms each weight as a weighted sum of several low-rank factor products, so the attainable rank grows linearly with the number of components while decoder size grows linearly rather than quadratically. Structure-Aware Spectral Inheritance keeps the pretrained singular values of the matched layer, generates only orthonormal bases, and ties the Q, K, V and O matrices of an attention block to three shared bases. Validation covers direct evaluation of generated ViTs on ImageNet-1K, further training, multitask generation on Decathlon and on two larger task sets, three unseen tasks, ResNets, and LLaMA language modeling, plus a component ablation, a decoder-size sweep, and rank measurements of the generated weights.

Critical strengths
  • The headline gain is large and consistent. Generated ViT-Small and ViT-Base models gain about ten points of ImageNet accuracy over the direct predecessor without any training, and the improvement has the same direction at every target size and on every benchmark. The ablation also shows that the new decoder alone, with no pretrained knowledge, already beats PITH on the larger targets. That is a useful finding about where prior hypernetwork decoders were bottlenecked.

  • The design is motivated by direct measurements rather than intuition: how a trained ViT degrades under rank truncation, how the existing decoder's size grows with rank, and how the Q/K and residual subspaces of pretrained attention align. The method is simple and should be reproducible.

Critical weaknesses
  • The ablations do not isolate the named contributions. Attention coupling, the "structure-aware" element in the title, is never ablated for ViTs. The step from the spectral-inheritance variant to the full model changes several things at once (decoder type, retained rank, fusion coefficient, an added residual transformation, learned spectrum scaling), so the source of that gain is unknown. The rank claim is not tested against a single-component decoder of matched size, and the generated weights' effective rank sits only slightly above the cap of the baseline decoder. The paper therefore does not establish which mechanism produces the improvement.

  • Several comparisons are not like-for-like. In the twenty-task experiment the authors' own PITH re-run is at chance level on three tasks at every model size, while the same method does well on two of those tasks in the fourteen-task experiment. Correcting this would roughly halve the reported gap. TAL likewise sits at the majority-class rate on SVHN in both task sets but handles SVHN well on Decathlon. Further-training results use best-epoch selection and, on some Decathlon tasks, a per-task hyperparameter search for SAHyper only, against numbers copied from the PITH paper. Everything is run with a single seed, and several reported gains are under one point. No parameter-count or training-cost comparison with PITH is given, although a compact hypernetwork is a central claim.

Questions for authors
  • What does spectral inheritance without attention coupling achieve on ViTs, all else equal?
  • Why do PITH and TAL collapse to chance on several tasks in your multitask re-runs? Were those baselines tuned?
  • Did PITH's further-training numbers use the same best-epoch selection and per-task hyperparameter search as SAHyper?
2 comments
AuthorsOpenReview ↗

Response to Reviewer uJ91 (1/2)

Thank you for your questions and suggestions. We address them individually below.

Q1. Attention coupling, the "structure-aware" element in the title, is never ablated for ViTs. The step from the spectral-inheritance variant to the full model changes several things at once (decoder type, retained rank, fusion coefficient, an added residual transformation, and learned spectrum scaling), so the source of that gain is unknown. What does spectral inheritance without attention coupling achieve on ViTs, all else equal? (W1, Q1)

A1. SAHyper introduces two core improvements: the Multi Component Decoder (Section 3.1) and Structure-Aware Spectral Inheritance (Section 3.2). The latter first performs spectral inheritance by combining pretrained singular values with generated orthogonal bases, then shares bases across related attention matrices through Attention Coupling (AC). Appendix A.7 provides implementation details and combined configurations for the two paths, without introducing additional independent methodological contributions.

Table 5 in the paper already compares the MC decoder with LoGAH and SA Inherit with PITH. Neither the MC decoder nor LoGAH uses pretrained inheritance, allowing a direct comparison of basic parameter generation capability. To address the reviewer's question about AC in ViTs, Table 1 below adds Spectral Inherit (w/o AC), explicitly reporting spectral inheritance results without AC. This configuration already achieves higher average accuracy than PITH. SA Inherit with AC achieves an even higher average accuracy, and the full SAHyper performs best at all three model sizes.

ConfigurationViT-TinyViT-SmallViT-BaseAverage
LoGAH26.3144.8244.8538.66
MC decoder46.3557.5057.5653.81
PITH46.3553.2753.3550.99
Spectral Inherit (w/o AC)49.2955.9755.5153.59
SA Inherit48.9259.3358.9955.75
SAHyper51.7563.7963.8659.80

Table 1. Direct Top-1 accuracy (%) of generated 12-layer ViTs after 200 epochs of hypernetwork training on ImageNet-1K. Average is computed across the three model sizes.

Q2. The rank claim is not tested against a single-component decoder of matched size, and the generated weights' effective rank sits only slightly above the cap of the baseline decoder. (W1)

A2. The MC decoder is designed to efficiently generate high-rank weights (Figure 2(a) and Section 3.1 in the paper). Directly increasing the rank of LoGAH causes its dominant parameter cost to grow approximately quadratically. The MC decoder reduces this cost through multiple compact components. Following the settings and accuracy metric in Figure 6 of the paper, we additionally compare against single-component LoGAH with increased ranks. Table 2 below shows that LoGAH remains less accurate than the MC decoder even with more parameters.

DecoderDecoder params (M)Epoch-25 Acc. (%)
LoGAH-r9020.1917.11
LoGAH-r12838.4217.93
LoGAH-r19282.5220.31
LoGAH-r256143.4321.52
MC decoder (18$\times$32)24.1123.23

Table 2. Acc. (%) on sampled architectures at epoch 25 for LoGAH with different rank settings and the MC decoder.

Section 5.2 and Appendix A.8 of the paper already report median numerical ranks of 192/380/568.5 for weights generated by the MC decoder, clearly exceeding LoGAH's rank-90 limit. r95r_{95} measures spectral energy concentration, not a rank ceiling. High-rank representations do not require uniform spectral energy, and low-energy directions cannot simply be considered ineffective.

AuthorsOpenReview ↗

Response to Reviewer uJ91 (2/2)

Q3. No parameter-count or training-cost comparison with PITH is given, although a compact hypernetwork is a central claim. (W2)

A3. With similar hypernetwork parameter counts, SAHyper achieves higher average accuracy and a lower calculated training cost, as shown in Table 3 below.

MethodGHN params (M)GPU-hoursAverage Acc. (%)
PITH50.5289550.99
SAHyper50.1525559.80

Table 3. Performance and cost comparison of PITH and SAHyper trained on ImageNet-1K for 200 epochs, with costs calculated for a single H100.

In addition, our training pipeline optimizations reduce CPU memory usage from approximately 200–294+ GB to 58–63 GB and model library disk storage from 96.4 GB to 222 MB (Section 5.4, Table 7, and Appendix A.10 in the paper).

Q4. In the twenty-task experiment the authors' own PITH re-run is at chance level on three tasks at every model size, while the same method does well on two of those tasks in the fourteen-task experiment. Correcting this would roughly halve the reported gap. TAL likewise sits at the majority-class rate on SVHN in both task sets but handles SVHN well on Decathlon. Everything is run with a single seed, and several reported gains are under one point. Why do PITH and TAL collapse to chance on several tasks in your multitask re-runs? Were those baselines tuned? (W2, Q2)

A4. The 14-task and 20-task experiments in the paper extend beyond the Decathlon benchmark used by TAL/PITH to examine parameter generation over larger task sets. Different task sets correspond to different joint optimization problems, and a given task's performance is influenced by the other tasks trained jointly. Following the multitask settings in the original TAL/PITH papers, we evaluate the checkpoint at epoch 100 (Section 4.3 and Appendix A.3 in the paper). All methods follow fair and consistent training and evaluation protocols, and baseline configurations were determined through multiple rounds of preliminary tuning.

We have repeatedly observed instability in TAL/PITH multitask training, including failure to learn individual tasks or degradation later in training.

Even after excluding DTD, Oxford-IIIT Pet, and Food101 from both methods' original 100-epoch results, SAHyper still has an average accuracy advantage of 3.93/5.50/5.40 pp over the remaining 17 tasks (Tiny/Small/Base).

We examined historical checkpoint records and confirmed that PITH had not yet degraded on these three tasks at epoch 75. After evaluating the 75-epoch checkpoints of PITH and SAHyper under the same protocol, we find that SAHyper still outperforms PITH by 7.57/7.35/7.62 pp in average accuracy across all 20 tasks (Tiny/Small/Base). Thus, SAHyper's multitask performance advantage holds even before the baseline experiences degradation on these tasks.

To address seed sensitivity, we conduct additional paired tests with four fixed seeds and a uniform 10-epoch budget, keeping each method's training configuration fixed and changing only the random seed. PITH exhibits learning failures on different tasks across seeds and has lower overall accuracy and greater variability. SAHyper achieves higher average accuracy across tasks for all four seeds and all three model sizes, with smaller standard deviations across seeds (Table 4 below). This further supports its performance advantage and robustness across seeds under this fixed budget.

MethodViT-TinyViT-SmallViT-Base
PITH22.33 ±\pm 8.6724.13 ±\pm 9.7124.05 ±\pm 9.54
SAHyper53.74 ±\pm 3.6663.79 ±\pm 3.0063.72 ±\pm 3.23

Table 4. Average Acc. (%) of PITH and SAHyper after 10 epochs of training in the 20-task setting. Results are the mean and sample standard deviation across four seeds (1234, 2026, 42, and 2222).

Q5. Further-training results use best-epoch selection and, on some Decathlon tasks, a per-task hyperparameter search for SAHyper only, against numbers copied from the PITH paper. Did PITH's further-training numbers use the same best-epoch selection and per-task hyperparameter search as SAHyper? (W2, Q3)

A5. Yes. PITH and SAHyper use the same further-training settings, per-task hyperparameter search ranges and budgets, and best-epoch selection rule. Both report the best validation accuracy within the specified budget. Per-task tuning applies to both methods, and the cited PITH results also follow this protocol.

Reviewer_xbAqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAHyper, a graph hypernetwork for generating parameters of target neural networks. It introduces two components: a multi-component decoder that sums low-rank factors to increase the attainable rank of generated weights, and structure-aware spectral inheritance that combines pretrained singular values with generated orthonormal bases while coupling related attention matrices. Experiments cover ImageNet-1K, multitask image classification, transfer to unseen tasks, ResNets, and LLaMA models. The strongest result is a 10.51 percentage-point improvement over PITH for directly evaluated ViT-Base on ImageNet-1K.

Critical strengths
  1. Substantial improvements in direct parameter generation. On ImageNet-1K, SAHyper improves ViT-Base accuracy from 53.35% to 63.86% without further target-model training. Improvements are also reported after further training and across several task collections. These results suggest that the proposed parameterization could meaningfully improve the quality of generated initializations.

  2. A motivated approach with evidence across model families. The paper connects the limited rank of existing decoders to a concrete representational constraint and introduces attention-aware inheritance based on observed relationships between pretrained subspaces. Experiments beyond ViTs, including ResNets and LLaMA, provide useful evidence that the overall approach is applicable across different architectures.

Critical weaknesses
  1. The experiments do not sufficiently isolate the proposed contributions from changes in capacity and parameterization. Appendix A.7 states that the full model differs from the component configurations in several additional respects, including the retained spectral rank, architecture-conditioned fusion, residual transformations across components, and learned scaling of higher-order singular values. Consequently, Table 5 does not establish how much improvement comes specifically from multi-component decoding or attention coupling. Moreover, summing low-rank products is algebraically equivalent to a higher-rank factorization, so the key contribution should be demonstrated through a favorable accuracy–cost trade-off against appropriately configured single-component decoders. The paper lacks a convincing matched-budget comparison and a controlled separation of spectral inheritance, orthogonalization, and attention coupling. This limits the evidence for the claimed methodological advance.

  2. Several aspects of the evaluation leave uncertainty about the magnitude and reliability of the gains. In Table 11, PITH achieves near-chance accuracy on DTD, Oxford-IIIT Pet, and Food101. For ViT-Base, these three tasks account for approximately 7.83 percentage points of the reported 12.42-point average improvement. Avoiding such failures could itself be valuable, but their cause and reproducibility need clarification. In addition, some baseline results are imported from prior work, whereas SAHyper reports the best accuracy within its training budget and includes task-specific hyperparameter searches in parts of the Decathlon evaluation. The paper does not establish that model-selection rules and tuning budgets are equivalent. The newly trained experiments also use a single seed, leaving uncertainty particularly around the smaller gains. These issues weaken confidence in interpreting the reported improvements as a robust advantage under comparable conditions.

Questions for authors
  1. Can you provide controlled comparisons that keep retained rank, fusion, residual transformations, and spectral scaling fixed while separately varying multi-component decoding and attention coupling? These would help determine which proposed mechanisms explain the improvement.

  2. How does SAHyper compare with a single-component decoder under matched hypernetwork parameter counts or training compute? Please clarify the complete models’ parameter counts and runtime costs. Table 7 measures implementation optimizations rather than a comparison between final SAHyper and PITH.

  3. What causes PITH’s near-chance results on DTD, Oxford-IIIT Pet, and Food101 in the 20-task setting? Do these failures persist across seeds and reasonable hyperparameter settings? Please also clarify whether the further-training comparisons use equivalent checkpoint-selection rules and hyperparameter-search budgets.

2 comments
AuthorsOpenReview ↗

Response to Reviewer xbAq (1/2)

Thank you for your questions and suggestions. We address them individually below.

Q1. Can you provide controlled comparisons that keep retained rank, fusion, residual transformations, and spectral scaling fixed while separately varying multi-component decoding and attention coupling? The paper also lacks a controlled separation of spectral inheritance, orthogonalization, and attention coupling. (W1, Q1)

A1. SAHyper introduces two core improvements: the Multi Component Decoder (Section 3.1) and Structure-Aware Spectral Inheritance (Section 3.2). The latter first performs spectral inheritance by combining pretrained singular values with generated orthogonal bases, then shares bases across related attention matrices through Attention Coupling (AC). QR ensures the basis orthogonality required for spectral inheritance and is part of its construction. Appendix A.7 provides implementation details and combined configurations for the two paths, without introducing additional independent methodological contributions.

Table 5 in the paper (Section 5.1) already compares the MC decoder with LoGAH and SA Inherit with PITH, respectively. LoGAH is the low-rank decoder shared by TAL and PITH, so this comparison directly examines the improvement that MC brings to basic parameter generation. Table 1 below adds Spectral Inherit (w/o AC), yielding a two-step comparison: PITH →\rightarrow spectral inheritance →\rightarrow spectral inheritance + AC. The results show that MC improves direct generation. Spectral inheritance improves average accuracy, and SA Inherit with AC improves it further. The full SAHyper achieves the best results at all three model sizes.

ConfigurationViT-TinyViT-SmallViT-BaseAverage
LoGAH26.3144.8244.8538.66
MC decoder46.3557.5057.5653.81
PITH46.3553.2753.3550.99
Spectral Inherit (w/o AC)49.2955.9755.5153.59
SA Inherit48.9259.3358.9955.75
SAHyper51.7563.7963.8659.80

Table 1. Direct Top-1 accuracy (%) of generated 12-layer ViTs after 200 epochs of hypernetwork training on ImageNet-1K. Average is computed across the three model sizes. Neither LoGAH nor the MC decoder uses pretrained inheritance. Spectral Inherit (w/o AC) uses spectral inheritance without Attention Coupling.

Q2. Summing low-rank products is algebraically equivalent to a higher-rank factorization, so the key contribution should be demonstrated through a favorable accuracy–cost trade-off against appropriately configured single-component decoders. How does SAHyper compare with a single-component decoder under matched hypernetwork parameter counts or training compute? (W1, Q2)

A2. The MC decoder is designed to efficiently generate high-rank weights (Figure 2(a) and Section 3.1 in the paper). Directly increasing the rank of LoGAH causes its dominant parameter cost to grow approximately quadratically. The MC decoder reduces this cost through multiple compact components. Following the settings and accuracy metric in Figure 6 of the paper, we additionally compare against single-component LoGAH with increased ranks. Table 2 below shows that LoGAH remains less accurate than the MC decoder even with more parameters.

DecoderDecoder params (M)Epoch-25 Acc. (%)
LoGAH-r9020.1917.11
LoGAH-r12838.4217.93
LoGAH-r19282.5220.31
LoGAH-r256143.4321.52
MC decoder (18$\times$32)24.1123.23

Table 2. Acc. (%) on sampled architectures at epoch 25 for LoGAH with different rank settings and the MC decoder.

AuthorsOpenReview ↗

Response to Reviewer xbAq (2/2)

Q3. Please clarify the complete models’ parameter counts and runtime costs. Table 7 measures implementation optimizations rather than a comparison between final SAHyper and PITH. (Q2)

A3. With similar hypernetwork parameter counts, SAHyper achieves higher average accuracy (59.80% vs. 50.99%) and a lower calculated training cost (55 vs. 95 GPU-hours), as shown in Table 3 below.

MethodGHN params (M)GPU-hoursAverage Acc. (%)
PITH50.5289550.99
SAHyper50.1525559.80

Table 3. Performance and cost comparison of PITH and SAHyper trained on ImageNet-1K for 200 epochs, with costs calculated for a single H100.

In addition, our training pipeline optimizations reduce CPU memory usage from approximately 200–294+ GB to 58–63 GB and model library disk storage from 96.4 GB to 222 MB (Section 5.4, Table 7, and Appendix A.10 in the paper).

Q4. What causes PITH’s near-chance results on DTD, Oxford-IIIT Pet, and Food101 in the 20-task setting? Do these failures persist across seeds and reasonable hyperparameter settings? For ViT-Base, these three tasks account for approximately 7.83 percentage points of the reported 12.42-point average improvement. The newly trained experiments also use a single seed, leaving uncertainty particularly around the smaller gains. (W2, Q3)

A4. The 14-task and 20-task experiments in the paper extend beyond the Decathlon benchmark used by TAL/PITH to examine parameter generation over larger task sets. Following the multitask settings in the original TAL/PITH papers, we evaluate the checkpoint at epoch 100 (Section 4.3 and Appendix A.3 in the paper). Hypernetwork training and direct evaluation of generated models in the paper follow fair and consistent comparison protocols, and the training configurations for all methods were determined through multiple rounds of preliminary tuning.

We have repeatedly observed instability in TAL/PITH multitask training, including failure to learn individual tasks or degradation later in training. Even after excluding DTD, Oxford-IIIT Pet, and Food101, on which PITH degraded, from the original 100-epoch results, SAHyper still outperforms PITH by 3.93/5.50/5.40 pp in average accuracy over the remaining 17 tasks (Tiny/Small/Base).

We examined historical checkpoint records and confirmed that PITH had not yet degraded on these three tasks at epoch 75. After evaluating the 75-epoch checkpoints of PITH and SAHyper under the same protocol, we find that SAHyper still outperforms PITH by 7.57/7.35/7.62 pp in average accuracy across all 20 tasks (Tiny/Small/Base). Thus, SAHyper's multitask performance advantage holds even before the baseline experiences degradation on these tasks.

To address seed sensitivity, we additionally conduct paired tests with four fixed seeds and a uniform 10-epoch budget, keeping each method's training configuration fixed and changing only the random seed. PITH exhibits learning failures on different tasks across seeds and has lower overall accuracy and greater variability. SAHyper achieves higher average accuracy across tasks for all four seeds and all three model sizes, with smaller standard deviations across seeds (Table 4 below). This further supports its stronger multitask learning ability and robustness across seeds under this fixed budget.

MethodViT-TinyViT-SmallViT-Base
PITH22.33 ±\pm 8.6724.13 ±\pm 9.7124.05 ±\pm 9.54
SAHyper53.74 ±\pm 3.6663.79 ±\pm 3.0063.72 ±\pm 3.23

Table 4. Average Acc. (%) of PITH and SAHyper after 10 epochs of training in the 20-task setting. Results are the mean and sample standard deviation across four seeds (1234, 2026, 42, and 2222).

Q5. Some baseline results are imported from prior work, whereas SAHyper reports the best accuracy within its training budget and includes task-specific hyperparameter searches in parts of the Decathlon evaluation. Please also clarify whether the further-training comparisons use equivalent checkpoint-selection rules and hyperparameter-search budgets. (W2, Q3)

A5. PITH and SAHyper use the same further-training settings, per-task hyperparameter search ranges and budgets, and both report the best validation accuracy within the specified budget. The cited PITH results follow the same rule.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Paper Revision: Added Overview Figure

We thank the reviewer for the helpful suggestion. In response, we have added an overview figure (Figure 8, Appendix B). The rest of the paper remains unchanged.

AuthorsOpenReview ↗

Summary of Core Contributions and Responses to Key Reviewer Concerns

We thank the reviewers for their feedback. We briefly summarize the contributions of our work and the supplementary evidence.

This work studies the use of hypernetworks to directly predict the complete parameters of target neural networks.

Core contributions. SAHyper addresses the rank bottleneck in generated weights and the lack of structural modeling in pretrained weight inheritance. It introduces the MC decoder for efficient high-rank weight generation and SA Inherit for exploiting pretrained spectra and attention subspace relationships. Analysis of the generated weights verifies the intended rank and structural properties. Compared with the SOTA method PITH, full SAHyper improves the direct evaluation accuracy of ViT-Base on ImageNet-1K from 53.35% to 63.86%, without increasing the total number of hypernetwork parameters.

We summarize our responses to the reviewers’ main concerns below.

  1. Component contributions. We added a spectral inheritance configuration without Attention Coupling, which achieves higher average accuracy than PITH, while SA Inherit with Attention Coupling achieves further improvement. Together with the MC decoder comparison already provided in the paper, these comparisons cover the core designs in the Method section and jointly support the effectiveness of the improvements to both paths.

  2. Parameter efficiency. We added comparisons with single-component LoGAH configurations using increased ranks. The MC decoder achieves higher accuracy with substantially fewer parameters, consistent with the motivation presented in the Introduction and supporting its design objective of achieving high-rank generation with a compact hypernetwork.

  3. Multitask performance advantages. PITH’s task degradation reflects its stability limitations in multitask training. Supplementary comparisons show that SAHyper achieves more stable performance and higher average accuracy, and that its advantage is not solely attributable to PITH’s task degradation.

We also provided comparisons of overall model costs, clarification of the training protocols, and an overview figure in Appendix B.

We welcome further discussion and hope this summary helps the reviewers and the AC assess the contributions of this work and the supplementary evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.