- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SAHyper is a graph hypernetwork that generates model weights from computational graphs.
Specifically, it introduces two components: (1) a multi component decoder, which sums low-rank factors to increase the rank of generated weights efficiently, and (2) structure-aware spectral inheritance, which combines pre-trained singular values with generated orthonormal bases and shares selected bases across related attention matrices.
Experiments cover several downstream tasks, including image classification, multitask learning, and unseen tasks transferability.
Critical strengths
-
S1. The paper clearly identify the limitations of existing methods.
-
S2. Experiments are comprehensive that they cover diverse downstream tasks.
-
S3. Reduction in parameter cost is promising in the use cases.
Critical weaknesses
-
W1. Missing end-to-end training-time comparison. Reducing training cost is a key motivation for this line of research, yet the paper does not establish this benefit through end-to-end runtime comparisons. Although the authors analyze decoder parameter complexity and report implementation-level efficiency improvements, they should also compare wall-clock training time against competing methods, ideally reporting the time required to reach comparable accuracy.
-
W2. Insufficient motivation for basis sharing. The observed overlap between attention subspaces is intriguing, but it does not directly justify enforcing shared bases across attention matrices. Could the authors provide clearer intuition connecting the empirical observations to this design choice, particularly why exact basis sharing is appropriate rather than a weaker alignment constraint?
-
W3. Unclear high-level presentation of the method. The main method figure focuses heavily on implementation details, making it difficult to grasp the central idea and how the proposed components work together. The authors should revise the figure to provide a clearer high-level overview, highlighting the roles of the two pathways and their key differences from prior methods.
Questions for authors
See Weaknesses.
2 comments
Thank you for your responses. I maintain my score from the positive side.
Response to Reviewer v69k
Thank you for your questions and suggestions. We address them individually below.
Q1. Reducing training cost is a key motivation for this line of research, yet the paper does not establish this benefit through end-to-end runtime comparisons. Although the authors analyze decoder parameter complexity and report implementation-level efficiency improvements, they should also compare wall-clock training time against competing methods, ideally reporting the time required to reach comparable accuracy. (W1)
A1. Table 1 below provides the performance and training costs of the complete PITH and SAHyper hypernetworks. With similar hypernetwork parameter counts, SAHyper achieves higher average accuracy and a lower calculated training cost.
Table 1. Performance and cost comparison of PITH and SAHyper trained on ImageNet-1K for 200 epochs, with costs calculated for a single H100.
In addition, our training pipeline optimizations reduce CPU memory usage from approximately 200–294+ GB to 58–63 GB and model library disk storage from 96.4 GB to 222 MB (Section 5.4, Table 7, and Appendix A.10 in the paper).
Q2. The observed overlap between attention subspaces is intriguing, but it does not directly justify enforcing shared bases across attention matrices. Could the authors provide clearer intuition connecting the empirical observations to this design choice, particularly why exact basis sharing is appropriate rather than a weaker alignment constraint? (W2)
A2. SAHyper follows PITH's two-path framework: one path directly generates target model parameters, while the other inherits pretrained parameters. Shared bases constrain only the inheritance path, leaving the direct generation path unrestricted in this respect. SAHyper adaptively balances the two paths through learnable fusion. Thus, shared bases provide structural guidance for the overall model, rather than imposing a hard constraint that the final generated parameters must share bases exactly (Section 3 of the paper).
We directly examine the ViT-Base attention weights generated by the trained hypernetworks. Compared with PITH, SAHyper's Q/K output subspaces are more aligned within each layer, and the gap in Subspace Affinity between within-layer and cross-layer comparisons is closer to the pretrained observation (Figure 2(b) in the paper and Table 2 below).
Table 2. Comparison of Q/K output subspace relationships in ViT-Base parameters from different sources. Subspace Affinity reports the mean within-layer affinity across 12 layers, and Affinity Gap is the within-layer mean minus the cross-layer mean.
These results show that the expected within-layer relationships are reflected in the final generated parameters, consistent with the pretrained observations. The generated models also achieve higher accuracy (Table 5 in the paper).
Q3. The main method figure focuses heavily on implementation details, making it difficult to grasp the central idea and how the proposed components work together. The authors should revise the figure to provide a clearer high-level overview, highlighting the roles of the two pathways and their key differences from prior methods. (W3)
A3. The original paper provides an overall SAHyper workflow in Figure 7 of the appendix, referenced in Section 3. We agree that a clearer overview would help, and have added Figure 8 in Appendix B. The new figure compares PITH and SAHyper side by side, showing the roles of the two paths and the improvements introduced by the MC decoder and SA Inherit.