- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies “cross-architecture transfer of context-internalization hypernetworks”. A context-internalization hypernetwork is a neural network that maps a document into LoRA parameter for a frozen LLM without normal LoRA training. A trained hypernetwork is tightly coupled to its backbone (input projection must match hidden dims; output generator must match layer count/dims), so adapting to a new LLM would normally require retraining from scratch, which is expensive in data and compute. The authors ask whether a trained source hypernetwork can be reused to make target hypernetwork training more effective under limited resources.
They first conduct a systematic cross-architecture analysis on a 600-document probe set across Gemma2-2B, Qwen3-4B and Mistral-7B, yielding three observations: (i) input projections across backbones are misaligned but can be recovered by a linear map; (ii) Perceiver encoder/decoder representations are substantially shared/transferable; (iii) the LoRA generation head is backbone-specific (near-full-rank spectrum).
Based on these findings they propose HYPERBRIDGE: ALIGN (fit a linear map and fold it into the target input projection), PRESERVE (initialize target Perceiver from source, plus low-rank LoRA adapters), and RE-LEARN (layer-interpolate the post-Perceiver, re-initialize the generation head), trained with the standard KL-distillation + L1 objective. Validation covers all six source→target directions at 5K/10K/15K/20K training steps on SQuAD/DROP/ROPES (with FineWeb-Edu pretraining), reporting up to 28.5% relative gains over from-scratch baselines, with ablations on input alignment, Perceiver adaptation, and data composition.
Critical strengths
-
Clear, well-motivated problem and a genuinely informative analysis. Introducing cross-architecture hypernetwork transfer as a problem is timely and practical, and the RSA/CKA-driven decomposition into "align / preserve / re-learn" components turns the analysis directly into an actionable recipe. This makes the paper read as an honest empirical study of where hypernetwork knowledge is reusable, which is consequential because it can guide future backbones without repeating the full pipeline.
-
Consistent, budget-aware empirical validation. The evaluation covers all six transfer directions, multiple datasets, and four training budgets; HyperBridge improves over from-scratch Native on most directions and budgets (14.7% at 5K, up to 28.5% on Mistral), with ablations isolating each design choice (input alignment, Perceiver adaptation, data composition). The robustness across data budgets directly supports the "limited resources" motivation.
Critical weaknesses
-
The analysis conclusions are tied to the specific Doc-to-LoRA architecture and their generality across the broader space of context-internalization approaches is not established. The paper fixes the hypernetwork decomposition into the "input projection P + Perceiver E + LoRA generator G" trio, and all three core observations (projection linearly alignable, Perceiver preservable, generator backbone-specific) are derived on this one specific architecture. However, context internalization admits other implementations: peer context→LoRA hypernetworks (e.g., SHINE) may use different compressors; task-conditioned hypernetworks (HyperLoRA/ZHyper, etc.) do not use a Perceiver at all; Shin et al. 2025, Qi et al. 2025, and Kujanpää et al. 2025 pursue per-context optimization/self-distillation rather than hypernetworks; and Caccia et al. 2025 follow a fundamentally different plug-n-play "knowledge module + deep context distillation" paradigm. The paper neither studies the sensitivity/robustness of its findings to architectural choices nor discusses how the framework would transfer to these alternatives. Consequently, the claimed generality of "which hypernetwork components are reusable across backbones" is only supported for Doc-to-LoRA-style LoRA-generating hypernetworks, which weakens its value to the broader community and, in turn, the significance of the contribution.
-
Limited generality and modest, noisy margins. Only small backbones (2B–7B), 512-token documents, extractive-QA ROUGE metrics, and no significance/variance reporting temper the headline result; several gains (especially DROP) are small and some Native@20K cells rival the transferred model. The "up to 28.5%" figure, while correct, sits atop a heterogeneous set of much smaller improvements, and the mismatch of evaluation metrics across targets makes cross-backbone comparisons fragile. The causal generality of the analysis (e.g., does the linear-alignability observation hold for larger or non-decoder-only backbones?) is untested.
Questions for authors
(Clarifying) Are the reported numbers from a single run or multiple seeds? If multiple, please report mean/variance; if not, are the observed gaps (often a few points on DROP) statistically meaningful?
8 comments
Re: We note that the review references several works without providing proper citations.
All the works I mentioned can be found in the References in your paper:
(Shin et al. 2025, Qi et al. 2025, and Kujanpää et al. 2025):
- Haebin Shin, Lei Ji, Yeyun Gong, Sungdong Kim, Eunbi Choi, and Minjoon Seo. Generative prompt internalization. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 7338–7363, 2025.
- Siyuan Qi, Bangcheng Yang, Kailin Jiang, Xiaobo Wang, Jiaqi Li, Yifan Zhong, Yaodong Yang, and Zilong Zheng. In-context editing: Learning knowledge from self-induced distributions. In International Conference on Learning Representations, volume 2025, pp. 77563–77585, 2025.
- Kalle Kujanp¨a¨a, Pekka Marttinen, Harri Valpola, and Alexander Ilin. Effi cient knowledge injection in LLMs via self-distillation. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=drYpdSnRJk.
(Caccia 2025):
- Lucas Caccia, Alan Ansell, Edoardo Ponti, Ivan Vuli´c, and Alessandro Sordoni. Training plug-n-play knowledge modules with deep context distillation. arXiv preprint arXiv:2503.08727, 2025.
We thank the reviewer for noticing that these works come from our paper's Introduction, where readers can clearly see that they are mentioned only as background knowledge on context internalization for LLMs.
We would like to emphasize that using these background citations to criticize a hypernetwork-based method for not applying to non-hypernetwork approaches might reflect a clear misunderstanding of the scope of our work.
We thank the reviewer for the thoughtful feedback. We address these concerns as follows.
(W1) Specificity to Doc-to-LoRA
We thank the reviewer for the comment, but would like to clarify several important distinctions.
We note that the review references several works without providing proper citations. We encourage the reviewer to provide precise references, particularly when using prior literature to substantiate a criticism of our paper.
The review conflates distinct CI paradigms. Several CI frameworks the author mentioned do not involve transferring a pre-trained context-to-parameter hypernetwork. Consequently, they do not face the cross-backbone generator-transfer problem addressed by HyperBridge.
Our philosophy extends beyond Doc-to-LoRA. For SHINE [1], we can naturally align backbone-specific memory states, preserve transferable M2P Transformer components, and re-learn target-specific modules. While the exact operations differ, the underlying principle of representation-guided selective transfer remains applicable. More details please refer to our response to Reviewer 3HSp (Q1).
(W2) Limited generality and noisy gains
We clarify each of the points as follows:
(Q1) Clarification question
The reported number in the paper is from a single run. To further address the reviewer's concern, we replicate our main experiment results in Table 1 with two additional random seeds and record mean ± std over three seeds. Mean metrics over three datasets are reported. As shown in the following tables, HyperBridge consistently outperforms Native across all target backbones and training budgets, demonstrating robust transfer gains.
[1] SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass, ICML 2026.