- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates LoRA fine-tuning for multi-head attention, focusing on the potential mismatch between the shared latent space and head-specific adaptation needs. To address this issue, the authors propose HyRA, which introduces a input-dependent Householder-like transformation for each attention head. The transformation selectively modulates latent features along input-dependent direction while preserving the orthogonal residual components. Experiments across different datasets and model architectures demonstrate that HyRA achieves improved parameter efficiency compared with LoRA.
Critical strengths
- The paper provides a well-motivated approach to addressing head-wise heterogeneity in LoRA fine-tuning. The observation in Figure 2 is particularly interesting, as the diminishing performance gains of standard LoRA under increasing head-wise heterogeneity suggest potential limitations of the shared low-rank latent space.
- The proposed Householder-like adaptation mechanism is novel. The idea of introducing head-specific transformations within the shared latent space offers a promising approach to improving LoRA's adaptation capability. Moreover, the paper is generally well written and clearly structured, with informative illustrations, comprehensive experiments, and meaningful ablation studies that support the proposed approach.
- The experimental results are promising. The consistent improvements across different model architectures and datasets suggest that the proposed method can provide effective adaptation beyond a specific model or task setting.
Critical weaknesses
- The evaluation of training efficiency is insufficient. The paper does not provide sufficient information about training time, peak GPU memory consumption, or computational overhead. As a result, it remains unclear whether the reported performance improvements come at the cost of increased computational resources.
- The paper lacks a direct comparison with independent head-wise LoRA. Given that the motivation concerns the mismatch between the shared latent space and head-specific adaptation needs, a natural alternative is to assign independent LoRA projections to each head, i.e., ($B_h$$A_h$). Although the authors discuss the potential parameter overhead and redundant subspace learning associated with this approach in Lines 176–179, these arguments would benefit from stronger experimental evidence.
- The input-dependent transformation may introduce additional inference overhead. Unlike standard LoRA, whose low-rank updates can be merged into the pretrained weights, HyRA employs input-dependent Householder-like transformations that generally prevent direct weight merging. This may introduce additional computation and inference latency. However, the paper does not sufficiently discuss or evaluate this potential limitation.
Questions for authors
- Could the authors clarify the additional parameter and computational overhead introduced by HyRA? In particular, it would be helpful to provide a detailed breakdown of parameter-sharing strategies and trainable parameter counts of the additional linear layers and transformation generators. I would also suggest the authors to report training time and peak GPU memory consumption relative to standard LoRA.
- Could the authors provide a direct experimental comparison with independent head-wise LoRA, where each attention head has its own low-rank projection ( )? This seems to be a natural baseline given the paper's motivation.
- Could the authors discuss the inference implications of the input-dependent Householder-like transformation? My understanding is that, since () depends on the input, the resulting updates cannot generally be merged into the pretrained weights in the same way as standard LoRA.
- What motivates the choice of Householder-like transformations over other structured transformations, particularly orthogonal transformations? Existing approaches such as OFT[1], BOFT[2], and PSOFT[3] have explored orthogonal transformations for parameter-efficient fine-tuning. Could the proposed head-wise adaptation framework be extended to alternative transformations, such as rotation-based parameterizations? Furthermore, what are the specific advantages and limitations of the proposed Householder-like mechanism?
[1] Qiu et al. Controlling Text-to-Image Diffusion by Orthogonal Finetuning. NeurIPS 2023. [2] Liu et al. Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization. ICLR 2024. [3] Wu et al. Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation. ICLR 2026.
5 comments
Response to Reviewer 88CC (2/3)
W2&Q2: Comparison with independent LoRA projections to each head .
Thank you for your suggestion about addition comparison with independent . To directly evaluate independent head-wise LoRA and investigate the potential redundancy of independently learned projections, we conducted two additional experiments: (1) A direct comparison with HyRA shows that independent head-wise LoRA uses more than 2 times trainable parameters at the same rank without improving accuracy. (2) A cross-head similarity analysis shows that the independently learned increasingly overlap during training.
- Comparison with independent head-wise LoRA. We train and independently for each head and compare with HyRA at ranks 8 and 16 on seven datasets under the same training protocol.
| Method | Trainable Para. | MMLU-Pro | ARC-C | ARC-E | OBQA | CSQA | CoLA | RTE | Avg |
|---|---|---|---|---|---|---|---|---|---|
| HyRA () | 0.31% | 55.86 | 84.28 | 92.46 | 89.89 | 84.48 | 87.93 | 91.03 | 83.70 |
| HyRA () | 0.59% | 58.43 | 85.45 | 93.30 | 90.18 | 84.81 | 88.36 | 93.53 | 84.87 |
| Independent LoRA () | 0.71% | 57.74 | 83.39 | 92.66 | 89.69 | 83.95 | 87.96 | 89.80 | 83.60 |
| Independent LoRA () | 1.43% | 56.12 | 84.65 | 92.98 | 89.87 | 84.23 | 88.17 | 90.47 | 83.78 |
At , the two methods are comparable on average 83.70 vs. 83.60, and HyRA is higher on four of seven datasets. At , HyRA is higher on all seven 84.87 vs. 83.78. At a closer parameter budget, HyRA with and 0.59% parameters outperforms independent LoRA with and 0.71% parameters on all seven datasets. Independent LoRA at uses 1.43% parameters and reaches an average of 83.78. HyRA reaches a comparable 83.70 with 0.31% parameters. Thus, assigning each head its own projection adds parameters without adding accuracy, which supports the parameter-overhead argument in our paper at lines 176–179.
- Redundancy among independently learned projections. We extract on real inputs from Q projections of three random layers. We measure cross-head similarity with linear CKA and with the of predicting one head’s linearly from another’s at initialization and after training.
| Dataset | Rank | Linear CKA (Init → Trained) | Prediction (Init → Trained) |
|---|---|---|---|
| MMLU-Pro | 8 | 0.111 → 0.316 | 0.086 → 0.268 |
| MMLU-Pro | 16 | 0.193 → 0.336 | 0.140 → 0.291 |
| ARC-C | 8 | 0.116 → 0.437 | 0.100 → 0.369 |
| ARC-C | 16 | 0.204 → 0.348 | 0.158 → 0.425 |
| ARC-E | 8 | 0.137 → 0.306 | 0.114 → 0.393 |
| ARC-E | 16 | 0.219 → 0.375 | 0.169 → 0.521 |
| OBQA | 8 | 0.147 → 0.418 | 0.123 → 0.502 |
| OBQA | 16 | 0.230 → 0.369 | 0.182 → 0.586 |
| CSQA | 8 | 0.140 → 0.232 | 0.115 → 0.441 |
| CSQA | 16 | 0.231 → 0.278 | 0.178 → 0.566 |
| CoLA | 8 | 0.261 → 0.352 | 0.248 → 0.546 |
| CoLA | 16 | 0.391 → 0.697 | 0.365 → 0.711 |
| RTE | 8 | 0.110 → 0.242 | 0.092 → 0.287 |
| RTE | 16 | 0.206 → 0.397 | 0.160 → 0.518 |
Both measures increase significantly after training in all 14 settings. For example, prediction reaches up to 0.711 and linear CKA up to 0.697, compared with 0.365 and 0.391 at initialization. At , exceeds 0.5 on five of the seven datasets, so more than half of the variance of one head’s latent is linearly predictable from another head’s.
The projections are initialized and trained independently. The growing similarity indicates that training drives them toward the same information, and part of each head’s parameter budget is spent on relearning it. This is the redundant subspace learning described in Lines 176–179.
In summary, independent head-wise LoRA requires much more parameters without improving accuracy, and its projections become increasingly similar during training, which supports HyRA’s design of a shared A with lightweight head-wise, input-dependent transformations.
Response to Reviewer 88CC (3/3)
Q3&W3: Inference latency and potential limitation
We additionally present the actual inference cost. We measure input-dependent adapters on Llama-3.1-8B and use one forward pass over 256 token sequences on a single NVIDIA RTX PRO 6000 GPU.
Table: Inference latency of input-dependent (non-mergeable) adapters on Llama-3.1-8B.
| Method | Trainable Para. | Latency, bs=1 (ms) | Latency, bs=4 (ms) |
|---|---|---|---|
| TopLoRA () | 0.77% | 72.7 | 148.2 |
| MoE-LoRA (, , top-) | 2.21% | 247.5 | 327.0 |
| LD-MoLE (, ) | 2.33% | 162.2 | 262.6 |
| HyRA () | 0.59% | 72.3 | 148.2 |
HyRA matches the latency of TopLoRA and runs 2.2-3.4$\times$ faster than MoE-LoRA and 1.8-2.2$\times$ faster than LD-MoLE, since it replaces multiple experts with a single shared and a lightweight generator.
Potential Limitation. Thank you for your suggestion! We agree our paper should include an explicit discussion about limitations. As you mentioned, a potential limitation is that unlike standard LoRA, the input-dependent transformations of HyRA cannot be merged into the pretrained weights. This is an inherent trade-off of input-dependent adaptation, as also seen in existing dynamic LoRA approaches such as TopLoRA, MoE-LoRA, LD-MoLE and so on, and we report our inference latency in above table.
However, the Householder modulation is computationally efficient, requiring only operations per head without explicitly constructing an full rank matrix. And the lightweight generator leverages existing frozen head outputs without an additional backbone forward pass. More important, the input dependence provides a strong benefit beyond static head-wise adaptation shown from our experimental results and demonstrates the practical value of input-dependent modulation. We believe this represents a meaningful trade-off between inference efficiency and adaptation capability.
Q4: Why Householder-like transformations?
We chose the Householder-like transformation for its direction-selective geometry. Householder-like transformation allows for the selective adjustment of components along specific directions in the latent representation while preserving the orthogonal residuals. Unlike strictly orthogonal transformations, which preserve vector norms, Householder-like transformations allow for the adjustment of feature amplitudes along selected directions. This property aligns with the “head-dependent directional sensitivity” analyzed in our Appendix A: different attention heads could dynamically select modulation directions and intensities suited to their specific needs based on the input.
OFT and BOFT multiply the pretrained weights by an orthogonal matrix and are therefore restricted to norm-preserving transformations. PSOFT applies an orthogonal transformation within the principal subspace of the pretrained weights and relaxes strict orthogonality with two learnable scaling vectors. The essential difference is that these methods have to transform the whole space, but ours rescales a single direction. HyRA directly applies an identity-plus-rank-1 transformation directly in the LoRA latent space, achieving adjustments through only a single direction vector and a modulation coefficient. It never constructs a full transformation matrix, and offers a more intuitive geometric interpretation and a simpler modulation structure. This keeps the modulation lightweight and suited to head-dependent adaptation.
Thank you again for your positive feedback and helpful suggestions to broaden our evaluation. We remain happy to clarify anything further before the discussion closes!
Response to Reviewer 88CC (1/3)
Thank you for your insightful review and feedback. We are delighted that you found our novel conceptual design. Below, we address your concerns and comments in detail:
W1&Q1: Additional parameter and computational overhead.
We sincerely thank you for your question about training efficiency, and we agree that only reports parameters is insufficient. As discussed in our Section 3, HyRA shares LoRA's A across heads and partitions LoRA's B into head-specific up-projections. The generator weights are also shared across the heads. Compared with LoRA, the additional parameters per adapted layer come from this lightweight generator, which includes a low-rank linear layer projecting Whx to rh for 2kdhead parameters where k is the rank, Ww for rdhead parameters and Wβ for dhead parameters. Thus, HyRA adds only (r+2k+1)dhead parameters per adapted layer on top of LoRA. The generator takes 2H(r+2k+1)dhead FLOPs, and the Householder update requires O(Hr) FLOPs. Overall, HyRA requires about 3% more FLOPs than LoRA.
To reflect the actual training cost, we measured the wall-clock time and peak memory of one training epoch on the CommonsenseQA dataset using the Llama-3.1-8B model on a single NVIDIA RTX PRO 6000 GPU with a batch size of 4.
Table: Computational cost and memory usage of different methods using the Llama-3.1-8B model.
* MoE-LoRA places E rank-r LoRA experts in each adapted layer, and a learned router activates the top-2 experts for each token.
As shown in Table, the HyRA's trainable parameters, memory, and training time are comparable to LoRA's.
The memory usage of HyRA is comparable to that of TopLoRA, but TopLoRA trains more parameters. The per-step of DoRA weight normalization doubles the epoch time comparing LoRA's. The mixture-based methods are substantially more expensive as MoE-LoRA and LD-MoLE train eight experts in every adapted layer. HyRA achieves input-dependent and head-wise adaptation with a single shared A and a lightweight generator instead of multiple experts. Thus, it trains 2.7-3.4$\times$ faster than MoE-LoRA and LD-MoLE, with 14-18,GB less memory and about 4$\times$ fewer trainable parameters.