acceptodds
Under review as a conference paper at ICLR 2027

HyRA: Head-wise Reflective Adaptation via Input-Dependent Householder Transformations

Abstract

Low-rank adaptation (LoRA) has become a dominant approach to parameter-efficient fine-tuning, which is widely used in large foundation models. LoRA can be conceptualized as projecting the input space into a low-dimensional latent adaptation space, where the dimensionality is determined by the rank of LoRA. However, under multi-head attention, standard LoRA constrains all heads to share a single latent adaptation space, despite attention heads typically encoding functionally distinct roles in pretrained models. We identify this mismatch between the shared adaptation and head-wise specialization as a structural bottleneck that limits LoRA's expressiveness. In this paper, we propose Head-wise Reflective Adaptation (HyRA), which dynamically generates an input-dependent Householder transformation for each head. This Householder generator enables compact head-wise adaptation by modulating feature components along an input-dependent direction while preserving the orthogonal residual. Formally, the head-wise HyRA update can be expressed as , where and are low-rank matrices and is a head-wise, input-dependent Householder transformation that breaks the shared subspace bottleneck. Notably, HyRA does not increase the LoRA rank, but enables each head to learn its own input-dependent adaptation. Extensive experiments across both language and vision architectures demonstrate that HyRA consistently outperforms LoRA and its competitive variants across diverse downstream benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parameter-Efficient Fine-Tuning, LoRA

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_88CCOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates LoRA fine-tuning for multi-head attention, focusing on the potential mismatch between the shared latent space and head-specific adaptation needs. To address this issue, the authors propose HyRA, which introduces a input-dependent Householder-like transformation for each attention head. The transformation selectively modulates latent features along input-dependent direction while preserving the orthogonal residual components. Experiments across different datasets and model architectures demonstrate that HyRA achieves improved parameter efficiency compared with LoRA.

Critical strengths
  1. The paper provides a well-motivated approach to addressing head-wise heterogeneity in LoRA fine-tuning. The observation in Figure 2 is particularly interesting, as the diminishing performance gains of standard LoRA under increasing head-wise heterogeneity suggest potential limitations of the shared low-rank latent space.
  2. The proposed Householder-like adaptation mechanism is novel. The idea of introducing head-specific transformations within the shared latent space offers a promising approach to improving LoRA's adaptation capability. Moreover, the paper is generally well written and clearly structured, with informative illustrations, comprehensive experiments, and meaningful ablation studies that support the proposed approach.
  3. The experimental results are promising. The consistent improvements across different model architectures and datasets suggest that the proposed method can provide effective adaptation beyond a specific model or task setting.
Critical weaknesses
  1. The evaluation of training efficiency is insufficient. The paper does not provide sufficient information about training time, peak GPU memory consumption, or computational overhead. As a result, it remains unclear whether the reported performance improvements come at the cost of increased computational resources.
  2. The paper lacks a direct comparison with independent head-wise LoRA. Given that the motivation concerns the mismatch between the shared latent space and head-specific adaptation needs, a natural alternative is to assign independent LoRA projections to each head, i.e., ($B_h$$A_h$). Although the authors discuss the potential parameter overhead and redundant subspace learning associated with this approach in Lines 176–179, these arguments would benefit from stronger experimental evidence.
  3. The input-dependent transformation may introduce additional inference overhead. Unlike standard LoRA, whose low-rank updates can be merged into the pretrained weights, HyRA employs input-dependent Householder-like transformations that generally prevent direct weight merging. This may introduce additional computation and inference latency. However, the paper does not sufficiently discuss or evaluate this potential limitation.
Questions for authors
  1. Could the authors clarify the additional parameter and computational overhead introduced by HyRA? In particular, it would be helpful to provide a detailed breakdown of parameter-sharing strategies and trainable parameter counts of the additional linear layers and transformation generators. I would also suggest the authors to report training time and peak GPU memory consumption relative to standard LoRA.
  2. Could the authors provide a direct experimental comparison with independent head-wise LoRA, where each attention head has its own low-rank projection (BhB_h AhA_h)? This seems to be a natural baseline given the paper's motivation.
  3. Could the authors discuss the inference implications of the input-dependent Householder-like transformation? My understanding is that, since (Hh(x)\mathcal H_h(x)) depends on the input, the resulting updates cannot generally be merged into the pretrained weights in the same way as standard LoRA.
  4. What motivates the choice of Householder-like transformations over other structured transformations, particularly orthogonal transformations? Existing approaches such as OFT[1], BOFT[2], and PSOFT[3] have explored orthogonal transformations for parameter-efficient fine-tuning. Could the proposed head-wise adaptation framework be extended to alternative transformations, such as rotation-based parameterizations? Furthermore, what are the specific advantages and limitations of the proposed Householder-like mechanism?

[1] Qiu et al. Controlling Text-to-Image Diffusion by Orthogonal Finetuning. NeurIPS 2023. [2] Liu et al. Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization. ICLR 2024. [3] Wu et al. Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation. ICLR 2026.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 88CC (1/3)

Thank you for your insightful review and feedback. We are delighted that you found our novel conceptual design. Below, we address your concerns and comments in detail:

W1&Q1: Additional parameter and computational overhead.

We sincerely thank you for your question about training efficiency, and we agree that only reports parameters is insufficient. As discussed in our Section 3, HyRA shares LoRA's AA across heads and partitions LoRA's BB into head-specific up-projections. The generator weights are also shared across the heads. Compared with LoRA, the additional parameters per adapted layer come from this lightweight generator, which includes a low-rank linear layer projecting WhxW^h x to rhr_h for 2kdhead2k d_{\text{head}} parameters where kk is the rank, WwW_w for rdheadrd_{\text{head}} parameters and WβW_\beta for dheadd_{\text{head}} parameters. Thus, HyRA adds only (r+2k+1) dhead(r+2k+1)\,d_{\text{head}} parameters per adapted layer on top of LoRA. The generator takes 2H(r+2k+1) dhead2H(r+2k+1)\,d_{\text{head}} FLOPs, and the Householder update requires O(Hr)\mathcal{O}(Hr) FLOPs. Overall, HyRA requires about 3% more FLOPs than LoRA.

To reflect the actual training cost, we measured the wall-clock time and peak memory of one training epoch on the CommonsenseQA dataset using the Llama-3.1-8B model on a single NVIDIA RTX PRO 6000 GPU with a batch size of 4.

Table: Computational cost and memory usage of different methods using the Llama-3.1-8B model.

MethodTrainable Para.Memory (GB)Time (Min)
LoRA (r=16r=16)0.52%49.45.08
DoRA (r=16r=16)0.55%53.211.23
TopLoRA (r=16r=16)0.77%52.36.40
MoE-LoRA* (r=8r=8, E=8E=8, top-22)2.21%66.021.15
LD-MoLE (r=8r=8, E=8E=8)2.33%69.716.49
HyRA (r=16r=16)0.59%51.96.15

* MoE-LoRA places EE rank-rr LoRA experts in each adapted layer, and a learned router activates the top-2 experts for each token.

As shown in Table, the HyRA's trainable parameters, memory, and training time are comparable to LoRA's.

The memory usage of HyRA is comparable to that of TopLoRA, but TopLoRA trains more parameters. The per-step of DoRA weight normalization doubles the epoch time comparing LoRA's. The mixture-based methods are substantially more expensive as MoE-LoRA and LD-MoLE train eight experts in every adapted layer. HyRA achieves input-dependent and head-wise adaptation with a single shared AA and a lightweight generator instead of multiple experts. Thus, it trains 2.7-3.4$\times$ faster than MoE-LoRA and LD-MoLE, with 14-18,GB less memory and about 4$\times$ fewer trainable parameters.

AuthorsOpenReview ↗

Response to Reviewer 88CC (2/3)

W2&Q2: Comparison with independent LoRA projections to each head (Bh,Ah)(B_h, A_h).

Thank you for your suggestion about addition comparison with independent (Bh,Ah)(B_h, A_h). To directly evaluate independent head-wise LoRA and investigate the potential redundancy of independently learned projections, we conducted two additional experiments: (1) A direct comparison with HyRA shows that independent head-wise LoRA uses more than 2 times trainable parameters at the same rank without improving accuracy. (2) A cross-head similarity analysis shows that the independently learned AhA_h increasingly overlap during training.

  1. Comparison with independent head-wise LoRA. We train AhA_h and BhB_h independently for each head and compare with HyRA at ranks 8 and 16 on seven datasets under the same training protocol.
MethodTrainable Para.MMLU-ProARC-CARC-EOBQACSQACoLARTEAvg
HyRA (r=8r=8)0.31%55.8684.2892.4689.8984.4887.9391.0383.70
HyRA (r=16r=16)0.59%58.4385.4593.3090.1884.8188.3693.5384.87
Independent LoRA (r=8r=8)0.71%57.7483.3992.6689.6983.9587.9689.8083.60
Independent LoRA (r=16r=16)1.43%56.1284.6592.9889.8784.2388.1790.4783.78

At r=8r=8, the two methods are comparable on average 83.70 vs. 83.60, and HyRA is higher on four of seven datasets. At r=16r=16, HyRA is higher on all seven 84.87 vs. 83.78. At a closer parameter budget, HyRA with r=16r=16 and 0.59% parameters outperforms independent LoRA with r=8r=8 and 0.71% parameters on all seven datasets. Independent LoRA at r=16r=16 uses 1.43% parameters and reaches an average of 83.78. HyRA reaches a comparable 83.70 with 0.31% parameters. Thus, assigning each head its own projection adds parameters without adding accuracy, which supports the parameter-overhead argument in our paper at lines 176–179.

  1. Redundancy among independently learned projections. We extract zh=Ahxz_h=A_hx on real inputs from Q projections of three random layers. We measure cross-head similarity with linear CKA and with the R2R^2 of predicting one head’s zhz_h linearly from another’s at initialization and after training.
DatasetRankLinear CKA (Init → Trained)Prediction R2R^2 (Init → Trained)
MMLU-Pro80.111 → 0.3160.086 → 0.268
MMLU-Pro160.193 → 0.3360.140 → 0.291
ARC-C80.116 → 0.4370.100 → 0.369
ARC-C160.204 → 0.3480.158 → 0.425
ARC-E80.137 → 0.3060.114 → 0.393
ARC-E160.219 → 0.3750.169 → 0.521
OBQA80.147 → 0.4180.123 → 0.502
OBQA160.230 → 0.3690.182 → 0.586
CSQA80.140 → 0.2320.115 → 0.441
CSQA160.231 → 0.2780.178 → 0.566
CoLA80.261 → 0.3520.248 → 0.546
CoLA160.391 → 0.6970.365 → 0.711
RTE80.110 → 0.2420.092 → 0.287
RTE160.206 → 0.3970.160 → 0.518

Both measures increase significantly after training in all 14 settings. For example, prediction R2R^2 reaches up to 0.711 and linear CKA up to 0.697, compared with 0.365 and 0.391 at initialization. At r=16r=16, R2R^2 exceeds 0.5 on five of the seven datasets, so more than half of the variance of one head’s latent is linearly predictable from another head’s.

The projections are initialized and trained independently. The growing similarity indicates that training drives them toward the same information, and part of each head’s parameter budget is spent on relearning it. This is the redundant subspace learning described in Lines 176–179.

In summary, independent head-wise LoRA requires much more parameters without improving accuracy, and its projections become increasingly similar during training, which supports HyRA’s design of a shared A with lightweight head-wise, input-dependent transformations.

AuthorsOpenReview ↗

Response to Reviewer 88CC (3/3)

Q3&W3: Inference latency and potential limitation

We additionally present the actual inference cost. We measure input-dependent adapters on Llama-3.1-8B and use one forward pass over 256 token sequences on a single NVIDIA RTX PRO 6000 GPU.

Table: Inference latency of input-dependent (non-mergeable) adapters on Llama-3.1-8B.

MethodTrainable Para.Latency, bs=1 (ms)Latency, bs=4 (ms)
TopLoRA (r=16r=16)0.77%72.7148.2
MoE-LoRA (r=8r=8, E=8E=8, top-22)2.21%247.5327.0
LD-MoLE (r=8r=8, E=8E=8)2.33%162.2262.6
HyRA (r=16r=16)0.59%72.3148.2

HyRA matches the latency of TopLoRA and runs 2.2-3.4$\times$ faster than MoE-LoRA and 1.8-2.2$\times$ faster than LD-MoLE, since it replaces multiple experts with a single shared AA and a lightweight generator.

Potential Limitation. Thank you for your suggestion! We agree our paper should include an explicit discussion about limitations. As you mentioned, a potential limitation is that unlike standard LoRA, the input-dependent transformations of HyRA cannot be merged into the pretrained weights. This is an inherent trade-off of input-dependent adaptation, as also seen in existing dynamic LoRA approaches such as TopLoRA, MoE-LoRA, LD-MoLE and so on, and we report our inference latency in above table.

However, the Householder modulation is computationally efficient, requiring only O(r)O(r) operations per head without explicitly constructing an full rank matrix. And the lightweight generator leverages existing frozen head outputs without an additional backbone forward pass. More important, the input dependence provides a strong benefit beyond static head-wise adaptation shown from our experimental results and demonstrates the practical value of input-dependent modulation. We believe this represents a meaningful trade-off between inference efficiency and adaptation capability.

Q4: Why Householder-like transformations?

We chose the Householder-like transformation for its direction-selective geometry. Householder-like transformation allows for the selective adjustment of components along specific directions in the latent representation while preserving the orthogonal residuals. Unlike strictly orthogonal transformations, which preserve vector norms, Householder-like transformations allow for the adjustment of feature amplitudes along selected directions. This property aligns with the “head-dependent directional sensitivity” analyzed in our Appendix A: different attention heads could dynamically select modulation directions and intensities suited to their specific needs based on the input.

OFT and BOFT multiply the pretrained weights by an orthogonal matrix and are therefore restricted to norm-preserving transformations. PSOFT applies an orthogonal transformation within the principal subspace of the pretrained weights and relaxes strict orthogonality with two learnable scaling vectors. The essential difference is that these methods have to transform the whole space, but ours rescales a single direction. HyRA directly applies an identity-plus-rank-1 transformation directly in the LoRA latent space, achieving adjustments through only a single direction vector and a modulation coefficient. It never constructs a full transformation matrix, and offers a more intuitive geometric interpretation and a simpler modulation structure. This keeps the modulation lightweight and suited to head-dependent adaptation.


Thank you again for your positive feedback and helpful suggestions to broaden our evaluation. We remain happy to clarify anything further before the discussion closes!

Reviewer_anhhOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Head-wise Reflective Adaptation (HyRA) is proposed, which is a parameter-efficient fine-tuning method motivated by the limitation of standard LoRA in adapting shared low-rank representations to different attention heads. Given a shared latent representation (u=Ax), HyRA employs a lightweight generator to produce an input-dependent direction (w_h(x)) and modulation coefficient (β_h(x)) for each head. It then applies a Householder-like transformation, (H_h(x) = I − β_h(x)w_h(x)w_h(x)^T), before the head-specific up-projection. This transformation selectively modulates the latent component along the generated direction while preserving the orthogonal component, enabling head-wise, input-dependent adaptation without increasing the nominal LoRA rank. The method is evaluated on Llama-3.1-8B, Llama-3.2-3B, and Qwen3-1.7B for language-understanding tasks, as well as ViT-Base and ViT-Large on image-classification benchmarks. Comparisons with LoRA, DoRA, TopLoRA, mixture-of-LoRA-experts methods, and FourierFT show that HyRA achieves competitive performance across the evaluated settings. Ablation studies further examine the contributions of head-wise adaptation and input dependence, while comparisons with alternative gating mechanisms and visualization analyses provide additional evidence for the proposed design.

Critical strengths
  1. The paper proposes a simple and interpretable approach to head-wise, input-dependent LoRA adaptation. By introducing a lightweight Householder-like transformation, HyRA enables flexible feature modulation without increasing the LoRA rank, offering a good balance between expressiveness and parameter efficiency.
  2. The paper provides extensive evaluations across language and vision tasks, demonstrating consistent improvements over several PEFT baselines. Component ablations and gating comparisons further support the effectiveness of head-wise adaptation, input dependence, and the proposed modulation mechanism.
Critical weaknesses
  1. The motivation for treating the shared latent transformation as a structural bottleneck is insufficiently justified. While polar decomposition reveals a shared operator MM, it does not establish that this sharing limits LoRA's performance, especially since attention heads retain distinct output mappings. More direct empirical evidence is needed to support this central claim.
  2. The computational overhead of HyRA is insufficiently analyzed. Although it maintains the LoRA rank, the additional generator introduces extra parameters and computational costs. The paper should report training time, memory usage, and inference latency, and provide comparisons under matched parameter budgets to better demonstrate its practical efficiency.
Questions for authors
  1. Do the authors have empirical evidence beyond the polar decomposition analysis to demonstrate that sharing the latent transformation across attention heads is a significant limitation of LoRA?
  2. Could the authors clarify the computational overhead introduced by the input-dependent generator? Specifically, are measurements of training time, peak GPU memory usage, and inference latency available?
5 comments
AuthorsOpenReview ↗

Response to Reviewer anhh (1/2)

Thank you for your valuable feedback, and the time taken to provide constructive feedback to strengthen our work further! Your tow questions map closely onto what the revision adds. We hope our following responses can address your concerns. We answer W1&Q1 here and W2&Q2 in the next comment.

W1&Q1: Empirical evidence for the bottleneck of the shared latent transformation.

We would like to further clarify that MM is a potential driver of the limitation, as all head-wise output mappings BhB_h clearly share it.These mappings static, so they cannot adjust the combination of latent features according to the current input. This restricts low-rank adaptation from fully leveraging linear combinations within the current space. To verify whether this limitation actually exists and whether it affects the performance of downstream tasks, we first point to the evidence already in the manuscript and then report two additional experiments.

Existing Evidence:

  1. In the ablation study, Table 3 compares four variants that differ only in the latent operator. The results answer the reviewer’s two points directly: whether sharing limits performance, and whether distinct output mappings already compensate for it. Replacing the shared latent operator with a head-wise one improves average accuracy by 1.58, with the shared AA and the head-wise BhB_h unchanged. Distinct output mappings therefore do not compensate for the shared operator. The static ThT_h adds only +0.57. The cost of sharing appears when heads need transformations that differ with the input.

  2. Figure 2 is consistent with this result. LoRA obtains lower margin gains on inputs with higher head-wise gradient heterogeneity, consistently across ranks, which is where a shared latent operator is expected to be most restrictive. MoLE and TopLoRA add capacity or input dependence but keep it shared across heads, and they show the same trend. The limitation is therefore tied to sharing across heads, and not to capacity or input dependence alone.

Additional experiments:

  1. Optimal static output-mapping fit. The reviewer notes that heads retain distinct output mappings. This experiment illustrates that distinct output mappings alone are not sufficient to express head-wise, input-dependent updates. For each of the 896 heads (Q/V adapters, 28 layers), we fit an unconstrained static output mapping to HyRA’s updates in closed form: B~h∗=arg⁡min⁡B~h∑i∥B~hAxi−BhHh(xi)Axi∥2\widetilde{B}_h^{\ast}=\arg\min_{\widetilde{B}_h}\sum_i\lVert\widetilde{B}_h Ax_i-B_h\mathcal{H}_h(x_i)Ax_i\rVert^2. We report the residual as a norm ratio on the inputs used for the fit and on held-out inputs. The fitting residual is the norm of the fitting error relative to the norm of HyRA’s update on the inputs used to fit B~h∗\widetilde B_h^*, and the held-out residual is the same ratio on inputs not used in the fit.
DatasetFitting Error (Norm Ratio)Held-Out Error (Norm Ratio)
ARC-E19.47%19.48%
ARC-C15.35%15.27%
MMLU-Pro23.72%23.70%

Even with the optimal static mapping, 15.27–23.70% of the norm of HyRA’s update remains unexplained on held-out inputs. The residual is nearly identical on fitting and held-out inputs, so it is a systematic input-dependent component and not noise. Distinct static output mappings therefore cannot reproduce the head-wise, input-dependent updates.

  1. PCA Analysis of Latent and Output Increments. To further investigate how sharing restricts the latent operator itself, we compare the latent-space and output-space increments generated by shared and head-wise input-dependent modulation, using static LoRA as the reference. PCA visualization results in revised manuscript (Figure 5 in Section 5.2 and Figure 7 and Figure 8 in Appendix D) show that Shared and HyRA produce different distributions of modulation increments in latent space. The increments learned by HyRA are clearly more broadly distributed than the shared method. More importantly, these differences remain observable in the adapter output space after applying the same head-specific output mappings BhB_h. These findings indicate that the head-wise modulation learned by HyRA is not merely manifested as numerical differences in the latent space but further influences the actual adapter outputs.

In summary, replacing the shared M with a head-wise operator improves accuracy by 1.58 points (Table 3), and LoRA gains less where heads are more heterogeneous (Figure 2). Even optimal static output mappings leave 15.27–23.70% of HyRA’s update unexplained, and the PCA shows that shared modulation stays confined to a narrow region. These results support that sharing the latent operator limits LoRA despite distinct output mappings. We have added the new experiments to the revised manuscript in Section 5.2 and Appendix D.

AuthorsOpenReview ↗

Response to Reviewer anhh (2/2)

W2&Q2: More sufficient computational overhead analysis of HyRA.

We sincerely thank you for your suggestion, we agree that only reports parameters is insufficient. As discussed in our Section 3, HyRA shares LoRA's AA across heads and partitions LoRA's BB into head-specific up-projections. The generator weights are also shared across the heads. Compared with LoRA, the additional parameters per adapted layer come from this lightweight generator, which includes a low-rank linear layer projecting WhxW^h x to rhr_h for 2kdhead2k d_{\text{head}} parameters where kk is the rank, WwW_w for rdheadrd_{\text{head}} parameters and WβW_\beta for dheadd_{\text{head}} parameters. Thus, HyRA adds only (r+2k+1) dhead(r+2k+1)\,d_{\text{head}} parameters per adapted layer on top of LoRA. The generator takes 2H(r+2k+1) dhead2H(r+2k+1)\,d_{\text{head}} FLOPs, and the Householder update requires O(Hr)\mathcal{O}(Hr) FLOPs. Overall, HyRA requires about 3% more FLOPs than LoRA.

Training Stage. To reflect the actual training cost, we measured the wall-clock time and peak memory of one training epoch on the CommonsenseQA dataset using the Llama-3.1-8B model on a single NVIDIA RTX PRO 6000 GPU with a batch size of 4. And we add the trade-off between accuracy and computational overhead.

Table: Computational cost and memory usage of different methods using the Llama-3.1-8B model.

MethodAcc.Trainable Para.Memory (GB)Time (Min)
LoRA (r=16r=16)83.25%0.52%49.45.08
DoRA (r=16r=16)83.90%0.55%53.211.23
TopLoRA (r=16r=16)84.23%0.77%52.36.40
MoE-LoRA* (r=8r=8, E=8E=8, top-22)83.24%2.21%66.021.15
LD-MoLE (r=8r=8, E=8E=8)83.01%2.33%69.716.49
HyRA (r=16r=16)84.81%0.59%51.96.15

* MoE-LoRA places EE rank-rr LoRA experts in each adapted layer, and a learned router activates the top-2 experts for each token.

As shown in Table, the accuracy of HyRA outperforms other methods while the HyRA's trainable parameters, memory, and training time are comparable to LoRA's.

The memory usage of HyRA is comparable to that of TopLoRA, but TopLoRA trains more parameters. The per-step of DoRA weight normalization doubles the epoch time comparing LoRA's. The mixture-based methods are substantially more expensive as MoE-LoRA and LD-MoLE train eight experts in every adapted layer. HyRA achieves input-dependent and head-wise adaptation with a single shared AA and a lightweight generator instead of multiple experts. Thus, it trains 2.7-3.4$\times$ faster than MoE-LoRA and LD-MoLE, with 14-18,GB less memory and about 4$\times$ fewer trainable parameters.

Inference Statge. To reflect the actual inference cost, we measure input-dependent adapters on Llama-3.1-8B and use one forward pass over 256 token sequences on a single NVIDIA RTX PRO 6000 GPU.

Table: Inference latency of input-dependent (non-mergeable) adapters on Llama-3.1-8B.

MethodTrainable Para.Latency, bs=1 (ms)Latency, bs=4 (ms)
TopLoRA (r=16r=16)0.77%72.7148.2
MoE-LoRA (r=8r=8, E=8E=8, top-22)2.21%247.5327.0
LD-MoLE (r=8r=8, E=8E=8)2.33%162.2262.6
HyRA (r=16r=16)0.59%72.3148.2

HyRA matches the latency of TopLoRA and runs 2.2-3.4$\times$ faster than MoE-LoRA and 1.8-2.2$\times$ faster than LD-MoLE, since it replaces multiple experts with a single shared AA and a lightweight generator.


Thank you again for your constructive comments. We hope the revision addresses your concerns, and we would be glad to clarify anything further during the discussion period.

AuthorsOpenReview ↗

Follow up on our rebuttal

Dear Reviewer anhh:

Thank you again for your valuable feedback. As the discussion period closes on November 18 (AoE), we would like to check whether our responses and the revised paper have addressed your concerns. If so, we would sincerely appreciate it if you could consider raising the score.

Reviewer_CbCbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how LoRA can accommodate different adaptation needs across attention heads without increasing its rank. HyRA inserts an input-dependent, head-specific transformation between the shared down-projection A and each head's up-projection B_h. A lightweight generator uses the frozen head output to produce a direction and modulation strength. The resulting identity-plus-rank-one transformation modifies the latent representation along that direction while preserving its orthogonal component. The method is evaluated on three language backbones and two vision transformers, with reported improvements over LoRA and the compared adaptation methods. Ablations compare static and input-dependent transformations, as well as scalar, diagonal, FiLM, and directional modulation, to examine which parts of the head-wise design contribute to the gains.

Critical strengths

The modification is compact, and the experiments show positive results across three language backbones and two ViTs. The scalar, diagonal, FiLM, and directional controls are useful: they test more than simply whether adding parameters helps.

Critical weaknesses

The evaluation-split descriptions are inconsistent. Appendix C says results are evaluated on validation and the test split is unused. Appendix E.1 says validation selects hyperparameters and the final results are test accuracy. It is not clear which description applies to the main NLP table.

Section 2 does not clearly distinguish static reparameterization from input-dependent expressiveness. A static per-head T_h can be absorbed into B_h, and for fixed x, row(B_h H_h(x) A) remains within row(A). The input-dependent generator can still improve expressiveness, but this is different from removing a limitation through static head-specific transformations. The static T_h ablation could reflect an optimization benefit; the explanation should distinguish this from a change in expressiveness.

The evaluation protocol and the interpretation of the static ablation need to be resolved before the main claims can be assessed reliably.

Questions for authors
  1. Which splits are used for model selection and final reporting on each NLP task?
  2. How should the static T_h ablation in Table 3 be interpreted, given that T_h can be absorbed into B_h?
2 comments
AuthorsOpenReview ↗

Response to Reviewer CbCb

Thank you for your insightful review and feedback. We are delighted that you found our novel conceptual design. Below, we address your concerns and comments in detail:

W1&Q1: Clarification of evaluation-split descriptions and splits used for NLP tasks.

Thank you for your careful reading of our paper and sorry for the confusion! The description in Appendix C is correct and applies to all NLP results in Table 1. All methods are trained on the training split and evaluated on the validation split. We use the validation split because the test labels of several benchmarks are not publicly released, including CommonsenseQA, SWAG, HellaSwag, CoLA, and RTE. This follows the evaluation setup of [1] and [2]. For consistency, we apply the same protocol to all datasets. HyRA and all baselines use the same data partitions to ensure a fair comparison. The sentence in Appendix E.1 should read validation accuracy, and we have corrected it in the revised manuscript.

W2&Q2: Section 2 does not clearly distinguish static reparameterization from input-dependent expressiveness.

We sincerely thank you for the careful analysis. We clarify that the expressiveness gain of HyRA comes from input dependence, as stated in Section 3 and Appendix A.3: the head-wise structure anchors the input-dependent transformation at each attention head, so that each head recombines the shared latent features according to the current input. We further explain the gain in HyRA’s expressiveness as follows.

The static ThT_h variant is an ablation in our Section 5.1 that studies head-wise structure in isolation. With a static transformation alone, the function class is unchanged, and the effect is on optimization. Let Ch=BhThC_h=B_hT_h and Gh=∂L/∂ChG_h=\partial\mathcal{L}/\partial C_h. LoRA updates the up-projection of head hh as ΔChLoRA=−ηGh\Delta C_h^{\mathrm{LoRA}}=-\eta G_h, whereas the static variant gives ΔChstatic=−η(GhTh⊤Th+BhBh⊤Gh)+O(η2)\Delta C_h^{\mathrm{static}}=-\eta\left(G_hT_h^\top T_h+B_hB_h^\top G_h\right)+O(\eta^2), so each head’s update has its own preconditioner. We interpret the +0.57+0.57 gain of the static variant over LoRA in Table 3 as this optimization effect. Input dependence enlarges the function class. In LoRA, BhAxB_hAx is linear in xx and depends on xx only through AxAx. In HyRA, Hh(x)\mathcal{H}_h(x) is generated from WhxW_hx, so BhHh(x)AxB_h\mathcal{H}_h(x)Ax is a nonlinear function of xx that no static (Bh,A)(B_h, A) can represent. This holds even though, for a fixed xx, BhHh(x)AB_h\mathcal{H}_h(x)A remains within row(A)\mathrm{row}(A) with rank at most rr. Input dependence can be introduced in several ways, such as per token with one modulation shared across heads [3] or through routing over multiple LoRA experts [1] [2]. Section 2 and Figure 2 identify the importance of attention heads, so HyRA anchors the input-dependent transformation at the head level. Table 3 supports this choice: a shared H(x)\mathcal{H}(x) gives +0.78+0.78 and the head-wise Hh(x)\mathcal{H}_h(x) gives +2.36+2.36, so we attribute the additional +1.58+1.58 to the head-wise structure.

We have revised our paper to state the distinction between static reparameterization and input-dependent expressiveness in Appendix A.3.

[1]: Ld-mole: Learnable dynamic routing for mixture of lora experts

[2]: Higher layers need more lora experts

[3]: Beyond higher rank: Token-wise input-output projections for efficient low-rank adaptation


Thank you again for your constructive feedback and for helping us improve the paper. We will include all additional clarification and analysis into our revised paper!

AuthorsOpenReview ↗

Follow up on our rebuttal

Dear Reviewer CbCb:

Thank you again for your valuable feedback. As the discussion period closes on November 18 (AoE), we would like to check whether our responses and the revised paper have addressed your concerns. If so, we would sincerely appreciate it if you could consider raising the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.