acceptodds
Under review as a conference paper at ICLR 2027

HoRA: Rethinking Multimodal Low-Rank Adaptation as Implicit Parameterization

Abstract

While multimodal low-rank adaptation (LoRA) has achieved strong empirical success, its explicit parameterization makes adaptation capacity closely tied to the dimensions and design of low-rank factors, making modality-wise parameter allocation less directly controllable once the rank is fixed. To address this limitation, we propose feature h ashing-based multimodal l o w-r ank adaptation (HoRA), a novel framework that reformulates multimodal LoRA as an implicit parameterization problem. HoRA constructs low-rank adaptations via feature hashing-based implicit parameterization, generating matrix entries on-the-fly from compact parameter buckets instead of explicitly storing them. By independently controlling the bucket size of each modality, HoRA introduces an additional degree of freedom for fine-grained control over modality-wise adaptation parameters within a fixed-rank formulation. We theoretically characterize the feature hashing-based low-rank parameterization through its unbiased reconstruction and collision-dependent approximation bound under the assumed hashing scheme. We further establish the convergence property of HoRA, with the convergence bound characterized by modality-specific bucket sizes. Extensive experiments across diverse multimodal benchmarks demonstrate that HoRA achieves competitive or superior performance with comparable or fewer trainable parameters. The code is available at https://anonymous.4open.science/r/HoRA-E512.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal LoRA, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2t6UOpenReview ↗
Rating
4(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HoRA, a framework that reformulates multimodal LoRA as an implicit parameterization problem. It implicitly represents modality-specific low-rank factors using compact parameter buckets, allowing each modality’s trainable parameter budget to be adjusted independently at a fixed rank. It provides theoretical analyses and optimization, and evaluate on QWen2-7B and LLAMA2-7B, demonstrating its compective performence.

But I think there are some questions to be solved:

  1. The unbiased reconstruction analysis starts from a given matrix and studies hash-based aggregation and reconstruction, whereas the implemented method directly initializes and optimizes bucket parameters.
  2. Several gains on Qwen2-7B are small and require statistical evidence.
  3. The current capacity analysis changes both bucket sizes and the total parameter budget. Comparing uniform and nonuniform modality allocations under a fixed total parameter budget would better isolate the benefits of allocation.
  4. The modality notation is inconsistent: Figure 1(b) uses A^t, while line 134 uses A^a. Please check and unify the notation throughout the paper.
Critical strengths
  1. The method is simple and well motivated, allowing flexible control of modality-specific parameter budgets at a fixed rank.

  2. The experiments cover several multimodal tasks and include useful ablations, showing promising performance with fewer parameters. The paper is also clearly organized.

Critical weaknesses
  1. The unbiased reconstruction analysis starts from a given matrix and studies hash-based aggregation and reconstruction, whereas the implemented method directly initializes and optimizes bucket parameters.

  2. Several gains on Qwen2-7B are small and require statistical evidence.

  3. The current capacity analysis changes both bucket sizes and the total parameter budget. Comparing uniform and nonuniform modality allocations under a fixed total parameter budget would better isolate the benefits of allocation.

  4. The modality notation is inconsistent: Figure 1(b) uses A^t, while line 134 uses A^a. Please check and unify the notation throughout the paper.

Questions for authors
  1. How does the unbiased reconstruction analysis apply to directly trained bucket parameters?

  2. Are the small gains on Qwen2-7B consistent across repeated experiments?

  3. Does nonuniform modality allocation outperform uniform allocation under a fixed total parameter budget?

  4. Could you clarify the inconsistent notation between Figure 1(b) and line 134 and unify it throughout the paper?

3 comments

Thank you for your insightful comments. We provide detailed responses to your concerns below.

Role of Unbiased Reconstruction in Direct Bucket Training: Thank you for raising this point. The unbiasedness result does not assume that HoRA explicitly compresses Ao{\bf A}^o during training. Firstly, Eqs. (6)–(7) of the original paper provide an analytical construction under which: Eho,ξo[A^o]=Ao.\mathbb E_{h^o,\xi^o}\left[\widehat{\bf A}^o\right]={\bf A}^o.

This result characterizes the representational and collision-induced approximation properties of the hashed parameterization. Secondly, the implementation directly optimizes Wo{\bf W}^o, with Eq. (7) of the original paper used in the forward pass and the exact chain-rule gradient ∂L∂Wko=∑(i,j):ho(i,j)=kξo(i,j)∂L∂A^i,jo.\frac{\partial\mathcal L}{\partial{W}_k^o}=\sum_{(i,j):h^o(i,j)=k}\xi^o(i,j)\frac{\partial\mathcal L}{\partial\widehat{A}_{i,j}^o}.

Therefore, Eq. (6) is an analytical construction rather than a training operation. We will clarify this distinction between representational analysis and direct bucket optimization.

Relevant discussion will be included in the final version.

Robustness of the Qwen2-7B Results across Random Seeds: We thank the reviewer for raising this important concern and agree that small performance margins should be supported by repeated experiments. Firstly, Appendix Table IV already reports four random-seed runs on AVE, where HoRA consistently outperforms MokA. Secondly, we conducted one additional independent run on the other five benchmarks, with the results summarized in Table R1.

Table R1: Robustness analysis of HoRA with Qwen2-7B. Results are reported as mean ±\pm std over two random seeds.

Method/SeedM-AVQAMMBenchPOPESEED-BenchMME
HoRA (ϵ=0\epsilon=0)79.60%76.36%81.98%58.43%1252.65
HoRA (ϵ=1\epsilon=1)79.34%76.36%81.70%58.29%1352.48
HoRA (Avg.)79.47%$\pm$0.13%76.36%$\pm$0.00%81.84%$\pm$0.14%58.36%$\pm$0.07%1302.57$\pm$49.92
MokA79.17%76.25%81.83%58.59%1245.76

Furthermore, the repeated runs support consistent improvements on M-AVQA and MMBench and an improvement on MME, although MME shows larger variation. We also acknowledge that HoRA is essentially tied with MokA on POPE and slightly lower on SEED-Bench, changing its ranking from second to third.

Finally, we acknowledge that two seeds provide limited statistical evidence. In the final version, we will run additional random-seed experiments on all Qwen2-7B benchmarks and update the reported means, standard deviations, rankings, and corresponding conclusions.

Fixed-Budget Capacity Allocation: We thank the reviewer for this important observation. Firstly, although Fig. 2 jointly varies bucket sizes and total parameters, it already contains controlled comparisons with the same total budget, because the shared factor B\bf B is unchanged and fixing ∑oMˉo\sum_o\bar M^o fixes the total trainable parameters. For example, at ∑oMˉo=1.8\sum_o\bar M^o=1.8, the uniform allocation (Mˉa,Mˉt,Mˉv)=(0.6,0.6,0.6)(\bar M^a,\bar M^t,\bar M^v)=(0.6,0.6,0.6) achieves 76.77%, while the non-uniform allocations (0.8,0.4,0.6)(0.8,0.4,0.6) and (0.4,0.8,0.6)(0.4,0.8,0.6) achieve 77.59% and 77.11%, respectively. Secondly, allocations with identical budgets produce different performance, confirming that the gain is affected by the allocation ratio rather than merely by additional parameters. Furthermore, we do not claim that non-uniform allocation is universally superior; instead, these results support HoRA’s ability to explore task-dependent modality allocations under a fixed budget. We will clarify this controlled interpretation of Fig. 2 in the final version.

Notation Consistency: Thank you for pointing out the inconsistent modality notation. Firstly, in the revised manuscript, we have unified the superscripts aa, vv, and tt to consistently denote the audio, visual, and text modalities, respectively. Secondly, we have carefully checked and corrected the notation throughout Figure 1, the main text, equations, and appendix. This writing issue has now been fully resolved without affecting our method or results.

Thank you for your response. It addresses most of my concerns. The paper represents a substantial amount of work, and the experimental evaluation is thorough. If the paper is accepted, please double-check the notation and formatting throughout the final version. I will raise my overall rating to 4 and increase my confidence score.

Thank you for your thoughtful follow-up and constructive feedback throughout the discussion. We are glad that our responses have addressed most of your concerns. We will carefully double-check the notation and formatting throughout the final version and incorporate the clarifications and additional experimental results promised in our rebuttal. We appreciate your time and careful evaluation of our work.

Reviewer_cPvBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issues of the coupling of the trainable parameters and the size of the low-rank factor in multimodal low-rank adaptation, as well as the difficulty in flexibly allocating the parameter budgets for each modality. To this end, it proposes the implicit parameterization method HoRA based on feature hashing. Retain the shared low-rank matrix, generate modality-specific matrices through modality-independent parameter buckets and signed hash functions, thereby adjusting the trainable parameters of each modality while maintaining the rank. Its main contribution is to introduce hash parameter sharing into multimodal low-rank adaptation and conduct theoretical analysis around the unbiasedness of reconstruction, the approximate error caused by collisions, and the optimization convergence properties. In terms of verification, this paper compares HoRA with various adaptation methods on six benchmarks, including LLaMA2-7B and Qwen2-7B, and examines the method's characteristics through experiments on parameter budget matching, bucket size allocation, hash mechanism and position, rank, and running overhead. The results show that HoRA achieves competitive or superior performance with comparable or fewer trainable parameters in most test settings.

Critical strengths
  1. In multimodal adaptation, different modalities may require different parameter budgets. This paper introduces a variable called bucket size to adjust the parameters, eliminating the reliance solely on the selection of rank and providing a feasible approach for model adaptation under resource-constrained conditions.
  2. HoRA achieves parameter sharing for low-rank factors through signed hashing without the need to introduce an additional complex parameter generation network. HoRA also retains the explicit shared matrix BB, enabling a more natural integration into the existing multimodal low-rank adaptation framework.
  3. This paper provides a more granular way of modality parameter configuration. Each modality uses an independent parameter bucket, allowing the number of trainable parameters to be adjusted separately while keeping the size of the low-rank factors unchanged.
Critical weaknesses
  1. There is no main figure, making it difficult to understand the author's intention.
  2. Hash parameter sharing, signed reconstruction, and gradient aggregation have already established a mature research foundation. Although the paper mentions NOLA, there is no direct comparison. It also misses Uni-LoRA, which is similar to its core idea. Thus, it is currently difficult to determine whether the performance gain comes from the existing parameter sharing mechanism or the modality-independent design proposed in this paper.
  3. The unbiasedness proof analyzes "given the complete matrix, compress and then reconstruct", while the actual method directly trains the parameter buckets. There is a lack of a strict connection between the two. The convergence analysis also mixes the update matrix ΔW\Delta W with the bucket parameters and the gradient of the matrix BB in the space, without fully handling the chain rule of reparameterization and the dependency between the fixed hash and the training parameters. Therefore, the current theory is not yet sufficient to directly support the convergence guarantee of the actual HoRA.
  4. In Eq. 2, BAa+BAvBA^a + BA^v can be combined into B(Aa+Av)B(A^a + A^v), but this itself does not explain how different inputs use the corresponding modality branches. The authors need to clarify the modality selection at the token level, the text branch, and the handling of the MokA cross-modal interaction module.
  5. The bucket size analysis simultaneously changes the budget of each modality and the total number of parameters, making it difficult to distinguish whether the gain comes from the increase in the total budget or a more reasonable allocation method. The existing results mainly prove the flexibility of parameter configuration, but have not fully demonstrated that non-uniform modality allocation is superior to uniform allocation or cross-modal sharing under the same total budget.
Questions for authors

Please see "Critical Weaknesses".

4 comments
AuthorsOpenReview ↗

rebuttal (1/2)

Thank you for your insightful comments. We provide detailed responses to your concerns below.

Clarification and Addition of the Main Overview Figure: Thank you for this helpful suggestion. Firstly, HoRA specifically modifies the parameterization of the LoRA module while retaining the overall multimodal model pipeline. Therefore, the original Figure 1 focuses on the main methodological change: reconstructing modality-specific low-rank factors from compact parameter buckets through feature hashing. Secondly, we agree that a more comprehensive overview would improve clarity. We have added a new main figure (Figure VI, Appendix E) to the revised manuscript that integrates HoRA into the complete LoRA workflow and clearly illustrates the input processing, modality-specific branches, hashed reconstruction, and model output.

Comparison with NOLA and Uni-LoRA: Thank you for pointing out the close relationship with NOLA and Uni-LoRA. Overall, our contribution is not parameter sharing, signed reconstruction or gradient aggregation. Instead, HoRA introduces a modality-aware implicit parameterization that assigns independently configurable parameter buckets to different modalities, thereby enabling fine-grained modality-wise capacity allocation under a fixed-rank formulation.

Firstly, NOLA [1] and Uni-LoRA [2] were originally developed for unimodal LoRA compression and do not explicitly consider modality-specific branches or capacity allocation across modalities. Therefore, their original formulations cannot be directly applied to our multimodal setting without adaptation.

Secondly, to provide a controlled empirical comparison, we extend NOLA and Uni-LoRA to the same multimodal framework and use the same training and evaluation protocol. We configured their ranks to match HoRA’s trainable adapter parameter budget. As shown in Table R1, all methods use approximately 15.01M adapter parameters.

Table R1. Budget-matched comparison with the multimodal extensions of NOLA and Uni-LoRA on AVE using LLaMA2-7B.

MethodRankAdapter Params (M)AVE Acc. (%)
NOLA [1]815.0176.39
Uni-LoRA [2]815.0177.04
HoRA (Ours)415.0178.39

Furthermore, under the same adapter parameter budget, HoRA achieves 78.39% accuracy, outperforming NOLA and Uni-LoRA by 2.00 and 1.35 percentage points, respectively. Moreover, HoRA achieves this performance with a lower rank (r=4r=4 versus r=8r=8), demonstrating that its advantage cannot be explained solely by a larger rank or parameter budget. These results support the effectiveness of modality-specific implicit parameterization beyond generic LoRA compression and parameter-sharing mechanisms. We will include these comparisons and further clarify the distinctions from NOLA and Uni-LoRA in the revised manuscript.

References:

[1]. Abbasi Koohpayegani, Soroush, et al. Nola: Compressing lora using linear combination of random basis. ICLR, 2024.

[2]. Li, Kaiyang, et al. Uni-LoRA: One vector is all you need. NeurIPS, 2025.

Unbiasedness and Direct Bucket Optimization: Thank you for highlighting this distinction. Firstly, Proposition 3.1 is intended as a representational result, rather than a claim that a directly trained bucket remains an unbiased estimator at every iteration. Specifically, Eq. (6) constructs Wo{\bf W}^o from any reference factor AoA^o, and Eq. (7) reconstructs A^o\widehat{\bf A}^o. The result E[A^o]=Ao\mathbb{E}[\widehat{\bf A}^o]={\bf A}^o therefore shows that the hashed parameterization can approximate any explicit factor in expectation over the hash initialization. In actual HoRA training, AoA^o is never instantiated or optimized; Wo{\bf W}^o is directly optimized, while A^o\widehat{\bf A}^o is only an induced intermediate variable.

Secondly, the optimization follows the exact chain rules given in Eq. (11) of original paper. Thus, updates are performed entirely in the trainable parameter space {Wo,B}\{{\bf W}^o,{\bf B}\}, without requiring independent convergence of A^o\widehat{\bf A}^o. Moreover, the implementation samples {ho,ξo}\{h^o,\xi^o\} once and keeps them fixed. We will revise the notation that may suggest per-iteration hash resampling, state convergence for the fixed-hash optimization, and reserve the expectation over hashing for the approximation guarantee across random hash initializations.

Relevant discussion will be included in the final version.

AuthorsOpenReview ↗

rebuttal (2/2)

Clarification of Modality-Specific Routing: Thank you for identifying this ambiguity. Eq. (2) is a simplified module-level expression that omits token-level modality masks. Specifically, for mi∈a,vm_i\in \\{a,v\\} and token xi{\bf x}_i, the actual computation is:

hi=Wxi+BA^mixi,h_i={\bf W}{\bf x}_i+{\bf B}\widehat{\bf A}^{m_i}{\bf x}_i,

where mim_i is determined by the known modality boundary in the input sequence. Equivalently, H=WX+∑o∈a,vBA^oXPo{\bf H}={\bf W}{\bf X}+\sum_{o\in\\{a,v\\}}{\bf B}\widehat{\bf A}^o{\bf X}{\bf P}^o, where Pa{\bf P}^a and Pv{\bf P}^v are mutually exclusive token masks. Therefore, Aa{\bf A}^a and Av{\bf A}^v cannot be operationally combined, since they act on different token subsets. Firstly, we use two modalities here only for notational simplicity; the text branch is handled identically using At{\bf A}^t and its corresponding mask. Secondly, HoRA only replaces MokA’s modality-specific factors with hashed factors, while retaining its original cross-modal interaction mechanism. We will revise the equation and figure to make this routing explicit.

Relevant discussion will be included in the final version.

Fixed-Budget Modality-Wise Allocation: We thank the reviewer for highlighting that the original heatmap does not explicitly separate the allocation ratio from the total budget. However, Figure 2 already contains controlled comparisons with identical total bucket budgets. Firstly, with ∑oMˉo=0.9\sum_o\bar M^o=0.9, the uniform allocation (Mˉa,Mˉt,Mˉv)=(0.3,0.3,0.3)(\bar M^a,\bar M^t,\bar M^v)=(0.3,0.3,0.3) achieves 76.6%, whereas the non-uniform allocation (0.2,0.4,0.3)(0.2,0.4,0.3) achieves 78.8% (+2.2 points). Secondly, with ∑oMˉo=1.8\sum_o\bar M^o=1.8, (0.5,0.7,0.6)(0.5,0.7,0.6) achieves 78.3%, outperforming the uniform allocation (0.6,0.6,0.6)(0.6,0.6,0.6) at 76.8% (+1.5 points). Since the rank, shared factor B\bf B, adapted layers, and ∑oMˉo\sum_o\bar M^o are fixed, each pair has exactly the same number of trainable parameters. Furthermore, Figure 3 shows that HoRA consistently outperforms MokA at all three matched-parameter operating points on both AVE and MMBench. We will reorganize these existing results into explicit fixed-budget comparisons. Our claim is not that every non-uniform allocation is superior, but that HoRA enables task-dependent allocations that can outperform uniform allocation under the same budget.

Relevant discussion will be included in the final version.

The convergence proof has mixed the gradients of two different parameter spaces. The current derivation cannot prove the convergence of HoRA, including Eqs. (15)-(18) and Appendix Eqs. (75)-(99).

It is more like applying a general proof based on biased random gradients, and it has not yet been proven that the actual HoRA optimization process meets its prerequisites. I maintain my rating.

Thank you for clarifying this concern. We understand that the current presentation does not clearly distinguish the gradient with respect to the effective update ΔW\Delta{\bf W} from the gradient with respect to the trainable parameters θ=(B,{Wo})\theta=({\bf B},\{{\bf W}^o\}). Our original intention was to characterize how hashing perturbs the effective LoRA update and its gradient, since ΔW\Delta{\bf W} is the common object through which LoRA and HoRA modify the pretrained model. However, we agree that this analysis alone should not be presented as an O(∑o1/Mo)\mathcal{O}(\sum_o1/M^o) convergence guarantee for the actual HoRA optimization trajectory.

We will therefore clarify the scope by separating two results:

  • HoRA directly optimizes B{\bf B} and {Wo}\{{\bf W}^o\} using their exact chain-rule gradients, and its optimization behavior should be characterized in this trainable-parameter space.
  • The O(∑o1/Mo)\mathcal{O}(\sum_o1/M^o) result characterizes the hashing-induced approximation and gradient deviation for a hash-independent reference update, rather than a training-time convergence residual.

This correction narrows the theoretical claim but does not affect the HoRA formulation, implementation, or empirical results.

We sincerely thank the reviewer for helping us make this distinction precise.

Reviewer_ZvokOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose feature hashing-based multimodal low-rank adaptation (HoRA), a novel framework that reformulates multimodal LoRA as an implicit parameterization problem. HoRA constructs low-rank adaptations via feature hashing-based implicit parameterization, generating matrix entries on-the-fly from compact parameter buckets instead of explicitly storing them.

Critical strengths

1.By independently controlling the bucket size of each modality, HoRA introduces an additional degree of freedom for modality-wise parameter allocation within a fixed-rank formulation. 2.Formulate modality-wise adaptation through independently configurable parameter budgets and develop forward and backward computation schemes 3.Establish theoretical guarantees for both the approximation and optimization behavior of HoRA. We establish an unbiased reconstruction property and derive a collision-dependent approximation bound. We further show that the hashing-induced gradient bias is controlled by the modality-specific bucket sizes, leading to a corresponding convergence bound for the optimization process.

Critical weaknesses

1.HoRA controls modality-wise parameter budgets through bucket sizes under a fixed-rank formulation, but a similar goal can potentially be achieved by assigning different LoRA ranks to different modalities. The advantage of bucket-based allocation over heterogeneous-rank LoRA therefore needs stronger empirical and theoretical justification. 2.Feature hashing inevitably introduces parameter collisions, especially when the bucket size is small. Although the paper provides collision-dependent approximation bounds, excessive collisions may still reduce representation flexibility and hurt adaptation performance in practice. 3.HoRA introduces modality-specific bucket sizes as extra hyperparameters. Determining an appropriate parameter budget for each modality may require additional tuning, and the optimal allocation can vary across datasets and tasks, which may reduce the simplicity and generality of the method.

Questions for authors

1.The main motivation of HoRA is fine-grained modality-wise parameter allocation under a fixed rank. Why is there no direct comparison against heterogeneous-rank LoRA, where different modalities are assigned different ranks under the same total trainable-parameter budget? 2.The theoretical analysis takes expectation over hashing randomness, but the implementation appears to use a fixed xxhash mapping during training. How should the expectation overζt in Eqs. (15)–(18) be interpreted when the hash functions are sampled only once and then kept fixed? Does the convergence guarantee hold for a fixed realization of the hash functions, or only in expectation over random hash initializations? 3.The approximation and convergence bounds predict in Eqs.(16) dependence on modality-specific bucket sizes. Can you empirically verify this theoretical relationship by measuring reconstruction error, gradient deviation, and downstream performance as the bucket sizes vary?

3 comments
AuthorsOpenReview ↗

rebuttal (1/3)

Thank you for your insightful comments. We provide detailed responses to your concerns below.

Comparison with Heterogeneous-Rank LoRA: Thank you for suggesting this comparison. We address this concern from two perspectives: architectural applicability and empirical performance.

Firstly, HoRA supports parameter allocation across both independent and partially shared architectures. With independent modality-specific (Ao,Bo)({\bf A}^o,{\bf B}^o) pairs, both heterogeneous ranks and HoRA's bucket sizes enable modality-wise parameter allocation. With partially shared factors, such as a shared B{\bf B} and modality-specific Ao{\bf A}^o, the shared factor fixes a common inner dimension. Heterogeneous ranks therefore require modifying the shared-factor usage, e.g., through slicing or masking. HoRA instead adjusts modality-specific parameter counts while preserving the original sharing structure and factor dimensions. Thus, HoRA directly accommodates both settings, offering broader architectural applicability.

Secondly, we provide a structure- and budget-matched empirical comparison in Table R1. Specifically, Independent-branch HoRA uses separately hashed (A^o,B^o)(\widehat{\bf A}^o,\widehat{\bf B}^o) factors for each modality with rank 4. We set the normalized A-bucket sizes to 0.8/0.6/0.70.8/0.6/0.7 (T/V/A) and the normalized B-bucket size to 1/31/3 for every modality, yielding 15.01M adapter parameters. This configuration matches the independent modality-specific structure and parameter budget of heterogeneous-rank LoRA. Both methods use the same MokA framework, token routing, and training settings. We exhaustively evaluate all 10 positive-integer rank allocations satisfying rT+rV+rA=6r_T+r_V+r_A=6, including the uniform allocation. The best allocation, (3,2,1)(3,2,1), achieves 76.82% accuracy with 14.99M adapter parameters, whereas Independent-branch HoRA achieves 77.21% with 15.01M parameters, an improvement of 0.39 percentage points.

Table R1. Comparison with heterogeneous-rank LoRA on AVE (LLaMA2-7B-Chat).

MethodRank (T/V/A)Adapter Params (M)Accuracy (%)
HoRA (Ours)4/4/415.0178.39
Independent-branch HoRA4/4/415.0177.21
Heterogeneous-Rank LoRA1/1/414.9974.85
1/2/314.9975.15
1/3/214.9974.38
1/4/114.9976.42
2/1/314.9974.38
2/2/214.9974.20
2/3/114.9976.44
3/1/214.9975.27
3/2/114.9976.82
4/1/114.9975.40

Relevant discussion and experimental results will be included in the final version.

Impact of Hash Collisions: We agree that collisions constrain expressivity and may hurt performance when buckets are too small. Nevertheless, our results suggest a useful trade-off between parameter sharing and adaptation performance. Firstly, parameter redundancy may allow adaptation with fewer independent parameters, while collision-induced sharing may provide implicit regularization. These are only possible explanations and do not guarantee that collisions are beneficial. HoRA directly optimizes the shared buckets and controls sharing through bucket sizes. Secondly, under matched parameter budgets (Fig. 3), HoRA achieves competitive or better performance than MokA despite hash collisions. Furthermore, explicit L1L_1 regularization improves MokA (Fig. 7), and parameter sharing in HoRA may offer a similar regularization benefit. We will clarify this distinction and include the relevant discussion in the final version.

AuthorsOpenReview ↗

rebuttal (2/3)

Bucket-Size Selection and Tuning Cost: We agree that modality-specific bucket sizes introduce additional hyperparameters and therefore require tuning as part of the standard model-selection procedure. We have explicitly acknowledged this issue in the Limitations section, where we identify the heuristic or adaptive determination of modality-wise parameter allocation as an important direction for future work. Our additional analysis is not intended to claim that tuning is unnecessary, but to quantify the sensitivity of HoRA to bucket-size choices and provide practical starting configurations.

Firstly, as shown in Table R2, uniform allocation provides a simple starting point before modality-specific tuning. Under the same adapter parameter budget as MokA, uniform HoRA with (Mˉa,Mˉt,Mˉv)=(1.0,1.0,1.0)(\bar M^a,\bar M^t,\bar M^v)=(1.0,1.0,1.0) achieves 77.51% accuracy, outperforming MokA by 0.45 percentage points (pp). Under a more compressed uniform setting, (0.6,0.6,0.6)(0.6,0.6,0.6), HoRA uses 13.64M adapter parameters—28.6% fewer than MokA—while sacrificing only 0.29 pp in accuracy. Secondly, tuning the modality-wise allocation further improves the performance–parameter trade-off: the reported non-uniform configuration achieves 78.39% with 15.01M parameters, outperforming MokA by 1.33 pp while using 21.4% fewer parameters.

Table R2. Uniform and tuned bucket-size configurations on AVE with LLaMA2-7B-Chat.

MethodNormalized bucket sizes (Audio/Text/Vision)Adapter parametersAccuracyAccuracy difference vs. MokA
MokAN/A19.10M77.06%N/A
HoRA, uniform matched-budget1.0 / 1.0 / 1.019.10M77.51%+0.45 pp
HoRA, uniform compressed0.6 / 0.6 / 0.613.64M76.77%−0.29 pp
HoRA, non-uniform0.7 / 0.8 / 0.615.01M78.39%+1.33 pp

Furthermore, the complete sweep of 192 bucket-size configurations indicates that the tuning landscape is not excessively sharp. Specifically, 140 out of 192 configurations remain within 1 pp of the uniform (0.6,0.6,0.6)(0.6,0.6,0.6) setting, and the standard deviation across all configurations is 0.90 pp. To separate allocation sensitivity from changes in the total parameter budget, we additionally group configurations with the same total normalized bucket budget. Among the 17 fixed-budget groups containing at least three allocations, 16 exhibit an allocation-wise standard deviation below 1 pp, with a median of 0.81 pp. These results suggest that, although bucket-size tuning remains necessary, a broad range of configurations yields relatively stable performance.

Finally, we agree that the best allocation can vary across datasets and tasks. Therefore, we do not claim that a single allocation is universally optimal. Instead, the uniform configurations in Table R2 provide practical starting points, after which modality-specific tuning can be conducted according to the target parameter budget and validation performance. We will revise the paper to clarify this tuning procedure, its additional cost, and the limitation that an adaptive allocation strategy is not yet available.

AuthorsOpenReview ↗

rebuttal (3/3)

Fixed Hashing and the Scope of the Guarantee: Thank you for raising this important question. Overall, the hash realization ζ=ho,ξoo\zeta=\\{h^o,\xi^o\\}_o is sampled once at initialization and remains fixed throughout training. Firstly, conditional on any fixed ζ\zeta, the factors are parameterized as A^i,jo=ξo(i,j)Who(i,j)o\widehat{\bf A}^o_{i,j}=\xi^o(i,j)W^o_{h^o(i,j)}, and Eq. (11) gives the exact bucket gradient by the chain rule. Therefore, HoRA performs standard SGD on a fixed reparameterized objective, whose convergence to stationarity does not require resampling the hash functions. Secondly, the collision-dependent O(∑o1/Mo)\mathcal{O}(\sum_o1/M^o) approximation bound is averaged over hash realizations sampled at initialization and is not a uniform guarantee for every realization. Moreover, we will replace ζt\zeta_t with a single ζ\zeta and clearly distinguish convergence of the fixed-hash objective from the across-initialization approximation guarantee.

Relevant discussion will be included in the final version.

Empirical Validation of Bucket-Size Dependence: Thank you for this valuable suggestion. Overall, Eqs. (14) and (16) are upper bounds whose coefficients involve unknown or iteration-dependent quantities, such as LgL_g, ∣Ato∣\\|{\bf A}_t^o\\|, and ∣Bt∣\\|{\bf B}_t\\|. Therefore, the complete bounds cannot be numerically evaluated directly. Instead, we examine their predicted dependence on the bucket size through observable quantities.

Firstly, under fixed reference factors and loss, we vary only the normalized bucket size Mˉ=M/(rk)\bar M=M/(rk). We measure the reconstruction error Erec=Eζ[MSE(BA^,BA)]E _{\mathrm{rec}}=\mathbb{E} _{\zeta}\left[\mathrm{MSE}({\bf B}\widehat{\bf A},{\bf B}{\bf A})\right], and the gradient deviation Egrad=Eζ[MSE(gh,g⋆)]E _{\mathrm{grad}}=\mathbb{E} _{\zeta}\left[\mathrm{MSE}(g^h,g^\star)\right].

Table R3: Approximation, gradient, and downstream results under different bucket sizes. The first two metrics are averaged over two representative matrix dimensions, three reference seeds, and 20 hash realizations.

Mˉ\bar MErecE_{\mathrm{rec}} ↓EgradE_{\mathrm{grad}} ↓AVE accuracy ↑
0.19.98465.058276.67%
0.61.66810.807176.77%
1.01.00010.498077.51%

Increasing Mˉ\bar M tenfold reduces both reconstruction error and gradient deviation by approximately tenfold, supporting the predicted 1/M1/M dependence.

Secondly, downstream accuracy broadly improves as the bucket size increases, although the complete sweep exhibits minor nonmonotonicity. However, this does not contradict the theory: the bounds characterize approximation and gradient deviations, rather than guaranteeing monotonic generalization performance. Thus, these results provide empirical support for the predicted bucket-size dependence without claiming an exact numerical verification of the complete convergence bound.

Relevant experimental results and discussion will be included in the final version.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion phase comes to an end, we'd appreciate it if you could take a moment to review the authors' responses. If you have any remaining concerns or questions, please raise them with the authors before the discussion closes.

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.