- Rating
- 4(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes HoRA, a framework that reformulates multimodal LoRA as an implicit parameterization problem. It implicitly represents modality-specific low-rank factors using compact parameter buckets, allowing each modality’s trainable parameter budget to be adjusted independently at a fixed rank. It provides theoretical analyses and optimization, and evaluate on QWen2-7B and LLAMA2-7B, demonstrating its compective performence.
But I think there are some questions to be solved:
- The unbiased reconstruction analysis starts from a given matrix and studies hash-based aggregation and reconstruction, whereas the implemented method directly initializes and optimizes bucket parameters.
- Several gains on Qwen2-7B are small and require statistical evidence.
- The current capacity analysis changes both bucket sizes and the total parameter budget. Comparing uniform and nonuniform modality allocations under a fixed total parameter budget would better isolate the benefits of allocation.
- The modality notation is inconsistent: Figure 1(b) uses A^t, while line 134 uses A^a. Please check and unify the notation throughout the paper.
Critical strengths
-
The method is simple and well motivated, allowing flexible control of modality-specific parameter budgets at a fixed rank.
-
The experiments cover several multimodal tasks and include useful ablations, showing promising performance with fewer parameters. The paper is also clearly organized.
Critical weaknesses
-
The unbiased reconstruction analysis starts from a given matrix and studies hash-based aggregation and reconstruction, whereas the implemented method directly initializes and optimizes bucket parameters.
-
Several gains on Qwen2-7B are small and require statistical evidence.
-
The current capacity analysis changes both bucket sizes and the total parameter budget. Comparing uniform and nonuniform modality allocations under a fixed total parameter budget would better isolate the benefits of allocation.
-
The modality notation is inconsistent: Figure 1(b) uses A^t, while line 134 uses A^a. Please check and unify the notation throughout the paper.
Questions for authors
-
How does the unbiased reconstruction analysis apply to directly trained bucket parameters?
-
Are the small gains on Qwen2-7B consistent across repeated experiments?
-
Does nonuniform modality allocation outperform uniform allocation under a fixed total parameter budget?
-
Could you clarify the inconsistent notation between Figure 1(b) and line 134 and unify it throughout the paper?
3 comments
Thank you for your response. It addresses most of my concerns. The paper represents a substantial amount of work, and the experimental evaluation is thorough. If the paper is accepted, please double-check the notation and formatting throughout the final version. I will raise my overall rating to 4 and increase my confidence score.
Thank you for your thoughtful follow-up and constructive feedback throughout the discussion. We are glad that our responses have addressed most of your concerns. We will carefully double-check the notation and formatting throughout the final version and incorporate the clarifications and additional experimental results promised in our rebuttal. We appreciate your time and careful evaluation of our work.
rebuttal
Thank you for your insightful comments. We provide detailed responses to your concerns below.
Role of Unbiased Reconstruction in Direct Bucket Training: Thank you for raising this point. The unbiasedness result does not assume that HoRA explicitly compresses Ao during training. Firstly, Eqs. (6)–(7) of the original paper provide an analytical construction under which: Eho,ξo[Ao]=Ao.
This result characterizes the representational and collision-induced approximation properties of the hashed parameterization. Secondly, the implementation directly optimizes Wo, with Eq. (7) of the original paper used in the forward pass and the exact chain-rule gradient ∂Wko∂L=∑(i,j):ho(i,j)=kξo(i,j)∂Ai,jo∂L.
Therefore, Eq. (6) is an analytical construction rather than a training operation. We will clarify this distinction between representational analysis and direct bucket optimization.
Relevant discussion will be included in the final version.
Robustness of the Qwen2-7B Results across Random Seeds: We thank the reviewer for raising this important concern and agree that small performance margins should be supported by repeated experiments. Firstly, Appendix Table IV already reports four random-seed runs on AVE, where HoRA consistently outperforms MokA. Secondly, we conducted one additional independent run on the other five benchmarks, with the results summarized in Table R1.
Table R1: Robustness analysis of HoRA with Qwen2-7B. Results are reported as mean ± std over two random seeds.
Furthermore, the repeated runs support consistent improvements on M-AVQA and MMBench and an improvement on MME, although MME shows larger variation. We also acknowledge that HoRA is essentially tied with MokA on POPE and slightly lower on SEED-Bench, changing its ranking from second to third.
Finally, we acknowledge that two seeds provide limited statistical evidence. In the final version, we will run additional random-seed experiments on all Qwen2-7B benchmarks and update the reported means, standard deviations, rankings, and corresponding conclusions.
Fixed-Budget Capacity Allocation: We thank the reviewer for this important observation. Firstly, although Fig. 2 jointly varies bucket sizes and total parameters, it already contains controlled comparisons with the same total budget, because the shared factor B is unchanged and fixing ∑oMˉo fixes the total trainable parameters. For example, at ∑oMˉo=1.8, the uniform allocation (Mˉa,Mˉt,Mˉv)=(0.6,0.6,0.6) achieves 76.77%, while the non-uniform allocations (0.8,0.4,0.6) and (0.4,0.8,0.6) achieve 77.59% and 77.11%, respectively. Secondly, allocations with identical budgets produce different performance, confirming that the gain is affected by the allocation ratio rather than merely by additional parameters. Furthermore, we do not claim that non-uniform allocation is universally superior; instead, these results support HoRA’s ability to explore task-dependent modality allocations under a fixed budget. We will clarify this controlled interpretation of Fig. 2 in the final version.
Notation Consistency: Thank you for pointing out the inconsistent modality notation. Firstly, in the revised manuscript, we have unified the superscripts a, v, and t to consistently denote the audio, visual, and text modalities, respectively. Secondly, we have carefully checked and corrected the notation throughout Figure 1, the main text, equations, and appendix. This writing issue has now been fully resolved without affecting our method or results.