- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces HRHA, a parameter-efficient fine-tuning method designed to bypass the rank bottleneck of LoRA. Instead of using a global low-rank approximation, HRHA partitions the weight update matrix into non-overlapping blocks. For each block, the method applies shared, frozen orthogonal bases to represent left and right singular vectors, and it learns only block-specific diagonal matrices. This approach allows the weight update to achieve a high effective rank and capture complex, localized features without inflating the trainable parameter budget.
The authors formally establish the theoretical convexity of their formulation and validate HRHA across a wide range of tasks, including commonsense reasoning, natural language understanding, and image classification, demonstrating strong performance against recent baselines.
Critical strengths
The paper demonstrates that a block-wise diagonal parameterization mapped through a shared orthogonal basis yields high effective and stable ranks in practice. This confirms that the model does not just nominally increase the rank mathematically, but genuinely utilizes a broader spectrum of active dimensions during learning.
Because the shared orthogonal bases remain frozen, the parameterization is linear with respect to the trainable block-diagonal coefficients. The authors neatly formalize how this linearity guarantees a convex optimization landscape under a single-layer squared loss, yielding favorable convergence properties.
Empirical validation is thorough across computer vision, natural language understanding, generative dialogue, and arithmetic reasoning. HRHA consistently matches or beats reported baselines like LoRA, HiRA, and SVFT.
Critical weaknesses
The primary weaknesses of the paper is missing citations and comparisons with highly relevant prior literature.
The core concept of freezing basis matrices and learning only sparse/diagonal scaling coefficients to decouple the trainable parameter count from the update's rank was already established by VeRA (Kopiczko et al., 2024) and NoLA (Koohpayegani et al., 2024). More recently, RandLoRA (Albert et al., 2025a) explicitly utilized diagonal scaling matrices applied to fixed random matrices to achieve full numerical rank updates with minimal parameters.
Additionally, HRHA's structural parameterization of formulating the weight update as a block-wise Kronecker product also heavily overlaps with established tensor-based PEFT methods. KronA (Edalati et al., 2022) originally pioneered replacing the low-rank bottleneck with Kronecker product decomposition to preserve higher rank flexibility while maintaining parameter efficiency.
Because HRHA combines the frozen-basis diagonal scaling of VeRA/RandLoRA with the Kronecker factorization paradigm established by KronA, its claim of introducing a fundamentally new framework for rank-unconstrained PEFT is overstated. Because the paper does not cite or compare with these works, the true marginal contribution of HRHA's specific orthogonal initialization is obscured.
Minor: executing block-wise tensor contractions will reduce rcomputational throughput compared to standard LoRA (as pointed out in RandLoRA for example), yet the paper lacks empirical wall-clock time and peak memory profiling to substantiate its practical system-level efficiency.
Edalati, A., et al. "KronA: Parameter Efficient Tuning with Kronecker Adapter." arXiv:2212.10650 (2022).
Kopiczko, D. J., et al. "VeRA: Vector-based Random Matrix Adaptation." ICLR (2024).
Koohpayegani, S. A., et al. "NOLA: Compressing LoRA using Linear Combination of Random Basis." ICLR (2024).
Albert, P., et al. "RandLoRA: Full-rank parameter-efficient fine-tuning of large models." arXiv:2502.00987 (2025a).
Questions for authors
Did the authors measure empirical wall-clock time and peak memory consumption comparisons between HRHA, LoRA, and GraLoRA for a standard training setup?
Can HRHA be understood as a block-wise version of SVFT with shared basis matrices across all blocks?