acceptodds
Under review as a conference paper at ICLR 2027

HRHA: A High Rank and High Applicability Method for Parameter-Efficient Fine-tuning

Abstract

Parameter-efficient fine-tuning (PEFT) is essential for adapting foundation models, yet the widely adopted low-rank paradigm imposes an explicit rank constraint that couples parameter efficiency with adaptation flexibility. This becomes a bottleneck when tasks, models, and layers exhibit heterogeneous intrinsic ranks and spectral structures, as no single small rank is universally sufficient. We propose HRHA, a High-Rank and High-Applicability PEFT method that decouples parameter efficiency from low-rank parameterization. HRHA partitions each weight update into non-overlapping blocks and represents them with an SVD-inspired parameterization: left and right spectral bases are shared and frozen within each layer, while only block-specific diagonal coefficients are learned. Parameter efficiency is thus achieved through structured parameter sharing rather than rank reduction, and the resulting update is not explicitly rank-constrained, allowing full-rank or near-full-rank adaptation with substantially fewer trainable parameters than full fine-tuning. We theoretically establish that HRHA imposes no additional rank constraint, and empirically show that it attains near-full numerical rank and higher stable and effective ranks than representative high-rank LoRA variants. Extensive experiments on five benchmarks across diverse architectures and scales demonstrate consistent, stable, and parameter-efficient performance, highlighting the broad applicability of HRHA.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Parameter-Efficient Fine-tuning, High Rank, High Applicability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_y6R2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HRHA, a parameter-efficient fine-tuning method designed to bypass the rank bottleneck of LoRA. Instead of using a global low-rank approximation, HRHA partitions the weight update matrix into non-overlapping blocks. For each block, the method applies shared, frozen orthogonal bases to represent left and right singular vectors, and it learns only block-specific diagonal matrices. This approach allows the weight update to achieve a high effective rank and capture complex, localized features without inflating the trainable parameter budget.

The authors formally establish the theoretical convexity of their formulation and validate HRHA across a wide range of tasks, including commonsense reasoning, natural language understanding, and image classification, demonstrating strong performance against recent baselines.

Critical strengths

The paper demonstrates that a block-wise diagonal parameterization mapped through a shared orthogonal basis yields high effective and stable ranks in practice. This confirms that the model does not just nominally increase the rank mathematically, but genuinely utilizes a broader spectrum of active dimensions during learning.

Because the shared orthogonal bases remain frozen, the parameterization is linear with respect to the trainable block-diagonal coefficients. The authors neatly formalize how this linearity guarantees a convex optimization landscape under a single-layer squared loss, yielding favorable convergence properties.

Empirical validation is thorough across computer vision, natural language understanding, generative dialogue, and arithmetic reasoning. HRHA consistently matches or beats reported baselines like LoRA, HiRA, and SVFT.

Critical weaknesses

The primary weaknesses of the paper is missing citations and comparisons with highly relevant prior literature.

The core concept of freezing basis matrices and learning only sparse/diagonal scaling coefficients to decouple the trainable parameter count from the update's rank was already established by VeRA (Kopiczko et al., 2024) and NoLA (Koohpayegani et al., 2024). More recently, RandLoRA (Albert et al., 2025a) explicitly utilized diagonal scaling matrices applied to fixed random matrices to achieve full numerical rank updates with minimal parameters.

Additionally, HRHA's structural parameterization of formulating the weight update as a block-wise Kronecker product ΔW=(Im′⊗U)Σ~(In′⊗V)⊤\Delta W = (I_{m'} \otimes U) \tilde{\Sigma} (I_{n'} \otimes V)^\top also heavily overlaps with established tensor-based PEFT methods. KronA (Edalati et al., 2022) originally pioneered replacing the low-rank bottleneck with Kronecker product decomposition to preserve higher rank flexibility while maintaining parameter efficiency.

Because HRHA combines the frozen-basis diagonal scaling of VeRA/RandLoRA with the Kronecker factorization paradigm established by KronA, its claim of introducing a fundamentally new framework for rank-unconstrained PEFT is overstated. Because the paper does not cite or compare with these works, the true marginal contribution of HRHA's specific orthogonal initialization is obscured.

Minor: executing block-wise tensor contractions will reduce rcomputational throughput compared to standard LoRA (as pointed out in RandLoRA for example), yet the paper lacks empirical wall-clock time and peak memory profiling to substantiate its practical system-level efficiency.

Edalati, A., et al. "KronA: Parameter Efficient Tuning with Kronecker Adapter." arXiv:2212.10650 (2022).

Kopiczko, D. J., et al. "VeRA: Vector-based Random Matrix Adaptation." ICLR (2024).

Koohpayegani, S. A., et al. "NOLA: Compressing LoRA using Linear Combination of Random Basis." ICLR (2024).

Albert, P., et al. "RandLoRA: Full-rank parameter-efficient fine-tuning of large models." arXiv:2502.00987 (2025a).

Questions for authors

Did the authors measure empirical wall-clock time and peak memory consumption comparisons between HRHA, LoRA, and GraLoRA for a standard training setup?

Can HRHA be understood as a block-wise version of SVFT with shared basis matrices across all blocks?

Reviewer_c7LNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes HRHA, a parameter-efficient fine-tuning method that represents weight updates using block-wise transformations with shared, frozen orthogonal bases and trainable diagonal coefficients. The resulting parameterization requires only a fraction of the full weight matrix's trainable parameters while allowing updates to reach full rank. The authors provide theoretical analyses of the representation and optimization properties and evaluate HRHA across language understanding, reasoning, dialogue, and image classification tasks. The results demonstrate promising quality-parameter trade-offs, particularly in low-resource commonsense reasoning and image classification.

Critical strengths
  1. The proposed parameterization is simple but well-motivated. By combining fixed orthogonal bases with trainable block-wise coefficients, HRHA provides a structured alternative to conventional low-rank adaptation. Its parameter efficiency is straightforward to characterize, and the construction allows full-rank updates despite using substantially fewer trainable parameters.

  2. The evaluation covers a broad range of models and tasks, including language understanding, reasoning, personalized dialogue, and image classification. Some results demonstrate compelling parameter-efficiency trade-offs.

  3. The paper includes ablations on block size, basis initialization, and update rank, providing some insight into the design choices and representational behavior of the method.

Critical weaknesses
  1. The theoretical claims about representational capacity and optimization are stronger than the results establish. While HRHA can produce full-rank updates, its trainable parameters span only an (mn/b)-dimensional subspace of the full (mn)-dimensional weight space when the bases are fixed. Therefore, full-rank reachability does not imply unrestricted representational capacity, contrary to the discussion in Section 2.4. Moreover, the optimization analysis treats the structured update as an unconstrained matrix rather than optimizing the actual trainable coefficients, so its gradient and convergence conditions do not directly characterize HRHA. The theory should explicitly account for the parameterization constraints and distinguish achievable rank from approximation capacity.

  2. The comparison with closely related PEFT methods is incomplete. Methods such as VeRA[1] and FouRA[2] also employ fixed transformations with a small number of trainable coefficients, while MoRA[3] explicitly explores high-rank updates under limited parameter budgets. Without positioning HRHA against these approaches, it is difficult to determine the novelty and benefits of its particular block-wise parameterization.

  3. The explanations based on rank and orthogonality are not fully supported by the experiments. The rank analysis is concentrated on a single model and task, and some interpretations of numerical and effective rank are not mathematically justified.

  4. The practical resource benefits are not sufficiently evaluated. Although HRHA reduces the number of trainable parameters, this does not necessarily imply lower training memory or faster training, particularly when reconstructing full weight updates and storing fixed transformation bases.

[1] Dawid J. Kopiczko, et, al, VeRA: Vector-based Random Matrix Adaptation, ICLR 2024 [2] Shubhankar Borse, et, al, FouRA: Fourier Low-Rank Adaptation, NeurIPS 2024 [3] Ting Jiang, et, al, MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

Questions for authors

How does HRHA compare with existing PEFT methods in training throughput, peak GPU memory footprint, and adapter storage under identical experimental settings?

Reviewer_xkazOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HRHA, a parameter-efficient fine-tuning method that represents each weight update using non-overlapping blocks. Each block is parameterized as ΔWij=Udiag⁡(cij)V⊤\Delta W_{ij}=U\operatorname{diag}(c_{ij})V^\top, where the orthogonal bases U,VU,V are shared across blocks within a layer and frozen during fine-tuning, while the block-specific coefficients are optimized independently. This gives mn/bmn/b trainable parameters for an m×nm\times n update with block size bb, under the stated divisibility assumption. The authors establish full-rank attainability and analyze the linear parameterization under a single-layer squared-loss setting. Experiments cover five task families across language and vision models, supplemented by spectral analyses and ablations on block size and basis initialization. The central claim is that structured parameter sharing enables high-rank, region-wise adaptation with a small trainable-parameter budget.

Critical strengths

1. A simple and compact parameterization with full-rank attainability.

The combination of shared frozen bases and independently trainable block coefficients yields a straightforward parameterization with an explicit parameter count. The rank analysis establishes that this constrained representation can attain the maximum matrix rank. This provides a concrete construction for exploring high-rank adaptation under a limited trainable-parameter budget.

2. Broad evaluation with promising reported results.

The evaluation spans commonsense reasoning, natural language understanding, image classification, personalized dialogue generation, and arithmetic reasoning. In particular, Table 1 reports the highest average accuracy among the compared methods on all five evaluated models in the 15K commonsense-reasoning setting. This breadth and the reported low-resource performance provide encouraging evidence of the method’s potential utility, subject to resolving the reporting concerns below.

Critical weaknesses

1. The motivation, theoretical analysis, and experiments do not establish the specific benefit of block-wise adaptation.

The paper motivates block-wise adaptation as a way to accommodate regional heterogeneity in weight updates, but its motivation, theoretical analysis, and experiments do not establish the specific benefit of this design.

Motivation. The paper motivates block-wise adaptation by suggesting that SVFT’s global spectral parameterization may limit heterogeneous adaptation across different regions of a weight matrix. However, it provides no direct analysis identifying the regional patterns that are difficult to capture, or explaining why the proposed contiguous partition is well suited to them. HRHA assigns independent diagonal coefficients to different blocks while sharing the same fixed bases, but the manuscript does not establish why this particular structure better addresses the claimed limitation.

Theory. These results characterize the rank and optimization properties of the proposed parameterization, but do not establish the specific advantage of partitioning the update into multiple blocks. The theoretical analysis therefore does not demonstrate that the proposed partition improves the ability to represent heterogeneous updates.

Experiments. Table 6 varies block size and trainable-parameter count simultaneously: reducing bb from 256 to 32 increases parameters from 196K to 1572K. Thus, the observed gains cannot be attributed specifically to finer partitioning. Comparisons with SVFT also differ in parameter budget and coefficient structure. A parameter-matched global-versus-block-wise comparison, ideally accompanied by an analysis of regional update-approximation errors, is needed to validate the proposed motivation.

2. Multiple inconsistencies in the experimental reporting undermine confidence in the empirical conclusions.

Dataset–value misalignment in Table 3. The FFT results are attributed to Zheng et al.’s AdaMSS. Cross-checking its Table 1 shows that HRHA retains the source’s numerical ordering while changing the dataset-column ordering. As a result, six of the seven dataset entries in each FFT row are assigned to the wrong datasets. For example, the source reports ViT-Base FFT accuracy of 98.92 on CIFAR10 and 79.78 on StanfordCars, whereas HRHA’s Table 3 assigns 93.14 and 99.05 to those datasets. This permutation preserves the overall average, but it invalidates the affected per-dataset comparisons and their interpretation.

An inconsistent average in the central block-size ablation. The seven scores reported for b=128b=128 in Table 6 are 94.19, 83.34, 99.06, 98.97, 62.90, 98.97, and 93.49. Their arithmetic mean is approximately 90.13, while the reported average is 89.58. The discrepancy is too large to be explained by rounding. At least one individual entry or the average requires correction against the original records. Until the underlying entries are verified, the quantitative justification for the default block size—including the reported 1.52-point improvement from b=128b=128 to b=32b=32—cannot be reliably assessed.

Inconsistent block-size and parameter-count reporting. Appendix D.5 specifies b=256b=256 for dialogue generation, whereas Appendix D.6 specifies b=128b=128 for arithmetic reasoning; both describe adapting the query and value projections. Nevertheless, Tables 4 and 5 report identical HRHA parameter counts of 4.19M and 2.62M for the same two backbones. Under the paper’s mn/bmn/b formula, halving the block size should double the adapter parameter count when the adapted matrices are unchanged. The actual configurations and counting conventions therefore need clarification before the parameter-efficiency comparisons can be assessed reliably.

Questions for authors
  1. Clarification of the block-wise motivation.

Can the authors provide evidence supporting the hypothesis that SVFT’s global spectral parameterization may limit its ability to capture heterogeneous adaptation patterns across different regions of a weight matrix? Can the authors further clarify what properties of these patterns motivate the proposed contiguous partition and the use of shared bases with independently learned diagonal coefficients?

  1. Experimental evidence for why block-wise adaptation is effective.

Can the authors provide additional controlled experiments that identify what actually improves when block-wise adaptation is introduced, and how that improvement translates into better downstream performance? In particular, a parameter-matched comparison against a global fixed-basis parameterization would help isolate the effect of blocking from the increased number of trainable coefficients in Table 6. Analyses of regional update-approximation errors, learned block-wise coefficients, or changes in spectral properties could then test whether the proposed partition captures the claimed heterogeneous adaptation patterns and whether these changes account for the observed gains. Such evidence is needed to connect the motivating hypothesis, the block-wise mechanism, and the empirical results.

  1. Reconciliation of the reported results and configurations.

Can the authors provide corrected FFT dataset assignments in Table 3, resolve the inconsistency in the b=128b=128 row of Table 6, and reconcile the identical parameter counts in Tables 4–5 with the different block sizes specified in Appendices D.5–D.6? Please identify the affected entries using the original experimental records and explain which comparisons or conclusions change after correction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.