HRHA: A High Rank and High Applicability Method for Parameter-Efficient Fine-tuning

Parameter-efficient fine-tuning (PEFT) is essential for adapting foundation models, yet the widely adopted low-rank paradigm imposes an explicit rank constraint that couples parameter efficiency with adaptation flexibility. This becomes a bottleneck when tasks, models, and layers exhibit heterogeneous intrinsic ranks and spectral structures, as no single small rank is universally sufficient. We propose HRHA, a High-Rank and High-Applicability PEFT method that decouples parameter efficiency from low-rank parameterization. HRHA partitions each weight update into non-overlapping blocks and represents them with an SVD-inspired parameterization: left and right spectral bases are shared and frozen within each layer, while only block-specific diagonal coefficients are learned. Parameter efficiency is thus achieved through structured parameter sharing rather than rank reduction, and the resulting update is not explicitly rank-constrained, allowing full-rank or near-full-rank adaptation with substantially fewer trainable parameters than full fine-tuning. We theoretically establish that HRHA imposes no additional rank constraint, and empirically show that it attains near-full numerical rank and higher stable and effective ranks than representative high-rank LoRA variants. Extensive experiments on five benchmarks across diverse architectures and scales demonstrate consistent, stable, and parameter-efficient performance, highlighting the broad applicability of HRHA.