Singular Energy-Aware Layer-Wise Rank Allocation for Accurate Low-Rank Fine-Tuning

Fine-tuning large-scale neural networks in a low-rank space has become a popular strategy in the era of large language models (LLMs). However, most existing methods, such as low-rank reparameterization and low-rank adaptation (LoRA), assume a uniform rank across all layers. We provide practical insights into exploiting the learning capacity of low-rank models through layer-wise rank allocation. In particular, we find that the singular energy of gradients plays a key role in determining layer-wise ranks for strong downstream performance. Based on this observation, we propose a singular energy-aware (SEA) rank allocation scheme that can be applied to a broad class of low-rank fine-tuning methods. We further provide a theoretical analysis of the optimal rank in terms of retained energy after pruning. Extensive experiments demonstrate consistent improvements in downstream performance with negligible additional computational and memory overhead.