acceptodds
Under review as a conference paper at ICLR 2027

When Does Adaptive LoRA Rank Allocation Help? Signal Reliability and Resource Accounting

Abstract

Adaptive LoRA redistributes a limited rank budget, yet final accuracy alone does not isolate the benefit of rank placement. Allocation scores depend on existing ranks, while rank transfers can change parameter cost and model state. We analyze the scores and test transfers at fixed active parameter count. For a score based on loss progress, a sign change shared by all modules can reverse their rankings even when the ordering of score magnitudes persists. We separate the immediate loss change caused by a transfer from the loss change during continued training. Experiments cover eight GLUE tasks and quantized Llama commonsense workloads. On commonsense, Spearman correlations between consecutive measurements average 0.944 for score magnitudes and -0.155 for signed scores. At selected RTE checkpoints, online allocation activates 16.4% more parameters than uniform at equal total rank. At a shared RTE checkpoint, one of three module pairs yields lower final loss after moving rank to the higher-scored module, but a less favorable loss change over 20 updates than the reverse transfer. Synthetic controls show that concentrating rank resolves local capacity shortages, while repeated adjustment helps after demand shifts.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.