SEESAW : BUDGET-PRESERVING RANK REALLOCATION FOR LOW-RANK ADAPTATION

Conventional LoRA assigns every Transformer layer the same rank, implicitly assuming uniform adaptation demand across layers. We instead cast layer-wise rank selection as a fixed-budget allocation problem and introduce Seesaw, which redistributes a constant global rank budget across layers using an exponentially-smoothed, rank-normalized gradient signal. At each allocation step, Seesaw identifies the highest-scoring receiver and lowest-scoring donor layer and, when their relative importance gap exceeds a threshold $\tau$, transfers a single rank unit between them—resizing the corresponding $A$/$B$ matrices and resetting optimizer state while leaving the total rank sum unchanged. On six NLU benchmarks with BERT-base, Seesaw outperforms uniform-rank LoRA, AdaLoRA, and DyLoRA at matched budgets; controlled ablations against fixed non-uniform and importance-inverted allocation directions show the gains stem specifically from correctly-directed reallocation, not non-uniformity alone. We further perform against standard LoRA on DeBERTa-v1, and benchmark against BitFit, Houlsby/Pfeiffer Adapters, LoRA, and AdaLoRA on DeBERTa-v3 under identical parameter budgets. Our results show that where a fixed adaptation budget is placed matters as much as how large it is, and that a lightweight first-order signal is enough to discover effective, architecture-dependent placements without second-order or SVD-based machinery.