acceptodds
Desk rejected at ICLR 2027

SEESAW : BUDGET-PRESERVING RANK REALLOCATION FOR LOW-RANK ADAPTATION

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Conventional LoRA assigns every Transformer layer the same rank, implicitly assuming uniform adaptation demand across layers. We instead cast layer-wise rank selection as a fixed-budget allocation problem and introduce Seesaw, which redistributes a constant global rank budget across layers using an exponentially-smoothed, rank-normalized gradient signal. At each allocation step, Seesaw identifies the highest-scoring receiver and lowest-scoring donor layer and, when their relative importance gap exceeds a threshold , transfers a single rank unit between them—resizing the corresponding / matrices and resetting optimizer state while leaving the total rank sum unchanged. On six NLU benchmarks with BERT-base, Seesaw outperforms uniform-rank LoRA, AdaLoRA, and DyLoRA at matched budgets; controlled ablations against fixed non-uniform and importance-inverted allocation directions show the gains stem specifically from correctly-directed reallocation, not non-uniformity alone. We further perform against standard LoRA on DeBERTa-v1, and benchmark against BitFit, Houlsby/Pfeiffer Adapters, LoRA, and AdaLoRA on DeBERTa-v3 under identical parameter budgets. Our results show that where a fixed adaptation budget is placed matters as much as how large it is, and that a lightweight first-order signal is enough to discover effective, architecture-dependent placements without second-order or SVD-based machinery.

Primary area:
optimization
Keywords:
Parameter-Efficient Fine-Tuning, Low-Rank Adaptation, Dynamic Rank Allocation, Adaptive Rank, Gradient-Based Optimization, Fixed-Budget Adaptation, Transformers

Related papers

Discussion (0)

Sign in to comment.