Beyond the Mean Gradient: Spectral Allocation for Conditional Low-Rank Adaptation

Low-rank adaptation reduces fine-tuning cost by constraining weight updates to low-rank matrices, but standard LoRA still applies the same learned update to every input. Gradient-aware methods improve this shared update using an aggregate calibration gradient, which can suppress systematic differences in the update directions preferred by individual samples. We propose Spectral Gradient Basis LoRA (SGB-LoRA), which turns this within-task gradient variation into an input-conditioned low-rank update under a fixed per-matrix rank budget. SGB-LoRA decomposes per-sample gradients into a shared mean and centered residuals, extracts reusable residual modes, and allocates rank across mean and residual directions according to both gradient energy and low-rank compressibility. The retained bases initialize LoRA branches, while coordinates refitted on the same bases supervise a lightweight prompt-side router that predicts signed branch weights. After warm-up, the router and branches are jointly optimized, and inference requires neither targets nor gradients. We show that the proposed allocation is optimal for calibration-gradient reconstruction within a fixed recovered spectral family. Across instruction following, mathematical reasoning, code generation, and commonsense reasoning, SGB-LoRA consistently outperforms standard LoRA and strong gradient-aware baselines. Further analyses show that within-task gradient variation is compressible, stable across calibration subsets, and predictable from the prompt, supporting the mechanism behind the observed gains.