Characterizing Local Loss Gaps in Low-Rank Adaptation

Low-rank adaptation (LoRA) makes fine-tuning efficient, but a suitable rank is not known in advance. Informed rank adjustment requires diagnosing whether the current rank limits attainable improvement and how much additional improvement a specific increase could unlock. Performance gaps alone cannot answer these questions, because optimization and update scale confound the comparison. We propose a local diagnostic framework that compares the best dense and low-rank updates to the same module within a fixed Frobenius ball. A local quadratic approximation makes this comparison tractable. With separable curvature and feasible unconstrained optima, its gap equals the spectral tail of a curvature-whitened gradient. We control the discrepancy from the actual loss and correct for optima outside the radius. Under explicit approximation-error bounds, the resulting conditional intervals distinguish rank limitation, local sufficiency, and inconclusive cases relative to a prescribed tolerance. The reduction in the gap to the same dense comparator quantifies the optimal benefit of increasing rank; newly captured spectral energy predicts this benefit in the interior regime. Controlled objectives evaluate interval coverage and width, while RoBERTa/SST-2 experiments assess fixed-rank prediction and comparison sensitivity. Newly captured energy ranks observed upgrade gains with Spearman correlations of 0.903 on SST-2 and 0.734 on MRPC, compared with 0.333 and 0.414 for the current tail. The framework informs local rank adjustment while distinguishing optimal capacity gains from finite-budget training outcomes.