- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how double descent behaves when scaling Low-Rank Adaptation (LoRA)'s rank under label noise. Double descent refers to test risk peaking near the interpolation threshold (where the model first fits the noisy training data) and then improving as capacity increases further. Across four vision backbones and a 7B language model, the authors find that test risk is lowest at rank one due to strong implicit regularization, rises steeply as the adapter learns the noisy labels to form an "interpolation cliff," and then descends slowly at higher ranks.
By comparing LoRA against dense fine-tuning, the paper proves that every post-peak rank that saves parameters performs worse than dense fine-tuning. Furthermore, even past full rank where LoRA is fully expressive, it continues to trail dense fine-tuning due to ill-conditioned factor geometry under AdamW. The authors demonstrate that optimizing directly in update space removes this condition-number penalty and recovers dense-equivalent performance.
Critical strengths
The paper provides clarity by separating five thresholds often confused in double descent studies: parameter count, empirical interpolation, peak risk, parameter parity, and expressivity parity. Demonstrating that parameter parity occurs before expressivity parity for projection layers cleanly explains why LoRA cannot both save parameters and match the generalization benefits of an expressive dense model.
Observations translate to real world label noise as highlighted by the CIFAR100N results.
The empirical analysis of optimization geometry at full rank is welcomed. By holding representation capacity fixed at full rank, the authors show that poor factor conditioning under coordinate-wise optimizers like AdamW causes the performance gap to dense fine-tuning. They confirm this mechanism by designing Lifted AdamW, which updates parameters in full matrix space and completely eliminates the condition-number penalty.
Critical weaknesses
The primary findings rely heavily on regimes with small sample sizes and artificial label noise. Under clean labels, test risk exhibits only a flat, mild rise without an interpolation cliff or visible second descent. Because practical fine-tuning datasets rarely contain enough label noise to trigger an interpolation peak, the prescriptive value of navigating the post-peak tail is limited for standard production workflows.
The empirical study focuses primarily on adapting query-key-value projections in the last six transformer blocks. Modern fine-tuning typically applies LoRA across all linear and attention layers throughout the entire network. Demonstrating that the parameter-saving window remains strictly inferior to dense fine-tuning when adapting the entire architecture would strengthen the generality of the conclusions.
Questions for authors
Does the performance gap between post-peak LoRA and dense fine-tuning persist when adapting all linear projections across every layer of the network, rather than only the attention projections of the final six blocks?
Did the authors observe any hints of an interpolation peak or rank non-monotonicity on standard generative language modeling or instruction-tuning benchmarks where label noise is absent?