acceptodds
Withdrawn from ICLR 2027

LoRA's Second Descent Extends Beyond Parameter Parity

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Double descent has sparked considerable interest, with recent work relating it to the data, the model and the learning configuration. Practical fine-tuning commonly involves training a small adapter on top of frozen pretrained weights, as in low-rank adaptation (LoRA). The adapter's rank is the hyperparameter that sets its capacity, yet how this rank relates to double descent has not been well explored. We quantify this relation under label noise on four vision backbones and a 7B language model with a module-matched rank sweep (MMRS), which extends past full rank and compares every rank with dense fine-tuning of the same modules, paired by seed. On DeiT-Tiny, risk is lowest at rank one and rises sharply as the adapter becomes able to fit the noisy labels, forming an interpolation cliff. Past the peak, risk falls again, but every tested post-peak rank that still saves parameters remains above dense risk. Rank-one LoRA outperforms dense fine-tuning on three of the four vision backbones, consistent with strong regularization at small rank. LoRA thus exhibits a second descent, but matches dense risk only after losing its parameter advantage, first on DeiT-Tiny at four times dense's projection weights. Code is available at https://anonymous.4open.science/r/lora-second-descent-C7B5.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
low-rank adaptation, LoRA, double descent, parameter-efficient fine-tuning, label noise, interpolation threshold, dense fine-tuning, rank selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_q5VfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how double descent behaves when scaling Low-Rank Adaptation (LoRA)'s rank under label noise. Double descent refers to test risk peaking near the interpolation threshold (where the model first fits the noisy training data) and then improving as capacity increases further. Across four vision backbones and a 7B language model, the authors find that test risk is lowest at rank one due to strong implicit regularization, rises steeply as the adapter learns the noisy labels to form an "interpolation cliff," and then descends slowly at higher ranks.

By comparing LoRA against dense fine-tuning, the paper proves that every post-peak rank that saves parameters performs worse than dense fine-tuning. Furthermore, even past full rank where LoRA is fully expressive, it continues to trail dense fine-tuning due to ill-conditioned factor geometry under AdamW. The authors demonstrate that optimizing directly in update space removes this condition-number penalty and recovers dense-equivalent performance.

Critical strengths

The paper provides clarity by separating five thresholds often confused in double descent studies: parameter count, empirical interpolation, peak risk, parameter parity, and expressivity parity. Demonstrating that parameter parity occurs before expressivity parity for projection layers cleanly explains why LoRA cannot both save parameters and match the generalization benefits of an expressive dense model.

Observations translate to real world label noise as highlighted by the CIFAR100N results.

The empirical analysis of optimization geometry at full rank is welcomed. By holding representation capacity fixed at full rank, the authors show that poor factor conditioning under coordinate-wise optimizers like AdamW causes the performance gap to dense fine-tuning. They confirm this mechanism by designing Lifted AdamW, which updates parameters in full matrix space and completely eliminates the condition-number penalty.

Critical weaknesses

The primary findings rely heavily on regimes with small sample sizes and artificial label noise. Under clean labels, test risk exhibits only a flat, mild rise without an interpolation cliff or visible second descent. Because practical fine-tuning datasets rarely contain enough label noise to trigger an interpolation peak, the prescriptive value of navigating the post-peak tail is limited for standard production workflows.

The empirical study focuses primarily on adapting query-key-value projections in the last six transformer blocks. Modern fine-tuning typically applies LoRA across all linear and attention layers throughout the entire network. Demonstrating that the parameter-saving window remains strictly inferior to dense fine-tuning when adapting the entire architecture would strengthen the generality of the conclusions.

Questions for authors

Does the performance gap between post-peak LoRA and dense fine-tuning persist when adapting all linear projections across every layer of the network, rather than only the attention projections of the final six blocks?

Did the authors observe any hints of an interpolation peak or rank non-monotonicity on standard generative language modeling or instruction-tuning benchmarks where label noise is absent?

Reviewer_Lu27OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies LoRA rank sweeps under label noise and compares the post-peak branch with module-matched dense fine-tuning

Critical strengths

The experiments include multiple backbones, paired seeds and several optimization interventions.

Critical weaknesses
  1. The central comparison has a model-selection problem: Appendix C.2 explicitly selects learning rates using test NLL, and changing training seeds does not restore test-set independence. Joint selection of learning rate, weight decay and stopping epoch substantially reduces the cliff, but includes no corresponding dense arm, leaving the practical comparison unresolved.

  2. MMRS largely packages standard experimental controls, while the frozen-factor interventions demonstrate coordinate dependence without explaining the second descent of ordinary LoRA.

  3. The scope remains narrow: even the 7B experiment is noisy AG News classification, providing little evidence about practical generative fine-tuning.

  4. The presentation also lacks a unified experimental pipeline; essential decisions are scattered across dozens of experiment blocks, and large gaps around sections and figures reinforce the fragmented, claim-first writing. Despite the extensive AI-assisted research disclosed by the authors, the paper reads as accumulated sweeps rather than a coherent causal investigation. Experiment volume does not compensate for limited explanatory progress

Questions for authors

please refer the weaknesses.

Reviewer_cYLuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates double descent in LoRA under label noise. The authors propose a module-matched rank sweep to compare different LoRA ranks with dense fine-tuning. Experiments on four vision backbones and a 7B language model reveal an interpolation cliff followed by a second descent. However, LoRA only approaches dense performance after losing its parameter-efficiency advantage. The paper also investigates how optimization affects test risk, even when LoRA and dense fine-tuning have the same expressive power.

Critical strengths

The paper provides extensive experiments across multiple models and LoRA ranks, with additional analyses of different training settings.

Critical weaknesses
  • The practical value and generality of the observed second descent remain unclear. Some results are counterintuitive, such as rank-8 LoRA performing worse than rank-0 under label noise, while hyperparameter tuning substantially reduces the interpolation cliff. Moreover, the second descent offers limited practical value, as it occurs after LoRA loses its parameter-efficiency advantage.
  • The experimental results are presented in a poorly organized style, particularly in the appendix. Several important tables lack proper captions, and some listed experiments have no reported results. The authors should substantially improve the organization, clarity, and completeness of the experimental results.
Questions for authors
  1. Appendix E.2 shows that hyperparameter tuning and early stopping significantly reduce the interpolation cliff, with all selected checkpoints remaining below interpolation. When would the observed second descent be useful in practice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.