acceptodds
Under review as a conference paper at ICLR 2027

LoRA vs. Full Fine-Tuning: A Theoretical Perspective

Abstract

Fine-tuning adapts a pre-trained model to downstream tasks using a small amount of labeled data. Low-Rank Adaptation (LoRA) is an efficient fine-tuning method that reduces memory and computation costs while often achieving performance close to full fine-tuning. Despite its widespread use, the theoretical behavior of LoRA in terms of generalization is not yet well understood. In this paper, we study LoRA in a simple linear regression setting and compare its excess risk with that of full fine-tuning. Our analysis identifies regimes in which LoRA achieves lower excess risk than full fine-tuning in both overdetermined and underdetermined settings. Specifically, our theory predicts that LoRA can outperform full fine-tuning when the difference between the pretraining and the downstream tasks is effectively low-rank. We further show how the choice of LoRA rank affects generalization performance, explaining why using a very small rank can improve test accuracy in certain settings, even though it limits model expressivity. Finally, we support our theoretical results with experiments on practical tasks, suggesting that the identified tradeoffs and insights extend beyond linear regression.

Primary area:
learning theory
Keywords:
Fine-tuning, LoRA, excess risk bounds, linear regression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aednOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper develops a theoretical perspective on the generalization behavior of Low-Rank Adaptation (LoRA) compared with full fine-tuning. The authors argue that despite the widespread use of LoRA, its theoretical generalization properties remain insufficiently understood. The paper studies LoRA in a simple linear regression setting, comparing its excess risk with that of full fine-tuning and identifying conditions under which LoRA can generalize better. In particular, the authors show that when the required adaptation is approximately low rank and training observations are noisy, LoRA can outperform full fine-tuning. Restricting the adaptation rank reduces noise fitting but introduces approximation error when the true adaptation requires additional dimensions.

Critical strengths

The paper advances the theoretical understanding of LoRA's generalization behavior, complementing previous results on expressivity and optimization. In particular, it provides a characterization in the linear regression setting, including minimum-norm solution selection, and derives excess-risk bounds while studying the interplay among adaptation rank, noise, sample size, and task-shift structure. Another strength is the combination of theoretical analysis with empirical validation, including synthetic experiments and experiments using Qwen 2.5.

Critical weaknesses

I have two reservations, and a related question that would follow.

First, the conclusion that LoRA can generalize better than full fine-tuning when the required adaptation is approximately low rank and the observations are noisy seems somewhat intuitive and may already be expected by the community. This raises questions about the extent of the conceptual novelty.

Second, the abstract states that LoRA can outperform full fine-tuning when the difference between the pre-training and downstream tasks is effectively low rank. I believe this statement should explicitly mention the noisy setting considered in the analysis. Without this qualification, the claim could appear questionable from an optimization perspective. Making the assumptions more explicit in the abstract would help clarify the result.

Questions for authors

How do the authors view the novelty of their theoretical contributions relative to previous NTK-based analyses of fine-tuning, such as:

Afzal, Zahra Rahimi, et al. "Linearization explains fine-tuning in large language models." arXiv preprint arXiv:2602.08239 (2026).

In particular, I would be interested in understanding how the present theoretical perspective differs from or advances beyond such existing analyses.

2 comments

We thank the reviewer for his comments.

Conceptual novelty and contribution:
Even if the qualitative conclusions were anticipated by the community, establishing them rigorously remains a contribution. Theorems 5 and 7 provide population excess-risk bounds that substantiate these expectations under explicit assumptions, covering both the overdetermined and underdetermined regimes.

Beyond these risk bounds, our paper studies solution selection in LoRA, identifying how initialization and regularization can lead to the estimators analyzed in our theory. Furthermore, our experiments demonstrate that properties of these solutions, such as robustness to label noise, also arise in real-world LLM fine-tuning. These are additional contributions that go beyond the intuitive expectation that low-rank adaptation can improve generalization in linear regression.

Explicit setting in the abstract:
We thank the reviewer for raising this issue! In the revised manuscript, we have updated the abstract to state explicitly that our theoretical setting includes noisy labels.

NTK and LLM fine-tuning:
NTK-based analyses provide a framework for understanding fine-tuning dynamics under specified conditions. Malladi et al. (2023) derived conditions under which prompt-based language-model fine-tuning exhibits kernel behavior and show that sufficiently large LoRA ranks approximately preserve the FFT kernel. Their results do not establish that standard LLM fine-tuning generally exhibits such behavior. Afzal et al. (2025) quantify how regularization toward pretrained parameters controls linearization error and relate the resulting NTK spectrum to empirical risk and layer selection. Their justification of linearization relies on this explicit regularization and does not automatically extend to conventional LLM fine-tuning. Tomihari and Sato (2024) use an NTK decomposition to explain how linear probing and classifier norms affect subsequent fine-tuning dynamics, feature preservation, and calibration, including an extension to LoRA. Yang et al. (2026) incorporate adaptive preconditioning into an optimizer-aware kernel and analyze errors in the kernel approximation. These studies primarily examine optimization dynamics and the validity or consequences of kernel approximations, whereas our main question concerns the population-risk consequences of restricting adaptation rank.

A few studies provide explicit generalization guarantees. Zeng et al. (2025) combine a PAC-Bayesian generalization bound with an NTK-based performance model for selecting pretrained LLMs, while Sun and Raskutti (2026) prove population prediction-error guarantees for early-stopped fine-tuning of sufficiently wide attention-based networks. More closely related, Jang et al. (2024) used sufficient rank conditions for favorable LoRA optimization landscapes and derived a nuclear-norm-dependent population excess-risk bound. Their guarantee concerns a penalized, perturbed objective and requires r(r+1)>2KNr(r+1)>2KN, where NN is the number of training examples and KK is the output dimension. It therefore does not cover low ranks, which is the operating mode of LoRA. Our risk bounds for the least-squares estimators impose no analogous lower bound on rr. Overall, our contribution is an explicit statistical comparison of LoRA and FFT estimators. We derive exact FFT risk expressions and rank-dependent LoRA upper bounds in both the overdetermined and underdetermined regimes in linear regression, quantifying dependence on sample size, dimensions, noise covariance, and the adaptation spectrum. This identifies regimes in which reduced noise fitting outweighs the approximation cost of restricting rank.

We will add this discussion to the related work section, thanks!

References:
Malladi et al. (2023) - A Kernel-Based View of Language Model Fine-Tuning
Afzal et al. (2025) - Linearization Explains Fine-Tuning in Large Language Models
Tomihari and Sato (2024) - Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective
Yang et al. (2026) - Preconditioning Neural Tangent Kernel for Adaptive Optimization
Zeng et al. (2025) - LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Sun and Raskutti (2026) - A Theoretical Framework for LLM Fine-tuning Using Early Stopping for Non-random Initialization
Jang et al. (2024) - LoRA Training in the NTK Regime has No Spurious Local Minima

Thank you for the clarifications. The distinction from NTK-based analyses is clearer, and the abstract revision addresses my second concern. While I recognize the value of the theoretical bounds, my reservation regarding the overall novelty remains. I will maintain my rating.

Reviewer_gFjAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores excess risk of Low-Rank Adaptation (LoRA) in comparison with that of Full Fine-Tuning (FFT) for a multivariate linear regression model.

To the end, low-rank adaptation problem is identified as a rank-r constrained empirical risk minimization problemwhere the global minimizer is characterized using a whitened Eckart--Young truncation of the least-squares estimator (Theorem 1).

Mathematical results show that this solution minimizes distance to the parameter matrix of a pre-trained model across unitarily invariant norms. The results also suggest hat Gradient Descent (GD) with standard initialization selects this solution. Finally, exact excess risk for unregularized FFT was compared against upper bounds for LoRA for overdetermined and underdetermined regimes.

Theoretical claims and results are examined on some benchmark using a Qwen2.5 model.

Critical strengths

Experimental analyses are interesting and go beyond standard benchmarking. The analyses demonstrate a reasonable qualitative alignment with the linear model's spectral decay predictions.

Critical weaknesses

First, the paper claims that they explore a fundamental problem regarding generalization properties of LoRA in comparison with FFT which is modeled as unregularized OLS. Unregularized OLS suffers from catastrophic variance blow-up under high noise. In experimental analyses, FFT is heavily regularized with weight decay and early stopping.

A more rigorous comparison requires testing LoRA against ridge regularized FFT. This conflates the benefit of generic ℓ2\ell_2 regularization with the specific benefit of low-rank spectral truncation.

Theorem 2 claims that GD with standard initialization selects the minimum-norm LoRA solution. However, the theorem assumes that GD converges to a global minimizer which is the entire difficulty of non-convex matrix factorization. Since the objective is non-convex, it contains strict saddles and rank-deficient invariant subspaces where GD can get permanently trapped. Moreover, the proof of Theorem 8 is simply an induction showing row-space invariance. Therefore, it is misleading to package a basic row-space invariance property as a major theorem on GD convergence while assuming away global convergence.

In the theoretical setup, there is no shared representation or covariance alignment modeled between pre-training and fine-tuning. Thereby, the setup reduces entirely to classical zero-initialized Reduced-Rank Regression (RRR) vs. OLS. Consequently, Theorem 1 is a restatement of the RRR estimator. Moreover, the excess risk bounds closely replicate high-dimensional oracle inequalities for RRR. As an application of RRR, this can be considered nice. However, this also limits theoretical novelty of the paper.

Questions for authors

If you include a curve for optimally tuned ridge-regularized FFT in Figure 1, does LoRA still hold a generalization advantage? If so, in what specific spectral regimes does hard rank-r truncation beat ℓ2\ell_2 shrinkage?

Is the linear factor of r in the cross-approximation term of Theorems 5 & 7 tight, or is it an artifact of the operator norm bound in Lemmas 9–10? Can you provide a matching lower bound for the excess risk of A^LoRA\hat{A}_{LoRA}?

Can you provide explicit conditions on C0C_0 and step size ηt\eta_t under which GD with B0=0B_0=0 is guaranteed to converge to the global minimizer?

In the experimental analyses, several class and word level noise generation operations are proposed and utilized. How do the different and reflect the theoretical noise using in the mathematical setup?

2 comments
AuthorsOpenReview ↗

Official Comment by Authors - Part 1

We thank the reviewer for his comments and his genuine interest in our work!

Analyzing weight decay and early stopping:
In this work, we study the fully optimized loss without any regularization as a first step. We agree that extending our analysis to incorporate also weight decay and early stopping is a valuable research direction. Unfortunately, these are out of scope for this paper. We leave these to future work.

Optimally tuned ridge-regularized FFT:
This paper studies how rank constraints, particularly those induced by LoRA, affect generalization; an analysis of ridge regularization is beyond its scope. Nevertheless, a fair comparison should allow both methods to be regularized and tuned. A natural oracle benchmark would therefore compare optimally tuned ridge-regularized FFT with optimally tuned LoRA using squared ℓ2\ell_2 penalties on its factors. In our linear setting, these factor penalties correspond, at global optima, to nuclear-norm regularization of the adaptation matrix. When the LoRA rank is large enough to approximate well the nuclear-norm solution, this comparison reduces to choosing between squared Frobenius and nuclear-norm regularization of the update. Prior empirical comparisons demonstrate that neither method consistently dominates: Yuan et al. (2007) find that nuclear-norm regularization can outperform ridge when the signal is concentrated in a few singular directions, whereas ridge can perform better when the spectrum is flat. Although their nuclear-norm method is formulated using a norm constraint, convex Lagrangian duality connects it to the penalized formulation for an appropriately chosen multiplier. This equivalence holds for each fixed dataset, with a data-dependent correspondence between the constraint radius and penalty coefficient.

The population-optimal regularization strength depends on unknown properties of the data distribution, so oracle tuning provides an idealized benchmark. In practice, regularization strengths can be selected using validation data, and performance additionally depends on the available validation samples and tuning budget. A comparison under matched tuning budgets would therefore be practically informative and would complement our analysis, while addressing an additional question beyond the present focus on rank constraints. We leave this to future work.

Factor of r\boldsymbol{r} in the cross-approximation term:
The factor rr in the cross-approximation term is an artifact of the operator-norm bounds in Lemmas 9–10. This becomes apparent in the infinite-sample limit of the overdetermined regime, where the empirical problem approaches its population counterpart and the limiting error is given by the third term of the bound (see Proposition 8 in the appendix). In our bound, however, only the noise term vanishes; the cross-approximation term remains, creating a nonvanishing gap above the limiting error.

We have an alternative derivation that eliminates the cross-approximation term but multiplies the final term by the expected condition number of Σ^xx−1Σxx\widehat{\boldsymbol{\Sigma}}_{xx}^{-1}\boldsymbol{\Sigma}_{xx}. The expected value of this condition number tends to one in the limit of infinite samples, which means that we can be tight in this limit. However, this factor can be large even with a moderate number of samples, so we chose to present the current bound.

Matching lower bound:
Although we do not have a matching lower bound in the general case, we can compute the exact excess risk in certain simple settings, such as when the ground-truth adaptation matrix is zero and the data are Gaussian. In this case, the exact risk matches the noise term in our bound, confirming that the bound captures the correct dependence on rank and dimensions. We can provide the analysis upon request of the reviewer.

Converge to a global minimizer:
Thanks for raising this issue! We added 6 pages of proof showing that under certain conditions, GD on LoRA converges to a global minimizer in linear regression. The main condition is that the initial column subspace of $ \boldsymbol{C} $ should not be too far from an optimal subspace. Please see App. VIII in the revised manuscript for the full details and proof.

References:
Yuan et al. (2007) - Dimension reduction and coefficient estimation in multivariate linear regression

AuthorsOpenReview ↗

Official Comment by Authors - Part 2

Theoretical and practical label noise:
In the LLM experiments, both noise protocols corrupt the training labels: for BoolQ, we flip the binary answer (“Yes”/“No”), while for CommonsenseQA, we replace the correct answer letter with one of the four incorrect alternatives, chosen uniformly at random. Since these labels are represented as answer words or letters, replacing the answer text implements class-label corruption; it is not a separate word-level noise mechanism.

Our theoretical model assumes y=A⋆x+εy=A^\star x+\varepsilon, where ε\varepsilon is independent, zero-mean Gaussian noise with covariance Σεε\Sigma_{\varepsilon\varepsilon}. The synthetic linear-regression experiments follow this model directly. The LLM corruptions provide a qualitative analogue of unreliable supervision, but do not satisfy these assumptions exactly. Although label replacement can be expressed as an additive perturbation of a one-hot label vector, this perturbation is discrete, label-dependent, and generally not zero-mean. Consequently, the corruption percentage does not directly correspond to the Gaussian noise variance. The LLM experiments assess whether the qualitative benefit of restricting noise fitting through low-rank adaptation extends beyond the precise theoretical setting. The experimental results suggest that these corruption mechanisms lead to the same qualitative behavior predicted by the theory: smaller LoRA ranks can improve robustness and generalization as training-label noise increases.

Misleading presentation:
Theorem 8 uses row-space invariance, but its argument does not rely on this property alone. It also builds on our characterization of LoRA minimizers in Eq. (47), derived in our proof of Theorem 1. Moreover, our result is stronger in several respects than previously published results. For example, Gunasekar et al. (2017) address implicit solution selection as a central research question. Their main result, Theorem 1, considers a full-dimensional two-factor parameterization without rank constraints, uses gradient flow, and imposes restrictions on the initialization of all factors. Our result applies to arbitrary depth, accommodates rank constraints through architectural bottlenecks, and concerns discrete GD. The only initialization condition used in our proof is zero initialization of the input-side factor; the nonzero initialization requirements on the remaining factors exclude trivial stationary initializations but are not used in the argument. These features justify presenting the result as a theorem.

The main purpose of Theorem 8 is to show a bias toward the solution we analyze. As in related implicit-bias results, our statement is conditional on convergence to a global minimizer. Proving convergence typically requires strong assumptions, whereas our selection result holds whenever GD converges globally, even when those restrictive sufficient conditions are not satisfied. Moreover, the revised paper now addresses convergence directly: Appendix VIII proves that GD converges to a global minimizer in the two-factor setting under specified conditions.

Finally, neither Theorem 8 nor its specialization, Theorem 2, is presented as a major theorem. A paper’s main contributions are typically displayed in the central sections of the main text, whereas our Theorem 8 appears on page 20 in the appendix and Theorem 2 in the preliminaries. Their placement and presentation reflect their supporting role in connecting optimization to the estimator we study. Therefore, the accusation of misleading presentation is unjustified.

References:
Gunasekar et al. (2017) - Implicit Regularization in Matrix Factorization

Reviewer_ochEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper compares LoRA and full fine-tuning in a linear regression setting, where adaptation means estimating a change from a fixed pretrained model. It characterizes the fitted LoRA solution, studies when gradient descent or vanishing regularization selects that solution, and derives excess-risk bounds in both overdetermined and underdetermined regimes. The central finding is a tradeoff: restricting the update rank can reduce noise fitting, but introduces approximation error when the required task adaptation has substantial components beyond that rank. Experiments on synthetic regression and Qwen2.5 fine-tuning examine whether these predictions extend beyond the theoretical setting.

Critical strengths
  • The analysis is framed well, emphasizing that what matters is the rank of the change required from pretraining, rather than the rank of the target model itself. The bounds distinguish the cost of fitting noise from the cost of discarding useful adaptation directions. In the underdetermined setting, they also identify the error from unobserved input directions. This gives a clear account of when restricting rank could help.
  • The controlled experiments illustrate the tradeoff clearly. The linear experiments vary noise, sample size, and adaptation rank with known ground truth. Figure 2a also provides a useful practical example: rank-1 LoRA loses some accuracy with clean labels but becomes preferable as corruption increases.
Critical weaknesses
  • The connection between the analyzed estimator and practical LoRA training is incomplete. The theory (1) assumes linear regression (2) assumes the selection of a certain global estimator. They show how that could arise under certain initialization conditions or regularizers, but it's not established that the LLM LoRa adapters training with gradient descent based optimizers will result in solutions that match the properties of the theory.
  • The LLM experiments provide partial qualitative support rather than a direct test of the theoretical explanation. Figure 2a shows the clearest approximation-noise tradeoff. Figure 2b supports greater robustness at small ranks, but rank 1 is already best without injected corruption, so the competing approximation cost is not apparent. Figure 3 varies sample size without establishing the corresponding theoretical regime or locating an analogous interpolation threshold, making it difficult to assess whether its trends test the sample-size predictions. These observations need not contradict the theory, which permits different regimes, but the paper does not estimate the relevant quantities to establish which regime applies. The correspondence is therefore less conclusive than the presentation suggests.
  • The paper offers limited practical guidance for choosing a method or rank. The comparison uses exact FFT risk expressions and LoRA upper bounds with hidden constants, conditioning factors, and unknown adaptation spectra. This identifies favorable scaling regimes but does not determine the crossover or best rank for a given task. It would also be useful to establish how much of the advantage remains against FFT with appropriately tuned regularization or early stopping, since those mechanisms can also reduce noise fitting.
Questions for authors
  • The approximation-estimation tradeoff is established in reduced-rank regression literature, notably Bunea et al. (2011). Can you discuss the additional technical contribution of Theorems 5 and 7 over those fixed-design bounds?

[1] Bunea, Florentina, Yiyuan She, and Marten H. Wegkamp. "Optimal selection of reduced rank estimators of high-dimensional matrices." (2011): 1282-1309.

2 comments

We thank the reviewer for his comments.

Q1: Technical contribution over prior work:
As the reviewer noted, prior work, including Bunea et al. (2011), had studied the fixed-design setting, which corresponds to evaluating a trained model on the training data. This notion of risk does not capture test-risk phenomena such as double descent and instability at the interpolation threshold, and the distinction between the over- and underdetermined regimes is less pronounced. In contrast, we study population (test) risk, which requires accounting for the mismatch between training and population statistics and thus poses additional challenges in the derivations. The two regimes also exhibit distinct behavior, where in the underdetermined regime, errors along input directions unobserved during training contribute to population risk.

Beyond these risk bounds (Theorems 5 and 7), our paper makes further contributions. While Bunea et al. study the same low-rank loss minimizer, we also show how it can be obtained in a learning setting. Furthermore, we demonstrate through experiments that properties of the low-rank linear regression estimator, such as robustness to label noise, also arise in real-world LLM fine-tuning.

Experiments, connection to LLMs, and practical guidance:
The three concerns are closely related. Our theory of reduced-rank linear regression identifies regimes in which LoRA improves generalization, particularly when the training data are noisy or the task shift is effectively low-rank. Extending these theoretical results to deep nonlinear models remains challenging, as discussed in the limitations section of the appendix. We demonstrate their relevance to practical LLM fine-tuning through experiments with three models across two tasks. We chose CommonsenseQA and BoolQ because they exhibit contrasting behavior: small ranks suffice for strong performance on CommonsenseQA, whereas BoolQ benefits from higher ranks in low-noise settings.

Determining the effective rank of the required adaptation in advance is difficult, so these findings cannot directly help to determine the optimal rank for a given task. Nevertheless, our results still suggest a practical guideline: when fine-tuning on noisy training data, smaller LoRA ranks can improve robustness and generalization by limiting noise fitting.

Thank you for the response. I think my question is answered, but the specific concerns I had relating to practicality, applying the theory to LLMs, and the applicability of the LLM experiments remain. The dynamics in practical setups might be very different than those studied in the paper. I don't think there is enough evidence to show that this theory would apply to those setups. I think it is particularly important for this paper since LoRA is pretty much exclusively used in those types of large scale non-linear regimes. I will be maintaining my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.