- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper develops a theoretical perspective on the generalization behavior of Low-Rank Adaptation (LoRA) compared with full fine-tuning. The authors argue that despite the widespread use of LoRA, its theoretical generalization properties remain insufficiently understood. The paper studies LoRA in a simple linear regression setting, comparing its excess risk with that of full fine-tuning and identifying conditions under which LoRA can generalize better. In particular, the authors show that when the required adaptation is approximately low rank and training observations are noisy, LoRA can outperform full fine-tuning. Restricting the adaptation rank reduces noise fitting but introduces approximation error when the true adaptation requires additional dimensions.
Critical strengths
The paper advances the theoretical understanding of LoRA's generalization behavior, complementing previous results on expressivity and optimization. In particular, it provides a characterization in the linear regression setting, including minimum-norm solution selection, and derives excess-risk bounds while studying the interplay among adaptation rank, noise, sample size, and task-shift structure. Another strength is the combination of theoretical analysis with empirical validation, including synthetic experiments and experiments using Qwen 2.5.
Critical weaknesses
I have two reservations, and a related question that would follow.
First, the conclusion that LoRA can generalize better than full fine-tuning when the required adaptation is approximately low rank and the observations are noisy seems somewhat intuitive and may already be expected by the community. This raises questions about the extent of the conceptual novelty.
Second, the abstract states that LoRA can outperform full fine-tuning when the difference between the pre-training and downstream tasks is effectively low rank. I believe this statement should explicitly mention the noisy setting considered in the analysis. Without this qualification, the claim could appear questionable from an optimization perspective. Making the assumptions more explicit in the abstract would help clarify the result.
Questions for authors
How do the authors view the novelty of their theoretical contributions relative to previous NTK-based analyses of fine-tuning, such as:
Afzal, Zahra Rahimi, et al. "Linearization explains fine-tuning in large language models." arXiv preprint arXiv:2602.08239 (2026).
In particular, I would be interested in understanding how the present theoretical perspective differs from or advances beyond such existing analyses.
2 comments
Thank you for the clarifications. The distinction from NTK-based analyses is clearer, and the abstract revision addresses my second concern. While I recognize the value of the theoretical bounds, my reservation regarding the overall novelty remains. I will maintain my rating.
We thank the reviewer for his comments.
Conceptual novelty and contribution:
Even if the qualitative conclusions were anticipated by the community, establishing them rigorously remains a contribution. Theorems 5 and 7 provide population excess-risk bounds that substantiate these expectations under explicit assumptions, covering both the overdetermined and underdetermined regimes.
Beyond these risk bounds, our paper studies solution selection in LoRA, identifying how initialization and regularization can lead to the estimators analyzed in our theory. Furthermore, our experiments demonstrate that properties of these solutions, such as robustness to label noise, also arise in real-world LLM fine-tuning. These are additional contributions that go beyond the intuitive expectation that low-rank adaptation can improve generalization in linear regression.
Explicit setting in the abstract:
We thank the reviewer for raising this issue! In the revised manuscript, we have updated the abstract to state explicitly that our theoretical setting includes noisy labels.
NTK and LLM fine-tuning:
NTK-based analyses provide a framework for understanding fine-tuning dynamics under specified conditions. Malladi et al. (2023) derived conditions under which prompt-based language-model fine-tuning exhibits kernel behavior and show that sufficiently large LoRA ranks approximately preserve the FFT kernel. Their results do not establish that standard LLM fine-tuning generally exhibits such behavior. Afzal et al. (2025) quantify how regularization toward pretrained parameters controls linearization error and relate the resulting NTK spectrum to empirical risk and layer selection. Their justification of linearization relies on this explicit regularization and does not automatically extend to conventional LLM fine-tuning. Tomihari and Sato (2024) use an NTK decomposition to explain how linear probing and classifier norms affect subsequent fine-tuning dynamics, feature preservation, and calibration, including an extension to LoRA. Yang et al. (2026) incorporate adaptive preconditioning into an optimizer-aware kernel and analyze errors in the kernel approximation. These studies primarily examine optimization dynamics and the validity or consequences of kernel approximations, whereas our main question concerns the population-risk consequences of restricting adaptation rank.
A few studies provide explicit generalization guarantees. Zeng et al. (2025) combine a PAC-Bayesian generalization bound with an NTK-based performance model for selecting pretrained LLMs, while Sun and Raskutti (2026) prove population prediction-error guarantees for early-stopped fine-tuning of sufficiently wide attention-based networks. More closely related, Jang et al. (2024) used sufficient rank conditions for favorable LoRA optimization landscapes and derived a nuclear-norm-dependent population excess-risk bound. Their guarantee concerns a penalized, perturbed objective and requires r(r+1)>2KN, where N is the number of training examples and K is the output dimension. It therefore does not cover low ranks, which is the operating mode of LoRA. Our risk bounds for the least-squares estimators impose no analogous lower bound on r. Overall, our contribution is an explicit statistical comparison of LoRA and FFT estimators. We derive exact FFT risk expressions and rank-dependent LoRA upper bounds in both the overdetermined and underdetermined regimes in linear regression, quantifying dependence on sample size, dimensions, noise covariance, and the adaptation spectrum. This identifies regimes in which reduced noise fitting outweighs the approximation cost of restricting rank.
We will add this discussion to the related work section, thanks!
References:
Malladi et al. (2023) - A Kernel-Based View of Language Model Fine-Tuning
Afzal et al. (2025) - Linearization Explains Fine-Tuning in Large Language Models
Tomihari and Sato (2024) - Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective
Yang et al. (2026) - Preconditioning Neural Tangent Kernel for Adaptive Optimization
Zeng et al. (2025) - LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Sun and Raskutti (2026) - A Theoretical Framework for LLM Fine-tuning Using Early Stopping for Non-random Initialization
Jang et al. (2024) - LoRA Training in the NTK Regime has No Spurious Local Minima