acceptodds
Under review as a conference paper at ICLR 2027

High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model

Abstract

We develop a high-dimensional statistical theory of low-rank adaptation (LoRA) in attention models, capturing the interplay between pre-training and fine-tuning. We introduce a solvable framework in which a single-head attention layer is first pre-trained on a data-abundant task and subsequently adapted via a rank-one LoRA update on limited data. In the high-dimensional limit, we obtain an asymptotic characterization of both stagesin terms of a finite set of order parameters, yielding explicit predictions for test errors and representation alignment. Our analysis shows that the impact of pre-training on LoRA is summarized by an effective noise component, from which we derive prescriptions for the optimal pre-training procedure. We also demonstrate a regime with a mismatch between the value of the test error and representation quality, and propose an application of our theory to active fine-tuning.

Primary area:
learning theory
Keywords:
fine-tuning, LoRA, attention, solvable model, high-dimensional

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vwR4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This study analyzes LoRA performance in a simplified high-dimensional attention model. They consider two stages learning, pretraining followed by LoRA fine-tuning, in a teacher-student setting. They focus on what happens when pretraining is insufficient. Their analysis shows that insufficient pretraining can be represented as effective noise. When pretraining and LoRA training use the same input data, high test performance does not necessarily mean accurate parameter prediction. Based on these findings, they also propose practical heuristics to improve performance.

Critical strengths
  • Given that it is certainly a non-trivial question how performance changes through post-training when pre-training is insufficient, the results of this study make a certain contribution to the success background of LoRA. In particular, it is interesting that, given the same inputs, high parameter prediction performance can sometimes coincide with low generalization performance.
  • Furthermore, it is interesting that the impact of insufficient pre-training is concentrated into effective noise, and that heuristics to reduce this in downstream tasks (such as selecting data where upstream task impacts do not appear, or optimizing the scale) are shown to be effective. I feel this provides suggestive insights for algorithmic design.
Critical weaknesses
  • The main concern with this study is the model setting. The study considers a teacher-student model, specifically assuming that the parameters to be estimated in post-training are truly low-rank, and that the learner knows which subspace to learn. However, the fundamental question regarding practical LoRA seems to lie one step prior to this. That is, why LLMs pre-trained on various tasks are able to generalize to downstream tasks (whose relation to those tasks is non-trivial) through only low-dimensional updates, or under what conditions they fail to do so. From that perspective, the model seems to fail to capture the essential aspects of LoRA. Furthermore, while this study considers the case where the inputs for pre-training and the LoRA task match as one of the scenarios, it was unclear whether investigating this is practically useful.
  • I feel there is a lack of experiments. In this work, a significant model simplification has been made where the key and query are the same matrix and the value is not learned, but the justification for this is weak. For example, I would like to check whether qualitatively similar trends can be obtained even when training with full parameters, and whether the heuristics remain effective.
Questions for authors

It is not clear how the results differ from existing transfer learning theory, except for the use of an attention model. For example, the setting where parameters learned in an upstream task are frozen and only a small number of parameters are trained in the downstream task has also been studied in transfer learning for high-dimensional linear regression[1]. Are the phenomena demonstrated here, such as the effect of effective noise and the divergence between parameter reconstruction accuracy and generalization performance caused by input data reuse, specific to Attention or LoRA? Or do they also appear in general two-stage transfer learning?

[1] Dar, Y., & Baraniuk, R. G. (2022). Double double descent: On generalization errors in transfer learning between linear regression tasks.

4 comments

We thank the reviewer for the comments and suggestions. We would like to address the weaknesses and questions below.

W1, Q1 : We agree that understanding why natural downstream tasks admit low-rank adaptation is an important question. Our paper addresses a complementary one: given a task-specific low-rank change, how does imperfect pre-training affect the amount of data needed to learn it, its prediction error, and recovery of the underlying task direction?

Importantly, in our setting, the learner is not given the downstream subspace. It optimizes over all w∈RDw\in\mathbb R^D and must discover the unknown direction; in the untied extension, both factor subspaces are learned. The shared extensive-rank teacher represents transferable structure, while the additional spike represents the task-specific change. The two stages have different statistical scales, N=Θ(D2)N=\Theta(D^2) and N′=Θ(D)N'=\Theta(D), yet both components contribute at order one to the attention scores. The construction therefore isolates how residual uncertainty in a frozen representation interferes with learning new structure from limited data, rather than assuming that the relevant direction is already known. Compared to [1], the key difference of our analysis relies in the different scales between pre-training and fine-tuning stages. This is specific to quadratic attention and LoRA. This allows the decoupling of the two learning problems in high-dimensions, and the subsequent effective noise mechanism and mismatch between parameter reconstruction accuracy and generalization performance with input reuse.

W2 : Our theory and our heuristics are valid in broader regimes. In the revised article we added several extensions.

Following the reviewer's suggestion, we added an experiment with distinct, jointly learned query, key and value matrices, as described in the new Appendix "Extension to complete self-attention". It shows that our effective-noise description holds in this broader setting. We compare a fully pre-trained attention to its effective low-dimensional Gaussian equivalent, that we derive in the Appendix, on both the test error and the test error of a subsequent fine-tuning. The errors of the two models are equal up to the numerical errors. This supports our effective-noise reduction holds for a general attention, beyond tied queries/keys and fixed values.

Following reviewer voya's suggestion, we show that our theory can straightforwardly be applied to standard LoRA: in the revised Appendix A8 "Extension to a generic untied low-rank update" we consider an attention with a general low-rank update uw⊤u w^\top (not necessarily rank one), as well as fine-tuning data with non-symmetric spikes u⋆(w⋆)⊤u^\star (w^\star)^\top. We extend all our theoretical results of part 3 to this broader setting. The effect of the pre-training is caught by the same effective noise mechanism.

Last, we performed an experiment on non-Gaussian data from cifar10. It is described in the revised Appendix "Supplementary figures", paragraph "Numerical extension to non-Gaussian data". We are able to qualitatively reproduce the phenomenology we observe when reusing sequences Fig. 3, i.e. to observe a mismatch between test error and reconstruction. This suggests that our theory and interpretation can extend to more realistic settings.

Thank you for the careful response and the additional experiments. The new experiments are meaningful and support the main claims of the paper.

However, I still have concerns about the practical motivation for the input-reuse setting. In my original comment, I asked how studying this setting helps us understand LoRA in practice. I do not think the response fully answers this question.

I think this issue is important because input reuse is behind some of the most interesting results in the paper. In particular, Section 4.2 show a non-trivial phenomenon. Overfitting in pretraining improves parameter recovery on the downstream task, but it makes the test error worse. This gap between test error and parameter recovery is one of the main contributions the paper highlights. But this result depends on the special input reuse setting, and it is not clear what real LoRA situation this setting corresponds to.

Of course, it is natural for theoretical work to simplify real training. I do not think the theoretical model has to match real LLM settings exactly. But for such a simplification to be meaningful, the model should capture the key features of the real problem. At least, the paper should give a convincing reason why studying these features matters.

For example, there may be prior practical work showing that input reuse is hard to avoid in real LoRA, and that it causes problems for training or evaluation. If so, that would strongly support this setting. Practical work showing that the gap between parameter recovery and test error is a real problem would also be good motivation.

For now, I will keep my score. If the authors give a convincing answer on this point, I will raise my score.

Thank you for clarifying the remaining concern, and for recognizing the value of the additional experiments. We would like to further clarify the practical motivation for studying input reuse and its relevance to multi-stage learning and LoRA fine-tuning.

A concrete motivation is task-adaptive pretraining (TAPT). Gururangan et al. [1] continues language-model pretraining on the downstream task's unlabeled training texts before supervised task learning. Thus, the same underlying inputs deliberately participate in two stages with different objectives. LoRA is also used in such workflows: Gu et al. [2] introduce LoRA-based TAPT followed by task classification. These methods are not identical to our model, but demonstrate that recurring task inputs across representation learning and adaptation is a practical design choice, not merely an artificial possibility. Our e=1e=1 setting isolates this form of dependence: downstream training inputs have already been encountered during pretraining, whereas evaluation uses unseen inputs. Importantly, multi-stage training does not always improve downstream performance. Shi and Lipani [3] show that conventional TAPT can even be detrimental to subsequent fine-tuning in some settings.

Regarding the gap between parameter recovery and test error, related empirical studies highlight a broader distinction between training objectives and downstream quality. Liu et al. [4] find that language models with comparable pretraining losses can exhibit different downstream performance. They show that to continue pre-training after convergence makes the loss landscape flatter and is beneficial for downstream tasks, which is similar to our overfitting case, with our inverse-curvature order parameter VV going to infinity. Isik et al. [5] identify machine-translation settings where downstream cross-entropy improves while translation-quality metrics worsen.

We believe our theory provides a possible mechanistic explanation for this broader class of discrepancies: the quality of the learned task-specific component and the test error of the complete predictor need not improve together, because the latter also depends on residual errors inherited from pretraining. Section 4.2 characterizes this separation in a controlled setting with recurring training inputs, where improved recovery of the downstream signal can coexist with worse predictions on unseen data.

We do not claim that the cited empirical findings necessarily arise from our specific mechanism, or that all practical LoRA pipelines exhibit this inversion. Rather, these studies motivate understanding why different measures of learning and downstream performance can disagree. We will clarify this connection and the scope of our model in the paper, including the distinction between input reuse and the additional shared-noise assumption addressed by our independent-noise analysis.

[1] Gururangan et al., Don't Stop Pretraining: Adapt Language Models to Domains and Tasks, ACL 2020.

[2] Gu et al, NLoPT: N-gram Enhanced Low-Rank Task Adaptive Pre-training for Efficient Language Model Adaption, LREC-COLING 2024.

[3] Shi and Lipani., Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner, NeurIPS 2023.

[4] Liu et al., Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models., ICML 2023.

[5] Isik et al., Scaling Laws for Downstream Task Performance in Machine Translation, ICLR 2025.

Thank you for your response. I think these are very important points, and I have raised my score on the premise that they will be reflected in the paper.

Reviewer_voyaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper develops a high-dimensional theory of LoRA fine-tuning in a simplified attention model. Using replica/AMP methods, it characterizes test error and representation recovery after pre-training, considering both independent and reused fine-tuning samples. The analysis identifies an effective-noise mechanism linking pre-training quality to downstream performance and explores weight recalibration and active fine-tuning. The theoretical predictions are compared with numerical simulations on synthetic Gaussian data.

Critical strengths

The paper provides a quantitative characterization of both stages within a unified high-dimensional model, including test error and representation recovery. It also considers both independent and reused fine-tuning samples. This is a meaningful theoretical contribution, since the interaction between pre-training and downstream adaptation is still not well understood.

Critical weaknesses
  1. The paper uses a symmetric PSD rank-one update ww⊤ww^\top, while standard LoRA uses a general low-rank update BABA. Together with tied queries and keys, single-head attention, and identity values, this makes the setting quite different from actual transformer LoRA. More importantly, all experiments use synthetic Gaussian data, with no real models or downstream tasks. It is therefore unclear whether the main findings reflect general properties of LoRA fine-tuning or are specific to this constrained model. This substantially limits the broader significance of the results.

  2. Appendix A acknowledges that a rigorous proof is missing, including the multi-token Gaussian equivalence and important steps in coupling pre-training with fine-tuning. While non-rigorous replica/AMP predictions can still be valuable, the paper relies heavily on numerical agreement to support them. Unfortunately, the supplementary code contains clear issues: SE_pre-training.py parses fractional alpha values as integers, the active-learning branch calls an undefined _softmax function, and the settings in Eq. (140) do not match Figure 6. I also could not find a complete Figure 6 reproduction script. These problems make it difficult to independently verify some of the paper's key results and weaken confidence in the numerical evidence.

Questions for authors
  1. How much of the theory is expected to apply to standard LoRA? In particular, would the main conclusions still hold for a general rank-one update ba⊤ba^\top, rather than the symmetric PSD update ww⊤ww^\top? Is there any empirical evidence, even from a small transformer or a non-Gaussian dataset, that the effective-noise mechanism carries over to more realistic settings? This would significantly affect my assessment of the paper's relevance.

  2. Which parts of the theoretical characterization are rigorously established, and which remain heuristic predictions? I am particularly concerned about the multi-token Gaussian equivalence and the decoupling argument when fine-tuning reuses pre-training samples. Could the authors clarify what justifies these steps beyond numerical agreement and replicon stability checks?

2 comments

We thank the reviewer for the comments and suggestions. We would like to address the weaknesses and questions below.

W1, Q1 : Our theory can straightforwardly be applied to standard LoRA: in the revised Appendix A8 "Extension to a generic untied low-rank update" we consider an attention with a general low-rank update uw⊤u w^\top (not necessarily rank one), as well as fine-tuning data with non-symmetric spikes u⋆(w⋆)⊤u^\star (w^\star)^\top. We extend all our theoretical results of part 3 to this broader setting. The decoupling still holds and importantly the effect of the pre-training is caught by the same effective noise. This is possible because of the different scalings between pre-training and fine-tuning. Consequently we expect the phenomenology we describe in part 4, on the effect of the pre-training on the fine-tuning, to be the same. New phenomena specifically related to the fine-tuning may appear; but we guess they would be very related to those already studied in the literature on narrow neural networks, committee machines, multi-indices retrieval or mismatched teacher-students; and they would not be specific to our two-stage setting.

We also extend the setting to non-Gaussian data, as suggested by the reviewer, by conducting an experiment on data from cifar10. It is described in the revised Appendix "Supplementary figures", paragraph "Numerical extension to non-Gaussian data". We are able to qualitatively reproduce the phenomenology we observe when reusing sequences Fig. 3, i.e. to observe a mismatch between test error and reconstruction. This suggests that our theory and interpretation can extend to more realistic settings.

Last, following reviewer vwR4's suggestion, we extended the setting to full key, query and value pre-training, as described in the Appendix "Extension to complete self-attention" of the revised article. It shows that our effective-noise description holds in this broader setting. We compare a fully pre-trained attention to its effective low-dimensional Gaussian equivalent, that we derive in the Appendix, on both the test error and the test error of a subsequent fine-tuning. The errors of the two models are equal up to the numerical errors.

Overall, these different extensions strongly enlarge the significance of our results.

W2 : We thank the reviewer for identifying these issues. The fractional-alpha parsing and undefined-helper errors were minor errors in the supplementary script versions we inadvertently uploaded, not in the versions used to generate the reported experiments. They therefore did not affect the numerical results in the article. We apologize for this packaging mistake and have corrected and execution-tested the supplied scripts.

We have also corrected the numerical-protocol description and included the original theory outputs, seed-level simulation data, configurations, generation scripts and plotting pipeline for original Figure 6 (Figure 7 in the revision). Recomputing its means and sample standard deviations from the original records reproduces the archived aggregates to numerical precision. The revised supplement also includes the CIFAR-10 implementation and the complete Q/K/V measured-overlap experiment. The theoretical equations and archived results are unchanged.

Finally, the predictions are analytically derived from established high-dimensional attention and sequence multi-index frameworks; they are not fitted to simulations. Numerical agreement is an independent consistency check, not their sole justification. We clarify the mathematical status of the individual steps in our answer to Q2.

Q2, W2 : We would like to mention firstly that our theoretical characterization relies on well-established techniques of high-dimensional inference; and that, while a rigorous proof for the overall characterization is missing, we are confident in the exactness of our results. The numerical agreement provides an independent consistency check of these predictions, not their main justification.

The reduction to a matrix-estimation problem (Appendix A1) is exact. The different predictions based on replica and AMP for non-convex losses have been made rigorous in Vilucchio et al. (2025), for a setting slightly different from ours.

For the multi-token Gaussian equivalence, required for the pre-training stage: the case T=1T=1 has rigorously been proven in Xu et al. (2025a). The generalization to multiple tokens requires promoting scalar outputs to finite T-dimensional vectorial ones in the proof and we do not foresee any technical roadblock here. The main requirement is that TT remains finite compared to DD. The same multi-token Gaussian equivalence is already used in Boncoraglio et al. (2025, 2026).

On the decoupling argument when fine-tuning reuses pre-training samples: the argument is exact, and holds for both new and reused sequences. The main point is that the low-rank update w^\hat w cannot correlate with the extensive-rank weights S⋆=P0−1/2W⋆W⋆⊤S^\star=P_0^{-1/2}W^\star W^{\star \top} and S^=P−1/2W^W^⊤\hat S=P^{-1/2}\hat W\hat W^\top, even if S^\hat S and w^\hat w are correlated with the same reused sequences; because S^\hat{S} and S⋆S^\star do not have any particular outlying eigenvector that w^\hat w can fit. This is explained in Appendix A4, where we show that any overlap between S^\hat S and w^\hat w vanishes. To make the point clearer we reformulate it in the revised article. The argument also extends to the general low-rank update uw⊤u w^\top that the reviewer suggested and to a full key-query-value pre-training.

Reviewer_aae7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies rank-one LoRA adaptation after pre-training in a solvable, single-head attention model. It derives high-dimensional predictions for test error and signal recovery, considering both fresh fine-tuning inputs and inputs reused from pre-training. The analysis identifies an effective-noise description for linear attention and uses it to study pre-training, recalibration, and active sample selection.

Critical strengths

The two-stage setting addresses a meaningful gap. The paper models a large-sample pre-training stage and a limited-sample adaptation stage within one framework. This makes it possible to examine how the quality of a frozen representation affects subsequent low-rank learning. The distinction between prediction and recovery is useful. The reused-input analysis shows a regime in which stronger recovery of the downstream signal does not yield lower test error. Reporting both quantities makes the interpretation more informative than a learning curve based only on prediction error. The paper connects its predictions to observable behavior. It examines regularization, effective noise, recalibration, and active selection, and compares the analytical predictions with finite-dimensional simulations. The revised appendix also makes the replica derivation and its assumptions easier to inspect.

Critical weaknesses

There exist the following concerns. (1) The authors might need to describe the theoretical contribution. Boncoraglio et al. [1] characterize the extensive-rank attention pre-training stage, while Cui [2] develops the sequence multi-index framework used for the low-rank stage. The paper's contribution appears to lie in coupling these stages, particularly when inputs are reused, rather than in either stage's equations individually. Could the authors show which predictions cannot be obtained by applying the existing fine-tuning analysis with pre-training order parameters supplied as fixed side information?

(2) Input reuse and noise reuse should be separated. In the paper's e=1 construction, the fine-tuning task reuses both the input sequence and its noise realization from pre-training. These are distinct sources of dependence, and matching the noise could contribute to the strong recovery effect. What happens if the input is reused but the fine-tuning noise is drawn independently? A controlled comparison, even only by simulation, would clarify how much of the test-error/recovery mismatch follows from input memorization itself.

Questions for authors

Could the authors show which predictions cannot be obtained by applying the existing fine-tuning analysis with pre-training order parameters supplied as fixed side information? What happens if the input is reused but the fine-tuning noise is drawn independently?

1 comment

We thank the reviewer for the comments and suggestions. We would like to address the weaknesses and questions below.

W1, Q1 : Our joint characterization is not just obtained by directly composing the previous theories in Boncoraglio et al. [1] and Cui et al. [2]. It requires proving the asymptotic decoupling between the extensive-rank quadratic fields and the rank-one fields (Appendix A4); deriving the fine-tuning channel, conditioned on the learned pre-training representation, distinguishing fresh-data and reused-data channels (Appendix A5, with full derivation in A5.1. [2] does not consider such additional channels or effective noise); stating the corresponding fine-tuning stability condition (Appendix A6. [2] does not provide it); the joint Bayes benchmark across the two sample-complexity scales (Appendix B); and establishing the resulting effective-noise, recalibration, reuse-induced mismatch, and active-selection mechanisms. Last, following reviewer vwR4's suggestion, we extended the setting to full key, query and value pre-training, as described in the Appendix "Extension to complete self-attention" of the revised article. Reference [1] only considers tied attention and identity values.

"What happens if the input is reused but the fine-tuning noise is drawn independently?" : please see the answer to W2.

W2 : The noise ξ\xi is thought as the part of the data XX that is not modeled by the attention scores of the target XW⋆W⋆⊤X⊤XW^\star W^{\star \top}X^\top ; this is why for reused sequences e=1e=1 we match the randomness of ξ\xi. We make this point more explicit in the revision, paragraph "Comments and limitations".

In the case when the sequences are reused but the noise ξ\xi is independent between pre-training and fine-tuning, our theory can still exactly characterize the resulting effective noise. Eq. 25 becomes Δeff(e=1)=c2(Q0−2M+Q)+Δ2+(1−c)2Δ2,c=(1+4VT−2)−1<1\Delta_\mathrm{eff}(e=1) = c^2(Q_0-2M+Q) + \frac{\Delta}{2}+(1-c)^2\frac{\Delta}{2},\quad c=(1+4VT^{-2})^{-1}< 1 The noise Δ/2\Delta/2 coming from ξ\xi is no longer reduced since WW cannot memorize it. Still, when Q0−2M+Q>Δ/2Q_0-2M+Q>\Delta/2, the overall effective noise is reduced, by memorization of the XμW⋆W⋆⊤Xμ⊤X_\mu W^\star W^{\star \top}X_\mu^\top: at small cc, Δeff(e=1)=Δ<Δeff(e=0)=Q0−2M+Q+Δ/2\Delta_\mathrm{eff}(e=1)=\Delta < \Delta_\mathrm{eff}(e=0) = Q_0-2M+Q + \Delta/2 and the test-error/recovery mismatch can hold. Overall, the change of the phenomenology would be minimal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.