acceptodds
Under review as a conference paper at ICLR 2027

LoLoRA: Locally Fine-Tuned Low Rank Adapters

Abstract

Low-Rank Adaptation (LoRA) is a memory-efficient fine-tuning method for large language models (LLMs) that approximates weight updates as , where , and . To maximize memory savings, one can freeze matrix , avoiding the storage of its input activations, but this often degrades performance. In this work, we mitigate this trade-off by introducing gradient-free updates to matrix during the forward pass. Our method computes these updates based on the layer’s immediate input, allowing it to adapt to input distribution shifts without storing activations for the backward pass. Experiments show that our approach achieves fine-tuning accuracy comparable to standard LoRA, while reducing peak memory consumption by up to 20% on LlaMA3-8B.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
efficient fine-tuning, language models, localized learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tYzpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  • The authors point out a limitation of LoRA: while it saves memory by approximating weight updates using low-rank matrices, it still requires storing the activations of the linear layers—to which LoRA is applied—for backpropagation, meaning that activation memory usage does not decrease significantly. To address this, the "LoRA-FA" method was proposed, which keeps matrix A fixed; however, the authors argue that this often leads to a performance trade-off, as A effectively becomes a static, random low-rank "feature extractor."
  • This paper aims to reduce activation memory while maintaining performance. The core idea is to update matrix A locally (during the forward pass) and matrix B globally (during the backpropagation phase). By doing so, there is no need to store input activations for the update of A, which reduces memory consumption and mitigates the performance degradation associated with using a fixed, random matrix A.
  • To demonstrate the superiority of their proposed method, the authors conducted experiments across various domains—including natural language understanding, mathematical reasoning, and multimodal tasks—comparing it against standard LoRA and LoRA-FA. They present both the performance and the computational efficiency of their approach for each task.
Critical strengths
  • This method builds upon the core concepts of LoRA while effectively identifying and addressing issues related to bottlenecks.
  • The training approach is both simple and effective.
  • Given the validity of the authors' initial assumptions, their conclusions are logical and compelling.
  • The paper presents experimental results and analyses that strongly support the authors' claims.
Critical weaknesses
  • The authors' proofs—including those in the appendix—rely on the assumption that Delta W_0 consists of i.i.d. Gaussian random variables. While this is a reasonable choice when no prior information is available, it creates a gap between theory and practice, as the Delta W values ​​encountered during the actual fine-tuning of LLMs often do not follow this distribution.
  • Consequently, the method proposed by the authors lacks theoretical persuasiveness. Their theoretical proofs do not account for the full complexity of LoRA fine-tuning in real-world scenarios; rather, they merely provide intuition regarding optimal subspaces and initialization within a simplified regression model.
  • The structure of the paper requires refinement to improve readability. For instance, having only a single subsection under a given section is poor organizational practice.
  • The paper fails to compare its findings with the latest research in the field from 2025 and 2026; such comparisons are necessary.
Questions for authors
  • Is it possible to provide a comparison—even a partial one—of the performance and efficiency of the efficient LoRA methods involving pruning that were introduced at ACL 2026?
  • If we assume that the Delta W (delta weight) in LLM fine-tuning follows a distribution other than an i.i.d. Gaussian random distribution—for instance, some other plausible distribution—can you provide any insights into the resulting performance and efficiency?
1 comment

Thank you for recognizing the simplicity of the training approach and the motivation from LoRA's activation-memory bottleneck.

W1 and Q2: Gaussian assumption and other target distributions

Response. We agree that Gaussianity was unnecessarily restrictive. The proof only used second moments, so the revised assumption requires the entries of ΔW0\Delta W_0 to have zero mean, a common finite variance, and pairwise zero correlation. The entries need not be Gaussian or mutually independent. Thus, the input-PCA result applies, for example, to any i.i.d. zero-mean, finite-variance entry distribution, including non-Gaussian distributions.

We also added a result for general finite target second moment G=E[ΔW0⊤ΔW0].G=\mathbb E[\Delta W_0^\top\Delta W_0].

For positive-definite input covariance Σzz\Sigma_{zz}, the minimizing rank-rr row space of AΣzz1/2A\Sigma_{zz}^{1/2} is a leading eigenspace of Σzz1/2GΣzz1/2.\Sigma_{zz}^{1/2}G\Sigma_{zz}^{1/2}.

Input PCA is recovered when GG is isotropic. Under anisotropic target uncertainty, the preferred subspace depends jointly on the input covariance and the target second moment and need not be the input-PCA subspace.

The activation-memory behavior of the current input-local update is unchanged by the assumed target distribution because the implementation still uses only the layer input during its local step. What changes is the theoretical claim: an input-only PCA rule is not generally the minimizing surrogate rule for anisotropic GG. Exploiting that target-weighted subspace would require additional target information or an estimator of GG, which we have not implemented or evaluated. The generalized result therefore gives a precise limitation, not a downstream-quality guarantee.

W2: Gap between the surrogate and real LLM fine-tuning

Response. We agree with the reviewer's characterization. The theory is an isolated, fixed-distribution population regression surrogate that provides intuition about adapter subspaces when task targets are unavailable. It does not capture the complete dynamics of multilayer LoRA fine-tuning: earlier adapters change the input distributions of later modules, the global loss couples all modules, and the practical local updates are mini-batch and Adam-scaled. We now state these limitations explicitly and no longer present the result as a convergence or optimality guarantee for full LLM fine-tuning.

W3: Paper organization

Response. We reorganized the Rationale into four peer subsections:

  1. From the Global Objective to a Local Surrogate;
  2. Optimal Adapter Subspaces in the Regression Surrogate;
  3. Interpretation and Scope;
  4. Local Autoencoder Alternative.

We also revised captions, table formatting, and experimental prose to improve readability.

W4 and Q1: Recent efficient-LoRA and pruning methods

Response. We added recent work and a conceptual comparison covering DPLoRA, HyperAdaLoRA, LoRAPrune, and LARS. These methods address different resource axes:

  • DPLoRA and HyperAdaLoRA primarily adapt or prune the adapter-rank allocation;
  • LoRAPrune uses LoRA information to guide structured pruning of base-model heads and channels;
  • LARS directly changes the adapter computation to reduce stored activation intermediates; and
  • LoLoRA retains a fixed rank in the reported experiments and targets retention of the full-rank adapter input.

Rank allocation and structured pruning may be complementary to LoLoRA, but a combination is not automatic: changing ranks changes the locally updated AA matrices, and pruning the base model changes the activation distributions that they track. We have not implemented or evaluated these combinations. We therefore provide a partial conceptual comparison of mechanisms, calibration requirements, and possible composability rather than numerical comparisons across incompatible models, tasks, hardware, and accounting conventions.

Reviewer_QhLROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes LoLoRA, a memory-efficient LoRA variant that updates the adapter matrix A using local PCA-based learning rules during the forward pass, while training matrix B through backpropagation. This avoids storing full input activations for updating A and allows it to adapt during fine-tuning. The paper provides a theoretical motivation based on a simplified regression model and evaluates the method on language understanding, mathematical instruction tuning, and multimodal fine-tuning. Experiments show reduced activation memory usage with validation losses comparable to standard LoRA in larger-model settings.

Critical strengths

The paper presents a simple way to update LoRA's A matrix without storing its input activations for backpropagation. The reported extra GPU memory is reduced by around 22–23% on the MetaMathQA and LLaVA experiments, while achieving comparable or better validation loss than standard LoRA. This is a useful result, as memory efficiency is an important concern in LLM fine-tuning. The idea of combining local updates with standard backpropagation is also interesting, even though the memory savings largely build on LoRA-FA.

Critical weaknesses
  1. Limited novelty and unclear benefits of online updates.

The main method combines LoRA-FA with existing PCA-based local learning rules such as Oja/SNL. EVA has already explored using the principal components of input activations for LoRA, so the main new idea seems to be updating A online rather than initializing it once. However, the experiments do not clearly show why these online updates are necessary. In Tables 5 and 6, LoLoRA and LoRA-FA with EVA initialization achieve almost identical perplexity across all ranks. At rank 8, the difference is only 0.001, much smaller than the reported standard deviations. The paper also does not compare against the full EVA method.

More importantly, LoLoRA performs worse than standard LoRA on all eight GLUE metrics, while the math and multimodal experiments only report loss or perplexity rather than actual downstream task accuracy. Overall, the experiments do not provide convincing evidence that online updates offer a meaningful advantage over simpler existing methods. This is my main concern, since it directly questions the need for the proposed method.

  1. The proof of Theorem 4.6 contains incorrect arguments.

The proof in Appendix A.3 appears to have two mathematical errors. First, the authors claim that if a selected subspace is not optimal, there must be an orthogonal direction with a higher Rayleigh quotient than every selected direction. This is not generally true. For example, with a covariance matrix diag(3, 2, 1), selecting the first and third eigenvectors gives a non-optimal subspace, but the only orthogonal direction has a Rayleigh quotient of 2, which is smaller than 3 for the first selected direction.

Second, the authors claim that a rotational perturbation gives p′(0)<0p'(0) < 0. However, at an eigenvector, the first derivative can be exactly zero, even when the direction is not locally optimal. A second-order argument is needed in such cases. These errors do not necessarily invalidate the theorem itself, but the current proof is not correct. Since the theoretical justification is presented as a key contribution, this issue significantly weakens the paper.

Questions for authors
  1. What is the actual benefit of continuously updating A compared with initializing it once using EVA? Tables 5 and 6 show almost identical results for LoLoRA and LoRA-FA (EVA). Could the authors provide stronger evidence that online updates improve downstream task performance or overall training efficiency? A comparison with the full EVA method would also help clarify the contribution.

  2. Could the authors revisit the proof of Theorem 4.6? In particular, how do the authors justify the existence of an orthogonal direction with a higher Rayleigh quotient than all selected directions? Also, how is the claim p′(0)<0p'(0) < 0 justified when the selected directions are eigenvectors, where the first derivative may be zero?

  3. How is the forward-pass update of A implemented without affecting gradient correctness? Algorithm 1 updates A immediately after computing u=Azu = Az, before the backward pass. However, computing the gradient with respect to the input zz requires the version of A used during the forward computation. Does the backward pass use A before or after the update, and how is this handled without storing additional activations? Releasing the implementation would help verify both the gradient computation and the reported memory savings.

3 comments

Before addressing the LoLoRA-specific comments, we respectfully note that the opening summary appears to describe a different submission. This paper proposes LoLoRA, not CARD-LoRA; it does not use suppression gates or pretrained spectral strength, and it does not report LLaMA-2-7B or Qwen2.5-7B experiments. We nevertheless address every LoLoRA-specific weakness and question below.

W1 and Q1: Benefit of online updates relative to EVA

Response. We agree that the current experiments do not establish a downstream-quality advantage for continuous updates over fixed-rank EVA initialization. In the TinyLlama ablations, the differences between LoLoRA and LoRA-FA with EVA initialization are smaller than the reported variation. LoLoRA also has lower mean task metrics than standard LoRA on all eight reported GLUE tasks, and the MetaMathQA and LLaVA experiments report validation loss or perplexity rather than downstream answer accuracy. We have therefore removed the stronger superiority framing.

The revised contribution is narrower and procedural. Starting from uniform initialization, LoLoRA estimates the activation subspace within the fine-tuning stream, avoiding a separate incremental-SVD initialization pass, and can update that subspace as preceding layers change. The existing reconstruction-error and projector-distance curves show that the local rules track the chosen instantaneous PCA target more closely in the measured layers. We now present these curves only as mechanistic evidence of subspace tracking, not as evidence that tracking improves downstream task metrics or overall training efficiency.

We also clarify that the evaluated "LoRA-FA (EVA init.)" baseline uses EVA's initialization component at a fixed rank in every layer. It does not implement EVA's cross-layer rank redistribution and is not a comparison with the full EVA method. Consequently, our supported empirical conclusion is a quality-memory trade-off relative to standard LoRA and randomly initialized LoRA-FA, together with a single-pass online alternative to fixed-rank PCA initialization. A full-EVA comparison and stronger evidence for a practical benefit of online tracking remain open experimental requirements.

We also agree that the activation-memory saving builds on the frozen-AA mechanism of LoRA-FA. The specific aim of LoLoRA is to update AA locally without restoring retention of its full-rank input, rather than to claim an unrelated activation-memory mechanism.

W2 and Q2: Errors in the proof of Theorem 4.6

Response. We agree with the counterexample and have replaced the flawed argument. The revised proof does not claim that an omitted direction has a higher Rayleigh quotient than every selected direction. After establishing that a stationary candidate is an invariant subspace, a non-dominant subspace must contain some selected eigenvector qjq_j and omit some eigenvector qiq_i with λi>λj\lambda_i>\lambda_j. Rotating that selected direction toward the omitted one yields

F(V(t))−F(V)=(λi−λj)sin⁡2(t)>0.F(V(t))-F(V)=(\lambda_i-\lambda_j)\sin^2(t)>0.

This is explicitly a second-order improvement: the derivative at zero is allowed to vanish. It rules out the stationary but non-dominant eigenspace as a local maximum and proves the corresponding claim for the autoencoder loss. We also corrected row/column orthogonality and trace-dimension issues and made the map from a nearby Stiefel path to a nearby adapter matrix explicit.

Q3: Which version of AA is used during backward propagation?

Response. The reported experiments use the post-update matrix during upstream feedback. We agree that the previous Algorithm 1 did not make this clear and that the resulting upstream derivative is not the exact derivative of the recorded forward computation.

Let A−A^- be the matrix used to compute u=A−zu=A^-z, and let A+=A−+ΔAA^+=A^-+\Delta A be its value after the immediate local step. The implementation retains uu, so for output gradient δ=∇hL\delta=\nabla_h\mathcal L, the gradient

∇BL=δu⊤\nabla_B\mathcal L=\delta u^\top

is the usual gradient for the recorded forward computation. The reported rule does not retain zz or a snapshot of A−A^-, however, so the adapter term in the upstream feedback uses (A+)⊤B⊤δ(A^+)^\top B^\top\delta rather than (A−)⊤B⊤δ(A^-)^\top B^\top\delta. Their discrepancy is

=(\Delta A)^\top B^\top\delta, \qquad \left\|\widetilde{\nabla}_z\mathcal L-\nabla_z\mathcal L\right\| \leq \|\Delta A\|_2\|B\|_2\|\delta\|.$$ The discrepancy is zero at the standard initialization $B=0$, but we do not claim that it remains negligible during training. The revised method section and algorithm now identify this as an updated-weight feedback approximation and state exactly which tensors are retained: the low-rank feature $u$, but neither the full-rank input $z$ nor a pre-update copy of $A$.

Presentation

Response. We reorganized the Rationale into four peer subsections, made the experimental captions self-contained, standardized the result tables, corrected spelling and numerical descriptions, and replaced ambiguous uses of "performance" with the actual quantity being discussed (validation loss, perplexity, or a named GLUE metric).

Thank you for the rebuttal and revisions. I appreciate the authors' efforts to correct the theoretical proof and clarify the implementation. However, my main concern about the necessity of online updates remains. The authors acknowledge that LoLoRA does not demonstrate a reliable performance advantage over one-time EVA initialization. For example, at rank 8, LoLoRA achieves a perplexity of 2.535 compared with 2.536 for LoRA-FA with EVA initialization, a difference much smaller than the reported standard deviations. Since the main novelty lies in continuously updating A rather than initializing it once, I would have liked to see stronger evidence that this additional mechanism provides a meaningful benefit.

The experimental results also leave this question unresolved. LoLoRA has lower mean scores than standard LoRA on all eight GLUE tasks, while the mathematical reasoning and multimodal experiments report validation loss or perplexity rather than actual downstream task accuracy. The reported memory savings are useful, but they largely build on the existing LoRA-FA mechanism. The rebuttal now emphasizes avoiding a separate SVD initialization pass and tracking changes in activation subspaces. However, the authors acknowledge that neither improved downstream performance nor better overall training efficiency has been established. These limitations make it difficult to assess the practical significance of the proposed online updates.

I appreciate the corrections to the proof and the more precise description of the theoretical assumptions. Nevertheless, the revised theory is restricted to a simplified regression setting and does not establish the behavior of the implemented training procedure. The authors have addressed several issues by narrowing the original claims, but the main question of what LoLoRA contributes beyond simpler existing approaches remains open. I believe a more direct demonstration of the practical value of online updates would be important for supporting the paper's central contribution.

Reviewer_mG4GOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper aims to reduce LoRA activation memory without permanently freezing the adapter input projection. Motivated by an optimal-subspace result under random regression assumptions, LoLoRA updates this projection using local PCA-style learning while training the output projection through backpropagation. Validation covers language understanding, mathematical instruction tuning, multimodal fine-tuning, etc.

Critical strengths
  • The experiment demonstrates a useful quality–memory trade-off. In Table 4, LoLoRA reduces extra memory from 52 GB to 40 GB while obtaining a validation loss of 1.1554, compared with 1.1509 for LoRA and 1.2587 for randomly initialized LoRA-FA.

  • The method is simple and intuitive: it combines familiar local PCA updates with standard LoRA training, avoiding retention of A's input activations for its backward pass without permanently freezing A.

Critical weaknesses
  • The theoretical justification has gaps, and the applicability is limited. Appendix A.3 incorrectly claims p'(0) < 0: with Sigma = diag(2,1), r = 1, A = e_2^T, and v = e_1, its path gives p(t) = 2 - sin^2(t), hence p'(0) = 0. A second-order argument is needed. More substantially, a characterization of local minima does not establish convergence of the implemented Adam-scaled updates under changing input distributions. The assertion that global SGD/Adam updates ensure optimal B (lines 264-266) is also unsupported.

  • The benchmarks and models in experiments are relatively old and insufficient to establish practical competitiveness. Evaluation uses RoBERTa/GLUE, TinyLlama/Alpaca, LLaMA-3.1, and LLaVA-v1.5, without newer model families such as Qwen3.

Questions for authors

NA.

1 comment

Thank you for recognizing the useful quality-memory trade-off and the simplicity of combining local PCA-style updates with LoRA training.

W1: Proof gap and unsupported optimization claims

Response. We agree with both parts of this criticism. The previous proof incorrectly relied on a first-order derivative that can be zero at a non-optimal eigenspace. It also used an unnecessarily strong comparison between an omitted direction and every selected direction.

The revised proof first uses the constrained stationarity condition to show that a candidate local optimum spans an invariant subspace of the input covariance. If that subspace is not dominant, it contains a selected eigenvector qjq_j, while its orthogonal complement contains an omitted eigenvector qiq_i with λi>λj\lambda_i>\lambda_j. Rotating only qjq_j toward qiq_i gives

F(V(t))−F(V)=(λi−λj)sin⁡2(t)>0F(V(t))-F(V)=(\lambda_i-\lambda_j)\sin^2(t)>0

for every sufficiently small nonzero tt. Thus, the first derivative may indeed vanish at t=0t=0, but the second-order change rules out a non-dominant stationary eigensubspace as a local maximum. This argument only requires the omitted eigenvalue to exceed the eigenvalue of one selected direction, and it also handles repeated eigenvalues.

We also removed the statement that global SGD or Adam updates "ensure" an optimal BB. The revised results characterize nested population optima after exact minimization over one adapter matrix in an isolated linear regression surrogate. They do not prove convergence of the implemented mini-batch, Adam-scaled updates, and they do not model the coupled, non-stationary dynamics of multilayer Transformer fine-tuning. The abstract, Rationale, theorem interpretations, and conclusion now state this scope explicitly.

W2: Older models and practical competitiveness

Response. We agree that the evaluated models do not establish competitiveness on current model families. We have added an explicit limitation stating that the experiments do not include Qwen3 or other model families introduced in 2025--2026 and therefore do not support a state-of-the-art claim for current architectures.

We also added a conceptual comparison with DPLoRA, HyperAdaLoRA, LoRAPrune, and LARS. The comparison separates four different resource axes: retention of adapter activations, adapter-rank or parameter allocation, structured pruning of the base model, and additional calibration or selection stages. We do not make cross-paper numerical rankings because the models, tasks, hardware, and memory-accounting conventions are incompatible. Evaluation on newer model families remains necessary future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank the reviewers for their careful reading and constructive comments. We have revised the manuscript substantially. The main changes are:

  1. We corrected the proof concerning the autoencoder objective using a stationary-subspace argument and a second-order rotation.
  2. We restricted the theoretical claims to population optima of an isolated, fixed-distribution regression surrogate and removed the unsupported claim that SGD or Adam ensures an optimal adapter matrix.
  3. We documented the precise forward/backward rule used in the reported experiments, including its updated-weight feedback approximation and an explicit error bound.
  4. We reframed the contribution relative to EVA: the current evidence supports a single-pass online alternative that tracks activation-subspace drift, but not a statistically established downstream advantage over fixed-rank EVA initialization.
  5. We generalized the random-regression result beyond Gaussian targets and added a target-anisotropic result.
  6. We added a conceptual comparison with recent efficient-LoRA, pruning, and activation-memory methods; clarified the experimental scope; reconciled the memory accounting; and completed a presentation and reference audit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.