- Limited novelty and unclear benefits of online updates.
The main method combines LoRA-FA with existing PCA-based local learning rules such as Oja/SNL. EVA has already explored using the principal components of input activations for LoRA, so the main new idea seems to be updating A online rather than initializing it once. However, the experiments do not clearly show why these online updates are necessary. In Tables 5 and 6, LoLoRA and LoRA-FA with EVA initialization achieve almost identical perplexity across all ranks. At rank 8, the difference is only 0.001, much smaller than the reported standard deviations. The paper also does not compare against the full EVA method.
More importantly, LoLoRA performs worse than standard LoRA on all eight GLUE metrics, while the math and multimodal experiments only report loss or perplexity rather than actual downstream task accuracy. Overall, the experiments do not provide convincing evidence that online updates offer a meaningful advantage over simpler existing methods. This is my main concern, since it directly questions the need for the proposed method.
- The proof of Theorem 4.6 contains incorrect arguments.
The proof in Appendix A.3 appears to have two mathematical errors. First, the authors claim that if a selected subspace is not optimal, there must be an orthogonal direction with a higher Rayleigh quotient than every selected direction. This is not generally true. For example, with a covariance matrix diag(3, 2, 1), selecting the first and third eigenvectors gives a non-optimal subspace, but the only orthogonal direction has a Rayleigh quotient of 2, which is smaller than 3 for the first selected direction.
Second, the authors claim that a rotational perturbation gives p′(0)<0. However, at an eigenvector, the first derivative can be exactly zero, even when the direction is not locally optimal. A second-order argument is needed in such cases. These errors do not necessarily invalidate the theorem itself, but the current proof is not correct. Since the theoretical justification is presented as a key contribution, this issue significantly weakens the paper.
Thank you for recognizing the simplicity of the training approach and the motivation from LoRA's activation-memory bottleneck.
W1 and Q2: Gaussian assumption and other target distributions
Response. We agree that Gaussianity was unnecessarily restrictive. The proof only used second moments, so the revised assumption requires the entries of ΔW0 to have zero mean, a common finite variance, and pairwise zero correlation. The entries need not be Gaussian or mutually independent. Thus, the input-PCA result applies, for example, to any i.i.d. zero-mean, finite-variance entry distribution, including non-Gaussian distributions.
We also added a result for general finite target second moment G=E[ΔW0⊤ΔW0].
For positive-definite input covariance Σzz, the minimizing rank-r row space of AΣzz1/2 is a leading eigenspace of Σzz1/2GΣzz1/2.
Input PCA is recovered when G is isotropic. Under anisotropic target uncertainty, the preferred subspace depends jointly on the input covariance and the target second moment and need not be the input-PCA subspace.
The activation-memory behavior of the current input-local update is unchanged by the assumed target distribution because the implementation still uses only the layer input during its local step. What changes is the theoretical claim: an input-only PCA rule is not generally the minimizing surrogate rule for anisotropic G. Exploiting that target-weighted subspace would require additional target information or an estimator of G, which we have not implemented or evaluated. The generalized result therefore gives a precise limitation, not a downstream-quality guarantee.
W2: Gap between the surrogate and real LLM fine-tuning
Response. We agree with the reviewer's characterization. The theory is an isolated, fixed-distribution population regression surrogate that provides intuition about adapter subspaces when task targets are unavailable. It does not capture the complete dynamics of multilayer LoRA fine-tuning: earlier adapters change the input distributions of later modules, the global loss couples all modules, and the practical local updates are mini-batch and Adam-scaled. We now state these limitations explicitly and no longer present the result as a convergence or optimality guarantee for full LLM fine-tuning.
W3: Paper organization
Response. We reorganized the Rationale into four peer subsections:
We also revised captions, table formatting, and experimental prose to improve readability.
W4 and Q1: Recent efficient-LoRA and pruning methods
Response. We added recent work and a conceptual comparison covering DPLoRA, HyperAdaLoRA, LoRAPrune, and LARS. These methods address different resource axes:
Rank allocation and structured pruning may be complementary to LoLoRA, but a combination is not automatic: changing ranks changes the locally updated A matrices, and pruning the base model changes the activation distributions that they track. We have not implemented or evaluated these combinations. We therefore provide a partial conceptual comparison of mechanisms, calibration requirements, and possible composability rather than numerical comparisons across incompatible models, tasks, hardware, and accounting conventions.