acceptodds
Under review as a conference paper at ICLR 2027

On the Convergence of Low-Precision LoRA Training

Abstract

Low-rank adaptation (LoRA) has become a standard approach for parameter-efficient fine-tuning, substantially reducing the number of trainable parameters and the optimizer memory required during training. Quantization offers a further route to resource-efficient fine-tuning, but many existing quantized LoRA methods focus on weight-only or deployment-oriented compression, leaving unclear how low precision affects optimization when it is used throughout training. In this work, we initiate a systematic theoretical study of low-precision LoRA training in which the data, pretrained weights, adapters, gradients, and intermediate matrix multiplications are quantized during optimization. For spectrally initialized LoRA under a low-rank linear model with sub-Gaussian data and relative quantization error, we establish finite-horizon convergence upper bounds that consist of a full-precision linear convergence term plus a quantization-dependent additive error floor. Our results show that low-precision LoRA can retain convergence behavior close to its full-precision counterpart, provided that the mantissa length grows only logarithmically with the inverse of the full-precision contraction rate.

Primary area:
learning theory
Keywords:
LoRA, low-precision training

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_t7veOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies gradient descent for low-precision LoRA in a noiseless low-rank linear regression model. Building on spectral initialization, it derives finite-horizon recovery-error bounds comprising a geometric term and quantization-dependent terms. It also argues that overparameterized LoRA requires tighter precision control.

Critical strengths

Analyzing quantization throughout the LoRA training computation is a relevant question. The paper gives explicit bounds under a clearly stated quantization model and distinguishes exact-rank and overparameterized cases.

Critical weaknesses
  1. Insufficient positioning against prior work. Although the paper cites Mu and Klabjan, it does not clearly distinguish their general stationary-point guarantee from its more restrictive recovery guarantee. Given the similar titles and shared focus on LoRA convergence, the authors should explain why their low-precision analysis requires a noiseless linear model and spectral initialization, and what is gained under these assumptions.

  2. Limited theoretical novelty. Spectral initialization and geometric convergence are established in LoRA-One. The new analysis bounds quantization perturbations around that trajectory, but the paper does not clearly identify the key technical step beyond a standard perturbation argument. Its finite-horizon guarantees also require admissible quantization error to shrink as the horizon grows; claims about an error floor or required precision should be qualified.

  3. Unclear over-rank mechanism. In the noiseless model, X⊤XΔ/NX^\top X\Delta/N has rank at most r∗r^ *. For r>r∗r>r^ *, the extra adapter directions appear to start at zero. Since the relative-error assumption implies Q(0)=0Q(0)=0, the stated updates appear to keep them at zero. How can these directions accumulate the error claimed in Theorem 4.3, and what explains the rank-dependent differences in Figure 2?

  4. Limited empirical support. Section 4.3 evaluates only synthetic noiseless regression with d=k=64d=k=64. Experiments using a specified practical low-precision format, with recovery quality and training cost or memory measurements, would help establish the relevance of the theoretical regime.

  5. Presentation needs substantial improvement. For a primarily theoretical paper, the main insight is difficult to identify within the lengthy proof. The authors should provide a clearer proof roadmap, identify which lemmas are new relative to LoRA-One, and summarize how the assumptions and precision conditions affect the scope of the results. Unresolved cross-references in the theorem statements should also be corrected.

Questions for authors

See weaknesses. I encourage the authors to substantially improve the writing, particularly the explanation of the main technical contribution, the proof structure, and the implications of the assumptions.

Reviewer_8pXpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the convergence of LoRA fine-tuning when low-precision arithmetic is used throughout training, rather than only for storing pretrained weights. Unlike methods such as QLoRA, which typically quantize the frozen backbone but keep adapters and much of the training computation in higher precision, this work models quantization of the inputs, pretrained weights, LoRA factors, output residuals, and intermediate matrix products in both forward and backward passes. The analysis considers an idealized noiseless multi-output linear regression problem with sub-Gaussian inputs, a low-rank target update, spectral initialization, and full-batch gradient descent. The main results show that, when the relative quantization error is sufficiently small, low-precision LoRA preserves the geometric convergence rate of full-precision LoRA up to an additional quantization-induced error floor.

Critical strengths

The paper’s main strength is that it studies low-precision computation throughout the LoRA training process rather than focusing only on quantized weights. It provides a clear theoretical result showing that low-precision LoRA can preserve the convergence rate of full-precision LoRA, with an additional quantization-dependent error floor. The paper also gives a useful distinction between exact-rank and over-parameterized LoRA, showing that extra rank directions may accumulate quantization errors over time. In addition, its quantization model covers data, weights, adapters, residuals, and intermediate matrix products, and the synthetic experiments support the main theoretical predictions.

Critical weaknesses

The experiments use only an abstract quantization-error parameter on a small synthetic linear problem, without evaluating concrete formats such as FP8, FP16, or BF16, real models and datasets, or efficiency metrics; therefore, they do not sufficiently demonstrate the practical effectiveness of the proposed analysis.

Questions for authors

None

Reviewer_j5tQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper derives convergence bounds for LoRA adaptation of a frozen pre-trained matrix to a linear regression task where all matrix multiplications use quantized matrices, including the inputs, frozen weights, adapters A,B, output gradients, and intermediate tensors. The authors assume there exists a target update Δ=W∗−W♮\Delta = W^* - W^{\natural} that satisfies rank(Δ)=r∗≤rrank(\Delta) = r^* \leq r; then the paper bounds ∣∣ATBT–Δ∣∣F||A_TB_T – \Delta||_F and studies how the bound depends on the quantization precision (mantissa length) and adapter rank (whether r=r∗r = r^* or r>r∗r> r^*).

Critical strengths

· The central question of the paper (Line 64) is valuable to the field of resource-efficient fine-tuning.

· The theoretical bounds and experiments highlight the value of matching the rank of the LoRA modules A,B to the rank of the ideal update Delta under a quantized training process, giving a practical takeaway of handling low-compression regimes with LoRA.

· The authors provide conditions under which full-precision convergence rate is asymptotically recovered, which can help practically determine when to apply these methods.

Critical weaknesses

· The experiments report recovery error and compare convergence rates with full precision but do not quantify the practical efficiency. Comparing the quantized training with full precision’s performance regarding runtime and memory would better highlight the practical benefit of the training method.

· The authors motivate the work through papers that focus on quantization in LLM fine-tuning. However, the theoretical analysis covers noiseless linear regression, which limits its implications to nonlinear LLM fine-tuning. Extending it to LoRA-One’s ReLU setting would better connect the theory to its motivation; see Equation (2), Section 4 of [1].

· The authors rely on spectral-initialized adapters from a full-precision gradient of the base model’s weights for each adapted layer. This is very expensive compared to LoRA, which conflicts with the resource-efficient setting that quantization focuses on.

· In Lines 256-261, the authors denote \epsilon_max. The authors note, in the abstract, that many existing quantized LoRA methods focus on weight only (or deployment) quantization. Can the authors comment of the effect of just \epsilon_A, \epsilon_B, and \epsilon_w on the approximation error ||A_tB_t - \Delta||_F given their relevance to other practical quantized LoRA variants?

Suggestions

· Define notation at first use. Please specify W^\natural’s dimensions around Line 67, define T and N when they first appear around Lines 79 and 128, respectively.

· Experimental reproducibility. Please provide more information on the generated data for the experiments, for example, the seed values used to generate the data and the target update.

· In Line 238, “imply” should be “implies”.

· In practice, it is not ensured that r^* \leq r, it would be useful to include empirical results on the under-ranked regime to see how the observations persist in that setting; refer to the LoRA-One paper, Figure 4(a), parge 7 for an example [1].

[1] Zhang, Yuanhe, Fanghui Liu, and Yudong Chen. ”LoRA-One: One-step full gradient could suffice for fine-tuning large language models, provably and efficiently.” arXiv preprint arXiv:2502.01235 (2025)

Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.