- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies gradient descent for low-precision LoRA in a noiseless low-rank linear regression model. Building on spectral initialization, it derives finite-horizon recovery-error bounds comprising a geometric term and quantization-dependent terms. It also argues that overparameterized LoRA requires tighter precision control.
Critical strengths
Analyzing quantization throughout the LoRA training computation is a relevant question. The paper gives explicit bounds under a clearly stated quantization model and distinguishes exact-rank and overparameterized cases.
Critical weaknesses
-
Insufficient positioning against prior work. Although the paper cites Mu and Klabjan, it does not clearly distinguish their general stationary-point guarantee from its more restrictive recovery guarantee. Given the similar titles and shared focus on LoRA convergence, the authors should explain why their low-precision analysis requires a noiseless linear model and spectral initialization, and what is gained under these assumptions.
-
Limited theoretical novelty. Spectral initialization and geometric convergence are established in LoRA-One. The new analysis bounds quantization perturbations around that trajectory, but the paper does not clearly identify the key technical step beyond a standard perturbation argument. Its finite-horizon guarantees also require admissible quantization error to shrink as the horizon grows; claims about an error floor or required precision should be qualified.
-
Unclear over-rank mechanism. In the noiseless model, has rank at most . For , the extra adapter directions appear to start at zero. Since the relative-error assumption implies , the stated updates appear to keep them at zero. How can these directions accumulate the error claimed in Theorem 4.3, and what explains the rank-dependent differences in Figure 2?
-
Limited empirical support. Section 4.3 evaluates only synthetic noiseless regression with . Experiments using a specified practical low-precision format, with recovery quality and training cost or memory measurements, would help establish the relevance of the theoretical regime.
-
Presentation needs substantial improvement. For a primarily theoretical paper, the main insight is difficult to identify within the lengthy proof. The authors should provide a clearer proof roadmap, identify which lemmas are new relative to LoRA-One, and summarize how the assumptions and precision conditions affect the scope of the results. Unresolved cross-references in the theorem statements should also be corrected.
Questions for authors
See weaknesses. I encourage the authors to substantially improve the writing, particularly the explanation of the main technical contribution, the proof structure, and the implications of the assumptions.