- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this manuscript, the authors study the convergence of LoRA optimization when the original loss is smooth and lower bounded, but the factorized objective is not globally smooth. It improves the deterministic LoRA-GD complexity to (O(\epsilon^{-4})), provides a finite-variance counterexample for plain LoRA-SGD, and establishes stochastic oracle complexities of (O(\epsilon^{-8})) for normalized momentum and (O(\epsilon^{-6})) for a STORM variant.
Critical strengths
-
The deterministic analysis improves the previous exponential sufficient bound to a polynomial bound for the same algorithm and stationarity criterion.
-
The scalar counterexample explains why finite-variance product-space gradients do not suffice for ordinary simultaneous factor updates.
-
The stochastic proofs account for factor-dependent smoothness and variance without assuming uniformly bounded factors. I did not identify a specific error in the main arguments.
Critical weaknesses
-
The stochastic algorithms adapt established methods. Further discussion of rate tightness would help clarify the theoretical contribution.
-
The experiments use tuned parameters and schedules that differ from those analyzed, so they do not directly validate the convergence rates.
-
Comparisons mainly involve prior adaptive LoRA-GD rules. Tuned SGD with momentum and AdamW, repeated runs, and comparisons under matched computational budgets would strengthen the evaluation.
-
Factor-space stationarity need not imply a useful solution or stationarity in the product variable. This limitation deserves more discussion.
Questions for authors
-
Could sharper trajectory control improve the stochastic rates, or are the current bounds expected to be tight?
-
Does STORM’s advantage remain when comparing gradient evaluations or runtime rather than iterations?
-
In the multilayer experiments, is normalization performed jointly across adapters or separately by layer? How does this correspond to the analysis?
-
Can a controlled experiment measure the full factor-gradient norm under the theoretical schedules?