- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper uses gradient flow to study the optimization dynamics and rank-dependent implicit bias of LoRA. Mainly, it derives the gradient-flow equations from discrete updates and gives exact limiting solutions for a trace-squared objective and low-rank approximation under specific initializations.
Critical strengths
The paper finds exact gradient-flow trajectories and limiting solutions for the two objectives under the given initializations. This lets us directly see how rank and initialization affect the optimization.
Critical weaknesses
-
The technical novelty of Theorem 2.5 is limited. The analysis appears to follow standard techniques for first-order ODE approximation. The paper does not clearly identify the additional technical or conceptual contribution beyond these standard techniques.
-
The motivation for the two examples and their relevance to LoRA are insufficiently developed. The authors choose these two objectives since their gradient-flow dynamics can be solved explicitly. However, the paper does not sufficiently explain what these examples reveal about LoRA.
In addition, the spectral initialization assumption in the low-rank approximation setting requires exact spectral alignment and therefore seems too strong, although the authors acknowledge this limitation.
Questions for authors
-
Since Eqs. (26) and (32) already give the exact limiting solutions, Eq. (33) can be obtained directly by simple algebra. Could the authors explain what extra insights into LoRA we get from the rank-dependent results in Theorem 2.8?
-
Why did the authors choose the trace-squared and low-rank approximation problems, other than the fact that their gradient flows have closed-form solutions? Specifically, what properties of LoRA do these two examples capture, and what do the results tell us about real fine-tuning tasks?
-
Theorem 2.5 only holds on finite time intervals, while Theorems 2.8 and 2.11 study the limit as . Could the authors explain how these results connect to each other? Also, what do they imply for the discrete algorithm when using a fixed, small step size?
-
The authors state that their contribution is methodological. Could you clarify what is new in the proposed method itself, and what theoretical insights or practical guidance it gives for using LoRA?