- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work analyzes the learning-rate scaling law across batch size, rank, and model width. It derives a state-dependent linear operator to characterize the weight-space dynamics induced by LoRA updates and the resulting effect on local curvature. Experiments on three models are used to verify the laws across different batch sizes and LoRA ranks.
Critical strengths
- The paper includes systematic learning-rate sweeps across multiple batch sizes and LoRA ranks. These experiments provide a useful empirical basis for understanding how the optimal learning rate changes across different configurations.
Critical weaknesses
-
The motivation is unclear. At line 33, the paper states, “Although FullFT benefits from well-established learning-rate scaling rules, directly transferring these rules to LoRA consistently leads to degraded optimization and a persistent performance gap.” However, it is unclear to me which FullFT scaling laws this statement refers to. I checked the cited reference [1], but I could not find a discussion of learning-rate scaling laws with batch size. My understanding is that the reference shows that LoRA is less tolerant of larger batch sizes, rather than explaining how the learning rate should scale with batch size. This makes the motivation difficult to follow. In addition, [1] reports that both LoRA and FullFT achieve their best loss at smaller batch sizes. Since small batch sizes are usually used in practice, this also leaves me questioning the practical value of analyzing learning-rate scaling with batch size.
-
The abstract and contributions overstate the reduction in search cost. The paper states, “A two-point calibration scheme derived from the law reduces learning-rate search cost by over 30×.” However, this reduction is based on searching across 30 configurations spanning multiple batch sizes and ranks. Practitioners do not usually train LoRA under 30 different combinations of rank and batch size. They usually select a batch size and try only one or two ranks. Therefore, I think the current claim overstates the practical benefit and is misleading without this context. It should be revised to clearly state the search setting in which the reported reduction is achieved.
[1] John Schulman et al. LoRA Without Regret. Thinking Machines Blog, 2025.
Questions for authors
-
From Figure 3, it appears that a rank of 16 usually gives the best performance, while FullFT usually does not perform best. In some cases, FullFT even give the worst performance (B=8). This contradicts the paper’s claim that FullFT generally has greater capacity and gives better performance. However, the authors do not explain why FullFT underperforms in these experiments or what causes this discrepancy. Is this due to instable training occured? More analysis is needed to explain these results and how they relate to the paper’s motivation.
-
The theory relies on strong assumptions that need more justification. For example, Section 3.4 and Appendix A.3 assume approximately isotropic gradient noise, meaning similar noise variance in all directions, and that the gradient signal is concentrated in a few high-curvature directions. These assumptions are central to the claimed d/r increase in the noise scale, but it is unclear how well they hold in the reported experiments.