acceptodds
Withdrawn from ICLR 2027

Understanding Learning-Rate Scaling with Batch Size in LoRA

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LoRA and full fine-tuning (FullFT) exhibit different learning-rate scaling behaviors as batch size increases, with LoRA requiring substantially larger learning rates at large batch sizes. The origin of this difference is not well understood and is often attributed to the low-rank nature of LoRA. We show that the gap reflects a learning-rate scaling mismatch rather than a capacity deficit. Starting from the coupled updates of the low-rank factors, we derive a state-dependent linear operator that characterizes the induced weight-space dynamics and the resulting effect on local curvature. This analysis reveals that LoRA changes the relative retention of gradient signal and stochastic noise, leading to a larger effective gradient-noise scale and consequently a different batch-size dependence of the optimal learning rate. Building on this mechanism, we derive a closed-form learning-rate scaling law that captures the dependence on batch size, rank, and model width. Fitted on the measured optima, the law accurately reproduces the observed scaling and, with no refitting, transfers to unseen models of different widths and architectures. A two-point calibration scheme derived from the law reduces learning-rate search cost by over 30.

Primary area:
learning theory
Keywords:
LoRA, Learning-rate scaling, Batch size

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6mFtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work analyzes the learning-rate scaling law across batch size, rank, and model width. It derives a state-dependent linear operator to characterize the weight-space dynamics induced by LoRA updates and the resulting effect on local curvature. Experiments on three models are used to verify the laws across different batch sizes and LoRA ranks.

Critical strengths
  1. The paper includes systematic learning-rate sweeps across multiple batch sizes and LoRA ranks. These experiments provide a useful empirical basis for understanding how the optimal learning rate changes across different configurations.
Critical weaknesses
  1. The motivation is unclear. At line 33, the paper states, “Although FullFT benefits from well-established learning-rate scaling rules, directly transferring these rules to LoRA consistently leads to degraded optimization and a persistent performance gap.” However, it is unclear to me which FullFT scaling laws this statement refers to. I checked the cited reference [1], but I could not find a discussion of learning-rate scaling laws with batch size. My understanding is that the reference shows that LoRA is less tolerant of larger batch sizes, rather than explaining how the learning rate should scale with batch size. This makes the motivation difficult to follow. In addition, [1] reports that both LoRA and FullFT achieve their best loss at smaller batch sizes. Since small batch sizes are usually used in practice, this also leaves me questioning the practical value of analyzing learning-rate scaling with batch size.

  2. The abstract and contributions overstate the reduction in search cost. The paper states, “A two-point calibration scheme derived from the law reduces learning-rate search cost by over 30×.” However, this reduction is based on searching across 30 configurations spanning multiple batch sizes and ranks. Practitioners do not usually train LoRA under 30 different combinations of rank and batch size. They usually select a batch size and try only one or two ranks. Therefore, I think the current claim overstates the practical benefit and is misleading without this context. It should be revised to clearly state the search setting in which the reported reduction is achieved.

[1] John Schulman et al. LoRA Without Regret. Thinking Machines Blog, 2025.

Questions for authors
  1. From Figure 3, it appears that a rank of 16 usually gives the best performance, while FullFT usually does not perform best. In some cases, FullFT even give the worst performance (B=8). This contradicts the paper’s claim that FullFT generally has greater capacity and gives better performance. However, the authors do not explain why FullFT underperforms in these experiments or what causes this discrepancy. Is this due to instable training occured? More analysis is needed to explain these results and how they relate to the paper’s motivation.

  2. The theory relies on strong assumptions that need more justification. For example, Section 3.4 and Appendix A.3 assume approximately isotropic gradient noise, meaning similar noise variance in all directions, and that the gradient signal is concentrated in a few high-curvature directions. These assumptions are central to the claimed d/r increase in the noise scale, but it is unclear how well they hold in the reported experiments.

Reviewer_AZQ5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how optimal learning rates scale with batch size and LoRA rank. It derives a state-dependent operator describing the first-order weight-space dynamics of LoRA under SGD and proposes that LoRA increases the effective gradient noise scale by a factor proportional to the width-to-rank ratio. Based on this mechanism, the authors introduce a three-parameter learning-rate scaling law and evaluate it on three language models. They further develop a two-point calibration scheme to reduce learning-rate tuning costs and test it on additional models.

Critical strengths
  1. Relevant and practically motivated problem. The paper studies the interaction between batch size, LoRA rank, and learning rate, which is important for efficient fine-tuning. Its focus on understanding existing optimization behavior, rather than introducing another LoRA variant, is valuable.
  2. Simple and potentially useful predictive framework. The proposed scaling law provides a compact description of the empirical learning-rate trends. The two-point calibration approach shows promising potential for reducing hyperparameter search costs.
Critical weaknesses
  1. The proposed scaling law relies on several strong assumptions that are neither rigorously justified nor directly validated experimentally. These include the isotropy of gradient noise, the concentration of gradient signal along dominant Hessian eigendirections, and the stronger suppression of signal than noise by LoRA's low-rank subspace. In particular, the key d/rd/r inflation of the gradient noise scale is not convincingly derived from the proposed operator. Moreover, the theoretical analysis is based on SGD, while the experiments use AdamW.
  2. The proposed scaling law shows only marginal improvements over a simple power law with the same number of parameters, and even performs slightly worse on EXP-03. Although the authors attribute this to the saturation regime lying beyond the measured batch-size range, this explanation also highlights that the experiments do not sufficiently validate the proposed functional form. In particular, the claimed advantage of modeling the transition to saturation remains largely untested.
Questions for authors
  1. Can the authors provide theoretical justification or direct empirical evidence for the assumptions underlying the proposed noise-scale analysis? In particular, can they verify that LoRA suppresses gradient signal more strongly than noise, and that the resulting noise-scale inflation follows the claimed d/rd/r relationship?
  2. The paper defines B∞B_\infty as the gradient noise scale of FullFT, arguing that FullFT is recovered when r=dr=d. However, a full-rank LoRA parameterization does not necessarily recover the optimization dynamics of FullFT, since PtP_t is not generally the identity even when r=dr=d. What justifies this identification? Has the fitted B∞B_\infty been independently verified against the actual gradient noise scale of FullFT?
  3. Given that the proposed scaling law provides only marginal improvements over a simple power law within the evaluated range, can the authors demonstrate its advantage in regimes where the two formulations make substantially different predictions, particularly near or beyond the predicted saturation point?
Reviewer_xjEFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work studies the difference in learning rate scaling between LoRA and full-rank training. The work argues that the low-rank subspace of LoRA is not sufficient to explain performance differences between LoRA and full-rank training. Instead, the authors argue that the differences can be attributed to different optimal learning rates for the two schemes. Their analysis involves taking a first-order approximation of the LoRA gradient step and applying the resulting effective Hessian to the scaling rule from McCandlish et al. (2018), which suggests that LoRA requires a larger learning rate than full finetuning. Based on this, they provide some heuristics for scaling the learning rate. They then propose a method for choosing the LoRA learning rate that is 30 times less computation than an exhaustive grid search; however, the method requires multiple full rank training runs.

Critical strengths

The problem of learning rate selection for LoRA is highly relevant.

Critical weaknesses
  1. There are a few key weaknesses of the main argument of the paper. First, the paper begins with the premise that LoRA requires a different learning-rate scaling rule than full-rank finetuning. Their only citation to support this is a blog post by Thinking Machines. Indeed, other works cited in the introduction (Wang et al. (2025); Yen et al. (2025)) do not cite this blogpost and are not motivated by scaling laws, just general improvement to LoRA. This work should provide convincing evidence in a wide range of settings that this is a core issue, and that using a scaling rule dictated by full-rank finetuning leads to worse performance.

Second, the authors' main claim is that the gap between LoRA and full finetuning is due to learning rate scaling, "not a structural defect." This claim is too strong. One key point they make is that the performance gap does not disappear as the rank increases. However, in reality the difference can be attributed to both the low-rank restricted subspace and the additional structure imposed by LoRA, and there is a clear explanation for why the gap does not disappear. LoRA imposes additional structure by introducing a layer of multiplication between weights. A simple example is if we optimize L(W)L(W), where WW is n×nn \times n, and we reparametrize W=BAW = BA, where BB and AA are also n×nn \times n. Then optimizing over B and A will have the same rank as optimizing WW, but the loss landscape will have changed to be more ill-conditioned. As discussed in several prior works (see below Mu and Klabjan, ICML 2026) the LoRA reparametrization can cause a slowdown in gradient descent convergence regardless of rank. In conclusion, just because the performance gap does not disappear does not imply that rank is irrelevant; moreover, there is a clear explanation for the gap that goes beyond learning rate scaling.

  1. The mathematical argument is weak and relies on a simple first-order expansion of the gradient step, leading to an effective Hessian that can be plugged into the scaling law from McCandlish et al. (2018). This prior work is presented with no explanation of the assumptions on the noise; in particular, the specific sampling scheme or loss function are not stated, both of which may affect the behavior of the noise. The analysis essentially relies on approximating the loss function as a local quadratic function, which is overly simplistic. With these arguments, the authors observe the ratio of BnoiseB_{noise} for LoRA and full finetuning (Equation 7) and make some qualitative statements about what the ratio might imply. Ultimately, they conclude that BnoiseLoRA>BnoiseFullFTB^{LoRA}_{noise} > B^{FullFT}_{noise}, but their scaling rules and guidelines are heuristically chosen.

  2. The experimental section is messy and poorly formatted. The purpose of the experimental section is to explain what you did and what your experiments show, in prose, not to provide an exhaustive list of hyperparameters in partial sentences. The authors propose a two-point method for predicting the learning rate, and state that it is 30 times faster than the grid search approach. However, first, their method requires multiple full-rank training runs to calibrate the method, which is impractical and misses the point of LoRA, a method for reducing computation especially when full-rank training is computationally intractable. Second, there are far more sophisticated methods for hyperparameter tuning than grid search that are used in practice, which the paper does not compare against.

  3. The writing is dense and difficult to parse. An example of this is Lines 055-059:

"Spectral analysis reveals that this operator asymmetrically filters gradient signal, which is concentrated in sharp curvature directions, far more than stochastic noise, which is diffusely distributed, inflating the gradient noise scale in proportion to the ratio of model width to adapter rank."

I suggest the authors revise to improve readability and clarity. In addition, the citations are not properly formatted in parentheses, which is confusing to read; in particular, using the ICLR LaTex submission template, the citations should use \citep instead of \cite.

  1. The related work is far too short and does not provide a clear picture of the literature. The paper's contribution and novelty compared to existing work is unclear. Some papers the authors should consider citing include:

Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models, Zhang and Pilanci, ICML 2024

LoRA Learns Less and Forgets Less, Biderman et al., TMLR 2024

LoRA vs Full Fine-tuning: An Illusion of Equivalence, Shuttleworth et al., NeurIPS 2025

On the Convergence Rate of LoRA Gradient Descent, Mu and Klabjan, ICML 2026

Questions for authors

What assumptions on the noise are you using to determine the noise scale?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.