acceptodds
Withdrawn from ICLR 2027

LoRA Memorization Scaling Laws

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LoRA adapters have become the canonical method for injecting new knowledge and behavior into a pretrained model, yet their physical limits are not yet fully understood. This work measures the capacity of adapters to memorize and generalize by training on synthetic datasets of both random facts and hidden modular rules. On the DataDecide OLMo checkpoints between 20M and 1B parameters, we observe that an adapter stores up to 2.2 bits per projection parameter, with an efficiency that falls with rank and rises with base model quality. Our memorization scaling laws predict the target held-out 1B x rank within 11% errors. While capacity alone does not decide whether an adapter can generalize to modular rules, it requires a sufficient number of directions to represent facts and can grok faster on a pretrained base checkpoint of higher quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LoRA, parameter-efficient finetuning, scaling laws, memorization, capacity, grokking, pretraining, post-training, low-rank adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FQhzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work develops scaling laws for LoRA adaptors for language modeling; in particular they characterize the memorization and generalization dynamics of LoRA adaptors as they relate to model size, LoRA rank, and base model capability. They fit scaling laws that predict the "capacity" of an adaptor. In particular, they find that the capacity of the adaptor increases with rank, but with diminishing returns. However, storage capacity alone is not predictive of ‘grokking’; instead, the base model capability and LoRA rank also play a significant role. Higher base model capability (to a certain extent) raises the likelihood of grokking, as does LoRA rank. They find that “real” tasks (e.g., those that are more closely reflected in the pre-training data) require lower rank than synthetic tasks which validate the use of LoRA.

Critical strengths

This work studies a timely topic: efficient model adaption via low-rank adaptors. Their scaling laws are convincingly fit (figure 3), and their findings about the role of rank and base model capacity are both intuitive and lend themselves to actionable guidance for how to tailor training recipes to be more effective.

Critical weaknesses

As an avenue to make model adaptation more effective, the authors suggest investing in raising the base model’s capabilities (line 463) and at the same time they determine that improving base model capabilities is only useful to a certain extent (e.g., until loss ~5). This raises the question: would the model memorize and grok more effectively (in fewer training steps) by directly fine-tuning the model’s parameters (rather than low rank adaptors)? When comparing the number of FLOPs needed to achieve a similar loss via full fine-tune vs. low-rank adaption; which one is more effective? I do recognize that this work is focused on analyzing the practice of using LoRA to adapt models rather than recommending LoRA vs. Fine-tuning; however, since a facet of this analysis involves the base model’s capabilities, I think this is a fair question.

Questions for authors

How is “grokking” explicitly defined in this work?

Reviewer_bt3BOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper measures how much information LoRA adapters can hold. Using the compression-based definition of unintended memorization from prior work, it trains adapters on a synthetic table of random facts. Experiments cover six DataDecide OLMo bases (20M-1B), ranks 1-16 (up to 640 for the grokking experiments), and intermediate pretraining checkpoints. The paper shows that adapters store up to ~2.2 bits per projection parameter, and this efficiency decreases as rank grows. The authors fit a law that is linear in the adapter’s projection parameters, discounted by a rank-efficiency term and scaled by a saturating factor of the base’s C4 loss. The proposed memorization law predicts 5 held-out 1B points with 11% mean error. Grokking on modular addition is also studied. Adapters need roughly 100-160 directions and a sufficiently pretrained base, and they grok far more slowly than full fine-tuning, while real-world tasks need far fewer directions.

Critical strengths
  1. The paper shows how LoRA learning on real datasets differs from synthetic ones. For example, trained rank-16 adapters on real tasks have an effective rank of only 1.5-2.1, versus 14.1 on random facts.

  2. The proposed scaling law is simple and interpretable; every factor is motivated by an observation, and alternative functional forms are analyzed (Table 5). The law is fit across ranks, base model sizes and checkpoints, and tested on a held-out 1B base.

Critical weaknesses
  1. The largest base model is 1B and the law is fit on models with at most 530M parameters. LoRA is typically used on much larger models to save compute, so it is unclear whether the conclusions replicate on such models.

  2. Back et al. (2026) already studies LoRA as knowledge memory (the submission cites it only in the introduction). On 8B bases with ranks from 2 to 1024, they find that capacity grows with rank and that knowledge stored per trainable parameter is highest at low ranks. I would like the authors to state clearly which results could not have been drawn from previous work.

I lean towards rejection mainly because the law is fit only up to 530M and checked at a single larger size (1B), where all five predictions overshoot, so it is unclear whether it holds at the scales where LoRA is typically used.

Questions for authors

Could you extend Figure 7 with a per-layer breakdown of stored bits, ideally crossed with a per-module breakdown? Does the bits-based measure agree with the result of Back et al. (2026) result of stronger memorization when the adapter is in early layers?

Could you test the law on at least one base of 4B or larger? It would be insightful to see how accurately the law, fitted on models of up to 530M (or up to 1B) parameters, scales to a much larger model.

Why do you use 200 epochs for each training run, and did you ablate this choice? An epoch number ablation on a few cells would show whether 200 is too large or insufficient. It would also show whether the peak-then-decline shape repeats at longer schedules and whether it is a training artifact.

References:

Back, S., Lee, D., Kang, N., Lee, T., Hong, S. K., Gwon, Y., & Ahn, S. (2026). Understanding LoRA as knowledge memory: An empirical analysis. arXiv preprint arXiv:2603.01097.

Reviewer_M3hiOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper measures how much information LoRA adapters can store, following the compression-based capacity definition of Morris et al. (2025). Adapters are trained on a synthetic table of random facts (a ∘ b = c, with c drawn uniformly from V = 2,000 single-token answers), and stored bits are computed as Σ (log₂ V − NLL₂), with negative contributions retained. Capacity is the peak of this quantity over dataset size. Using DataDecide OLMo checkpoints from 20M to 1B parameters and ranks 1 to 16, the paper reports about 2.1 bits per projection parameter at rank 1, falling to about 1.2 at rank 16, and a decline of up to 36% past the peak. It fits the law C = k · p · η(r) · g(ℓ), with η(r) = 1/(1 + r/r₀) and g(ℓ) = 1/(1 + (ℓ/ℓ₀)^m), to bases up to 530M and predicts the held-out 1B points with 11% mean error. On a + b mod 97, adapters grok only above about 100–160 ranks and only on sufficiently pretrained bases, and much more slowly than full fine-tuning. Truncated full fine-tuning and spectral analysis suggest that PopQA, SuperNI and Aya tasks need few directions.

Critical strengths
  1. The paper conducts wide experiments. The capacity measurement is careful.

  2. The paper writing is generally readable.

Critical weaknesses
  1. The rank-efficiency term may be an artefact of the α/r scaling.

  2. The real-task analysis doesn't measure knowledge injection. The conclusion that budgets “should instead be spent on the data and on improving base model quality” can't follow from these experiments. The low effective ranking of the training adapter may reflect an optimization bias in the LoRA algorithm, i.e., a tendency to favor a certain dominant direction.

  3. The headline numbers are reported inconsistently: “up to 2.2” bits per parameter in the abstract, “2.1 at rank 1, falling to about 1.1 at rank 16” in the contributions, and “2.1 to 1.2” and “about 2.05” in Section 3. The number of fitted points also differs between Section 3 (22 final checkpoints, 76 points) and Table 5 (19 + 5 and 71 + 5). Several typos remain (“base;s”, “capcity”, “via is h m”, “prioritizes”).

Questions for authors

I worked on the LoRA, but this paper is outside my field, so I cannot assess its contribution to the field or its research significance.

  1. The effective step size of the rank-16 adapter is only 1/16 that of the rank-1 adapter for the same learning rate and a fixed budget of 200 epochs. The number of bits stored per parameter becomes 1.2, and the resulting fitted η(r) likely reflects slower optimization rather than reduced storage efficiency. I suggest the authors control the variables, not change the α/r scaling relationship, and verify the results using multiple parameters.

  2. Why does the discovery that PopQA requires fewer directions lead to the conclusion that knowledge injection is low-ranked?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.