- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work develops scaling laws for LoRA adaptors for language modeling; in particular they characterize the memorization and generalization dynamics of LoRA adaptors as they relate to model size, LoRA rank, and base model capability. They fit scaling laws that predict the "capacity" of an adaptor. In particular, they find that the capacity of the adaptor increases with rank, but with diminishing returns. However, storage capacity alone is not predictive of ‘grokking’; instead, the base model capability and LoRA rank also play a significant role. Higher base model capability (to a certain extent) raises the likelihood of grokking, as does LoRA rank. They find that “real” tasks (e.g., those that are more closely reflected in the pre-training data) require lower rank than synthetic tasks which validate the use of LoRA.
Critical strengths
This work studies a timely topic: efficient model adaption via low-rank adaptors. Their scaling laws are convincingly fit (figure 3), and their findings about the role of rank and base model capacity are both intuitive and lend themselves to actionable guidance for how to tailor training recipes to be more effective.
Critical weaknesses
As an avenue to make model adaptation more effective, the authors suggest investing in raising the base model’s capabilities (line 463) and at the same time they determine that improving base model capabilities is only useful to a certain extent (e.g., until loss ~5). This raises the question: would the model memorize and grok more effectively (in fewer training steps) by directly fine-tuning the model’s parameters (rather than low rank adaptors)? When comparing the number of FLOPs needed to achieve a similar loss via full fine-tune vs. low-rank adaption; which one is more effective? I do recognize that this work is focused on analyzing the practice of using LoRA to adapt models rather than recommending LoRA vs. Fine-tuning; however, since a facet of this analysis involves the base model’s capabilities, I think this is a fair question.
Questions for authors
How is “grokking” explicitly defined in this work?