- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a method for alleviating forgetting in LoRA-based learning. They propose thinking of LoRA's weight update BAx as a reader (up projection A) and a writer (down projection B) for input X. Here, A is a feature selector of X, and then B is the writer that turns the selected features into updates. This interpretation helps explain forgetting in LoRA-based learning. Based on this, the authors propose Inherit LoRA, which constructs A using the task's gradients, filters it against the previous task's reader direction, and freezes it while learning B. This shows a better retention-acquisition tradeoff
Critical strengths
- The papers show competitive results on task retention with fewer trainable parameters (LoRA rank 8)
- I like the reader-writer interpretation. It shows a useful framework for understanding LoRA updates (and the authors have provided a good rationale for it)
- The authors have done a good job with extensive ablations, including reader construction, preconditioning, AdamW reset, etc. These help assess how different components contribute to the performance
Critical weaknesses
- The reader ablation Table 1's claims are a bit confusing. The results don't necessarily show that this produces better acquisition accuracy than a randomly initialized. Better final accuracy indicates methods like O-Lora seem to work better with a task-specific reader. But the reasoning behind this is not clear
- The reader construction ablation (Table 4) shows GradResp is not necessarily the best reader constructor
- The pre-conditioning ablation (Table 5 and 30) is a bit unclear. Table 5 shows degradation in Final ACC w/o preconditioning. But for different lr, Table 30 shows not much difference.
Questions for authors
- How general is the effect for the reader? Did the authors find that the result (table 1 and 16) hold with llama models too?
- Table 5 shows a pre-conditioning benefit, but Table 30 for some learning rates produces nearly similar final ACC. What is authors' takeaway from this? Is pre-conditioning improving robustness across different learning rates or showing a better task retention trade-off, rather than improving the best achievable final accuracy?
- Resetting the optimizer state every update vs resetting every 4 updates shows very similar performance. What led to the choice of resetting every update?