- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how LoRA learns a new task while retaining knowledge from a previous task. The authors develop a dynamical theory in a two-task teacher–student setting and compare its predictions with numerical simulations. They examine how adapter rank and task similarity affect forgetting and analyze a masking strategy that protects features important to the first task. A sequential MNIST experiment provides additional qualitative validation.
Critical strengths
- The problem setting in this paper is practical and useful, and the entire study design is reasonable.
- The agreement between theoretical predictions and numerical simulations is encouraging. The inverse-selection and ReLU experiments are also useful checks that the masking results are not limited to one particular choice of protected features or activation function.
Critical weaknesses
- The overall approach is understandable. However, the discussion sometimes moves from observations in a particular setting to broader explanations without making the supporting evidence clear.
- The explanation is not well established, and if you compare forgetting when the methods have reached the same performance on the second task, it would make the conclusion more convincing.
- The experiments don't explain your theory and cannot convince me.
Questions for authors
- How should the results in Appendix E change our interpretation of the main findings?
2 comments
Response to Reviewer KMck (2/2)
W3: We thank the reviewer for this comment. We agree that the MNIST experiment of the original submission, which uses the same two-layer network as the theory, is only a qualitative check of the ordering of the methods and does not test the mechanisms behind it. In the revised version, Fig. 2 shows the theory, sequential MNIST and the deep benchmark side by side, all compared at matched transfer (Fig. 2a–c). On the deep multi-task benchmark described in the general answer (ResNet-18, six tasks of Split CIFAR-100), we therefore measure the deep analogues of the quantities through which the theory explains forgetting, rather than only error curves.
The theory attributes the forgetting of LoRA to the change of the Task 1 alignment, in Eq. (18) of the main text, where the rows of set which hidden units are modified. The low rank of the update does not constrain these rows. Their deep analogue is the fraction of the update that falls on the channels most important for Task 1, presented in Fig. 10a. For vanilla LoRA, this fraction stays at chance level for every rank: as predicted, the low-rank constraint does not steer the update away from the channels that carry Task 1. SDGM sets this fraction to zero by construction, and reduces forgetting at matched transfer as can be seen in Fig. 9.
At the level of the representation, the theory explains the differences between methods through the overlaps of the student with the Task 1 teacher (Fig. 1c of the main text). Their deep analogue is the CKA similarity of the Task 1 features to their state at the first task switch, exposed in Fig. 10b. The ordering matches the theory: standard fine-tuning moves the Task 1 representation the most, LoRA less, and LoRA + SDGM the least. Together, these measurements connect the deep results to the mechanism identified by the theory: the rank alone does not determine which units the update modifies, and protecting the units used by Task 1 is what reduces forgetting.
Q1: We thank the reviewer for this question, which led us to substantially revise Appendix E. In the revised version, Task 2 is trained for , long enough for methods to reach the specialized phase, and all methods are compared at equal Task 2 performance rather than at a fixed training time, see Fig. 8. The results refine the interpretation of the main findings in three ways.
- The central claim is confirmed, and SDGM becomes the only method that protects Task 1. Protecting the hidden units that carry Task 1 information through SDGM reduces forgetting at equal Task 2 performance. From Fig. 8b, at every level of Task 2 error, both SDGM variants forget less than standard fine-tuning and LoRA. The same holds over the whole range of task similarity when the transfer is matched across methods, see Fig. 8d-e.
- The benefit of low rank alone is regime-dependent. In the unspecialized regime of the main text, vanilla LoRA forgets less than standard fine-tuning. In the specialized regime it forgets as much. The theory gives the reason: once each student unit is aligned with one teacher feature, Task 2 is learned by re-using these units (Lee et al., 2022). This requires an update of rank at most , which a rank- adapter with can perform. The rank constraint is therefore not binding, and LoRA rewrites the Task 1 representation just as standard fine-tuning does. This is visible directly in the Task 1 overlaps shown in Fig. 8f. Under both standard fine-tuning and LoRA, the overlap of the specialized units with the Task 1 teacher drops from about 1 to about , the value expected once they are realigned with the Task 2 teacher, while SDGM keeps it unchanged. This sharpens the main message of the paper: what controls forgetting is not the rank of the update but its orientation relative to the features used by Task 1. Only the state-dependent mask protects Task 1 in both regimes.
- The dependence on task similarity changes. In the unspecialized regime, forgetting peaks at intermediate similarity. In the specialized regime, at matched transfer, the forgetting of standard fine-tuning and LoRA decreases with , because the re-used units end up aligned with the Task 2 teacher, whose overlap with the Task 1 teacher is (Lee et al., 2022). As can be seen in Fig. 8d, with SDGM, forgetting remains well below standard fine-tuning and LoRA and nearly independent of .
The main additional cost of SDGM is speed. For LoRA + SDGM, the plastic units must first be recruited from scratch, which produces a long plateau before Task 2 is learned (Fig. 8a,c). We have updated Appendix E accordingly and added a sentence in Section 5.1 stating that the advantage of vanilla LoRA over standard fine-tuning holds in the unspecialized regime, while the advantage of SDGM holds in both.
Response to Reviewer KMck (1/2)
We thank the reviewer for assessing the relevance of our problem setting. Below, we address all weaknesses and questions. We hope these answers resolve the main concerns, and would be grateful if the reviewer would consider raising our score.
W1: We thank the reviewer for this comment, which we found fair. In several places, the original text stated conclusions obtained in a specific setting (the unspecialized regime, at fixed task similarity and rank) in more general terms than the evidence supports. We revised the manuscript so that each claim states the setting in which it holds and points to the evidence supporting it, and we now separate the conclusions that are robust from those that depend on the regime:
We also rephrased the sentences that presented interpretations as general mechanisms, so that they explicitly refer to the teacher-student model.
W2: We thank the reviewer for this important point. The concern is legitimate: LoRA is often reported to learn less and forget less (Biderman et al., 2024), although this is not always the case (Shuttleworth et al., 2025), so a lower forgetting could in principle simply reflect a weaker adaptation to Task 2. Following this remark, Fig. 2 of the revised manuscript now compares all methods at matched transfer: it plots Task 1 forgetting against Task 2 transfer along training (Eq. (25)), so that a method cannot appear more stable simply because it learns less. In the theory (Fig. 2a), LoRA + SDGM forgets the least over the whole transfer range, and the same ordering holds on sequential MNIST (Fig. 2b) and in the deep benchmark (Fig. 2c).
The other theory figures support the same conclusion. In Fig. 3a-b, where methods are compared after a fixed number of samples, all four methods reach almost the same final transfer over the whole range of task similarity, while their forgetting differs markedly. The rank sweep of Fig. 3c reports forgetting against transfer directly. In the specialized regime, where learning speeds differ strongly, we now compare all methods at matched Task 2 performance (revised Appendix E, see our answer to Q1): SDGM forgets substantially less than standard fine-tuning and LoRA at every level of Task 2 error and for all task similarities.
We also performed this comparison on the deep multi-task benchmark described in the general answer (ResNet-18, six tasks of Split CIFAR-100), where the methods are compared explicitly at matched transfer on the new task, across learning rates shown in Fig. 9. Panel (a), now also shown in the main text as Fig. 2c, follows the two-task protocol of the theory, with Task 1 forgetting against transfer on Task 2, and panel (b) averages the forgetting of all previous tasks over the six-task sequence. In both cases, the ordering predicted by the theory holds at matched transfer, with LoRA + SDGM forgetting the least. In particular, at rank L=8, LoRA still forgets less than standard fine-tuning at equal transfer, so in this setting its lower forgetting is not only a consequence of learning less (Biderman et al., 2024).