acceptodds
Under review as a conference paper at ICLR 2027

A Dynamical Theory of LoRA in Continual Learning

Abstract

Despite the widespread use of Low-Rank Adaptation (LoRA), little is known about its dynamics in continual learning and the mechanisms by which low-rank updates affect catastrophic forgetting. We provide an asymptotically exact dynamical characterization of LoRA in a solvable two-task teacher-student model. In the high-dimensional online-learning limit, we derive a closed system of ordinary differential equations for a finite set of macroscopic order parameters, yielding exact expressions for the generalization errors throughout both the initial Task 1 learning phase and the subsequent LoRA fine-tuning on Task 2. The theory quantitatively matches finite-dimensional simulations and exposes two characteristic effects of LoRA: low-rank adaptation reduces interference with features learned on the first task, but its initialization slows adaptation to the second task. Building on this mechanistic picture, we analyze a state-dependent masking strategy that freezes hidden units carrying the strongest first-task representations and restricts adaptation to the complementary subspace. This structural partitioning markedly reduces forgetting, while preserving plasticity on the new task. Our framework further clarifies the role of adapter rank: transfer improves only up to the intrinsic dimensionality of the target task and saturates beyond it, while forgetting continues to grow with rank. These results provide a dynamical and geometric account of how low-rank adaptation organizes information across sequential tasks and are qualitatively reproduced on a sequential MNIST benchmark.

Primary area:
learning theory
Keywords:
machine learning theory, statistical physics, low-rank adaptation, continual learning, catastrophic forgetting, online learning, sequential training, training dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KMckOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how LoRA learns a new task while retaining knowledge from a previous task. The authors develop a dynamical theory in a two-task teacher–student setting and compare its predictions with numerical simulations. They examine how adapter rank and task similarity affect forgetting and analyze a masking strategy that protects features important to the first task. A sequential MNIST experiment provides additional qualitative validation.

Critical strengths
  1. The problem setting in this paper is practical and useful, and the entire study design is reasonable.
  2. The agreement between theoretical predictions and numerical simulations is encouraging. The inverse-selection and ReLU experiments are also useful checks that the masking results are not limited to one particular choice of protected features or activation function.
Critical weaknesses
  1. The overall approach is understandable. However, the discussion sometimes moves from observations in a particular setting to broader explanations without making the supporting evidence clear.
  2. The explanation is not well established, and if you compare forgetting when the methods have reached the same performance on the second task, it would make the conclusion more convincing.
  3. The experiments don't explain your theory and cannot convince me.
Questions for authors
  1. How should the results in Appendix E change our interpretation of the main findings?
2 comments
AuthorsOpenReview ↗

Response to Reviewer KMck (1/2)

We thank the reviewer for assessing the relevance of our problem setting. Below, we address all weaknesses and questions. We hope these answers resolve the main concerns, and would be grateful if the reviewer would consider raising our score.

W1: We thank the reviewer for this comment, which we found fair. In several places, the original text stated conclusions obtained in a specific setting (the unspecialized regime, at fixed task similarity and rank) in more general terms than the evidence supports. We revised the manuscript so that each claim states the setting in which it holds and points to the evidence supporting it, and we now separate the conclusions that are robust from those that depend on the regime:

  • LoRA reduces interference with Task 1 compared to standard fine-tuning. This is supported by the generalization errors and by the overlaps with both teachers (Fig. 1b–d), in theory and simulations, and across task similarities (Fig. 3a). The revised Appendix E shows that it holds in the unspecialized regime only: once the Task 1 representation is specialized, LoRA forgets as much as standard fine-tuning. We now state this restriction in Section 4 and in the abstract.
  • LoRA slows early adaptation to Task 2. This is supported by Fig. 1b,d, and the explanation is read directly from the order parameters: the zero-initialized down-projection must first build up its overlaps with the Task 2 teacher through the rank-LL bottleneck. The same slowdown is observed in the specialized regime (Appendix E).
  • State-dependent masking reduces forgetting at equal Task 2 performance. This is supported in theory and simulations (Figs. 2a and 3a), by the inverse and random selection controls at equal capacity (Appendix D.2), with ReLU activations (Appendix D.3), in the specialized regime (Appendix E), and in a deep multi-task benchmark (Fig. 2c and Appendix G), as well as qualitatively on MNIST (Fig. 2b and Appendix H). This is the most robust conclusion of the paper.
  • Transfer saturates once the adapter rank reaches the dimension of the target task. This is supported by the rank sweep of Fig. 3c, obtained at fixed task similarity in the unspecialized regime; we now state this setting explicitly.

We also rephrased the sentences that presented interpretations as general mechanisms, so that they explicitly refer to the teacher-student model.

W2: We thank the reviewer for this important point. The concern is legitimate: LoRA is often reported to learn less and forget less (Biderman et al., 2024), although this is not always the case (Shuttleworth et al., 2025), so a lower forgetting could in principle simply reflect a weaker adaptation to Task 2. Following this remark, Fig. 2 of the revised manuscript now compares all methods at matched transfer: it plots Task 1 forgetting against Task 2 transfer along training (Eq. (25)), so that a method cannot appear more stable simply because it learns less. In the theory (Fig. 2a), LoRA + SDGM forgets the least over the whole transfer range, and the same ordering holds on sequential MNIST (Fig. 2b) and in the deep benchmark (Fig. 2c).

The other theory figures support the same conclusion. In Fig. 3a-b, where methods are compared after a fixed number of samples, all four methods reach almost the same final transfer over the whole range of task similarity, while their forgetting differs markedly. The rank sweep of Fig. 3c reports forgetting against transfer directly. In the specialized regime, where learning speeds differ strongly, we now compare all methods at matched Task 2 performance (revised Appendix E, see our answer to Q1): SDGM forgets substantially less than standard fine-tuning and LoRA at every level of Task 2 error and for all task similarities.

We also performed this comparison on the deep multi-task benchmark described in the general answer (ResNet-18, six tasks of Split CIFAR-100), where the methods are compared explicitly at matched transfer on the new task, across learning rates shown in Fig. 9. Panel (a), now also shown in the main text as Fig. 2c, follows the two-task protocol of the theory, with Task 1 forgetting against transfer on Task 2, and panel (b) averages the forgetting of all previous tasks over the six-task sequence. In both cases, the ordering predicted by the theory holds at matched transfer, with LoRA + SDGM forgetting the least. In particular, at rank L=8L = 8, LoRA still forgets less than standard fine-tuning at equal transfer, so in this setting its lower forgetting is not only a consequence of learning less (Biderman et al., 2024).

AuthorsOpenReview ↗

Response to Reviewer KMck (2/2)

W3: We thank the reviewer for this comment. We agree that the MNIST experiment of the original submission, which uses the same two-layer network as the theory, is only a qualitative check of the ordering of the methods and does not test the mechanisms behind it. In the revised version, Fig. 2 shows the theory, sequential MNIST and the deep benchmark side by side, all compared at matched transfer (Fig. 2a–c). On the deep multi-task benchmark described in the general answer (ResNet-18, six tasks of Split CIFAR-100), we therefore measure the deep analogues of the quantities through which the theory explains forgetting, rather than only error curves.

The theory attributes the forgetting of LoRA to the change of the Task 1 alignment, βBΛ⊤\beta \mathbf{B} \boldsymbol{\Lambda}^\top in Eq. (18) of the main text, where the rows of B\mathbf{B} set which hidden units are modified. The low rank of the update does not constrain these rows. Their deep analogue is the fraction of the update ΔW\Delta W that falls on the channels most important for Task 1, presented in Fig. 10a. For vanilla LoRA, this fraction stays at chance level for every rank: as predicted, the low-rank constraint does not steer the update away from the channels that carry Task 1. SDGM sets this fraction to zero by construction, and reduces forgetting at matched transfer as can be seen in Fig. 9.

At the level of the representation, the theory explains the differences between methods through the overlaps of the student with the Task 1 teacher (Fig. 1c of the main text). Their deep analogue is the CKA similarity of the Task 1 features to their state at the first task switch, exposed in Fig. 10b. The ordering matches the theory: standard fine-tuning moves the Task 1 representation the most, LoRA less, and LoRA + SDGM the least. Together, these measurements connect the deep results to the mechanism identified by the theory: the rank alone does not determine which units the update modifies, and protecting the units used by Task 1 is what reduces forgetting.

Q1: We thank the reviewer for this question, which led us to substantially revise Appendix E. In the revised version, Task 2 is trained for α=8000\alpha = 8000, long enough for methods to reach the specialized phase, and all methods are compared at equal Task 2 performance rather than at a fixed training time, see Fig. 8. The results refine the interpretation of the main findings in three ways.

  • The central claim is confirmed, and SDGM becomes the only method that protects Task 1. Protecting the hidden units that carry Task 1 information through SDGM reduces forgetting at equal Task 2 performance. From Fig. 8b, at every level of Task 2 error, both SDGM variants forget less than standard fine-tuning and LoRA. The same holds over the whole range of task similarity when the transfer is matched across methods, see Fig. 8d-e.
  • The benefit of low rank alone is regime-dependent. In the unspecialized regime of the main text, vanilla LoRA forgets less than standard fine-tuning. In the specialized regime it forgets as much. The theory gives the reason: once each student unit is aligned with one teacher feature, Task 2 is learned by re-using these units (Lee et al., 2022). This requires an update of rank at most MM, which a rank-LL adapter with L≥ML \geq M can perform. The rank constraint is therefore not binding, and LoRA rewrites the Task 1 representation just as standard fine-tuning does. This is visible directly in the Task 1 overlaps shown in Fig. 8f. Under both standard fine-tuning and LoRA, the overlap of the specialized units with the Task 1 teacher drops from about 1 to about cc, the value expected once they are realigned with the Task 2 teacher, while SDGM keeps it unchanged. This sharpens the main message of the paper: what controls forgetting is not the rank of the update but its orientation relative to the features used by Task 1. Only the state-dependent mask protects Task 1 in both regimes.
  • The dependence on task similarity changes. In the unspecialized regime, forgetting peaks at intermediate similarity. In the specialized regime, at matched transfer, the forgetting of standard fine-tuning and LoRA decreases with cc, because the re-used units end up aligned with the Task 2 teacher, whose overlap with the Task 1 teacher is cc (Lee et al., 2022). As can be seen in Fig. 8d, with SDGM, forgetting remains well below standard fine-tuning and LoRA and nearly independent of cc.

The main additional cost of SDGM is speed. For LoRA + SDGM, the plastic units must first be recruited from scratch, which produces a long plateau before Task 2 is learned (Fig. 8a,c). We have updated Appendix E accordingly and added a sentence in Section 5.1 stating that the advantage of vanilla LoRA over standard fine-tuning holds in the unspecialized regime, while the advantage of SDGM holds in both.

Reviewer_PRgsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the theoretical analysis of LoRA update system dynamics in continual learning across two tasks. The authors constructed two interconnected teacher models to serve as data generators and trained them on student models with shared feature layers. The authors analyze how LoRA learns new knowledge at the feature level when representations are frozen. Under varying degrees of task similarity, they observe that moderate task similarity has a greater impact on the basic LoRA method. Based on this observation, they propose the SDGM method to reduce forgetting.

Critical strengths

The analysis and modeling of Continual LoRA presented in this paper are sound. The corresponding experimental results also support the validity of the theory.

This paper presents some novel observations and explanations, including the impact of task similarity on Continual LoRA. The SDGM proposed in this paper is also a simple yet practical method.

Critical weaknesses

The theoretical framework, which is based on two layers, two tasks, and online learning, still falls significantly short of true continual learning.

Although the experiments validated the theoretical results and the authors proposed a new algorithm, these experiments and the new algorithm have not been extended to more realistic datasets and scenarios.

Questions for authors

Could you design deep neural networks and conduct multi-task experiments to validate the performance and stability of SDGM?

I understand the challenges involved in the theoretical analysis of multi-task learning. My question is: If we want to generalize the current theoretical results to a general multi-task setting, what is the biggest challenge? Which of the existing observations and conclusions would still hold true?

1 comment
AuthorsOpenReview ↗

Response to Reviewer PRgs

We thank the reviewer for acknowledging the soundness, as well as the observations on continual LoRA made in the paper.

Q1: We thank the reviewer for this suggestion. We implemented it as the deep multi-task experiment described in the general answer: a ResNet-18 trained on six sequential tasks of Split CIFAR-100, with SDGM transposed to the output channels of each convolution. This benchmark also addresses the limitations raised in the weaknesses, since it involves a deep network, a sequence of six tasks, natural images, and minibatch training over several epochs rather than online learning.

Regarding performance, LoRA + SDGM forgets less than standard fine-tuning, standard + SDGM and vanilla LoRA when the methods are compared at matched transfer, averaged over all tasks of the sequence (Fig. 9b). In terms of accuracy (Table 1), it reaches the highest final average accuracy, on par with standard + SDGM, and the smallest backward transfer of the four methods, at the price of a slightly lower plasticity than standard fine-tuning, since fewer channels remain available for the new tasks.

Regarding stability, Fig. 10b tracks the similarity of the Task 1 features to their state at the first task switch along the whole sequence, measured with linear centered kernel alignment (CKA; Kornblith et al., 2019). CKA compares two representations of the same inputs and equals 1 when they coincide up to a rotation and a rescaling, so a decreasing CKA indicates that the Task 1 representation drifts. Under standard fine-tuning and vanilla LoRA, the drift accumulates with every new task. Under LoRA + SDGM, it slows down markedly after the first tasks, and the same plateau appears for standard + SDGM, which shows that it is caused by the mask rather than by the low-rank parameterization. The representation of the first task therefore remains more stable as more tasks are learned, and the results are consistent across seeds.

Q2: We see three levels at which the theoretical extension to T>2T > 2 sequential tasks can be discussed.

First, the closure of the theory carries over. If the adapter learned on task tt is merged into the frozen weights before task t+1t + 1 (or, equivalently, if a new adapter is added for each task while the previous ones are kept frozen), the frozen overlaps at each switch remain explicit functions of the order parameters of the previous stage. For T=O(1)T = O(1), the number of order parameters grows linearly in TT, and the system remains closed and deterministic in the high-dimensional limit. The cost is computational: the covariance matrix the theory computes has dimension 2K+TM+L2K + TM + L, and the number of four-point Gaussian integrals (which is the numerically most demanding part) grows accordingly.

Second, the space of task relations becomes much richer. With two tasks, task similarity is controlled by a single scalar cc. With TT tasks, it is controlled by a full similarity matrix and by the order in which tasks are presented, and the forgetting of a given task depends on all subsequent ones. Characterizing this space, and in particular how interference accumulates over long sequences, is still an open problem.

Third, capacity becomes a constraint. With a fixed number KK of hidden units, protecting units after every task progressively exhausts the plastic budget, so that beyond some TT, new tasks can only be learned by re-using protected units. This connects to the node re-use versus node activation picture of Lee et al. (2022), and to the optimal-protocol view of Mori et al. (2025).

The mechanisms we identify are properties of each task switch, so we expect them to hold at every stage of a longer sequence:

  • As long as the representation of previous tasks is not specialized, LoRA freezes its overlaps and reduces interference relative to standard fine-tuning, while the zero-initialized adapter slows early adaptation to each new task. Once the representation is specialized, LoRA re-uses the specialized units and forgets as much as standard fine-tuning (Appendix E).
  • Most importantly, rank alone does not control forgetting; the orientation of the update relative to previously used directions does. We expect this conclusion to become stronger with more tasks, as suggested by the deep benchmark, where the Task 1 drift accumulates under vanilla LoRA but not with a mask as shown in Fig. 10b of Appendix G. This is consistent with the observation that LoRA can forget more than standard fine-tuning over sequences of tasks (Shuttleworth et al., 2025). Multi-task LoRA methods such as InfLoRA (Liang and Li, 2024) and O-LoRA (Wang et al., 2023) address this problem by constraining the orientation of each new adapter, and have been shown experimentally to outperform vanilla LoRA. SDGM can be seen as a simplified, analytically tractable instance of this idea.
Reviewer_KeG7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies LoRA in a two-task teacher–student setting. The authors derive a set of ODEs for the first training stage and subsequent LoRA adaptation, then use the model to examine forgetting, task similarity, and adapter rank. They also study a masking rule (SDGM) that protects hidden units associated with Task 1. The theory agrees well with simulations, and a sequential MNIST experiment shows a similar trend.

Critical strengths

The paper provides a dynamical analysis of LoRA in continual learning by deriving a closed system of ODEs for a two-task teacher-student model. Unlike previous work focusing on a fixed pretrained state, this study follows both the initial learning stage and subsequent LoRA adaptation. The theoretical predictions closely match finite-dimensional simulations. The authors also investigate how adapter rank, task similarity, and state-dependent masking affect forgetting and transfer, with additional experiments on MNIST supporting some of the findings.

Critical weaknesses

W1 — Limited validation on real-world tasks. Most experiments are conducted using a simplified two-task teacher-student model. The only real-data experiment uses MNIST with two binary classification tasks (Figure 2b). The MNIST results provide some support for the theory, but experiments on deeper models and more than two sequential tasks are still missing. W2 — Insufficient comparison for SDGM. The proposed SDGM fixes the LoRA up-projection matrix while selecting which hidden units to protect. However, the paper does not provide a direct comparison with a fixed-matrix LoRA baseline using random masking. The inverse SDGM experiment in Appendix D.2 is useful, but it does not show how much of the improvement comes from selecting important features rather than fixing the LoRA matrix. W3 — The effect of adapter rank is not isolated. In Figure 3c, the authors set the number of frozen hidden units to κ = K − L. Consequently, increasing the adapter rank also reduces the number of protected units. This makes it difficult to determine whether the observed changes in forgetting come from the adapter rank itself or from the changing number of frozen units.

Questions for authors

Q1. Have the authors tested SDGM against random masking or a LoRA baseline with a fixed B matrix? Such a comparison would help explain where the improvement comes from. Q2. In Figure 3c, both the adapter rank L and the number of frozen units are changed. Can the authors provide re sults where only the rank is varied? Q3. The paper describes its ODE framework as “asymptotically exact.” Does this claim rely on a formal high-dimensional derivation, or is there a mathematical convergence proof?

2 comments
AuthorsOpenReview ↗

Response to Reviewer KeG7

We thank the reviewer for pointing out what we agree are the main strengths of our work: the theoretical dynamical study of both the initial learning stage, as well as the second one using LoRA. Below, we address all weaknesses (W) and questions (Q).

W1: We thank the reviewer for this suggestion. We added the deep multi-task benchmark described in the general answer, which addresses both points raised: a deeper model, a ResNet-18, and a longer sequence, six tasks of Split CIFAR-100, on natural images rather than MNIST. As detailed there, the ordering of the methods predicted by the theory is preserved when they are compared at matched transfer, both in the two-task protocol of the theory and over the whole six-task sequence (Fig. 2c in the main text, and Fig. 9). Table 1 reports the corresponding accuracies. The benchmark is summarized in Section 5.1 and described in full in Appendix G.

W2+Q1: Our inverse control is a worst case; following your proposal, we extended Fig. 4 of Appendix D.2. In the random control (LoRA + random SDGM), κ\kappa hidden units drawn uniformly at random are frozen. SDGM, inverse SDGM and random SDGM all fix B\mathbf{B} to a mask with the same number of frozen units and the same adapter rank, and differ only in which units are frozen. Comparing them with vanilla LoRA, where B\mathbf{B} is trained, separates the effect of fixing B\mathbf{B} from that of selecting the protected units:

  • Fixing B\mathbf{B} alone does not explain the gain. Inverse SDGM also fixes B\mathbf{B}, yet it forgets more than vanilla LoRA: freezing the units least important for Task 1 concentrates the update on the units that carry it.
  • The choice of the protected units does. Random SDGM forgets less than vanilla LoRA, and SDGM, which protects the units with the largest Task 1 readouts, reduces forgetting further.

All variants reach comparable Task 2 performance, so these differences in forgetting do not come at the cost of transfer.

W3 + Q2: We thank the reviewer for this remark. Varying LL in Fig. 3c indeed changes both the adapter rank and the number of frozen units. Our choice κ=K−L\kappa = K - L follows from the structure of SDGM. The up-projection matrix B\mathbf{B} has nonzero rows only on the K−κK - \kappa plastic units, so the update BA\mathbf{B}\mathbf{A} has rank at most min⁡(K−κ,L)\min(K - \kappa, L). Setting κ=K−L\kappa = K - L gives rank exactly LL, so every adapter direction is used, and vanilla LoRA and SDGM are compared at equal adapter rank. With κ\kappa fixed independently of LL, adapter directions beyond L=K−κL = K - \kappa add no rank to the update.

For completeness, we ran the rank sweep at fixed κ∈{2,5,8}\kappa \in \lbrace 2, 5, 8 \rbrace, both in the unspecialized regime of the main text (Fig. 6a) and in the specialized regime of Appendix E (Fig. 6b). In the unspecialized regime, increasing the number of protected units reduces forgetting, at little cost in transfer, for every rank LL. A large κ\kappa retains most of the transfer because the student has not yet specialized to the individual teacher features at the training horizon considered, and the Task 2 error it reaches can be attained with few plastic units.

In the specialized regime presented in Fig. 6b, representing Task 2 requires on the order of MM specialized units, and the effective rank min⁡(L,K−κ)\min(L, K - \kappa) of the update becomes the limiting factor. LoRA + SDGM escapes the Task 2 plateau only if min⁡(L,K−κ)≥M\min(L, K - \kappa) \geq M: for L<ML < M, or for κ=8\kappa = 8 at any rank, Task 2 stays on the plateau. In this regime, a large κ\kappa does limit transfer. In summary, κ\kappa sets the forgetting at a given transfer in both regimes, while in the specialized regime the effective rank min⁡(L,K−κ)\min(L, K - \kappa) also determines whether Task 2 can be learned at all. We added this analysis in Appendix D.4.

Q3: Our derivation follows the standard statistical-physics route and is not a formal convergence proof. A rigorous derivation for this class of models, two-layer networks trained by online SGD in the teacher-student setup, was given by Goldt et al. (2019). Using concentration arguments, they show that the order parameters concentrate on the solution of the ODEs, with fluctuations that vanish as N→∞N \to \infty. Our setting shares the ingredients of this proof. The LoRA factors A\mathbf{A} and B\mathbf{B} are updated with the same scalings as the first layer (η/N\eta/\sqrt{N}) and the readout (η/N\eta/N) of their model, and the generalization errors depend on the inputs only through a finite set of jointly Gaussian fields. We therefore expect the same argument to carry over, although the bilinear coupling between A\mathbf{A} and B\mathbf{B} requires checking its conditions explicitly, which we leave for future work. Consistently, finite-size simulations at N=103N = 10^3 closely match the theoretical predictions in all our figures.

Thank you for the clarification. My concerns have been largely addressed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General answer

We thank the three reviewers for their careful reading and constructive feedback. We first summarize the revisions that address points raised by multiple reviewers. In particular, we describe an additional experiment in a more realistic setting involving deep neural networks and more than two sequential tasks. The revised manuscript has been updated accordingly in several places.

Summary of revisions:

Each claim now states the setting in which it holds, in particular the distinction between the unspecialized regime of the main text and the specialized regime. Appendix E is rewritten: in the specialized regime, all methods are compared at matched Task 2 performance. Appendix D.2 now includes a random-selection control and vanilla LoRA, next to the inverse-selection control. Appendix D.4 (new) reports the rank sweep at a fixed number of frozen units, in both regimes. Appendix G (new) presents the deep multi-task experiment described below. In the main text, Fig. 2 now compares all methods at matched transfer by plotting Task 1 forgetting against Task 2 transfer for the theory (Fig. 2a), sequential MNIST (Fig. 2b; details in Appendix H), and the deep multi-task experiment (Fig. 2c). We modified the abstract accordingly.

Deep multi-task benchmark (Appendix G)

To test whether the mechanisms identified by the theory survive beyond the solvable model, we train a ResNet-18 on Split CIFAR-100 with six sequential tasks. Task 1 contains 50 classes and is learned from scratch with full SGD; each of the five following tasks contains 10 classes. As in the theory, the setting is task-incremental, with one readout head per task, so that forgetting is caused by changes in the shared layers alone. During Tasks 2 to 6, LoRA adapters of rank L=8L = 8 are placed on every 3×33\times3 convolution, with the same prefactor γ/L\gamma/\sqrt{L} as in the paper, and merged into the weights after each task. SDGM carries over directly: the hidden units of the theory become the output channels of each convolution, controlled by the rows of LoRA's up-projection matrix B\mathbf{B}, and the readout magnitude ∣hi†∣|\mathbf{h}^\dagger_i| is replaced by a first-order Taylor importance score computed on the task just learned (Molchanov et al., 2017). Half of the channels of each layer are protected after Task 1, and additional channels after each new task. We compare standard fine-tuning, standard + SDGM, LoRA and LoRA + SDGM, over a grid of learning rates with three seeds each.

In order to compare the different methods, we plot forgetting against transfer as defined in Eq.(77) of Appendix G along training and compare the methods at the largest transfer reached by all of them. A method therefore cannot appear more stable simply because it learns less.

First, the ordering predicted by the theory carries over: at matched transfer, standard fine-tuning forgets the most, standard + SDGM and LoRA forget less, and LoRA + SDGM forgets the least, both in the two-task protocol of the theory and averaged over the six-task sequence (Fig. 2c in the main text, and Fig. 9). Second, the similarity (linear CKA) of the Task 1 features to their state at the first task switch shows that, without a mask, the drift of the representation accumulates with each new task, whereas with a mask, in both standard + SDGM and LoRA + SDGM, it slows down markedly (Fig. 10b). Third, the fraction of the update that falls on the channels most important for Task 1 stays at chance level for vanilla LoRA at every rank, as predicted by the theory, and is zero for SDGM by construction (Fig. 10a).

This benchmark departs from the theoretical setting in several important respects, including network depth, convolutional structure, minibatch training over multiple epochs, and natural-image inputs. The fact that the ordering of the methods and the qualitative mechanisms identified above are preserved suggests that these effects are not specific to the solvable model. Given the use of a single architecture and three random seeds, however, we emphasize the consistency of the method ordering rather than the precise magnitude of the performance gaps.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.