acceptodds
Under review as a conference paper at ICLR 2027

Learning to Evolve: Enabling LLM Continual Learning via the Stable Anchor Dynamics

Abstract

Continual learning seeks to enable sequential mastery of a stream of tasks, simultaneously securing competent performance on the newly introduced task while preserving the proficiency acquired on earlier ones. In the contemporary era of large language models (LLMs), the performance degradation on old tasks, i.e., catastrophic forgetting, is partially mitigated by the inherently massive parameter scale and the extensive pre-training corpora. Nevertheless, due to the fundamental mechanics of gradient-based training, the acquisition of a new task invariably leaves an imprint on the parameter configuration, thereby exerting a nontrivial impact on previously established capabilities. In this paper, we probe into the very nature of forgetting and introduce the notion of Stable Anchor, which theoretically traces forgetting to the propagation of stable anchor changes from the new task to the old one (via the parameter space), ultimately altering its performance. Building upon this insight, we propose a loss function designed to minimize the influence transmitted via the stable anchor. Extensive experiments across diverse task streams substantiate the effectiveness of the proposed method.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM Continual Learning, Stable Anchor, Catastrophic Forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mbr5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the "classical" continual learning (CL) setting, where a model is trained on tasks introduced in incremental fashion and evaluates forward and backward transfer, here measured in ROUGE-L terms.

This paper proposes a way to mitigate forgetting that can be classified a s a functional regularization technique. The method works as follows:

  • estimate the set of top-k important head for each task on the base model before CL (needs data of all future tasks?) using causal mediation analysis from Todd et al. (2024); Jiang et al. (2025) (uses inverse prompt technique)
  • then after/during each task's learning calculate the stable anchor of current and each past task (requires storing some of the past tasks' data) --- function vector, which is the average (over examples from some support set of that task) representation of each important head for the last token
  • each new task is trained while minimizing the trace of the m x m matrices, where each is the product of two Jacobians: the current task's Jacobian of the function vector w.r.t. params and each previous task's Jacobian of its function vector w.r.t. current model's params

Now what it is supposed to do intuitively: Minimizing the m x m Jacobian product of task anchors w.r.t current parameters essentially incentivises the model to learn the new task by updating parameters in directions that does not change the old anchor. Applying this at each task can accumulate knowldge of the past tasks.

Critical strengths

I think the main strength of the paper is connecting continual learning to the idea of task vectors, which comes from a different ML domain. However, afaik this has already been done by the previous work, Jiang et al. (2025), and neither the connection to this previous work seem to be discussed, nor the method from it is used as a baseline. Without reading Jiang et al. (2025) it is hard for me to understand how noverl the approach presented in this work is.

My understanding of the hypothesis this paper builds on is that is that forgetting is linked to the change in some canonical representations (anchor point), to which extend this hypothesis is novel is hard to say without reading Jiang et al. (2025) and Todd et al. (2024), since discussion provided in this manuscript is not sufficient. Hence, trying to keep this canonical representations of past tasks stable is a logical idea for a method that regularizes the gradients updates on new tasks to be orthogonal to gradient updates that would change old tasks's anchors. One has to say that similar ideas but with anchor point being directly the loss on past tasks has been explored in works like GEM and A-GEM, both not mentioned in this manuscript.

Critical weaknesses

I see two fundamental weaknesses of this work, with further secondary weaknesses outlined in the questions section:

  1. Practicality of the addressed scenario: It seems to me that the setting this work addresses has rather trivial solution of: train adapter per task and retrieve it for inference or train on all data jointly. Hence, works addressing more sophisticated methods like the one presented should at least justify why these options are not available (data cannot be stored? but it is stored in this method; task ids are not available at inference? but can we do task id prediction? etc.) as well as have these options as baselines in the main results table. What seems more worth targeting in the current era is the problem of preserving general abilities like reasoning in CL setting (e.g. would be interesting to see numbers on some general benchmarks after the task stream has been learned), long stream continual learning under bounded compute constraints. None of this is discussed in this work.

  2. Novelty is unclear to me: how exactly does the method differ from Jiang et al. (2025) that already connects forgetting and function vectors?

Questions for authors

Related works & Motivation:

  • How does the method differ from Jiang et al. (2025)? Why the method from Jiang et al. (2025) is not a baseline?
  • How does proposed method relate to GEM, A-GEM and similar CL methods from the pre-LLM era?

Theory and algorithm:

  • What is the inverse prompt exactly? It is unclear how the semantically "inverted"model is obtained?
  • Proof Theorem 1: the meaning of h_lk → ĥ_lk notation is somewhat confusing to me; in line 220 it meant that one single head was injected however in the proof it looks like its denoting top-k heads replaced being replaced
  • Phase 1 precomputes important heads, which requires access to the full task stream before continual learning even starts? This seem to contradict any reasonable practical CL situation

Experiments:

  • I am not really convinced that better numbers on the CL metrics (like less forgetting) is an artifact of real forgetting reduction and not an artifact of better per task learning of the proposed method as compared to the baselines (for reasons such as better hp tuning, model selection or maybe just regularizing effect of the proposed aux. loss). For this reason it would be very important to also report absolute task performance numbers (each method's performance on each task on all three task streams right after the task was learned, not only on the mixed stream as in Fig. 4)
  • additionally, I would like to see the classical CL baselines like join training (is it the SFT baselien) on all data and a separate LORa per task (i.e. one LoRa per task + oracle retrieval)
  • how many training and test samples per task and how is is model selection performed?
  • it is unclear to me what the Vanilla and SFT baselines are (SFT is likely the joint baseline)
  • what exactly is the replay baseline? how does the compute spent on the replay baseline compare to the compute spent on the proposed method? (there seem to be a tradeoff between compute and umber sample required for the proposed method and replay)
  • Table 1 shows negative F.Ra, which should be impossible according t the definition?
  • Table 3: I am assuming that the blocks per model denote task steams? Ni-Seq-C being the upper one and Ni-Seg-M the lower? But then task 618 from Figure 4, that is supposed to show Ni-Seg-M stream, does not show up in the Ni-Seg-M stream from Table 3
  • also negative F.Ra in Table 3
8 comments
AuthorsOpenReview ↗

discussion 1

Thank you for your careful review. We address your concerns below.

Weakness 1. Unified-model continual learning

Our goal is to retain and accumulate knowledge in one shared model, merging each stage's LoRA module into it. Separate adapters preserve task-specific solutions and require inference-time selection, while joint training assumes simultaneous access to all task data. Both are useful references, but address different settings.

Stable Anchors are computed from historical task data to identify representations with the strongest causal contribution to stable task outputs. We acknowledge that our approach is better suited to optimizing the overall training process when the task stream is known in advance. Its effectiveness when training continues on randomly arriving tasks remains to be verified.

Weakness 2 & Question 1. Relation to Function Vectors and other Baselines

We build on Todd et al. (2024) and Jiang et al. (2025). Restoring old-task Function Vector activations can partially recover task performance; this prior observation motivates our approximate Stable Anchors. Our contribution is to analyze cross-task representation changes and regularize their Jacobian coupling during training, rather than introduce Function Vectors or their connection to forgetting.

GEM / A-GEM constrain updates through old-task loss gradients; we target cross-task representation Jacobian coupling.

Comparison with the FV baseline (Jiang et al., 2025)

Thank you for suggesting this comparison. We now include the FV replacement baseline below. Performance is measured by ROUGE-L (%), rounded to two decimal places. Bold BWT and F.Ra values at the end of each stream are averages over its first four tasks, not metrics for the last task.

ModelTask NameStep 1Step 2Step 3Step 4Step 5BWTF.Ra
Qwen3-8BNI-Seq-C
task19589.0071.2068.5275.9576.4712.5312.53
task1343—97.5598.7998.9097.85-0.301.05
task1310——61.0950.4452.168.938.93
task1292———81.0181.50-0.49-0.49
task363————86.67BWT=5.17F.Ra=5.51
NI-Seq-G
task61842.0222.1222.9123.0123.3918.6318.63
task1290—33.1417.5818.9419.9113.2313.23
task589——11.919.8910.391.521.52
task511———18.9916.352.642.64
task1357————27.24BWT=9.01F.Ra=9.01
NI-Seq-M
task36012.3710.3610.5610.419.862.512.51
task363—88.0175.2886.0287.180.830.83
task1290——30.9717.2618.5812.3912.39
task339———61.9244.1317.7917.79
task1510————99.67BWT=8.38F.Ra=8.38
Llama3.1-8BNI-Seq-C
task19588.1081.2786.4683.0882.535.575.57
task1343—98.4898.1997.2997.161.321.32
task1310——61.2355.7057.463.773.77
task1292———81.2980.061.231.23
task363————88.35BWT=2.97F.Ra=2.97
NI-Seq-G
task61844.8530.3929.5926.1820.9823.8723.87
task1290—33.1724.5523.7921.6811.4911.49
task589——14.9012.0010.864.044.04
task511———20.5919.461.131.13
task1357————28.77BWT=10.13F.Ra=10.13
NI-Seq-M
task36012.4910.089.969.589.133.363.36
task363—89.1080.2488.2388.770.330.33
task1290——30.9621.1920.3310.6310.63
task339———64.0353.5110.5210.52
task1510————99.69BWT=6.21F.Ra=6.21

AA (mean final-checkpoint score across all five tasks), in NI-Seq-C / NI-Seq-G / NI-Seq-M order: Qwen3-8B: 78.93 / 19.46 / 51.88; Llama3.1-8B: 81.11 / 20.35 / 54.29.

Overall, the FV baseline performs between O-LoRA and our method, although the ranking varies across individual task streams and metrics.

AuthorsOpenReview ↗

discussion 2

Theory and algorithm 1.

The Inverse Prompt is an input instruction, not a separately trained or semantically inverted model. Head selection uses the same model throughout:

  1. Estimate each candidate head's mean activation from standard task-support samples.
  2. Apply the inverse prompt to perturb task behavior.
  3. Replace one candidate head's activation with its mean activation from Step 1 and measure the change in correct-answer probability relative to Step 2.

This change measures the candidate head's causal contribution. We will provide the prompt templates and input-output examples.

Theory and algorithm 2.

During candidate evaluation, hlkh_{lk} denotes the activation of one head kk in layer ll, and h^lk\hat{h}_{lk} denotes its mean activation estimated from the standard support samples. The intervention hlk→h^lkh_{lk}\to\hat{h}_{lk} replaces that single head, not all top-kk heads. After ranking the candidates, the selected heads collectively define the task representation. We will use separate notation for this representation and any joint intervention.

Theorem 1 should be read as an approximate motivation, not a rigorous invariance guarantee. We decompose generation into representation acquisition RR and downstream generation GG. Partial recovery after restoring old-task Function Vector activations motivates stabilizing RR, but does not prove invariance under arbitrary changes to GG's parameters. We will distinguish this empirical motivation from the ideal Stable Anchor definition.

Theory and algorithmc3.

Algorithm 1 places head selection for all tasks before continual training. Although head selection is defined per task and could in principle occur upon task arrival, the reported results do not establish this online variant. We will clarify the actual execution order and information-access assumptions, explicitly disclosing any advance access to future-task support sets.

AuthorsOpenReview ↗

discussion 3

Experiments.1. Per-task performance across all three streams

Qwen3-8B / NI-Seq-C (Classification; ROUGE-L)

MethodTasktask195task1343task1310task1292task363BWT_iF.Ra_i
vanillatask19585.3885.0785.0785.3384.151.231.23
task1343—99.6999.6999.6199.77-0.08-0.08
task1310——61.5457.8560.610.930.93
task1292———78.5477.920.620.62
task363————87.38BWT=0.68F.Ra=0.68
Replaytask19586.6285.0883.5483.2382.923.693.69
task1343—99.6999.6999.6999.620.080.08
task1310——64.4667.2366.00-1.541.23
task1292———80.2380.000.230.23
task363————90.3BWT=0.62F.Ra=1.31
O-LoRAtask19585.6981.0862.4645.2150.4935.2035.20
task1343—99.0097.5876.3488.2310.7710.77
task1310——61.8539.8557.444.414.41
task1292———77.8574.813.043.04
task363————88.77BWT=13.36F.Ra=13.36
Tree-LoRAtask19585.6977.6971.6952.8759.1826.5126.51
task1343—99.3897.5473.8381.9417.4417.44
task1310——56.0040.5151.494.514.51
task1292———78.6974.484.214.21
task363————88.31BWT=13.17F.Ra=13.17
Ablationtask19586.6185.8486.0085.8486.92-0.31-0.31
task1343—99.3099.3099.3099.230.070.07
task1310——60.2551.1753.077.187.18
task1292———78.4373.035.405.40
task363————86.76BWT=3.09F.Ra=3.09
Ourtask19586.6187.3887.2387.0786.92-0.310.46
task1343—99.0099.0799.1599.000.000.15
task1310——56.2557.3357.17-0.920.16
task1292———72.2573.03-0.78-0.78
task363————87.07BWT=-0.50F.Ra=0.00

Qwen3-8B / NI-Seq-G (Generation; ROUGE-L)

MethodTasktask618task1290task589task511task1357BWT_iF.Ra_i
vanillatask61833.3716.218.7612.4216.8716.5016.50
task1290—22.8721.5121.8322.110.760.76
task589——6.326.755.620.701.13
task511———17.0617.020.040.04
task1357————22.65BWT=4.50F.Ra=4.61
Replaytask61844.6740.2616.5122.5226.1218.5518.55
task1290—29.0629.0929.0128.870.190.22
task589——14.1312.0912.311.831.83
task511———21.8922.11-0.22-0.22
task1357————28.43BWT=5.09F.Ra=5.09
O-LoRAtask61829.0813.458.1212.2812.0817.0017.00
task1290—25.6519.8320.9625.290.360.36
task589——4.294.725.14-0.85-0.42
task511———17.5315.901.631.63
task1357————24.63BWT=4.54F.Ra=4.64
Tree-LoRAtask61827.2811.097.469.6812.1815.1015.10
task1290—25.0117.9121.4525.69-0.68-0.68
task589——4.225.565.00-0.780.56
task511———16.1117.64-1.53-1.53
task1357————24.10BWT=3.03F.Ra=3.36
Ablationtask61844.9242.2438.5633.2234.8110.1110.11
task1290—23.2223.0723.0424.14-0.92-0.92
task589——11.809.477.724.084.08
task511———20.4620.420.040.04
task1357————23.46BWT=3.33F.Ra=3.33
Ourtask61845.0940.6030.3939.2335.729.379.37
task1290—25.6025.0124.7125.540.060.06
task589——10.739.158.002.732.73
task511———19.7319.630.100.10
task1357————25.74BWT=3.07F.Ra=3.07
Reviewer_o5crOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper discusses the limitations of utilizing independent and discrete parameter spaces in current continual learning methods, and thus studies forgetting formulation, then proposes the stable anchor, whose invariance reflects final model output invariance. This paper establishes the reciprocal perturbation between stable anchors of tasks and proposes a loss function to mitigate cross-task interference, reducing forgetting in continual learning. The experimental results across different task sequences show its effectiveness.

Critical strengths

The paper proposes a novel perspective for continual learning via the stable anchor instead of the parameter space, analyzes the propagation of the stable anchor between tasks, and shows that it mitigates forgetting in continual learning. Because current existing continual learning methods mostly focus on the parameter space, this paper gives a new perspective to address the forgetting issue in continual learning.

Critical weaknesses
  1. Experimental results lack enough evidence to show why the stable anchor works, such as no computational comparison and no enough ablation studies. For example, there is no hyperparameter kk sweeping to show the performance of the stable anchor under different kk.

  2. The proof of Theorem 1 is not solid for me. When reducing the contribution of both R\mathcal{R} and G\mathcal{G} to only R\mathcal{R}, the proof uses “Owing to the diversity of forms that w may assume” to support the conclusion, but this statement does not provide any other support reference or experimental results shown in the paper, making the proof weaker.

Questions for authors

Major:

  1. The paper defines the forward process as y=G(R(x))y=\mathcal{G}(\mathcal{R}(x)) in line 144 and Theorem 1 gives the conclusion that the output variation is governed by R\mathcal{R}-increment, but it’s not clear to me, in each base model used in the experiments, what functions or modules are defined as R\mathcal{R} and G\mathcal{G}.

  2. The experiments do not show the computation cost, such as GPU memory usage, compared to other baselines. Can authors compare this result to show the performance of stable anchor?

  3. The ablation study also lacks enough analysis on the stable anchor. For example, Algorithm 1 uses top-kk selection in line 14, and here kk is a hyperparameter, so it’s necessary to sweep the value of kk to see the performance of stable anchor under different kk.

Minor:

  1. For the experimental metric BWT, by the traditional definition of BWT [1], a higher score is better, where a positive score means the previous tasks are improved after learning new tasks. But this paper defined an opposite version of BWT, so I suggested the authors follow the traditional definition to avoid confusion.

    [1] “Gradient Episodic Memory for Continual Learning”, Advances in Neural Information Processing Systems 30 (NIPS 2017).

  2. Can authors refine Table 3 in the Appendix? For example, “Task Name” is out of the column space?

  3. Can authors modify Eq.(4) since it inappropriately separates a gradient using the chain rule into two lines?

3 comments

Thank authors for taking the time to respond to my questions. Based on the response, I decide to keep my original score.

We thank the reviewer for the further reply. However, we believe that your initial concerns are all concrete and verifiable. We are confident that our added content should response your concerns. Since your current reply does not seem to provide additional information, we do not understand whether there is an extra concern or something unreasonable in our response. We would be grateful if you could point it out so that we can further improve this work. Thank you very much.

Thanks for your reply. There is still no clear computational comparison in the added experiments. The response in major 2 only mentions "average memory usage for the baseline method is about 52 GB (65%) with batch size 32.", but this is not clear to compare each baseline method. Also, the method targets LoRA continual learning, but the baselines in the experiments only pick two LoRA continual learning baselines. Thus, the experimental results are not strong enough to show competitiveness in LoRA continual learning. Overall, I would like to keep my score unchanged.

Reviewer_oxPvOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes stable anchors as intermediate representations intended to isolate the source of forgetting. Function vectors instantiate these anchors, and a regularizer penalizes interactions between old- and new-task representation Jacobians. Experiments evaluate two 8B model families on classification, generation and mixed SuperNI sequences.

The introduction, method and conclusion consistently pursue a mechanistic explanation of forgetting. However, the claimed theoretical explanation is substantially stronger than the argument establishes.

Critical strengths

The paper connects representation-level analysis to an implementable continual-learning objective. Function vectors are a sensible object to investigate because selected heads have a measurable effect on task behavior.

Results are promising in several settings, particularly the mixed streams. The causal-head-versus-random-head ablation supports the usefulness of the selected representations. Testing both Qwen3-8B and Llama3.1-8B is helpful. The paper also acknowledges second-order derivative costs and reports a training-time comparison.

Critical weaknesses
  1. Definition (2) requires output invariance across arbitrary downstream parameter configurations. The proof of Theorem 1 instead discusses interventions under different prompts and contexts. Prompt robustness does not establish invariance to the downstream parameter vector. Selecting heads with high causal effects also does not establish that the omitted G-increment is negligible during fine-tuning.

  2. Theorem 2 promotes a sufficient interference-control strategy to an equivalence: a nonzero cross-Jacobian can produce zero drift for a particular loss-gradient direction. Subsequently, minimizing a lower bound does not generally minimize the Frobenius norm being bounded. Appendix A introduces a diagonal-dominance assumption, but does not establish it for the cross-task Jacobians. The matrix absolute-value notation in Equation (6) also needs clarification.

  3. The stated maximization includes the final checkpoint, so F.Ra cannot be negative; Table 3 nevertheless reports −0.78 for Qwen task1292. Tables 1 and 2 also report different Llama mixed-stream F.Ra values, 2.23 and 1.79. These discrepancies weaken confidence in the retention conclusions. Repeated-seed uncertainty and the old-task support/cache budget are insufficiently specified.

Questions for authors
  1. Can the authors replace Theorem 1’s prompt-based argument with a parameter-perturbation statement and directly measure the supposedly negligible G-increment?

  2. What assumptions make the trace surrogate control the complete cross-Jacobian norm, and do measured Jacobians satisfy them?

  3. Can the authors reconcile the metric definitions and tables, and provide evaluation code, seed results and support/cache accounting for a reproducible comparision?

Comments on the paper

1 comment
AuthorsOpenReview ↗

rebuttal summary

We would like to provide a complete summary of our rebuttal as the final response. We sincerely thank the reviewers and chairs for their comments and efforts.

Theoretical part

  1. Reviewers oxPv, o5cr, and mbr5 all questioned the rigor of Theorem 1. The motivation is grounded in the function-vector recovery phenomenon (line 222): restoring a task's function-vector activations after subsequent training partially restores that task's performance — a phenomenon validated in Jiang et al. (ICLR 2025) and indirectly supported by our continual-learning experiments. Accordingly, for any two new training processes i and j, G(Rfv,ϕi)≈G(Rfv,ϕj)G(R_{fv},\phi_i) \approx G(R_{fv}, \phi_j) approximately satisfies the invariance requirement in Definition 2.

  2. Regarding the equivalence in Theorem 2 raised by Reviewer oxPv: a nonzero cross-Jacobian can indeed produce zero drift along a single gradient direction. Our intended claim is uniform interference control — zero drift for all unit directions, which is equivalent to $|J_cross_F| = $|. The minimized quantity is the spectral-coherence component of the Frobenius norm: for the rank-one cross-gradient matrix A=gigjTA = g_i g_j^T, it equals ∣⟨gi,gj⟩∣|⟨g_i, g_j⟩|, so minimizing the bound directly penalizes cross-sample gradient coherence under the gradient-magnitude control imposed by the main objective.

  3. Regarding the information-access assumption raised by Reviewer mbr5 (head selection before continual learning): we acknowledged that our approach is best suited to optimizing the overall training process when the task stream is known in advance.

Experimental issues

  1. To address Reviewer o5cr's concern that the paper lacks evidence for why the stable anchor works, we added the requested k sweep (k = 2, 5, 10, 15, 20, 40 across NI-Seq-C/G/M). Performance first improves and then degrades as k grows, with the optimum near the value used in the paper — consistent with the design intent that k captures the primary storage locations of old-task knowledge.

  2. Regarding novelty, we clarified that our contribution is not function vectors themselves, but the analysis of cross-task representation interference and its regularization through Jacobian coupling — orthogonal to GEM/A-GEM (old-task loss-gradient constraints) and to the replacement-based use of function vectors in Jiang et al. (2025). As suggested by Reviewer mbr5, we added the FV-replacement baseline across all six model–stream combinations (AA on NI-Seq-C/G/M: Qwen3-8B 78.93/19.46/51.88; Llama3.1-8B 81.11/20.35/54.29); it performs between O-LoRA and our method.

  3. Regarding the metric inconsistencies raised by Reviewers oxPv and mbr5: they stemmed from an inconsistent averaging convention. The correct value for Llama3.1-8B / NI-Seq-M is 2.23 (Table 1 is correct; the 1.79 in Table 2 will be corrected). We clarified that our BWT uses the opposite sign convention to the traditional definition (lower is better; positive indicates forgetting), that F.Ra averages over the first N−1 tasks with the historical maximum excluding the final checkpoint (a negative value means the final score exceeds all earlier post-learning scores), and that Figure 4 corresponds to NI-Seq-G.

  4. Regarding absolute task performance and comparability, we provided complete per-task results for both models across all three streams and all methods, together with the checkpoint-selection protocol applied uniformly to every method. We acknowledge that reporting the best current-task checkpoint is not ideal practice; the revision will additionally report final-checkpoint results, and we will clarify the exact training configurations of each method and discuss the remaining first-task differences.

  5. Regarding computational cost raised by Reviewer o5cr: we reported memory usage (baseline ≈52 GB at batch size 32; ours ≈64 GB at batch size 4, reflecting the second-order regularizer), and the paper already reports the training-time comparison.

Finally, we would like to state that our primary motivation is to understand where forgetting comes from and how to control it at the representation level: forgetting is traced to the propagation of stable-anchor changes from new tasks to old ones through the parameter space, and the proposed loss minimizes the influence transmitted through this channel. This differs from methods that constrain updates in the parameter space via old-task loss gradients and from replay-based methods. Our theory makes two contributions. First, it reformulates forgetting in a shared representation space centered on function vectors, connecting causal-mediation analysis to continual learning. Second, and most importantly, it converts interference control into a tractable objective — minimizing the spectral coherence of cross-task gradients — with a concrete implementation (Algorithm 1) and extensive experimental validation on two model families and three stream types.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.