- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses the "classical" continual learning (CL) setting, where a model is trained on tasks introduced in incremental fashion and evaluates forward and backward transfer, here measured in ROUGE-L terms.
This paper proposes a way to mitigate forgetting that can be classified a s a functional regularization technique. The method works as follows:
- estimate the set of top-k important head for each task on the base model before CL (needs data of all future tasks?) using causal mediation analysis from Todd et al. (2024); Jiang et al. (2025) (uses inverse prompt technique)
- then after/during each task's learning calculate the stable anchor of current and each past task (requires storing some of the past tasks' data) --- function vector, which is the average (over examples from some support set of that task) representation of each important head for the last token
- each new task is trained while minimizing the trace of the m x m matrices, where each is the product of two Jacobians: the current task's Jacobian of the function vector w.r.t. params and each previous task's Jacobian of its function vector w.r.t. current model's params
Now what it is supposed to do intuitively: Minimizing the m x m Jacobian product of task anchors w.r.t current parameters essentially incentivises the model to learn the new task by updating parameters in directions that does not change the old anchor. Applying this at each task can accumulate knowldge of the past tasks.
Critical strengths
I think the main strength of the paper is connecting continual learning to the idea of task vectors, which comes from a different ML domain. However, afaik this has already been done by the previous work, Jiang et al. (2025), and neither the connection to this previous work seem to be discussed, nor the method from it is used as a baseline. Without reading Jiang et al. (2025) it is hard for me to understand how noverl the approach presented in this work is.
My understanding of the hypothesis this paper builds on is that is that forgetting is linked to the change in some canonical representations (anchor point), to which extend this hypothesis is novel is hard to say without reading Jiang et al. (2025) and Todd et al. (2024), since discussion provided in this manuscript is not sufficient. Hence, trying to keep this canonical representations of past tasks stable is a logical idea for a method that regularizes the gradients updates on new tasks to be orthogonal to gradient updates that would change old tasks's anchors. One has to say that similar ideas but with anchor point being directly the loss on past tasks has been explored in works like GEM and A-GEM, both not mentioned in this manuscript.
Critical weaknesses
I see two fundamental weaknesses of this work, with further secondary weaknesses outlined in the questions section:
-
Practicality of the addressed scenario: It seems to me that the setting this work addresses has rather trivial solution of: train adapter per task and retrieve it for inference or train on all data jointly. Hence, works addressing more sophisticated methods like the one presented should at least justify why these options are not available (data cannot be stored? but it is stored in this method; task ids are not available at inference? but can we do task id prediction? etc.) as well as have these options as baselines in the main results table. What seems more worth targeting in the current era is the problem of preserving general abilities like reasoning in CL setting (e.g. would be interesting to see numbers on some general benchmarks after the task stream has been learned), long stream continual learning under bounded compute constraints. None of this is discussed in this work.
-
Novelty is unclear to me: how exactly does the method differ from Jiang et al. (2025) that already connects forgetting and function vectors?
Questions for authors
Related works & Motivation:
- How does the method differ from Jiang et al. (2025)? Why the method from Jiang et al. (2025) is not a baseline?
- How does proposed method relate to GEM, A-GEM and similar CL methods from the pre-LLM era?
Theory and algorithm:
- What is the inverse prompt exactly? It is unclear how the semantically "inverted"model is obtained?
- Proof Theorem 1: the meaning of h_lk → ĥ_lk notation is somewhat confusing to me; in line 220 it meant that one single head was injected however in the proof it looks like its denoting top-k heads replaced being replaced
- Phase 1 precomputes important heads, which requires access to the full task stream before continual learning even starts? This seem to contradict any reasonable practical CL situation
Experiments:
- I am not really convinced that better numbers on the CL metrics (like less forgetting) is an artifact of real forgetting reduction and not an artifact of better per task learning of the proposed method as compared to the baselines (for reasons such as better hp tuning, model selection or maybe just regularizing effect of the proposed aux. loss). For this reason it would be very important to also report absolute task performance numbers (each method's performance on each task on all three task streams right after the task was learned, not only on the mixed stream as in Fig. 4)
- additionally, I would like to see the classical CL baselines like join training (is it the SFT baselien) on all data and a separate LORa per task (i.e. one LoRa per task + oracle retrieval)
- how many training and test samples per task and how is is model selection performed?
- it is unclear to me what the Vanilla and SFT baselines are (SFT is likely the joint baseline)
- what exactly is the replay baseline? how does the compute spent on the replay baseline compare to the compute spent on the proposed method? (there seem to be a tradeoff between compute and umber sample required for the proposed method and replay)
- Table 1 shows negative F.Ra, which should be impossible according t the definition?
- Table 3: I am assuming that the blocks per model denote task steams? Ni-Seq-C being the upper one and Ni-Seg-M the lower? But then task 618 from Figure 4, that is supposed to show Ni-Seg-M stream, does not show up in the Ni-Seg-M stream from Table 3
- also negative F.Ra in Table 3
8 comments
discussion 2
Theory and algorithm 1.
The Inverse Prompt is an input instruction, not a separately trained or semantically inverted model. Head selection uses the same model throughout:
- Estimate each candidate head's mean activation from standard task-support samples.
- Apply the inverse prompt to perturb task behavior.
- Replace one candidate head's activation with its mean activation from Step 1 and measure the change in correct-answer probability relative to Step 2.
This change measures the candidate head's causal contribution. We will provide the prompt templates and input-output examples.
Theory and algorithm 2.
During candidate evaluation, denotes the activation of one head in layer , and denotes its mean activation estimated from the standard support samples. The intervention replaces that single head, not all top- heads. After ranking the candidates, the selected heads collectively define the task representation. We will use separate notation for this representation and any joint intervention.
Theorem 1 should be read as an approximate motivation, not a rigorous invariance guarantee. We decompose generation into representation acquisition and downstream generation . Partial recovery after restoring old-task Function Vector activations motivates stabilizing , but does not prove invariance under arbitrary changes to 's parameters. We will distinguish this empirical motivation from the ideal Stable Anchor definition.
Theory and algorithmc3.
Algorithm 1 places head selection for all tasks before continual training. Although head selection is defined per task and could in principle occur upon task arrival, the reported results do not establish this online variant. We will clarify the actual execution order and information-access assumptions, explicitly disclosing any advance access to future-task support sets.
discussion 3
Experiments.1. Per-task performance across all three streams
Qwen3-8B / NI-Seq-C (Classification; ROUGE-L)
| Method | Task | task195 | task1343 | task1310 | task1292 | task363 | BWT_i | F.Ra_i |
|---|---|---|---|---|---|---|---|---|
| vanilla | task195 | 85.38 | 85.07 | 85.07 | 85.33 | 84.15 | 1.23 | 1.23 |
| task1343 | — | 99.69 | 99.69 | 99.61 | 99.77 | -0.08 | -0.08 | |
| task1310 | — | — | 61.54 | 57.85 | 60.61 | 0.93 | 0.93 | |
| task1292 | — | — | — | 78.54 | 77.92 | 0.62 | 0.62 | |
| task363 | — | — | — | — | 87.38 | BWT=0.68 | F.Ra=0.68 | |
| Replay | task195 | 86.62 | 85.08 | 83.54 | 83.23 | 82.92 | 3.69 | 3.69 |
| task1343 | — | 99.69 | 99.69 | 99.69 | 99.62 | 0.08 | 0.08 | |
| task1310 | — | — | 64.46 | 67.23 | 66.00 | -1.54 | 1.23 | |
| task1292 | — | — | — | 80.23 | 80.00 | 0.23 | 0.23 | |
| task363 | — | — | — | — | 90.3 | BWT=0.62 | F.Ra=1.31 | |
| O-LoRA | task195 | 85.69 | 81.08 | 62.46 | 45.21 | 50.49 | 35.20 | 35.20 |
| task1343 | — | 99.00 | 97.58 | 76.34 | 88.23 | 10.77 | 10.77 | |
| task1310 | — | — | 61.85 | 39.85 | 57.44 | 4.41 | 4.41 | |
| task1292 | — | — | — | 77.85 | 74.81 | 3.04 | 3.04 | |
| task363 | — | — | — | — | 88.77 | BWT=13.36 | F.Ra=13.36 | |
| Tree-LoRA | task195 | 85.69 | 77.69 | 71.69 | 52.87 | 59.18 | 26.51 | 26.51 |
| task1343 | — | 99.38 | 97.54 | 73.83 | 81.94 | 17.44 | 17.44 | |
| task1310 | — | — | 56.00 | 40.51 | 51.49 | 4.51 | 4.51 | |
| task1292 | — | — | — | 78.69 | 74.48 | 4.21 | 4.21 | |
| task363 | — | — | — | — | 88.31 | BWT=13.17 | F.Ra=13.17 | |
| Ablation | task195 | 86.61 | 85.84 | 86.00 | 85.84 | 86.92 | -0.31 | -0.31 |
| task1343 | — | 99.30 | 99.30 | 99.30 | 99.23 | 0.07 | 0.07 | |
| task1310 | — | — | 60.25 | 51.17 | 53.07 | 7.18 | 7.18 | |
| task1292 | — | — | — | 78.43 | 73.03 | 5.40 | 5.40 | |
| task363 | — | — | — | — | 86.76 | BWT=3.09 | F.Ra=3.09 | |
| Our | task195 | 86.61 | 87.38 | 87.23 | 87.07 | 86.92 | -0.31 | 0.46 |
| task1343 | — | 99.00 | 99.07 | 99.15 | 99.00 | 0.00 | 0.15 | |
| task1310 | — | — | 56.25 | 57.33 | 57.17 | -0.92 | 0.16 | |
| task1292 | — | — | — | 72.25 | 73.03 | -0.78 | -0.78 | |
| task363 | — | — | — | — | 87.07 | BWT=-0.50 | F.Ra=0.00 |
Qwen3-8B / NI-Seq-G (Generation; ROUGE-L)
| Method | Task | task618 | task1290 | task589 | task511 | task1357 | BWT_i | F.Ra_i |
|---|---|---|---|---|---|---|---|---|
| vanilla | task618 | 33.37 | 16.21 | 8.76 | 12.42 | 16.87 | 16.50 | 16.50 |
| task1290 | — | 22.87 | 21.51 | 21.83 | 22.11 | 0.76 | 0.76 | |
| task589 | — | — | 6.32 | 6.75 | 5.62 | 0.70 | 1.13 | |
| task511 | — | — | — | 17.06 | 17.02 | 0.04 | 0.04 | |
| task1357 | — | — | — | — | 22.65 | BWT=4.50 | F.Ra=4.61 | |
| Replay | task618 | 44.67 | 40.26 | 16.51 | 22.52 | 26.12 | 18.55 | 18.55 |
| task1290 | — | 29.06 | 29.09 | 29.01 | 28.87 | 0.19 | 0.22 | |
| task589 | — | — | 14.13 | 12.09 | 12.31 | 1.83 | 1.83 | |
| task511 | — | — | — | 21.89 | 22.11 | -0.22 | -0.22 | |
| task1357 | — | — | — | — | 28.43 | BWT=5.09 | F.Ra=5.09 | |
| O-LoRA | task618 | 29.08 | 13.45 | 8.12 | 12.28 | 12.08 | 17.00 | 17.00 |
| task1290 | — | 25.65 | 19.83 | 20.96 | 25.29 | 0.36 | 0.36 | |
| task589 | — | — | 4.29 | 4.72 | 5.14 | -0.85 | -0.42 | |
| task511 | — | — | — | 17.53 | 15.90 | 1.63 | 1.63 | |
| task1357 | — | — | — | — | 24.63 | BWT=4.54 | F.Ra=4.64 | |
| Tree-LoRA | task618 | 27.28 | 11.09 | 7.46 | 9.68 | 12.18 | 15.10 | 15.10 |
| task1290 | — | 25.01 | 17.91 | 21.45 | 25.69 | -0.68 | -0.68 | |
| task589 | — | — | 4.22 | 5.56 | 5.00 | -0.78 | 0.56 | |
| task511 | — | — | — | 16.11 | 17.64 | -1.53 | -1.53 | |
| task1357 | — | — | — | — | 24.10 | BWT=3.03 | F.Ra=3.36 | |
| Ablation | task618 | 44.92 | 42.24 | 38.56 | 33.22 | 34.81 | 10.11 | 10.11 |
| task1290 | — | 23.22 | 23.07 | 23.04 | 24.14 | -0.92 | -0.92 | |
| task589 | — | — | 11.80 | 9.47 | 7.72 | 4.08 | 4.08 | |
| task511 | — | — | — | 20.46 | 20.42 | 0.04 | 0.04 | |
| task1357 | — | — | — | — | 23.46 | BWT=3.33 | F.Ra=3.33 | |
| Our | task618 | 45.09 | 40.60 | 30.39 | 39.23 | 35.72 | 9.37 | 9.37 |
| task1290 | — | 25.60 | 25.01 | 24.71 | 25.54 | 0.06 | 0.06 | |
| task589 | — | — | 10.73 | 9.15 | 8.00 | 2.73 | 2.73 | |
| task511 | — | — | — | 19.73 | 19.63 | 0.10 | 0.10 | |
| task1357 | — | — | — | — | 25.74 | BWT=3.07 | F.Ra=3.07 |
discussion 1
Thank you for your careful review. We address your concerns below.
Weakness 1. Unified-model continual learning
Our goal is to retain and accumulate knowledge in one shared model, merging each stage's LoRA module into it. Separate adapters preserve task-specific solutions and require inference-time selection, while joint training assumes simultaneous access to all task data. Both are useful references, but address different settings.
Stable Anchors are computed from historical task data to identify representations with the strongest causal contribution to stable task outputs. We acknowledge that our approach is better suited to optimizing the overall training process when the task stream is known in advance. Its effectiveness when training continues on randomly arriving tasks remains to be verified.
Weakness 2 & Question 1. Relation to Function Vectors and other Baselines
We build on Todd et al. (2024) and Jiang et al. (2025). Restoring old-task Function Vector activations can partially recover task performance; this prior observation motivates our approximate Stable Anchors. Our contribution is to analyze cross-task representation changes and regularize their Jacobian coupling during training, rather than introduce Function Vectors or their connection to forgetting.
GEM / A-GEM constrain updates through old-task loss gradients; we target cross-task representation Jacobian coupling.
Comparison with the FV baseline (Jiang et al., 2025)
Thank you for suggesting this comparison. We now include the FV replacement baseline below. Performance is measured by ROUGE-L (%), rounded to two decimal places. Bold BWT and F.Ra values at the end of each stream are averages over its first four tasks, not metrics for the last task.
AA (mean final-checkpoint score across all five tasks), in NI-Seq-C / NI-Seq-G / NI-Seq-M order: Qwen3-8B: 78.93 / 19.46 / 51.88; Llama3.1-8B: 81.11 / 20.35 / 54.29.
Overall, the FV baseline performs between O-LoRA and our method, although the ranking varies across individual task streams and metrics.