acceptodds
Under review as a conference paper at ICLR 2027

Towards Transferable Parameter Updates for Continual Learning in Large Language Models

Abstract

Continual learning (CL) for large language models (LLMs) aims to enable sequential knowledge acquisition while mitigating catastrophic forgetting. However, most existing LLM CL methods primarily regulate parameter modifications, such as determining which parameters should be updated or preserved, while rarely exploring how effective task-specific update behaviors can be learned and generated. This limitation hinders their ability to facilitate knowledge transfer across tasks. To address this challenge, we rethink parameter updates in CL from the perspective of learning adaptive update strategies. Specifically, we introduce Meta-HA, a meta-learning framework that learns transferable update strategies through Hyper-Adapter modules. Hyper-Adapter module consists of a meta-learner and a controller, where the meta-learner captures transferable update priors across tasks through outer-loop optimization, and the controller instantiates these priors to generate task-specific updates and consolidate new knowledge into memory during inner-loop adaptation. Extensive experiments on multiple CL benchmarks demonstrate that our method consistently outperforms state-of-the-art baselines in both average performance and backward transfer.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, meta learning, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6EtSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Meta-HA studies replay-free continual adaptation of frozen LLMs. It meta-trains LoRA priors over simulated task sequences, trains a controller on each downstream task, and uses its outputs to write updates into persistent adapter memory. Experiments on three NLP continual-learning benchmarks report improved average performance and backward transfer, with additional backbone, task-order, and component studies.

Critical strengths
  1. Table 1 shows consistent gains across three benchmarks, including near-zero BWT. The additional models and task orders support the usefulness of the approach beyond one configuration.

  2. ANML receives the same meta-training stage, and Tables 2-3 test controller removal, random initialization, and multi-task pretraining. These are useful controls for attributing the gains.

Critical weaknesses
  1. Important 2026 comparisons are missing (Sections 2 and 5.1). Meta-UCF (Xiao et al., ICLR 2026) also generates LoRA updates with a shared hypernetwork for LLM continual learning. It uses replay, so that difference should be made explicit, but it is a close methodological comparison. SLAO, from Merge before Forget (Qiao and Mahdavi, ICLR 2026), continually merges updates into one LoRA and evaluates on closely related benchmarks. Neither appears in the references or experiments. Including FOREVER is welcome, but these omissions leave the claimed advantage over generated adapters and single-adapter consolidation uncertain.

  2. The novelty claim needs a narrower explanation (Section 4; Algorithm 1). The outer loop learns LoRA priors, while the controller is reset and trained conventionally for each task. This looks like meta-learned adapter state combined with a task-trained writer; the paper should explain what makes it a transferable update strategy beyond that combination. K-DeCore (Chen et al., NeurIPS 2025) is relevant background on separating reusable and task-dependent functions with fixed trainable parameter counts. It uses example replay and does not generate adapter updates, so it is a positioning reference rather than a required baseline.

  3. The experimental evidence does not fully establish resource-efficient transfer (Section 5.1; Appendices A and D). Meta-training uses 1,187 auxiliary tasks. Matched ANML training and Table 3 help, but total training cost and parameter budgets are not compared. Evaluation-task exclusion is stated without source-dataset or example-level exclusion details, leaving the strength of cross-dataset transfer unclear. Category-based pool ablations are useful but do not resolve this. Seed counts and uncertainty estimates for the main tables are also missing.

  4. A central implementation step remains underspecified (Section 4.2; Appendix C; Algorithm 1, line 13). The trained controller writes using the full task training set, but the paper does not explain how example-level writes are aggregated or whether each write changes the state used to compute subsequent writes. These choices affect update magnitude and order dependence. Explicit pseudocode separating controller fitting, memory consolidation, and prediction would make the method reproducible.

Questions for authors
  1. How does Meta-HA differ from Meta-UCF's parameter generator, and why was SLAO excluded from the comparisons?
  2. Were alternative formulations of the same source datasets excluded from meta-training? Please give the exclusion rule and clarify validation/test separation.
  3. How are full-dataset writes aggregated? Please also report the existing runs' seed counts, parameter counts, and meta-training versus downstream training cost.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 6EtS

Response to Reviewer 6EtS

We sincerely thank you for your valuable comments to help improve our paper.

For Weakness 1 and Q1: about "analysis of Meta-UCF and SLAO".

Meta-UCF. Both methods generate LoRA parameters, but differ in training strategy and knowledge consolidation. Meta-UCF continuously trains a shared hypernetwork to generate task-specific LoRA adapters conditioned on historical replay samples. In contrast, Meta-HA meta-learns transferable conditioning priors offline, which guide task-specific controllers to generate incremental updates for persistent memory consolidation without downstream replay. We attempted to reproduce Meta-UCF, but insufficient implementation details prevented a reliable comparison. We therefore clarified the methodological differences in the Related work of the revised manuscript.

SLAO. We have now reproduced SLAO following its algorithm and evaluated it on Qwen3-0.6B, averaging results across multiple task orders.

MethodStd. CL APStd. CL BWTLong Seq. APLong Seq. BWTSuperNI APSuperNI BWT
SLAO73.8-0.771.4-0.643.4-0.8
Meta-HA75.3-0.271.7-0.347.4-0.8

Meta-HA consistently achieves higher AP across all three benchmarks, with improvements of 1.5, 0.3, and 4.0 points on Standard CL, Long Sequence, and SuperNI, respectively. We have incorporated these comparisons into Table 1 of the revised manuscript.

For Weakness 2: about "explanation of transferable update strategy".

Meta-HA's contribution is not merely combining a meta-learned adapter state with a task-specific writer, but learning a transferable update strategy through the bi-level optimization of their interaction.

What makes the strategy transferable? The meta-learner fθf_\theta learns input-conditioned priors through outer-loop optimization over CL episodes, where its effectiveness is evaluated after sequential adaptation and memory consolidation. Crucially, the meta-learner and accumulated task memory fϕf_\phi jointly contribute to prediction (Eq. 6) and condition the controller's update generation (Eqs. 2–5). Although each task-specific controller gψg_{\psi} is reset and independently optimized, its adaptation and generated updates depend on these shared priors and the evolving memory state. Thus, what transfers is not the controller parameters, but a meta-learned prior optimized for the entire sequential update-and-consolidation process, rather than merely an initialization for predefined gradient-based adaptation.

Distinction from K-DeCore. K-DeCore separates task-agnostic schema filtering and task-specific query building, focusing on functional decomposition for structured knowledge reasoning. In contrast, Meta-HA separates transferable update conditioning, task-specific parameter update generation, and persistent knowledge consolidation. Unlike K-DeCore's replay-based approach, Meta-HA explicitly generates LoRA-factor updates without requiring historical sample replay during downstream adaptation.

We have clarified these distinctions and position K-DeCore appropriately in the Subsection 4.1 and Related Work of the revised manuscript.

For Weakness 4 and Q3: about "memory writing details and other key steps".

During controller optimization, each training batch generates a temporary update using Eqs. (3)–(4). This update is temporarily applied to compute the task loss and optimize the controller, but is discarded afterward, leaving the persistent task memory unchanged. Thus, updates generated by earlier training batches do not alter the memory state used by subsequent batches. After controller optimization, the trained controller is fixed and used to generate updates for all examples in the full task training set, starting from the same persistent memory state. Each example-level update is obtained by averaging over its target tokens (Eq. 4), and the resulting updates are then averaged equally across all training examples. The brief pseudocode of controller training, memory writing, consolidation and prediction are as follows:

Algorithm: Inner Optimization,Task-Level Memory Consolidation and Prediction

Input: Task training set DttrD_t^{tr}, sampled subset Dtsub⊂DttrD_t^{sub}\subset D_t^{tr}, meta-learner fθf_{\theta}, task memory fϕf_{\phi}, and controller gψg_{\psi}.

Stage 1: Controller Optimization

  1. Initialize the task-specific controller and freeze the meta-learner and persistent task memory.
  2. For each training batch B\mathcal{B} from DtsubD_t^{sub}:
  • Generate masked example-level updates ΔW^i\Delta \hat{W}_i using Eqs. (3)–(4).
  • Compute the temporary batch update: ΔW^B\Delta \hat{W}_\mathcal{B} = mean(ΔW^i\Delta \hat{W}_i for ii in B\mathcal{B}).
  • Reshape ΔW^B\Delta \hat{W}_\mathcal{B} into LoRA-factor increments and temporarily apply them to task memory for prediction and controller optimization.
  • Discard the temporary update after the optimization step, leaving persistent task memory unchanged.

Stage 2: Task-Level Memory Consolidation

  1. Fix the optimized controller.
  2. Traverse the full training set DttrD_t^{tr} and generate masked example-level updates ΔW^i\Delta \hat{W}_i using Eqs. (3)–(4).
  3. Compute the task-level update by averaging over all training examples: ΔW^\Delta \hat{W} = mean(ΔW^i\Delta \hat{W}_i for ii in DttrD_t^{tr}).
  4. Reshape ΔW^\Delta \hat{W} into LoRA-factor increments ΔA\Delta A and ΔB\Delta B.
  5. Write to persistent task memory exactly once: At←At+ΔAA_t \leftarrow A_t + \Delta A, Bt←Bt+ΔBB_t \leftarrow B_t + \Delta B.

Stage 3: Prediction

Disable the controller and perform prediction using the frozen base model, meta-learner, and consolidated task memory (Eq. 6), without generating additional updates.

We have added the aggregation formula and corresponding pseudocode to Subsection 4.3 and Algorithm 1 of the revised manuscript.

For Q3: about "more training details include seed counts, parameter counts and training cost".

  1. Seed Count. To evaluate the robustness of Meta-HA, we conducted additional experiments using three random seeds (40, 41, and 42) on Qwen3-0.6B across all three CL benchmarks. The results are reported as mean ± standard deviation:
SeedStd. CL APStd. CL BWTLong Seq. APLong Seq. BWTSuperNI APSuperNI BWT
4076.41.173.22.846.22.4
4174.6-0.370.8-0.846.4-1.8
4275.3-0.271.7-0.347.4-0.8
Mean ± Std.75.4 ± 0.910.2 ± 0.7871.9 ± 1.210.6 ± 1.9546.7 ± 0.64-0.1 ± 2.19

The results confirm that Meta-HA's performance gains are robust across random seeds. Compared with the strongest baseline reported in Table 1, Meta-HA achieves best performance on Standard CL, Long Sequence, and SuperNI, respectively. These additional results strengthen the empirical evidence that Meta-HA's continual learning improvements are not dependent on a particular random seed. We have included the seed counts and corresponding uncertainty statistics in Appendix D.6 of the revised manuscript.

  1. Parameter Budget. We provide a detailed parameter comparison on Qwen3-0.6B:
MethodLoRA MemoryController / ModulationTotal Trainable
LoRA (r=8)1.15M–1.15M
ANML1.15M0.041M1.19M
ANML (All-layer)1.15M1.15M2.29M
Meta-HA1.15M10.32M10.47M

ANML configuration. Following the original ANML design, we apply the modulation network to the final layer, resulting in 1.19M trainable parameters. We also investigated extending modulation to all LoRA layers to increase its parameter budget. However, despite increasing the trainable parameters to 2.29M, this variant performed worse than the standard implementation. Therefore, we retain the original configuration for a faithful and competitive comparison.

Meta-HA's advantage. Although Meta-HA introduces additional training-time parameters through the controller, these parameters are dedicated to learning task-adaptive update behaviors rather than expanding persistent knowledge capacity. The meta-learner and task memory operate within the same rank-8 LoRA space (1.15M), maintaining the same effective adapter rank as conventional LoRA. Moreover, the controller is disabled after memory consolidation and does not participate in inference. Thus, Meta-HA achieves stronger continual learning performance without increasing the effective persistent adapter rank or introducing additional controller inference overhead, despite its higher training-time parameter budget.

We have included this analysis in the Appendix D.7 of the revised manuscript.

  1. Training Cost. We compare Meta-HA and ANML under the same meta-training protocol using two NVIDIA RTX 4090 (48GB) GPUs.
MethodMeta-training (GPU-hours / 100 episodes)Long Seq. Training(GPU-hours)
ANML49.71.0
Meta-HA65.20.8

Meta-HA incurs 31.2% higher meta-training cost but achieves 20% lower downstream training cost than ANML. The additional meta-training overhead arises from repeatedly optimizing the controller and meta-learner through bi-level learning. However, this upfront investment enables more efficient downstream knowledge acquisition. Meta-HA trains its controller on only a subset of task samples, then uses the learned update mechanism to consolidate knowledge from the full dataset without further gradient-based optimization. In contrast, ANML relies on conventional gradient-based adaptation over the task training data. Consequently, Meta-HA reduces downstream training cost despite its substantially larger controller, demonstrating the efficiency benefit of learning transferable update behaviors.

We have included this analysis in the Appendix D.8 of the revised manuscript.

Reviewer_qkKbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Meta-HA, a meta-learning framework for continual learning in large language models. It uses Hyper-Adapter modules consisting of a meta-learner, a controller, and task memory. The meta-learner captures update priors across simulated task sequences, while the controller generates task-specific parameter updates that are accumulated in memory. Experiments on three continual learning benchmarks show improved average performance and reduced forgetting across different model sizes.

Critical strengths
  1. The paper is generally well written and easy to follow. The motivation and the roles of the main components are clearly presented.

  2. The experiments are comprehensive, covering multiple benchmarks, model sizes, and task orders. The comparisons with various continual learning baselines and the additional ablation studies help demonstrate the effectiveness of the framework.

Critical weaknesses

I have two main concerns:

  1. My first concern is that the contribution of the claimed transferable update strategy is not sufficiently clear.

The meta-learner is implemented as a LoRA module, while the controller is optimized for each task. I find it difficult to tell what update behavior is transferred across tasks and how it differs from learning a useful representation or initialization. The controller and initialization ablations are helpful, but their interpretation could be more precise. For example, removing the controller reduces AP on Long Sequence from 69.5 to 66.7, while improving BWT from -0.7 to -0.3. This supports its role in task adaptation, but does not consistently support its contribution to reducing forgetting.

I would like a clearer explanation of what each component learns and how it contributes to the reported continual learning gains.

  1. I also find the task-level memory-writing procedure insufficiently specified.

Section 5.3 states that the controller performs one final update to task memory, and Appendix C says that this update uses the full training set. However, Eqs. (3)–(5) describe updates generated from an individual example. It is unclear how these example-level updates are aggregated and normalized across the full dataset to produce the final task-level update. Since this operation determines what knowledge is consolidated into memory, I think it needs to be specified more explicitly.

I would consider raising my rating if the authors can clarify the memory-writing procedure and provide stronger evidence for the contribution of the learned update strategy.

Questions for authors
  1. Are the controllers reset to the same initialization before every downstream task, following the procedure used during meta-training in Algorithm 1?

  2. When removing overlap between the meta-training pool and evaluation tasks, is filtering performed at the task-ID level or the underlying-dataset level? Different Natural-Instructions tasks can be constructed from the same source dataset.

  3. A small suggestion: could you expand the caption of Figure 2 to briefly explain the main components and the inner- and outer-loop procedures? The current caption is quite brief, and a more informative description would make the framework easier to understand.

5 comments
AuthorsOpenReview ↗

Response to Reviewer qkKb

Response to Reviewer qkKb

We sincerely thank you for the effort and expertise that you contributed towards reviewing our paper.

For Weakness 1: about "the transferable update strategy and component contributions".

  1. What is transferred, and how does it differ from learning an initialization? Meta-HA learns a transferable update strategy, rather than simply a shared initialization. Specifically, the meta-learner fθf_\theta captures transferable conditioning priors through outer-loop optimization over CL episodes, where performance is evaluated after sequential adaptation and memory consolidation. These priors contribute directly to model prediction (Eq. 6) and, together with accumulated task memory fϕf_\phi, condition the task-specific controller gψg_{\psi} to generate LoRA-factor updates (Eqs. 2–5). The generated updates are then consolidated into persistent memory. Unlike initialization-based meta-learning, which adapts shared parameters through predefined gradient updates, Meta-HA uses a task-conditioned generator to determine how new knowledge is incorporated based on both the meta-learned prior and existing memory. Thus, the meta-learner provides the transferable prior, while the overall update-generation and consolidation process constitutes the transferable strategy.

  2. Ablation evidence for the learned update strategy. In Table 2, removing the controller reduces Meta-HA to FOMAML-style gradient-based adaptation, decreasing AP by 2.8/2.7 points on Long Sequence/SuperNI. Conversely, Table 3 shows that retaining the controller but replacing episodic meta-training with random initialization or conventional multi-task training substantially degrades both AP and BWT. These complementary results demonstrate that neither a meta-learned initialization nor a task-specific update generator alone is sufficient. The gains depend on their interaction through sequential meta-optimization.

  3. Component contributions and the BWT trade-off. The controller primarily enhances plasticity through task-adaptive update generation, while the relative objective LrelL_{\mathrm{rel}} explicitly promotes stability. Importantly, these objectives are coupled rather than independent. On Long Sequence, removing the controller improves BWT (-0.7 → -0.3) but reduces AP (69.5 → 66.7), whereas removing LrelL_{rel} improves AP (69.5 → 70.5) but worsens BWT (-0.7 → -1.3), illustrating the stability–plasticity trade-off. On SuperNI, removing either component degrades both AP and BWT, indicating that disrupting their balance can compromise both knowledge acquisition and retention. Therefore, forgetting mitigation should not be attributed to the controller alone, but to the coordinated adaptation and consolidation mechanism under the stability-aware meta-objective LrelL_{rel}.

We haved revised the manuscript to clarify these distinctions and component contributions. The details can be found in the Subsection 4.1 and 5.3.

For Weakness 2: about "the task-level memory-writing procedure".

We clarify that controller optimization and task-level memory consolidation are two separate stages. During controller training, each batch produces a temporary update used only for loss computation, while the persistent task memory remains unchanged. After training, the optimized controller generates updates over the full training set, which are averaged across all samples and written into memory exactly once.

Algorithm: Inner Optimization and Task-Level Memory Consolidation

Input: Task training set DttrD_t^{tr}, sampled subset Dtsub⊂DttrD_t^{sub}\subset D_t^{tr}, meta-learner fθf_{\theta}, task memory fϕf_{\phi}, and controller gψg_{\psi}.

Stage 1: Controller Optimization

  1. Initialize the task-specific controller and freeze the meta-learner and persistent task memory.
  2. For each training batch B\mathcal{B} from DtsubD_t^{sub}:
  • Generate masked example-level updates ΔW^i\Delta \hat{W}_i using Eqs. (3)–(4).
  • Compute the temporary batch update: ΔW^B\Delta \hat{W}_\mathcal{B} = mean(ΔW^i\Delta \hat{W}_i for ii in B\mathcal{B}).
  • Reshape ΔW^B\Delta \hat{W}_\mathcal{B} into LoRA-factor increments and temporarily apply them to task memory for prediction and controller optimization.
  • Discard the temporary update after the optimization step, leaving persistent task memory unchanged.

Stage 2: Task-Level Memory Consolidation

  1. Fix the optimized controller.
  2. Traverse the full training set DttrD_t^{tr} and generate masked example-level updates ΔW^i\Delta \hat{W}_i using Eqs. (3)–(4).
  3. Compute the task-level update by averaging over all training examples: ΔW^\Delta \hat{W} = mean(ΔW^i\Delta \hat{W}_i for ii in DttrD_t^{tr}).
  4. Reshape ΔW^\Delta \hat{W} into LoRA-factor increments ΔA\Delta A and ΔB\Delta B.
  5. Write to persistent task memory exactly once: At←At+ΔAA_t \leftarrow A_t + \Delta A, Bt←Bt+ΔBB_t \leftarrow B_t + \Delta B.

This procedure ensures that controller optimization does not modify persistent memory, while the final consolidation uses the full task training set with equal weighting across examples. We have included this procedure and the corresponding aggregation formula in the revised manuscript. The details can be found in Subsection 4.3 and Algorithm 1.

For Q1: about "the controllers reset to the same initialization before every downstream task".

Yes. During downstream evaluation, the controller is reset to the same initialization gψ0g_{\psi_0} before adapting to each new task, consistent with the procedure in Algorithm 1. The meta-learner remains frozen throughout downstream evaluation, while the task memory retains the accumulated updates across tasks. Thus, task-specific adaptation does not rely on carrying over controller parameters from previous tasks.

For Q2: about "the filter strategy of meta-training pool".

In our original implementation, overlapping tasks were excluded from the meta-training pool based on task IDs and task names. To further address the concern that different task IDs may originate from the same underlying dataset, we conducted an additional experiment with more stringent source-aware filtering. Specifically, we identified 25 dataset-specific keywords associated with the evaluation benchmarks: agnews, amazon, boolq, dbpedia, imdb, mnli, mrpc, qnli, qqp, rte, cb, sst2, copa, yahoo, yelp, quoref, commonsenseqa, reddit_tifu, sciq, woz_user_utterance, glucose, xsum, diplomacy, personachat, and samsum. We further filtered the meta-training tasks based on specific keywords in their names, in addition to the filtering based on task IDs and task names. This stricter filtering reduced the meta-training pool from 1,187 to 1,090 tasks. We then evaluated Meta-HA with the filtered meta-training pool using Qwen3-0.6B across all three CL benchmarks. As shown in the following Table, the results demonstrate comparable performance to the original setting across all benchmarks, indicating that the effectiveness of Meta-HA is robust to this stricter filtering strategy. These results provide further evidence that Meta-HA learns generalizable and transferable update strategies through meta-learning, rather than merely memorizing dataset-specific knowledge. We have clarified the original filtering protocol and include this additional experiment in the Appendix A and D.5.

MethodStd.CL APStd.CL BWTLong Seq. APLong Seq. BWTSuperNI APSuperNI BWT
Meta-HA75.3-0.271.7-0.347.4-0.8
Meta-HA (keywords filter)75.21.271.2-0.546.1-0.5

For Q3: about "expand the caption of Figure 2 to briefly explain the main components and the inner- and outer-loop procedures".

We have expanded the Figure 2 caption to describe the roles of the meta-learner, controller, and task memory, together with the inner-loop adaptation and outer-loop optimization procedures. The details are as follows:

Figure 2: Overview of the Meta-HA framework. During inner-loop adaptation, the controller generates task-specific parameter updates conditioned on the meta-learner and accumulated task memory, and consolidates them into memory. During outer-loop optimization, the meta-learner is optimized using held-out validation data across continual learning episodes to learn transferable update priors. The backbone LLM remains frozen throughout.

Reviewer_BK6rOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Meta-HA, a meta-learning framework for continual learning in LLMs. It introduces a meta-learner to learn transferable update strategies, a controller to generate task-specific LoRA parameter updates, and shared task memory to consolidate knowledge. Through bi-level optimization, Meta-HA improves knowledge retention and reduces catastrophic forgetting, achieving strong performance across multiple benchmarks.

Critical strengths

The key strength of this paper is its ability to develop task-specific update strategies for mitigating catastrophic forgetting. By combining transferable meta-learned priors with task-adaptive controllers, Meta-HA generates parameter updates tailored to each new task while encouraging the preservation of previously learned knowledge, rather than relying on fixed update rules or constraints.

Critical weaknesses

Meta-HA relies on a fixed-rank shared LoRA memory to consolidate knowledge across tasks, raising concerns about whether its limited parameter capacity can accommodate increasingly diverse tasks without interference. Although the current experiments demonstrate strong performance, further evaluations on longer task sequences and with different memory ranks would help assess the scalability and capacity limitations of the proposed method.

Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer BK6r

Response to Reviewer BK6r

We are grateful for your careful review, helpful comments and detailed questions that ensued.

For Weakness 1: about "evaluations on longer task sequences and with different memory". To investigate the scalability and capacity limitations of Meta-HA, we construct an extended 30-task continual learning scenario by combining Long Sequence and SuperNI, and evaluate Meta-HA with memory ranks of 8 and 16.

(1) Scalability to longer task sequences. As shown below, Meta-HA with rank 8 achieves an average AP of 61.0 and BWT of -2.8 across the 30-task sequence. Its AP is comparable to the average performance obtained when evaluating the two original 15-task benchmarks separately in Table 1, suggesting that the shared rank-8 memory can accommodate substantially longer task sequences without severe capacity-induced performance degradation.

(2) Effect of memory capacity. Interestingly, increasing the rank from 8 to 16 does not improve AP. Instead, AP decreases by 0.8 points, while BWT deteriorates by 2.2 points. This suggests that simply increasing memory capacity does not necessarily benefit continual learning. A higher rank provides greater flexibility for task-specific updates, but may also allow more aggressive updates that interfere with previously consolidated knowledge. This observation is consistent with recent studies on LoRA-based continual learning, which identify a trade-off between adaptation capacity and forgetting [1][2]. We have included these additional experiments and discussions in the Appendix D.4 of the revised manuscript.

RankAP Order 1AP Order 2AP AVGBWT Order 1BWT Order 2BWT AVG
r=861.260.761.0-2.6-2.9-2.8
r=1659.760.760.2-5.5-4.4-5.0

[1] Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learning. (EMNLP 2026)

[2] Merge before Forget: A Single LoRA Continual Learning via Continual Merging. (ICLR 2026)

Dear Reviewer BK6r,

Thank you again for your thoughtful review and valuable suggestions. We have carefully addressed your concerns and revised the manuscript accordingly, with changes highlighted in blue. As the discussion deadline approaches, we would sincerely appreciate it if you could revisit our responses and let us know whether any concerns remain. We would be happy to provide further clarification.

Thank you for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.