acceptodds
Under review as a conference paper at ICLR 2027

Continual Learning Mechanisms Compose for Long-Horizon Memorization

Abstract

Language models may need to internalize information that arrives over time and retain it through many subsequent updates. To study this challenge, we introduce long-horizon memorization, a setting in which a model learns 100 query-answer tasks through continual supervised fine-tuning without retaining earlier training examples or receiving task identifiers at inference. Sequential updates cause catastrophic forgetting, and no single continual learning mechanism we evaluate maintains strong retention at this horizon. We hypothesize that mechanisms addressing complementary sources of forgetting will be more effective when composed. We organize these compositions along two design dimensions. Data, function, and weight anchors specify what prior information each update should preserve, while low-rank allocation rules determine where successive updates are retained. To test this hypothesis systematically, we construct three distinct 100-task memorization datasets. We introduce task-level successive halving to search the combinatorial design space and use a factorial experiment to measure individual and interaction effects. Our best method combines all three anchors with merged LoRA, ranks among the top 3 methods in all datasets, and raises average final retention from 1.2% under naive sequential fine-tuning to 34.9%, a 28-fold improvement. The data anchor and merged LoRA provide the largest average gains and interact super-additively on all three datasets. Together, these results show that composing complementary mechanisms substantially improves long-horizon memorization beyond what any individual mechanism achieves.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, memorization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MmfwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the forgetting problem in a long-horizon continual learning of LLMs.They adopted LoRA method and conbines multiple methods to address the forgetting problem. They combined: SFT, regularization(anchor), and merge LoRA. They set 100 continual tasks and provide the performance under different combination of methods. Their results show that combining multiple methods outperforms a single method.

Critical strengths
  1. The provide the performance under different combination of mechanisms. The experiments are comprehensive.

  2. The resulrts supports a meaningful conclusion that data replay and merge lora outperformns other combination of two methods.

Critical weaknesses
  1. The motivation is not clear to me.

  2. The baseline is too weak.

  3. Lack of explanation.

Details see "Questions".

Questions for authors
  1. The motivation for the long-horizon setting is unclear to me. First, repeatedly updating shared LLM parameters introduces a risk of interference and forgetting. What practical scenario motivates sequential SFT over 100 tasks, and why is this an appropriate strategy for long-term knowledge accumulation? Second, what is distinctive about the long-horizon regime beyond the increased number of updates? Which conclusions specifically depend on evaluating 100 tasks, and would these conclusions differ from those drawn from shorter sequences? Explicit comparisons across horizons would help establish the insights gained from this setting.

  2. In this paper, there are 100 tasks. It is no surprising that the baseline is weak(naive fine-tuning). Therefore, the statement "a 28-fold improvement" sounds tricky for me. There are plenty of continual llm studies, and I think the auther should establish a stronger baseline.

  3. Is there any explanation on the mechanism combination? For example, Data + Merge is stronger than other combination of two. Can you explain the reason?

4 comments

We thank the reviewer for the constructive feedback, and for recognizing the comprehensiveness of the experiments and the benefit of combining replay with merged LoRA. We address the motivation, the baseline comparison, and the mechanism explanation below.

W1, Q1: Motivation for sequential SFT and what the 100-task horizon adds

Motivation. A practical scenario is a personal agent that periodically incorporates information from new interactions, documents, and experiences into a persistent memory model. Each update must acquire the newly available information while preserving the knowledge accumulated through earlier interactions.

Recent work gives concrete examples of this direction. MeMo [1] internalizes knowledge in a dedicated memory model queried by a frozen reasoning model, and SEAL [2] uses model-generated training data to incorporate new information through parameter updates. SEAL explicitly reports forgetting under repeated self-updates, which highlights retention as an unresolved obstacle to continual knowledge accumulation.

Why sequential SFT. Our sequential SFT setting abstracts this process into successive batches of newly arriving knowledge. SFT is a direct mechanism for incorporating these batches into parameters, particularly when repeatedly training on the complete historical corpus is impractical or earlier examples are unavailable.

We study which combinations of continual learning mechanisms allow repeated acquisition while limiting the forgetting of earlier knowledge. MeMo uses a dedicated memory model, so this question is relevant even when the agent's main reasoning model remains frozen.

Why 100 tasks. We use 100 tasks to examine retention across a sustained sequence of acquisition events. Memorization in real-world applications is long-horizon, and we think a 100-task setting reflects these challenges.

Explicit horizon comparison. We analyzed runs at horizons of 10, 20, 50, and 100 tasks. The table below shows retention on the first ten tasks for Replay+Merge.

DatasetAfter 10 tasksAfter 20 tasksAfter 50 tasksAfter 100 tasks
Symbol-QA79.6 ± 8.6%26.9 ± 9.7%0.1 ± 0.2%0.0 ± 0.0%
LLM-QA88.3 ± 2.3%52.6 ± 8.1%14.4 ± 1.9%5.8 ± 2.0%
Real-QA81.9 ± 2.0%62.5 ± 8.1%27.3 ± 4.8%9.5 ± 2.0%

Table 1: Replay+Merge retention (%) on the first ten tasks at four horizons. The evaluated associations stay fixed as training continues.

What the longer horizon reveals. This comparison shows why the longer task sequence matters. Tracking the same first ten tasks separates the durability of earlier knowledge from performance on newly acquired information. For Replay+Merge, retention on these Real-QA tasks falls from 81.9% after ten tasks to 9.5% after 100.

Strong retention over a short sequence can therefore substantially overstate the persistence of acquired knowledge. The longer evaluation also lets us measure memory lifetime and track retention as training continues (Section 5.2).

The horizon comparison also clarifies which conclusions are already visible in shorter runs. The Replay+Merge interaction is positive at horizon 10, and method rankings are strongly correlated across horizons. These qualitative findings therefore do not depend exclusively on evaluating 100 tasks.

What the longer horizon establishes is the severity of accumulated forgetting and the magnitude of compositional effects over continued learning. On Real-QA, the interaction estimate rises from 9.8 at horizon 10 to 15.5, 30.9, and 33.0 at horizons 20, 50, and 100, respectively. The interaction trend differs across datasets. We have revised Section 5.2 and Appendix A.6.4 to include these comparisons.

Future work. Building on these findings, we plan to combine the compositions identified through task-level successive halving with model-generated training data, and to evaluate them on continuous streams without explicit task boundaries. This would test their applicability to online adaptation in personal agents, complementing approaches based on expanding context windows. We leave these experiments to future work, because the present study focuses on demonstrating the benefits of composing complementary continual learning mechanisms.

Our findings suggest a broader design principle: new continual learning methods should be developed and evaluated with attention both to their standalone effectiveness and to their interactions with existing mechanisms.

[1] Ryan Wei Heng Quek et al. MeMo: Memory as a Model. 2026. arXiv:2605.15156.

[2] Adam Zweiger et al. Self-Adapting Language Models. 2025. arXiv:2506.10943.

W2, Q2: Stronger baselines and the "28-fold improvement" statement

We thank the reviewer for raising the concern about the "28-fold improvement" statement. Our paper already reports eight baselines (Naive SFT w/ shared LoRA, SFT w/ merged LoRA, SFT w/ O-LoRA, SFT w/ OSRM, Generative Replay, Elastic Weight Consolidation, Synaptic Intelligence, LwF/Self-Distillation), as well as their compositions, in Section 5.1 and Appendix A.5.2. We have revised the abstract to report the improvement over the strong baseline generative replay instead: from 8.1% to 34.9% average final retention.

We have also added a REMIX comparison to Section 5.1 and Appendix A.6.1. Across three seeds on Real-QA, the REMIX adaptations reach 1.4 ± 0.1% final retention with shared LoRA and 2.1 ± 0.2% with merged LoRA. Replay+Merge reaches 48.2 ± 2.8%, and SI+Replay+Merge reaches 54.8 ± 10.4%. The REMIX adaptations exceed 97% immediate acquisition, so their low retention after 100 tasks is not due to a failure to learn the current tasks.

W3, Q3: Why Data + Merge is stronger than other combinations

Replay and low-rank allocation act on different parts of an update. Replay retains training signal about earlier associations, while allocation determines how successive updates are represented. Shared LoRA repeatedly modifies a single rank-constrained update (Sections 3.2 and 3.3). Merged LoRA instead incorporates each task's update into the backbone and initializes a fresh adapter for the next task, so the accumulated change is not restricted to one rank-r adapter.

We hypothesize that the two mechanisms complement each other: replay provides the information to preserve, and merge provides the additional freedom (effective rank) to accommodate successive updates. We have revised Section 5.3 to include this explanation and the replay coverage analysis below.

Interaction. We compute the interaction effect of replay and merge as

I=A(Replay+Merge)−A(Replay)−A(Merge)+A(Naive SFT)I = A(\text{Replay+Merge}) - A(\text{Replay}) - A(\text{Merge}) + A(\text{Naive SFT})

where A is the mean accuracy over all tasks seen at the evaluated horizon, with SI and SD absent. At 100 tasks, the interaction is 11.7 ± 5.6 percentage points on Symbol-QA, 23.3 ± 6.5 on LLM-QA, and 33.0 ± 2.3 on Real-QA.

Replay coverage. To test this hypothesis, we analyzed 1782 saved replay batches across the three datasets, comparing Replay and Replay+Merge over three seeds. Each batch contains 300 generation attempts. The table below reports the number of distinct, correctly reproduced past question–answer pairs divided by those 300 attempts.

DatasetReplayReplay + Merge
Symbol-QA27.9 ± 0.3%32.1 ± 2.4%
LLM-QA33.5 ± 3.6%37.6 ± 5.3%
Real-QA22.9 ± 0.3%24.8 ± 0.8%

Table 2: Recovery of earlier associations in generated replay. Entries are distinct correct past pairs divided by 300 generation attempts (%), averaged over all 99 replay events in each run.

We average over all 99 replay-generation events in each 100-task run, and report the mean ± sample SD across three seeds. Replay begins at task 2, because no earlier tasks are available when learning task 1. A pair is correct if it exactly matches the canonical stored question and answer.

On Real-QA, the fraction of previous tasks represented by at least one correct replay pair also increases, from 19.6 ± 0.5% to 27.4 ± 0.6%. Merge is therefore associated with broader recovery of earlier associations across the full training sequence.

Dear Reviewer Mmfw,

Thank you again for your constructive review. We wanted to follow up before the author-reviewer discussion period closes.

In our response, we addressed your main concerns about the motivation for the long-horizon setting, the strength of the baselines, and why replay and merged LoRA work well together. In particular, we described the scenario motivating sequential SFT and added a comparison across horizons of 10, 20, 50, and 100 tasks. We revised the abstract to report the improvement over generative replay (8.1% to 34.9%) rather than over naive SFT, and added a REMIX comparison on top of the eight baselines already in the paper. We also added an interaction analysis and a replay-coverage analysis supporting our explanation that replay provides the information to preserve, while merging provides additional effective rank to accommodate successive updates.

Could you please let us know whether these explanations address your concerns? We would be happy to provide further clarification during the remaining discussion period. If our responses have resolved your concerns, we would be grateful if you would consider updating your assessment.

Thank you for your time and consideration.

Dear Reviewer Mmfw,

Please carefully review the authors' rebuttal and respond to each of your concerns. If any concerns remain unresolved, clearly specify what is missing and request concrete clarifications or additional evidence. In particular, if the baselines (references) are insufficient, please suggest specific baseline methods for comparison.

Bests,

Your AC.

Reviewer_4BCSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how language models can retain information learned across 100 sequential tasks without storing previous training examples. It investigate whether combining different continual learning mechanisms can reduce long-term forgetting more effectively than using individual methods. They combine three mechanisms: generative replay, self-distillation, and importance-based regularization, along with different LoRA update strategies.

Critical strengths

The paper addresses an important problem in continual learning by studying how models retain information over a long sequence of 100 tasks. This setting provides useful insights into long-term forgetting, which is often difficult to evaluate in shorter task sequences.

The paper systematically studies how different continual learning mechanisms work together rather than evaluating them individually. In particular, the finding that generative replay and merged LoRA provide complementary benefits is interesting and helps explain why combining certain methods can improve retention.

Critical weaknesses

The proposed approach mainly combines existing continual learning techniques, including generative replay, self-distillation, importance-based regularization, and LoRA merging. The paper does not introduce a fundamentally new mechanism for preventing forgetting.

The paper evaluates memorization using the same questions seen during training. While this measures how well the model retains learned associations, it does not show whether the retained knowledge can generalize to unseen or differently phrased questions.

Although the proposed method significantly improves retention of learned information, the experiments show substantial degradation in general capabilities such as mathematical reasoning and knowledge understanding. This raises concerns about whether the method can preserve the model's existing abilities while continuously learning new information.

Questions for authors

Please refer to the weaknesses listed above.

4 comments

We thank the reviewer for the thoughtful feedback, and for recognizing the importance of the long horizon and of the systematic analysis of complementary mechanisms. We respond to each concern below.

W1: Combining existing methods rather than introducing a new mechanism

We agree that the individual mechanisms are established. Our research question is whether their combinations remain effective through many sequential updates, and which interactions account for the gains we observe.

Our contribution is a controlled empirical evaluation of this question: we separate anchors from allocation rules, use three 100-task datasets, and analyze combinations factorially. Rather than evaluating one mechanism at a time, we treat composition itself as a design choice for continual learning. Its success across several mechanisms suggests a direction for developing and evaluating future methods. We have revised Section 1 to clarify this contribution.

W2: Evaluating memorization using the training questions

Scope. The paper explicitly studies memorization (Sections 4.1 and 6). Evaluating the trained question–answer associations measures whether they remain accessible after subsequent updates, and we acknowledge in the conclusion that we do not focus on generalization to new facts or tasks. We agree, however, that access under rephrased questions is a useful additional diagnostic of the retained associations, so we ran the experiment below.

Additional evaluation. We evaluated 500 trained Real-QA associations, five per task, using Qwen3-4B-Base. Each association is queried with both the original question and a rephrased question, keeping the same target answer. A question-only generator produced the rewordings. In the table below, SI denotes Synaptic Intelligence and SD denotes self-distillation.

MethodOriginal questionRephrased question
Naive SFT1.3 ± 0.2%0.4 ± 0.3%
Replay14.3 ± 1.3%10.7 ± 0.9%
Replay + Merge49.9 ± 3.5%41.3 ± 3.9%
SI + SD + Replay + Merge46.2 ± 10.4%37.0 ± 7.1%
SI + Replay + Merge56.5 ± 10.9%44.7 ± 10.6%

Table 1: Accuracy (%) on the same 500 REAL-QA associations under original and rephrased questions after 100 tasks.

We find that most retained associations remain accessible after rewording. SI+Replay+Merge reaches 44.7% on rephrased questions, compared with 0.4% for naive SFT and 10.7% for generative replay. We have revised Section 5.1 and Appendix A.6.3 to include this diagnostic of access to memorized associations, and Section 6 to clarify its scope.

W3: Degradation of pretrained capabilities

Our experiments show that retaining pretrained capabilities while continually learning new knowledge is a challenging problem. O-LoRA helps preserve more general capability on LLM-QA and Real-QA (Section 5.1, Figure 4). The comparison below keeps SI and Replay fixed and changes only the allocation rule.

Training datasetMethodGSM8KMATHMGSMMMLU-Redux
Before continual trainingBase model83.657.667.568.7
LLM-QASI + Replay + Merge9.3 ± 7.68.8 ± 4.42.5 ± 0.731.3 ± 7.8
LLM-QASI + Replay + O-LoRA27.2 ± 9.921.5 ± 6.412.5 ± 1.845.8 ± 3.0
Real-QASI + Replay + Merge7.5 ± 5.78.2 ± 5.44.8 ± 2.633.2 ± 7.5
Real-QASI + Replay + O-LoRA25.5 ± 10.722.1 ± 0.215.7 ± 5.952.0 ± 2.0

Table 2: Comparison of O-LoRA and Merge allocation rules for general capability preservation.

Entries for trained models are accuracy (%) after 100 tasks. These scores remain well below the base model, and O-LoRA uses state that grows with the number of tasks.

Additional experiment with math rehearsal. We trained Qwen3-4B-Base on the first ten Real-QA tasks, comparing naive SFT, Replay+Merge, and SI+SD+Replay+Merge with and without rehearsal of 500 GSM8K training examples. This gives 18 runs across three seeds. Mean accuracy improves on all four capability benchmarks for all three methods. The table below shows the Replay+Merge comparison.

Math rehearsalGSM8KMATHMGSMMMLU-Redux
Without19.2 ± 4.920.6 ± 8.714.6 ± 2.246.9 ± 1.6
With65.3 ± 4.846.2 ± 1.049.0 ± 1.157.8 ± 0.9
Math rehearsalReal-QA retention
Without81.7 ± 4.7
With83.2 ± 1.4

Table 3 and 4: General capability preservation with math rehearsal and 10-task final retention.

Real-QA retention evaluates all 500 trained associations. The improvements on MATH, MGSM, and MMLU-Redux show that the benefit extends beyond the rehearsed dataset.

Preserving pretrained capabilities while learning new associations remains an open challenge. We have revised Sections 5.1 and 6, and report the O-LoRA comparison in Appendix A.5.9 and the math-rehearsal experiment in Appendix A.6.5.

Dear Reviewer 4BCS,

Thank you again for your detailed review. We wanted to follow up before the author-reviewer discussion period closes.

In our response, we addressed your main concerns regarding novelty, evaluation on rephrased questions, and degradation of general capabilities. In particular, we clarified that our contribution is a controlled, factorial study of how continual learning mechanisms compose over a long horizon. We added an experiment on 500 Real-QA associations showing that most retained knowledge stays accessible under rephrased questions (44.7% for SI+Replay+Merge vs. 0.4% for naive SFT). We also added an O-LoRA comparison and a math-rehearsal experiment, which improved GSM8K, MATH, MGSM, and MMLU-Redux for all three methods tested.

Could you please let us know whether these explanations and new results address your concerns? We would be happy to provide further clarification during the remaining discussion period. If our responses have resolved your concerns, we would be grateful if you would consider updating your assessment.

Thank you for your time and consideration.

Thank you to the authors for the detailed rebuttal and additional experiments. I appreciate the efforts to address my concerns, particularly the new evaluation on rephrased questions. However, I am still concerned about the limited novelty of the proposed approach. Therefore, I will maintain my original rating.

Thank you for reading our response and for acknowledging that we resolved your concerns, particularly regarding the rephrased questions. We understand the remaining concern is novelty, and we would like to clarify one point.

We agree that we do not propose a new mechanism, and the paper says so explicitly. Our claim to novelty is empirical: (1) no single mechanism survives the 100-task horizon; (2) replay and merged LoRA interact super-additively on all three datasets (on Real-QA, their standalone gains sum to 13.9 points, while their combination yields 46.9); and (3) the long-horizon memorization setting, three datasets, and the factorial protocol that make these effects measurable. To our knowledge, none of these results appears in prior work. If you are aware of work that establishes them, we would be grateful for the references so we can position our paper accurately.

We appreciate your time and engagement throughout the discussion.

Reviewer_sYJoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper conducts an experimental study on how effective the two categories of methods, “anchors” (regularization toward the previous model) and “low-rank allocation” (shared or merged LoRA), and their combinations are for long-horizon memorization. The experimental protocol sequentially fine-tunes the pretrained model using task-specific training data across 100 tasks, and tests the levels of catastrophic forgetting on every previous task after SFT on a new task. The authors introduce three 100-task memorization datasets (two based on synthetic data and one based on real QA datasets) as well as the evaluation protocols. They conclude that the leading methods at 100 tasks are the ones that combine anchors and merged LoRA.

Critical strengths
  • The paper proposes an interesting 100-task horizon experimental setup and designs a protocol that evaluates three anchor methods and LoRA allocation rules, as well as their combinations.
  • The experimental study is quite comprehensive as it tests 90 configurations in a staged search and a full 2⁴ factorial across three datasets.
Critical weaknesses
  • The 100-task setting is still somewhat artificial. While this makes the controlled experimental study more manageable, it is not clear how the conclusion will apply to the real-world continual learning scenarios.
  • The work tests which method combination is the best recipe that alleviates forgetting (although even the best method usually can only retrain less than half of the memory of previous tasks), but provides limited insight on why such combination is the most effective one.
  • The novelty seems limited. For instance, the long-horizon memorization problem setting is not new (the paper should cite “Continual Memorization of Factoids in Language Models” by Chen et. al. 2024). Adapting the TSH method to the proposed problem is a reasonable choice but also somewhat straightforward.
Questions for authors
  • Do the authors have some explanation or insight on why replay and merged LoRA have a super-additive interaction?
  • How does this work relate to “Continual Memorization of Factoids in Language Models” by Chen et. al. 2024? Would their REMIX be a fair baseline to include?
4 comments

We thank the reviewer for the constructive feedback, and for recognizing the value of the 100-task protocol and of the systematic evaluation of mechanism combinations. We respond to each concern and question below.

W2, Q1: Why replay and merged LoRA have a super-additive interaction

Replay and low-rank allocation act on different parts of an update. Replay retains training signal about earlier associations, while allocation determines how successive updates are represented. Shared LoRA repeatedly modifies a single rank-constrained update (Sections 3.2 and 3.3). Merged LoRA instead incorporates each task's update into the backbone and initializes a fresh adapter for the next task, so the accumulated change is not restricted to one rank-r adapter.

We hypothesize that the two mechanisms complement each other: replay provides the information to preserve, and merge provides the additional freedom (effective rank) to accommodate successive updates. We have revised Section 5.3 to include this explanation and the replay coverage analysis below.

Interaction. We compute the interaction effect of replay and merge as

I=A(Replay+Merge)−A(Replay)−A(Merge)+A(Naive SFT)I = A(\text{Replay+Merge}) - A(\text{Replay}) - A(\text{Merge}) + A(\text{Naive SFT})

where A is the mean accuracy over all tasks seen at the evaluated horizon, with SI and SD absent. At 100 tasks, the interaction is 11.7 ± 5.6 percentage points on Symbol-QA, 23.3 ± 6.5 on LLM-QA, and 33.0 ± 2.3 on Real-QA.

Replay coverage. To test this hypothesis, we analyzed 1782 saved replay batches across the three datasets, comparing Replay and Replay+Merge over three seeds. Each batch contains 300 generation attempts. Table 1 below shows the recovery of earlier associations in the generated replay data.

DatasetReplayReplay + Merge
Symbol-QA27.9 ± 0.3%32.1 ± 2.4%
LLM-QA33.5 ± 3.6%37.6 ± 5.3%
Real-QA22.9 ± 0.3%24.8 ± 0.8%

Table 1: Recovery of earlier associations in generated replay. Entries are distinct correct past pairs divided by 300 generation attempts (%), averaged over all 99 replay events in each run.

We average over all 99 replay-generation events in each 100-task run, and report the mean ± sample SD across three seeds. Replay begins at task 2, because no earlier tasks are available when learning task 1. A pair is correct if it exactly matches the canonical stored question and answer.

On Real-QA, the fraction of previous tasks represented by at least one correct replay pair also increases, from 19.6 ± 0.5% to 27.4 ± 0.6%. Merge is therefore associated with broader recovery of earlier associations across the full training sequence.

Q2: Relationship to “Continual Memorization of Factoids in Language Models” and the REMIX baseline

We thank the reviewer for pointing us to this closely related work. It studies continual factoid memorization and proposes REMIX, which mixes random word sequences or generic text with the task data. Their main experiments use two training stages, with an additional three-stage analysis.

Our work instead studies memorization over 100 tasks, using three datasets of varying semantic naturalness. Our focus also differs: we study whether continual learning mechanisms are composable. We have revised Section 2 to cite Chen et al. and to discuss how the two works relate and differ.

Additional baseline. REMIX is a relevant baseline, so we ran it. We completed six additional 100-task Real-QA runs, using shared and merged LoRA with three seeds each. We adapted the random-first/generic-later mixing rule: random-word copying for the first task and Knowledge Pile continuations for later tasks, with two auxiliary examples per current QA example.

MethodMean immediate acquisitionFinal retention after 100 tasks
Naive SFT97.9 ± 1.6%1.3 ± 0.2%
Replay100.0 ± 0.0%12.5 ± 1.4%
Merge99.8 ± 0.1%4.0 ± 0.4%
REMIX adaptation, shared LoRA97.3 ± 0.3%1.4 ± 0.1%
REMIX adaptation, merged LoRA99.1 ± 0.2%2.1 ± 0.2%
Replay + Merge99.9 ± 0.0%48.2 ± 2.8%
SI + SD + Replay + Merge99.5 ± 0.0%44.3 ± 8.7%
SI + Replay + Merge99.9 ± 0.1%54.8 ± 10.4%

Table 2: REAL-QA memorization (%), mean ± sample SD over three seeds. Acquisition averages accuracy immediately after learning each task. Retention evaluates all 5,000 associations after task 100.

Acquisition averages each task's accuracy immediately after that task is learned. Final retention evaluates all 5,000 Real-QA associations after task 100. SI denotes Synaptic Intelligence and SD denotes self-distillation.

The REMIX adaptations acquire the current tasks successfully, but retain little after 100 tasks under this recipe. In this setting, our best composed method improves on the merged-LoRA REMIX adaptation by more than 25-fold. We have added this comparison to Section 5.1 and the full results to Appendix A.6.1.

W1: Practical motivation and the artificial nature of the setting

We thank the reviewer for raising the question of real-world relevance.

Motivation. An agent must continually acquire world knowledge and retain episodic experience. Current approaches mainly focus on retrieval-based memory; we instead study how parametric memory incorporates knowledge into its parameters continually. This has applications in continual learning agents for personal assistants, scientific discovery, and embodied AI. Our setting is a controlled abstraction of a practical requirement for such agents: newly acquired knowledge must remain accessible after many subsequent updates.

This motivation connects directly to recent work on parametric knowledge internalization. MeMo [1] trains a dedicated memory model to encode knowledge for a frozen reasoning model, and SEAL [2] incorporates new information through self-generated training data and parameter updates. SEAL explicitly identifies catastrophic forgetting under repeated self-updates as an outstanding challenge. These approaches motivate our central question: how can parametric memory accumulate knowledge while preserving earlier acquisitions?

Why a controlled setting. Our 100-task protocol isolates this retention problem over a sustained sequence of updates, without retaining earlier training examples or providing task identifiers at inference. This lets us analyze systematically how continual learning mechanisms and their interactions affect forgetting.

The three datasets differ in semantic naturalness (Section 4.2): arbitrary symbolic associations, fictional facts expressed in natural language, and real-world factual questions. Real-QA contains natural questions from ten public QA datasets, filtered to exclude those the base model already answers correctly. Composition helps consistently across the three datasets, which is evidence that our findings extend across different types of knowledge.

Retention under rephrased questions. To test whether retained knowledge remains accessible when the wording changes, we ran additional experiments with rephrased questions (see also our response to Reviewer 4BCS, W2). We evaluated 500 trained Real-QA associations, five per task, using Qwen3-4B-Base. Each association is queried with both the original question and a rephrased question, keeping the same target answer. A question-only generator produced the rewordings.

MethodOriginal questionRephrased question
Naive SFT1.3 ± 0.2%0.4 ± 0.3%
Replay14.3 ± 1.3%10.7 ± 0.9%
Replay + Merge49.9 ± 3.5%41.3 ± 3.9%
SI + SD + Replay + Merge46.2 ± 10.4%37.0 ± 7.1%
SI + Replay + Merge56.5 ± 10.9%44.7 ± 10.6%

Table 3: Accuracy (%) on the same 500 REAL-QA associations under original and rephrased questions after 100 tasks.

Our composed method reaches 44.7% on rephrased questions, compared with 0.4% for naive SFT and 10.7% for generative replay. This is evidence of retention beyond exact recall of the training queries.

Deployment in open-ended agent environments remains to be evaluated. What our study identifies are mechanisms that improve a necessary component of such systems: preserving acquired knowledge through continued learning. We have revised Section 1 to clarify this motivation and Section 5.1 to include these additional results.

Future work. We plan to combine the continual learning compositions identified through task-level successive halving with model-generated training data, and to evaluate them on continuous task streams without explicit task boundaries. This would test whether the retention benefits observed in our controlled setting extend to online adaptation, supporting personal agents that learn continually from ongoing interactions.

Such adaptation is complementary to expanding context windows, because it lets agents accumulate and retain knowledge within their parameters. This is how our observation on the composability of continual learning mechanisms can be applied to realistic settings.

[1] Ryan Wei Heng Quek et al. MeMo: Memory as a Model. 2026. arXiv:2605.15156.

[2] Adam Zweiger et al. Self-Adapting Language Models. 2025. arXiv:2506.10943.

W3: Novelty and the role of task-level successive halving

The novelty of our work is mainly twofold. First, we scale the horizon of memorization tasks to 100 in order to study long-term forgetting. Second, we study empirically how existing continual learning mechanisms compose.

Task-level successive halving (Section 4.3) adapts successive halving to the number of sequential tasks, which makes the combinatorial search manageable. It is therefore a supporting component of the research, used to seek preliminary evidence for our hypothesis.

Thanks for the additional experiments and explanations to address my concerns.

Thanks for taking the time to review our rebuttal and reconsider the score. We really appreciate it.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Area Chair and Reviewers

We thank the reviewers for their constructive feedback and for recognizing that our paper addresses an important problem in continual learning. Reviewers also highlighted these strengths:

  • An interesting 100-task setup, and a comprehensive study covering 90 configurations and a full 2⁴ factorial across three datasets (sYJo).
  • Useful insights into long-term forgetting, which is difficult to evaluate in shorter task sequences (4BCS).
  • A systematic study of how mechanisms work together, with the meaningful finding that replay and merged LoRA are complementary (4BCS, Mmfw).

We conducted the following new experiments and analyses for the rebuttal:

  • REMIX baseline: six additional 100-task Real-QA runs (sYJo, Mmfw).
  • Rephrased-question evaluation on 500 trained Real-QA associations (sYJo, 4BCS).
  • Replay+Merge interaction effect and replay coverage over 1782 saved replay batches (sYJo, Mmfw).
  • Retention on the first ten tasks at horizons of 10, 20, 50, and 100 (Mmfw).
  • Math rehearsal: 18 additional 10-task Real-QA runs, comparing three methods with and without GSM8K rehearsal over three seeds (4BCS).

Below, we summarize our responses to the main concerns and questions:

Motivation and novelty (sYJo, 4BCS, Mmfw). Our setting is a controlled abstraction of a practical requirement for continually learning agents: newly acquired knowledge must remain accessible after many subsequent updates. The individual mechanisms are established. Our contribution is a controlled study of whether their combinations remain effective over 100 sequential updates, and which interactions account for the gains.

Why replay and merged LoRA interact super-additively (sYJo, Mmfw). We hypothesize that replay provides the information to preserve, and merge provides the additional freedom (effective rank) to accommodate successive updates. At 100 tasks, the interaction is 11.7, 23.3, and 33.0 percentage points on Symbol-QA, LLM-QA, and Real-QA. Replay+Merge also recovers more distinct past question–answer pairs in its replay batches on all three datasets.

Baselines and REMIX (sYJo, Mmfw). Our REMIX adaptation exceeds 97% immediate acquisition on Real-QA, but retains at most 2.1% after 100 tasks, compared with 54.8% for SI+Replay+Merge. We have also revised the abstract to report the improvement over generative replay (8.1% to 34.9%) in place of the "28-fold improvement" statement.

Rephrased questions (sYJo, 4BCS). SI+Replay+Merge reaches 44.7% on rephrased questions, compared with 0.4% for naive SFT and 10.7% for generative replay. Most retained associations therefore remain accessible after rewording.

What the 100-task horizon adds (Mmfw). For Replay+Merge, retention on the first ten Real-QA tasks falls from 81.9% after ten tasks to 9.5% after 100. Short sequences can therefore substantially overstate the persistence of acquired knowledge.

Pretrained capabilities (4BCS). O-LoRA preserves more general capability than merged LoRA, but scores remain well below the base model. We also conducted 10-task Real-QA experiments with math rehearsal over three seeds. For Replay+Merge, rehearsing GSM8K training examples raises MATH accuracy from 20.6% to 46.2%, MGSM from 14.6% to 49.0%, and MMLU-Redux from 46.9% to 57.8%, while memorization retention changes from 81.7% to 83.2%. We have added these results to Section 5.1 and Appendix A.6.5, and retained capability preservation as an open challenge in Section 6.

All new results and revisions are included in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.