- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the forgetting problem in a long-horizon continual learning of LLMs.They adopted LoRA method and conbines multiple methods to address the forgetting problem. They combined: SFT, regularization(anchor), and merge LoRA. They set 100 continual tasks and provide the performance under different combination of methods. Their results show that combining multiple methods outperforms a single method.
Critical strengths
-
The provide the performance under different combination of mechanisms. The experiments are comprehensive.
-
The resulrts supports a meaningful conclusion that data replay and merge lora outperformns other combination of two methods.
Critical weaknesses
-
The motivation is not clear to me.
-
The baseline is too weak.
-
Lack of explanation.
Details see "Questions".
Questions for authors
-
The motivation for the long-horizon setting is unclear to me. First, repeatedly updating shared LLM parameters introduces a risk of interference and forgetting. What practical scenario motivates sequential SFT over 100 tasks, and why is this an appropriate strategy for long-term knowledge accumulation? Second, what is distinctive about the long-horizon regime beyond the increased number of updates? Which conclusions specifically depend on evaluating 100 tasks, and would these conclusions differ from those drawn from shorter sequences? Explicit comparisons across horizons would help establish the insights gained from this setting.
-
In this paper, there are 100 tasks. It is no surprising that the baseline is weak(naive fine-tuning). Therefore, the statement "a 28-fold improvement" sounds tricky for me. There are plenty of continual llm studies, and I think the auther should establish a stronger baseline.
-
Is there any explanation on the mechanism combination? For example, Data + Merge is stronger than other combination of two. Can you explain the reason?
4 comments
W2, Q2: Stronger baselines and the "28-fold improvement" statement
We thank the reviewer for raising the concern about the "28-fold improvement" statement. Our paper already reports eight baselines (Naive SFT w/ shared LoRA, SFT w/ merged LoRA, SFT w/ O-LoRA, SFT w/ OSRM, Generative Replay, Elastic Weight Consolidation, Synaptic Intelligence, LwF/Self-Distillation), as well as their compositions, in Section 5.1 and Appendix A.5.2. We have revised the abstract to report the improvement over the strong baseline generative replay instead: from 8.1% to 34.9% average final retention.
We have also added a REMIX comparison to Section 5.1 and Appendix A.6.1. Across three seeds on Real-QA, the REMIX adaptations reach 1.4 ± 0.1% final retention with shared LoRA and 2.1 ± 0.2% with merged LoRA. Replay+Merge reaches 48.2 ± 2.8%, and SI+Replay+Merge reaches 54.8 ± 10.4%. The REMIX adaptations exceed 97% immediate acquisition, so their low retention after 100 tasks is not due to a failure to learn the current tasks.
W3, Q3: Why Data + Merge is stronger than other combinations
Replay and low-rank allocation act on different parts of an update. Replay retains training signal about earlier associations, while allocation determines how successive updates are represented. Shared LoRA repeatedly modifies a single rank-constrained update (Sections 3.2 and 3.3). Merged LoRA instead incorporates each task's update into the backbone and initializes a fresh adapter for the next task, so the accumulated change is not restricted to one rank-r adapter.
We hypothesize that the two mechanisms complement each other: replay provides the information to preserve, and merge provides the additional freedom (effective rank) to accommodate successive updates. We have revised Section 5.3 to include this explanation and the replay coverage analysis below.
Interaction. We compute the interaction effect of replay and merge as
where A is the mean accuracy over all tasks seen at the evaluated horizon, with SI and SD absent. At 100 tasks, the interaction is 11.7 ± 5.6 percentage points on Symbol-QA, 23.3 ± 6.5 on LLM-QA, and 33.0 ± 2.3 on Real-QA.
Replay coverage. To test this hypothesis, we analyzed 1782 saved replay batches across the three datasets, comparing Replay and Replay+Merge over three seeds. Each batch contains 300 generation attempts. The table below reports the number of distinct, correctly reproduced past question–answer pairs divided by those 300 attempts.
| Dataset | Replay | Replay + Merge |
|---|---|---|
| Symbol-QA | 27.9 ± 0.3% | 32.1 ± 2.4% |
| LLM-QA | 33.5 ± 3.6% | 37.6 ± 5.3% |
| Real-QA | 22.9 ± 0.3% | 24.8 ± 0.8% |
Table 2: Recovery of earlier associations in generated replay. Entries are distinct correct past pairs divided by 300 generation attempts (%), averaged over all 99 replay events in each run.
We average over all 99 replay-generation events in each 100-task run, and report the mean ± sample SD across three seeds. Replay begins at task 2, because no earlier tasks are available when learning task 1. A pair is correct if it exactly matches the canonical stored question and answer.
On Real-QA, the fraction of previous tasks represented by at least one correct replay pair also increases, from 19.6 ± 0.5% to 27.4 ± 0.6%. Merge is therefore associated with broader recovery of earlier associations across the full training sequence.
Dear Reviewer Mmfw,
Thank you again for your constructive review. We wanted to follow up before the author-reviewer discussion period closes.
In our response, we addressed your main concerns about the motivation for the long-horizon setting, the strength of the baselines, and why replay and merged LoRA work well together. In particular, we described the scenario motivating sequential SFT and added a comparison across horizons of 10, 20, 50, and 100 tasks. We revised the abstract to report the improvement over generative replay (8.1% to 34.9%) rather than over naive SFT, and added a REMIX comparison on top of the eight baselines already in the paper. We also added an interaction analysis and a replay-coverage analysis supporting our explanation that replay provides the information to preserve, while merging provides additional effective rank to accommodate successive updates.
Could you please let us know whether these explanations address your concerns? We would be happy to provide further clarification during the remaining discussion period. If our responses have resolved your concerns, we would be grateful if you would consider updating your assessment.
Thank you for your time and consideration.
Dear Reviewer Mmfw,
Please carefully review the authors' rebuttal and respond to each of your concerns. If any concerns remain unresolved, clearly specify what is missing and request concrete clarifications or additional evidence. In particular, if the baselines (references) are insufficient, please suggest specific baseline methods for comparison.
Bests,
Your AC.
We thank the reviewer for the constructive feedback, and for recognizing the comprehensiveness of the experiments and the benefit of combining replay with merged LoRA. We address the motivation, the baseline comparison, and the mechanism explanation below.
W1, Q1: Motivation for sequential SFT and what the 100-task horizon adds
Motivation. A practical scenario is a personal agent that periodically incorporates information from new interactions, documents, and experiences into a persistent memory model. Each update must acquire the newly available information while preserving the knowledge accumulated through earlier interactions.
Recent work gives concrete examples of this direction. MeMo [1] internalizes knowledge in a dedicated memory model queried by a frozen reasoning model, and SEAL [2] uses model-generated training data to incorporate new information through parameter updates. SEAL explicitly reports forgetting under repeated self-updates, which highlights retention as an unresolved obstacle to continual knowledge accumulation.
Why sequential SFT. Our sequential SFT setting abstracts this process into successive batches of newly arriving knowledge. SFT is a direct mechanism for incorporating these batches into parameters, particularly when repeatedly training on the complete historical corpus is impractical or earlier examples are unavailable.
We study which combinations of continual learning mechanisms allow repeated acquisition while limiting the forgetting of earlier knowledge. MeMo uses a dedicated memory model, so this question is relevant even when the agent's main reasoning model remains frozen.
Why 100 tasks. We use 100 tasks to examine retention across a sustained sequence of acquisition events. Memorization in real-world applications is long-horizon, and we think a 100-task setting reflects these challenges.
Explicit horizon comparison. We analyzed runs at horizons of 10, 20, 50, and 100 tasks. The table below shows retention on the first ten tasks for Replay+Merge.
Table 1: Replay+Merge retention (%) on the first ten tasks at four horizons. The evaluated associations stay fixed as training continues.
What the longer horizon reveals. This comparison shows why the longer task sequence matters. Tracking the same first ten tasks separates the durability of earlier knowledge from performance on newly acquired information. For Replay+Merge, retention on these Real-QA tasks falls from 81.9% after ten tasks to 9.5% after 100.
Strong retention over a short sequence can therefore substantially overstate the persistence of acquired knowledge. The longer evaluation also lets us measure memory lifetime and track retention as training continues (Section 5.2).
The horizon comparison also clarifies which conclusions are already visible in shorter runs. The Replay+Merge interaction is positive at horizon 10, and method rankings are strongly correlated across horizons. These qualitative findings therefore do not depend exclusively on evaluating 100 tasks.
What the longer horizon establishes is the severity of accumulated forgetting and the magnitude of compositional effects over continued learning. On Real-QA, the interaction estimate rises from 9.8 at horizon 10 to 15.5, 30.9, and 33.0 at horizons 20, 50, and 100, respectively. The interaction trend differs across datasets. We have revised Section 5.2 and Appendix A.6.4 to include these comparisons.
Future work. Building on these findings, we plan to combine the compositions identified through task-level successive halving with model-generated training data, and to evaluate them on continuous streams without explicit task boundaries. This would test their applicability to online adaptation in personal agents, complementing approaches based on expanding context windows. We leave these experiments to future work, because the present study focuses on demonstrating the benefits of composing complementary continual learning mechanisms.
Our findings suggest a broader design principle: new continual learning methods should be developed and evaluated with attention both to their standalone effectiveness and to their interactions with existing mechanisms.
[1] Ryan Wei Heng Quek et al. MeMo: Memory as a Model. 2026. arXiv:2605.15156.
[2] Adam Zweiger et al. Self-Adapting Language Models. 2025. arXiv:2506.10943.