acceptodds
Under review as a conference paper at ICLR 2027

CLoRE: Drift-Controlled Continual Learning of LLMs via Expert Expansion with Reservoir Routing

Abstract

Continual learning of large language models (LLMs) by mixture-of-experts (MoE) expansion freezes earlier experts and adds new ones, which in principle prevents forgetting. However, the router is shared, and new experts added to it can displace or down-weight an earlier input's experts. Most prior methods constrain routing during or after training. Yet what must be preserved is the representation, which can drift as soon as new experts are added. We propose CLoRE, which controls this representational drift without storing past data. Concurrent router alignment trains the router at every step with the language-modeling loss, matching no previous router. Reservoir routing initializes new experts and their router rows so that expansion preserves the softmax denominator in expectation. Self-generated replay steers each question toward an earlier task by routing only its first token through that task's experts, and replays the backbone. On TRACE with two LLMs (e.g., Llama-3.1 and Qwen3) and on an MoE trained from scratch, CLoRE outperforms every baseline in average accuracy and forgetting, even one using original data, and largely preserves the backbone's general abilities.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, catastrophic forgetting, mixture-of-experts, large language models, representation drift, expert expansion, routing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eoPVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper develops the CLoRE algorithm to address continual learning when the base model is an LLM. The idea is based on expert expansion in mixture-of-experts (MoE) framework. The paper argues that freezing old MOE experts does not preserve the LLM computations for earlier inputs and hence will generate forgetting. The reason is that the shared router changes whenever new experts and router rows are added as more tasks are learned. As a result, representational drift emerge before any corrective training. CLoRE controls this drift during training and inference to enable CL and mitigates forgetting using synthetic experience replay. During training, concurrent router alignment updates the router at every training step together with new experts based on new-task inputs and self-generated replayed inputs are used to mitigate forgetting. At expansion step, reservoir routing reserves the softmax mass of non-included experts through a single reservoir. New experts are drawn from this reservoir with small noise and start with zero output, so the softmax denominator is preserved. Self-generated replay follows a Magpie-style procedure and the model writes questions from the chat template alone for repla. Experiments on 8B models show CLoRE reaches the best accuracy and lowest forgetting compared to the baselines with no-stored-data stored with low forgetting. Ablation experiments show that all main ideas contribute to an optimal performance.

Critical strengths
  1. The paper identifies an underexplored failure mode in expansion-based continual learning. CLoRE reframes the problem around representational drift and router-induced rerouting, which is a more accurate target for intervention compared to past work.

  2. Reservoir routing is the paper’s strongest technical contribution. Instead of initializing new router rows randomly or to zero and then hoping training repairs the damage, CLoRE reserves the future experts' share of the softmax denominator before those experts exist.

  3. CLoRE trains the router at every step together with new experts, using the language-modeling loss on new-task and replayed inputs. This avoids two common limitations: post-hoc alignment and distillation toward a previous router.

  4. CLoRE needs replay for router alignment but does not store original earlier-task data. It generates questions from the chat-template prefix alone. The rest of the sequence is generated with free routing.

  5. The paper does not just report final scores. It isolates drift at expansion, drift during training, router loss choice, reservoir routing, and self-generated replay.

Critical weaknesses
  1. CLoRE’s replay is generated by the model itself, not stored data. This is appealing, but it means the router is aligned to the model's current behavior on inputs that the model itself produced. If the model generates low-quality, repetitive, or off-domain questions for an earlier task, the router is trained to preserve behavior on that flawed distribution. The routing override at the first token steers the domain, but it does not guarantee coverage of the earlier task’s true input distribution. Over many tasks, errors could compound. The paper shows examples where generated questions reproduce task instructions, but it does not measure replay diversity, coverage, or failure cases.

  2. The empirical evaluation is narrow. It uses TRACE, which has eight text tasks, and Wiki→Code→Conversation, which has three domains. There are only two small backbones. There is no evaluation on multilingual data, long-context tasks, code execution, tool use, or safety-critical domains. The paper claims general continual learning of LLMs, but the evidence is mostly text classification, multiple-choice, math word problems, and next-token prediction. It is unclear whether the same drift-control mechanisms work for other domain.

  3. Some baselines are reimplemented on the LoRA-MoE architecture and hence are not compared with their original architectures.

  4. CLoRE introduces several hyperparameters but does not provide a systematic sensitivity analysis. It is unclear the improvement margins are easy to obtain with delicate hyperparameters tuning.

  5. CLoRE reports low backbone forgetting on MMLU, GSM8K, and PIQA but the routing analysis in Table 8 shows that the skip fraction for backbone inputs falls from 61.4% after the first task to 29.3% after the last. In other words, the router increasingly sends backbone inputs to added experts rather than the skip experts. Accuracy remains stable on those three benchmarks, but this does not prove that the backbone's general capability is preserved in unmeasured areas.

  6. CLoRE’s expert pool grows with each task. The router must score over all experts and skip experts, and the reservoir adds one logit per layer. The paper says the number of expansions is unbounded at the cost of one extra logit per layer, but inference still requires routing over a growing pool. It is unclear how CLoRE behaves at large number tasks.

  7. The code doesn't seem to be included which makes judgments about reproducibility challenging.

Questions for authors
  1. How sensitive is CLoRE to the quality and diversity of self-generated replay?

  2. Can CLoRE scale to many more tasks and larger expert pools?

  3. Is backbone preservation truly general, or only benchmark-specific? The skip fraction for backbone inputs drops sharply, yet MMLU, GSM8K, and PIQA remain stable. Does this mean the backbone’s behavior is preserved, or that these benchmarks are not sensitive to the changes?

  4. Many baselines are reimplemented on CLoRE's LoRA-MoE architecture, and hyperparameters are chosen by the authors. Would the results hold if baselines were tuned with equal effort, given the same inference budget, and evaluated on their original architectures?

5 comments
AuthorsOpenReview ↗

Response to Reviewer eoPV (1/3): W1

W1 / Q1. “…it does not measure replay diversity, coverage, or failure cases.” / “How sensitive is CLoRE to the quality and diversity of self-generated replay?”

Before each of the seven expansions on TRACE, we compared the replay generated for the first task, C-STANCE (500 questions), with the same number of its real training questions. Questions are embedded with the frozen backbone. Fidelity [1] is the share of generated questions that lie within the k-nearest-neighbor radius (k = 5) of some real question, and coverage [2] is the share of real questions with a generated question within their radius. Unique is the share of generated questions that are distinct, and distinct-2 [3] is the share of distinct bigrams among all bigrams; lower values of either indicate more repetition.

Replay (C-STANCE)FidelityUniquedistinct-2CoverageAA ↑F ↓
Real data0.77 → 0.771.00 → 1.000.47 → 0.470.81 → 0.8163.090.28
Generated, with override (CLoRE)0.72 → 0.611.00 → 1.000.42 → 0.410.78 → 0.7359.442.84
Generated, without override0.73 → –0.99 → –0.43 → –0.78 → 0.0057.027.23

The table compares real data, CLoRE’s generated replay with the routing override, and generated replay without it; each cell shows the value at the first → last expansion. With the routing override, the generated replay keeps covering C-STANCE: coverage changes little over the sequence (0.78 → 0.73), and repetition stays at the level of real data (unique 1.00 vs. 1.00 and distinct-2 0.41 vs. 0.47 at the last expansion). Fidelity decreases slightly (0.72 → 0.61). Without the override, coverage falls to 0.00 and the other metrics are not even defined at the last expansion, because no C-STANCE question is generated at all; this is the case for four of the seven earlier tasks. Coverage, not fidelity, is what carries over to performance (Q1): with the override, CLoRE forgets far less (F 2.84 vs. 7.23) and reaches a higher AA (59.44 vs. 57.02).

[1] Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019

[2] Reliable Fidelity and Diversity Metrics for Generative Models, ICML 2020

[3] A Diversity-Promoting Objective Function for Neural Conversation Models, NAACL 2016

AuthorsOpenReview ↗

Response to Reviewer eoPV (2/3): W2, W3, W4

W2. “The empirical evaluation is narrow… There is no evaluation on multilingual data, long-context tasks, code execution, tool use, or safety-critical domains.”

Following the reviewer’s suggestion, we prepended two tasks, KMMLU-ko [1] and KodCode [2], to the TRACE sequence. They add a new language (Korean) and execution-scored code generation. CLoRE retains the score each task reached right after it was first learned, through all subsequent tasks (KMMLU-ko: 42.65 → 42.70; KodCode: 67.20 → 65.80). EWC, the strongest baseline without stored data, learns both tasks about as well (42.65 and 65.00) but loses much of them over the same sequence (KMMLU-ko: 42.65 → 31.90; KodCode: 65.00 → 8.00). Over the full ten-task sequence, CLoRE reaches AA 57.56 and F 1.51, against 50.90 and 13.05 for EWC. We thank the reviewer for this suggestion and will add these results with broader exploration to the revision to further strengthen the paper.

MethodKMMLU-koKodCode
CLoRE42.65 → 42.7067.20 → 65.80
EWC42.65 → 31.9065.00 → 8.00

W3 / Q4. “Some baselines are reimplemented on the LoRA-MoE architecture and hence are not compared with their original architectures.” / “Would the results hold if baselines were tuned with equal effort, given the same inference budget, and evaluated on their original architectures?”

We first clarify that only two baselines, Lifelong-MoE and MoE-LPR, are reimplemented for TRACE. The other baselines use their original implementations, and EWC uses the implementation in the official TRACE repository (Table 1). Lifelong-MoE and MoE-LPR are originally FFN-MoE methods and cannot be run on TRACE without adaptation, so we reimplemented them on the same LoRA-MoE architecture as CLoRE. Furthermore, we compared them in their original setting, continual pre-training of an FFN-MoE on Wiki→Code→Conversation (Table 2). For the two reimplemented methods, Lifelong-MoE and MoE-LPR, we started from the hyperparameters recommended in their original papers and tuned them with the same search budget as CLoRE, reporting the best configuration. The inference budget is also matched: the two reimplemented methods activate the same number of adapter parameters per token as CLoRE (169M).

W4. “CLoRE introduces several hyperparameters but does not provide a systematic sensitivity analysis. It is unclear the improvement margins are easy to obtain with delicate hyperparameters tuning.”

We conduct three sets of experiments to measure the sensitivity to σ and η, (m, k), and the replay size N, using Llama-3.1 on TRACE. The results in the tables below show that the gains in forgetting hold across configurations: compared with EWC, the strongest baseline without original data (AA 58.69, F 7.48, F₀ 12.93), F is below EWC’s in all settings shown, and F₀ in every setting where it was measured, whereas AA stays comparable.

Varying the reservoir noise σ (at η = 0.01) and the failure probability η (at σ = 0.014), with m = k = 1 and N = 500.

σ0.00350.0070.0140.0140.014
η0.010.010.010.050.1
AA ↑57.2757.6660.2659.4156.86
F ↓6.535.922.373.624.39
F₀ ↓1.211.40−0.050.710.76

Varying the number of experts added per task (m) and selected per token (k), with active parameters per token (k × rank) held equal; σ = 0.014, η = 0.01. (1, 1) is the same run as σ = 0.014, η = 0.01 above.

(m, k)(1, 1)(2, 2)(4, 2)(4, 4)
Rank64323216
AA ↑60.2659.0856.3857.25
F ↓2.373.763.673.06

Varying the number of self-generated replay examples per task (N), in the default setting (σ = 0, m = k = 1). N = 500 is the run in Table 1.

N per task25050010002500
AA ↑58.0359.4459.1858.06
F ↓7.022.842.991.30

[1] KMMLU: Measuring Massive Multitask Language Understanding in Korean, NAACL 2025

[2] KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding, Findings of ACL 2025

AuthorsOpenReview ↗

Response to Reviewer eoPV (3/3): W5, W6, W7

W5 / Q3. “…the skip fraction for backbone inputs falls from 61.4% after the first task to 29.3% after the last… this does not prove that the backbone’s general capability is preserved in unmeasured areas.” / “Is backbone preservation truly general, or only benchmark-specific?”

Following the reviewer’s concern, we evaluated the final TRACE checkpoint on three additional benchmarks: KMMLU [1] (Korean knowledge), MGSM [2] (math in 10 languages), and BFCL [3] (tool use).

Preservation is not specific to MMLU, GSM8K, and PIQA: on KMMLU, MGSM, and BFCL, CLoRE stays within 2.0 points of the backbone, while the baselines lose 6 to 24 points. The falling skip fraction does not indicate forgetting. Routing backbone inputs to skip is not always the best way to preserve the backbone’s predictions. As experts are added and the router must also serve more tasks, holding every backbone input on skip becomes a harder constraint, and routing some of them to added experts that leave their predictions intact can be the easier solution. Since the router is trained with the language-modeling loss on backbone replay, it takes whichever routing preserves the predictions, skip or added experts. The low backbone forgetting F₀ (Table 1) and the results on the additional benchmarks show that the predictions are indeed preserved.

MethodKMMLUMGSMBFCL
Backbone41.4646.2867.14
CLoRE40.4844.4065.14
EWC35.0122.4453.44
O-LoRA33.2032.6453.74

W6 / Q2. “…inference still requires routing over a growing pool. It is unclear how CLoRE behaves at large number tasks.” / “Can CLoRE scale to many more tasks and larger expert pools?”

We agree that the expert pool grows with each task. This is common to all expansion-based methods, which add parameters for every new task; what CLoRE itself adds on top is a single reservoir logit per layer.

Evaluating a much larger number of tasks is beyond what we could run in the rebuttal period. We examined this on the ten-task sequence of W2. A new expert stays out of the existing top-k as long as the reservoir logit lies below the logits of the selected experts. As the table below shows, the share of tokens that satisfy this margin does not shrink as tasks are added: it stays at 99.9% or above at every step.

Tasks learned12345678910
Tokens satisfying the margin (%)100.00100.0099.9699.9199.9799.9899.9699.9599.9499.96

Forgetting does not grow either: F is 1.51 over the ten tasks, and the two tasks learned first keep their scores through all subsequent tasks (W2). For larger expert pools, adding two or four experts per task instead of one keeps F between 3.06 and 3.76 (W4). At inference, the router computes one logit per expert, so routing cost grows linearly with the pool. Only the top-k experts are evaluated per token, so the cost of the experts themselves stays constant.

W7. “The code doesn’t seem to be included which makes judgments about reproducibility challenging.”

We will release the code in an anonymous repository as soon as possible and post the link in this thread. It will include a run script and a configuration file for each table.

[1] KMMLU: Measuring Massive Multitask Language Understanding in Korean, NAACL 2025

[2] Language Models are Multilingual Chain-of-Thought Reasoners, ICLR 2023

[3] The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models, ICML 2025

Reviewer_5qLdOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CLoRE, a continual learning framework for LLMs based on MoE expansion. The authors identify representational drift as an important source of forgetting. To address this issue, the proposed method combines three components: (1) concurrent router alignment, (2) reservoir-based initialization to mitigate changes in routing during expert expansion, and (3) self-generated replay that steers generation toward previous tasks through a first-token routing override. Experiments report improvements in average performance and forgetting over the selected baselines.

Critical strengths
  • The problem addressed in this paper is important and well-motivated.
  • The analysis of representational drift during expert expansion is interesting. Table 3 provides useful empirical evidence that the initialization of new router rows can substantially affect existing routing decisions, even before training begins.
  • The proposed method demonstrates encouraging improvements in mitigating forgetting and preserving the backbone's general capabilities.
Critical weaknesses
  1. My primary concern is the novelty of the proposed approach. Several recent works have investigated MoE-based continual learning, including expert expansion, routing drift, and the stability–plasticity trade-off. For example, LLaVA-DyMoE [1], StaR-MoE [2], and PASs-MoE [3] identify routing instability or drift as a source of catastrophic forgetting and propose mitigation strategies. Furthermore, concurrent router alignment resembles standard replay-based continual learning applied to the routing network. Although combining these components may be useful, the paper does not sufficiently clarify its novel contributions beyond existing approaches. More direct comparisons and stronger ablations are needed to substantiate its novelty.

  2. The theoretical analysis does not fully support the claimed preservation of representations. Section 3.4 and Appendix H show that the softmax denominator is preserved in expectation under the proposed stochastic initialization. However, preserving the expected denominator does not necessarily imply preservation of the expected normalized routing weights or layer outputs.

  3. The improvements in final average performance are relatively modest in some comparisons. For example, with Llama-3.1-8B-Instruct, CLoRE achieves an average score of 59.44, compared with 58.69 for EWC and 58.44 for sequential LoRA. Furthermore, results across multiple random seeds or measures of variability are not reported.

  4. Computational efficiency is insufficiently evaluated. CLoRE requires two training branches and the generation of replay data for previous tasks. However, comparisons of training time, FLOPs, peak GPU memory usage, replay generation overhead, inference latency, and total versus active parameter counts are missing.

[1] On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models, CVPR 2026.

[2] Stable Routing for Mixture-of-Experts in Class-Incremental Learning, arXiv 2026.

[3] PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning, ACL 2026.

Questions for authors

Please refer to the weaknesses discussed above.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 5qLd (1/3): W1

W1-(a): My primary concern is the novelty of the proposed approach. Several recent works have investigated MoE-based continual learning, including expert expansion, routing drift, and the stability–plasticity trade-off. For example, LLaVA-DyMoE [1], StaR-MoE [2], and PASs-MoE [3] identify routing instability or drift as a source of catastrophic forgetting and propose mitigation strategies.

We thank the reviewer for the careful reading. The key difference from [1] and [2] is that CLoRE addresses drift exactly where it arises, and with the target objective itself. [1] and [2] study MoE-based continual learning with expert expansion and reduce routing drift with a regularizer on the router: [1] keeps tokens whose routing scores are old-dominant or ambiguous on old experts, and [2] aligns the routing of Gaussian samples, drawn from the stored mean and variance of earlier classes’ router inputs, to their historical routing distributions. Neither addresses the drift caused by the expansion itself, so earlier outputs have already moved before their regularizers act. This matters because restoring outputs once they have moved is itself another optimization problem, which must recover the result of large-scale training; router training cannot fully restore them (§4.3, Table 4a; without the reservoir, F rises from 2.84 to 5.41, see our response to Reviewer oKGb, W2 / Q2). Both also constrain routing rather than predictions, so routing can stay close to the past while outputs still change (Table 4b). CLoRE closes both gaps: it prevents drift at the moment of expansion, and during training aligns the router with the target-task loss, concurrently with the new experts. Since [1] and [2] were designed for vision-language models and class-incremental classification, respectively, we are reimplementing them on our LoRA-MoE for TRACE and will add the results to this thread as soon as they are available. [3] is not expansion-based: it keeps a fixed expert pool and stabilizes the rank directions important to earlier tasks. It is therefore complementary to CLoRE and could be combined with it; we leave this exploration to future work.

W1-(b): Furthermore, concurrent router alignment resembles standard replay-based continual learning applied to the routing network. Although combining these components may be useful, the paper does not sufficiently clarify its novel contributions beyond existing approaches. More direct comparisons and stronger ablations are needed to substantiate its novelty.

We understand the reviewer’s concern as follows: adding router rows for new experts is like adding new classes in class-incremental learning, so standard replay for that setting, such as iCaRL [4], could simply be applied to the router. Our ablation in Tab. 4b already does this. As in iCaRL, the router retains its earlier routing by distillation from the previous router on replayed data, and the new rows learn from the new task. This variant stays closer to the previous routing than CLoRE does (31.60% vs. 37.49% changed selections), but the layer outputs change just as much (11.67% vs. 11.59%), and it forgets more (F 0.64 vs. 0.40).

This is because a router is not a classifier. A classifier’s output is the prediction itself, so preserving its outputs preserves the prediction. A router’s output only decides how the experts are mixed in each layer, and even a small change in this mix shifts the representation passed to later layers. As discussed in W1-(a), matching the earlier routing therefore does not guarantee the same prediction. CLoRE instead trains the router directly on the target-task loss, so that the prediction is what is preserved.

[1] On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models, CVPR 2026.

[2] Stable Routing for Mixture-of-Experts in Class-Incremental Learning, arXiv 2026.

[3] PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning, ACL 2026.

[4] iCaRL: Incremental Classifier and Representation Learning, CVPR 2017

AuthorsOpenReview ↗

Response to Reviewer 5qLd (2/3): W2, W3

W2. “…preserving the expected denominator does not necessarily imply preservation of the expected normalized routing weights or layer outputs.”

We agree. The abstract, the introduction, and §3.4 state that the softmax denominator is preserved in expectation, which is what Appendix H proves. The caption of Figure 2(b), Appendix E.1, and the description of Algorithm 1 instead say that outputs are preserved. That was not our intent, and we will correct these three places as follows.

  • Which results have noise. All TRACE results in the paper (Tables 1 and 5, and the TRACE columns of Table 3) use σ = 0. Only the Wiki→Code→Conv results (Tables 2 and 4, and the corresponding columns of Table 3) use σ > 0.
  • σ = 0. The denominator is preserved exactly. The top-k set, and hence the output, is preserved exactly for tokens that satisfy the margin condition.
  • σ > 0. The denominator is preserved in expectation. The top-k set is preserved with probability at least 1 − η for tokens that satisfy the margin condition. The expected weights and outputs are not preserved exactly.
  • How large the gap is. We measured it on TRACE, feeding real inputs of earlier tasks to the checkpoint right before each expansion and repeating the expansion with 64 noise draws.
σ = 0σ > 0 (12 settings)
E[Z_after / Z_before]1 (exact)1 ± 10⁻⁵
Bias of expected weights, E[p_after / p_before] − 10+1.1 × 10⁻⁵ to +1.9 × 10⁻⁴
P(new expert enters top-k) for tokens satisfying the margin0≤ 3.4 × 10⁻⁵
Tokens satisfying the margin99.93%99.75–99.86%

The σ > 0 settings cover σ ∈ {0.0035, 0.007, 0.014}, η ∈ {0.001, 0.01, 0.05, 0.1}, and m ∈ {1, 2, 4} experts added per task. In the Wiki→Code→Conv setting itself, expansion changes 1.08% of top-k selections and moves the layer output by 0.85% (Table 3). We will add these measurements to Appendix H.

W3. “The improvements in final average performance are relatively modest in some comparisons… results across multiple random seeds or measures of variability are not reported.”

We agree that the AA gain over the strongest baselines is modest. In continual learning, however, AA alone does not capture the goal: a method can reach a similar AA by fitting later tasks well while forgetting earlier ones, which is why forgetting is reported alongside it. To measure variability, we repeated every method three times on both backbones and report the mean ± std.

Over three runs, CLoRE’s mean AA is the highest on both backbones. While its AA margin over EWC and Seq LoRA lies within the run-to-run variation, its margin in forgetting lies far beyond it: F is 2.93 on Llama and 1.43 on Qwen, against 8.76 and 9.06 for EWC, a reduction of 67% and 84%, and its range does not overlap with that of any baseline on either backbone. CLoRE also has the lowest backbone forgetting F₀ (Table 1). With past data, CLoRE improves over MoE-LPR in both AA (+4.1 on both backbones) and F, with no overlap in either. The revision will report these results in Table 1, and we are running additional seeds for the other experiments to report them as mean ± std as well.

MethodPast dataLlama AA ↑Llama F ↓Qwen AA ↑Qwen F ↓
Seq LoRA✗57.05 ± 1.69.78 ± 1.758.37 ± 0.610.20 ± 1.0
EWC✗57.83 ± 0.88.76 ± 1.059.27 ± 1.09.06 ± 1.1
O-LoRA✗49.22 ± 2.17.52 ± 2.551.81 ± 1.46.96 ± 1.3
S-LoRA✗55.36 ± 1.411.46 ± 1.450.79 ± 2.418.73 ± 3.1
Lifelong-MoE✗33.05 ± 2.518.32 ± 3.037.10 ± 7.815.89 ± 8.7
CLoRE✗58.97 ± 0.72.93 ± 1.161.90 ± 2.31.43 ± 0.9
MoE-LPR✓58.28 ± 1.73.32 ± 0.759.59 ± 1.24.52 ± 1.7
CLoRE✓62.35 ± 0.7−0.96 ± 0.963.68 ± 0.60.18 ± 0.6
AuthorsOpenReview ↗

Response to Reviewer 5qLd (3/3): W4

W4. “Computational efficiency is insufficiently evaluated… comparisons of training time, FLOPs, peak GPU memory usage, replay generation overhead, inference latency, and total versus active parameter counts are missing.”

We measured all of these for Llama-3.1-8B-Instruct on TRACE, at the last expansion. Among MoE-based methods, CLoRE’s training cost is comparable: with past data, it is faster than MoE-LPR (3.75 vs. 4.38 h) with fewer FLOPs, and its peak memory (40.6 GB) lies between those of MoE-LPR (37.3 GB) and Lifelong-MoE (55.9 GB). Without past data, replay generation adds 2.89 h. All MoE-based methods cost more than non-MoE methods, which use 31.5–33.9 GB, because of routing. Active parameters per token increase by only 2% (8.20B vs. 8.03B), while total parameters grow linearly with the number of tasks, as in any expansion-based method; the reservoir adds a negligible single row per router. After the eight tasks, the total is 9.37B for CLoRE and MoE-LPR and 8.99B for Lifelong-MoE, which shares one attention adapter across tasks. At inference, non-MoE adapters can be merged into the backbone (16.0 ms/token), whereas MoE-based methods must run the selected experts and the router (46.7 ms/token). We will discuss these costs as limitations in the revision.

Training cost (Llama-3.1-8B-Instruct)

MethodPast dataTrain time (h)Train FLOPs (×10¹⁸)Peak mem. (GB)Active params / tokenTotal params
Non-MoE (adapters merged)
Seq LoRA✗2.185.532.48.03B8.03B
EWC✗2.685.933.98.03B8.03B
O-LoRA✗2.015.031.58.03B8.03B
S-LoRA✗6.475.531.68.03B8.03B
MoE-based
Lifelong-MoE✗4.017.255.98.20B8.99B
CLoRE✗6.64 (3.75 + 2.89)6.8 (6.7 + 0.12)40.68.20B9.37B
MoE-LPR✓4.386.937.38.20B9.37B
CLoRE✓3.756.740.68.20B9.37B
Reviewer_oKGbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual learning of LLMs through expert expansion. Its central observation is that freezing previously learned experts does not preserve earlier behavior: newly added router rows can change both top-k expert selection and the softmax weights assigned to existing experts. The proposed method, CLoRE, combines three components: concurrent training of new experts and the router using a language-modeling objective; reservoir routing, which reserves softmax denominator mass for future experts and initializes their outputs to zero; and self-generated replay, with task-specific generation induced by overriding expert selection at the first generated token. Replay also includes inputs intended to preserve the backbone’s general capabilities. On the eight-task TRACE sequence, CLoRE achieves final average scores of 59.44 and 61.77, with forgetting of 2.84 and 0.79, on Llama-3.1-8B-Instruct and Qwen3-8B, respectively. The paper additionally reports experiments on an MoE trained from scratch, measurements of expansion-time drift, and component ablations.

Critical strengths
  1. The paper identifies a concrete failure mechanism in expert-expansion methods.Section 3.2 distinguishes expert displacement from changes caused by the shared softmax denominator. This explains why freezing both earlier experts and their router rows does not necessarily preserve the model’s function, and yields a directly testable research question.
  2. Reservoir routing provides a technically identifiable contribution.Equation (3) and Appendix H explain how reserved denominator mass is transferred to newly instantiated experts. Combined with zero-output initialization and a routing-margin condition, this substantially reduces expansion-time drift. Table 3 provides direct evidence: on TRACE, the proportion of tokens with changed expert selections decreases from 66.20% under random initialization to 0.37%.
  3. The evaluation considers both task retention and backbone capability preservation.Table 1 shows a favorable balance across two backbones. On Llama, CLoRE preserves general capabilities substantially better than Seq LoRA and EWC while attaining competitive or better TRACE performance. Evaluating both dimensions makes the results more informative than reporting forgetting alone.
  4. Several ablations address meaningful design choices.Table 4b shows that closer agreement with historical routing does not necessarily produce better task performance. Tables 4a and 5a also demonstrate differences between concurrent and post-hoc alignment under the evaluated configurations.
Critical weaknesses
  1. Inconsistent experimental protocols undermine the data-free claim. Section 4.2 and Table 2 explicitly state that CLoRE uses self-generated replay in the from-scratch MoE experiment, whereas Appendix K describes replaying a fixed 0.1% subset of the original corpora, repeated 200 times. Appendix E.1 also introduces output-KL-based initialization, whose relationship to the zero-output initialization described in Appendix K is unclear. These inconsistencies prevent verification of the experiment underlying a central claim. If Table 2 uses original data, its performance comparison may remain valid, but it cannot support the claim that CLoRE outperforms original-data replay without storing historical data.
  2. The distinctive components are not sufficiently isolated in the primary self-generated-replay setting. The main component ablations in Table 5 use original-data replay, although the paper emphasizes learning without stored historical data. Removing the reservoir changes AA from 63.09 to 62.79 and forgetting from 0.28 to 0.58, but the corresponding comparison under self-generated replay is absent. Table 5c compares the routing override with a learned prompt, without including equal-budget generation without an override, and no ablation isolates backbone replay. Consequently, the results establish the effectiveness of the complete recipe more clearly than the independent contributions of reservoir routing and task-directed generation.
  3. Differences in replay coverage and optimization budgets limit the interpretation of the comparisons. CLoRE replays the backbone domain, whereas MoE-LPR replays previously learned tasks, so their backbone-retention comparison changes replay coverage together with the routing method. Appendix J also states that CLoRE processes separate expert and router batches at every step, while MoE-LPR performs post-hoc router tuning for 20% of the task’s update count. The concurrent-versus-post-hoc ablations do not clearly establish whether router updates and replay exposure are matched. These differences do not invalidate the reported performance, but weaken the attribution of the gains specifically to concurrent alignment or the choice of routing objective.
  4. The preservation claims require narrower qualification, and small downstream improvements lack uncertainty estimates. Appendix H establishes preservation of the expected softmax denominator, but for nonzero initialization noise this does not imply preservation of expected gating weights or outputs, because taking the reciprocal does not commute with expectation. The implemented TRACE setting uses σ=0, allowing exact denominator preservation at insertion, although output preservation still requires unchanged top-k selection. The manuscript should distinguish these guarantees more explicitly. In addition, the relatively small downstream differences in Tables 4b and 5b are reported without cross-run uncertainty, making their robustness difficult to assess.
  5. The two TRACE backbones belong to different model families, but both are 8B models evaluated with the same top-1, one-expert-per-task configuration. This limits the evidence for robustness across model scales and routing configurations. More importantly, Table 3 shows a substantial reduction in expansion-time drift, whereas Table 5b shows a comparatively small improvement in final average performance. A targeted analysis tracking drift and task performance immediately before expansion, after expansion, and after subsequent training would help establish whether insertion-time drift predicts lasting forgetting and when reservoir routing provides a meaningful benefit.
  6. The stored-data comparison is limited to MoE-LPR and CLoRE. This does not make the comparison inherently unfair, but it leaves unclear whether the proposed routing mechanisms improve over a simpler replay-based alternative. A baseline using the same historical examples and a comparable replay and optimization budget would help distinguish the benefit of CLoRE from that of replay itself. Differences in backbone replay coverage should also be made explicit.
Questions for authors
  1. Does Table 2 use self-generated replay or subsets of the original corpora? Was the KL initialization in Appendix E.1 applied?Please identify the configuration corresponding to the reported results. Confirmation that the discrepancy is a documentation error and that the experiment satisfies the claimed replay constraint would substantially alleviate the main concern.
  2. How does CLoRE perform without the reservoir under self-generated replay, and with generation that omits the first-token routing override?Equal-budget comparisons would clarify whether these distinctive components provide independent benefits in the primary setting.
  3. Are concurrent and post-hoc alignment matched in router updates and replay exposure?Clarification, or a budget-matched comparison if available, would affect the interpretation that alignment timing is the decisive factor.
  4. Can the preservation claims distinguish denominator preservation, top-k preservation, and output preservation?A precise statement for the implemented σ=0 setting, together with appropriately qualified claims for nonzero noise, would resolve most of this concern.
5 comments
AuthorsOpenReview ↗

Response to Reviewer oKGb (1/4): W1, W2

W1 / Q1. “Does Table 2 use self-generated replay or subsets of the original corpora? Was the KL initialization in Appendix E.1 applied?”

Both inconsistencies are documentation errors in the appendix. The experiments were run as the main text describes, and they satisfy the replay constraint.

  • Table 2 uses self-generated replay, as Section 4.2 states. The sentence in Appendix K about replaying a 0.1% subset of the original corpora 200 times was misplaced. Original-data replay is used only in the component analyses. Table 2 is therefore obtained without any original data and supports the data-free claim. We will correct Appendix K.
  • The KL initialization in Appendix E.1 was not applied in any reported experiment. It is an alternative we also developed, and the appendix described it without relating it to what CLoRE uses. Both aim at an unchanged output distribution at expansion. KL initialization achieves it through an extra optimization stage. The reservoir construction achieves it by design, at no extra cost: experts start at zero output and router rows are drawn from the reservoir. Every run that uses the reservoir uses it alone; KL initialization is never applied on top of it. We will state this relation in Appendix E.1.

W2 / Q2. “How does CLoRE perform without the reservoir under self-generated replay, and with generation that omits the first-token routing override?” / “…no ablation isolates backbone replay.”

We thank the reviewer for such constructive comments. We clarify that we intentionally chose original-data replay since generated replay data may be affected by modified components in each ablation. By fixing the replay to the original data, we could isolate the ablated components. To further explore the effects of reservoir routing, the first-token routing override, and the backbone replay components, we add the three ablations the reviewer asks for. They are run on TRACE with Llama-3.1-8B-Instruct. Replay is self-generated, as in our main setting without original data, and each result comes from a single seed.

SettingAA ↑forgetting F ↓backbone forgetting F₀ ↓
CLoRE (full)59.442.840.82
w/o reservoir routing58.485.412.53
w/o first-token routing override (same generation budget)57.027.231.21
w/o backbone replay (same budget)58.134.871.81

Each component helps for a different reason.

  • Reservoir routing. Without it, every expansion displaces earlier representations before training starts. Self-generated replay repairs only part of that damage (W5). AA falls by 0.96 points, while F and F₀ almost double (2.84 to 5.41; 0.82 to 2.53, respectively). The effect is larger than under original-data replay (F 0.28 → 0.58).
  • First-token routing override. Without it, generation drifts toward a few tasks: the share of real examples that have a generated example nearby falls from 0.92 at the first expansion to 0.38 at the last (0.93 to 0.73 with the override). At the last expansion, four of the seven earlier tasks receive no replay that covers them, so the router is never aligned for these tasks. AA falls by 2.42 points and F rises to 7.23.
  • Backbone replay. Without it, nothing tells the router how to treat general inputs, so the backbone’s predictions are not protected. Removing backbone replay raises not only backbone forgetting F₀ from 0.82 to 1.81 (e.g., GSM8K 74.53 → 71.42) but also task forgetting F to 4.87.

Each component, removed alone, increases forgetting or backbone degradation. Each therefore contributes independently in the primary setting.

AuthorsOpenReview ↗

Response to Reviewer oKGb (2/4): W3, W4

W3-(1). “CLoRE replays the backbone domain, whereas MoE-LPR replays previously learned tasks, so their backbone-retention comparison changes replay coverage together with the routing method.”

Correct, and we tested it by retraining MoE-LPR with backbone replay.

  • Setup. CLoRE replays the backbone domain. MoE-LPR, following its paper, replays earlier tasks only. We retrained MoE-LPR with backbone replay generated exactly as in CLoRE (original data of earlier tasks in all rows; a single seed).
MethodBackbone replayAA ↑F ↓F₀ ↓
MoE-LPR✗55.913.819.51
MoE-LPR + backbone replay✓57.936.1311.06
CLoRE✓63.090.280.94
  • Result. Backbone replay raises MoE-LPR’s AA. It does not improve backbone retention: F0F_0 goes from 9.51 to 11.06 and FF from 3.81 to 6.13. CLoRE’s backbone retention is therefore not explained by replay coverage.
  • Reason. We attribute MoE-LPR’s lack of gain to when it uses replay. It tunes the router after the experts are trained. By then the backbone’s representations have already moved, and tuning the router afterwards does not restore them. The same holds inside CLoRE: with backbone replay but post-hoc alignment, F0F_0 is 5.89 instead of 0.94 (Table 5a).

W3-(2) / Q3. “…MoE-LPR performs post-hoc router tuning for 20% of the task’s update count.” / “Are concurrent and post-hoc alignment matched in router updates and replay exposure?”

Yes, they are matched. The mismatch comes from a wording error in Appendix J.

  • Appendix J. “20% of the task’s update count” should read “20% of the task’s training data”. MoE-LPR’s post-hoc router tuning runs the corresponding number of training steps on this data. Its compute therefore matches CLoRE’s router alignment. We will correct Appendix J.
  • Tables 4a and 5a. The concurrent and post-hoc settings have the same number of router updates and see the same number of replay tokens. They differ only in when the router is aligned. The gap therefore reflects alignment timing, not compute or replay exposure. We will state this in both captions.

W4 / Q4. “Can the preservation claims distinguish denominator preservation, top-k preservation, and output preservation?” / “…the relatively small downstream differences in Tables 4b and 5b are reported without cross-run uncertainty…”

We agree. The abstract, the introduction, and §3.4 state that the softmax denominator is preserved in expectation, which is what Appendix H proves. The caption of Figure 2(b), Appendix E.1, and the description of Algorithm 1 instead say that outputs are preserved. That was not our intent, and we will correct these three places as follows.

  • Which results have noise. All TRACE results in the paper (Tables 1 and 5, and the TRACE columns of Table 3) use σ=0\sigma = 0. Only the Wiki→Code→Conv results (Tables 2 and 4, and the corresponding columns of Table 3) use σ>0\sigma > 0.
  • σ=0\sigma = 0. The denominator is preserved exactly. The top-kk set, and hence the output, is preserved exactly for tokens that satisfy the margin condition (99.93% of tokens).
  • σ>0\sigma > 0. The denominator is preserved in expectation. The top-kk set is preserved with probability at least 1−η1 - \eta for tokens that satisfy the margin condition (99.75–99.86% of tokens). The expected weights and outputs are not preserved exactly. On TRACE the bias is at most 2×10−42 \times 10^{-4} (our response to Reviewer 5qLd, W2). In the Wiki→Code→Conv setting, expansion shifts the layer output by 0.85% (Table 3).
  • Tables 4b and 5b. Both are single-run results. We will report means and standard deviations over multiple seeds in the final version.
AuthorsOpenReview ↗

Response to Reviewer oKGb (3/4): W5

W5-(1). “…both are 8B models evaluated with the same top-1, one-expert-per-task configuration. This limits the evidence for robustness across model scales and routing configurations.”

Correct for the TRACE backbones. Two results extend the evidence on routing configurations.

  • Table 2. It covers a different scale and configuration: a FLAME-MoE trained from scratch, with eight experts per task and top-4 routing.
  • Routing configuration on TRACE. We varied (m, k), the experts added per task and selected per token, with the active parameters per token held equal. Forgetting stays between 2.37 and 3.76. All four rows use σ = 0.014 and η = 0.01, so the (1, 1) row is not the σ = 0 run of Table 1.
(m, k)(1, 1)(2, 2)(4, 2)(4, 4)
Rank64323216
AA ↑60.2659.0856.3857.25
F ↓2.373.763.673.06
  • Model scale. Experiments on backbones larger than 8B could not be completed within the rebuttal period, and we will add those results in the final version.

W5-(2). “Table 3 shows a substantial reduction in expansion-time drift, whereas Table 5b shows a comparatively small improvement in final average performance.”

Both observations are correct, and one pattern explains them. The damage at expansion is large, most of it is repaired during the next task, and the rest remains as forgetting. Table 3 measures before this repair and Table 5b after it.

  • Setup. We ran the analysis the reviewer proposes. We conducted one run with the reservoir and one without it (new router rows initialized randomly), both with self-generated replay on TRACE with Llama-3.1-8B-Instruct and a single seed (AA / F of 59.44 / 2.84 and 58.48 / 5.41). At each of the seven expansions, and for every task learned so far, we measured drift and the task score before expansion, right after it, after training the new task, and at the end. This gives 28 (expansion, earlier task) pairs.
  • At expansion. Without the reservoir, hidden states change by 30–36% and top-k selections by 31–66%. With the reservoir these are at most 1.2% and 0.3%, in line with Table 3.
  • After training. Without the reservoir, the 8.0-point drop shrinks to 1.3 points while the router is aligned on replay. The repair is incomplete: the final drop is 3.84 points compared to 2.08.
Mean drop from before expansion (points, 28 pairs)Right after expansionAfter training the new taskFinal
With reservoir−0.130.792.08
Without reservoir8.011.343.84
  • Weaker repair, larger gap. Table 5b uses original-data replay, which repairs more. Removing the reservoir increases F from 0.28 to 0.58 there, compared to 2.84 → 5.41 under self-generated replay.

W5-(3). “A targeted analysis… would help establish whether insertion-time drift predicts lasting forgetting and when reservoir routing provides a meaningful benefit.”

Partly, and the benefit is largest for tasks whose damage is not repaired. Both answers come from the analysis in W5-(2).

  • Does insertion-time drift predict lasting forgetting? Without the reservoir, the score drop right after expansion correlates with final forgetting at r = 0.56 (r = 0.43 for the top-k change). The run with the reservoir forgets less in 19 of the 28 pairs. The pairs share tasks and checkpoints, so we read this as a trend.
  • When does the reservoir matter? MeetingBank collapses at expansion and recovers only in part: 64.41 → 41.15 → 63.61 → 49.94 without the reservoir, against 65.46 → 65.38 → 65.63 → 59.94 with it (after learning → right after expansion → after the next task → final). On average it finally loses 10.64 points without the reservoir and 5.16 with it. FOMC (5.17 vs. 2.52) and NumGLUE-cm (4.94 vs. 1.85) behave the same way. Py150 and ScienceQA also drop at expansion but recover almost fully. The average therefore understates the benefit.

We will add this analysis to the appendix and explain the relation between Tables 3 and 5b in §4.4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.