- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper develops the CLoRE algorithm to address continual learning when the base model is an LLM. The idea is based on expert expansion in mixture-of-experts (MoE) framework. The paper argues that freezing old MOE experts does not preserve the LLM computations for earlier inputs and hence will generate forgetting. The reason is that the shared router changes whenever new experts and router rows are added as more tasks are learned. As a result, representational drift emerge before any corrective training. CLoRE controls this drift during training and inference to enable CL and mitigates forgetting using synthetic experience replay. During training, concurrent router alignment updates the router at every training step together with new experts based on new-task inputs and self-generated replayed inputs are used to mitigate forgetting. At expansion step, reservoir routing reserves the softmax mass of non-included experts through a single reservoir. New experts are drawn from this reservoir with small noise and start with zero output, so the softmax denominator is preserved. Self-generated replay follows a Magpie-style procedure and the model writes questions from the chat template alone for repla. Experiments on 8B models show CLoRE reaches the best accuracy and lowest forgetting compared to the baselines with no-stored-data stored with low forgetting. Ablation experiments show that all main ideas contribute to an optimal performance.
Critical strengths
-
The paper identifies an underexplored failure mode in expansion-based continual learning. CLoRE reframes the problem around representational drift and router-induced rerouting, which is a more accurate target for intervention compared to past work.
-
Reservoir routing is the paper’s strongest technical contribution. Instead of initializing new router rows randomly or to zero and then hoping training repairs the damage, CLoRE reserves the future experts' share of the softmax denominator before those experts exist.
-
CLoRE trains the router at every step together with new experts, using the language-modeling loss on new-task and replayed inputs. This avoids two common limitations: post-hoc alignment and distillation toward a previous router.
-
CLoRE needs replay for router alignment but does not store original earlier-task data. It generates questions from the chat-template prefix alone. The rest of the sequence is generated with free routing.
-
The paper does not just report final scores. It isolates drift at expansion, drift during training, router loss choice, reservoir routing, and self-generated replay.
Critical weaknesses
-
CLoRE’s replay is generated by the model itself, not stored data. This is appealing, but it means the router is aligned to the model's current behavior on inputs that the model itself produced. If the model generates low-quality, repetitive, or off-domain questions for an earlier task, the router is trained to preserve behavior on that flawed distribution. The routing override at the first token steers the domain, but it does not guarantee coverage of the earlier task’s true input distribution. Over many tasks, errors could compound. The paper shows examples where generated questions reproduce task instructions, but it does not measure replay diversity, coverage, or failure cases.
-
The empirical evaluation is narrow. It uses TRACE, which has eight text tasks, and Wiki→Code→Conversation, which has three domains. There are only two small backbones. There is no evaluation on multilingual data, long-context tasks, code execution, tool use, or safety-critical domains. The paper claims general continual learning of LLMs, but the evidence is mostly text classification, multiple-choice, math word problems, and next-token prediction. It is unclear whether the same drift-control mechanisms work for other domain.
-
Some baselines are reimplemented on the LoRA-MoE architecture and hence are not compared with their original architectures.
-
CLoRE introduces several hyperparameters but does not provide a systematic sensitivity analysis. It is unclear the improvement margins are easy to obtain with delicate hyperparameters tuning.
-
CLoRE reports low backbone forgetting on MMLU, GSM8K, and PIQA but the routing analysis in Table 8 shows that the skip fraction for backbone inputs falls from 61.4% after the first task to 29.3% after the last. In other words, the router increasingly sends backbone inputs to added experts rather than the skip experts. Accuracy remains stable on those three benchmarks, but this does not prove that the backbone's general capability is preserved in unmeasured areas.
-
CLoRE’s expert pool grows with each task. The router must score over all experts and skip experts, and the reservoir adds one logit per layer. The paper says the number of expansions is unbounded at the cost of one extra logit per layer, but inference still requires routing over a growing pool. It is unclear how CLoRE behaves at large number tasks.
-
The code doesn't seem to be included which makes judgments about reproducibility challenging.
Questions for authors
-
How sensitive is CLoRE to the quality and diversity of self-generated replay?
-
Can CLoRE scale to many more tasks and larger expert pools?
-
Is backbone preservation truly general, or only benchmark-specific? The skip fraction for backbone inputs drops sharply, yet MMLU, GSM8K, and PIQA remain stable. Does this mean the backbone’s behavior is preserved, or that these benchmarks are not sensitive to the changes?
-
Many baselines are reimplemented on CLoRE's LoRA-MoE architecture, and hyperparameters are chosen by the authors. Would the results hold if baselines were tuned with equal effort, given the same inference budget, and evaluated on their original architectures?
5 comments
Response to Reviewer eoPV (2/3): W2, W3, W4
W2. “The empirical evaluation is narrow… There is no evaluation on multilingual data, long-context tasks, code execution, tool use, or safety-critical domains.”
Following the reviewer’s suggestion, we prepended two tasks, KMMLU-ko [1] and KodCode [2], to the TRACE sequence. They add a new language (Korean) and execution-scored code generation. CLoRE retains the score each task reached right after it was first learned, through all subsequent tasks (KMMLU-ko: 42.65 → 42.70; KodCode: 67.20 → 65.80). EWC, the strongest baseline without stored data, learns both tasks about as well (42.65 and 65.00) but loses much of them over the same sequence (KMMLU-ko: 42.65 → 31.90; KodCode: 65.00 → 8.00). Over the full ten-task sequence, CLoRE reaches AA 57.56 and F 1.51, against 50.90 and 13.05 for EWC. We thank the reviewer for this suggestion and will add these results with broader exploration to the revision to further strengthen the paper.
| Method | KMMLU-ko | KodCode |
|---|---|---|
| CLoRE | 42.65 → 42.70 | 67.20 → 65.80 |
| EWC | 42.65 → 31.90 | 65.00 → 8.00 |
W3 / Q4. “Some baselines are reimplemented on the LoRA-MoE architecture and hence are not compared with their original architectures.” / “Would the results hold if baselines were tuned with equal effort, given the same inference budget, and evaluated on their original architectures?”
We first clarify that only two baselines, Lifelong-MoE and MoE-LPR, are reimplemented for TRACE. The other baselines use their original implementations, and EWC uses the implementation in the official TRACE repository (Table 1). Lifelong-MoE and MoE-LPR are originally FFN-MoE methods and cannot be run on TRACE without adaptation, so we reimplemented them on the same LoRA-MoE architecture as CLoRE. Furthermore, we compared them in their original setting, continual pre-training of an FFN-MoE on Wiki→Code→Conversation (Table 2). For the two reimplemented methods, Lifelong-MoE and MoE-LPR, we started from the hyperparameters recommended in their original papers and tuned them with the same search budget as CLoRE, reporting the best configuration. The inference budget is also matched: the two reimplemented methods activate the same number of adapter parameters per token as CLoRE (169M).
W4. “CLoRE introduces several hyperparameters but does not provide a systematic sensitivity analysis. It is unclear the improvement margins are easy to obtain with delicate hyperparameters tuning.”
We conduct three sets of experiments to measure the sensitivity to σ and η, (m, k), and the replay size N, using Llama-3.1 on TRACE. The results in the tables below show that the gains in forgetting hold across configurations: compared with EWC, the strongest baseline without original data (AA 58.69, F 7.48, F₀ 12.93), F is below EWC’s in all settings shown, and F₀ in every setting where it was measured, whereas AA stays comparable.
Varying the reservoir noise σ (at η = 0.01) and the failure probability η (at σ = 0.014), with m = k = 1 and N = 500.
| σ | 0.0035 | 0.007 | 0.014 | 0.014 | 0.014 |
|---|---|---|---|---|---|
| η | 0.01 | 0.01 | 0.01 | 0.05 | 0.1 |
| AA ↑ | 57.27 | 57.66 | 60.26 | 59.41 | 56.86 |
| F ↓ | 6.53 | 5.92 | 2.37 | 3.62 | 4.39 |
| F₀ ↓ | 1.21 | 1.40 | −0.05 | 0.71 | 0.76 |
Varying the number of experts added per task (m) and selected per token (k), with active parameters per token (k × rank) held equal; σ = 0.014, η = 0.01. (1, 1) is the same run as σ = 0.014, η = 0.01 above.
| (m, k) | (1, 1) | (2, 2) | (4, 2) | (4, 4) |
|---|---|---|---|---|
| Rank | 64 | 32 | 32 | 16 |
| AA ↑ | 60.26 | 59.08 | 56.38 | 57.25 |
| F ↓ | 2.37 | 3.76 | 3.67 | 3.06 |
Varying the number of self-generated replay examples per task (N), in the default setting (σ = 0, m = k = 1). N = 500 is the run in Table 1.
| N per task | 250 | 500 | 1000 | 2500 |
|---|---|---|---|---|
| AA ↑ | 58.03 | 59.44 | 59.18 | 58.06 |
| F ↓ | 7.02 | 2.84 | 2.99 | 1.30 |
[1] KMMLU: Measuring Massive Multitask Language Understanding in Korean, NAACL 2025
[2] KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding, Findings of ACL 2025
Response to Reviewer eoPV (3/3): W5, W6, W7
W5 / Q3. “…the skip fraction for backbone inputs falls from 61.4% after the first task to 29.3% after the last… this does not prove that the backbone’s general capability is preserved in unmeasured areas.” / “Is backbone preservation truly general, or only benchmark-specific?”
Following the reviewer’s concern, we evaluated the final TRACE checkpoint on three additional benchmarks: KMMLU [1] (Korean knowledge), MGSM [2] (math in 10 languages), and BFCL [3] (tool use).
Preservation is not specific to MMLU, GSM8K, and PIQA: on KMMLU, MGSM, and BFCL, CLoRE stays within 2.0 points of the backbone, while the baselines lose 6 to 24 points. The falling skip fraction does not indicate forgetting. Routing backbone inputs to skip is not always the best way to preserve the backbone’s predictions. As experts are added and the router must also serve more tasks, holding every backbone input on skip becomes a harder constraint, and routing some of them to added experts that leave their predictions intact can be the easier solution. Since the router is trained with the language-modeling loss on backbone replay, it takes whichever routing preserves the predictions, skip or added experts. The low backbone forgetting F₀ (Table 1) and the results on the additional benchmarks show that the predictions are indeed preserved.
| Method | KMMLU | MGSM | BFCL |
|---|---|---|---|
| Backbone | 41.46 | 46.28 | 67.14 |
| CLoRE | 40.48 | 44.40 | 65.14 |
| EWC | 35.01 | 22.44 | 53.44 |
| O-LoRA | 33.20 | 32.64 | 53.74 |
W6 / Q2. “…inference still requires routing over a growing pool. It is unclear how CLoRE behaves at large number tasks.” / “Can CLoRE scale to many more tasks and larger expert pools?”
We agree that the expert pool grows with each task. This is common to all expansion-based methods, which add parameters for every new task; what CLoRE itself adds on top is a single reservoir logit per layer.
Evaluating a much larger number of tasks is beyond what we could run in the rebuttal period. We examined this on the ten-task sequence of W2. A new expert stays out of the existing top-k as long as the reservoir logit lies below the logits of the selected experts. As the table below shows, the share of tokens that satisfy this margin does not shrink as tasks are added: it stays at 99.9% or above at every step.
| Tasks learned | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Tokens satisfying the margin (%) | 100.00 | 100.00 | 99.96 | 99.91 | 99.97 | 99.98 | 99.96 | 99.95 | 99.94 | 99.96 |
Forgetting does not grow either: F is 1.51 over the ten tasks, and the two tasks learned first keep their scores through all subsequent tasks (W2). For larger expert pools, adding two or four experts per task instead of one keeps F between 3.06 and 3.76 (W4). At inference, the router computes one logit per expert, so routing cost grows linearly with the pool. Only the top-k experts are evaluated per token, so the cost of the experts themselves stays constant.
W7. “The code doesn’t seem to be included which makes judgments about reproducibility challenging.”
We will release the code in an anonymous repository as soon as possible and post the link in this thread. It will include a run script and a configuration file for each table.
[1] KMMLU: Measuring Massive Multitask Language Understanding in Korean, NAACL 2025
[2] Language Models are Multilingual Chain-of-Thought Reasoners, ICLR 2023
[3] The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models, ICML 2025
Response to Reviewer eoPV (1/3): W1
W1 / Q1. “…it does not measure replay diversity, coverage, or failure cases.” / “How sensitive is CLoRE to the quality and diversity of self-generated replay?”
Before each of the seven expansions on TRACE, we compared the replay generated for the first task, C-STANCE (500 questions), with the same number of its real training questions. Questions are embedded with the frozen backbone. Fidelity [1] is the share of generated questions that lie within the k-nearest-neighbor radius (k = 5) of some real question, and coverage [2] is the share of real questions with a generated question within their radius. Unique is the share of generated questions that are distinct, and distinct-2 [3] is the share of distinct bigrams among all bigrams; lower values of either indicate more repetition.
The table compares real data, CLoRE’s generated replay with the routing override, and generated replay without it; each cell shows the value at the first → last expansion. With the routing override, the generated replay keeps covering C-STANCE: coverage changes little over the sequence (0.78 → 0.73), and repetition stays at the level of real data (unique 1.00 vs. 1.00 and distinct-2 0.41 vs. 0.47 at the last expansion). Fidelity decreases slightly (0.72 → 0.61). Without the override, coverage falls to 0.00 and the other metrics are not even defined at the last expansion, because no C-STANCE question is generated at all; this is the case for four of the seven earlier tasks. Coverage, not fidelity, is what carries over to performance (Q1): with the override, CLoRE forgets far less (F 2.84 vs. 7.23) and reaches a higher AA (59.44 vs. 57.02).
[1] Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019
[2] Reliable Fidelity and Diversity Metrics for Generative Models, ICML 2020
[3] A Diversity-Promoting Objective Function for Neural Conversation Models, NAACL 2016