- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposed a novel MoE framework, AEI, for continual learning of LLMs. Instead of using a fixed number of experts, AEI utilizes an adaptive expert expansion mechanism encouraging to learn complementary knowledge, which reduces interference and parameter redundancy by constrained complementary spaces. Also, AEI has a context-aware gating module to adaptively identify and combine relevant experts. The experiments across different LLMs and settings show that AEI outperforms existing continual learning methods using only 1% rehearsal data.
Critical strengths
The paper proposes a novel adaptive expandable MoE framework for continual learning. Since previous works mostly focus on using a fixed number of experts, although GoDMoE also proposes an expansion method in MoE for continual learning, the performance of GoDMoE is not better than AEI. The idea and perspective of this MoE for continual learning are novel and interesting to me, since it's different from conventional continual MoE frameworks and it progressively introduces new experts.
Critical weaknesses
It’s not clear for the motivation of the design in AEI. Since AEI already stores 1% rehearsal data for previous tasks and uses them to refine gating vector when learning new tasks, it makes me confused about why not also use this historical data to help update newly introduced expert. We know that historical data can be used as the input representation which can also help new experts reduce interference from old experts in continual learning.
Questions for authors
-
For newly added experts, how to initialize them? Does the algorithm give high priority to learning new tasks on the newly added experts rather than on old experts?
-
Can authors explain how to initialize the newly added column in the gating vector in AEI? In line 306, it said that “the parameters learned at stage are used to initialize the gating module at stage , and the entire is then updated for the new task”. But if the entire gating vector is updated, the learned parameters would not be the last added column in gating vector, since other columns are also updated. In this case, the dimension of all columns will be a problem for newly added column.
-
What’s the principle for the asymmetric label-smoothing strategy in Eq(15)?
-
Can authors compare the performance of AEI and SAME [1]? And compare the cost of GoDMoE in Table 5?
[1] “SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning”, ICML 2026.
-
Providing a completed algorithm makes it clear to see how different parts of AEI work together.
4 comments
Part2: Responses to Question 4 and Question 5
Question 4: Can authors compare the performance of AEI and SAME [1]? And compare the cost of GoDMoE in Table 5?
Table 9. Comparison with SAME on SuperNI-10 using LLaMA-2-7B.
5 tasks
| Method | Order 1 AR ↑ | Order 1 BWT ↑ | Order 2 AR ↑ | Order 2 BWT ↑ | Order 3 AR ↑ | Order 3 BWT ↑ | Average AR ↑ | Average BWT ↑ |
|---|---|---|---|---|---|---|---|---|
| SAME | 52.55 | -1.01 | 51.97 | -1.60 | 51.59 | -2.31 | 52.04 | -1.64 |
| AEI | 53.46 | +0.03 | 53.26 | -0.02 | 53.26 | -0.19 | 53.33 | -0.06 |
10 tasks
| Method | Order 1 AR ↑ | Order 1 BWT ↑ | Order 2 AR ↑ | Order 2 BWT ↑ | Order 3 AR ↑ | Order 3 BWT ↑ | Average AR ↑ | Average BWT ↑ |
|---|---|---|---|---|---|---|---|---|
| SAME | 62.61 | -3.13 | 62.89 | -2.78 | 62.53 | -3.36 | 62.68 | -3.09 |
| AEI | 65.56 | +0.04 | 65.32 | +0.01 | 65.64 | -0.10 | 65.51 | -0.01 |
Response. We compare AEI with SAME on SuperNI-10 using LLaMA-2-7B and SAME's official implementation. As shown in Table 9, AEI consistently achieves higher AR and BWT across all evaluated task orders and sequence lengths. The advantage is particularly evident in the 10-task setting, demonstrating better adaptation and knowledge retention than SAME.
Table 10. Computational cost comparison on LLaMA-3.1-8B.
| Method | Additional Params. | Training Time (TT) | Inference Time (IT) |
|---|---|---|---|
| GoDMoE | 102M | ~10.1 h | ~1.21 h |
| Gain | 210M | ~5.41 h | ~0.79 h |
| SERS | — | ~25.6 h | ~0.40 h |
| AEI (Ours) | 235M | ~5.13 h | ~0.44 h |
We also include GoDMoE in the cost comparison in Table 10. Although AEI introduces more additional parameters than GoDMoE, it requires substantially shorter training and inference times in this setting. These results show that AEI trades a larger number of additional parameters for lower measured runtimes. Both comparisons have been incorporated into the revised manuscript.
Question 5: Providing a completed algorithm makes it clear to see how different parts of AEI work together.
Response. We have provided a complete algorithm (Algorithm 1 in Appendix F of the revised manuscript) to show how expert expansion, router training, expert training, and pruning fit together. In particular, it specifies that both training phases use current and replay data, while the router is frozen during expert training.
Thanks for authors taking time to reply to my concerns. Most of them are addressed, so I would keep my original score. The reason I wanna keep my score is that the current algorithm makes sense to me and has strong performance in experiments. One remaining concern is about the router update rule in the current algorithm. I think it will cause router drift on previously learned tasks due to router weight changes. I think perhaps exploring how to improve the current algorithm to reduce router drift may be future work.
We sincerely thank you for your careful consideration of our responses and for your continued positive assessment. We will consider this constructive suggestion in our future work.
Part1: Responses to Weakness 1, Question 1, Question 2, and Question 3
We sincerely thank the reviewer for the thoughtful and constructive comments. We have carefully addressed the concerns below, and the corresponding revisions are highlighted in blue in the revised manuscript.
Weakness 1: It’s not clear for the motivation of the design in AEI. Since AEI already stores 1% rehearsal data for previous tasks and uses them to refine gating vector when learning new tasks, it makes me confused about why not also use this historical data to help update newly introduced expert. We know that historical data can be used as the input representation which can also help new experts reduce interference from old experts in continual learning.
Response. We apologize for the ambiguity. Indeed, historical replay data are used in both training phases: router training and new-expert training.
At stage t, both router training and expert training use Dt, which contains the current-task data and the retained samples from previous tasks. We first update the router using task-identity supervision. We then freeze the router and optimize the newly introduced expert using the next-token prediction loss on Dt, together with the subspace regularizer. In this way, historical samples serve not only as supervision for expert identification and routing, but also as training inputs for the newly introduced expert, allowing it to learn from historical representations.
We have clarified this training procedure in the revised manuscript and provided Algorithm 1 in Appendix F to illustrate how historical replay samples are used in both training phases.
Question 1: For newly added experts, how to initialize them? Does the algorithm give high priority to learning new tasks on the newly added experts rather than on old experts?
Response. The new expert's down-projection matrix is initialized to zero, while its other projection matrices use Kaiming-uniform initialization. The zero down-projection makes the new expert's initial output zero. During expert training, the backbone, historical experts, and trained router are frozen, so parameter updates are confined to the newly introduced expert. Historical experts can still contribute through the router, allowing their knowledge to be reused.
Question 2: Can authors explain how to initialize the newly added column in the gating vector in AEI? In line 306, it said that “the parameters learned at stage t + 1 are used to initialize the gating module at stage , and the entire is then updated for the new task”. But if the entire gating vector is updated, the learned parameters would not be the last added column in gating vector, since other columns are also updated. In this case, the dimension of all columns will be a problem for newly added column.
Response We apologize for the unclear description of the matrix annotations. We expand the router's output projection matrix, not the token-dependent gating vector itself. At stage t, each attention head's output projection has shape t×dh after task t. At the next stage, we copy these t rows and append one randomly initialized row, obtaining a matrix of shape (t+1)×dh and thus outputting a gating vector α∈Rt+1. The query, key, and value projection dimensions are unchanged, and their learned parameters are also retained as initialization. All parameters of the expanded router are then updated using current and replay data. Updating the values of the existing rows does not change their dimensions, so it causes no dimensional mismatch. After router training, we freeze the router and train the new expert, and the historical experts remain frozen throughout. We have corrected the corresponding descriptions in the revised manuscript.
Question 3: What’s the principle for the asymmetric label-smoothing strategy in Eq(15)?
Response. The principle is to encourage reuse of earlier knowledge while limiting interference from experts introduced later. For a token from task Tc with c>1, its corresponding expert receives target weight 1−ε, and experts from tasks 1,…,c−1 share the remaining mass ε equally. Experts introduced after task c receive zero target weight. This preserves the temporal direction of knowledge reuse when training on replay samples (earlier experts share the remaining mass ε). The targets discourage later experts from interfering with earlier tasks (later experts receive zero target weight).