acceptodds
Under review as a conference paper at ICLR 2027

Theoretical Understanding of Dynamic Mixture of Experts in Continual Learning

Abstract

The use of Mixture-of-Experts (MoE) architectures to mitigate catastrophic forgetting in continual learning (CL) has attracted increasing research interest. In particular, dynamic MoE frameworks that progressively expand the expert pool achieve strong empirical performance by allocating dedicated capacity to incoming tasks. Despite this success, a rigorous theoretical understanding of their underlying mechanisms remains largely underexplored. This work provides a formal analysis of dynamic MoE in CL, introducing a framework that integrates expert expansion, freezing, and inheritance. Our analysis reveals that as task diversity grows, expert specialization proceeds in distinct phases, motivating a phased expansion mechanism. We prove that only a finite set of experts is required to cover an infinite task stream within any prescribed error tolerance, justifying an efficient expansion strategy. Furthermore, we derive explicit bounds on forgetting and generalization error, showing that dynamic expansion with expert inheritance ensures stable learning and fast adaptation. Experiments on both synthetic and real-world benchmarks validate our theoretical insights and offer principled guidelines for designing scalable MoE-based CL systems.

Primary area:
learning theory
Keywords:
Mixture-of-Experts, Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YH1xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the role of dynamic Mixture-of-Experts (MoE) models in continual learning, focusing on how expert expansion, freezing, and parameter inheritance affect learning as tasks continually arrive. The authors develop a theoretical framework based on a growing pool of linear regression tasks, showing that task diversity leads to phased expert specialization. They discuss the conditions under which a finite set of experts can cover an infinite task stream within a prescribed error tolerance and further derive bounds on forgetting and generalization error. Experiments on synthetic data and multiple image classification benchmarks examine these theoretical insights and the practical effects of the corresponding mechanisms.

Critical strengths

1.The paper develops a formal analytical framework for continual learning with a growing task pool and dynamically expanding experts. It discusses the conditions for finite expert coverage and derives bounds on forgetting and generalization error, providing a theoretical perspective on the roles of expert expansion, freezing, and parameter inheritance.

2.The paper presents a theoretical observation of phased expert specialization, characterizing differences in specialization completion times among experts within the same task cluster. This analysis helps explain the limitations of a common freezing time and provides a basis for freezing experts according to their individual states.

3.The experiments cover synthetic data and multiple image classification benchmarks and examine the effects of freezing and parameter inheritance. The class-incremental experiments in the appendix also provide preliminary evidence regarding whether these theoretical insights can extend to other continual learning settings.

Critical weaknesses

1.The positioning of the theoretical contribution is not sufficiently clear. Expert expansion, freezing, and parameter inheritance have been used in prior work. The paper does not sufficiently distinguish these existing engineering components from its new theoretical analysis or clearly explain its advances over theoretical analyses of fixed-expert models.

2.A gap remains between finite coverage and the online algorithm. Proposition 5.1 establishes the existence of a finite expert set, but does not prove that the proposed online algorithm can generate this set and route tasks correctly. It therefore does not directly guarantee that the actual expansion process will eventually stop.

3.The connection between the specialization criterion and the deep-network experiments is insufficient. The theory defines the specialization completion round through differences in gating outputs, but there is insufficient validation of whether this criterion is reliable in deep networks and can guide expert freezing.

4.The experiments do not sufficiently isolate the effect of dynamic expansion. Changing the expansion threshold affects both the number of experts and the timing of expansion. The dynamic and static models are also not sufficiently matched in capacity, freezing rules, and training budgets, making it difficult to identify which mechanism primarily accounts for the performance improvements.

Questions for authors

1.How does the number of experts required in Proposition 5.1 vary with the error tolerance and task-space complexity? Can the authors provide an explicit upper bound and clarify the conditions under which the proposed online algorithm can achieve this finite coverage? 2.In a deep MoE model, how can one determine online whether an expert has completed specialization and can be frozen? How is the threshold on gating-output differences chosen, and does this criterion perform better than freezing at a fixed time or freezing based on validation error? 3.With the same total capacity, freezing rules, and training budget, how much benefit does dynamic MoE provide over static MoE? In light of the diminishing marginal coverage gains in Proposition 5.2, can the authors explain when further expansion offers limited benefits or may even degrade performance?

3 comments

W1. The position of the theoretical contribution.

We agree that the expansion with components of freezing and inheritance have existing engineering implementations in some empicical work. Our work is parallel to these empirical work, which aims to conduct theoretical analyses of these mechanisms under the setting of dynamic CL. Specifically, the total number of tasks and the diversities in real world naturally cannot be determined in advance, yet existing studies focus on a static task pool. As a result, existing studies fundamentally fail to identify or solve the problems presented in this work. The results proposed in this work are novel theoretical findings centered on the dynamic expansion framework with a continuously expanding task pool. We make revisions in lines 52-54 to better highlight the novelty of this work.

Q1&W2. Gap between Proposition 5.1 and the framework.

We appreciate the valuable suggestions. We have strengthened Proposition 5.1, which shows that under a bounded parameter deviation δ\delta, we can guarantee a finite expansion of the number of experts. Specifically, based on the gap between the parameters of each expert when frozen and the task wtw_t, we directly present an upper bound on the number of experts within this theoretical framework, which is M0+(D+1)(1+2c/(ϵ0−δ))M_0+(D+1)(1+2c/(\epsilon_0-\delta)). A tighter error tolerance means a decrease in ϵ\epsilon and more experts; The complexity of the task space is related to the dimension dd of this space and the bound cc of the task norm; a larger dd or cc implies higher task complexity and likewise requires more experts. The revisions can be seen in lines 338-343 and 1141-1172.

Q2 & W3. Connection between the specialization criterion and the experiments.

We thank the reviewer for raising these points. In the experiments with the deep MoE model in section 6.2, a threshold-based approach is also adopted to determine when specialization completes, where the selection of this threshold is treated as a hyperparameter. For practical complex tasks, we do not deny that more sophisticated strategies are needed to be adopted to make judgments to guarantee optimal performance. We supplement experiments covering three schemes: the validation-based method, the threshold-based method, and freezing at specific fixed times. The results show that the threshold-based method outperforms the baseline and achieves performance comparable to the validation-set-error-based method. This demonstrates that the judgment criterion proposed in this paper is valid for deep MoE models. The revisions can be seen in lines 920-926.

W4. Alignment of Dynamic and Static Models.

We thank the reviewer for raising this concern. Our experiments on both synthetic and real-world datasets are fairly matched in terms of model capacity and training budget. Specifically, we set the static models to have the same total number of experts as the final expanded expert count of the dynamic expansion models to guarantee identical capacity. For the scope of this study, we do not report the training budget directly in the paper. Nevertheless, the framework with dynamic expansion actually consumes a smaller budget than the model with fixed experts. The revision can be seen in lines 425-427.

Q3. Benefits of the dynamic model.

We thank the reviewer for this thoughtful comment. Our experiments on dynamic and static MoE are fully aligned in terms of total capacity and training budget,which show that dynamic MoE exhibits notable advantages over static MoE. For instance, in the incremental task experiments on CIFAR-100 and Tiny-ImageNet described, the gaps in both forgetting rate and generalization error between the two can exceed 0.1. It should be noted that the number of expanded experts and the timing of expansion are closely tied to the expansion threshold. As clearly stated in Table 1(line 503), the benefits of dynamic MoE decline rapidly when the expansion thresholds are too low or too high, which also aligns with the conclusion of diminishing marginal benefits proposed in Proposition 5.2.

Thank you for your response. I would like to keep my score.

While the theoretical derivation is thorough, the practical contribution remains unclear, and the experimental evaluation is not sufficiently comprehensive (the datasets are relatively simple, and the number of baseline methods is limited). Furthermore, there are still concerns regarding the method's application in practical deep learning.

Thank you for your comment. We would like to emphasize that this is a theoretical work, intended to provide theoretical insights into existing dynamic MoE methods in CL. Should any specific concerns remain, we would be glad to receive your detailed comments and will provide additional clarification as needed.

Reviewer_u55tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focus on the dynamic expansion of the MoE architecture in continual learning. It explores a framework combining expert expansion, freezing, and parameter inheritance. Besides, the paper analyzes routing dynamics and phased expert specialization. Its main theoretical claims prove that only a finite set of experts is required to cover an infinite task stream within any prescribed error tolerance and derive explicit bounds on forgetting and generalization. Empirical studies verify the proposed mechanisms on synthetic and real-world benchmarks such as CIFAR100 and Tiny-ImageNet, with comparisons involving fixed-expert models and ablations.

Critical strengths

• The paper explores a dynamic mechanism additionally inherited on the fixed numbers and tasks and provides a prime insight on theoretical analysis.

• The paper combines synthetic experiments with real-world benchmarks to evaluate the proposed mechanism, which help support its theoretical perspectives.

Critical weaknesses
  1. Proposition 5.1 shows that a finite expert cover exists, but it does not show that the proposed online algorithm will keep the total number of experts bounded. Appendix A limits how many experts can be added per round, which is not the same as bounding the total. This leaves a gap in the paper’s main claim about controlled expansion.

  2. The explanation of phased specialization is not fully supported. Proposition 4.2 bounds the gating-score gap but does not show that routing actively reduces it. Proposition 4.4 assumes stable effective update frequencies and comparable update requirements, without explaining why the algorithm satisfies these conditions.

  3. The theoretical analysis is mainly on overparameterized linear regression cases, which might be a limitation in this work as nonlinear deep neural network cases can be more practical.

Questions for authors

What are the challenges from the fixed number of tasks and a fixed model architecture to the dynamic one, like how dynamic expansion affects routing load balancing or freezing-inheritance trade-offs? A clear challenge statement would help justify the proposed mechanisms and clarify the theoretical contributions.

Does ‖X_t ‖=O(1)  in Lemma 4.1 follow from the data-generation model, or is it an additional assumption? What normalization or probability guarantee is required?

Following Eq. (7), the paper claims that the model reduces the gating-score gap by increasing |T_m'^aux |. How does this behavior follow from Proposition 4.2, which only provides an upper bound on |h_t^((m) )-h_t^((m') ) |? What ensures that the routing updates actually increase this count and reduce the gap?

Proposition 4.4 assumes comparable cumulative effective update requirements and uniformly controlled realized update frequencies until specialization is completed. These conditions are not sufficiently justified, are they a strong or weak assumptions that can be achieved or found in related works or real world? Please explain.

2 comments

W1. Gap between Proposition 5.1 and the framework.

We thank Reviewer u55t for raising this concern. We have strengthened Proposition 5.1, which shows that under the framework design in section 3, we can guarantee a finite expansion of the number of experts. Specifically, based on the gap between the parameters of each expert when frozen and the task wtw_t, we directly present an upper bound on the number of experts within this theoretical framework, which is M0+(D+1)(1+2c/(ϵ0−δ))M_0+(D+1)(1+2c/(\epsilon_0-\delta)). The revisions can be seen in lines 345-350 and 1141-1173.

W2 & Q3. Clarification of Proposition 4.2.

We thank the reviewer for raising this concern. The reduction of the discrepancy is related to an analytical result from prior work. For rigor, we include the corresponding lemma 2 in lines 806-830 and provide explanations in lines 265-269. We also revise the statement concerning Tm′aux\mathcal{T}_{m'}^{\text{aux}} accordingly.

W2 & Q4. Clarification of Proposition 4.4.

We thank the reviewer for this valuable comment regarding the assumption of Proposition 4.4. We clarify that the comparable update frequency adopted in Proposition 4.4 is a simplified but reasonable assumption, which puts forward a basic stability requirement for the MoE system. Specifically, a number of empirical studies show that MoE can achieve stable update frequencies under proper conditions [4,5,6], while existing theoretical work also show that the routing update is not fully random but restricted by load balancing [1,3]. The relevant standard deviation can be referred to Figure (b) in Section 6.1. We supplement experiments to examine the updating behavior of experts within the same group, which show that the update frequencies of the experts are comparable. The revision can be seen in lines 860-863.

W3. Gap between linear framework and realistic nonlinear training.

We appreciate this valuable comment. Although this theoretical work is developed within a simplified linear regression framework, we argue that this does not substantially diminish the value of our contribution. First, a considerable body of existing theoretical work is similarly grounded in linear models [1][2][3]. Besides, we also conduct experiments on deep MoE to verify the transferability of the conclusions derived from linear models.

Q1. Challenges from a fixed model to the dynamic one.

We appreciate this suggestion and make revisions in lines 59-64. We emphasize that the task pool in real CL scenarios is continuously expanding, yet existing fixed expert models focus on static task pool and deviate from this requirement. The challenges in transitioning from a fixed framework to a dynamic one are stated as follows. A continuously expanding task pool renders the required expert capacity unknown and variable with growing task diversity. In particular, the expansion of new experts will cause sudden changes in routing load. This invalidates approaches that predefine the number of experts for a fixed task pool and derive specialization convergence, forgetting, and generalization through static methods. Furthermore, the freezing and inheritance schedules during expansion will jointly affect the retention and adaptation of knowledge. This is an issue unique to dynamic frameworks. Therefore, this work centers on these challenges, which analyzes expansion containing freezing and inheritance as a unified process and reestablishes corresponding guarantees for forgetting and generalization. Section 4 describes phased specialization, and Section 5 establishes the forgetting and generalization properties. We have incorporated this more focused statement of the challenge into the paper to clarify the mechanism design and the theoretical results.

Q2. Clarification of XtX_t.

We thank the reviewer for this comment and make a clarification. Xt=O(1)X_t=\mathcal{O}(1) follows from the generation setting in Section 3.1. In the paragraph of Data Generation (line 150-152), one column of XtX_t is set to cvncv_n, while other s−1s-1 columns are noise vectors of o(1)o(1). Since c=O(1)c=\mathcal{O}(1) and ∣vn∣=O(1)\\|v_n\\|=\mathcal{O}(1), we have ∣Xt∣=O(1)\\|X_t\\|=\mathcal{O}(1). Here, vnv_n is obtained by normalizing wnw_n, and no additional probabilistic guarantees are required.

[1]Towards understanding the mixture-of-experts layer in deep learning, NeurIPS, 2022.

[2]Theory on mixture-of-experts in continual learning. ICLR, 2025.

[3]Theory on forgetting and generalization of continual learning. ICML, 2023.

[4]StableMoE: Stable Routing Strategy for Mixture of Experts, ACL, 2022.

[5]Advancing Expert Specialization for Better MoE, NeurIPS, 2026.

[6]SAME: Stabilized Mixture-of-Experts for Multimodel Continual Instruction Tuning, ICML 2026.

The authors clarified some point yet my other concerns such as W3 remained. I thus keep my rating.

Reviewer_PhT8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the theoretical properties of dynamic MoE architectures for continual learning. The authors propose a framework that integrates expert expansion, freezing, and parameter inheritance to accommodate sequentially arriving tasks. Their theoretical analysis characterizes phased expert specialization, establishes a finite-expert coverage result, and derives bounds on forgetting and generalization error. Experiments on synthetic and real-world datasets are conducted to validate the theoretical findings.

Critical strengths
  • The paper addresses an interesting and relevant problem: understanding the theoretical properties of dynamically expanding MoE architectures in continual learning.
  • The paper includes experiments on both synthetic and real-world datasets to investigate the theoretical findings.
Critical weaknesses
  • My primary concern is the novelty of the theoretical contributions relative to existing work. In particular, [1] have already investigated expert specialization, routing dynamics, forgetting, and generalization in MoE-based continual learning using an overparameterized linear regression framework. The current paper adopts a closely related theoretical setting and extends it with dynamic expert expansion, freezing, and parameter inheritance. While this extension is potentially meaningful, the paper does not sufficiently clarify which fundamentally new theoretical insights arise specifically from dynamic expansion, beyond those established in prior work.
  • The paper lacks comparisons with existing dynamic expansion-based continual learning methods, making it difficult to assess the practical advantages of the proposed framework beyond fixed-expert baselines.

[1] Theory on Mixture-of-Experts in Continual Learning, ICLR 2025

Questions for authors

Please refer to the weaknesses discussed above.

3 comments

W1. Distinction from prior work.

We appreciate the reviewer's recognition of our research perspective, inference efficiency, and presentation. To further address the reviewer's concerns, we make the following clarifications. All the insights in this work are novel results centered on the dynamic expansion framework with a setting of continuously expanding task pool, which is closer to a real CL scenario. In the real world, the total number of tasks and the diversities among them naturally cannot be determined in advance, yet existing studies focus on a static task pool. As a result, these studies fundamentally fail to identify or solve the problems presented in this work. Specifically:

  • In Section 4, we illustrate that under high task diversity, each expert develops phased specialization. However, under the setting of a static task pool in [1] as an example, all experts are initialized at the beginning of the entire training process, making it impossible to capture such phased features. More importantly, this directly motivates phased expert expansion and freezing, which is entirely overlooked under the static setting.

  • In Section 5, we first demonstrate that the number of experts required under this dynamic expansion framework is finite (in the revision we further give an upper bound). We then prove that unlimited expansion of experts will lead to the diminish of marginal benefits. These are unique and important issues that arise only when the task pool keeps expanding. A fixed task pool naturally requires only a certain set of fixed experts and fails to capture the issues.

  • Finally, we derive the bound of forgetting and generalization error under the dynamic framework. Although existing work includes estimates of forgetting and generalization error, the bounds cannot characterize the result for the dynamic task pool, esspecially integrating freezing and inheritance designs. Therefore, under this more complex and realistic setup, we have to re-establish these bounds.

In summary, based on a feature analysis, this work clarifies several critical issues under the setting of a dynamic task pool. We have also made revisions in lines 52-54 to better highlight the novelty of this work.

W2. Comparision with existing work.

We thank the reviewer for this comment and we clarify that our work is parallel to existing empirical studies. Many existing studies have adopted dynamic expansion frameworks similar to the one proposed in this paper and achieved state-of-the-art results in CL [2][3][4] and we aims to model the dynamic expansion framework and provide theoretical guarantees.

[1]Towards understanding the mixture-of-experts layer in deep learning, NeurIPS, 2022.

[2]Moe-adapters++: Towards more efficient continual learning of vision-language models via dynamic mixture-of-experts adapters, TPAMI, 2025.

[3]Divide and not forget: Ensemble of selective trained experts in continual learning, ICLR, 2024.

[4]Learning adaptive and expandable mixture model for continual learning, AAAI, 2026.

Thank you for your response. I would like to keep my score.

Thank you for your valuable comments and for acknowledging our work.  Your valuable suggestions significantly improved the quality of our manuscripts. We appreciate your time and effort in reviewing our work. Thanks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.