- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the role of dynamic Mixture-of-Experts (MoE) models in continual learning, focusing on how expert expansion, freezing, and parameter inheritance affect learning as tasks continually arrive. The authors develop a theoretical framework based on a growing pool of linear regression tasks, showing that task diversity leads to phased expert specialization. They discuss the conditions under which a finite set of experts can cover an infinite task stream within a prescribed error tolerance and further derive bounds on forgetting and generalization error. Experiments on synthetic data and multiple image classification benchmarks examine these theoretical insights and the practical effects of the corresponding mechanisms.
Critical strengths
1.The paper develops a formal analytical framework for continual learning with a growing task pool and dynamically expanding experts. It discusses the conditions for finite expert coverage and derives bounds on forgetting and generalization error, providing a theoretical perspective on the roles of expert expansion, freezing, and parameter inheritance.
2.The paper presents a theoretical observation of phased expert specialization, characterizing differences in specialization completion times among experts within the same task cluster. This analysis helps explain the limitations of a common freezing time and provides a basis for freezing experts according to their individual states.
3.The experiments cover synthetic data and multiple image classification benchmarks and examine the effects of freezing and parameter inheritance. The class-incremental experiments in the appendix also provide preliminary evidence regarding whether these theoretical insights can extend to other continual learning settings.
Critical weaknesses
1.The positioning of the theoretical contribution is not sufficiently clear. Expert expansion, freezing, and parameter inheritance have been used in prior work. The paper does not sufficiently distinguish these existing engineering components from its new theoretical analysis or clearly explain its advances over theoretical analyses of fixed-expert models.
2.A gap remains between finite coverage and the online algorithm. Proposition 5.1 establishes the existence of a finite expert set, but does not prove that the proposed online algorithm can generate this set and route tasks correctly. It therefore does not directly guarantee that the actual expansion process will eventually stop.
3.The connection between the specialization criterion and the deep-network experiments is insufficient. The theory defines the specialization completion round through differences in gating outputs, but there is insufficient validation of whether this criterion is reliable in deep networks and can guide expert freezing.
4.The experiments do not sufficiently isolate the effect of dynamic expansion. Changing the expansion threshold affects both the number of experts and the timing of expansion. The dynamic and static models are also not sufficiently matched in capacity, freezing rules, and training budgets, making it difficult to identify which mechanism primarily accounts for the performance improvements.
Questions for authors
1.How does the number of experts required in Proposition 5.1 vary with the error tolerance and task-space complexity? Can the authors provide an explicit upper bound and clarify the conditions under which the proposed online algorithm can achieve this finite coverage? 2.In a deep MoE model, how can one determine online whether an expert has completed specialization and can be frozen? How is the threshold on gating-output differences chosen, and does this criterion perform better than freezing at a fixed time or freezing based on validation error? 3.With the same total capacity, freezing rules, and training budget, how much benefit does dynamic MoE provide over static MoE? In light of the diminishing marginal coverage gains in Proposition 5.2, can the authors explain when further expansion offers limited benefits or may even degrade performance?
3 comments
Thank you for your response. I would like to keep my score.
While the theoretical derivation is thorough, the practical contribution remains unclear, and the experimental evaluation is not sufficiently comprehensive (the datasets are relatively simple, and the number of baseline methods is limited). Furthermore, there are still concerns regarding the method's application in practical deep learning.
Thank you for your comment. We would like to emphasize that this is a theoretical work, intended to provide theoretical insights into existing dynamic MoE methods in CL. Should any specific concerns remain, we would be glad to receive your detailed comments and will provide additional clarification as needed.
We agree that the expansion with components of freezing and inheritance have existing engineering implementations in some empicical work. Our work is parallel to these empirical work, which aims to conduct theoretical analyses of these mechanisms under the setting of dynamic CL. Specifically, the total number of tasks and the diversities in real world naturally cannot be determined in advance, yet existing studies focus on a static task pool. As a result, existing studies fundamentally fail to identify or solve the problems presented in this work. The results proposed in this work are novel theoretical findings centered on the dynamic expansion framework with a continuously expanding task pool. We make revisions in lines 52-54 to better highlight the novelty of this work.
We appreciate the valuable suggestions. We have strengthened Proposition 5.1, which shows that under a bounded parameter deviation δ, we can guarantee a finite expansion of the number of experts. Specifically, based on the gap between the parameters of each expert when frozen and the task wt, we directly present an upper bound on the number of experts within this theoretical framework, which is M0+(D+1)(1+2c/(ϵ0−δ)). A tighter error tolerance means a decrease in ϵ and more experts; The complexity of the task space is related to the dimension d of this space and the bound c of the task norm; a larger d or c implies higher task complexity and likewise requires more experts. The revisions can be seen in lines 338-343 and 1141-1172.
We thank the reviewer for raising these points. In the experiments with the deep MoE model in section 6.2, a threshold-based approach is also adopted to determine when specialization completes, where the selection of this threshold is treated as a hyperparameter. For practical complex tasks, we do not deny that more sophisticated strategies are needed to be adopted to make judgments to guarantee optimal performance. We supplement experiments covering three schemes: the validation-based method, the threshold-based method, and freezing at specific fixed times. The results show that the threshold-based method outperforms the baseline and achieves performance comparable to the validation-set-error-based method. This demonstrates that the judgment criterion proposed in this paper is valid for deep MoE models. The revisions can be seen in lines 920-926.
We thank the reviewer for raising this concern. Our experiments on both synthetic and real-world datasets are fairly matched in terms of model capacity and training budget. Specifically, we set the static models to have the same total number of experts as the final expanded expert count of the dynamic expansion models to guarantee identical capacity. For the scope of this study, we do not report the training budget directly in the paper. Nevertheless, the framework with dynamic expansion actually consumes a smaller budget than the model with fixed experts. The revision can be seen in lines 425-427.
We thank the reviewer for this thoughtful comment. Our experiments on dynamic and static MoE are fully aligned in terms of total capacity and training budget,which show that dynamic MoE exhibits notable advantages over static MoE. For instance, in the incremental task experiments on CIFAR-100 and Tiny-ImageNet described, the gaps in both forgetting rate and generalization error between the two can exceed 0.1. It should be noted that the number of expanded experts and the timing of expansion are closely tied to the expansion threshold. As clearly stated in Table 1(line 503), the benefits of dynamic MoE decline rapidly when the expansion thresholds are too low or too high, which also aligns with the conclusion of diminishing marginal benefits proposed in Proposition 5.2.