acceptodds
Withdrawn from ICLR 2027

Stable Routing for Mixture-of-Experts in Class-Incremental Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Class-incremental learning (CIL) requires models to learn new classes sequentially while preserving prior knowledge. Recently, approaches that combine pre-trained models with mixture-of-experts (MoE) have received increasing attention in CIL: they typically expand experts during learning and employ a router to assign weights across experts. However, existing MoE methods often overlook routing drift induced by expert expansion. Once new experts are introduced, the router may reassign samples from earlier classes to newly added experts, thereby perturbing previously established expert compositions and causing interference even when old experts remain frozen. We argue that expandable MoE in CIL requires two complementary properties: stable old-class routing for knowledge preservation and sufficient capacity utilization for new-class adaptation. To this end, we propose Stable Routing for MoE (StaR-MoE), a routing-level framework for expandable MoE in CIL. By incorporating sensitivity-aware routing alignment, StaR-MoE aligns current old-class routing behavior with historical routing distributions through sensitivity-guided constraints. Complementarily, StaR-MoE introduces asymmetric capacity regularization to encourage effective utilization of the expanded expert pool without compromising class-specific routing specialization. Extensive experiments across four standard CIL benchmarks demonstrate that StaR-MoE consistently improves both average and last accuracy over state-of-the-art methods, highlighting the importance of stable routing.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Mixture-of-Experts, Routing Stability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LYaeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies routing drift in expandable mixture-of-experts models for class-incremental learning. As new experts are introduced and the router continues to be updated, the expert composition for previously learned classes may change even when the old experts are frozen. The authors propose StaR-MoE, which uses SARA to align current routing distributions with historical ones and ACR to improve expert utilization. The method is evaluated on four CIL benchmarks, together with ablations, controlled routing interventions, and capacity-matched comparisons.

Critical strengths
  1. The paper identifies that freezing historical experts does not guarantee stable computation paths for old classes. The correlation analysis and controlled routing intervention provide evidence that routing drift contributes to forgetting, offering a useful mechanism-level explanation for the observed performance gains.

  2. In addition to comparisons with several representative pretrained-model-based continual learning methods, the paper includes a matched expansion-schedule comparison with SEMA, evaluations with different pretrained backbones, component ablations, and analyses of runtime and storage overhead. These experiments partially rule out the possibility that the improvement over SEMA is solely due to the use of more experts and demonstrate applicability across different pretraining paradigms.

Critical weaknesses
  1. The method and theoretical analysis primarily constrain dense routing distributions, whereas the actual forward computation uses discrete Top-kk routing. A small dense-routing KL divergence does not necessarily imply that the selected expert set remains unchanged, particularly when the kk-th and (k+1)(k+1)-th routing logits are close. The paper does not report the expert-switch rate, Top-kk overlap, or the frequency with which newly introduced experts enter the computation paths of old-class samples. The central routing-stability claim therefore lacks direct evidence at the level of the routing decisions used by the model.

  2. Most main experiments use a single class order generated with seed 1993, while multi-seed results are provided only for ImageNet-A. Routing allocation, expert specialization, and forgetting may all depend on the class order. The current evaluation therefore does not sufficiently establish whether the main findings remain robust across datasets and task orders.

Questions for authors
  1. The ICLR 2026 paper One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning (SMoPE; Le et al., 2026) also employs input-dependent Top-kk expert selection with a frozen ViT to reduce interference in rehearsal-free CIL. Could the authors compare StaR-MoE with SMoPE using the same backbone, task splits, and class orders, while reporting accuracy, parameter count, and computational cost? If a direct comparison is infeasible, please clarify the specific incompatibilities between the two experimental protocols.

  2. Does SARA significantly reduce changes in the actual Top-kk expert sets for old-class samples, rather than only reducing dense-routing KL? It would be helpful to report the expert-switch rate, Top-kk overlap, and the frequency with which newly added experts are selected for old-class samples.

  3. Most main experiments use the class order generated with seed 1993, and multi-seed evaluation is reported only for ImageNet-A. Do the main performance gains and the relationship between routing drift and forgetting remain consistent across different class orders on the other datasets?

  4. Table 13 reports 1.88M additional learnable parameters for StaR-MoE, compared with 0.05M for L2P, 0.25M for DualPrompt, and 1.35M for SEMA. Could the authors report results under a fixed total number of experts or a matched parameter budget? This comparison would help distinguish the benefit of routing stabilization from that of increasing model capacity over the task sequence.

Reviewer_i6C1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Stable Routing for Mixture-of-Experts (StaR-MoE), a framework designed to mitigate routing drift in expandable Mixture-of-Experts (MoE) models for class-incremental learning. The method introduces Sensitivity-Aware Routing Alignment (SARA), which preserves historical routing behavior by aligning routing probabilities for old classes with stored routing targets, and Asymmetric Capacity Regularization (ACR), which encourages effective utilization of newly added experts. Theoretical analysis and experiments on CIFAR-100, ImageNet-R, ImageNet-A, and VTAB demonstrate the effectiveness of the proposed approach in reducing routing drift and improving incremental learning performance.

Critical strengths
  1. The paper identifies routing drift as an important source of forgetting in expandable MoE-based class-incremental learning and provides a clear motivation for explicitly preserving historical routing behavior.
  2. SARA and ACR address different but related challenges: preserving routing stability and promoting expert utilization. The ablation studies provide evidence that both components contribute to the overall performance.
  3. The method is evaluated on multiple benchmarks and compared with several baselines and alternative routing-preservation strategies. The paper also includes ablation studies, hyperparameter analyses, and theoretical analysis connecting routing drift with forgetting.
Critical weaknesses
  1. Insufficient causal evidence linking routing drift to forgetting. The paper observes a strong correlation between routing divergence and forgetting, but correlation alone does not establish causality. Although the analysis suggests that routing drift is an important factor, it remains unclear to what extent the performance degradation is directly caused by routing changes rather than by other factors, such as changes in feature representations. More controlled experiments are needed to isolate the causal effect of routing drift on forgetting.
  2. The independent contribution of sensitivity-aware weighting is insufficiently validated. SARA introduces sensitivity-aware layer weighting to prioritize routing alignment at more influential MoE layers. However, the paper does not sufficiently isolate the contribution of this weighting strategy from that of the routing alignment objective itself. A direct comparison with uniform layer weighting and alternative sensitivity estimation strategies would help establish whether sensitivity-aware weighting provides consistent benefits and justify the additional complexity.
  3. A gap remains between the theoretical analysis and practical sparse routing. The theoretical guarantee focuses on bounding sensitivity-weighted dense routing divergence, whereas the actual MoE model employs sparse Top-k expert selection. Small differences in dense routing probabilities do not necessarily guarantee consistent expert selection, particularly when routing scores are close to the Top-k decision boundary. The paper would benefit from a more explicit analysis of how the proposed bound translates into practical routing stability and reduced forgetting.
Questions for authors

Please refer to the Weaknesses section.

Reviewer_xD7WOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies routing drift in expandable mixture-of-experts for class-incremental learning. StaR-MoE combines sensitivity-aware routing alignment with asymmetric capacity regularization to preserve old-class routing while utilizing newly added experts. Experiments report improvements on four visual class-incremental learning benchmarks.

Critical strengths

Maintaining stable routing while expanding expert capacity is a relevant problem for continual learning.

Critical weaknesses
  • LLaVA-DyMoE (CVPR 2026) already analyzes forgetting caused by old inputs being attracted to newly added experts despite frozen old experts, and SAME (ICML 2026) directly studies routing stability. SMoPE (ICLR 2026) also penalizes experts whose activation frequencies exceed the mean, which is closely related to ACR. The paper does not sufficiently distinguish its contributions from these works or provide targeted comparisons with their designs.

  • Closely related methods such as CaRE/BR-MoE (ICML 2026) and SMoPE are missing. Experiments are limited to relatively small visual benchmarks and two pretraining variants of ViT-B/16, leaving long-term expansion stability, computational costs, and cross-model applicability unclear. With DINO, the final-accuracy gain over the strongest baseline is only 0.60 percentage points. Convincing experiments on modern LLMs or MLLMs are needed to establish the method’s broader relevance.

  • A single class-level routing target may suppress useful within-class expert specialization, while updates to earlier routers can make stored input statistics outdated. The t-SNE visualization and distribution-family ablations do not adequately address these concerns. The current-task gradient norm is also not validated as a measure of old-task sensitivity, and its gains over uniform weighting are only 0.38/0.59 percentage points in average/final accuracy. ACR still requires uniform loads to achieve zero loss, so accuracy gains alone do not establish better preservation of specialization. Moreover, the distillation comparison does not control input information: SARA uses historical synthetic features, whereas the LwF baseline uses current-task inputs.

  • In Table 4, both quantities are strongly associated with task age, without controlling for the greater number of subsequent updates experienced by earlier tasks. The oracle intervention provides useful evidence but lacks detailed per-task and cross-seed results. Table 3 also shows that the full model forgets less despite having higher routing KL than SARA alone, highlighting the need to distinguish harmful drift from beneficial expert reuse. The theoretical bound only covers dense routing on synthetic inputs and does not establish the stability of actual Top-k paths or predictions.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.