- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies routing drift in expandable mixture-of-experts models for class-incremental learning. As new experts are introduced and the router continues to be updated, the expert composition for previously learned classes may change even when the old experts are frozen. The authors propose StaR-MoE, which uses SARA to align current routing distributions with historical ones and ACR to improve expert utilization. The method is evaluated on four CIL benchmarks, together with ablations, controlled routing interventions, and capacity-matched comparisons.
Critical strengths
-
The paper identifies that freezing historical experts does not guarantee stable computation paths for old classes. The correlation analysis and controlled routing intervention provide evidence that routing drift contributes to forgetting, offering a useful mechanism-level explanation for the observed performance gains.
-
In addition to comparisons with several representative pretrained-model-based continual learning methods, the paper includes a matched expansion-schedule comparison with SEMA, evaluations with different pretrained backbones, component ablations, and analyses of runtime and storage overhead. These experiments partially rule out the possibility that the improvement over SEMA is solely due to the use of more experts and demonstrate applicability across different pretraining paradigms.
Critical weaknesses
-
The method and theoretical analysis primarily constrain dense routing distributions, whereas the actual forward computation uses discrete Top- routing. A small dense-routing KL divergence does not necessarily imply that the selected expert set remains unchanged, particularly when the -th and -th routing logits are close. The paper does not report the expert-switch rate, Top- overlap, or the frequency with which newly introduced experts enter the computation paths of old-class samples. The central routing-stability claim therefore lacks direct evidence at the level of the routing decisions used by the model.
-
Most main experiments use a single class order generated with seed 1993, while multi-seed results are provided only for ImageNet-A. Routing allocation, expert specialization, and forgetting may all depend on the class order. The current evaluation therefore does not sufficiently establish whether the main findings remain robust across datasets and task orders.
Questions for authors
-
The ICLR 2026 paper One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning (SMoPE; Le et al., 2026) also employs input-dependent Top- expert selection with a frozen ViT to reduce interference in rehearsal-free CIL. Could the authors compare StaR-MoE with SMoPE using the same backbone, task splits, and class orders, while reporting accuracy, parameter count, and computational cost? If a direct comparison is infeasible, please clarify the specific incompatibilities between the two experimental protocols.
-
Does SARA significantly reduce changes in the actual Top- expert sets for old-class samples, rather than only reducing dense-routing KL? It would be helpful to report the expert-switch rate, Top- overlap, and the frequency with which newly added experts are selected for old-class samples.
-
Most main experiments use the class order generated with seed 1993, and multi-seed evaluation is reported only for ImageNet-A. Do the main performance gains and the relationship between routing drift and forgetting remain consistent across different class orders on the other datasets?
-
Table 13 reports 1.88M additional learnable parameters for StaR-MoE, compared with 0.05M for L2P, 0.25M for DualPrompt, and 1.35M for SEMA. Could the authors report results under a fixed total number of experts or a matched parameter budget? This comparison would help distinguish the benefit of routing stabilization from that of increasing model capacity over the task sequence.