- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Control-MoE, a dynamic routing framework for Mixture-of-Experts models. It combines parallel Top-1 gates with a lightweight skip expert to enable token-dependent expert activation. An MPC controller adjusts the skip-loss weight to regulate training-time sparsity, while the skip expert bias allows test-time activation scaling without retraining. Experiments on 2B–10B models show improved performance over fixed Top-K routing at comparable activation levels.
Critical strengths
-
The proposed routing mechanism is simple and enables flexible expert activation, allowing different tokens to use different numbers of experts.
-
The experiments cover multiple model scales and show improvements over fixed Top-K routing at comparable activation levels.
-
Control-MoE supports test-time activation scaling, allowing a single checkpoint to operate under different computational budgets without retraining.
Critical weaknesses
-
The paper lacks direct comparisons with existing dynamic MoE methods.
-
The contribution of the MPC controller is not sufficiently analyzed. It remains unclear whether MPC offers substantial advantages over fixed skip-loss weights or simpler feedback controllers in terms of sparsity control and training stability.
-
Equation (4) appears to contain a sign inconsistency. Minimizing the skip loss as written would decrease the skip expert probability, contradicting its stated purpose of encouraging skip expert selection.
Questions for authors
-
How does Control-MoE compare with existing dynamic routing methods under matched computational budgets?
-
How does MPC compare with fixed skip-loss weights or simpler feedback controllers?
-
How does the training sparsity target affect test-time activation scaling?
-
Could the authors clarify the negative sign in Equation (4)?
2 comments
Thank you for the detailed response and additional experiments. The authors have adequately addressed my other concerns, particularly regarding the MPC controller and the correction to Eq. (4).
Although the comparison with existing dynamic MoE methods is still ongoing, the preliminary results already show a promising advantage for Control-MoE. Given the limited time remaining in the discussion period, I understand that the complete results may not yet be available.
I have decided to raise my rating from 2 to 3 and look forward to seeing the completed experimental results.
Weakness #1
Response: Control-MoE contributes a predictive control mechanism that adjusts each layer’s skip-loss weight using an online model of its sparsity response. For dynamic-routing methods without automatic target tracking, matching the average activation budget requires additional calibration of routing thresholds or sparsity hyperparameters.
We thank the reviewer for pointing out ProbMoE, which we have decided to add to our paper as the new baseline. We have implemented ProbMoE Exact-4 and started a comparison on 2B models with 64 experts. Control-MoE uses 16 gates with a target of four effective experts per token. Across the first 3,415 shared training steps, Control-MoE achieves lower cross-entropy loss during the recent training stage, reaching approximately 2.30 versus 2.40 for ProbMoE. The training is ongoing, and we will report the completed results to determine whether this loss advantage translates into benchmark gains.
Weakness #2
Response: We have added a full ablation section showing that manually tuning fixed skip-loss weights struggles to reach the target sparsity, and new PID-control results demonstrating high fluctuation.
Action: We revised Section 4.3, added ablation tables and discussion, and included full experimental details in Appendix F.
Weakness #3
Response: The reviewer correctly identified a sign error in Eq. (4). Our implementation adds the positive squared penalty to the training objective, which encourages higher skip probabilities. We have removed the leading minus sign in the revision.
Question #1
Response: Please refer to Weakness #1.
Question #2
Response: Our ablations show that fixed skip-loss weights can approach the target sparsity after manual tuning, but different weights produce substantially different activation levels, as shown in the updated Section 4.3 and Appendix F. Several trials were terminated early because they remained below the target (Figure 7, Appendix F). MPC automates this adjustment using feedback from the realized skip rate. Against a PID baseline on the same 2B, 64-expert routing architecture, the tested PID configuration overshoots above 70% for a 50% target and continues to oscillate, as shown in Figure 8. MPC maintains smoother sparsity tracking, while both controllers achieve comparable final training losses.
MPC adds a small ARX fit and a closed-form scalar update per layer during training. The observed trade-off is therefore additional controller computation in exchange for more stable, automatic budget tracking.
Question #3
Response: The training sparsity target shapes the range of useful test-time activation levels. In Figure 2, the 2B–384 checkpoints trained to use 4 and 6 effective experts peak near their respective training targets. Larger departures in either direction can reduce accuracy. The width of the useful range varies by checkpoint. For example, the 7B model maintains similar accuracy across roughly four to eight effective experts. We therefore use the training target as a starting point and sweep the skip bias to select an operating point for the desired inference budget.
Question #4
Response: It is a clear typo. Thank you for pointing it out. We have modified it accordingly in the new draft.