acceptodds
Under review as a conference paper at ICLR 2027

Control-MoE: Mixture of Experts with Controllable Dynamic Sparsity

Abstract

Mixture-of-Experts (MoE) has become a foundational approach for scaling large models by enabling selective activation of parameters within the multi-layer perceptron (MLP) layers. However, traditional expert selection employs a single router per layer, often leading to inflexible activation rate. Furthermore, the use of a predetermined number of experts per token constrains the adaptability of network sparsity. We propose One Choice at A Time (OCAT), an innovative routing strategy that utilizes parallel Top-1 routers for expert assignment in each layer. By integrating a skip mechanism into each router, which allows bypassing when predictive precision is maintained, OCAT determines both the relevant experts and their ideal count. To automatically regulate the overall sparsity, we further incorporate a Model Predictive Control (MPC) framework that dynamically adjusts the skip-loss weight to steer the realized skip probability toward a target setpoint without manual tuning. Comprehensive evaluations demonstrate that OCAT substantially improves both model performance and operational efficiency. Comprehensive evaluations demonstrate that OCAT substantially improves both model performance and operational efficiency, with OCAT models achieving an average 2% improvement over Top-K router baselines at comparable activation levels across different model scales. It further exhibits test-time scaling capability, allowing the average number of activated parameters to be dynamically adjusted.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, Router, Controllable Sparsity

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pcAfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Control-MoE, a dynamic routing framework for Mixture-of-Experts models. It combines parallel Top-1 gates with a lightweight skip expert to enable token-dependent expert activation. An MPC controller adjusts the skip-loss weight to regulate training-time sparsity, while the skip expert bias allows test-time activation scaling without retraining. Experiments on 2B–10B models show improved performance over fixed Top-K routing at comparable activation levels.

Critical strengths
  1. The proposed routing mechanism is simple and enables flexible expert activation, allowing different tokens to use different numbers of experts.

  2. The experiments cover multiple model scales and show improvements over fixed Top-K routing at comparable activation levels.

  3. Control-MoE supports test-time activation scaling, allowing a single checkpoint to operate under different computational budgets without retraining.

Critical weaknesses
  1. The paper lacks direct comparisons with existing dynamic MoE methods.

  2. The contribution of the MPC controller is not sufficiently analyzed. It remains unclear whether MPC offers substantial advantages over fixed skip-loss weights or simpler feedback controllers in terms of sparsity control and training stability.

  3. Equation (4) appears to contain a sign inconsistency. Minimizing the skip loss as written would decrease the skip expert probability, contradicting its stated purpose of encouraging skip expert selection.

Questions for authors
  1. How does Control-MoE compare with existing dynamic routing methods under matched computational budgets?

  2. How does MPC compare with fixed skip-loss weights or simpler feedback controllers?

  3. How does the training sparsity target affect test-time activation scaling?

  4. Could the authors clarify the negative sign in Equation (4)?

2 comments

Weakness #1

The paper lacks direct comparisons with existing dynamic MoE methods.

Response: Control-MoE contributes a predictive control mechanism that adjusts each layer’s skip-loss weight using an online model of its sparsity response. For dynamic-routing methods without automatic target tracking, matching the average activation budget requires additional calibration of routing thresholds or sparsity hyperparameters.

We thank the reviewer for pointing out ProbMoE, which we have decided to add to our paper as the new baseline. We have implemented ProbMoE Exact-4 and started a comparison on 2B models with 64 experts. Control-MoE uses 16 gates with a target of four effective experts per token. Across the first 3,415 shared training steps, Control-MoE achieves lower cross-entropy loss during the recent training stage, reaching approximately 2.30 versus 2.40 for ProbMoE. The training is ongoing, and we will report the completed results to determine whether this loss advantage translates into benchmark gains.

Weakness #2

The contribution of the MPC controller is not sufficiently analyzed. It remains unclear whether MPC offers substantial advantages over fixed skip-loss weights or simpler feedback controllers in terms of sparsity control and training stability.

Response: We have added a full ablation section showing that manually tuning fixed skip-loss weights struggles to reach the target sparsity, and new PID-control results demonstrating high fluctuation.

Action: We revised Section 4.3, added ablation tables and discussion, and included full experimental details in Appendix F.

Weakness #3

Equation (4) appears to contain a sign inconsistency. Minimizing the skip loss as written would decrease the skip expert probability, contradicting its stated purpose of encouraging skip expert selection.

Response: The reviewer correctly identified a sign error in Eq. (4). Our implementation adds the positive squared penalty to the training objective, which encourages higher skip probabilities. We have removed the leading minus sign in the revision.

Question #1

How does Control-MoE compare with existing dynamic routing methods under matched computational budgets?

Response: Please refer to Weakness #1.

Question #2

How does MPC compare with fixed skip-loss weights or simpler feedback controllers?

Response: Our ablations show that fixed skip-loss weights can approach the target sparsity after manual tuning, but different weights produce substantially different activation levels, as shown in the updated Section 4.3 and Appendix F. Several trials were terminated early because they remained below the target (Figure 7, Appendix F). MPC automates this adjustment using feedback from the realized skip rate. Against a PID baseline on the same 2B, 64-expert routing architecture, the tested PID configuration overshoots above 70% for a 50% target and continues to oscillate, as shown in Figure 8. MPC maintains smoother sparsity tracking, while both controllers achieve comparable final training losses.

MPC adds a small ARX fit and a closed-form scalar update per layer during training. The observed trade-off is therefore additional controller computation in exchange for more stable, automatic budget tracking.

Question #3

How does the training sparsity target affect test-time activation scaling?

Response: The training sparsity target shapes the range of useful test-time activation levels. In Figure 2, the 2B–384 checkpoints trained to use 4 and 6 effective experts peak near their respective training targets. Larger departures in either direction can reduce accuracy. The width of the useful range varies by checkpoint. For example, the 7B model maintains similar accuracy across roughly four to eight effective experts. We therefore use the training target as a starting point and sweep the skip bias to select an operating point for the desired inference budget.

Question #4

Could the authors clarify the negative sign in Equation (4)?

Response: It is a clear typo. Thank you for pointing it out. We have modified it accordingly in the new draft.

Thank you for the detailed response and additional experiments. The authors have adequately addressed my other concerns, particularly regarding the MPC controller and the correction to Eq. (4).

Although the comparison with existing dynamic MoE methods is still ongoing, the preliminary results already show a promising advantage for Control-MoE. Given the limited time remaining in the discussion period, I understand that the complete results may not yet be available.

I have decided to raise my rating from 2 to 3 and look forward to seeing the completed experimental results.

Reviewer_DMwPOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Control-MoE, a dynamic routing method that combines multiple independent Top-1 gates with a lightweight skip expert to enable dynamic expert activation. A skip loss encourages sparse activation, while an MPC controller models the router’s response and adaptively adjusts the loss weight to track a target activation rate. For models with 2B–10B total parameters trained on 600B tokens, the reported average downstream scores improve by approximately 0.9–2.0% points over fixed Top-K baselines at matched active expert counts.

Critical strengths
  • The empirical evaluation spans 2B, 7B, and 10B total parameters trained on 600B tokens. The analysis in Section 4.4 offers useful insights into how expert allocation varies across tokens with different linguistic characteristics.
  • The test-time activation sweeps in Figure 2 demonstrate that a single checkpoint can accommodate different inference compute budgets through skip-expert bias adjustment without retraining.
Critical weaknesses
  • The discussion and comparison with prior dynamic-sparsity MoE methods are incomplete. The distinction from MoE++ is insufficiently clarified, particularly between the proposed skip expert and MoE++’s low-cost experts. The related work discussion also omit ReMoE (https://arxiv.org/abs/2412.14711), which already combines dynamic sparsity with an adaptively weighted sparsity loss. The experiments also lack controlled comparisons with existing dynamic-sparsity methods, while the MoE++ comparison in Table 2 is not compute-matched, as MoE++ activates fewer parameters. The evidence therefore positions Control-MoE as another dynamic-sparsity method rather than establishing a new paradigm.
  • The necessity of MPC controller is not established. There are no ablations against fixed skip-loss weights or simpler feedback rules under the same routing architecture, leaving its contribution and the justification for its added complexity unclear.
Questions for authors
  • How do standard MoE baselines perform when K is varied at inference? Comparing their accuracy–compute curves with Figure 2 would clarify the advantage of skip-bias adjustment.
  • How sensitive are the results to the MPC hyperparameters in Table 6, in terms of sparsity tracking, training stability, and model quality?
  • Could the authors report training and validation language-model loss curves for Control-MoE and the matched Top-K baselines, alongside the final downstream results?
  • What are the measured end-to-end training and inference throughput and latency, accounting for routing, expert deduplication, and training-time MPC updates?
4 comments

We thank the reviewer for the constructive feedback.

Weakness #1

The discussion and comparison with prior dynamic-sparsity MoE methods are incomplete. The distinction from MoE++ is insufficiently clarified, particularly between the proposed skip expert and MoE++’s low-cost experts. The related work discussion also omit ReMoE, which already combines dynamic sparsity with an adaptively weighted sparsity loss. The experiments also lack controlled comparisons with existing dynamic-sparsity methods, while the MoE++ comparison in Table 2 is not compute-matched, as MoE++ activates fewer parameters. The evidence therefore positions Control-MoE as another dynamic-sparsity method rather than establishing a new paradigm.

Response: We agree that the related-work discussion should acknowledge MoE++ and ReMoE more explicitly. We have incorporated the ReMoE into the relative works (See Relative Work Section, marked in blue).

MoE++ uses a Top-k router with zero, copy, and constant experts. Control-MoE gives each of K parallel Top-1 gates access to a shared affine skip expert, so the per-token cap and average activation target can be set separately. Reaching a prescribed activation budget in MoE++ requires extensive human labor to calibrate across multiple training runs, and needs to be retuned for different model scale and configuration. On the other hand, Control-MoE uses per-layer MPC to predict the effect of skip-loss weight changes and track a target skip rate. MPC can adjust the skip-loss weight during training using feedback from the realized skip rate.

ReMoE tracks model-wide sparsity through a sign-based multiplicative update of its L1 weight, analogous to integral control on the sign of the error in log space. Its ablation shows that a larger update factor can shorten settling time while increasing oscillations. ReMoE’s positive ReLU outputs determine both expert selection and mixing weights, so shifting logits to change sparsity would also change the weights of experts that remain active. Control-MoE applies the skip bias only during selection and computes mixing weights from the original logits, which provides better training stability.

ControlMoE offers the ability of testing time scaling, which is not achievable in any of these approaches.

Weakness #2

The necessity of MPC controller is not established. There are no ablations against fixed skip-loss weights or simpler feedback rules under the same routing architecture, leaving its contribution and the justification for its added complexity unclear.

Response: We add a new section in the ablation study with fixed skip-loss weights. We also implemented the PID controller in the same setting. However, the PID requires substantial tuning to achieve the target sparsity level. We have appended some “painful” parameter tuning results in Fig 7, Appendix F.

Question #1

How do standard MoE baselines perform when K is varied at inference? Comparing their accuracy–compute curves with Figure 2 would clarify the advantage of skip-bias adjustment.

Response: We have added the requested motivation study in Appendix H, training a series standard 2B Top-K MoE models with different K (4 to 64). The results show that accuracy–compute curves are highly non-monotonic. While some benchmarks improve with larger K, many exhibit a clear "sweet point" where adding compute degrades performance. This directly clarifies the motivation of developing dynamic activation MoE models. Therefore, tuning the skip-bias to achieve testing time scaling is an effective approach to balance the performance and efficiency.

Question #2: Sensitivity of MPC hyperparameters

Response: We have incorporated the curves in the MPC training for all our experiments for all the models in Appendix C. The MPC parameters are not changed despite the model scale and the activation cap K change. The initial MPC parameters are determined by the runs in Fig 4, Appendix C.

Question #3: Loss Curves

Response: We have incorporated the training curves in Fig 4, Appendix C.

Question #4

What are the measured end-to-end training and inference throughput and latency, accounting for routing, expert deduplication, and training-time MPC updates?

Response: We have added a section in Appendix G and updated the Training & Inference Computation Implications Section on Page 10 to discuss the computation cost for routing and MPC updates.

The additional computation in ControlMoE is concentrated in the router, where the K parallel Top-1 projections can be combined into a single, highly parallelizable matrix multiplication. For a fixed ARX model order, each training-time controller update requires O(M) work to fit a window of M observations, followed by a closed-form scalar MPC solve. We consider this overhead an acceptable trade-off for controllable sparsity and the observed accuracy gains.

Dear Reviewer DMwP,

We have added further computational and overhead analyses, along with MPC and loss curves. We have also included comparisons with other dynamic MoE baselines (e.g., MoE++ and ReMoE) and alternative design choices, such as a fixed skip-loss weight and a PID controller.

We would be happy to provide further clarification. We thank you for your time and valuable feedback.

Thank you for the response and additional experiments. The revised discussion clarifies how Control-MoE fits into prior work on dynamic sparse MoEs. The fixed-weight and PID ablations in Appendix F also address my concern about what MPC adds. I found Figure 8 particularly helpful in showing that MPC tracks the skip probability more consistently than PID controller.

The reviewer now see this work as a modest but solid contribution to the existing literature, although some concerns remain. I would be willing to raise my rating from 2 to 3 if the authors can include a fair comparison with MoE++ and ReMoE under matched activation budgets in a future revision.

Thank you for your positive feedback; we are glad that our recent efforts have addressed most of your concerns. We now believe the key motivation of the paper "achieving controllable sparsity in a dynamic-activation MoE model" is well established.

We agree that a fair comparison with MoE++ and ReMoE under matched activation budgets would strengthen our evaluation. However, original MoE++ and ReMoE serve as two dynamic-activation MoE baselines without self-adaptive controllers, and manually controlling these methods to match activation budgets requires substantial parameter tuning. We are currently working on such hyperparameters tuning and will include a full comparison with both MoE++ and ReMoE under matched activation budgets in a forthcoming revision.

As a direct comparison, we also provide a ProbMoE baseline (Zhao, Heng, et al., “ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts,” ICML 2026), since it combines dynamic routing with a simple feedback controller similar to the I control in PID. We have implemented ProbMoE Exact-4 and started a comparison on 2B models with 64 experts. Control-MoE uses 16 gates with a target of 4 effective experts per token. Across the first 24,495 shared training steps, Control-MoE achieves lower CE loss—approximately 1.94 versus 2.09 for ProbMoE. Training is ongoing, and we will report the final results to determine whether this loss advantage translates into benchmark gains.

Reviewer_m4HDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Control-MoE, a routing scheme that replaces the single Top-K router in each MoE layer with K parallel, independent Top-1 gates, each choosing among the normal experts plus a cheap skip expert. Sparsity is further encouraged by a skip loss whose weight is adjusted online by a per-layer MPC controller to track a target skip probability. Load balancing uses an auxiliary-free bias update with a soft cap and elasticity on the skip-expert bias. Experiment results on OLMoE models trained on 600B tokens are presented with comparison with Top-K baselines on various tasks.

Critical strengths
  • The paper is clearly written and easy to follow: the method is motivated well, each component is introduced step by step, and the notation is consistent throughout.
  • Figure 1 gives a clear overview of the whole system and is very helpful for understanding the method.
  • The parallel Top-1 formulation is simple, removes the activation lower bound that most dynamic-activation methods retain, and is plug-in compatible with existing MoE layers.
  • Framing sparsity regulation as a feedback-control problem is a reasonable idea, and the same controller configuration is reportedly used across all scales without retuning.
Critical weaknesses
  • The choice of K is still a key, untreated hyperparameter. The method trades Top-k's fixed k for the number of gates K (16 or 24) plus a target p*, and Table 1 shows K matters. No guidance or sweep over K is provided, so the claim of removing manual tuning is only partially met.
  • No ablations of the many introduced components. Section 4.3 is titled "Ablations" but only contains a comparison to public models. The method stacks a skip expert, skip loss, MPC controller, skip-bias soft cap, bias elasticity, bias mean-normalization, and a modified z-loss; none is ablated. In particular there is no comparison of MPC against simpler alternatives: a fixed skip-loss weight or directly learning the skip probability.
  • Lack of dynamic-activation baselines. The related work lists Top-P/DTop-P, DA-MoE, DynaMoE, DARE, BEAM, MoE++, AdaMoE, and LongCat, yet none is trained under the matched setup. The core claim that this routing beats existing dynamic routers, not just Top-K is therefore unsupported. A closely related dynamic-routing method is missing: Zhao, Heng, et al., "ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts," ICML 2026
  • Gate diversity/collapse is not analyzed. Since nothing prevents all gates from selecting the same expert, the method could degenerate to a few distinct experts per token. The paper reports no statistics on how many distinct normal experts are selected, how often gates coincide, or per-layer skip rates.
  • All experiments use OLMoE-derived models, so it is unclear how well the method generalizes to other MoE architectures. Results on at least one additional backbone would strengthen the generality claim.
Questions for authors
  • How should a practitioner choose K? Is performance sensitive to K at fixed target effective experts, and what is the router FLOP/parameter cost as K grows?
  • Does "effective experts" count duplicate gate selections of the same expert? If so, what is the average number of distinct normal experts per token, and how often do gates coincide?
  • How does MPC compare to a fixed w_skip swept over a grid or a directly learned skip probability? Is MPC necessary, or just one convenient way to hit a target?
  • Can you provide a controlled comparison to at least a few of the dynamic-activation baselines in Section 2?
2 comments

We thank the reviewer for the constructive feedback.

Weakness 1: Choice of K

The choice of K is still a key, untreated hyperparameter. The method trades Top-k's fixed k for the number of gates K (16 or 24) plus a target p*, and Table 1 shows K matters. No guidance or sweep over K is provided, so the claim of removing manual tuning is only partially met.

Response: We agree that K remains an architectural choice. Control-MoE separates the per-token activation cap from the average expert budget. Practitioners can choose the cap K to fit their training and inference resources, then set p*=1-k/K to target k non-skip selections per token on average. MPC automates the tuning of the skip-loss weight to track that target. At 4 effective experts, the K=16 and K=24 variants in Table 1 score 42.45% and 42.28%, respectively, versus 41.14% for Top-4. The gain holds at both tested settings. A broader sweep would help characterize the trade-off across resource budgets.

Weakness 2: Missing Ablations

No ablations of the many introduced components. Section 4.3 is titled "Ablations" but only contains a comparison to public models. The method stacks a skip expert, skip loss, MPC controller, skip-bias soft cap, bias elasticity, bias mean-normalization, and a modified z-loss; none is ablated. In particular there is no comparison of MPC against simpler alternatives: a fixed skip-loss weight or directly learning the skip probability.

Response: We agree that the original Sec 4.3 did not adequately isolate the introduced components. We have added the requested ablations.

Action: We have updated Sec 4.3 to include component-wise ablations for the K parallel Top-1 routing mechanism, the skip expert, and the fixed skip-loss weight without MPC. The main result is in Table 3 and the full benchmark results are in Appendix D.

We further provide Figure 7 in Appendix F on the manual tuning of fixed skip-loss weights with MPC disabled. The traces illustrate the sensitivity of the realized skip probability to the weight. Manually selected weights could bring the model sparsity close to the target, but finding these weights requires repeated training trials, including computation spent on candidates that are subsequently abandoned. This motivates the MPC controller which adjusts w_skip online using feedback from realized skip prob to track the target.

Weakness 3

Lack of dynamic-activation baselines. The related work lists Top-P/DTop-P, DA-MoE, DynaMoE, DARE, BEAM, MoE++, AdaMoE, and LongCat, yet none is trained under the matched setup. The core claim that this routing beats existing dynamic routers, not just Top-K is therefore unsupported. A closely related dynamic-routing method is missing: ProbMoE.

Response: Control-MoE contributes a predictive control mechanism that adjusts each layer’s skip-loss weight using an online model of its sparsity response. For dynamic-routing methods without automatic target tracking, matching the average activation budget requires additional calibration of routing thresholds or sparsity hyperparameters.

We have decided to add ProbMoE as the new baseline. We have implemented ProbMoE Exact-4 and started a comparison on 2B models with 64 experts. Control-MoE uses 16 gates with a target of 4 effective experts per token. Across the first 3,415 shared training steps, Control-MoE achieves lower CE loss, reaching approximately 2.30 versus 2.40 for ProbMoE. Training is ongoing, and we will report the final results to see whether this loss advantage translates into benchmark gains.

Question 1:

How should a practitioner choose K? Is performance sensitive to K at fixed target effective experts, and what is the router FLOP/parameter cost as K grows?

Response: As noted in Weakness #1, practitioners can choose the upper limit K to match their training/inference budget. We further provide Table 15 in Appendix G comparing the computation of Top-K and ControlMoE below. ControlMoE uses K parallel Top-1 gates; although the router’s FLOP and parameter costs grow linearly with K, the gates can be evaluated in a single wider matrix multiplication, which is highly parallelizable. Please see Appendix G for the details.

Question 2:

Does "effective experts" count duplicate gate selections of the same expert? If so, what is the average number of distinct normal experts per token, and how often do gates coincide?

Response: We count only distinct non-skip experts per token and layer as “effective experts”, since duplicate gate selections can be merged before the actual expert computation. The collision rate for two gates select the same expert is very low even after training (<1%, around 1/N, N is total number of experts). We provide a new Figure 6 in Appendix D describing the distribution of token dispatching at inference time.

Question 3: MPC v.s. Fixed w_skip

Response: Please refer to Weakness #2.

Question 4: New Baseline

Response: Please see Weakness #3.

Thanks for the additional results. Still, my concern about the limited diversity of MoE architectures remains. I'll keep my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.