acceptodds
Under review as a conference paper at ICLR 2027

Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation

Abstract

Mixture-of-Experts (MoE) models are increasingly deployed alongside Speculative Decoding (SD) to accelerate inference, but combining the two is challenging. SD improves the inference speed of dense models by verifying groups of tokens in parallel. However, the inference speedup for SD with MoEs depends heavily on the number of tokens being verified. Using more verification tokens results in more experts being transferred from DRAM to the Neural Processing Unit (NPU), which increases the memory transfer cost. This negatively impacts model runtime, as memory transfer is typically the bottleneck in inference. In this work, we investigate the impact of MoE router design during training on the speed of MoEs with SD. We find that routers with high degrees of expert coactivation result in much faster runtimes, mitigating the impact of using more verification tokens. Motivated by this observation, we assess the impact of various router design choices on expert coactivation and runtime using billion-parameter transformer models. We find that combining a global load-balancing loss, shared experts, a consistency loss, and an autoregressive expert selection mechanism during training results in significantly stronger expert coactivation. This increased coactivation translates into higher overall runtime throughput: our exploration yields a model that improves throughput by over MoE baselines, while maintaining on-par accuracy with the baseline MoE.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Speculative Decoding, Efficient Inference, Expert Coactivation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NvAqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reduced effectiveness of speculative decoding (SD) in Mixture-of-Experts (MoE) models. The authors observe that verifying multiple tokens can activate a larger set of distinct experts, increasing memory-access costs. To address this, they propose improving expert coactivation through global load balancing, shared experts, a consistency loss, and autoregressive expert selection. Experiments on approximately 1B-parameter MoE models report a 21% throughput improvement over the standard MoE baseline while maintaining comparable model quality.

Critical strengths

S1: The paper studies a practically relevant interaction between Mixture-of-Experts routing and speculative decoding. It connects larger verification windows to a greater number of distinct activated experts and shows a corresponding increase in MoE FFN runtime, providing a clear motivation for improving expert coactivation.

S2: The proposed training-oriented approach is interesting, as it improves expert reuse without modifying the speculative decoding algorithm itself. The experiments demonstrate consistent throughput improvements across different verification window sizes and tasks, with a reported 21% speedup over the standard MoE baseline while maintaining comparable model quality.

Critical weaknesses

W1: The compatibility of autoregressive expert selection with the exactness guarantee of speculative decoding is insufficiently specified. Equation (9) is defined within each SD verification window, with the i=0i=0 token selecting experts independently and later tokens depending on the preceding expert set. As written, it is unclear whether this expert-selection chain is reset at each verification-window boundary. If it is reset, routing for a token may depend on how the sequence is partitioned into verification windows, making it unclear what fixed autoregressive target distribution speculative decoding is preserving.

Sequential dependence alone would not necessarily violate exactness if the expert-selection state is part of the target model's causal state and is carried across windows. However, the paper does not explain how this state is initialized, committed after partial acceptance, or discarded or recomputed after rejection. It also does not provide a correctness argument or report whether the sequential expert-selection overhead is included in the runtime measurements.

W2: The inference setup and throughput methodology lack details needed to interpret and reproduce the reported speedups. The appendix identifies NVIDIA H100 GPUs for training but does not clearly specify the inference hardware, batch size, numerical precision, expert-weight residency, memory hierarchy, or routing implementation. It is also unclear whether throughput is measured using an end-to-end speculative decoding implementation or computed from Equation (2) using separately measured runtimes and task-dependent acceptance rates.

The paper attributes the improvement primarily to reduced expert-weight transfers, but reports neither direct memory-traffic measurements nor a runtime breakdown separating routing, expert computation, memory access, draft generation, and verification. These omissions make it difficult to establish the source of the reported 21% gain and assess its generality.

Questions for authors

Q1: In Equation (9), does Ei−1E_{i-1} refer to the preceding token globally or only to the preceding token within the current verification window? Please define the corresponding token-by-token autoregressive decoding procedure and explain how expert-selection state is propagated, committed after partial acceptance, and discarded or recomputed after rejection. Can the authors provide either a correctness argument or an empirical distribution-equivalence test?

Q2: Could the authors clarify the inference configuration and throughput methodology, including the hardware, batch size, numerical precision, expert-weight residency, routing kernel, draft-model cost, and source of the task-dependent acceptance rates? Are the reported results end-to-end measurements, and do they include the overhead of autoregressive expert selection?

Q3: In the shared-expert setting, does k=4k=4 denote the total number of active experts or the number of routed experts in addition to the SS shared experts? Equation (8) appears to activate SS shared experts plus a top-kk routed set, while the main comparison reports 353M active parameters for k=4k=4. Please clarify whether the router instead selects k−Sk-S routed experts and how active parameters and computation are matched across methods.

Q4: What value of the consistency-loss coefficient λC\lambda_C is used? Could the authors also report the full method without the consistency loss, namely global load balancing, shared experts, and autoregressive expert selection only, so that the contribution of the consistency loss can be isolated?

1 comment
AuthorsOpenReview ↗

Response to Reviewer NvAq

We thank Reviewer NvAq for their time and detailed feedback! We are glad the reviewer finds the MoE/SD interaction practically relevant, our motivation clear (larger verification windows activate more distinct experts and increase MoE FFN runtime), and our training-based approach interesting since it improves expert reuse without modifying the SD algorithm. We address each point below.

W1 / Q1. AES and exactness of SD. We thank the reviewer for this precise question. SD verification itself is unchanged (Sec. 2): drafts are accepted or rejected by the standard rule against the distribution the verifier computes. In our implementation, the AES chain of Eq. 9 is reset at the start of each verification pass: the first token of the window selects its top-kk experts independently, and later tokens in the same window depend on the preceding token's expert set. Nothing is carried across passes, so there is no expert-selection state to commit or discard after acceptance or rejection. Consequently, SD is exact with respect to the verifier's per-pass distribution, but routing at positions i>0i>0 is constrained by AES, so the resulting outputs are not identical to those of unconstrained top-kk decoding of the same weights, and the routing a token receives can depend on where its verification window begins. We therefore regard AES as part of how the model is trained and run with SD, not as an exact simulation of independent top-kk routing. Its effect on quality is measured, not assumed: the models are trained with this routing and their accuracy matches the MoE baseline (Sec. 5.3, 5.4). A variant that resets the chain at fixed absolute positions would make routing independent of the window partition and define a single fixed autoregressive model; this is a natural refinement, and we will describe it and the above precisely in Sec. 4.4. We did not run a distribution-equivalence test.

W2 / Q2. Inference setup and source of the 21% gain. Throughput follows Eq. 2: we use the measured verification runtime M(γ+1)\mathcal{M}(\gamma+1) of each MoE at each window size at batch size 1, the draft cost A\mathcal{A} (a small approximation model, 3% of the target's size, Sec. 5.1), and per-task acceptance rates (Sec. 3, 5.5, Table 1, Figs. 6, 7). The measured runtime includes the AES routing, which only adds a top-kk over the N=16N=16 router scores per token (Eq. 9).

The gain comes from a chain that the paper measures at each step. (i) Verifying a window loads the union of its tokens' experts, and this union grows with window size (Fig. 2a). (ii) MoE FFN runtime grows linearly with the number of active experts (Fig. 2b), so a larger union directly increases M(γ+1)\mathcal{M}(\gamma+1) (Fig. 2c). (iii) Our components lower the number of active experts: at a 16-token window the standard MoE loads about 9.3 of 16 experts, versus about 5.3 for our method, with each component alone also reducing it (Fig. 3c). (iv) The smaller union lowers M(γ+1)\mathcal{M}(\gamma+1), which raises throughput at every window size (Fig. 3d), up to the 21% gain over the standard MoE at matched accuracy (Sec. 5.3, Fig. 4d). In Table 1, the dense model and the MoE have the same single-token speed (240 tok/s), yet SD speeds up the MoE less (e.g. 3.38×3.38\times vs. 4.45×4.45\times on message drafting), a gap due to verification cost, which our method reduces. We did not measure DRAM traffic directly or provide a per-component runtime breakdown; the linear relation between active experts and runtime in Fig. 2b is our evidence for transfer-bound behavior.

Q3. Shared experts and kk. k=4k=4 is the total number of active experts per token, including the S=2S=2 shared experts (2 shared + 2 routed), so active parameters are matched at 353M across the MoE variants (Table 2). We will clarify Eq. 8 and Table 2 accordingly.

Q4. Consistency loss coefficient and ablation. λC=0.01\lambda_C=0.01; we will state it in App. A. We do not have a run of the full method without the consistency loss, and agree it would isolate its contribution. The paper shows that each component alone (global loss, shared experts, AES) reduces active experts relative to the MoE baseline, and that the combination with the consistency loss gives the fewest active experts and highest throughput at all window sizes (Fig. 3c,d; Fig. 5). Sec. 5.4 also notes that AES alone gives a small quality loss that the combined method recovers.

Thank you again for the careful reading, which helped us improve the clarity of the paper. We hope our responses address your concerns, and we are happy to answer any further questions. We would be grateful if the reviewer would reconsider their score in light of these clarifications.

Reviewer_HKksOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper applies classical MoE inference optimization techniques to speculative decoding verification. While the proposed ideas may be worthwhile, the experimental evidence is unconvincing, with insufficient emphasis on relevant metrics and a lack of clean, interpretable experiments. I lean toward rejection without substantial revisions to the experimental methodology and presentation. More fundamentally, the proposed modifications to the verification step raise concerns about whether the method preserves the lossless property of speculative decoding, a foundational guarantee of the technique.

Critical strengths
  1. The experimental results demonstrate promising throughput improvements with this MoE inference optimization method
Critical weaknesses
  1. Missing multi-batch inference experiments: The experiments focus on verification-window size without systematically varying the number of concurrent decoding sequences.

  2. Limited evaluation with modern speculative decoding methods: The paper does not include experiments with EAGLE-3, DFlash, or other competitive drafting approaches.

  3. Applicability beyond speculative decoding: The underlying principle of expert reuse is not inherently specific to speculative decoding, yet the paper focuses exclusively on speculative verification.

  4. Insufficient experimental documentation: The speculative decoding setup is inadequately documented, particularly regarding the draft model and its training methodology. Additionally, acceptance rates are not reported alongside throughput metrics, making the results difficult to interpret.

  5. Correctness and losslessness of speculative decoding: It is unclear whether the proposed modifications to the verification step preserve speculative decoding's lossless guarantee.

Questions for authors
  1. Correctness and losslessness of speculative decoding: Do the proposed modifications to the verification step preserve speculative decoding's lossless guarantee? How is this ensured?

  2. Missing multi-batch inference experiments: The experiments focus on verification-window size without systematically varying the number of concurrent decoding sequences. Can the authors provide experiments across different inference batch sizes?

  3. Limited evaluation with modern speculative decoding methods: Can the authors evaluate their approach with more competitive speculative decoding methods, such as EAGLE-3 and DFlash?

  4. Applicability beyond speculative decoding: Given that the underlying principle of expert reuse is not inherently specific to speculative decoding, why is the proposed approach applied specifically to speculative verification? Could it also benefit regular autoregressive decoding?

  5. Insufficient documentation of the speculative decoding setup: Can the authors provide more details about the speculative decoding configuration and report acceptance rates alongside throughput metrics?

  6. Draft model training and configuration: How are the draft models trained? Are separate draft models trained for each target model configuration? How do acceptance rates vary across different target models, and are these differences accounted for in the reported throughput comparisons?

  7. Interpretation of Table 1: Since speculative decoding throughput depends on acceptance rates, how much of the observed differences in Table 1 can be attributed to verification latency versus differences in draft-model quality and acceptance rates?

  8. Missing acceptance-rate metrics: Throughout the paper, several sections and figures discuss throughput without reporting the corresponding acceptance rates. Can the authors provide these metrics to clarify the interpretation of the reported throughput improvements?

1 comment
AuthorsOpenReview ↗

Response to Reviewer HKks

We thank Reviewer HKks for their time and detailed feedback! We are glad the reviewer finds the throughput improvements promising, which is the central claim of our paper. We address each point below.

W1 / Q2. Multi-batch experiments. Our method is a training-time change to the router, so the learned behavior does not depend on how many sequences are decoded concurrently. The typical SD deployment, especially on memory-limited devices, pairs a draft model with a verifier (target) model and decodes a single sequence, which is the setting we study (Sec. 1, Fig. 2a, 3c). There, the reduction in distinct experts per verification window translates directly into lower transfer cost and higher throughput. With several concurrent sequences, the same reduction applies within every sequence, so each one loads fewer distinct experts per window.

W2 / Q3. Modern SD methods (EAGLE-3, DFlash). We use standard SD (Sec. 2): a small approximation model (3% of the target's size, Sec. 5.1) drafts γ\gamma tokens and the target verifies γ+1\gamma+1 tokens in one pass. Our method does not depend on how draft tokens are produced. The problem arises whenever the target processes a set of candidate tokens in one pass: neighboring tokens choose different experts, so more expert weights are transferred from DRAM (Sec. 3, Fig. 2). We change only the target's router during training (Sec. 4) and leave drafting and verification untouched (Sec. 1, 6). In Eq. 2 the drafter affects only the accepted length vv and cost A\mathcal{A}, while we reduce only M(γ+1)\mathcal{M}(\gamma+1). Tree-based and block drafters verify more tokens per step, which makes the problem larger. Our four tasks also span very different acceptance rates (MoE SD speedup from 1.42×1.42\times to 3.38×3.38\times, Table 1, Figs. 6, 7), and our method has the highest peak throughput on all four, with larger gains on easier tasks (Sec. 5.5), which behave like a stronger drafter. Since nothing in our method is specific to a drafter, we believe the results generalize.

W3 / Q4. Applicability beyond SD. We agree that expert reuse is not inherently specific to SD, which we see as a strength: the same coactivation should help whenever several tokens of one sequence are processed against the same loaded weights, e.g. chunked prefill or expert caching (Sec. 6, Skliar et al.). We focus on SD because it is where the problem arises: with SD the target receives many tokens at once in a single inference call, so the experts needed by all of them must be loaded into compute memory together, which slows verification for MoEs (Sec. 3, Fig. 2). In plain autoregressive decoding of a single sequence, each step loads exactly kk experts per layer regardless of routing, so there are no extra transfers to remove. SD is what creates the problem our method solves.

W4 / Q5, Q6, Q8. SD setup, draft model, acceptance rates. SD itself is not the focus of the paper; we focus on how the target runs when it verifies a set of tokens from SD, which requires loading the union of the experts they select (Sec. 1, 3). The setup is in Sec. 2 and 5.1. Our method modifies only the target's router and works with any draft model, so draft training is not part of our contribution. Throughput follows Eq. 2 with per-task acceptance rates (Sec. 5.5, Table 1, Figs. 6, 7) that are the same for all target models, so differences between models come from verification cost. We will report acceptance rates next to the throughput figures.

W5 / Q1. Losslessness. We do not modify SD verification. The draft-and-verify procedure (Sec. 2) is unchanged and is exact with respect to whatever target model it is given; what changes is the target model itself, an MoE trained with a different router (Sec. 4). All quality results (Sec. 5.3, 5.4) are measured on this model with its routing active. Global load balancing, shared experts and the consistency loss are training-time or architecture choices that do not interact with verification. Only AES (Sec. 4.4) makes a token's experts depend on the previous token's; the expert set of each token is a deterministic function of the preceding tokens, so the target remains a fixed autoregressive model and SD's guarantee holds for it. We will define the expert-selection state precisely in Sec. 4.4.

Thank you again for the thoughtful questions. We hope our responses clarify the scope and correctness of our method, and we are happy to answer any further questions. We would be grateful if the reviewer would reconsider their score in light of these clarifications.

Reviewer_Ajb5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Existing speculative decoding methods' acceleration ratio degrade when applied to MoE models, since loading a significantly larger number of experts in the verification step can cause the inference bounded by the speed of memory access. To alleviate this problem, this paper proposes a training based method and achieves a higher decoding throughput with the model's task performance being on-par or slightly higher than the baseline model. In order to increase the locality of expert access during decoding, the proposed method uses a KL-divergence based loss to reduce the mismatch of the expert selection between the current and previous decoding token. By training the model with this loss, the model's diversity in expert selection is decreased, thereby reducing the memory traffic at MoEs at each verification step. Extensive experiments in this paper demonstrate that the proposed method (i) does not degrade the model, compared with similar size dense models or similar activation MoE models; (ii) achieves a higher decoding thoughput for various tasks.

Critical strengths
  1. This paper targets at a significant drawback of speculative decoding, that the locality of MoE memory access is degraded due to the massive number of tokens in the verification batch. The background section supports this argument with solid analysis and empirical results, that fully show the severeness of the existing methods. The discussion of the related works on both MoEs and speculative decoding are thorough.

  2. The presentation of this paper is clear and into details. The arguments and hyperparameters of the empirical study are fully described and discusses, and the method presentation is easy to read.

  3. The results shows the effectiveness of the proposed method. The empirical studies can fully show that the performance of the proposed method has not been degraded while achiving a significant higher decoding thoughput.

Critical weaknesses
  1. This paper omits an important type of MoE models, the auxillary loss free MoEs [1]. By removing the load balancing loss, the MoE model obtains significant performance gain, and it is considered as a more "end-to-end" method. Would adding a KL loss of expert selection between adjacent tokens still doesn't harm the model performance under this setting? Please support your arguments with additional empirical results.

  2. For finer grain settings and latent MoEs, would the proposed method still be useful? Please provide a brief discussion regarding to this question.

  3. Which speculative decoding method is tested in this paper? Recent advancements of speculative decoding introduce lots of tree-based verification methods, e.g. EAGLE-3 [2], or diffusion based methods [3]. What is the performance of the proposed method under these SD methods? Please provide some additional empirical results.

[1] Deepseek-V3 Technical Report

[2] EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

[3] DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

Questions for authors

See above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Ajb5

We thank Reviewer Ajb5 for their time and detailed feedback! We are glad the reviewer finds that our background section supports the argument with solid analysis and empirical results that show the severity of the problem for existing methods, and that our results show the effectiveness of the method, with performance not degraded and a significantly higher decoding throughput.

W1. Auxiliary-loss-free MoEs (DeepSeek-V3).

  • Built on the standard MoE recipe. Our baseline is the standard MoE with load balancing (Sec. 4.1, App. A), the most widely used setting. Our aim was to show that expert coactivation can be improved in this setup.

  • Layered on top of balancing, not tied to it. Only the consistency loss (Sec. 4.3) adds a loss on the router: a KL term between adjacent tokens' router outputs. It does not specify how load is spread over experts, so it does not push against any balancing mechanism; it only asks adjacent tokens to move between experts smoothly. AES (Sec. 4.4) is a selection rule and shared experts (Sec. 4.2) are an architecture choice, so neither adds a loss that could conflict with balancing.

  • Why we expect it to carry over. Aux-loss-free routing adds a per-expert bias to the selection scores. AES ranks by those same scores and can use them unchanged, the consistency loss does not depend on how balance is enforced, and shared experts are the DeepSeek-V3 design itself.

  • Additional ablations. Our paper already has broad experiments in the standard setup: different numbers of active experts (k=2,4,6k=2,4,6) and dense sizes (Table 2), per-component ablations (Figs. 3, 5), three perplexity corpora plus a seven-benchmark QA average (Fig. 4), and throughput across window sizes and tasks (Figs. 6, 7). Repeating all of this on a different base recipe means retraining every model for 6T tokens (Sec. 5.1), beyond our compute. Aux-loss-free MoEs are a specialized setting; we demonstrate the method in the standard one, believe it applies here too, and will list this as future work.

W2. Fine-grained and latent MoEs.

  • Why this matters in practice. We target devices where expert weights sit in DRAM and are transferred to a limited-memory accelerator, so transfer is the bottleneck (Sec. 1, 6). There, extra expert transfers during SD verification can erase the benefit of SD (Table 1, Fig. 2). Reducing distinct experts per window makes SD practical for MoEs, which we believe is worth having even with a small quality trade-off, though accuracy is on par in our experiments (Sec. 5.3).

  • Fine-grained MoEs. The problem gets worse (e.g., N=256N=256, k=8k=8 in DeepSeek-V3): a window of TT tokens can touch up to kTkT experts, and with large NN accidental overlap is rare. As an illustration, if each token picked kk of NN experts uniformly at random, a window would load N(1−(1−k/N)T)N\left(1-(1-k/N)^{T}\right) distinct experts. For T=16T=16 this is about 102102 of 256256 for N=256,k=8N=256, k=8, versus about 15.815.8 of 1616 for our N=16,k=4N=16, k=4. Real routers are more correlated (our baseline loads about 9.39.3 of 1616, Fig. 3c), but headroom is much larger. AES bounds the experts per window at k+γlk+\gamma l regardless of NN (Sec. 4.4).

  • Latent MoEs. Each transfer is cheaper, but the number of distinct experts per window still determines the bytes moved, so we expect the benefit to persist, with a smaller absolute gain.

W3. Which SD method is tested; EAGLE-3, tree-based and diffusion-based drafters.

  • SD method used. Standard speculative decoding (Sec. 2): a small approximation model (3% of the target's size, Sec. 5.1) drafts γ\gamma tokens, and the target verifies the γ+1\gamma+1 tokens in one parallel pass.

  • Our method does not depend on how draft tokens are produced. The problem arises whenever the target processes a set of candidate tokens in one pass: neighboring tokens choose different experts, so more expert weights are transferred from DRAM (Sec. 3, Fig. 2). We change only the target's router during training (Sec. 4) and leave drafting and verification untouched (Sec. 1, 6). In Eq. 2 the drafter affects only vv and A\mathcal{A}, while we reduce only M(γ+1)\mathcal{M}(\gamma+1). Tree and diffusion drafters verify more tokens per step, which makes the problem larger; for trees, AES would condition each node on its parent's expert set.

  • Evidence across drafter strength. Our four tasks span different acceptance rates (MoE SD speedup from 1.42×1.42\times to 3.38×3.38\times, Table 1, Figs. 6, 7). Our method has the highest peak throughput on all four, with larger gains on easier tasks (Sec. 5.5), which behave like a stronger drafter. Since nothing in our method is specific to a drafter, we believe the results generalize, and we will list EAGLE-3 and similar drafters as future work.

We hope this clarifies our design choices, and we would be grateful if the reviewer would consider this in their assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.