- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the reduced effectiveness of speculative decoding (SD) in Mixture-of-Experts (MoE) models. The authors observe that verifying multiple tokens can activate a larger set of distinct experts, increasing memory-access costs. To address this, they propose improving expert coactivation through global load balancing, shared experts, a consistency loss, and autoregressive expert selection. Experiments on approximately 1B-parameter MoE models report a 21% throughput improvement over the standard MoE baseline while maintaining comparable model quality.
Critical strengths
S1: The paper studies a practically relevant interaction between Mixture-of-Experts routing and speculative decoding. It connects larger verification windows to a greater number of distinct activated experts and shows a corresponding increase in MoE FFN runtime, providing a clear motivation for improving expert coactivation.
S2: The proposed training-oriented approach is interesting, as it improves expert reuse without modifying the speculative decoding algorithm itself. The experiments demonstrate consistent throughput improvements across different verification window sizes and tasks, with a reported 21% speedup over the standard MoE baseline while maintaining comparable model quality.
Critical weaknesses
W1: The compatibility of autoregressive expert selection with the exactness guarantee of speculative decoding is insufficiently specified. Equation (9) is defined within each SD verification window, with the token selecting experts independently and later tokens depending on the preceding expert set. As written, it is unclear whether this expert-selection chain is reset at each verification-window boundary. If it is reset, routing for a token may depend on how the sequence is partitioned into verification windows, making it unclear what fixed autoregressive target distribution speculative decoding is preserving.
Sequential dependence alone would not necessarily violate exactness if the expert-selection state is part of the target model's causal state and is carried across windows. However, the paper does not explain how this state is initialized, committed after partial acceptance, or discarded or recomputed after rejection. It also does not provide a correctness argument or report whether the sequential expert-selection overhead is included in the runtime measurements.
W2: The inference setup and throughput methodology lack details needed to interpret and reproduce the reported speedups. The appendix identifies NVIDIA H100 GPUs for training but does not clearly specify the inference hardware, batch size, numerical precision, expert-weight residency, memory hierarchy, or routing implementation. It is also unclear whether throughput is measured using an end-to-end speculative decoding implementation or computed from Equation (2) using separately measured runtimes and task-dependent acceptance rates.
The paper attributes the improvement primarily to reduced expert-weight transfers, but reports neither direct memory-traffic measurements nor a runtime breakdown separating routing, expert computation, memory access, draft generation, and verification. These omissions make it difficult to establish the source of the reported 21% gain and assess its generality.
Questions for authors
Q1: In Equation (9), does refer to the preceding token globally or only to the preceding token within the current verification window? Please define the corresponding token-by-token autoregressive decoding procedure and explain how expert-selection state is propagated, committed after partial acceptance, and discarded or recomputed after rejection. Can the authors provide either a correctness argument or an empirical distribution-equivalence test?
Q2: Could the authors clarify the inference configuration and throughput methodology, including the hardware, batch size, numerical precision, expert-weight residency, routing kernel, draft-model cost, and source of the task-dependent acceptance rates? Are the reported results end-to-end measurements, and do they include the overhead of autoregressive expert selection?
Q3: In the shared-expert setting, does denote the total number of active experts or the number of routed experts in addition to the shared experts? Equation (8) appears to activate shared experts plus a top- routed set, while the main comparison reports 353M active parameters for . Please clarify whether the router instead selects routed experts and how active parameters and computation are matched across methods.
Q4: What value of the consistency-loss coefficient is used? Could the authors also report the full method without the consistency loss, namely global load balancing, shared experts, and autoregressive expert selection only, so that the contribution of the consistency loss can be isolated?
Response to Reviewer NvAq
We thank Reviewer NvAq for their time and detailed feedback! We are glad the reviewer finds the MoE/SD interaction practically relevant, our motivation clear (larger verification windows activate more distinct experts and increase MoE FFN runtime), and our training-based approach interesting since it improves expert reuse without modifying the SD algorithm. We address each point below.
W1 / Q1. AES and exactness of SD. We thank the reviewer for this precise question. SD verification itself is unchanged (Sec. 2): drafts are accepted or rejected by the standard rule against the distribution the verifier computes. In our implementation, the AES chain of Eq. 9 is reset at the start of each verification pass: the first token of the window selects its top-k experts independently, and later tokens in the same window depend on the preceding token's expert set. Nothing is carried across passes, so there is no expert-selection state to commit or discard after acceptance or rejection. Consequently, SD is exact with respect to the verifier's per-pass distribution, but routing at positions i>0 is constrained by AES, so the resulting outputs are not identical to those of unconstrained top-k decoding of the same weights, and the routing a token receives can depend on where its verification window begins. We therefore regard AES as part of how the model is trained and run with SD, not as an exact simulation of independent top-k routing. Its effect on quality is measured, not assumed: the models are trained with this routing and their accuracy matches the MoE baseline (Sec. 5.3, 5.4). A variant that resets the chain at fixed absolute positions would make routing independent of the window partition and define a single fixed autoregressive model; this is a natural refinement, and we will describe it and the above precisely in Sec. 4.4. We did not run a distribution-equivalence test.
W2 / Q2. Inference setup and source of the 21% gain. Throughput follows Eq. 2: we use the measured verification runtime M(γ+1) of each MoE at each window size at batch size 1, the draft cost A (a small approximation model, 3% of the target's size, Sec. 5.1), and per-task acceptance rates (Sec. 3, 5.5, Table 1, Figs. 6, 7). The measured runtime includes the AES routing, which only adds a top-k over the N=16 router scores per token (Eq. 9).
The gain comes from a chain that the paper measures at each step. (i) Verifying a window loads the union of its tokens' experts, and this union grows with window size (Fig. 2a). (ii) MoE FFN runtime grows linearly with the number of active experts (Fig. 2b), so a larger union directly increases M(γ+1) (Fig. 2c). (iii) Our components lower the number of active experts: at a 16-token window the standard MoE loads about 9.3 of 16 experts, versus about 5.3 for our method, with each component alone also reducing it (Fig. 3c). (iv) The smaller union lowers M(γ+1), which raises throughput at every window size (Fig. 3d), up to the 21% gain over the standard MoE at matched accuracy (Sec. 5.3, Fig. 4d). In Table 1, the dense model and the MoE have the same single-token speed (240 tok/s), yet SD speeds up the MoE less (e.g. 3.38× vs. 4.45× on message drafting), a gap due to verification cost, which our method reduces. We did not measure DRAM traffic directly or provide a per-component runtime breakdown; the linear relation between active experts and runtime in Fig. 2b is our evidence for transfer-bound behavior.
Q3. Shared experts and k. k=4 is the total number of active experts per token, including the S=2 shared experts (2 shared + 2 routed), so active parameters are matched at 353M across the MoE variants (Table 2). We will clarify Eq. 8 and Table 2 accordingly.
Q4. Consistency loss coefficient and ablation. λC=0.01; we will state it in App. A. We do not have a run of the full method without the consistency loss, and agree it would isolate its contribution. The paper shows that each component alone (global loss, shared experts, AES) reduces active experts relative to the MoE baseline, and that the combination with the consistency loss gives the fewest active experts and highest throughput at all window sizes (Fig. 3c,d; Fig. 5). Sec. 5.4 also notes that AES alone gives a small quality loss that the combined method recovers.
Thank you again for the careful reading, which helped us improve the clarity of the paper. We hope our responses address your concerns, and we are happy to answer any further questions. We would be grateful if the reviewer would reconsider their score in light of these clarifications.