- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents GLoRE, a dynamic expert-skipping method for batched Mixture-of-Experts (MoE) inference, aiming to reduce the memory access overhead associated with activating a large number of distinct experts across tokens. The authors identify the challenge of balancing batch-level expert reduction with token-level accuracy preservation in existing methods. To address this issue, they propose a two-level optimization strategy: (1) Global Mass-based Masking, which dynamically skips less important experts based on their aggregated router gate mass across the batch; and (2) Local Mass-based Protection, which preserves important expert activations based on each token's top-1 expert gate weight, mitigating potential accuracy degradation caused by global pruning. The proposed method requires neither additional training nor offline calibration and is integrated into vLLM through fused CUDA kernels. Experiments across four MoE models and nine datasets demonstrate that GLoRE achieves up to 1.39× higher decoding throughput while largely preserving model accuracy and maintaining consistent performance across different batch sizes.
Critical strengths
-
Simple and well-motivated method design: By combining batch-level global expert importance with token-level local importance, GLoRE reduces the number of activated experts while preserving important token-level routing decisions. This design provides a reasonable approach to balancing inference efficiency and generation quality.
-
Low deployment overhead and practical applicability: GLoRE requires neither additional training nor offline calibration, relying solely on existing router gate weights for expert selection. Its integration into vLLM through fused CUDA kernels further enhances its practical applicability, making it a lightweight and plug-and-play optimization for MoE inference.
Critical weaknesses
-
Limited methodological novelty: Several prior studies, such as LYNX, OEA, and XShare, have explored global expert pruning and local expert protection for MoE inference. Although GLoRE acknowledges these works, the paper lacks a sufficiently detailed comparison of their methodological differences and empirical performance. This makes it difficult to assess the distinct contributions and novelty of the proposed approach relative to existing methods.
-
Insufficient end-to-end evaluation of expert dropping versus re-routing: The paper highlights expert dropping, rather than re-routing experts excluded by the global masking strategy, as a key distinction from prior approaches. However, it lacks direct end-to-end comparisons of these two strategies in terms of both inference efficiency and model accuracy. Figure 13 provides only indirect evidence through intermediate metrics, which is insufficient to fully substantiate the claimed advantages of expert dropping over re-routing.
Questions for authors
-
Could the authors provide direct experimental comparisons with closely related methods, such as LYNX, OEA, and XShare, and quantitatively analyze the advantages of GLoRE over these approaches? In particular, a more detailed comparison of their expert selection and protection strategies would help clarify the methodological novelty and practical benefits of GLoRE.
-
Could the authors provide a quantitative comparison between directly dropping and re-routing token assignments to experts excluded by global masking, evaluating their impact on both end-to-end inference efficiency and model accuracy across multiple benchmarks? Such experiments would help substantiate the claimed advantages of the proposed expert-dropping strategy over existing re-routing approaches.
5 comments
Response to W1 & Q1 (part 2): Methodological and Empirical Comparison with Related Methods
2. Empirical comparison.
We compare GLoRE with related methods under matched serving workloads to evaluate their accuracy–throughput trade-offs and stability across batch sizes. We have added main results to the revised manuscript (Section 4, Table 2, Figures 3–4).
The results demonstrate that GLoRE provides a more favorable and stable accuracy–throughput trade-off across different serving batch sizes:
- Accuracy–throughput trade-off. Across four models and four batch sizes (16 settings), GLoRE achieves Pareto-optimal accuracy–throughput trade-offs in 13, more often than any other method (SERE and OEA: 10/16; XShare: 9/16; LynX: 7/16). With one shared configuration (, ) and no per-model tuning, it retains 96.8–99.3% of vanilla accuracy on three models, while its throughput gain grows from 1.00–1.16× at B=8 to 1.21–1.39× at B=64. Each baseline loses in a different way. SERE is fast but retains only 58.7–77.8% of vanilla accuracy at B=8, because its small substitution pool at small batches approximates removed experts poorly. XShare reaches 1.30× on Qwen3.6-35B-A3B at B=64, but its accuracy drops from 99.7% to 95.3–96.5%, as its fixed expert count enables speedup by limiting the active set but fails to cover the increasingly diverse routing patterns and growing expert union at larger batches. LynX and OEA preserve accuracy but achieve at most 1.09× speedup on Qwen3-30B-A3B. On Gemma-4, methods that preserve accuracy provide no speedup, while SERE and GLoRE achieve speedup with accuracy loss.
- Stability across batch sizes. Continuous batching causes the effective batch size to vary from step to step, making both accuracy and throughput stability important. GLoRE's accuracy varies by 0.9–2.5 points across B=8–64 (1.4 on average), comparable to OEA (1.6) and XShare (2.1), and well below LynX (5.6) and SERE (28.0). GLoRE is the only method whose throughput gain increases monotonically with batch size across all four models, enabling greater throughput gains at larger batches when serving systems may prioritize latency under high load. SERE, for example, peaks at B=16 or 32 and declines at B=64 on every model. This behavior is consistent with GLoRE's fractional mass budget, which adapts the number of masked experts to the current batch's routing distribution rather than imposing a fixed count.
Table. Accuracy–throughput trade-offs across batch sizes on Qwen3-30B-A3B.
| B | SERE | XShare | LynX | OEA | GLoRE (ours) |
|---|---|---|---|---|---|
| 8 | 58.74 / 1.190⋆ | 98.54 / 1.014 | 100.37 / 1.024⋆ | 99.05 / 1.063⋆ | 98.69 / 1.162⋆ |
| 16 | 82.28 / 1.261 | 98.42 / 1.102 | 99.52 / 1.033⋆ | 98.68 / 1.055⋆ | 98.46 / 1.279⋆ |
| 32 | 93.57 / 1.290 | 98.27 / 1.211⋆ | 99.41 / 1.085 | 99.48 / 1.089⋆ | 97.72 / 1.341⋆ |
| 64 | 97.20 / 1.163 | 100.55 / 1.189⋆ | 100.16 / 1.042 | 99.58 / 1.085 | 98.02 / 1.389⋆ |
Accuracy retention (%) / throughput gain (×) relative to vanilla. ⋆ indicates Pareto-optimal results within each batch size; bold indicates the highest throughput.
Response to W2 & Q2: Drop vs. Reroute Comparison
We thank the reviewer for this valuable suggestion. We conduct end-to-end experiments comparing dropping and rerouting strategies on Qwen3-30B-A3B across mixed-domain workloads. We construct mixed-domain batches by combining requests from three representative domains: math (Math500, 500 samples), general knowledge (MMLU-Pro, 400 samples), and coding (HumanEval+ and LiveCodeBench, 339 samples), for a total of 1,239 samples.
End-to-end comparison. We further implement several rerouting alternatives within GLoRE, including offline-calibrated output-similarity-based rerouting following SERE and token-wise next-rank rerouting with different treatments of the final routing weights. All methods are evaluated under the same mixed-domain workload described above. We have added these results to Appendix D.3 (Table 20). As shown in the result table, rerouting incurs a 3.65–5.19-point accuracy drop relative to vanilla (60.69), while increasing length-cap hits from 10.9% to 12.8–15.0%. SERE-style similarity rerouting performs worst in both accuracy (55.50) and length-cap hit rate (15.0%). In contrast, dropping maintains a comparable accuracy to vanilla (58.46 vs. 60.69), with fewer length-cap hits. The same trend holds for SERE: rerouting yields lower accuracy (41.57 vs. 52.94) and a higher length-cap hit rate (26.4% vs. 13.1%) than dropping.
Table. End-to-end comparison of dropping and rerouting strategies.
| Method | Variant | Macro acc | math500 | mmlu_pro | humaneval+ | lcb | Hit cap |
|---|---|---|---|---|---|---|---|
| Vanilla | Vanilla | 60.69 ±0.33 | 76.2 | 72.1 | 63.6 | 30.9 | 10.9% |
| GLoRE | Drop | 58.46 ±0.37 | 77.3 | 70.1 | 60.4 | 26.1 | 9.7% |
| Next-rank reroute, orig weight | 57.04 ±0.51 | 76.8 | 68.7 | 54.9 | 27.8 | 12.8% | |
| Next-rank reroute, renorm | 56.17 ±0.79 | 74.6 | 70.2 | 54.9 | 24.9 | 14.2% | |
| SERE similarity reroute | 55.50 ±0.83 | 75.6 | 65.9 | 56.5 | 24.0 | 15.0% | |
| SERE | Similarity reroute | 41.57 ±0.80 | 59.6 | 57.1 | 34.4 | 15.2 | 26.4% |
| Drop | 52.94 ±0.65 | 69.5 | 65.6 | 55.5 | 21.1 | 13.1% |
Impact on generation behavior. Poorly aligned substitutes can make the model prone to repetitive generation, sometimes continuing until the maximum output length even after producing a valid answer. This is reflected in the increased cap-hit rate under rerouting, a trend observed with both GLoRE and SERE. These results suggest that unsuitable substitutes can disrupt generation behavior beyond simply failing to recover masked contributions.
Scope of our claim. We do not claim that dropping is universally better than rerouting; rather, our findings highlight the risks of using poorly aligned substitutes. As discussed in Appendix G, combining dropping and rerouting through layer-aware decisions can be a promising future direction.
Response to W1 & Q1: Methodological and Empirical Comparison with Related Methods
We sincerely thank the reviewer for the thoughtful review and constructive questions, which led us to add a detailed comparison with related methods and an end-to-end drop-versus-reroute analysis that clarify GLoRE's design choices and strengthen the paper.
We thank the reviewer for highlighting the need for a more detailed comparison with closely related methods. We provide a comprehensive methodological comparison of these approaches, along with additional experimental results. We have added the comparison below to the revised manuscript (Table 5, Appendix A.2), together with a strategy-level summary (Table 1, Section 2).
1. Methodological comparison
Expert selection. Existing methods adopt different strategies for constructing the active expert pool. Union-based methods (SERE and OEA) retain the union of per-token top-k0/top-S routes, making the active-set size dependent on route coverage across the batch. Fixed-count methods (XShare, ExFold, and BASE) retain a predetermined number of experts according to batch-level rankings. These strategies have different limitations under varying batch sizes: union-based methods may provide insufficient coverage at small batches or limited pruning at large batches, while fixed-count methods cannot adapt the expert budget to the current routing distribution. For example, on Qwen3-30B-A3B, keeping each token's top-1 expert leaves a shared pool of only 7.2 experts on average at B=8, whereas keeping the top-4 at B=128 avoids only 19% of expert loads (Section 2.2). A fixed count calibrated at B=64 removes only 0.5% of gate mass at B=8 but 20.5% at B=128, while GLoRE's mass budget removes 19.2–19.6% at every batch size (Table 21).
LynX employs confidence-based binning and batch-level expert scoring to construct an adaptive active set. However, its active-set size is indirectly determined by the batch's confidence distribution, without an explicit mechanism to control how the expert budget adapts to varying batch sizes. Empirically, the same LynX setting yields at most 1.09× throughput on three models, but on GLM-4.5-Air at B=8 it removes much more (1.18×) and its accuracy drops to 83.6% (Table 8).
Route protection. The methods also differ in how they handle important routes that may be excluded by global expert selection. Union-based methods inherently preserve each token's top-k0 experts, regardless of their importance. Batch-level ranking in fixed-count methods, in contrast, does not explicitly guarantee the preservation of locally important experts for individual tokens. LynX provides confidence-based top-1 route protection, but couples it with active-set selection through confidence binning, making the expert budget and route protection difficult to control independently. Without any protection, a batch-level mask drops the top-1 expert of 10.1–11.7% of tokens at B ≥ 64 (Table 22).
GLoRE's global selection–local protection. GLoRE combines a fractional mass budget for global expert selection that adapts to varying batch sizes with local protection of each token's dominant top-1 route. These two mechanisms are independently configurable, providing a controllable accuracy–efficiency trade-off for workload-dependent serving priorities. Each has a measured effect: