acceptodds
Under review as a conference paper at ICLR 2027

GLoRE: Global-Local Router-Mass Expert Skipping for Batched MoE Serving

Abstract

Mixture-of-Experts (MoE) scales model capacity without proportionally increasing per-token compute and now underpins many large language models. This sparsity is less effective during serving: decode is dominated by weight movement, so each step's cost depends on the batch's union of selected experts rather than any one token's top- experts. We introduce GLoRE, a runtime expert-masking method that reduces this union through global expert masking for efficiency and local protection for quality. At each decoding step, GLoRE masks experts with low aggregate router gate mass under a batch-level budget while protecting a token's top expert when its routing weight exceeds a threshold. This adapts expert reduction to each batch while preserving important token-level assignments, and remains robust as batch sizes vary. Implemented as a plug-and-play CUDA kernel for vLLM, GLoRE achieves up to decode throughput at batch size 64 with only 2% accuracy loss on Qwen3-30B-A3B, and remains on the accuracy–throughput Pareto frontier across four MoE models. On long chain-of-thought test-time scaling, GLoRE maintains vanilla-level maj@8 accuracy on AIME and GPQA-Diamond while using up to fewer GPU-seconds.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts, MoE serving, LLM serving, efficient inference, expert skipping, batched decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yd5JOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GLoRE, a dynamic expert-skipping method for batched Mixture-of-Experts (MoE) inference, aiming to reduce the memory access overhead associated with activating a large number of distinct experts across tokens. The authors identify the challenge of balancing batch-level expert reduction with token-level accuracy preservation in existing methods. To address this issue, they propose a two-level optimization strategy: (1) Global Mass-based Masking, which dynamically skips less important experts based on their aggregated router gate mass across the batch; and (2) Local Mass-based Protection, which preserves important expert activations based on each token's top-1 expert gate weight, mitigating potential accuracy degradation caused by global pruning. The proposed method requires neither additional training nor offline calibration and is integrated into vLLM through fused CUDA kernels. Experiments across four MoE models and nine datasets demonstrate that GLoRE achieves up to 1.39× higher decoding throughput while largely preserving model accuracy and maintaining consistent performance across different batch sizes.

Critical strengths
  1. Simple and well-motivated method design: By combining batch-level global expert importance with token-level local importance, GLoRE reduces the number of activated experts while preserving important token-level routing decisions. This design provides a reasonable approach to balancing inference efficiency and generation quality.

  2. Low deployment overhead and practical applicability: GLoRE requires neither additional training nor offline calibration, relying solely on existing router gate weights for expert selection. Its integration into vLLM through fused CUDA kernels further enhances its practical applicability, making it a lightweight and plug-and-play optimization for MoE inference.

Critical weaknesses
  1. Limited methodological novelty: Several prior studies, such as LYNX, OEA, and XShare, have explored global expert pruning and local expert protection for MoE inference. Although GLoRE acknowledges these works, the paper lacks a sufficiently detailed comparison of their methodological differences and empirical performance. This makes it difficult to assess the distinct contributions and novelty of the proposed approach relative to existing methods.

  2. Insufficient end-to-end evaluation of expert dropping versus re-routing: The paper highlights expert dropping, rather than re-routing experts excluded by the global masking strategy, as a key distinction from prior approaches. However, it lacks direct end-to-end comparisons of these two strategies in terms of both inference efficiency and model accuracy. Figure 13 provides only indirect evidence through intermediate metrics, which is insufficient to fully substantiate the claimed advantages of expert dropping over re-routing.

Questions for authors
  1. Could the authors provide direct experimental comparisons with closely related methods, such as LYNX, OEA, and XShare, and quantitatively analyze the advantages of GLoRE over these approaches? In particular, a more detailed comparison of their expert selection and protection strategies would help clarify the methodological novelty and practical benefits of GLoRE.

  2. Could the authors provide a quantitative comparison between directly dropping and re-routing token assignments to experts excluded by global masking, evaluating their impact on both end-to-end inference efficiency and model accuracy across multiple benchmarks? Such experiments would help substantiate the claimed advantages of the proposed expert-dropping strategy over existing re-routing approaches.

5 comments
AuthorsOpenReview ↗

Response to W1 & Q1: Methodological and Empirical Comparison with Related Methods

We sincerely thank the reviewer for the thoughtful review and constructive questions, which led us to add a detailed comparison with related methods and an end-to-end drop-versus-reroute analysis that clarify GLoRE's design choices and strengthen the paper.


We thank the reviewer for highlighting the need for a more detailed comparison with closely related methods. We provide a comprehensive methodological comparison of these approaches, along with additional experimental results. We have added the comparison below to the revised manuscript (Table 5, Appendix A.2), together with a strategy-level summary (Table 1, Section 2).

1. Methodological comparison

Design aspectSEREXShareLynXOEAExFoldBASEGLoRE (ours)
Expert retention criterionTop-k0k_0 routing unionAccumulated gate massConfidence-binned votesTop-k0k_0 routing unionGate × calibrated normGate² × predicted energyAccumulated gate mass
Budget formulationUnion-basedFixed expert countAdaptive active setUnion-basedFixed expert countFixed expert countFractional mass budget (β\beta)
Batch-size adaptabilityPartlyNoPartlyPartlyNoNoYes
Route protectionTop-k0k_0 routesTop-k0k_0 routesImplicit confidence-based top-1Top-k0k_0 routesNoneNoneExplicit dominant top-1 (μ\mu)
Excluded-route handlingReroute (similarity-based)Reroute (recomputed Top-kk)Reroute (confidence-aware remapping)Reroute (next-ranked backfilling)Reroute (scaled projection-loss-based)Reroute (next-ranked backfilling)Drop (no rerouting)
Offline calibrationSimilarity matrixNoneNoneNoneProjector matricesLinear predictorsNone

Expert selection. Existing methods adopt different strategies for constructing the active expert pool. Union-based methods (SERE and OEA) retain the union of per-token top-k0k_0/top-SS routes, making the active-set size dependent on route coverage across the batch. Fixed-count methods (XShare, ExFold, and BASE) retain a predetermined number of experts according to batch-level rankings. These strategies have different limitations under varying batch sizes: union-based methods may provide insufficient coverage at small batches or limited pruning at large batches, while fixed-count methods cannot adapt the expert budget to the current routing distribution. For example, on Qwen3-30B-A3B, keeping each token's top-1 expert leaves a shared pool of only 7.2 experts on average at B=8, whereas keeping the top-4 at B=128 avoids only 19% of expert loads (Section 2.2). A fixed count calibrated at B=64 removes only 0.5% of gate mass at B=8 but 20.5% at B=128, while GLoRE's mass budget removes 19.2–19.6% at every batch size (Table 21).

LynX employs confidence-based binning and batch-level expert scoring to construct an adaptive active set. However, its active-set size is indirectly determined by the batch's confidence distribution, without an explicit mechanism to control how the expert budget adapts to varying batch sizes. Empirically, the same LynX setting yields at most 1.09× throughput on three models, but on GLM-4.5-Air at B=8 it removes much more (1.18×) and its accuracy drops to 83.6% (Table 8).

Route protection. The methods also differ in how they handle important routes that may be excluded by global expert selection. Union-based methods inherently preserve each token's top-k0k_0 experts, regardless of their importance. Batch-level ranking in fixed-count methods, in contrast, does not explicitly guarantee the preservation of locally important experts for individual tokens. LynX provides confidence-based top-1 route protection, but couples it with active-set selection through confidence binning, making the expert budget and route protection difficult to control independently. Without any protection, a batch-level mask drops the top-1 expert of 10.1–11.7% of tokens at B ≥ 64 (Table 22).

GLoRE's global selection–local protection. GLoRE combines a fractional mass budget for global expert selection that adapts to varying batch sizes with local protection of each token's dominant top-1 route. These two mechanisms are independently configurable, providing a controllable accuracy–efficiency trade-off for workload-dependent serving priorities. Each has a measured effect:

  • Protection enables a larger global budget. At β=0.20\beta=0.20, protection improves five-task macro accuracy from 57.56 to 70.90 (+13.3 points), and by 40.4 points at β=0.30\beta=0.30 (Table 3).
  • Protection matters more under mixed batches. On a mixed math/knowledge/coding workload, disabling top-1 protection lowers macro accuracy from 63.4 to 56.5, a larger drop than under single-domain batching (Table 13).
AuthorsOpenReview ↗

Response to W1 & Q1 (part 2): Methodological and Empirical Comparison with Related Methods

2. Empirical comparison.

We compare GLoRE with related methods under matched serving workloads to evaluate their accuracy–throughput trade-offs and stability across batch sizes. We have added main results to the revised manuscript (Section 4, Table 2, Figures 3–4).

The results demonstrate that GLoRE provides a more favorable and stable accuracy–throughput trade-off across different serving batch sizes:

  • Accuracy–throughput trade-off. Across four models and four batch sizes (16 settings), GLoRE achieves Pareto-optimal accuracy–throughput trade-offs in 13, more often than any other method (SERE and OEA: 10/16; XShare: 9/16; LynX: 7/16). With one shared configuration (β=0.20\beta=0.20, μ=1.5\mu=1.5) and no per-model tuning, it retains 96.8–99.3% of vanilla accuracy on three models, while its throughput gain grows from 1.00–1.16× at B=8 to 1.21–1.39× at B=64. Each baseline loses in a different way. SERE is fast but retains only 58.7–77.8% of vanilla accuracy at B=8, because its small substitution pool at small batches approximates removed experts poorly. XShare reaches 1.30× on Qwen3.6-35B-A3B at B=64, but its accuracy drops from 99.7% to 95.3–96.5%, as its fixed expert count enables speedup by limiting the active set but fails to cover the increasingly diverse routing patterns and growing expert union at larger batches. LynX and OEA preserve accuracy but achieve at most 1.09× speedup on Qwen3-30B-A3B. On Gemma-4, methods that preserve accuracy provide no speedup, while SERE and GLoRE achieve speedup with accuracy loss.
  • Stability across batch sizes. Continuous batching causes the effective batch size to vary from step to step, making both accuracy and throughput stability important. GLoRE's accuracy varies by 0.9–2.5 points across B=8–64 (1.4 on average), comparable to OEA (1.6) and XShare (2.1), and well below LynX (5.6) and SERE (28.0). GLoRE is the only method whose throughput gain increases monotonically with batch size across all four models, enabling greater throughput gains at larger batches when serving systems may prioritize latency under high load. SERE, for example, peaks at B=16 or 32 and declines at B=64 on every model. This behavior is consistent with GLoRE's fractional mass budget, which adapts the number of masked experts to the current batch's routing distribution rather than imposing a fixed count.

Table. Accuracy–throughput trade-offs across batch sizes on Qwen3-30B-A3B.

BSEREXShareLynXOEAGLoRE (ours)
858.74 / 1.190⋆98.54 / 1.014100.37 / 1.024⋆99.05 / 1.063⋆98.69 / 1.162⋆
1682.28 / 1.26198.42 / 1.10299.52 / 1.033⋆98.68 / 1.055⋆98.46 / 1.279⋆
3293.57 / 1.29098.27 / 1.211⋆99.41 / 1.08599.48 / 1.089⋆97.72 / 1.341⋆
6497.20 / 1.163100.55 / 1.189⋆100.16 / 1.04299.58 / 1.08598.02 / 1.389⋆

Accuracy retention (%) / throughput gain (×) relative to vanilla. ⋆ indicates Pareto-optimal results within each batch size; bold indicates the highest throughput.

AuthorsOpenReview ↗

Response to W2 & Q2: Drop vs. Reroute Comparison

We thank the reviewer for this valuable suggestion. We conduct end-to-end experiments comparing dropping and rerouting strategies on Qwen3-30B-A3B across mixed-domain workloads. We construct mixed-domain batches by combining requests from three representative domains: math (Math500, 500 samples), general knowledge (MMLU-Pro, 400 samples), and coding (HumanEval+ and LiveCodeBench, 339 samples), for a total of 1,239 samples.

End-to-end comparison. We further implement several rerouting alternatives within GLoRE, including offline-calibrated output-similarity-based rerouting following SERE and token-wise next-rank rerouting with different treatments of the final routing weights. All methods are evaluated under the same mixed-domain workload described above. We have added these results to Appendix D.3 (Table 20). As shown in the result table, rerouting incurs a 3.65–5.19-point accuracy drop relative to vanilla (60.69), while increasing length-cap hits from 10.9% to 12.8–15.0%. SERE-style similarity rerouting performs worst in both accuracy (55.50) and length-cap hit rate (15.0%). In contrast, dropping maintains a comparable accuracy to vanilla (58.46 vs. 60.69), with fewer length-cap hits. The same trend holds for SERE: rerouting yields lower accuracy (41.57 vs. 52.94) and a higher length-cap hit rate (26.4% vs. 13.1%) than dropping.

Table. End-to-end comparison of dropping and rerouting strategies.

MethodVariantMacro accmath500mmlu_prohumaneval+lcbHit cap
VanillaVanilla60.69 ±0.3376.272.163.630.910.9%
GLoREDrop58.46 ±0.3777.370.160.426.19.7%
Next-rank reroute, orig weight57.04 ±0.5176.868.754.927.812.8%
Next-rank reroute, renorm56.17 ±0.7974.670.254.924.914.2%
SERE similarity reroute55.50 ±0.8375.665.956.524.015.0%
SERESimilarity reroute41.57 ±0.8059.657.134.415.226.4%
Drop52.94 ±0.6569.565.655.521.113.1%

Impact on generation behavior. Poorly aligned substitutes can make the model prone to repetitive generation, sometimes continuing until the maximum output length even after producing a valid answer. This is reflected in the increased cap-hit rate under rerouting, a trend observed with both GLoRE and SERE. These results suggest that unsuitable substitutes can disrupt generation behavior beyond simply failing to recover masked contributions.

Scope of our claim. We do not claim that dropping is universally better than rerouting; rather, our findings highlight the risks of using poorly aligned substitutes. As discussed in Appendix G, combining dropping and rerouting through layer-aware decisions can be a promising future direction.

Reviewer_ovz5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce an expert pruning algorithm and a fused kernel implementation for improving the throughput of MoE. Experiments are designed to show it's effectiveness on various popular open-source models. However, it should be noted that since a new kernel is introduced, it is unclear to this reviewer whether the throughput gains from this approach come from the proposed pruned approach or the fused kernel. Further ablations on this need to be done before the paper should be accepted.

Critical strengths
  1. Throughput gains showed in realistic multi-concurrency experiments with different open-source models with minimal quality degradation.
Critical weaknesses
  1. It is unclear whether the throughput gains from this approach come from the proposed pruned approach or the fused kernel. Further ablations on this need to be done before the paper should be accepted.
Questions for authors

I have questions that are related to the performance of the proposed approach vs baselines:

  1. Matched-kernel comparison: Are GLoRE and SERE implemented using comparable MoE dispatch and reduction kernels? Can the authors compare both routing policies using the same optimized kernels, ideally including OEA on the same backend?

  2. Kernel ablation: Can the authors provide an ablation comparing vanilla vLLM, vanilla vLLM with GLoRE's optimized reduction kernel where applicable, and full GLoRE, to isolate the performance gains from the routing algorithm versus kernel optimizations?

  3. Latency breakdown: Can the authors provide a detailed latency breakdown covering routing overhead, expert execution time, output-reduction time, and total MoE-layer latency, alongside the number of distinct experts executed?

4 comments
AuthorsOpenReview ↗

Response to Q1: Matched-Kernel Implementation and Baseline Comparison

We sincerely thank the reviewer for the careful review and insightful questions, which led us to add matched-kernel baseline comparisons, a kernel-overhead ablation, and an MoE-layer latency breakdown that make our efficiency claims more transparent and reproducible.


We clarify the kernel implementations used by GLoRE and SERE and extend our evaluation with additional baselines on a consistent kernel backend.

Both GLoRE and SERE use vLLM's stock dispatch (moe_align_block_size) and expert GEMM (fused_moe). GLoRE additionally enables vLLM's built-in expert_map / ignore_invalid_experts path (normally used for expert parallelism) with an identity map to exclude dropped slots from dispatch. It also replaces moe_sum with masked_moe_sum, which sums only live slots. This masked reduction introduces overhead but does not accelerate expert execution. We have added this clarification to Appendix A.2 (Kernel backend).

We have further added three baselines, XShare, LynX, and OEA, all implemented on the same kernel backend under a consistent execution environment (Section 4 and Appendix A.2). The new Reproducibility Statement summarizes the implementation details for all methods.

Overall comparison results. The results demonstrate that GLoRE provides a more favorable and stable accuracy–throughput trade-off across different serving batch sizes:

  • Accuracy–throughput trade-off. Across four models and four batch sizes (16 settings), GLoRE achieves Pareto-optimal accuracy–throughput trade-offs in 13, more often than any other method (SERE and OEA: 10/16; XShare: 9/16; LynX: 7/16). With one shared configuration (β=0.20\beta=0.20, μ=1.5\mu=1.5) and no per-model tuning, it retains 96.8–99.3% of vanilla accuracy on three models, while its throughput gain grows from 1.00–1.16× at B=8 to 1.21–1.39× at B=64. Each baseline loses in a different way. SERE is fast but retains only 58.7–77.8% of vanilla accuracy at B=8, because its small substitution pool at small batches approximates removed experts poorly. XShare reaches 1.30× on Qwen3.6-35B-A3B at B=64, but its accuracy drops from 99.7% to 95.3–96.5%, as its fixed expert count enables speedup by limiting the active set but fails to cover the increasingly diverse routing patterns and growing expert union at larger batches. LynX and OEA preserve accuracy but achieve at most 1.09× speedup on Qwen3-30B-A3B. On Gemma-4, methods that preserve accuracy provide no speedup, while SERE and GLoRE achieve speedup with accuracy loss.
  • Stability across batch sizes. Continuous batching causes the effective batch size to vary from step to step, making both accuracy and throughput stability important. GLoRE's accuracy varies by 0.9–2.5 points across B=8–64 (1.4 on average), comparable to OEA (1.6) and XShare (2.1), and well below LynX (5.6) and SERE (28.0). GLoRE is the only method whose throughput gain increases monotonically with batch size across all four models, enabling greater throughput gains at larger batches when serving systems may prioritize latency under high load. SERE, for example, peaks at B=16 or 32 and declines at B=64 on every model. This behavior is consistent with GLoRE's fractional mass budget, which adapts the number of masked experts to the current batch's routing distribution rather than imposing a fixed count.

The table below presents representative results on Qwen3-30B-A3B (Table 2 in the revised manuscript); full results across four models are in Appendix B (Table 8, Tables 9–12).

Table. Accuracy–throughput trade-offs across batch sizes on Qwen3-30B-A3B.

BSEREXShareLynXOEAGLoRE (ours)
858.74 / 1.190⋆98.54 / 1.014100.37 / 1.024⋆99.05 / 1.063⋆98.69 / 1.162⋆
1682.28 / 1.26198.42 / 1.10299.52 / 1.033⋆98.68 / 1.055⋆98.46 / 1.279⋆
3293.57 / 1.29098.27 / 1.211⋆99.41 / 1.08599.48 / 1.089⋆97.72 / 1.341⋆
6497.20 / 1.163100.55 / 1.189⋆100.16 / 1.04299.58 / 1.08598.02 / 1.389⋆

Accuracy retention (%) / throughput gain (×) relative to vanilla. ⋆ indicates Pareto-optimal results within each batch size; bold indicates the highest throughput.

AuthorsOpenReview ↗

Response to Q2: Disentangling Kernel Impact from Algorithmic Benefits

We thank the reviewer for this helpful suggestion to disentangle the performance impact of GLoRE's kernel optimizations from the benefits of expert skipping. We have added this ablation to Appendix F.4 (Table 27).

What GLoRE's kernels do. GLoRE neither replaces nor accelerates vLLM's compute kernels. The expert GEMMs use vLLM's stock fused_moe kernels in all configurations. GLoRE adds two components:

  1. Fused mask kernel: Builds the per-step skip mask and applies it to router outputs in a single launch per layer. This is work vanilla vLLM does not perform.
  2. Masked reduction (masked_moe_sum): Replaces vLLM's moe_sum. Skipped slots are redirected to a sentinel expert, which this kernel excludes from each token's sum. This preserves correctness but does not accelerate the reduction.

Ablation setup. Following the reviewer's suggestion, we compare three configurations on Qwen3-30B-A3B (TP=2, H100, vLLM 0.18.1, CUDA graphs on). The (a)–(b) comparison isolates the overhead of GLoRE's kernels, while (b)–(c) isolates the routing algorithm: both use identical kernels and differ only in the mask.

ArmConfiguration
(a) vanillaStock vLLM
(b) GLoRE kernels, no skipping(a) + all GLoRE kernels, with an empty skipped mask
(c) full GLoRE(b) with the actual mask

Table. Throughput impact of GLoRE kernel overhead and expert skipping.

Regime (ISL/OSL, concurrency)(a) Vanilla(b) GLoRE kernels, no skipping(c) Full GLoREKernels overhead (a→b)Skipping gain (b→c)
128/256, c=81.0×0.915×1.106×−8.5%+20.9%
128/256, c=161.0×0.937×1.287×−6.3%+37.3%
128/256, c=321.0×0.936×1.327×−6.4%+41.8%
128/256, c=641.0×0.950×1.400×−5.0%+47.4%

Throughput values are normalized to vanilla (a). Kernel overhead is calculated as b/a−1b/a-1, and skipping gain as c/b−1c/b-1.

The results confirm that expert skipping is responsible for GLoRE's throughput improvements, while its kernels introduce small overhead.

  • GLoRE's kernels introduce overhead. Arm (b) executes every GLoRE kernel, including the masked reduction, but skips no experts. It is slower than vanilla across all regimes, reducing throughput by 5.0–8.5%.

  • Expert skipping more than offsets this overhead. Arms (b) and (c) use identical kernels and differ only in the routing mask. Enabling expert skipping increases throughput by 20.9–47.4% over arm (b), yielding 1.11–1.40× throughput over vanilla. The skipping gain increases with decoding concurrency, as larger batches offer greater opportunities to reduce expert-weight loading.

AuthorsOpenReview ↗

Response to Q3: MoE-layer Latency Breakdown

We thank the reviewer for this helpful suggestion. We provide a phase-wise latency breakdown to quantify both GLoRE's kernel overhead and the savings from expert skipping, which we have added to Appendix F.4 (Table 28).

We break each MoE layer into three phases:

  • Routing: Gate projection and top-k selection, including GLoRE's fused mask kernel.
  • Experts: Token grouping and expert GEMMs (gate/up, SiLU, and down projections). Experts receiving no tokens are not loaded.
  • Reduction: Summation of expert outputs (moe_sum in vLLM; masked_moe_sum in GLoRE, excluding skipped slots).

We report total MoE-layer latency and the number of distinct experts receiving tokens (out of 128) on Qwen3-30B-A3B across different batch sizes.

Table. MoE-layer latency breakdown on Qwen3-30B-A3B across batch sizes.

Tokens/stepArmRouting (µs)Experts (µs)Reduction (µs)Total (µs)Distinct experts
8Vanilla9.981.95.397.140.3
GLoRE18.055.26.980.023.1
16Vanilla10.5108.05.7124.156.9
GLoRE19.268.37.795.230.1
32Vanilla10.9135.25.8151.973.6
GLoRE20.580.59.5110.536.8
64Vanilla11.1162.26.2179.588.5
GLoRE21.693.310.1125.042.9

These results are consistent with the throughput improvements reported above: although GLoRE's kernels introduce overhead in routing and reduction, the savings in expert execution time outweigh this overhead, reducing overall MoE-layer latency and translating into end-to-end throughput gains.

AuthorsOpenReview ↗

Follow-up on Kernel Fairness Experiments

We thank the reviewer again for the constructive suggestions. Following them, we added the requested kernel ablation and MoE-layer latency breakdown (Appendix F.4, Tables 27–28), showing that the throughput gains come from expert skipping rather than kernel differences. We would appreciate it if the reviewer could let us know whether these address the concerns.

Reviewer_tTjvOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper focuses on the challenge of batched inference of MoE models. While MoEs are trained with load balancing loss which ensures experts are roughly equally used in large batch sizes, in low batch size regimes, tokens activate experts very sparsely. Therefore, the balance of load may vary significantly with certain experts carrying more importance compared to others. This paper proposes a method to dynamically prune experts in each decoding step by skipping execution of low load experts. In principle, the method involves first computing the sum of routing weights across tokens for each expert and then dropping the lowest-scoring experts under a cumulative threshold budget. Next, it safeguards experts with sufficiently strong top-1 scores. Contrary to many prior works, the authors conclude that fully dropping the experts and effectively reducing the number of expert set (to lower than K) is less harmful that substituting them with other experts in the pool. The method is tested on 4 MoE models with the highest speed up gained for Qwen3-30B with almost 1.4X speed up at roughly 2% accuracy drop. In comparison, Gemma-4 shows a modest speed up gain of 1.23X while incurring a substantially larger relative accuracy loss of 6.4% (5 point absolute drop).

Critical strengths
  • The paper is well-written and very easy to follow
  • Applying the method to four MoE models is very useful for comparing the effectiveness across different model families
  • The authors implemented a custom CUDA kernel for GLoRE in vLLM
Critical weaknesses
  1. The first concern I have about this paper is importance of contribution compared to prior art and novelty. While the method is incredibly simple (that can be a strength rather than weakness), consisting of two steps: 1) computing an aggregate score per expert and 2) safeguarding some experts, even these simple two steps are not new and have precedence.
  • As cited in the related work section, the method XShare [1] is the most direct relevant work. Importantly, XShare’s Algorithm 1 already describes a cumulative retained-mass threshold. Its requirement to retain at least a fraction τ\tau of total mass is identical to GLoRE’s requirement to drop at most a fraction β\beta. Both use the same exact ranking. Thus, a claim that GLoRE uniquely introduces fractional aggregate mass control would be too broad.

  • Similarly for step two, methods such as Lynx [2] have a conditional protection of confident top-1 assignments.

Therefore, both components of the method have already been proposed and the small variations such as what happens to excluded routes is primarily an ablation for existing methods rather than a new contribution.

  1. I think the choice of comparison to baselines is sub-optimal. While comparison to SERE is meaningful, comparison to REAP as a permanent pruning method seems to be less fair. REAP comes with the advantage of massive memory footprint reduction for the total parameter size that dynamic pruning methods do not possess and can potentially be more suitable for batch size one on-device inference. The more suitable baselines would be XShare, OEA, BASE [3], Lynx, ExFold, etc. which the paper does not compare to.

  2. The authors correctly acknowledge dependence on batch composition in Appendix G, however, the batch-size sweeps in section 4.3 do not isolate the effect of processing batched requests from different domains. It would be very interesting to see how the proposed method and competing approaches compare in this more challenging real-life setting. It can also demonstrate the influence of the top-1 safe-guarding approach to mitigate the risk of missing important experts for specific domains.

  3. The throughput gains for some of the MoE models are very modest with significant drops in accuracy. In particular, Gemma-4 does not seem to tolerate pruning well. Could the authors discuss what explains this larger degradation compared with the other evaluated models? E.g. is the architecture impacting the results, or is it sensitivity to the choice of percentile?

  4. Given the more modest results on Gemma, it would be important to provide a sensitivity analysis of β\beta and μ\mu on the Gemma model as well and report possible throughput improvement.

[1] Vankov, Daniil, et al. "XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inference." arXiv preprint arXiv:2602.07265 (2026).

[2] Gupta, Vima, et al. "Lynx: Enabling efficient moe inference through dynamic batch-aware expert selection." arXiv preprint arXiv:2411.08982 (2024).

[3] Abbasi, Ali, Justin Shi, and Soheil Kolouri. "BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding." arXiv preprint arXiv:2609.36222 (2026).

Questions for authors

Can the authors elaborate more on their intuition on why entirely dropping excluded routes is more benificial than rerouting them to the next high ranked experts in the pool. Their finding is in contrast to other works such as Lynx, Xshare, OEA, SERE, BASE [3].

7 comments
AuthorsOpenReview ↗

Response to W1: Differences from Prior Work

We sincerely thank the reviewer for the thorough review and constructive suggestions, which led to new baselines, a clearer comparison with prior work, and additional analyses that have substantially strengthened the paper.


We thank the reviewer for pointing us to XShare's Algorithm 1 and LynX, which are now discussed in the revision. While GLoRE shares high-level concepts with these methods, namely a cumulative router-mass threshold (XShare's Algorithm 1) and top-1 protection (LynX), our novelty lies in how the two are combined: GLoRE pairs a gate-mass budget for batch-level reduction with a separately controlled top-1 protection to explicitly address the tension between batch-level expert reduction and token-level quality across varying batch sizes. This combination has measurable impact: the mass budget keeps the reduction level stable across batch sizes (Table 21), local protection makes this budget usable without accuracy collapse (+13.3 points at the default budget; Table 3), and end-to-end GLoRE is Pareto-optimal in 13/16 model–batch settings vs. 9/16 for XShare and 7/16 for LynX (Table 8). Details follow.

 

Methodological differences from XShare and LynX

XShare. Although XShare's Algorithm 1 describes a similar cumulative router-mass threshold, the authors adopt fixed-count selection as their default and report that it performs better in practice, without further evaluating Algorithm 1. In contrast, we argue that a fixed count does not adapt to varying batch sizes, whereas a mass budget provides a workload-adaptive reduction budget (Section 2.3). GLoRE further adds local protection of a token's top-1 expert when its routing weight is sufficiently high, controlled separately by μ\mu. Our original Related Work discussed only XShare's default configuration; the revision now covers its Algorithm 1 (Section 5).

Supporting evidence.

  • Mass budget vs. fixed count (Table 21, Appendix D.4). On Qwen3-30B-A3B, the mass budget (β=0.20\beta=0.20) removes 19.2–19.6% of gate mass at every B from 8 to 128, whereas a fixed count calibrated at B=64 (K=45, a simplified XShare-style budget) removes only 0.5% at B=8 but 20.5% at B=128.
  • End-to-end consequence (Table 8). On Qwen3.6-35B-A3B, XShare's accuracy falls from 99.7% at B=8 to 95.3–96.5% at B=32–64, while GLoRE stays at 96.9–98.0%.
  • Impact of local protection (Table 3, Section 4.3). At β=0.20\beta=0.20, disabling protection lowers five-task macro accuracy from 70.90 to 57.56 (−13.3 points), and the gap widens to 40.4 points at β=0.30\beta=0.30.

LynX. LynX's top-1 protection differs from GLoRE in both criterion and control. LynX protects the top-1 expert when few competing experts lie within 1/α1/\alpha logits, whereas GLoRE does so when its routing weight exceeds μ/k\mu/k of the token's total routed mass. These capture different notions of confidence: relative logit separation versus top-expert mass concentration. Moreover, LynX's bin width 1/α1/\alpha jointly controls top-1 protection and batch-wide expert preferences, whereas GLoRE uses a dedicated μ\mu for token-level protection, separately from the global reduction budget β\beta.

Supporting evidence.

  • Mass-conditioned protection is targeted and cheaper (Table 22, Appendix D.4). Without protection, 10.1–11.7% of tokens lose their top-1 expert at B ≥ 64. Protection with μ=1.5\mu=1.5 reduces this to about 3% while loading fewer experts than protecting every token's top-1 (48.9 vs. 50.8 at B=64; 55.4 vs. 58.6 at B=128).
  • Coupled control leaves LynX's reduction level implicit (Table 8). With one setting, LynX achieves at most 1.09× throughput on the three 26–35B models, yet on GLM-4.5-Air at B=8 it removes much more (1.18×) and its accuracy drops to 83.6%, whereas GLoRE's accuracy varies by at most 2.5 points on every model.

We have added a strategy-level comparison (Table 1, Section 2) and a per-method design comparison (Table 5, Appendix A.2).

 

Dropping rather than rerouting masked assignments

Our choice to drop rather than reroute excluded assignments is a deliberate design decision, motivated by our reconstruction analysis (Appendix D.3). We also added end-to-end comparisons (Appendix D.3, Table 20; details in Q1): even an oracle substitute recovers at most 0.1% of the dropping error in interior layers, and dropping outperforms all rerouting variants within GLoRE (58.46 vs. 55.50–57.04 macro accuracy) and within SERE (52.94 vs. 41.57), with fewer length-cap hits. More broadly, each design choice in GLoRE is motivated by a rationale, supported by analytical insights or empirical evidence.

AuthorsOpenReview ↗

Response to W2: Expanded Baseline Comparison

We thank the reviewer for this suggestion to broaden the baseline comparisons. We agree that REAP is a less direct baseline for dynamic MoE serving. We originally included it as a representative static-pruning baseline and have moved it to the appendix (Appendix B.2).

Following the suggestion, we have added comparisons with OEA, XShare, and LynX to the main results (Section 4, Table 2, Figures 3–4). We reimplement these methods based on their papers and publicly available code, using matched MoE kernels and the same evaluation setup. Full implementation details are in Appendix A.2 and summarized in the new Reproducibility Statement.

BASE was released recently (arXiv:2609.36222) and is therefore treated as concurrent work. Due to time and computational resource constraints, we could not include ExFold in the empirical comparison. Nevertheless, both methods are included in our methodological comparison (Table 1 and Table 5) and Related Work (Section 5).

Overall comparison results. The results demonstrate that GLoRE provides a more favorable and stable accuracy–throughput trade-off across different serving batch sizes:

  • Accuracy–throughput trade-off. Across four models and four batch sizes (16 settings), GLoRE achieves Pareto-optimal accuracy–throughput trade-offs in 13, more often than any other method (SERE and OEA: 10/16; XShare: 9/16; LynX: 7/16). With one shared configuration (β=0.20\beta=0.20, μ=1.5\mu=1.5) and no per-model tuning, it retains 96.8–99.3% of vanilla accuracy on three models, while its throughput gain grows from 1.00–1.16× at B=8 to 1.21–1.39× at B=64. Each baseline loses in a different way. SERE achieves high throughput gains but retains only 58.7–77.8% of vanilla accuracy at B=8, because its small substitution pool at small batches approximates removed experts poorly. XShare reaches a 1.30× throughput gain on Qwen3.6-35B-A3B at B=64, but its accuracy drops from 99.7% to 95.3–96.5%, as its fixed expert count increases throughput by limiting the active set but fails to cover the increasingly diverse routing patterns and growing expert union at larger batches. LynX and OEA preserve accuracy but achieve at most a 1.09× throughput gain on Qwen3-30B-A3B. On Gemma-4, methods that preserve accuracy provide no throughput gain, while SERE and GLoRE achieve throughput gains with accuracy loss.
  • Stability across batch sizes. Continuous batching causes the effective batch size to vary from step to step, making both accuracy and throughput stability important. GLoRE's accuracy varies by 0.9–2.5 points across B=8–64 (1.4 on average), comparable to OEA (1.6) and XShare (2.1), and well below LynX (5.6) and SERE (28.0). GLoRE is the only method whose throughput gain increases monotonically with batch size across all four models, enabling greater throughput gains at larger batches, i.e., under high load, when serving systems typically prioritize throughput. SERE, for example, peaks at B=16 or 32 and declines at B=64 on every model. This behavior is consistent with GLoRE's fractional mass budget, which adapts the number of masked experts to the current batch's routing distribution rather than imposing a fixed count.

The table below presents representative results on Qwen3-30B-A3B (Table 2 in the revised manuscript); full results across four models are in Appendix B (Table 8, Tables 9–12).

Table. Accuracy–throughput trade-offs across batch sizes on Qwen3-30B-A3B.

BSEREXShareLynXOEAGLoRE (ours)
858.74 / 1.190⋆98.54 / 1.014100.37 / 1.024⋆99.05 / 1.063⋆98.69 / 1.162⋆
1682.28 / 1.26198.42 / 1.10299.52 / 1.033⋆98.68 / 1.055⋆98.46 / 1.279⋆
3293.57 / 1.29098.27 / 1.211⋆99.41 / 1.08599.48 / 1.089⋆97.72 / 1.341⋆
6497.20 / 1.163100.55 / 1.189⋆100.16 / 1.04299.58 / 1.08598.02 / 1.389⋆

Accuracy retention (%) / throughput gain (×) relative to vanilla. ⋆ indicates Pareto-optimal results within each batch size; bold indicates the highest throughput.

AuthorsOpenReview ↗

Response to W3: Evaluation under Mixed-Domain Workloads

We thank the reviewer for the suggestion to evaluate the methods under more realistic mixed-domain serving.

We construct mixed-domain batches by combining requests from three representative domains: math (Math500, 500 samples), general knowledge (MMLU-Pro, 400 samples), and coding (HumanEval+ and LiveCodeBench, 339 samples), for a total of 1,239 requests. We report accuracy separately for each domain and measure throughput using a serving benchmark at a matched concurrency of 16. We have added this evaluation to Appendix B.3 (Table 13).

Table. Mixed-domain vs. single-domain accuracy on Qwen3-30B-A3B (B=16B=16).

armmathΔmathmmluΔmmlucodingΔcodingmacrothr. gain
Vanilla75.6−0.371.7+0.146.60.064.61.00×
SERE58.9−7.558.4−4.425.6−8.547.61.26×
XShare74.7−1.069.8−0.740.4−3.861.71.10×
LynX75.1−1.270.8+0.545.4+0.263.81.03×
OEA74.8+1.470.7−1.046.3+0.263.91.07×
GLoRE76.9+0.170.2+0.843.0−0.363.41.28×
GLoRE disable top-1 protection72.9−62.5−34.0−56.5

Δ denotes the accuracy difference between mixed-domain and single-domain evaluation. Throughput gain is relative to vanilla.

Robustness to mixed-domain workloads. Compared with single-domain evaluation, GLoRE shows minimal accuracy changes under mixed-domain batching (+0.1 on math, +0.8 on MMLU, and −0.3 on coding), with no substantial degradation across the three domains. In contrast, SERE and XShare experience larger coding accuracy drops (−8.5 and −3.8 points, respectively).

Impact of top-1 protection. Consistent with the ablation in Appendix C, disabling top-1 protection degrades accuracy. This degradation is even more pronounced under mixed-domain than single-domain evaluation (single-domain results are reported in Appendix B.2 and the protection ablation in Appendix C), highlighting the importance of local protection under varying batch compositions.

Accuracy–throughput trade-off. GLoRE achieves the highest throughput gain (1.28×) among the compared methods, with a macro accuracy of 63.4, only 1.2 points below vanilla. While LynX and OEA better preserve coding accuracy in absolute terms, their throughput gains are substantially smaller (1.03× and 1.07×). Overall, GLoRE offers a favorable accuracy–efficiency trade-off while maintaining stable performance across mixed-domain workloads.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response: Key Responses, Summary of Revisions and Paper Overview

We sincerely thank all reviewers for their thoughtful comments and constructive suggestions. We are encouraged that the reviewers recognized the following strengths of GLoRE:

  • Simple and well-motivated (yd5J): combining batch-level expert importance with token-level route protection reasonably balances efficiency and generation quality.
  • Practical and easy to deploy (tTjv, yd5J): no additional training or offline calibration, with plug-and-play CUDA kernels in vLLM.
  • Empirically effective in realistic serving (tTjv, ovz5): throughput gains with minimal quality degradation across four MoE model families.
  • Clearly presented (tTjv): well-written and easy to follow.

Below we summarize our key responses and the revisions, followed by an overview of GLoRE. Detailed responses are in each reviewer's thread.


Key Responses at a Glance

  • Novelty over XShare and LynX (tTjv W1, yd5J Q1). GLoRE combines a fractional gate-mass budget for batch-level reduction with a separately controlled, mass-conditioned top-1 protection. The mass budget removes a stable 19–20% of gate mass at every batch size, whereas a fixed count calibrated at B=64 removes only 0.5% at B=8 (Table 21); protection adds 13.3 accuracy points at the default budget (Table 3) while loading fewer experts than protecting every token's top-1 (Table 22).
  • Expanded baselines on matched kernels (tTjv W2, ovz5 Q1, yd5J Q1). GLoRE is Pareto-optimal in 13/16 model–batch settings (SERE and OEA: 10; XShare: 9; LynX: 7), retaining 96.8–99.3% of vanilla accuracy on three models with throughput gains growing from 1.00–1.16× at B=8 to 1.21–1.39× at B=64 (Table 8). Its accuracy varies by 1.4 points on average across B=8–64 (LynX: 5.6; SERE: 28.0), and it is the only method whose throughput gain increases monotonically with batch size on all four models.
  • Gains come from skipping, not kernels (ovz5 Q1–Q3). With an empty mask, GLoRE's added kernels reduce throughput by 5.0–8.5% (Table 27); enabling skipping on the same kernels adds 20.9–47.4% throughput and cuts MoE-layer latency by 18–30% (Table 28).
  • Mixed-domain workloads (tTjv W3). GLoRE's per-domain accuracy changes by at most 0.8 points versus single-domain batching, with the highest throughput gain (1.28×) at 1.2 points below vanilla (Table 13). Disabling top-1 protection lowers macro accuracy from 63.4 to 56.5.
  • Gemma-4 (tTjv W4+5). Gemma-4 is inherently more sensitive to expert removal: masking a full MoE layer yields 6.3× higher KL than on Qwen3 (2.70 vs. 0.43; Table 23). β controls the trade-off: β=0.05 preserves vanilla accuracy (78.78 vs. 78.61) at 1.04×, while β=0.20 reaches 1.24× at 73.59 (Table 24).
  • Dropping vs. rerouting (tTjv Q1, yd5J Q2). Even an oracle substitute recovers at most 0.1% of the dropping error in interior layers (Appendix D.3). End-to-end, dropping outperforms all rerouting variants within GLoRE (58.46 vs. 55.50–57.04) and SERE (52.94 vs. 41.57), with fewer length-cap hits (Table 20).

Summary of Revisions

  • New baselines (tTjv, ovz5, yd5J). XShare, LynX, and OEA are added to the main comparison (Section 4, Table 2, Figures 3–4; full results in Appendix B, Tables 8–12). REAP is moved to Appendix B.2.
  • Comparison with prior methods (tTjv, yd5J). A strategy-level comparison (Table 1), a per-method design comparison including ExFold and BASE (Table 5, Appendix A.2), and a rewritten Related Work (Section 5).
  • Mixed-domain workloads (tTjv). Accuracy and throughput when batches mix math, knowledge, and coding requests, including the effect of top-1 protection (Appendix B.3, Table 13).
  • Dropping versus rerouting (tTjv, yd5J). End-to-end comparison within GLoRE and SERE (Appendix D.3, Table 20).
  • Gemma-4 sensitivity (tTjv). Sensitivity analysis and β–μ sweep (Appendix D.5, Tables 23–24).
  • Implementation fairness and reproducibility (ovz5). Kernel backend and baseline details (Appendix A.2), kernel-overhead ablation and MoE-layer latency breakdown (Appendix F.4, Tables 27–28), and a new Reproducibility Statement.
  • Clearer motivation. Section 2 is revised to clarify the rationale behind GLoRE's design choices.

Overview of GLoRE


1. Practical Motivation

GLoRE is motivated by two practical considerations in LLM serving.

First, continuous batching leads to fluctuating effective batch sizes due to heterogeneous request lengths and dynamic request arrivals and completions. Expert-skipping methods should therefore remain effective across varying batch sizes rather than being optimized for a single operating point.

Second, serving priorities vary with workload intensity. Under heavy workloads, serving systems may tolerate some accuracy degradation in exchange for higher throughput while meeting latency SLOs.

Existing expert-skipping methods do not directly address the efficiency–quality trade-off across varying batch sizes and workload priorities.

 

2. Key Contributions

  • Global–local expert skipping: GLoRE introduces a controllable global–local mechanism combining a fractional mass budget with dominant-route protection.
  • Efficient vLLM implementation: We implement GLoRE with plug-and-play custom CUDA kernels in vLLM, enabling efficient integration into existing serving systems. An anonymized code link is provided in the paper.
  • Comprehensive evaluation: We evaluate GLoRE across four models and nine tasks against vanilla serving and four runtime expert-reduction baselines, demonstrating stable efficiency–quality trade-offs across batch sizes (8–64), and further explore its application to reducing inference costs in long-CoT test-time scaling.

 

3. Design Comparison with Prior Runtime Expert Reduction Methods

GLoRE differs from prior methods in three key design aspects:

  • Batch-adaptive expert reduction: A global fractional gate-mass budget adapts the extent of expert reduction to varying batch sizes.

  • Global reduction and local protection: GLoRE combines batch-level expert reduction with token-level route protection to maintain a stable quality–efficiency trade-off across varying batch sizes.

  • Drop instead of reroute: A deliberate design choice motivated by reconstruction analysis; excluded routes are dropped instead of rerouted.

Table. Comparison of expert selection and route protection strategies.

Selection strategyAdapts reduction to batch sizesProtects local routesProtection type
Per-token fixed count (OEA, SERE)Partly✓Fixed top-k0k_0
Per-batch fixed count (XShare)×✓Explicit top-k0k_0 bootstrap
Per-batch fixed count (ExFold, BASE)××None
Confidence-binned selection (LynX)Partly✓Router-confidence-based, implicit, coupled
Fractional mass budget (GLoRE, ours)✓✓Router-gate-mass-based, explicit, separable

 

4. Key Quality–Efficiency Results

Accuracy–throughput trade-off. Across four models and four batch sizes, GLoRE lies on the accuracy–throughput Pareto frontier in 13/16 settings, more often than any other method (SERE and OEA 10/16, XShare 9/16, LynX 7/16). It achieves 96.8–99.3% accuracy retention on three models and 93.6–94.5% on Gemma-4-26B, while throughput gains increase from 1.00–1.16× at B=8 to 1.21–1.39× at B=64.

Stability across batch sizes. GLoRE's accuracy varies by 0.9–2.5 points across B=8–64 (1.4 on average), comparable to OEA (1.6) and XShare (2.1) and well below LynX (5.6) and SERE (28.0). Notably, GLoRE is the only method whose throughput gain increases monotonically with batch size across all four models.

Long-CoT test-time scaling. As an additional application, GLoRE on Qwen3-30B-A3B achieves 1.05–1.20× lower GPU-seconds per problem across tested GPU-time budgets on AIME and GPQA-Diamond, with no statistically significant change in maj@8 accuracy.

Table. Accuracy–throughput trade-offs across batch sizes on Qwen3-30B-A3B.

BSEREXShareLynXOEAGLoRE (ours)
858.74 / 1.190⋆98.54 / 1.014100.37 / 1.024⋆99.05 / 1.063⋆98.69 / 1.162⋆
1682.28 / 1.26198.42 / 1.10299.52 / 1.033⋆98.68 / 1.055⋆98.46 / 1.279⋆
3293.57 / 1.29098.27 / 1.211⋆99.41 / 1.08599.48 / 1.089⋆97.72 / 1.341⋆
6497.20 / 1.163100.55 / 1.189⋆100.16 / 1.04299.58 / 1.08598.02 / 1.389⋆

Accuracy retention (%) / throughput gain (×) relative to vanilla. ⋆ indicates Pareto-optimal results within each batch size; bold indicates the highest throughput.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.