acceptodds
Under review as a conference paper at ICLR 2027

BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding

Abstract

Large language models continue to grow in parameter count and capacity, making them increasingly expensive to serve. In large-scale serving systems, autoregressive decoding is often bottlenecked by transferring model weights from accelerator high-bandwidth memory into on-chip SRAM. This memory bottleneck becomes more severe as models and context lengths grow, placing increasing pressure on memory bandwidth. Mixture-of-experts (MoE) models reduce computation by activating only a small subset of experts per token, but this sparsity does not translate directly to batched decoding. Different requests select different experts; therefore, the combined active set across many concurrent requests can span a substantial fraction of the expert pool and require significantly more expert weights to be transferred. Most expert-reduction techniques make retention decisions independently for each token and therefore do not address this batch-level expansion. More recently, batch-aware methods have attempted to coordinate expert use across concurrent requests and reuse experts already fetched for the batch. Yet their selection criteria are based primarily on router rankings or expert statistics collected during calibration. Consequently, these criteria are not directly tied to the output error caused by dropping an expert, nor do they capture how an expert's contribution changes across tokens at inference time. We instead rank experts according to how much their removal would change the MoE-layer output. To apply this criterion during serving, we train a lightweight linear predictor during calibration that estimates the expert removal cost for each incoming token. We also develop custom GPU kernels to efficiently perform cost prediction, batch-level score aggregation, and expert selection. Across three MoE architectures, BASE improves the quality-efficiency tradeoff without retraining. On Qwen3-30B-A3B, it improves average accuracy by 29.5 points over the strongest baseline at comparable throughput under a tight expert budget. At a higher expert budget, it is 60% faster than dense inference while remaining within 0.4 accuracy points.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, large language model inference, batched decoding, expert selection, efficient model serving, memory-efficient inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xTCgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

BASE selects a fixed batch-level expert set using predicted expert-removal error. It trains one task-agnostic linear regressor per expert to estimate token-specific output energy and reports a better quality–throughput tradeoff than existing batch-aware methods.

Critical strengths

The paper achieves a good trade-off between throughput and performance when loading a fixed number of experts into the GPU adaptively given the current batch.

Critical weaknesses
  1. The cross-term error is not fully explored. The current evaluation only compares the additive approximation with a cross-term-aware greedy search, which is itself an approximation rather than the optimal selection. It remains unclear whether explicitly modeling cross terms would lead to better expert selection.

  2. The batch-size experiment increases the expert budget as the batch size grows. Therefore, it does not show how BASE performs under a fixed loading budget. With more tokens sharing the same limited set of experts, the selected experts may cover a smaller fraction of the token-level routing preferences. The authors should report results across different batch sizes while keeping the expert budget fixed.

  3. The evaluation does not consider batches containing different tasks. For a single-task batch, tokens may activate a relatively small and consistent subset of experts. In a mixed-task batch, different requests may require different experts, making selection under a shared budget more difficult. The method should be evaluated under three settings for mixed tasks: a fixed budget and fixed batch size, a fixed budget with a growing batch size, and a growing budget with a growing batch size.

  4. The robustness of the regressor is not tested in the mixed-task setting. The regressors are trained using a general calibration dataset, but their predictions may behave differently across tasks and domains. The authors should evaluate whether the regressors remain accurate and whether performance is balanced across tasks in a mixed-task setting.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Reviewer xTCg Response, Part 1

We thank the reviewer for their careful evaluation. Their concerns regarding cross-term approximations, fixed expert budgets, and mixed-task batching have motivated additional experiments that provide a more comprehensive assessment of BASE's robustness and limitations.

W1: Cross-Term Error and Optimal Expert Selection

We appreciate the reviewer's observation that the cross-term-aware greedy search in Figure 1(c) is itself an approximation rather than the exact optimum. Importantly, this analysis is conducted in an oracle setting, where the true outputs of all routed experts are available. Such information is not accessible during practical inference without executing the experts we aim to skip. Therefore, these experiments are intended to evaluate the accuracy of our additive approximation and the potential benefit of explicitly modeling cross terms, rather than propose an additional inference-time selection method. To address the reviewer's concern, we conducted two additional experiments: (1) comparing additive selection against the globally optimal expert subset under the exact removal-error objective, and (2) evaluating whether explicitly modeling cross terms improves downstream accuracy.

We implemented an exact branch-and-bound algorithm that identifies the subset of MM experts minimizing the true batch-level removal error, including all cross terms:

\qquad D(A)=\sum_t\left\|\sum_{u\in S_t\setminus A}w_{t,u}E_u(z_t)\right\|_2^2,$$ where $U$ is the union of routed experts across the batch, $S_t$ is the original routed expert set for token $t$, and $D(A)$ measures the squared error caused by discarding experts outside the retained set $A$. Unlike our additive approximation, this objective explicitly accounts for interactions between omitted expert contributions, including their potential cancellation or reinforcement. Branch-and-bound systematically explores candidate expert subsets while using mathematically valid lower bounds on $D(A)$ to eliminate branches that cannot improve upon the best solution found so far. Once all remaining branches are explored or safely eliminated, the algorithm certifies that the selected subset achieves the global minimum of $D(A)$, rather than merely a locally optimal solution. Using the same experimental setting as Figure 1(c), we successfully certified global optimality for **99.7–100%** of evaluated decoding positions across Qwen3-30B-A3B, Qwen1.5-MoE, and DeepSeek-V2-Lite. Table: Exact removal-error ratio of additive selection to the globally optimal subset: | Model | Budget $M$ | Median ratio | 95th percentile | Greedy exactly optimal | |---|---:|---:|---:|---:| | Qwen3-30B-A3B | 12 | 1.0013 | 1.0314 | 99.1% | | Qwen1.5-MoE | 12 | 1.0000 | 1.0172 | 99.7% | | Qwen1.5-MoE | 20 | 1.0000 | 1.0112 | 99.8% | | DeepSeek-V2-Lite | 12 | 1.0000 | 1.0256 | 99.8% | *The ratio compares the exact removal error of additive selection with the global optimum; a ratio of 1 indicates optimality.* The additive approximation achieves **near-optimal removal error across all three architectures**. At the median, its error is within 0.13% of the optimum on Qwen3 and exactly optimal on the other two architectures. Even at the 95th percentile, the excess removal error remains below 3.2%. Furthermore, the cross-term-aware greedy search used in Figure 1(c) reaches the certified optimum at more than 99% of positions, confirming that our original comparison provides a close approximation to the true optimum. Impact of explicitly modeling cross terms on downstream accuracy: We additionally evaluated a cross-term-aware oracle that uses the true expert outputs to select experts by minimizing the exact removal error. We compare it against the additive oracle using the same expert budget, evaluation examples, and backfill procedure as Table 2. On 1,920 sampled Qwen3 decoding positions, we verified that the cross-term-aware greedy search reaches the certified global optimum in 99.6% of cases. Additive versus cross-term-aware oracle selection: | Selection method | Qwen3 | Qwen1.5-MoE | |---|---:|---:| | Additive oracle | 73.58 | **45.82** | | Cross-term-aware oracle | **74.46** | 45.34 | On both architectures, explicitly modeling cross terms produces no significant improvement in downstream accuracy compared with additive oracle selection. This holds even when the true expert outputs are available, whereas a practical cross-term-aware selector would require evaluating the expert outputs it aims to skip. Together, these results show that the additive removal-error formulation closely approximates the globally optimal selection objective, while explicitly accounting for cross terms provides no significant downstream accuracy improvement. We will include the exact optimum analysis and cross-term-aware oracle experiments in the revised manuscript.
AuthorsOpenReview ↗

Reviewer xTCg Response, Part 2

W2: Fixed Expert Budget Across Different Batch Sizes

We agree with the reviewer that increasing the expert budget alongside batch size does not directly characterize performance under a fixed loading constraint. To address this concern, we conducted an additional batch-size sweep on Qwen3-30B-A3B-Instruct-2507, holding the expert budget fixed at M∈{10,16,32}M\in\{10,16,32\} while varying batch size over B∈{8,16,32,64,128}B\in\{8,16,32,64,128\}. We use the same vLLM inference engine, sampling configuration, and full HumanEval, MBPP, MATH-401, and BoolQ benchmarks as in our original evaluation. We additionally evaluate ExFold under the same batch-level expert budget (D=MD=M), providing a direct comparison under identical loading constraints.

Accuracy under fixed expert budgets across batch sizes (average across four benchmarks):

MethodBudget M=DM=DB=8B=8B=16B=16B=32B=32B=64B=64B=128B=128
Dense (reference)Unrestricted84.3984.3984.3984.3984.39
ExFold1076.5059.4746.0036.5331.76
BASE1082.2377.4668.5858.9451.09
ExFold1684.3477.6571.3465.7964.49
BASE1684.3683.1381.9477.9976.62
ExFold3284.0784.7384.0681.7079.83
BASE3283.6784.6484.5983.7984.00

Effect of increasing batch size under a fixed budget: The results confirm the reviewer's observation that increasing batch size makes expert selection more challenging when the loading budget remains fixed. At M=10M=10, BASE's average accuracy decreases from 82.23% at B=8B=8 to 51.09% at B=128B=128. At M=16M=16, the decrease is substantially smaller, from 84.36% to 76.62%. With a larger budget of M=32M=32, BASE maintains near-dense accuracy across all evaluated batch sizes, ranging from 83.67% to 84.64%, demonstrating that a sufficiently large fixed budget can accommodate increasing batch sizes without substantial accuracy degradation.

These results demonstrate that although increasing batch size under a fixed expert budget can reduce accuracy, BASE maintains a favorable accuracy-throughput tradeoff relative to ExFold. We will include these fixed-budget experiments in the revised manuscript.

W3-W4: Mixed-Task Batching, Expert Budget Scaling, and Regressor Robustness:

We agree with the reviewer that mixed-task batches introduce additional challenges for batch-level expert selection, as requests from different tasks may activate different experts while sharing the same loading budget. To investigate this, we conducted additional experiments on Qwen3-30B-A3B using vLLM's continuous batching. We construct a mixed-task request stream by interleaving examples from HumanEval, MBPP, MATH-401, and BoolQ. As requests finish, new requests enter the batch, continuously changing its task composition. We compare mixed-task execution against running each benchmark separately under the same scheduler, and evaluate ExFold using the same expert budget (D=MD=M). We consider all three configurations requested by the reviewer.

Mixed-task continuous-batching accuracy (average across four benchmarks, %):

Batch-size limit BBBudget M=DM=DBASE (single-task)BASE (mixed-task)ExFold (mixed-task)
161078.6974.2726.40
161682.9082.5767.05
641058.5143.2812.26
641679.2870.1621.37
643284.4582.2768.14

(1) Fixed budget and batch-size limit: At B=16B=16 and M=16M=16, BASE achieves 82.57% accuracy under mixed-task execution, compared with 82.90% under single-task execution. At the more restrictive budget M=10M=10, mixed-task accuracy decreases to 74.27%, compared with 78.69% for single-task execution.

(2) Fixed budget with growing batch size: Holding M=16M=16 fixed while increasing the batch-size limit from 16 to 64 reduces mixed-task accuracy from 82.57% to 70.16%. At the tighter budget M=10M=10, accuracy decreases from 74.27% to 43.28%. These results confirm the reviewer's point that a fixed expert budget becomes increasingly restrictive as more heterogeneous requests share the selected experts.

AuthorsOpenReview ↗

Reviewer xTCg Response, Part 3

W3-W4 continue:

(3) Growing budget with growing batch size: Increasing MM from 16 at B=16B=16 to 32 at B=64B=64 maintains mixed-task accuracy at 82.57% and 82.27%, respectively. Results show that increasing the expert budget with concurrency can substantially mitigate the accuracy degradation introduced by sharing experts across different tasks.

Regressor robustness under mixed-task execution: Despite calibration exclusively on general web text, BASE retains 94.2-97.1% of the oracle's expert energy in mixed-task batches, compared with 94.2-98.0% in single-task batches. The difference remains within 1.3%, indicating that task mixing does not substantially degrade expert-selection quality.

Per-task accuracy under mixed-task execution: To determine whether mixing disproportionately affects particular tasks, we measured per-benchmark accuracy under single-task and mixed-task execution for both BASE and ExFold. We also evaluated the unpruned dense model under the identical mixed-task request stream.

Per-task accuracy under single-task and mixed-task continuous batching (%):

Method(B,M/D)(B,M/D)ExecutionHumanEvalMBPPMATH-401BoolQMean
Dense(16, unrestricted)Single-task88.4178.6079.5589.8284.10
Dense(16, unrestricted)Mixed-task87.8078.8080.5589.8584.25
ExFold(16, 16)Single-task74.3965.2083.5489.6378.19
ExFold(16, 16)Mixed-task65.8530.4082.2989.6367.05
BASE(16, 16)Single-task84.7674.4082.0490.4082.90
BASE(16, 16)Mixed-task87.2070.2082.7990.0982.57
Dense(64, unrestricted)Single-task89.0278.4080.3089.9484.42
Dense(64, unrestricted)Mixed-task88.4178.6080.0589.9484.25
ExFold(64, 32)Single-task82.9372.8082.7989.4581.99
ExFold(64, 32)Mixed-task70.1233.2079.3089.9468.14
BASE(64, 32)Single-task87.8077.8081.5590.6484.45
BASE(64, 32)Mixed-task86.5971.2081.0590.2482.27

The dense model maintains comparable accuracy across single-task and mixed-task execution. Similarly, at (B,M)=(16,16)(B,M)=(16,16) and (64,32)(64,32), BASE shows relatively small changes on HumanEval, MATH-401, and BoolQ. MBPP is more sensitive to mixing, with accuracy reductions of 4.2 and 6.6 percent respectively.

At tighter budgets, mixing produces larger accuracy reductions despite the regressor maintaining near-oracle expert-selection quality. This suggests that the main challenge is competition between heterogeneous requests under a shared expert budget rather than a substantial deterioration in selection quality.

These experiments show that BASE can accommodate heterogeneous requests under continuous batching while maintaining consistent expert-selection quality across tasks. Although tight expert budgets increase competition between requests, increasing the budget with concurrency substantially preserves accuracy. We will include these mixed-task, regressor robustness, and per-task accuracy analyses in the revised manuscript.

Reviewer_83HfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

BASE selects a batch-level active expert set using predicted removal error. A lightweight per-expert predictor estimates output energy before execution, and the estimated error is aggregated over the current decode batch. Backfill then ensures that each token can use enough experts from the loaded set. The method includes custom GPU kernels and is integrated into vLLM.

Critical strengths
  • The removal-error objective is more principled than using router scores or expert popularity alone.
  • The linear predictor performs close to the oracle based on real expert outputs.
  • The evaluation covers three different MoE models and several generation tasks, with strong quality-throughput results.
Critical weaknesses
  • Although LYNX is included, the paper does not state whether it uses the authors' official fused, CUDA-Graph-compatible implementation or only reimplements the selection policy in vLLM 0.9.2. This matters for the throughput comparison.
  • The current ablations show that predicting expert output energy is useful, but they do not isolate the proposed squared removal-error objective from a simpler REAP-style importance score using the same predicted output norm.
  • The serving setup uses controlled batch sizes and fixed-length generation. In continuous batching, batch size and composition change over time, while BASE relies on a batch-level expert budget M.
  • Each model requires a separately calibrated predictor. Its robustness to domain shift, quantization, or model fine-tuning is not evaluated, and the main predictor/backfill ablations are concentrated on Qwen3.
Questions for authors
  1. Does the LYNX baseline use its official fused and CUDA-Graph-compatible kernels? Please provide a full quality-throughput frontier for LYNX and BASE under the same engine, FusedMoE backend, graph mode, and workload.
  2. Using the same predictor, budget, and backfill rule, could the authors compare a dynamic REAP-style score sum_t w_{t,u} * predicted(||E_u(z_t)||) against BASE's removal-error score sum_t w_{t,u}^2 * predicted(||E_u(z_t)||^2)? This would clarify whether the gain comes from the squared removal-error formulation or simply from adding token-dependent output magnitude.
  3. How is M chosen under mixed-length continuous batching? Results with changing batch composition and tail-latency measurements would strengthen the serving evaluation.
  4. How well does the predictor transfer to a different prompt domain or a quantized/fine-tuned checkpoint? Repeating the predictor and backfill ablations on another MoE architecture would also be useful.
3 comments
AuthorsOpenReview ↗

Reviewer 83Hf Response, Part 1

We thank the reviewer for their constructive feedback. The suggestions regarding continuous-batching performance, and predictor generalization have motivated several additional experiments that strengthen our evaluation.

W1, Q1: LYNX Implementation and Quality-Throughput Comparison

We use the authors' official LYNX selection implementation and Triton kernels, integrated into the same vLLM 0.9.2 engine with CUDA-graph support as BASE.

To address this concern, we additionally evaluate the quality-throughput frontier of LYNX and BASE on Qwen3-30B-A3B using the same H100 GPU, vLLM engine, FusedMoE backend, CUDA-graph configuration, batch size of 16, and complete sets of all eight benchmarks.

LYNX throughput (tok/s)LYNX accuracyBASE throughput (tok/s)BASE accuracy
954.882.66960.682.58
1007.081.721017.582.67
1047.981.721051.582.81
1200.351.731207.282.49
1304.946.871312.081.92
1415.223.881417.676.18
1447.219.551463.576.38

Accuracy is averaged across the eight benchmarks in Table 1. Throughput is measured end-to-end under identical serving conditions. Additional operating points and corresponding hyperparameters will be included in the revised manuscript.

The results show that both methods achieve accuracy close to dense inference at approximately 1,000 tokens/s. However, as throughput increases, LYNX experiences substantially greater accuracy degradation, while BASE maintains considerably higher accuracy.

These additional experiments demonstrate that BASE provides a better quality-throughput tradeoff in the higher-throughput operating region under the same serving conditions. We will include the additional comparisons in the revised manuscript.

W2, Q2: Squared Removal Error versus Dynamic REAP-Style Importance

We conducted an additional controlled ablation to isolate the contribution of the squared removal-error formulation from the use of token-dependent expert output magnitudes. Specifically, we compare the dynamic REAP-style score

Sulinear=∑twt,u∥Eu(zt)∥22^S_u^{\mathrm{linear}}=\sum_t w_{t,u}\sqrt{\widehat{\|E_u(z_t)\|_2^2}}

against BASE's squared removal-error score

SuBASE=∑twt,u2∥Eu(zt)∥22^.S_u^{\mathrm{BASE}}=\sum_t w_{t,u}^2\widehat{\|E_u(z_t)\|_2^2}.

Both methods use the same calibrated energy predictor, expert budget, backfill procedure, evaluation examples, and random seed. We evaluate them on Qwen3-30B-A3B using the seven benchmarks from Table 2.

Selection scoreHumanEvalMATH-401MBPPGSM8KBBHMATHBoolQMean
Dynamic REAP-style (linear)53.0584.5027.0067.5059.7256.0076.5060.61
BASE (squared)68.2993.0049.0082.5065.7470.5085.5073.50

BASE's squared formulation improves mean accuracy by 12.89 percentage points, outperforming the linear score on all seven benchmarks.

We will include this controlled comparison in the revised manuscript.

W3, Q3: Expert Budget Selection and Tail Latency Under Continuous Batching

To address this concern, we conducted additional experiments using vLLM's continuous batching, with requests of varying lengths and task compositions. We additionally measured per-request tail latency and compared BASE against the unpruned dense model and existing batch-aware methods.

Expert budget MM selection: Similar to ExFold, OEA, and SERE, BASE configures its operating parameters offline based on the desired accuracy-throughput tradeoff. Specifically, MM defines the maximum number of distinct experts loaded per layer at each decoding step. Under continuous batching, MM remains fixed for a given serving configuration. At each decoding step, BASE recomputes the selected expert set based on the currently active sequences and their predicted expert-removal costs. Consequently, the expert-loading budget remains bounded by MM even as requests enter and leave the batch. We evaluate both maintaining a fixed budget as concurrency increases and configuring a larger budget to preserve accuracy at higher concurrency.

Changing batch composition: We evaluated BASE on Qwen3-30B-A3B using vLLM's continuous batching with HumanEval, MBPP, MATH-401, and BoolQ requests interleaved in a single stream. As requests finish, the scheduler admits new requests, continuously changing batch size and composition. We compare mixed-task execution against running each benchmark separately under the same scheduler. We vary the maximum number of concurrent sequences (max_num_seqs), while the actual batch size varies dynamically during execution. We compare mixed-task execution against running each benchmark separately under the same scheduler.

AuthorsOpenReview ↗

Reviewer 83Hf Response, Part 2

W3, Q3 continue:

Accuracy under mixed-task continuous batching (average across four benchmarks):

Batch-size limit BBBudget M=DM=DBASE (single-task)BASE (mixed-task)ExFold (mixed-task)
161078.6974.2726.40
161682.9082.5767.05
641058.5143.2812.26
641679.2870.1621.37
643284.4582.2768.14

At B=16B=16 and M=16M=16, BASE achieves 82.57% mixed-task accuracy, compared with 82.90% under single-task execution. When the batch-size limit increases to 64 while keeping M=16M=16, accuracy decreases to 70.16%, illustrating the increased competition for a fixed expert budget. Increasing MM to 32 at B=64B=64 restores accuracy to 82.27%. The dense model maintains 84.25% mixed-task accuracy at both batch-size limits, indicating that the observed degradation primarily arises from restricting expert selection rather than task mixing alone.

Tail latency under continuous batching: We additionally measured per-request time per output token (TPOT) using the same mixed-task stream. Each configuration was evaluated twice in clean serving sessions. We report mean TPOT and its 95th percentile (p95), including results for the unpruned dense model.

Table: Per-request latency under mixed-task continuous batching (ms):

Batch-size limit BBMethodExpert BudgetMean TPOTp95 TPOT
16DenseUnrestricted20.6323.59
16BASE1012.7414.07
16ExFold1011.9312.56
16BASE1614.4016.55
16ExFold1615.0417.06
64DenseUnrestricted39.2146.19
64BASE3231.1035.68
64ExFold3232.7937.89

BASE reduces p95 TPOT relative to dense inference by approximately 40% at (B,M)=(16,10)(B,M)=(16,10), 30% at (16,16)(16,16), and 23% at (64,32)(64,32). At the larger expert budgets, BASE also achieves lower p95 TPOT than ExFold, with 16.55 vs. 17.06 ms at (16,16)(16,16) and 35.68 vs. 37.89 ms at (64,32)(64,32). At the tighter budget (16,10)(16,10), ExFold reports lower TPOT, but its generations are substantially longer (approximately 1,141–1,179 tokens per request, compared with 203 for BASE). These long generations reduce request turnover and associated prefill interruptions, affecting the latency comparison.

The experiments demonstrate that BASE supports continuous batching with changing request lengths and task compositions while maintaining a bounded expert-loading budget. Although larger batches can require a higher expert budget to preserve accuracy, BASE provides substantial tail-latency reductions relative to dense inference and maintains competitive latency against ExFold. We will include these continuous-batching and tail-latency results in the revised manuscript.

W4, Q4: Predictor Robustness and Cross-Architecture Ablations

We conducted additional experiments to evaluate predictor robustness under quantization, checkpoint changes, and prompt-domain shifts, and extended the expert-selection and backfill ablations to Qwen1.5-MoE.

Predictor transfer across checkpoints: We calibrated the predictor once on BF16 Qwen3-30B-A3B-Instruct-2507 and applied it unchanged to its released FP8 checkpoint and the original Qwen3-30B-A3B checkpoint. We also compared against predictors recalibrated separately on each target checkpoint.

Target checkpointBudgetDense accuracyTransferred predictorRecalibrated predictor
FP8 Qwen3-Instruct1083.8176.9376.30
FP8 Qwen3-Instruct1683.8183.7683.20
Original Qwen3-30B-A3B1084.9072.3772.89

Accuracy is averaged across HumanEval, MBPP, MATH-401, and BoolQ.

Recalibration provides no statistically significant improvement over transferring the original predictor. The original Qwen3-30B-A3B checkpoint exhibits greater degradation under restricted expert selection, but recalibration does not significantly reduce this loss.

AuthorsOpenReview ↗

Reviewer 83Hf Response, Part 3

W4, Q4 continue:

Robustness to prompt-domain shifts: Since our predictor is calibrated exclusively on English web text, the existing evaluations on code generation, mathematical reasoning, and question answering already test its robustness beyond the calibration domain. To further investigate robustness under more substantial domain shifts, we additionally evaluated multilingual mathematical reasoning on MGSM in German, Japanese, Bengali, and Swahili, without recalibration, using 250 examples per language.

Multilingual MGSM accuracy (%) without predictor recalibration:

RegimeMethodGermanJapaneseBengaliSwahiliMean
ReferenceDense86.0076.0032.0057.2062.80
RestrictedSERE22.805.200.000.007.00
RestrictedOEA30.4020.404.000.8013.90
RestrictedLYNX39.6033.608.408.4022.50
RestrictedExFold40.0043.201.600.0021.20
RestrictedBASE (M=10M=10)83.6064.4023.6023.6048.80
RelaxedSERE75.2068.0023.2014.8045.30
RelaxedOEA76.0067.6019.2041.2051.00
RelaxedLYNX85.2074.4036.4052.8062.20
RelaxedExFold83.6075.2028.8046.4058.50
RelaxedBASE (M=16M=16)85.6077.6034.4050.0061.90

At M=10M=10, BASE achieves 48.80% average accuracy compared with 22.50% for the strongest restricted baseline, outperforming all competing methods across all four languages. At M=16M=16, BASE achieves 61.90%, comparable to the strongest relaxed baseline (62.20%) and dense inference (62.80%).

Cross-architecture ablations: We extend the expert-selection and backfill ablations from Tables 2 and 3 to Qwen1.5-MoE. Additionally, following Reviewer mqfT's suggestion, we incorporate ExFold's selection criterion and folding mechanism into the comparisons for both architectures.

Expert-selection ablation (average accuracy, %):

Expert-selection ruleQwen3Qwen1.5-MoE
Top-1 union58.6839.48
Squared router weights55.5041.46
Static expert energy64.4642.74
ExFold selection score53.9138.73
Nonlinear predictor72.0744.90
Oracle expert energy73.5845.82
BASE linear predictor73.5045.42

Backfill ablation (average accuracy, %):

Routing ruleQwen3Qwen1.5-MoE
Drop74.2027.67
Substitution27.4913.91
ExFold folding74.1529.58
Backfill (BASE)76.4531.65

The results on Qwen1.5-MoE are consistent with our findings on Qwen3, demonstrating the transferability of the design choices.

We will include these additional robustness experiments and cross-architecture ablations in the revised manuscript.

Reviewer_mqfTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors address the problem of batched token generation with MoEs. During token generation, latency is dominated by data transfer to on-chip SRAM. During batched token generation, the number of experts required for the full batch can be much larger than the number of experts for each individual token in the batch. This work aims to reduce data transfer latency by choosing a set of active experts for each batch of tokens, that is smaller than the set of experts required by all tokens in the batch. The authors aim to achieve this by setting a total maximum of active experts for all tokens in a batch, omitting experts whose omission would yield the smallest increase in output error. To achieve this, the authors must predict the output energy of each expert in a layer for a given input embedding, weighted by the expert weight given by the router. Building on prior work that used a (fixed) average output energy [1], the authors devise a small linear predictor of each expert's output energy given an input embedding. In extensive experimental validation, the authors show that this linear predictor gives a better prediction of expert output energy, allowing a better selection of experts.

While the paper could benefit from further experimental analysis (see "weaknesses" and "questions"), the paper is strong enough that a weak accept rating is warranted.

Critical strengths

The paper is well written, with a clear exposition and derivation of the method.

The method introduced in the paper is clever and efficient, and shows clear improvement over the prior work on which it is based [1]

The paper offers a thorough experimental validation, covering a range of relevant MoE models and benchmarks to show the benefit of BASE over competing methods. Particularly the ablations are useful and provide good insights into the model vis-a-vis other approaches (but see related weak points)

[1] ExFold, Wu et al. 2026

Critical weaknesses
  1. The method is useful, and well validated, but from a scientific standpoint yields a rather incremental improvement over prior work. Of specific interest would be a better description of the design choices by the specific predictor used by the authors, and a more thorough investigation of other linear or non-linear predictors.

  2. This is partially offered by the authors in the ablation on different types of predictors in Table 2. In this table, the authors show that the BASE predictor gives near-oracle performance. However, variance of performance across tasks is rather large, as seen in Table 1. It would therefore be strengthen the author's arguments to show a per-task breakdown, as well as an analysis of the cause of the drop in avg accuracy for the nonlinear predictor.

  3. ExFold employs a weight update step in expert folding during token generation which the authors appear to omit in the ablation in Table 2.

Questions for authors
  1. Experts are selected based on the predicted output energies jointly over all tokens in a batch. However, this could disadvantage tokens for which either output energies or expert weights are distributed more uniformly than tokens for which weights and energies are distributed with larger outliers. Could the authors provide an analysis?
  2. The authors remove the contribution of omitted experts from the layer's output. However, [2] showed that replacing experts can boost accuracy if the original router weights are used.
  3. Do the authors have an explanation why their methods outperforms other methods so much more strongly on Qwen3-30B-A3B than on the other models evaluated?
  4. Likewise, do the authors have an explanation as to why their model underperforms on Math401 compared to other benchmarks?

[2] Mixture of cache-conditional experts for efficient mobile device inference, Skliar et al. 2024

4 comments
AuthorsOpenReview ↗

Reviewer mqfT Response, Part 1

We thank the reviewer for their thoughtful assessment of our work. The constructive suggestions have helped us incorporate additional analyses and experimental results that further strengthen the paper.

W1, W2: Predictor Design Choices, Per-Task Performance, and Additional Ablations

Following the reviewer's suggestions, we provide additional ablations of expert-selection criteria and nonlinear predictor capacity, along with a per-task breakdown of performance across benchmarks.

We provide a per-task breakdown of the selection-rule ablation in Table 2, extending it to include ExFold's selection score and nonlinear predictors with MLP hidden widths of 16, 64, and 256. These nonlinear variants extend the expert-specific ridge predictor with a shared MLP and lightweight expert-specific output heads. All three variants use the same calibration data.

Selection scoreHumanEvalMATH-401MBPPGSM8KBBHMATHBoolQMean
Top-1 union50.6173.5036.0065.0060.6537.5087.5058.68
Router weight sum17.0775.0021.0048.0043.5230.5065.0042.87
Square weight sum40.2477.0020.5063.5059.2652.5075.5055.50
Static expert energy42.0790.0040.5074.5060.6561.0082.5064.46
ExFold selection score32.3288.5027.0060.5055.5642.0071.5053.91
Nonlinear (16)65.8591.5042.5082.0067.1372.5083.0072.07
Nonlinear (64)68.9094.0035.0082.5068.5265.5086.0071.49
Nonlinear (256)63.4193.0040.0081.5071.3066.5085.0071.53
BASE linear predictor68.2993.0049.0082.5065.7470.5085.5073.50
Oracle energy74.3992.5043.0083.5066.6771.0084.0073.58

We evaluate nonlinear predictors with MLP hidden widths of 16, 64, and 256. The results show no statistically significant performance differences relative to BASE's linear predictor, as confirmed by the paired comparisons below.

We additionally conduct paired comparisons using the exact McNemar test on the same 1,380 evaluation examples per architecture. The following table reports the p-values for each selection criterion and nonlinear predictor relative to BASE.

ModelTop-1 unionRouter weight sumSquare weight sumStatic expert energyExFold scoreNonlinear (16)Nonlinear (64)Nonlinear (256)Oracle energy
Qwen3-30B-A3B1e-255e-913e-411.5e-128e-460.210.0760.111.00
Qwen1.5-MoE-A2.7B3.6e-61.5e-70.00120.0362.1e-70.670.500.350.86

At the nominal significance threshold of 0.05, the first five selection-rule baselines show statistically significant differences from BASE on both architectures. In contrast, none of the three nonlinear predictor variants shows a statistically significant difference from BASE on either architecture. We will include these additional analyses in the revised manuscript.

W3: ExFold Weight Updates

Following the reviewer's suggestion, we clarify how ExFold's expert-selection and weight-update mechanisms are evaluated and provide an additional controlled ablation isolating its folding mechanism.

Main comparison (Table 1): We use ExFold's official implementation, including both its expert-selection and calibrated weight-update mechanisms.

Selection-rule ablation (Table 2): We additionally evaluate ExFold's selection score under the same expert cap and BASE backfill rule as the other selection criteria. This isolates expert-selection quality from differences in how omitted expert contributions are handled.

Fill-rule ablation (Table 3): The manuscript compares dropping, substitution, and BASE backfill under the same selected expert set and expert cap. To directly address the reviewer's question, we conduct an additional controlled experiment that includes ExFold's calibrated folding mechanism alongside these fill rules.

Fill ruleQwen3-30B-A3BQwen1.5-MoE-A2.7B
Drop74.2027.67
Substitution27.4913.91
ExFold folding (weight update)74.1529.58
BASE backfill76.4531.65

Mean accuracy across HumanEval, GSM8K, MATH, and BBH (780 evaluation examples), using identical selected expert sets and expert caps across fill rules (M=10M=10 for Qwen3 and M=11M=11 for Qwen1.5).

The results show that BASE backfill achieves higher accuracy than dropping, substitution, and ExFold folding on both architectures.

We will clarify the distinction between ExFold's full implementation in Table 1, its isolated selection score in Table 2, and the additional folding ablation in the revised manuscript.

AuthorsOpenReview ↗

Reviewer mqfT Response, Part 2

Q1: Effect of Token-Level Routing Flatness

We thank the reviewer for raising this concern. To investigate whether batch-level expert selection disproportionately affects tokens with more uniformly distributed routing weights or expert output energies, we conduct an additional token-level analysis on Qwen3-30B-A3B with M=10M=10.

We group tokens into quartiles according to router-weight entropy, where higher entropy indicates more uniformly distributed routing weights. We compare the flattest and peakiest quartiles using two sets of measurements: reconstruction errors within the MoE layers and changes in the final output distributions. All results are evaluated under BASE's inference configuration, with dense prefill followed by expert selection during decoding.

We examine two metrics:

  • Relative MoE error (rr): The squared reconstruction error introduced by expert selection, normalized by the squared norm of the original MoE output, r=∥Δy∥22/∥y∥22r=\|\Delta y\|_2^2/\|y\|_2^2.
  • Residual perturbation (ee): The squared reconstruction error normalized by the squared norm of the residual-stream representation, e=∥Δy∥22/∥h∥22e=\|\Delta y\|_2^2/\|h\|_2^2. This accounts for the scale of the MoE contribution relative to the residual stream.

Here, yy denotes the original MoE output, Δy\Delta y the perturbation introduced by expert selection, and hh the residual-stream representation.

Token groupRelative MoE error (rr)Residual perturbation (ee)
Flattest quartile0.5930.0132
Peakiest quartile0.2230.0165

The results confirm that flat tokens experience larger relative MoE reconstruction errors. However, this disparity does not translate into larger residual-stream perturbations. This distinction is important because BASE minimizes an approximation of the absolute squared reconstruction error rather than enforcing equal relative error across tokens.

We next examine whether these reconstruction errors translate into changes in the model's predictions using three metrics:

  • KL divergence: The divergence between the next-token probability distributions produced by BASE and the dense model, measured in nats. Lower values indicate closer agreement with the dense model.
  • Top-1 prediction flip rate: The percentage of token positions where BASE's highest-probability next token differs from that of the dense model.
  • Cross-entropy increase (ΔCE\Delta\mathrm{CE}): The increase in next-token prediction loss relative to the dense model, evaluated on the same fixed sequence of dense-model-generated tokens. Lower values indicate smaller degradation in predictive loss.

We compute these metrics using teacher forcing on identical token sequences to isolate the effects of expert selection.

Token groupKL divergenceTop-1 flip rate (%)Cross-entropy increase (ΔCE\Delta\mathrm{CE})
Flattest quartile0.1806.310.199
Peakiest quartile0.31410.880.343

Despite their larger relative reconstruction errors, flat tokens exhibit lower KL divergence, fewer next-token prediction changes, and smaller cross-entropy increases. Further analysis suggests that flat tokens tend to have smaller MoE contributions relative to the residual stream and occur at positions where the dense model has higher prediction confidence. We also observe the same qualitative behavior when grouping tokens by expert-energy flatness and when using oracle expert energies for selection.

These results suggest that flat tokens can tolerate larger relative reconstruction errors without proportionally greater degradation in their predictions. This supports BASE's batch-level selection strategy, which prioritizes expert contributions with larger predicted removal costs under a limited expert budget, rather than enforcing equal relative reconstruction error across tokens. This interpretation is also consistent with the main results in Table 1 of the manuscript, where BASE achieves higher overall benchmark accuracy than competing batch-aware expert-selection methods.

We will include these additional analyses in the revised manuscript.

AuthorsOpenReview ↗

Reviewer mqfT Response, Part 3

Q2: Expert Replacement and Original Router Weights

We agree with the observation in [2] that replacing unavailable experts with available alternatives while retaining their original router scores can improve model performance. We clarify that BASE already incorporates this principle through its backfill procedure in Section 3.5. The removal-error formulation is used to select the batch-level active expert set, rather than to determine the final execution rule. After selection, each token backfills unavailable experts with its highest-ranked experts from the active set, using their original router scores and the normalization denominator of the original top-KK experts. Our original fill-rule ablation in Table 3 confirms the benefit of this design. While [2] focuses on cache-aware expert reranking for single-token inference, BASE determines which experts to fetch jointly across a batch using predicted removal costs. We will clarify the distinction between our selection objective and backfill filling rule procedure in Section 3.5 and add the corresponding reference to [2].

Q3: Explaining the Larger Accuracy Gains on Qwen3

We conducted additional experiments across five MoE architectures to investigate the larger accuracy improvements on Qwen3-30B-A3B. We compare the models at approximately matched levels of retained routed experts. Expert coverage measures how many of a token's preferred experts are included in the selected expert set.

Accuracy degradation and recovery at approximately 25% expert coverage:

ModelRouted expertsBudget MMUnion accuracy lossBASE gain over union
Qwen3-30B-A3B1281023.1615.26
DeepSeek-V2-Lite6497.394.26
DeepSeekMoE-16B6498.544.98
OLMoE-1B-7B6496.963.24
Qwen1.5-MoE-A2.7B6087.112.77

The results confirm the reviewer's observation that BASE achieves larger gains on Qwen3. At comparable expert coverage, Qwen3 experiences approximately 3 times greater accuracy degradation under union-based selection than the other architectures, creating substantially more opportunity for BASE to recover lost accuracy.

We hypothesize that Qwen3's larger routed-expert pool (128 versus 60-64) may encourage greater expert specialization, making expert contributions less interchangeable and their omission more costly. This could explain why BASE's removal-error-based selection provides larger accuracy improvements on Qwen3. We will include this five-model comparison in the revised manuscript.

Q4: Smaller Accuracy Improvements on MATH-401

We thank the reviewer for this question. We clarify that BASE achieves accuracy close to dense inference on MATH-401, remaining within 1.75 percentage points of the dense model across all six model-budget configurations. The smaller improvements over competing methods therefore reflect a reduced opportunity for further accuracy gains rather than substantial degradation under BASE.

Our additional analysis suggests that MATH-401 exhibits lower expert-routing diversity across tokens, resulting in greater overlap between each token's originally routed top-KK experts and the batch-level active set. At the same expert budget, approximately 59% of the original routed experts are retained on MATH-401, compared with approximately 33% on HumanEval for DeepSeek-V2-Lite and Qwen1.5-MoE. This is consistent with MATH-401's relatively homogeneous, arithmetic-focused prompts, which may lead different sequences to favor similar sets of experts. Consequently, a smaller shared expert set can satisfy more tokens' routing preferences, reducing the impact of restricted expert selection.

These observations suggest that MATH-401 is less sensitive to restricted expert selection, leaving less room for BASE to improve upon strong baselines. We will include the expert-routing diversity analysis in the revised manuscript.

I thank the authors for their thorough response. My concerns have mostly been addressed. However, my main concern regarding incremental improvement over prior work remains and I retain my original rating (3)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal

We would like to thank all three reviewers for their positive assessment of our work and their constructive feedback. We found the experiments and analyses suggested by the reviewers particularly valuable, as they provided new insights into the effectiveness, practical implications, and limitations of BASE. During the rebuttal period, we conducted extensive additional experiments to address the concerns raised by the reviewers. We believe these findings strengthen our work and will include them in the final version of the paper.

Below, we summarize the main concerns raised by the reviewers and our corresponding findings:

  • Continuous batching, mixed tasks, and fixed expert budgets: A common concern among the reviewers was the applicability of BASE to realistic serving scenarios with heterogeneous requests and varying batch sizes. Our additional experiments demonstrate that BASE maintains a superior tradeoff between accuracy and throughput under these conditions. We also characterize the limitations of fixed expert budgets as concurrency increases and demonstrate that BASE reduces tail latency by approximately 23-40% compared to dense inference.

  • Effectiveness of the removal error objective: We conducted a controlled comparison against a dynamic REAP formulation using the same energy predictor. BASE's squared removal error criterion improves average accuracy by 12.89 percentage points and outperforms the alternative across all seven benchmarks.

  • Justification of the additive approximation: We implemented an exact branch-and-bound algorithm to evaluate our additive approximation against the globally optimal expert selection. Across three MoE architectures, the median removal error remains within 0.13% of the global optimum. Furthermore, explicitly accounting for cross terms provides no statistically significant improvement in downstream accuracy, even in an oracle setting.

  • Predictor robustness and generalization: We demonstrated the robustness of our linear predictor across multilingual reasoning tasks, quantized checkpoints, different model checkpoints, and heterogeneous batches without recalibration. We also evaluated three additional nonlinear predictor configurations and found no statistically significant improvement over our linear predictor.

  • Additional baseline comparisons and ablation studies: We extended our comparison against LYNX using its official kernels under identical serving conditions. We also incorporated ExFold's selection criterion and folding mechanism into our controlled ablation studies, and extended our ablations to another MoE architecture.

  • Further insights into the behavior of BASE: We conducted additional analyses of token routing entropy, expert energy distributions, and routing diversity across benchmarks. We find these results particularly interesting, as they shed light on the underlying factors contributing to the gains achieved by BASE and help explain the variations in its effectiveness across benchmarks and MoE architectures.

We hope that the additional experiments address the main concerns raised by the reviewers and provide a more comprehensive understanding of the practical benefits and limitations of BASE. We once again thank all reviewers for their time and constructive suggestions, which have helped us improve the quality of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.