- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
BASE selects a fixed batch-level expert set using predicted expert-removal error. It trains one task-agnostic linear regressor per expert to estimate token-specific output energy and reports a better quality–throughput tradeoff than existing batch-aware methods.
Critical strengths
The paper achieves a good trade-off between throughput and performance when loading a fixed number of experts into the GPU adaptively given the current batch.
Critical weaknesses
-
The cross-term error is not fully explored. The current evaluation only compares the additive approximation with a cross-term-aware greedy search, which is itself an approximation rather than the optimal selection. It remains unclear whether explicitly modeling cross terms would lead to better expert selection.
-
The batch-size experiment increases the expert budget as the batch size grows. Therefore, it does not show how BASE performs under a fixed loading budget. With more tokens sharing the same limited set of experts, the selected experts may cover a smaller fraction of the token-level routing preferences. The authors should report results across different batch sizes while keeping the expert budget fixed.
-
The evaluation does not consider batches containing different tasks. For a single-task batch, tokens may activate a relatively small and consistent subset of experts. In a mixed-task batch, different requests may require different experts, making selection under a shared budget more difficult. The method should be evaluated under three settings for mixed tasks: a fixed budget and fixed batch size, a fixed budget with a growing batch size, and a growing budget with a growing batch size.
-
The robustness of the regressor is not tested in the mixed-task setting. The regressors are trained using a general calibration dataset, but their predictions may behave differently across tasks and domains. The authors should evaluate whether the regressors remain accurate and whether performance is balanced across tasks in a mixed-task setting.
Questions for authors
See weakness.
3 comments
Reviewer xTCg Response, Part 2
W2: Fixed Expert Budget Across Different Batch Sizes
We agree with the reviewer that increasing the expert budget alongside batch size does not directly characterize performance under a fixed loading constraint. To address this concern, we conducted an additional batch-size sweep on Qwen3-30B-A3B-Instruct-2507, holding the expert budget fixed at while varying batch size over . We use the same vLLM inference engine, sampling configuration, and full HumanEval, MBPP, MATH-401, and BoolQ benchmarks as in our original evaluation. We additionally evaluate ExFold under the same batch-level expert budget (), providing a direct comparison under identical loading constraints.
Accuracy under fixed expert budgets across batch sizes (average across four benchmarks):
| Method | Budget | |||||
|---|---|---|---|---|---|---|
| Dense (reference) | Unrestricted | 84.39 | 84.39 | 84.39 | 84.39 | 84.39 |
| ExFold | 10 | 76.50 | 59.47 | 46.00 | 36.53 | 31.76 |
| BASE | 10 | 82.23 | 77.46 | 68.58 | 58.94 | 51.09 |
| ExFold | 16 | 84.34 | 77.65 | 71.34 | 65.79 | 64.49 |
| BASE | 16 | 84.36 | 83.13 | 81.94 | 77.99 | 76.62 |
| ExFold | 32 | 84.07 | 84.73 | 84.06 | 81.70 | 79.83 |
| BASE | 32 | 83.67 | 84.64 | 84.59 | 83.79 | 84.00 |
Effect of increasing batch size under a fixed budget: The results confirm the reviewer's observation that increasing batch size makes expert selection more challenging when the loading budget remains fixed. At , BASE's average accuracy decreases from 82.23% at to 51.09% at . At , the decrease is substantially smaller, from 84.36% to 76.62%. With a larger budget of , BASE maintains near-dense accuracy across all evaluated batch sizes, ranging from 83.67% to 84.64%, demonstrating that a sufficiently large fixed budget can accommodate increasing batch sizes without substantial accuracy degradation.
These results demonstrate that although increasing batch size under a fixed expert budget can reduce accuracy, BASE maintains a favorable accuracy-throughput tradeoff relative to ExFold. We will include these fixed-budget experiments in the revised manuscript.
W3-W4: Mixed-Task Batching, Expert Budget Scaling, and Regressor Robustness:
We agree with the reviewer that mixed-task batches introduce additional challenges for batch-level expert selection, as requests from different tasks may activate different experts while sharing the same loading budget. To investigate this, we conducted additional experiments on Qwen3-30B-A3B using vLLM's continuous batching. We construct a mixed-task request stream by interleaving examples from HumanEval, MBPP, MATH-401, and BoolQ. As requests finish, new requests enter the batch, continuously changing its task composition. We compare mixed-task execution against running each benchmark separately under the same scheduler, and evaluate ExFold using the same expert budget (). We consider all three configurations requested by the reviewer.
Mixed-task continuous-batching accuracy (average across four benchmarks, %):
| Batch-size limit | Budget | BASE (single-task) | BASE (mixed-task) | ExFold (mixed-task) |
|---|---|---|---|---|
| 16 | 10 | 78.69 | 74.27 | 26.40 |
| 16 | 16 | 82.90 | 82.57 | 67.05 |
| 64 | 10 | 58.51 | 43.28 | 12.26 |
| 64 | 16 | 79.28 | 70.16 | 21.37 |
| 64 | 32 | 84.45 | 82.27 | 68.14 |
(1) Fixed budget and batch-size limit: At and , BASE achieves 82.57% accuracy under mixed-task execution, compared with 82.90% under single-task execution. At the more restrictive budget , mixed-task accuracy decreases to 74.27%, compared with 78.69% for single-task execution.
(2) Fixed budget with growing batch size: Holding fixed while increasing the batch-size limit from 16 to 64 reduces mixed-task accuracy from 82.57% to 70.16%. At the tighter budget , accuracy decreases from 74.27% to 43.28%. These results confirm the reviewer's point that a fixed expert budget becomes increasingly restrictive as more heterogeneous requests share the selected experts.
Reviewer xTCg Response, Part 3
W3-W4 continue:
(3) Growing budget with growing batch size: Increasing from 16 at to 32 at maintains mixed-task accuracy at 82.57% and 82.27%, respectively. Results show that increasing the expert budget with concurrency can substantially mitigate the accuracy degradation introduced by sharing experts across different tasks.
Regressor robustness under mixed-task execution: Despite calibration exclusively on general web text, BASE retains 94.2-97.1% of the oracle's expert energy in mixed-task batches, compared with 94.2-98.0% in single-task batches. The difference remains within 1.3%, indicating that task mixing does not substantially degrade expert-selection quality.
Per-task accuracy under mixed-task execution: To determine whether mixing disproportionately affects particular tasks, we measured per-benchmark accuracy under single-task and mixed-task execution for both BASE and ExFold. We also evaluated the unpruned dense model under the identical mixed-task request stream.
Per-task accuracy under single-task and mixed-task continuous batching (%):
| Method | Execution | HumanEval | MBPP | MATH-401 | BoolQ | Mean | |
|---|---|---|---|---|---|---|---|
| Dense | (16, unrestricted) | Single-task | 88.41 | 78.60 | 79.55 | 89.82 | 84.10 |
| Dense | (16, unrestricted) | Mixed-task | 87.80 | 78.80 | 80.55 | 89.85 | 84.25 |
| ExFold | (16, 16) | Single-task | 74.39 | 65.20 | 83.54 | 89.63 | 78.19 |
| ExFold | (16, 16) | Mixed-task | 65.85 | 30.40 | 82.29 | 89.63 | 67.05 |
| BASE | (16, 16) | Single-task | 84.76 | 74.40 | 82.04 | 90.40 | 82.90 |
| BASE | (16, 16) | Mixed-task | 87.20 | 70.20 | 82.79 | 90.09 | 82.57 |
| Dense | (64, unrestricted) | Single-task | 89.02 | 78.40 | 80.30 | 89.94 | 84.42 |
| Dense | (64, unrestricted) | Mixed-task | 88.41 | 78.60 | 80.05 | 89.94 | 84.25 |
| ExFold | (64, 32) | Single-task | 82.93 | 72.80 | 82.79 | 89.45 | 81.99 |
| ExFold | (64, 32) | Mixed-task | 70.12 | 33.20 | 79.30 | 89.94 | 68.14 |
| BASE | (64, 32) | Single-task | 87.80 | 77.80 | 81.55 | 90.64 | 84.45 |
| BASE | (64, 32) | Mixed-task | 86.59 | 71.20 | 81.05 | 90.24 | 82.27 |
The dense model maintains comparable accuracy across single-task and mixed-task execution. Similarly, at and , BASE shows relatively small changes on HumanEval, MATH-401, and BoolQ. MBPP is more sensitive to mixing, with accuracy reductions of 4.2 and 6.6 percent respectively.
At tighter budgets, mixing produces larger accuracy reductions despite the regressor maintaining near-oracle expert-selection quality. This suggests that the main challenge is competition between heterogeneous requests under a shared expert budget rather than a substantial deterioration in selection quality.
These experiments show that BASE can accommodate heterogeneous requests under continuous batching while maintaining consistent expert-selection quality across tasks. Although tight expert budgets increase competition between requests, increasing the budget with concurrency substantially preserves accuracy. We will include these mixed-task, regressor robustness, and per-task accuracy analyses in the revised manuscript.
Reviewer xTCg Response, Part 1
We thank the reviewer for their careful evaluation. Their concerns regarding cross-term approximations, fixed expert budgets, and mixed-task batching have motivated additional experiments that provide a more comprehensive assessment of BASE's robustness and limitations.
W1: Cross-Term Error and Optimal Expert Selection
We appreciate the reviewer's observation that the cross-term-aware greedy search in Figure 1(c) is itself an approximation rather than the exact optimum. Importantly, this analysis is conducted in an oracle setting, where the true outputs of all routed experts are available. Such information is not accessible during practical inference without executing the experts we aim to skip. Therefore, these experiments are intended to evaluate the accuracy of our additive approximation and the potential benefit of explicitly modeling cross terms, rather than propose an additional inference-time selection method. To address the reviewer's concern, we conducted two additional experiments: (1) comparing additive selection against the globally optimal expert subset under the exact removal-error objective, and (2) evaluating whether explicitly modeling cross terms improves downstream accuracy.
We implemented an exact branch-and-bound algorithm that identifies the subset of M experts minimizing the true batch-level removal error, including all cross terms:
\qquad D(A)=\sum_t\left\|\sum_{u\in S_t\setminus A}w_{t,u}E_u(z_t)\right\|_2^2,$$ where $U$ is the union of routed experts across the batch, $S_t$ is the original routed expert set for token $t$, and $D(A)$ measures the squared error caused by discarding experts outside the retained set $A$. Unlike our additive approximation, this objective explicitly accounts for interactions between omitted expert contributions, including their potential cancellation or reinforcement. Branch-and-bound systematically explores candidate expert subsets while using mathematically valid lower bounds on $D(A)$ to eliminate branches that cannot improve upon the best solution found so far. Once all remaining branches are explored or safely eliminated, the algorithm certifies that the selected subset achieves the global minimum of $D(A)$, rather than merely a locally optimal solution. Using the same experimental setting as Figure 1(c), we successfully certified global optimality for **99.7–100%** of evaluated decoding positions across Qwen3-30B-A3B, Qwen1.5-MoE, and DeepSeek-V2-Lite. Table: Exact removal-error ratio of additive selection to the globally optimal subset: | Model | Budget $M$ | Median ratio | 95th percentile | Greedy exactly optimal | |---|---:|---:|---:|---:| | Qwen3-30B-A3B | 12 | 1.0013 | 1.0314 | 99.1% | | Qwen1.5-MoE | 12 | 1.0000 | 1.0172 | 99.7% | | Qwen1.5-MoE | 20 | 1.0000 | 1.0112 | 99.8% | | DeepSeek-V2-Lite | 12 | 1.0000 | 1.0256 | 99.8% | *The ratio compares the exact removal error of additive selection with the global optimum; a ratio of 1 indicates optimality.* The additive approximation achieves **near-optimal removal error across all three architectures**. At the median, its error is within 0.13% of the optimum on Qwen3 and exactly optimal on the other two architectures. Even at the 95th percentile, the excess removal error remains below 3.2%. Furthermore, the cross-term-aware greedy search used in Figure 1(c) reaches the certified optimum at more than 99% of positions, confirming that our original comparison provides a close approximation to the true optimum. Impact of explicitly modeling cross terms on downstream accuracy: We additionally evaluated a cross-term-aware oracle that uses the true expert outputs to select experts by minimizing the exact removal error. We compare it against the additive oracle using the same expert budget, evaluation examples, and backfill procedure as Table 2. On 1,920 sampled Qwen3 decoding positions, we verified that the cross-term-aware greedy search reaches the certified global optimum in 99.6% of cases. Additive versus cross-term-aware oracle selection: | Selection method | Qwen3 | Qwen1.5-MoE | |---|---:|---:| | Additive oracle | 73.58 | **45.82** | | Cross-term-aware oracle | **74.46** | 45.34 | On both architectures, explicitly modeling cross terms produces no significant improvement in downstream accuracy compared with additive oracle selection. This holds even when the true expert outputs are available, whereas a practical cross-term-aware selector would require evaluating the expert outputs it aims to skip. Together, these results show that the additive removal-error formulation closely approximates the globally optimal selection objective, while explicitly accounting for cross terms provides no significant downstream accuracy improvement. We will include the exact optimum analysis and cross-term-aware oracle experiments in the revised manuscript.