- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper characterizes neuron-level activation sparsity inside active MoE experts, observes that a small fraction of “critical tokens” account for much of the dense–sparse divergence, trains a lightweight per-token/per-expert threshold predictor, and integrates sparse routed-expert execution into vLLM with custom Triton/CUDA/HIP-compatible kernels and dense fallback. Across six models from 1B to 397B parameters, the paper reports increasing robustness to routed-expert neuron sparsity.
Critical strengths
The paper goes beyond measuring activation sparsity and addresses how to exploit it in an optimized serving engine. Preserving graph capture, expert-batched execution, and compatibility with surrounding MoE operations is an important engineering contribution. The paper also provided fairly broad evaluation - six models spanning substantially different sizes and expert configurations
Critical weaknesses
The core masking algorithm remains incompletely specified: the thresholded quantity is identified, but exact hard-mask semantics and sparsity-control constants are missing. Predictor validated on only one model: the six-model characterization does not validate the learned solution across models, particularly at the largest claimed sparsity levels.
Questions for authors
What is the exact neuron score and mask used in characterization, training, and deployment? At 93% achieved sparsity, how do static, per-layer, per-expert, CATS-style, and learned predictors compare?
3 comments
Continued...
2. Larger Qwen3.5-122B-A10B model results with TP/EP
We train the sparsity cutoff predictor on Qwen3.5-122B-A10B and present the accuracy and performance results below.
(1) Accuracy
| Benchmark | Dense | 10% | 30% | 60% | 90% | 95% | 100% | Predictor (sparsity) |
|---|---|---|---|---|---|---|---|---|
| ARC-Challenge | 61.3 | 63.6 | 64.7 | 62.5 | 62.4 | 63.3 | 22.4 | 61.2 (88.8%) |
| Winogrande | 70.4 | 73.0 | 69.4 | 68.6 | 69.3 | 69.9 | 51.7 | 70.5 (90.0%) |
| GSM8K | 93.6 | 89.6 | 80.7 | 75.8 | 69.9 | 57.9 | 1.3 | 88.7 (91.1%) |
(2) E2E Serving Performance, TP=4
| Mode | Rate/s | TTFT p50/p95/p99 (ms) | TPOT p50/p95/p99 (ms) | E2EL p50/p95/p99 (s) | E2EL speedup p50/p95/p99 |
|---|---|---|---|---|---|
| Dense | 1 | 39/103/111 | 8.4/10.7/11.2 | 1.42/5.61/7.15 | -- |
| Sparse | 1 | 37/98/118 | 7.1/8.2/8.6 | 1.14/4.56/5.93 | 1.25x/1.23x/1.20x |
| Dense | 4 | 50/103/124 | 13.1/14.4/15.6 | 1.99/8.38/10.75 | -- |
| Sparse | 4 | 44/96/103 | 9.7/11.1/12.5 | 1.55/6.01/7.79 | 1.29x/1.39x/1.38x |
| Dense | 16 | 90/161/206 | 24.6/29.8/35.5 | 4.16/13.77/17.22 | -- |
| Sparse | 16 | 85/169/302 | 19.2/24.1/60.8 | 3.20/11.16/13.79 | 1.30x/1.23x/1.25x |
(3) E2E Serving Performance, EP=4
| Mode | Rate/s | TTFT p50/p95/p99 (ms) | TPOT p50/p95/p99 (ms) | E2EL p50/p95/p99 (s) | E2EL speedup p50/p95/p99 |
|---|---|---|---|---|---|
| Dense | 1 | 44/108/125 | 9.9/12.2/12.9 | 1.64/6.56/8.53 | -- |
| Sparse | 1 | 48/110/123 | 9.4/10.8/12.4 | 1.56/6.01/7.99 | 1.05x/1.09x/1.07x |
| Dense | 4 | 60/118/135 | 15.6/17.6/19.7 | 2.47/10.00/12.79 | -- |
| Sparse | 4 | 54/111/131 | 12.3/13.8/16.6 | 1.98/7.89/10.19 | 1.25x/1.27x/1.25x |
| Dense | 16 | 109/192/228 | 29.3/37.5/55.5 | 4.88/15.61/19.33 | -- |
| Sparse | 16 | 103/207/248 | 24.5/34.3/57.0 | 4.06/13.47/16.20 | 1.20x/1.16x/1.19x |
(4) Token batch size per step
| Mode | Rate/s | 1-15 | 16-63 | 64-255 | 256-1023 | 1024+ | Total | Sparse Steps (%) |
|---|---|---|---|---|---|---|---|---|
| Sparse | 1 | 161,088 | 31,152 | 1,584 | 6,192 | 96 | 200,112 | 97.70 |
| Sparse | 4 | 23,472 | 44,448 | 3,168 | 6,336 | 384 | 77,808 | 92.72 |
| Sparse | 16 | 19,248 | 15,456 | 14,256 | 4,032 | 1,536 | 54,528 | 90.85 |
We measure p50/p95/p99 TTFT (time to first token), TPOT (time per output token), and E2EL (end-to-end latency) using vLLM’s serving benchmark with inputs from the ShareGPT dataset with Poisson arrival at rates of 1/4/16, with TP=4 or EP=4 on Nvidia H200 GPUs. We also report the token batch size per step distribution, gathered from the TP run. As can be seen from the results, the predictor-based sparsity translates well into both accuracy and performance even on the bigger 122B model.
Continued...
... Continued
3. Predictor Comparisons
We present dense, shared, calibrated per-layer (static), learned per-layer, learned per-expert, and our predictor-based sparsity threshold accuracy. “Shared” uses a constant threshold shared across all layers, experts, and tokens, and sparsity is averaged across all dimensions. “Calibrated per-layer (Static)” uses a per-layer threshold, where each layer achieves the target sparsity independently, and this is what we use as static sparsity in the paper. The learned results are based on our predictor design with the sparsity gradient feedback restricted to apply per layer or per expert.
Results are below.
| Method | Achieved sparsity | GSM8K |
|---|---|---|
| Dense | 0.000 | 0.9067 |
| Shared | 0.9021 | 0.6694 |
| Calibrated per-layer (static) | 0.8907 | 0.7657 |
| Learned per-layer | 0.8999 | 0.7854 |
| Learned per-expert | 0.8980 | 0.7756 |
| CATS, abs(SiLU(gate)) | 0.9047 | 0.8112 |
| Learned per-layer, per-expert, and per-token (Ours) | 0.9281 | 0.8946 |
As shown above, our per-token prediction and activation balancing achieves the best accuracy near that of the dense model, whereas the learned per-layer and per-expert results achieve only slightly better accuracy compared to calibrated per-layer. Note that both learned predictors' states are initialized to output the calibrated threshold before training, with uniform activation across experts and tokens.
This is expected, as the training loss is per-token in the first place. Our loss is calculated as the difference between dense logit outputs and sparse logit outputs. Larger loss generates larger gradients that push the predictor to output lower thresholds and achieve less sparsity, while smaller loss does the opposite. Therefore, activation rebalance is enabled by the difference in loss across tokens, and per-layer or per-expert training cannot achieve meaningful rebalancing of the activation budget under the given loss definition.
Meaningful per-expert or per-layer activation rebalancing independent of tokens may be possible using a different loss definition, but we find defining loss as the output token differences between dense and sparse executions to be the most intuitive for recovering accuracy.
We also add CATS-style pruning results, where the neuron score used is abs(silu(gate)), on top of our per-layer calibrated static baseline. Using the absolute value for neuron pruning yields higher accuracy, and we missed it in our paper because we mistakenly attributed CATS to use the non-absolute neuron score, as we do in our approach. We thank the reviewer for pointing it out.
We tried training a predictor with the CATS-style absolute neuron score. However, the training was largely unstable, especially near the 0.2785 threshold where the neurons with a negative portion of the SiLU output begin to enter the masked set, with a low 0.6725 GSM8K score at achieved sparsity of 0.9163. It seems that better parameterization and handling of the sparsity is required to enable training of CATS-style thresholds.
We also note a typo in our paper: Section 3.2’s “Critical Neurons” header should be “Critical Tokens”. We apologize for any confusion that this may have caused.
We thank you for the valuable feedback and questions on our work, and we apologize for the late reply; our 122B model predictor training took longer and required more resources than we anticipated.
1. Neuron score and mask used in characterization, training, and deployment
Here are the exact neuron score and mask used throughout the paper.
Characterization (Section 3)
Training (Section 4.1)
Deployment (Section 4.2, vLLM)
We describe the training masking as "sigmoid over gate output subtracted by threshold", which conflates the sigmoid of the activation with the sigmoid of the soft sparsity calculation. We also found that the fused activation Triton kernel we included in the appendix thresholds on the raw gate output, which is not what we use in our training and evaluations. We mistakenly included a previous version of the kernel in the appendix. We apologize for any confusion caused by the different neuron score and mask generation techniques presented in the paper and appendix, and we will make the choice clear in the revised paper.
Note that exact neuron scores and threshold numbers are both model- and input-dependent, as each layer and expert has a different activation output distribution that depends on the given token. For example, the threshold usually falls between 0.15 and 0.2 on expert 0, layer 0 of Qwen3.5-35B-A3B, whereas the activation output spans between -0.28 and 2.
Continued...