acceptodds
Under review as a conference paper at ICLR 2027

Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Execution

Abstract

Mixture of Experts (MoE) architecture has become the standard for state-of-the-art large language models, owing to its computational efficiency through sparse expert activation. However, sparsity through finer expert granularity is becoming increasingly difficult to achieve due to fundamental training challenges such as expert collapse and load imbalance. In this work, we explore and leverage intra-expert activation sparsity as a complementary and underexplored dimension of sparsity in MoE models. Surprisingly, substantial intra-expert sparsity is readily available in off-the-shelf pre-trained MoE models, providing up to 98% sparsity on top of existing inter-expert sparsity of the routed experts without significant accuracy loss. We explore intra-expert activation sparsity across six open source MoE models ranging from 1B to 397B parameters, and extend the MoE execution pipeline of vLLM to leverage intra-expert activation sparsity while preserving the existing MoE layer optimizations, achieving up to 2.0x speedup in MoE layer execution and 1.2x end-to-end speedup compared to the dense vLLM baseline.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
ML systems, sparsity, sparse execution, Mixture of Experts

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_CKD1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper characterizes neuron-level activation sparsity inside active MoE experts, observes that a small fraction of “critical tokens” account for much of the dense–sparse divergence, trains a lightweight per-token/per-expert threshold predictor, and integrates sparse routed-expert execution into vLLM with custom Triton/CUDA/HIP-compatible kernels and dense fallback. Across six models from 1B to 397B parameters, the paper reports increasing robustness to routed-expert neuron sparsity.

Critical strengths

The paper goes beyond measuring activation sparsity and addresses how to exploit it in an optimized serving engine. Preserving graph capture, expert-batched execution, and compatibility with surrounding MoE operations is an important engineering contribution. The paper also provided fairly broad evaluation - six models spanning substantially different sizes and expert configurations

Critical weaknesses

The core masking algorithm remains incompletely specified: the thresholded quantity is identified, but exact hard-mask semantics and sparsity-control constants are missing. Predictor validated on only one model: the six-model characterization does not validate the learned solution across models, particularly at the largest claimed sparsity levels.

Questions for authors

What is the exact neuron score and mask used in characterization, training, and deployment? At 93% achieved sparsity, how do static, per-layer, per-expert, CATS-style, and learned predictors compare?

3 comments

We thank you for the valuable feedback and questions on our work, and we apologize for the late reply; our 122B model predictor training took longer and required more resources than we anticipated.


1. Neuron score and mask used in characterization, training, and deployment

Here are the exact neuron score and mask used throughout the paper.

Characterization (Section 3)

  • We used the post-activation output (silu(gate)) for neuron scores and top-k to mask out the neurons. This choice was made to achieve the exact sparsity values.

Training (Section 4.1)

  • We also use post-activation output (silu(gate)) to calculate the neuron scores. For hard sparsity calculation during forward, we use silu(gate) >= threshold, and for the soft sparsity calculation during backward, we use sigmoid on (silu(gate) - threshold) to enable gradient backpropagation.

Deployment (Section 4.2, vLLM)

  • We also use post-activation output (silu(gate)) to calculate the neuron scores, and generate the mask using silu(gate) >= threshold. These two operations are fused in a single Triton kernel.

We describe the training masking as "sigmoid over gate output subtracted by threshold", which conflates the sigmoid of the activation with the sigmoid of the soft sparsity calculation. We also found that the fused activation Triton kernel we included in the appendix thresholds on the raw gate output, which is not what we use in our training and evaluations. We mistakenly included a previous version of the kernel in the appendix. We apologize for any confusion caused by the different neuron score and mask generation techniques presented in the paper and appendix, and we will make the choice clear in the revised paper.

Note that exact neuron scores and threshold numbers are both model- and input-dependent, as each layer and expert has a different activation output distribution that depends on the given token. For example, the threshold usually falls between 0.15 and 0.2 on expert 0, layer 0 of Qwen3.5-35B-A3B, whereas the activation output spans between -0.28 and 2.


Continued...

Continued...


2. Larger Qwen3.5-122B-A10B model results with TP/EP

We train the sparsity cutoff predictor on Qwen3.5-122B-A10B and present the accuracy and performance results below.

(1) Accuracy

BenchmarkDense10%30%60%90%95%100%Predictor (sparsity)
ARC-Challenge61.363.664.762.562.463.322.461.2 (88.8%)
Winogrande70.473.069.468.669.369.951.770.5 (90.0%)
GSM8K93.689.680.775.869.957.91.388.7 (91.1%)

(2) E2E Serving Performance, TP=4

ModeRate/sTTFT p50/p95/p99 (ms)TPOT p50/p95/p99 (ms)E2EL p50/p95/p99 (s)E2EL speedup p50/p95/p99
Dense139/103/1118.4/10.7/11.21.42/5.61/7.15--
Sparse137/98/1187.1/8.2/8.61.14/4.56/5.931.25x/1.23x/1.20x
Dense450/103/12413.1/14.4/15.61.99/8.38/10.75--
Sparse444/96/1039.7/11.1/12.51.55/6.01/7.791.29x/1.39x/1.38x
Dense1690/161/20624.6/29.8/35.54.16/13.77/17.22--
Sparse1685/169/30219.2/24.1/60.83.20/11.16/13.791.30x/1.23x/1.25x

(3) E2E Serving Performance, EP=4

ModeRate/sTTFT p50/p95/p99 (ms)TPOT p50/p95/p99 (ms)E2EL p50/p95/p99 (s)E2EL speedup p50/p95/p99
Dense144/108/1259.9/12.2/12.91.64/6.56/8.53--
Sparse148/110/1239.4/10.8/12.41.56/6.01/7.991.05x/1.09x/1.07x
Dense460/118/13515.6/17.6/19.72.47/10.00/12.79--
Sparse454/111/13112.3/13.8/16.61.98/7.89/10.191.25x/1.27x/1.25x
Dense16109/192/22829.3/37.5/55.54.88/15.61/19.33--
Sparse16103/207/24824.5/34.3/57.04.06/13.47/16.201.20x/1.16x/1.19x

(4) Token batch size per step

ModeRate/s1-1516-6364-255256-10231024+TotalSparse Steps (%)
Sparse1161,08831,1521,5846,19296200,11297.70
Sparse423,47244,4483,1686,33638477,80892.72
Sparse1619,24815,45614,2564,0321,53654,52890.85

We measure p50/p95/p99 TTFT (time to first token), TPOT (time per output token), and E2EL (end-to-end latency) using vLLM’s serving benchmark with inputs from the ShareGPT dataset with Poisson arrival at rates of 1/4/16, with TP=4 or EP=4 on Nvidia H200 GPUs. We also report the token batch size per step distribution, gathered from the TP run. As can be seen from the results, the predictor-based sparsity translates well into both accuracy and performance even on the bigger 122B model.


Continued...

... Continued


3. Predictor Comparisons

We present dense, shared, calibrated per-layer (static), learned per-layer, learned per-expert, and our predictor-based sparsity threshold accuracy. “Shared” uses a constant threshold shared across all layers, experts, and tokens, and sparsity is averaged across all dimensions. “Calibrated per-layer (Static)” uses a per-layer threshold, where each layer achieves the target sparsity independently, and this is what we use as static sparsity in the paper. The learned results are based on our predictor design with the sparsity gradient feedback restricted to apply per layer or per expert.

Results are below.

MethodAchieved sparsityGSM8K
Dense0.0000.9067
Shared0.90210.6694
Calibrated per-layer (static)0.89070.7657
Learned per-layer0.89990.7854
Learned per-expert0.89800.7756
CATS, abs(SiLU(gate))0.90470.8112
Learned per-layer, per-expert, and per-token (Ours)0.92810.8946

As shown above, our per-token prediction and activation balancing achieves the best accuracy near that of the dense model, whereas the learned per-layer and per-expert results achieve only slightly better accuracy compared to calibrated per-layer. Note that both learned predictors' states are initialized to output the calibrated threshold before training, with uniform activation across experts and tokens.

This is expected, as the training loss is per-token in the first place. Our loss is calculated as the difference between dense logit outputs and sparse logit outputs. Larger loss generates larger gradients that push the predictor to output lower thresholds and achieve less sparsity, while smaller loss does the opposite. Therefore, activation rebalance is enabled by the difference in loss across tokens, and per-layer or per-expert training cannot achieve meaningful rebalancing of the activation budget under the given loss definition.

Meaningful per-expert or per-layer activation rebalancing independent of tokens may be possible using a different loss definition, but we find defining loss as the output token differences between dense and sparse executions to be the most intuitive for recovering accuracy.

We also add CATS-style pruning results, where the neuron score used is abs(silu(gate)), on top of our per-layer calibrated static baseline. Using the absolute value for neuron pruning yields higher accuracy, and we missed it in our paper because we mistakenly attributed CATS to use the non-absolute neuron score, as we do in our approach. We thank the reviewer for pointing it out.

We tried training a predictor with the CATS-style absolute neuron score. However, the training was largely unstable, especially near the 0.2785 threshold where the neurons with a negative portion of the SiLU output begin to enter the masked set, with a low 0.6725 GSM8K score at achieved sparsity of 0.9163. It seems that better parameterization and handling of the sparsity is required to enable training of CATS-style thresholds.


We also note a typo in our paper: Section 3.2’s “Critical Neurons” header should be “Critical Tokens”. We apologize for any confusion that this may have caused.

Reviewer_6ArQOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper focuses on neuron-level sparsity within each activated expert in mixture-of-expert (MoE) architecture, known as intra-expert activation sparsity, which manifests through the claim that when a token is rooted to an expert, it only needs to activate a small subset of that expert's neurons. Although activation sparsity is known to occur in MoE architectures as well as in ReLU-LLMs, little knowledge is known for intra-expert activation sparsity. For this purpose, the authors take pre-trained MoE LLMs and measure their benchmark scores at different levels of intra-expert activation sparsity, by sorting each expert’s post-activation outputs and zeroing out the lowest-scoring neurons without any architectural modifications. The testing is performed across six open-source MoE models ranging from 1B to 397B parameters, and then redesigned the vLLM's MoE execution pipeline to accommodate sparse execution. The testing showed increased performance of up to 2× speedup in MoE layer execution and 1.2× speedup in end-to-end inference.

Critical strengths

Showed the possibility to extend the well known Activation Sparsity in LLM ReLU to neuron level sparsity within each activated expert, referred to as intra-expert activation sparsity in MoE LLM architectures, which is next extended to vLLM as well. Through a set of experimentation across six open sources MoE models of different scales, together with reshaping architecture to accommodate vLLM, leveraging intra-expert activation sparsity is shown to improve speedup in MoE layer execution and 1.2 end-to-end speed up as compared to vLLM baseline.

Critical weaknesses
  • The problem formulating is very much missing. The authors could have used known illustration of MoE sparsity formulating to elucidate the intra-expert activation sparsity mathematically, and use appendix to further expand the problem formulating, ablation and evaluation metrics, which are restricted to only execution time and inference speed up, and later token batch size. Possible memory gains have not been investigated nor discussed. The discussion section is very short to reflect on the innovation part and scalability of the results. This makes the paper far from ICLR standard.

  • Most of the results are available at https://arxiv.org/abs/2605.08575. The authors did not refer to it in the manuscript, neither mentioned in the metadata of the submission

Questions for authors
  • How is the work in the paper differ from https://arxiv.org/abs/2605.08575?

  • Why not use an appendix to elaborate more on theoretical part of intra-expert activation sparsity as well as empirical analysis

  • It is known that properties and performances of MoE do not scale proportionally when moving from small models to medium/ large models. it is interesting to see how these inference speed up, execution time vary with model size.

  • The paper seems to be a very preliminary work that still requires a lot of improvement both from problem statement perspective and empirical analysis

2 comments

We thank you for the valuable feedback and questions on our work, and we apologize for the late reply.


1. Problem Formulation

We agree that the problem statement of intra-expert activation sparsity is not well-defined in the paper. We intended the paper as an empirical findings paper that builds on the existing activation sparsity works (Sparsing Law, CATS, etc.) to extend the concept of activation sparsity to the popular Mixture of Experts models, and verify its efficacy in real-world serving engines such as vLLM. As such, the paper's real estate was mostly spent on observations, solution design, and evaluations, and not on the problem formulation.

We will strengthen the theoretical problem formulation, including a formal statement of intra-expert activation sparsity in standard MoE notation, defining the scored quantity, the mask, and the sparsity level as explicit functions, and relating them to conventional router-level sparsity, in the main paper and in detail in the appendix.


2. Relationships to the arXiv paper

Our submission extends the mentioned paper with the critical token observation and threshold predictor design, increasing generative benchmark accuracy. I apologize for the confusion; we acknowledge its existence and informed the AC of the arXiv paper.


3.Scaling with model size

We train the sparsity cutoff predictor on Qwen3.5-122B-A10B and present the accuracy and performance results below.

(1) Accuracy

BenchmarkDense10%30%60%90%95%100%Predictor (sparsity)
ARC-Challenge61.363.664.762.562.463.322.461.2 (88.8%)
Winogrande70.473.069.468.669.369.951.770.5 (90.0%)
GSM8K93.689.680.775.869.957.91.388.7 (91.1%)

(2) E2E Serving Performance, TP=4

ModeRate/sTTFT p50/p95/p99 (ms)TPOT p50/p95/p99 (ms)E2EL p50/p95/p99 (s)E2EL speedup p50/p95/p99
Dense139/103/1118.4/10.7/11.21.42/5.61/7.15--
Sparse137/98/1187.1/8.2/8.61.14/4.56/5.931.25x/1.23x/1.20x
Dense450/103/12413.1/14.4/15.61.99/8.38/10.75--
Sparse444/96/1039.7/11.1/12.51.55/6.01/7.791.29x/1.39x/1.38x
Dense1690/161/20624.6/29.8/35.54.16/13.77/17.22--
Sparse1685/169/30219.2/24.1/60.83.20/11.16/13.791.30x/1.23x/1.25x

(3) E2E Serving Performance, EP=4

ModeRate/sTTFT p50/p95/p99 (ms)TPOT p50/p95/p99 (ms)E2EL p50/p95/p99 (s)E2EL speedup p50/p95/p99
Dense144/108/1259.9/12.2/12.91.64/6.56/8.53--
Sparse148/110/1239.4/10.8/12.41.56/6.01/7.991.05x/1.09x/1.07x
Dense460/118/13515.6/17.6/19.72.47/10.00/12.79--
Sparse454/111/13112.3/13.8/16.61.98/7.89/10.191.25x/1.27x/1.25x
Dense16109/192/22829.3/37.5/55.54.88/15.61/19.33--
Sparse16103/207/24824.5/34.3/57.04.06/13.47/16.201.20x/1.16x/1.19x

(4) Token batch size per step

ModeRate/s1-1516-6364-255256-10231024+TotalSparse Steps (%)
Sparse1161,08831,1521,5846,19296200,11297.70
Sparse423,47244,4483,1686,33638477,80892.72
Sparse1619,24815,45614,2564,0321,53654,52890.85

We measure p50/p95/p99 TTFT (time to first token), TPOT (time per output token), and E2EL (end-to-end latency) using vLLM’s serving benchmark with inputs from the ShareGPT dataset with Poisson arrival at rates of 1/4/16, with TP=4 or EP=4 on Nvidia H200 GPUs. We also report the token batch size per step distribution, gathered from the TP run. As can be seen from the results, the predictor-based sparsity translates well into both accuracy and performance even on the bigger 122B model.


4. Memory Gains

The difference in memory footprint during execution is minimal, as our sparse execution does not prune neurons and uses the same set of weights as the dense execution. Similarly, as our method only targets the MoE layer, there is no reduction in KV cache storage as well.

Our approach reduces total memory reads, and thus the memory bandwidth usage, as we skip loading the masked-out neurons from HBM. This is why we see a more constant speedup of 1 to 1.5 times in the batch 1 to 4 regions of Figure 6 in the paper. In these memory-bound regions, the speedup is not from reduced computation, but from the reduced memory reads, which is constant across batch sizes. Above a batch size of 16, we see an increase in speedup as the reduced compute also takes effect, and at a larger batch size above 512, we lose speedup as the arithmetic intensity of our kernel is lower than that of dense GEMM. We will make this memory analysis more explicit in the revised paper.

Reviewer_6ArQOpenReview ↗

Feedback

We thank the authors for the explanations. I will maintain my score unchanged. Basically the authors should upload the revised version for now and not stating that they will do the work later. The analysis of memory impact is very incomplete and open to debate without clear discussion grounds.

Reviewer_przMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces intra-expert activation sparsity as a complementary approach to conventional MoE sparsity. Unlike traditional MoE architectures, which perform routing at the expert level, the proposed method further exploits neuron-level sparsity within the selected experts by dynamically identifying and skipping less important neurons. Experimental results demonstrate that the proposed approach reduces computational overhead and accelerates inference while largely preserving the original model's accuracy.

Critical strengths

The main strength of this paper lies in its effective optimization of the trade-off between inference efficiency and model accuracy in MoE architectures. By exploiting intra-expert activation sparsity, the proposed method significantly reduces computational overhead while largely preserving the original model's accuracy. The integration with vLLM further demonstrates the practical applicability of the proposed approach, offering a promising solution for efficient MoE inference.

Critical weaknesses

One major limitation of this work is the lack of comprehensive validation of the proposed method across different model families. Although the authors investigate intra-expert activation sparsity in six MoE models, the proposed sparsity threshold predictor and its end-to-end inference performance are primarily evaluated on Qwen3.5-35B-A3B. Consequently, it remains unclear whether the reported efficiency gains and accuracy preservation can be consistently achieved across other MoE architectures. Additional experiments on different model families would strengthen the generalizability of the proposed approach.

Questions for authors

None

1 comment

We thank you for the valuable feedback and questions on our work, and we apologize for the late reply. 

We sincerely apologize for the limited evaluation provided for the predictor and end-to-end inference. We are rather resource-constrained, and building the right training hyperparameters and environment took considerable time during our experiments. While not a different model family, we used our existing environment to train the Qwen3.5-122B-A10B predictor to present accuracy and end-to-end serving benchmark results on a bigger model under tensor and expert parallelism. We present the results below.

(1) Accuracy

BenchmarkDense10%30%60%90%95%100%Predictor (sparsity)
ARC-Challenge     61.3 63.664.762.562.463.322.461.2 (88.8%)
Winogrande        70.4 73.069.468.669.369.951.770.5 (90.0%)
GSM8K             93.689.680.775.869.957.9  1.388.7 (91.1%)

(2) E2E Serving Performance, TP=4

ModeRate/sTTFT p50/p95/p99 (ms)TPOT p50/p95/p99 (ms)E2EL p50/p95/p99 (s)E2EL speedup p50/p95/p99
Dense139/103/1118.4/10.7/11.21.42/5.61/7.15--
Sparse137/98/1187.1/8.2/8.61.14/4.56/5.931.25x/1.23x/1.20x
Dense450/103/12413.1/14.4/15.61.99/8.38/10.75--
Sparse444/96/1039.7/11.1/12.51.55/6.01/7.791.29x/1.39x/1.38x
Dense1690/161/20624.6/29.8/35.54.16/13.77/17.22--
Sparse1685/169/30219.2/24.1/60.83.20/11.16/13.791.30x/1.23x/1.25x

(3) E2E Serving Performance,  EP=4

ModeRate/sTTFT p50/p95/p99 (ms)TPOT p50/p95/p99 (ms)E2EL p50/p95/p99 (s)E2EL speedup p50/p95/p99
Dense144/108/1259.9/12.2/12.91.64/6.56/8.53--
Sparse148/110/1239.4/10.8/12.41.56/6.01/7.991.05x/1.09x/1.07x
Dense460/118/13515.6/17.6/19.72.47/10.00/12.79--
Sparse454/111/13112.3/13.8/16.61.98/7.89/10.191.25x/1.27x/1.25x
Dense16109/192/22829.3/37.5/55.54.88/15.61/19.33--
Sparse16103/207/24824.5/34.3/57.04.06/13.47/16.201.20x/1.16x/1.19x

(4) Token batch size per step

ModeRate/s1-1516-6364-255256-10231024+TotalSparse Steps (%)
Sparse1161,08831,1521,5846,19296200,11297.70
Sparse423,47244,4483,1686,33638477,80892.72
Sparse1619,24815,45614,2564,0321,53654,52890.85

We measure p50/p95/p99 TTFT (time to first token), TPOT (time per output token), and E2EL (end-to-end latency) using vLLM’s serving benchmark with inputs from the ShareGPT dataset with Poisson arrival at rates of 1/4/16, with TP=4 or EP=4 on Nvidia H200 GPUs. We also report the token batch size per step distribution, gathered from the TP run. As can be seen from the results, the predictor-based sparsity translates well into both accuracy and performance even on the bigger 122B model, with higher end-to-end speedup achieved in the serving case, up to 1.39 times speedup on the TP=4, 4 req/s case.

Reviewer_QieEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether pretrained MoE models can be made more efficient by also exploiting activation sparsity inside each expert. The authors measure this sparsity in six models ranging from 1B to 397B parameters. They find that routed experts can be sparsified much more when the shared experts are kept dense. They also find a problem with applying the same sparsity to every token: a small number of sensitive tokens can cause a large drop in quality on generative reasoning tasks. To address this, they use distillation to train a small predictor that sets a separate threshold for each token and each expert. They implement sparse execution in vLLM with custom kernels, and the system switches between sparse and dense execution depending on batch size. The paper reports layer-level speedups on both AMD and NVIDIA GPUs and up to 1.2× end-to-end speedup on Qwen3.5-35B-A3B.

Critical strengths

The main strength of the paper is that it turns intra-expert sparsity into real speedups in an optimized serving engine, not only in isolated kernels. The implementation handles practical issues such as graph capture, kernel fusion, and the fact that sparse kernels lose their advantage at larger batch sizes. The baseline is unmodified vLLM, and the paper reports end-to-end serving numbers, so the results matter in practice. The 1.2× end-to-end speedup is smaller than the layer-level gains, but it is still useful.

I also found the analysis of where sparsification works and where it fails informative. The difference between shared and routed experts, and the drop on generative reasoning tasks, are more useful to know than average sparsity numbers. In Table 2, the predictor recovers most of the GSM8K and BBH performance at about 93% routed-expert sparsity, and it does better than both the 90% and 95% static settings. This is good evidence that adaptive allocation helps.

Critical weaknesses

My main concern is that the experiments do not show which part of the predictor matters. Compared with fixed thresholds, the predictor changes several things at once. Its thresholds depend on the token, they are learned through distillation, and they are set per expert. The natural baseline would be a learned threshold that does not depend on the token, trained on the same data, and the paper does not include one. There is also no ablation of the critical-token margin loss. Without these, I can’t tell how much of the gain comes from token-dependent allocation and the new loss. Comparing at the same achieved sparsity would also help. None of this changes the fact that the predictor beats the static settings, but it does limit what we can conclude about the method.

Second, the sparsity claims go further than the deployment results support. The characterization covers six models, but the predictor and the end-to-end evaluation only use Qwen3.5-35B-A3B. The very high sparsity cutoffs for the larger models come from discriminative benchmarks. The paper itself shows that these benchmarks can miss large drops in generation quality. I would suggest either adding a generative evaluation on at least one more model or stating these cutoffs more carefully.

Questions for authors
  1. At the same achieved sparsity, how does the predictor compare with per-layer or per-expert thresholds that are learned from the same distillation data but don’t depend on the token? How much does the critical-token margin term add over KL distillation alone?
  2. Do the high sparsity cutoffs reported for the larger models also hold on generative reasoning tasks? Results on one more model would show whether the findings and the predictor carry over beyond Qwen3.5-35B-A3B.
  3. In Table 5, at 16 requests/s, p95 TTFT goes from 499 ms to 583 ms even though end-to-end latency improves. Is this reproducible, and what causes it?
2 comments

We thank you for the valuable feedback and questions on our work, and we apologize for the late reply. Our 122B model predictor training took longer and required more resources than we anticipated.


1. Predictor Comparisons 

We present dense, shared, calibrated per-layer (static), learned per-layer, learned per-expert, and our learned per-layer, per-expert, and per-token sparsity threshold accuracy. “Shared” uses a constant threshold shared across all layers, experts, and tokens, and sparsity is averaged across all dimensions. “Calibrated per-layer (Static)” uses a per-layer threshold, where each layer achieves the target sparsity independently, and this is what we use as static sparsity in the paper. The learned results are based on our predictor design with the sparsity gradient feedback restricted to apply per layer or per expert, by removing the token terms in the predictor formula in equation (1).

We present the results below.

Method                             Achieved sparsityGSM8K 
Dense                 0.000            0.9067
Shared       0.9021          0.6694
Calibrated per-layer (static)0.8907           0.7657
Learned per-layer          0.8999           0.7854
Learned per-expert          0.8980           0.7756
Learned per-layer, per-expert, and per-token (Ours)       0.9281       0.8946

As shown above, our per-token prediction and activation balancing achieves the best accuracy near that of the dense model, whereas the learned per-layer and per-expert result achieves only slightly better accuracy compared to calibrated per-layer. Note that all learned predictors' states are initialized to output the calibrated threshold before training, with uniform activation across experts and tokens.

This is expected, as the training loss is per-token in the first place. Our loss is calculated as the difference between dense logit outputs and sparse logit outputs. Larger loss generates larger gradients that push the predictor to output lower thresholds and achieve less sparsity, while smaller loss does the opposite. Therefore, activation rebalance is enabled by the difference in loss across tokens, and per-layer or per-expert training cannot achieve meaningful rebalancing of the activation budget under the given loss definition.

Meaningful per-expert or per-layer activation rebalancing independent from tokens may be possible using a different loss definition, but we find defining loss as the output token differences between dense and sparse executions to be the most intuitive for recovering accuracy.


Continued...

...Continued


2. Qwen3.5-122B-A10B results 

We train the sparsity cutoff predictor on Qwen3.5-122B-A10B and present the accuracy and performance results below.

(1) Accuracy

BenchmarkDense10%30%60%90%95%100%Predictor (sparsity)
ARC-Challenge     61.3 63.664.762.562.463.322.461.2 (88.8%)
Winogrande        70.4 73.069.468.669.369.951.770.5 (90.0%)
GSM8K             93.689.680.775.869.957.9  1.388.7 (91.1%)

(2) E2E Serving Performance, TP=4

ModeRate/sTTFT p50/p95/p99 (ms)TPOT p50/p95/p99 (ms)E2EL p50/p95/p99 (s)E2EL speedup p50/p95/p99
Dense139/103/1118.4/10.7/11.21.42/5.61/7.15--
Sparse137/98/1187.1/8.2/8.61.14/4.56/5.931.25x/1.23x/1.20x
Dense450/103/12413.1/14.4/15.61.99/8.38/10.75--
Sparse444/96/1039.7/11.1/12.51.55/6.01/7.791.29x/1.39x/1.38x
Dense1690/161/20624.6/29.8/35.54.16/13.77/17.22--
Sparse1685/169/30219.2/24.1/60.83.20/11.16/13.791.30x/1.23x/1.25x

(3) E2E Serving Performance, EP=4

ModeRate/sTTFT p50/p95/p99 (ms)TPOT p50/p95/p99 (ms)E2EL p50/p95/p99 (s)E2EL speedup p50/p95/p99
Dense144/108/1259.9/12.2/12.91.64/6.56/8.53--
Sparse148/110/1239.4/10.8/12.41.56/6.01/7.991.05x/1.09x/1.07x
Dense460/118/13515.6/17.6/19.72.47/10.00/12.79--
Sparse454/111/13112.3/13.8/16.61.98/7.89/10.191.25x/1.27x/1.25x
Dense16109/192/22829.3/37.5/55.54.88/15.61/19.33--
Sparse16103/207/24824.5/34.3/57.04.06/13.47/16.201.20x/1.16x/1.19x

(4) Token batch size per step

ModeRate/s1-1516-6364-255256-10231024+TotalSparse Steps (%)
Sparse1161,08831,1521,5846,19296200,11297.70
Sparse423,47244,4483,1686,33638477,80892.72
Sparse1619,24815,45614,2564,0321,53654,52890.85

We measure p50/p95/p99 TTFT (time to first token), TPOT (time per output token), and E2EL (end-to-end latency) using vLLM’s serving benchmark with inputs from the ShareGPT dataset with Poisson arrival at rates of 1/4/16, with TP=4 or EP=4 on Nvidia H200 GPUs. We also report the token batch size per step distribution, gathered from the TP run. As can be seen from the results, the predictor-based sparsity translates well into both accuracy and performance even on the bigger 122B model. 


3.P95 TTFT regression 

We agree that the 583ms of p95 TTFT in Table 5 is somewhat (17%) larger than the dense execution p95 TTFT, but we were not able to reproduce the behavior in further tests. It is also not visible in either TP or EP runs of Qwen3.5-122B-A10B.

We do repeatedly see a long tail in the TTFT distribution, but it is not specific to sparse execution and is also observed in dense executions. We find that vLLM takes longer to schedule the execution when the number of concurrent requests is larger, which is also partially visible in Figure 5 of the paper. 


We also note a typo in our paper: Section 3.2’s “Critical Neurons” header should be “Critical Tokens”. We apologize for any confusion that this may have caused.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.