- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SP-MoE aligns intermediate neurons across experts and replaces low-sensitivity neurons with one shared component while retaining a private residual for every routed expert. The shared component stays resident, so offloading transfers only the smaller private part. Decomposed execution computes the shared branch once per token and exactly reproduces the transformed model.
Critical strengths
- Sharing neurons while preserving every routed expert is an interesting alternative to expert pruning and merging.
- The method reduces both the expert parameter count and the offloading transfer size.
- The serving evaluation carefully charges the resident shared component against the same weight budget and shows clear gains in I/O-bound settings.
Critical weaknesses
- SP-MoE uses the same sharing ratio in every MoE layer, although different layers may have very different tolerance to neuron sharing.
- The quantitative distortion analysis is limited to OLMoE layer 0 and does not show how approximation error accumulates across layers or models. It is therefore unclear whether uniform sharing is close to optimal.
- The system evaluation considers single-GPU host offloading. In expert-parallel inference, the shared branch must be replicated or sharded, and either choice may change memory, computation, and collective-communication costs.
- The paper shows that the advantage over HC-SMoE shrinks with batch size and reverses at batch 8. This suggests that the current benefit is concentrated in strongly I/O-bound deployments; quality differences such as the 0.4-point Mixtral gain are also based on single runs.
Questions for authors
- Could the authors report per-layer reconstruction error or sensitivity as the sharing ratio changes?
- Under the same total parameter budget, how does a layer-adaptive sharing allocation compare with the current uniform ratio?
- How should the shared branch be placed under multi-GPU or multi-node expert parallelism: replicated on every rank, tensor-parallel sharded, or assigned to separate ranks? What additional collective communication would each design require?
- A multi-GPU EP experiment comparing replicated and sharded shared branches would help establish whether SP-MoE remains useful beyond PCIe offloading. Please also report repeated quality runs when the margin over a baseline is small.
2 comments
Dear Reviewer MhP5,
Thank you again for your review and the specific questions. As the discussion period closes soon, we would like to check whether our response addresses your concerns.
Following your questions, we compared uniform, depth-decreasing, and error-adaptive layer allocations at the same total budget over three seeds. All three stay within 0.5 points of each other, inside seed noise. We also measured per-layer and accumulated error on all three models: accumulated error stays at about 1.5× the local error rather than compounding. In vLLM on DeepSeek-V4, we measured a replicated and a tensor-parallel-sharded shared branch, and SP-MoE improves throughput over the uncompressed model by 21.0% and 19.8%. Finally, three-seed runs on Mixtral put the margin over NAEE and REAP at about 1.4 points.
If any point remains unclear, or you would like additional details, we would be glad to respond before the deadline. If our response resolves your concerns, we would be grateful if you would consider it in your final assessment.
Best regards,
The Authors
Reply to Reviewer MhP5
Thank you for your review. We added a comparison of three layer allocations over three seeds, per-layer and accumulated error on all three models, a multi-GPU expert-parallel deployment, and three-seed runs for the Mixtral comparison. Responses follow in order, with each question answered under its weakness.
W1 / Q2. Uniform sharing ratio across layers.
In our test, layer-adaptive allocation does not beat the uniform ratio under the same total budget. On OLMoE at 37.5% sparsity, we compared three allocations with the same mean sparsity, each over three calibration seeds:
The adaptive rule was fixed before evaluation. It moves sparsity substantially (from 26% to 56% across layers, and 75% on the least sensitive layer), yet the paired difference from uniform is −0.2 ± 0.9 points and changes sign across seeds. Across the three allocations, quality stays within 0.5 points, inside seed noise, so we keep the uniform ratio as the default: it needs no per-layer calibration.
W2 / Q1. Error across layers, ratios, and models.
We measured the relative error of every MoE block output on held-out C4, for all layers of all three models. Per-layer error on OLMoE has the same profile at every ratio and scales smoothly with it: the mean is 0.34 / 0.43 / 0.50 at 25 / 37.5 / 50% sparsity, with layer 2 the least sensitive (0.07–0.10) and layers 0, 5–7, and 11–13 the most.
To measure accumulation, we compared each block's output on the compressed model's own hidden states with the original model's (accumulated) against its output on the original hidden states (local). At 50%:
Accumulated error stays at about 1.5× the local error rather than compounding layer after layer. The relative drift of each MoE layer's input rises over the first quarter of depth, plateaus, and falls in the last layers: on Mixtral it is 0.61, 0.63, 0.59, and 0.42 at layers 8, 16, 24, and 31.
W3 / Q3–Q4. Expert parallelism.
The three placements trade memory against communication:
We measured the first two placements in vLLM on DeepSeek-V4 (not in the paper; 25% sharing ratio, three hash-routed layers unchanged, weights resident on GPUs): replicated on V4-Flash (TP1/DP2/EP2) and sharded on V4-Pro (TP8/EP8). SP-MoE raises throughput over the uncompressed model by 21.0% and 19.8% on average, and on all ten workloads, mainly because it has fewer weight bytes.
W4 / Q4. Batch size and repeated runs.
Memory-constrained, small-batch offloading is the setting SP-MoE targets, and there it is fastest: on OLMoE it is 1.16× faster than HC-SMoE at batch 1 and 1.08× at batch 4. Its lead over pruning narrows as batching amortizes transfers. At batch 8 it is 4.0% slower, yet transfers less (113 vs 190 MB per token) and is 6.9 points more accurate (Table 5). Its speedup over the uncompressed model does not narrow: 1.33×, 1.44×, and 1.43× at batch 1, 4, and 8 (App. D.3).
The batch-8 loss is launch overhead, not arithmetic. SP-MoE computes about 44% fewer expert FLOPs per token than the uncompressed model, but our runtime launches each active expert separately, and SP-MoE keeps 64 experts against HC-SMoE's 32. In vLLM, whose fused MoE kernel runs all experts together, SP-MoE matches expert pruning (EASY-EP) at the same sparsity, with all weights on GPUs and 16–256 concurrent requests (not in the paper): +0.7% on Qwen3-Coder-480B and +5.4% on Qwen3-Next-80B.
The 0.4-point Mixtral margin over NAEE was a single run. Over three calibration seeds in our current evaluation environment, SP-MoE averages 64.9 ± 0.3, NAEE 63.5 ± 0.5, and REAP 63.4 ± 0.4, a lead of about 1.4 points over each, four to five times its standard error.