acceptodds
Under review as a conference paper at ICLR 2027

SP-MoE: Post-Training Neuron Sharing for Memory-Constrained Inference

Abstract

Mixture-of-Experts (MoE) language models activate a few experts per token, but all experts occupy storage, making deployment memory a bottleneck. Existing work addresses this at two levels. At the system level, offloading keeps a subset of experts resident and fetches the rest from host memory; under a tight budget these full-expert transfers can dominate decoding: a Mixtral-8x7B baseline spends 86.8% of decode time waiting on transfers at a 16GiB weight budget, even with prefetching. At the model level, pruning removes experts and merging consolidates them, reducing storage but potentially losing expert-specific capacity; each retained expert remains a whole transfer unit. Both levels leave the transfer unit at expert granularity. We introduce SP-MoE, a post-training neuron-sharing method that reduces the size of this unit while retaining every routed expert. It aligns intermediate neurons, selects low-sensitivity ones by reconstruction-gradient magnitude, and aggregates them into one shared component per layer, leaving a private residual per expert. The shared component stays resident, so a miss transfers only that residual. Because a SwiGLU neuron is a gate column, an up column, and a down row whose contributions add in the down projection, decomposed execution computes the shared branch once per token and combines it with routing-weighted private outputs, reproducing the transformed model exactly. Across OLMoE, Qwen1.5-MoE, and Mixtral, SP-MoE achieves the highest average zero-shot accuracy among tested compression baselines at 50% expert-parameter sparsity. On one RTX 4090 at batch size one, under per-model weight budgets including resident shared weights, it improves throughput at every tested sparsity, by up to 1.71 over HC-SMoE and 2.09 over Full MoE. On Mixtral at 50% sparsity, each miss transfers 144 rather than 336 MiB.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, neuron sharing, post-training compression, memory-efficient inference, expert offloading

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MhP5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SP-MoE aligns intermediate neurons across experts and replaces low-sensitivity neurons with one shared component while retaining a private residual for every routed expert. The shared component stays resident, so offloading transfers only the smaller private part. Decomposed execution computes the shared branch once per token and exactly reproduces the transformed model.

Critical strengths
  • Sharing neurons while preserving every routed expert is an interesting alternative to expert pruning and merging.
  • The method reduces both the expert parameter count and the offloading transfer size.
  • The serving evaluation carefully charges the resident shared component against the same weight budget and shows clear gains in I/O-bound settings.
Critical weaknesses
  • SP-MoE uses the same sharing ratio in every MoE layer, although different layers may have very different tolerance to neuron sharing.
  • The quantitative distortion analysis is limited to OLMoE layer 0 and does not show how approximation error accumulates across layers or models. It is therefore unclear whether uniform sharing is close to optimal.
  • The system evaluation considers single-GPU host offloading. In expert-parallel inference, the shared branch must be replicated or sharded, and either choice may change memory, computation, and collective-communication costs.
  • The paper shows that the advantage over HC-SMoE shrinks with batch size and reverses at batch 8. This suggests that the current benefit is concentrated in strongly I/O-bound deployments; quality differences such as the 0.4-point Mixtral gain are also based on single runs.
Questions for authors
  1. Could the authors report per-layer reconstruction error or sensitivity as the sharing ratio changes?
  2. Under the same total parameter budget, how does a layer-adaptive sharing allocation compare with the current uniform ratio?
  3. How should the shared branch be placed under multi-GPU or multi-node expert parallelism: replicated on every rank, tensor-parallel sharded, or assigned to separate ranks? What additional collective communication would each design require?
  4. A multi-GPU EP experiment comparing replicated and sharded shared branches would help establish whether SP-MoE remains useful beyond PCIe offloading. Please also report repeated quality runs when the margin over a baseline is small.
2 comments
AuthorsOpenReview ↗

Reply to Reviewer MhP5

Thank you for your review. We added a comparison of three layer allocations over three seeds, per-layer and accumulated error on all three models, a multi-GPU expert-parallel deployment, and three-seed runs for the Mixtral comparison. Responses follow in order, with each question answered under its weakness.

W1 / Q2. Uniform sharing ratio across layers.

In our test, layer-adaptive allocation does not beat the uniform ratio under the same total budget. On OLMoE at 37.5% sparsity, we compared three allocations with the same mean sparsity, each over three calibration seeds:

AllocationNine-task mean
Uniform (paper)52.6 ± 0.6
Decreasing with depth (57.5% → 17.5%)52.1 ± 0.6
Adaptive: inversely proportional to each layer's measured error52.3 ± 0.4

The adaptive rule was fixed before evaluation. It moves sparsity substantially (from 26% to 56% across layers, and 75% on the least sensitive layer), yet the paired difference from uniform is −0.2 ± 0.9 points and changes sign across seeds. Across the three allocations, quality stays within 0.5 points, inside seed noise, so we keep the uniform ratio as the default: it needs no per-layer calibration.

W2 / Q1. Error across layers, ratios, and models.

We measured the relative error of every MoE block output on held-out C4, for all layers of all three models. Per-layer error on OLMoE has the same profile at every ratio and scales smoothly with it: the mean is 0.34 / 0.43 / 0.50 at 25 / 37.5 / 50% sparsity, with layer 2 the least sensitive (0.07–0.10) and layers 0, 5–7, and 11–13 the most.

To measure accumulation, we compared each block's output on the compressed model's own hidden states with the original model's (accumulated) against its output on the original hidden states (local). At 50%:

Model (layers)LocalAccumulatedRatio
OLMoE (16)0.500.771.53×
Mixtral (32)0.450.651.46×
Qwen3-30B-A3B (48)0.600.871.44×

Accumulated error stays at about 1.5× the local error rather than compounding layer after layer. The relative drift of each MoE layer's input rises over the first quarter of depth, plateaus, and falls in the last layers: on Mixtral it is 0.61, 0.63, 0.59, and 0.42 at layers 8, 16, 24, and 31.

W3 / Q3–Q4. Expert parallelism.

The three placements trade memory against communication:

  • Replicated (data parallel). Each rank holds a full copy and applies it to its own tokens, so no communication is needed. The cost is memory, which grows as each rank holds fewer experts: on DeepSeek-V4-Flash at EP2, the copy is about 1% of each rank's routed-expert bytes.
  • Tensor-parallel sharded. Like any tensor-parallel MLP (gate/up split by column, down by row), it needs an all-reduce of its partial outputs. On DeepSeek-V4-Pro (TP8/EP8) the routed outputs are all-reduced anyway, and vLLM adds the shared output before that all-reduce, so no collective is added.
  • One shared part per EP rank. Each rank stores the shared part of its own experts and applies it to the tokens dispatched there, so nothing is replicated and no communication is added. Our Hugging Face implementation already supports this grouping; our vLLM implementation does not yet. We leave this to future work.

We measured the first two placements in vLLM on DeepSeek-V4 (not in the paper; 25% sharing ratio, three hash-routed layers unchanged, weights resident on GPUs): replicated on V4-Flash (TP1/DP2/EP2) and sharded on V4-Pro (TP8/EP8). SP-MoE raises throughput over the uncompressed model by 21.0% and 19.8% on average, and on all ten workloads, mainly because it has fewer weight bytes.

W4 / Q4. Batch size and repeated runs.

Memory-constrained, small-batch offloading is the setting SP-MoE targets, and there it is fastest: on OLMoE it is 1.16× faster than HC-SMoE at batch 1 and 1.08× at batch 4. Its lead over pruning narrows as batching amortizes transfers. At batch 8 it is 4.0% slower, yet transfers less (113 vs 190 MB per token) and is 6.9 points more accurate (Table 5). Its speedup over the uncompressed model does not narrow: 1.33×, 1.44×, and 1.43× at batch 1, 4, and 8 (App. D.3).

The batch-8 loss is launch overhead, not arithmetic. SP-MoE computes about 44% fewer expert FLOPs per token than the uncompressed model, but our runtime launches each active expert separately, and SP-MoE keeps 64 experts against HC-SMoE's 32. In vLLM, whose fused MoE kernel runs all experts together, SP-MoE matches expert pruning (EASY-EP) at the same sparsity, with all weights on GPUs and 16–256 concurrent requests (not in the paper): +0.7% on Qwen3-Coder-480B and +5.4% on Qwen3-Next-80B.

The 0.4-point Mixtral margin over NAEE was a single run. Over three calibration seeds in our current evaluation environment, SP-MoE averages 64.9 ± 0.3, NAEE 63.5 ± 0.5, and REAP 63.4 ± 0.4, a lead of about 1.4 points over each, four to five times its standard error.

Dear Reviewer MhP5,

Thank you again for your review and the specific questions. As the discussion period closes soon, we would like to check whether our response addresses your concerns.

Following your questions, we compared uniform, depth-decreasing, and error-adaptive layer allocations at the same total budget over three seeds. All three stay within 0.5 points of each other, inside seed noise. We also measured per-layer and accumulated error on all three models: accumulated error stays at about 1.5× the local error rather than compounding. In vLLM on DeepSeek-V4, we measured a replicated and a tensor-parallel-sharded shared branch, and SP-MoE improves throughput over the uncompressed model by 21.0% and 19.8%. Finally, three-seed runs on Mixtral put the margin over NAEE and REAP at about 1.4 points.

If any point remains unclear, or you would like additional details, we would be glad to respond before the deadline. If our response resolves your concerns, we would be grateful if you would consider it in your final assessment.

Best regards,

The Authors

Reviewer_fMENOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SP-MoE, a post-training method that shares aligned intermediate neurons across the routed experts of each MoE layer. Low-sensitivity neurons, ranked by reconstruction-gradient magnitude, are replaced by one routing-weighted shared component, and each expert keeps the rest as a private residual. The router and expert set are unchanged, and the shared branch is computed exactly once per token. In offloaded serving, the shared component stays on the GPU, so each cache miss transfers only the smaller private part. On OLMoE, Qwen1.5-MoE, and Mixtral, SP-MoE achieves the optimal zero-shot accuracy among expert pruning and merging baselines at matched size.

Critical strengths
  • The paper changes the offloading unit rather than only the number of experts, which is a practical and well-motivated idea for memory-constrained MoE inference. The serving evaluation charges the resident shared weights to the same budget and reports cache hit rate, traffic per token, and stall fraction. This makes the source of the speedup clear: smaller per-miss transfers rather than better cache coverage. The Mixtral gains are substantial under a realistic consumer-GPU setting.

  • Decomposed execution is exact for the transformed model and reuses the shared computation across active experts, so the serving path adds no approximation beyond the sharing itself. The method also keeps every routed expert and the original router, and it outperforms expert pruning and merging baselines on quality at matched parameter counts across all three main models.

Critical weaknesses
  • The paper is missing the most direct baseline: intra-expert channel or neuron pruning at matched parameters. Such methods also shrink every expert and therefore the transfer unit, without needing a resident shared component. This contradicts the framing that model-level compression always leaves the transfer unit at expert granularity. The paper’s own removal ablation, which is essentially this baseline, is only 1.2 points below SP-MoE on average, wins four of nine tasks, and its advantage comes mainly from MMLU. Without a serving and quality comparison against channel pruning, the benefit of sharing over simply removing neurons is not established.

  • The serving comparison covers only HC-SMoE and the full model. It omits mixed-precision or quantized offloading systems such as HOBBIT, which target the same per-miss transfer cost. The paper argues quantization is orthogonal, but it shows no combined result and no head-to-head comparison.

  • The speedup is limited to small-batch decoding. At batch size eight on OLMoE, SP-MoE is already slower than the pruned baseline because it still executes all routed experts. Serving results also cover only the three smaller models on one consumer GPU, while the frontier-model experiments make no serving claims.

  • The dimension-selection criterion has a weak justification. The reconstruction target compares a single expert’s output with the full mixture output, which an individual expert is not meant to match. The score uses gradient magnitude alone and ignores the size of the actual replacement perturbation, so a first-order argument does not support it directly. The supporting diagnostic is a correlation on a single layer of OLMoE.

  • The quality cost at the reported operating points is large. At 50 percent sparsity, OLMoE drops from 65.7 to 48.4 average accuracy and Qwen1.5 from 66.3 to 58.2. On Mixtral, the margin over NAEE and REAP is under one point. The headline throughput gain at Mixtral 50 percent therefore comes with a quality loss that limits practical use.

  • The frontier-model generative results are hard to interpret. HC-SMoE and EASY-EP collapse at only 20 to 25 percent sparsity, for example to near-zero HumanEval. SP-MoE substantially exceeds the uncompressed model on some tasks, such as formal fallacies on Qwen3-Coder (0.864 versus 0.456) and MATH on DeepSeek-R1. These anomalies suggest evaluation or baseline-configuration issues. REAP, the most relevant baseline for generative settings, is not included there.

  • The conceptual novelty is limited. Permutation alignment follows MC-SMoE, and the shared component effectively acts as a post-hoc shared expert built by weighted averaging. The contribution is mainly the combination of these pieces with an offloading runtime.

Questions for authors

please see questions in weakness

3 comments
AuthorsOpenReview ↗

Reply to Reviewer fMEN - Part1

Thank you for your review. We added a channel-pruning baseline on three models, INT4 results, three-seed runs for the Mixtral comparison, two replacement-aware selection criteria, and a corrected frontier evaluation that includes REAP. Responses follow in order.

W1. Missing intra-expert channel-pruning baseline at matched parameters.

We added it. SP-MoE is more accurate than channel pruning (CP) on all three models; CP is faster on Mixtral, where the shared branch is large relative to the budget. CP removes the lowest-scoring intermediate neurons from every expert at the same parameter count as SP-MoE. The abstract's statement concerns expert pruning and merging; CP also shrinks the transfer unit, so it is the right control for sharing versus removal. At 50% sparsity (RTX 4090, batch size 1; ± = std over three calibration seeds, rerun in our current evaluation environment, so values differ from the paper's tables; rows without ± are single runs):

Model (experts/layer)Acc. SP-MoEAcc. CP-Wandatok/s SP-MoEtok/s CP-Wanda
OLMoE (64), 3 GiB49.5 ± 0.647.2 ± 0.211.7811.93
Mixtral (8), 16 GiB64.9 ± 0.356.62.272.67
Qwen3-30B-A3B (128)62.058.4––

The accuracy gap is 2.3–8.3 points; on Mixtral, SP-MoE leads on 8 of 9 tasks. On Qwen3 it also leads CP at 25% and 37.5%, with either Wanda or our gradient score.

Throughput depends on the number of experts N, because the shared branch holds 1/(N−1) of the parameters it removes. With 64 experts it is negligible, and the two run within 1.2% on OLMoE. With 8 experts, Mixtral's shared branch takes 6.4 GB of the budget, so CP caches 79 experts against 49 and is 17.7% faster, but 8.3 points less accurate. Recent MoEs use 128–512 experts, for example, on Qwen3-30B at 50% the shared branch is 0.23 GB.

W2. No combined or head-to-head result with quantized offloading (HOBBIT).

The two combine: with INT4, SP-MoE loses only 0.2 points and the byte savings multiply. With simulated INT4 (round-to-nearest, group size 128) on all expert weights, including the shared branch, SP-MoE at 50% loses 0.2 points on both Mixtral and OLMoE, less than the uncompressed models (0.6 and 1.1). By byte count, INT4 cuts an SP-MoE miss on Mixtral from 151 to 40 MB (a full expert: 352 to 94 MB) and the shared branch from 6.4 to 1.7 GB. Our runtime has no INT4 kernel, so we report no INT4 throughput. HOBBIT loads low-precision copies of experts on a miss and could load SP-MoE's private parts the same way; we did not compare against it directly.

W3. Speedup limited to small batches; no serving results for frontier models.

Keeping all routed experts costs launches, not arithmetic: at 50% on OLMoE, SP-MoE computes about 44% fewer expert FLOPs per token than the uncompressed model. Our offloading runtime launches each active expert separately, and SP-MoE keeps 64 experts against HC-SMoE's 32, so a batch activates more of them. On OLMoE, SP-MoE is 1.16× faster than HC-SMoE at batch 1 and 1.08× at batch 4. At batch 8 it is 4.0% slower, yet still transfers less (113 vs 190 MB per token) and is 1.43× faster than the uncompressed model (App. D.3) and 6.9 points more accurate than HC-SMoE (Table 5).

We also served two of the paper's frontier models in vLLM, whose fused MoE kernel runs all experts together, with all weights on GPUs (not offloading; not in the paper), at 16–256 concurrent requests, with the same parallel configuration for both methods. Averaged over ten workloads, SP-MoE matches expert pruning (EASY-EP) at the same sparsity: +0.7% on Qwen3-Coder-480B (20%) and +5.4% on Qwen3-Next-80B (25%). In the separate quality rerun in W6, SP-MoE also scores 15 HumanEval points above EASY-EP on Qwen3-Coder.

With offloading on the newer Qwen3-30B-A3B (8 GiB, batch 1), SP-MoE is the fastest method at 25 / 37.5 / 50%: 3.40 / 3.73 / 3.94 tok/s, against 3.31 / 3.32 / 3.47 for REAP.

AuthorsOpenReview ↗

Reply to Reviewer fMEN - Part2

W4. Weak justification of the dimension-selection criterion.

The criterion is a sensitivity heuristic; criteria that also model the replacement size rank neurons better but are at most 0.6 points more accurate. Eq. 4 compares each expert with the mixture output only to obtain a gradient measuring each neuron's sensitivity; no expert is trained to match it, and with the expert's own output as target every gradient would be zero. All our diagnostics measure the MoE-block output.

We tested two criteria that include the replacement size: perturbation-gradient, ∣⟨g,θS−θe⟩∣|\langle g, \theta_S - \theta_e \rangle|, and output substitution, the output change when a neuron is replaced by its shared copy. On every OLMoE and Mixtral layer, gradient magnitude correlates positively with the measured replacement error (mean r = 0.54 / 0.42) and above every structural criterion (Wanda: 0.14 / 0.24). The two replacement-aware criteria correlate better (up to 0.65 / 0.51), but at 50% sparsity they are at most 0.6 points more accurate than the gradient (three seeds; OLMoE 49.4 and 49.6 vs 49.5; Mixtral output substitution 65.4 vs 64.9). Ignoring the replacement size thus costs little, whereas structural criteria cost up to 5.8 points (W7).

W5. Large quality cost at 50% sparsity; small margins over NAEE and REAP on Mixtral.

SP-MoE loses the least of all tested methods at 50%, and its Mixtral lead holds over three seeds. Halving expert parameters costs every method substantial accuracy: at 50% the best baseline reaches 45.9 on OLMoE and 55.0 on Qwen1.5, against 48.4 and 58.2. Lower sparsities are the practical operating points: on Mixtral at 37.5%, SP-MoE loses 3.5 points (Table 7) and runs 1.68× faster than the uncompressed model (App. D.1).

The Mixtral margins over NAEE and REAP were single runs. Over three calibration seeds, rerun together in our current environment (so absolute scores differ from Table 7), SP-MoE averages 64.9 ± 0.3, NAEE 63.5 ± 0.5, and REAP 63.4 ± 0.4, a lead of about 1.4 points, four to five times its standard error. Each NAEE or REAP miss also transfers a full 336 MiB expert, against 144 MiB for SP-MoE.

W6. Hard-to-interpret frontier generative results; REAP missing.

Both anomalies come from the evaluation setup. In a corrected rerun that includes REAP, SP-MoE is still the most accurate compressed model.

Baseline collapse. Every method in Table 9 was calibrated on 4×2048 tokens, too few for Qwen3-Coder's 160 experts: in layer 61, 47 experts receive no routing mass. Pruning and merging choose experts from these statistics. SP-MoE removes or merges no expert; in the same setting it lost 8.5 HumanEval points against 61 for HC-SMoE. We rebuilt all methods on Qwen3-Coder at 20% sparsity with 16×2048 tokens and evaluated them on the same day:

MethodHumanEvalMATH-500
Uncompressed0.7300.928
SP-MoE0.7440.920
REAP0.7380.870
HC-SMoE0.6590.920
EASY-EP0.5910.904

The baselines recover, so the 22–33-point losses in Table 9 overstate their gap to SP-MoE, which remains highest on HumanEval and ties HC-SMoE on MATH-500. Its lead over the uncompressed model, 0.744 vs 0.730, is about 2 of 164 problems. Only Qwen3-Coder was rerun; the DeepSeek-R1 and Qwen3-Next baselines in Table 9 still use 4×2048.

Scores above the uncompressed model. These come from two evaluation errors. On BBH formal fallacies, the stop string "\n\n" cut off the chain of thought before the answer in 124 of 250 uncompressed responses, against 7 for SP-MoE; without that stop string, SP-MoE no longer exceeds the uncompressed model. On DeepSeek-R1 MATH, the SP-MoE run omitted the chat template; with the template on both, SP-MoE at 25% scores 0.880 against 0.868, within noise for 500 problems.

W7. Limited conceptual novelty: MC-SMoE alignment plus a post-hoc shared expert.

Problem. We study how to cut offloading overhead without losing model quality. Pruning and merging trade quality for it: they remove experts, yet each remaining expert still transfers whole. SP-MoE shrinks the transfer unit below an expert while keeping every expert, its router entry, and its private neurons exact.

Technique. Alignment follows MC-SMoE, as the paper states, but to share neurons, not to merge experts. Selection asks which neurons a shared copy can replace, not which to remove, and pruning criteria do worse: on OLMoE, Wanda, min norm, and activation sparsity score 0.6–2.0 points below gradient selection, and max similarity up to 5.8 below (Table 3). The shared branch is not an added shared expert: the shared neurons remain part of each routed expert, weighted by its routing weight, so execution is exact and computes them once per token. The resident part stays small: 0.23 GB on Qwen3-30B at 50%, against 6.8–9.7 GB of MoBE bases.

Dear Reviewer fMEN,

Thank you again for your detailed review. As the discussion period closes soon, we would like to check whether our response addresses your concerns.

We added the channel-pruning baseline you suggested on three models. SP-MoE is 2.3–8.3 points more accurate, and we report the throughput trade-off on Mixtral. We also added INT4 results, which show that the two methods combine, and three-seed runs for the Mixtral comparison: SP-MoE leads NAEE and REAP by about 1.4 points, four to five times the standard error. For the frontier evaluation, we traced both anomalies you noted to evaluation-setup errors, a stop string and a missing chat template. We then reran Qwen3-Coder with REAP included and adequate calibration, and SP-MoE remains the most accurate compressed model.

If any point remains unclear, or you would like additional details, we would be glad to respond before the deadline. If our response resolves your concerns, we would be grateful if you would consider it in your final assessment.

Best regards,

The Authors

Reviewer_xZMXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

MoE LLMs are known for its large memory footprint, and expert offloading has been one of the techniques that mitigates the memory consumption. The authors argue that prior works focus on expert pruning and expert merging, both leads to a cache-miss to transfer the whole expert via the PCIe, and leads to a huge latency. As a result, the authors propose SP-MoE, a post-training mechanism that extracts shared component for each layer of experts, so each expert only hodls a private component that's much smaller than a full expert. On OLMoE/Qwen 1.5 and Mixtral 8x7B and various configurations, the SP-MoE has achieved highest zero-shot accuracy across a set of baselines selected.

Critical strengths
  • The paper is clearly written and concise without sacrificing readability. In particular, the SwiGLU decomposition and its execution scheme are explained well.

  • The authors have not only reported the cache-hit rate, but also included throughput experiments based on real hardware benchmarks.

Critical weaknesses
  • The comparison with closely related work is insufficient. The authors states that the issue in expert offloading is that expert pruning and expert merging "leave the transfer unit at expert granularity." (L018) but prior approaches already explore shared representations with expert-specific components, e.g. D2D^2 -MoE (MobiCom 2025) /MoBE (ICLR 2026, cited on L123) /ResMoE (KDD 2025) -- all of them are published papers. Although the paper discusses MoBE and explains a structural distinction, it does not provide an empirical comparison. Comparisons with closely related methods under matched parameter and GPU memory budgets would help establish the benefits of complete-neuron sharing for both model quality and offloading performance.

  • The model used are also coming from earlier model families, and unfortunately quite outdated. It seems to me that this manuscript has been written for a while. I can resonate with the authors feelings -- this is very unfortunate. But I think sufficient comparison against prior work is needed to establish the novelty of this paper and justify its value of being published. It also needs to be evaluated with relatively newer models. While some such experiments are briefly mentioned in Sec. 4.2, the evaluation seems to be upon quality only without any offloading throughput experiments. The MoE architecture has improved significantly, so more work is needed to prove the relevancy of the proposed method.

  • Experiments were mostly conducted at Batch size B = 1. At higher batch sizes, in section D3, the proposed method is worse than the HC-SMoE method.

  • The choice of the Eq 4 may need some more justification. Let's say an expert is dominant on a token and the weight is very close to 1, in this case the reconstruction error for this expert is essentially 0, and then the score, which is the gradient of the loss is also close to 0.

  • Code assets is not provided alongside the paper in the initial phase.

Questions for authors

How would the proposed method compare against the methods proposed above, particularly on later models?

Why do we choose to use the Eq 4 and Eq 5 to decide the score of an expert component? See the potential counter example above.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer xZMX - Part1

Thank you for your review. We added a quality and offloading comparison with D²-MoE, MoBE, and REAP on Qwen3-30B-A3B, and tested the selection criterion on every layer of OLMoE and Mixtral.

W1 / Q1. Comparison with closely related methods (D²-MoE, MoBE, ResMoE) under matched budgets.

SP-MoE has the highest throughput of all methods that fit the budget at every tested sparsity, and the highest accuracy at 37.5% and 50%. MoBE is more accurate at 25%, but at that sparsity its shared bases alone exceed the 8 GiB budget.

Which D²-MoE. Two 2025 papers use this name. The one that shares a representation across experts is the delta-decompression method (ICML 2025, arXiv 2502.17298), which we compare against. The MobiCom 2025 D²MoE (arXiv 2504.15299) is a dynamic bit-width serving system: it reduces bits per parameter, SP-MoE reduces parameters fetched per miss, so the two are complementary.

How the methods differ. The sentence at L018 refers to the two approaches named just before it: offloading, and expert pruning or merging. Shared-representation methods are a third approach; with the shared part resident, they also transfer less than a full expert per miss. They differ from SP-MoE in what they share and what each expert computes:

MethodShared (resident)Per expert (fetched on a miss)Each active expert computes
D²-MoEa full-width base experta low-rank deltafull width
MoBEgate/up basis matricesgate/up coefficients, full down projectionfull width, after combining the bases into gate/up weights
ResMoEa barycenter experta pruned or low-rank residualfull width, after restoring the expert
SP-MoEa subset of complete neurons, one copythe remaining neurons, uncompressedprivate neurons only; shared ones run once per token

D²-MoE and ResMoE approximate each expert's difference from a shared center, and MoBE approximates gate/up through shared bases; SP-MoE keeps private neurons exact and replaces only the shared ones with one copy. We did not measure ResMoE: its released Mixtral code adds the barycenter output without the routing weight, unlike the restored expert F(x;C+Ri)F(x; C+R_i) in the paper, and we could not validate a faithful Qwen3 implementation in time.

Setup. Qwen3-30B-A3B-Instruct-2507 (released July 2025; 128 fine-grained experts with top-8 routing, against Mixtral's 8 experts with top-2). Sparsity is the fraction of routed-expert parameters removed; shared parameters count as retained. Accuracy: mean of the paper's nine zero-shot tasks, one calibration seed (uncompressed: 74.4). Serving: one RTX 4090, MoE-Infinity, batch size 1, 8 GiB weight budget including all resident shared weights. D²-MoE uses its official code, adapted to Qwen3. MoBE at 25% is its authors' released checkpoint; we built 37.5% and 50% with their Qwen3 recipe (1,000 epochs of weight-reconstruction optimization). The other methods are one-shot.

SparsityMethodAcc.tok/sMB/token
25%SP-MoE71.03.40720
D²-MoE68.13.25726
MoBE73.3does not fit–
REAP70.23.311079
37.5%SP-MoE67.03.73525
D²-MoE62.93.42539
MoBE65.9does not fit–
REAP64.83.321039
50%SP-MoE62.03.94338
D²-MoE55.33.86370
MoBE52.73.41*526
REAP60.43.47965

*Byte-matched surrogate; see Serving.

Quality. SP-MoE is above D²-MoE at every sparsity (by 3–7 points) and above REAP (by 0.7–2.2). MoBE leads at 25% by 2.3 points; SP-MoE leads at 37.5% by 1.1 and at 50% by 9.3. On OLMoE and Mixtral, SP-MoE's standard deviation over three calibration seeds is 0.3–0.6 points.

Serving. SP-MoE is fastest at every sparsity: 2.5–13.7% above REAP, which transfers whole experts, and 4.5–9.1% above D²-MoE at 25–37.5%. At 50%, D²-MoE comes within 2.1% because its delta is about as small as SP-MoE's private part (370 vs 338 MB/token), but there it is 6.8 points less accurate. At 25% and 37.5%, MoBE's bases alone take 9.66 and 6.77 GB; with 3.08 GB of non-expert weights, both exceed the 8 GiB (8.59 GB) budget before any expert is cached. The MoBE 50% row is a byte-matched surrogate: a random-weight checkpoint in SP-MoE's format with MoBE's resident and per-expert byte sizes, with routing replayed from the real MoBE model. It omits rebuilding each loaded expert's gate/up weights from the bases (about 0.2 GFLOP), which favors MoBE. Even so, the bases leave less room for cached experts: hit rate 0.37, against 0.64 for SP-MoE, whose shared part is 0.23 GB.

W2. Newer models. The paper already evaluates generative quality on DeepSeek-R1, Qwen3-Coder-480B, and Qwen3-Next-80B (App. C.3). The table above adds offloading throughput on the July 2025 Qwen3-30B-A3B, where SP-MoE is fastest at every tested sparsity.

Dear Reviewer xZMX,

Thank you again for your review. As the discussion period closes soon, we would like to check whether our response addresses your concerns.

In response to your main points, we added a quality and offloading-throughput comparison with D²-MoE, MoBE, and REAP on the newer Qwen3-30B-A3B (128 experts, top-8). SP-MoE is the fastest method that fits the 8 GiB budget at every tested sparsity, and the most accurate at 37.5% and 50%. On Eq. 4–5, we tested the criterion against the measured replacement error on every layer of OLMoE and Mixtral. We also built a variant that weights dominant tokens most, and it changes accuracy by at most 0.6 points. The code is now available at the anonymous link in Part 2.

If any point remains unclear, or you would like additional details, we would be glad to respond before the deadline. If our response resolves your concerns, we would be grateful if you would consider it in your final assessment.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.