acceptodds
Under review as a conference paper at ICLR 2027

PaCES: Pattern-Coded Element Sharing for Lightweight Mixture-of-Experts Inference

Abstract

Sparsely activated Mixture-of-Experts (MoE) models improve computational efficiency via token-wise sparse activation, but still require full dense parameter storage, severely limiting their practical deployment. Existing MoE compression approaches fail to achieve a favorable trade-off between model accuracy and storage overhead. In this work, we propose PaCES (Pattern-Coded Element Sharing), a post-training compression method for MoE language models. Based on the key observation that intra-layer experts exhibit consistent Gaussian weight distributions, PaCES groups similar experts and performs distribution-aware element-wise magnitude sharing. It compactly encodes expert participation and sign information using lightweight bit-level codes. Equipped with a fused GPU kernel that avoids dense weight reconstruction during inference, PaCES enables true end-to-end storage and latency optimization. Evaluated on Qwen-MoE and DeepSeek-V2-Lite over eight zero-shot benchmarks, our method reduces checkpoint storage by 32.6%–57.3% and achieves up to 4.05x inference speedup with less accuracy degradation. It consistently outperforms unstructured pruning and expert merging baselines under aggressive compression, providing a practical deployment-friendly compact representation for real-world MoE serving.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Model Compression, Element-wise Weight Sharing, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gtKvOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PaCES is a post-training MoE compression and execution framework. Experts are grouped using routing frequency and router-row cosine similarity. For every matching weight coordinate in a group, the method stores one shared BF16 magnitude and a ternary state per expert indicating positive participation, negative participation, or pruning. A Triton kernel decodes these pattern codes during matrix multiplication. Experiments on Qwen1.5-MoE-A2.7B and DeepSeek-V2-Lite report reduced checkpoint size, better accuracy than expert deletion or whole-expert merging at nominally similar compression, and end-to-end acceleration on an RTX 4090.

Critical strengths
  1. The paper treats representation and execution as one system. PaCES is implemented with a fused Triton kernel instead of being evaluated only through nominal storage or sparsity. The hardware and workload details make the systems result concrete and useful.

  2. The paper reports actual checkpoint sizes and tests more than one MoE regime. This is more useful than nominal sparsity alone. The method also preserves the logical expert interface without retraining, and the Qwen and DeepSeek results reveal meaningful differences between execution regimes.

Critical weaknesses
  1. I found the novelty claim and the motivation for the selection rule unclear. PuzzleMoE already combines element-wise expert merging, per-expert participation and sign information, bit-packed storage, and fused decode–GEMM execution. PaCES differs in router-guided grouping, the mean-plus/minus-standard-deviation support rule, groups larger than two, its packed layout, and distance-weighted shared magnitudes. However, the paper does not cite PuzzleMoE or show that these differences lead to a clear empirical advantage on the shared Qwen1.5-MoE-A2.7B setting. The distribution plots also do not show that central coordinates are functionally redundant or that outliers capture specialization, and there is no ablation of the threshold or of Equation (6) under fixed support.

  2. I am not yet convinced by the claimed quality–storage–latency advantage. The quality–storage plot mixes measured PaCES checkpoint files with BF16 payload estimates for some baselines, while sparse baselines stored in dense format are counted as having no actual saving. The resulting frontier therefore depends heavily on the chosen accounting. Table 3 compares PaCES with a path that explicitly reconstructs dense weights on every forward pass, but it does not report raw native-MoE timing, and the measurements use full-sequence forward passes rather than autoregressive decoding. The baseline set also misses the expert-channel-level structured pruning methods HEAPr (ICLR 2026) and CAMERA-P (AAAI 2026). A direct comparison would make it much easier to judge PaCES on both effectiveness and efficiency.

Questions for authors
  1. What are the concrete differences from PuzzleMoE in support selection, shared-magnitude construction, expert grouping, packed representation, and fused execution? On the shared Qwen1.5-MoE-A2.7B checkpoint and the same evaluation harness, how do the two methods compare at a matched actual checkpoint or model-memory footprint?

  2. What evidence shows that weights inside the mean-plus/minus-standard-deviation interval are functionally redundant? How sensitive are the results to this threshold, and how does Equation (6) compare with a least-squares shared magnitude when the support is fixed?

  3. Could the authors separate the two latency claims: avoiding per-forward dense reconstruction and outperforming native MoE execution? For the current sequence-length settings, what are the raw native-MoE and integrated-PaCES latencies? What deployment setting is represented by the explicit-reconstruction baseline, and what happens during prefill and token-by-token decoding?

  4. How does PaCES compare in effectiveness and efficiency with the expert-channel-level structured pruning methods HEAPr and CAMERA-P under comparable settings?

Reviewer_mk24OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper addresses the challenge of reducing the substantial storage overhead of MoE models while preserving model accuracy and inference efficiency. The authors propose PACES, a retraining-free compression framework that combines router-guided expert grouping, distribution-aware element sharing, and fused GPU kernels for efficient inference. Experiments on Qwen1.5-MoE and DeepSeek-V2-Lite across eight zero-shot benchmarks demonstrate 32.6%–57.3% checkpoint storage reduction.

Critical strengths
  1. The proposed router-guided expert grouping and distribution-aware element sharing effectively exploit statistical redundancy across experts while preserving expert-specific characteristics and the original routing structure.

  2. The compact pattern-coded representation, combined with a fused GPU kernel, enables direct computation on compressed weights without dense reconstruction, providing practical storage savings and inference acceleration (potentially, see my weaknesses and questions below).

Critical weaknesses

Lack of end-to-end inference performance data, which should includes at least two experiments: 1) Experiments regarding end-to-end serving performance, with throughput and latency as metrics. 2) Experiments regarding if the proposed fused kernel is compatible with common optimiazation such as CUDA Graph

Questions for authors

In Table 3 title, it says "Full-model forward latency", but in the text (Line 422) it says "execution efficiency of our integrated fused kernel". Authors needs to clarify what does the figure means.

Reviewer_qZV5OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to compress an MoE model. The proposed approach groups experts based on their routing similarity, allowing experts with similar routing decisions and magnitudes to be merged. This reduces the number of experts and, consequently, the overall size of the MoE model. Experiments with several small and slightly outdated MoE models show that some of the resulting accuracy loss can be recovered on a few datasets.

Critical strengths

S1: Addressed the MoE compression problem, which has significant practical value.

S2: The proposed method was compared with several straightforward baselines, including expert merging and unstructured pruning.

Critical weaknesses

W1: The merging algorithm is heuristic and training-free, making it difficult to adapt and potentially difficult to deploy widely, as adaptation could affect accuracy and there is no universal way to recover the lost accuracy.

W2: The selected MoE models will not be representative in late 2026 and will probably not interest the ICLR 2027 audience.

W3: The selected dataset is limited. For MMLU, accuracy could drop significantly.

W4: The selected baselines omit some obvious ones, such as expert sharing and quantization.

Questions for authors

Q1: Expert sharing and quantization are much more suitable alternatives. Can you compare them with the most recent approaches?

Q2: What about more recent MoE models?

Q3: How can the accuracy be recovered? Can you show a compelling case?

Comments on the paper

1 comment

Thank you for the detailed rebuttal and additional experiments. The authors have addressed most of my concerns by:

  1. Clarifying the differences from PuzzleMoE and providing direct comparisons.
  2. Providing additional analyses and ablation studies to justify the design choices.
  3. Reporting actual storage costs and additional inference efficiency measurements.
  4. Including comparisons with expert-channel pruning baselines.

If these results and clarifications, particularly the discussion and comparisons with PuzzleMoE given its close relationship to the proposed method, are properly incorporated into the revised manuscript, I will positively reconsider my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.