- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PaCES is a post-training MoE compression and execution framework. Experts are grouped using routing frequency and router-row cosine similarity. For every matching weight coordinate in a group, the method stores one shared BF16 magnitude and a ternary state per expert indicating positive participation, negative participation, or pruning. A Triton kernel decodes these pattern codes during matrix multiplication. Experiments on Qwen1.5-MoE-A2.7B and DeepSeek-V2-Lite report reduced checkpoint size, better accuracy than expert deletion or whole-expert merging at nominally similar compression, and end-to-end acceleration on an RTX 4090.
Critical strengths
-
The paper treats representation and execution as one system. PaCES is implemented with a fused Triton kernel instead of being evaluated only through nominal storage or sparsity. The hardware and workload details make the systems result concrete and useful.
-
The paper reports actual checkpoint sizes and tests more than one MoE regime. This is more useful than nominal sparsity alone. The method also preserves the logical expert interface without retraining, and the Qwen and DeepSeek results reveal meaningful differences between execution regimes.
Critical weaknesses
-
I found the novelty claim and the motivation for the selection rule unclear. PuzzleMoE already combines element-wise expert merging, per-expert participation and sign information, bit-packed storage, and fused decode–GEMM execution. PaCES differs in router-guided grouping, the mean-plus/minus-standard-deviation support rule, groups larger than two, its packed layout, and distance-weighted shared magnitudes. However, the paper does not cite PuzzleMoE or show that these differences lead to a clear empirical advantage on the shared Qwen1.5-MoE-A2.7B setting. The distribution plots also do not show that central coordinates are functionally redundant or that outliers capture specialization, and there is no ablation of the threshold or of Equation (6) under fixed support.
-
I am not yet convinced by the claimed quality–storage–latency advantage. The quality–storage plot mixes measured PaCES checkpoint files with BF16 payload estimates for some baselines, while sparse baselines stored in dense format are counted as having no actual saving. The resulting frontier therefore depends heavily on the chosen accounting. Table 3 compares PaCES with a path that explicitly reconstructs dense weights on every forward pass, but it does not report raw native-MoE timing, and the measurements use full-sequence forward passes rather than autoregressive decoding. The baseline set also misses the expert-channel-level structured pruning methods HEAPr (ICLR 2026) and CAMERA-P (AAAI 2026). A direct comparison would make it much easier to judge PaCES on both effectiveness and efficiency.
Questions for authors
-
What are the concrete differences from PuzzleMoE in support selection, shared-magnitude construction, expert grouping, packed representation, and fused execution? On the shared Qwen1.5-MoE-A2.7B checkpoint and the same evaluation harness, how do the two methods compare at a matched actual checkpoint or model-memory footprint?
-
What evidence shows that weights inside the mean-plus/minus-standard-deviation interval are functionally redundant? How sensitive are the results to this threshold, and how does Equation (6) compare with a least-squares shared magnitude when the support is fixed?
-
Could the authors separate the two latency claims: avoiding per-forward dense reconstruction and outperforming native MoE execution? For the current sequence-length settings, what are the raw native-MoE and integrated-PaCES latencies? What deployment setting is represented by the explicit-reconstruction baseline, and what happens during prefill and token-by-token decoding?
-
How does PaCES compare in effectiveness and efficiency with the expert-channel-level structured pruning methods HEAPr and CAMERA-P under comparable settings?
Thank you for the detailed rebuttal and additional experiments. The authors have addressed most of my concerns by:
If these results and clarifications, particularly the discussion and comparisons with PuzzleMoE given its close relationship to the proposed method, are properly incorporated into the revised manuscript, I will positively reconsider my rating.