- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem of compressing large Mixture-of-Experts (MoE) models and proposes GS-MoE, which partitions experts into groups and constructs shared subspaces. Building upon factorization and low-rank projection, the method further reduces the intermediate width of experts, thereby decreasing both the parameter count and the intermediate layer dimensions. The main contribution lies in integrating group-shared subspace factorization and Nyström-based width reduction into a unified MoE compression pipeline, together with the construction and derivation of the corresponding projection matrices. For validation, the paper compares the original models, factorization-only compression, Nyström-only compression, and the complete GS-MoE method on DeepSeekMoE-16B-Base and Qwen models. The evaluation includes perplexity on Wiki and PTB, as well as performance on OpenB, ARC-e, WinoG, HellaS, ARC-c, PIQA, and MathQA. The results show that GS-MoE generally outperforms factorization-only compression and achieves performance close to that of width reduction alone, suggesting that the proposed method can effectively compress MoE architectures with relatively limited performance degradation.
Critical strengths
The most critical strength of this paper lies in its proposal of a unified compression framework for Mixture-of-Experts (MoE) models. The framework compresses shared representations through group-shared subspace and rank reduction, while also reducing the intermediate width of individual experts. Furthermore, it leverages calibration statistics to allocate compression budgets at the layer, group, and expert levels without requiring any post-compression training.
Critical weaknesses
The full GS-MoE method does not consistently demonstrate clear advantages over simpler compression approaches. For example, in Table 6, under a 20% compression ratio on DeepSeekMoE-16B-Base, GS-MoE achieves an average score of 0.552, which is lower than the 0.558 achieved by the Nyström-only width reduction method. Its Wiki and PTB perplexity results are also slightly worse than those of Nyström. These results suggest that the additional benefits of incorporating factorization and shared subspaces are not sufficiently clear. The core allocation of groups, ranks, and widths relies on calibration surrogates and a heuristic multiplier search. As the paper explicitly acknowledges, this approach is primarily motivated by continuous resource allocation and does not guarantee optimality in the resulting discrete allocation.
Questions for authors
- In the subset comparisons, GS-MoE does not always clearly outperform the Nyström-only or width-reduction approaches. Could the authors clarify under which compression ratios, model architectures, and calibration datasets the full pipeline consistently provides additional benefits? How sensitive are these improvements to settings such as βscale, rmin/rmax, and calibration set size?
- Does the bisection and multiplier search used for discrete group-rank allocation provide any theoretical guarantees of approximate optimality with respect to the objective function? Alternatively, could the authors provide a sensitivity analysis to assess the stability of the allocation strategy?
6 comments
Rebuttals 2 / 6
Q.1 Part B How sensitive are these improvements to settings such as βscale, rmin/rmax, and calibration set size?
Calibration set size. GS-MoE degrades gradually as the calibration set shrinks (Table R3). Mean perplexity and average accuracy both improve monotonically with more samples. Cutting from 512 to 256 samples raises mean perplexity by 0.15 and lowers accuracy by 0.4 pts; cutting to 128 raises it by 0.38 and lowers accuracy by 0.6 pts. Even with 128 samples, GS-MoE stays ahead of Nyström-only calibrated on the full 512 (mean perplexity 20.87 vs. 24.28, accuracy 0.497 vs. 0.486, Table 1).
| Samples | Wiki. ↓ | C4 ↓ | PTB ↓ | Mean ↓ | Avg. ↑ |
|---|---|---|---|---|---|
| 128 | 17.57 | 22.30 | 22.74 | 20.87 | 0.497 |
| 256 | 17.34 | 22.07 | 22.51 | 20.64 | 0.499 |
| 512 (default) | 17.55 | 21.71 | 22.21 | 20.49 | 0.503 |
Table R3. Calibration-size sensitivity, DeepSeekMoE-16B-Base, 60% compression, Alpaca samples of 2048 tokens. Mean: WikiText/C4/PTB.
Group Rank Bounds. The bounds rarely constrain group size. In the full pipeline at our 60% setting on DeepSeek, none of the 216 groups sits at or ; the closest group is one rank above , and every group is at least 303 ranks below . Because is concave, relaxing a bound that no group touches leaves the multiplier solution unchanged. Tightening does bind at and would raise 12 groups. Table R4.
| r_min | r_max | Groups raised | Wiki. ↓ | C4 ↓ | PTB ↓ | Mean ↓ |
|---|---|---|---|---|---|---|
| 0.35 r_ℓ (default) | 2.5 r_ℓ (default) | – | 17.48 | 21.96 | 22.18 | 20.54 |
| 0.01 r_ℓ (none) | d (none) | 0 | 17.48 | 21.96 | 22.17 | 20.54 |
| 0.50 r_ℓ | d (none) | 12 | 17.56 | 21.99 | 22.17 | 20.57 |
| 0.70 r_ℓ | d (none) | 26 | 17.51 | 22.05 | 22.20 | 20.59 |
Table R4. Rank-bound sensitivity, DeepSeekMoE-16B-Base, 60% compression.
Beta scale. Generally, our method does not employ a tunable parameter; instead, is the fraction of up pre-activation energy retained by neuron selection, computed from calibration statistics. To test DeepSeek-MoE's sensitivity to , we scale this by a multiplier ranging from 0.7 to 1.7. Scaling it by 0.7 to 1.7 changes mean perplexity by at most 0.28, and every setting remains well below Nyström-only compression (24.28) (Table R5).
| β_scale multiplier | 0.7 | 0.85 | 1.0 | 1.15 | 1.3 | 1.7 |
|---|---|---|---|---|---|---|
| Wiki. ↓ | 18.12 | 17.36 | 17.55 | 17.67 | 17.42 | 17.91 |
| C4 ↓ | 21.76 | 21.87 | 21.71 | 21.92 | 21.80 | 21.98 |
| PTB ↓ | 22.45 | 22.28 | 22.21 | 22.40 | 22.20 | 22.35 |
| Mean ↓ | 20.77 | 20.50 | 20.49 | 20.66 | 20.47 | 20.75 |
Table R5. sensitivity, DeepSeekMoE-16B-Base, 60% compression (, ); 1.0 is the default.
Rebuttals 3 / 6
Q.2 Part A Does the bisection and multiplier search used for discrete group-rank allocation provide any theoretical guarantees of approximate optimality with respect to the objective function?
Optimality. When each group contains the same number of experts and all experts have the same intermediate width, increasing any group’s rank by one requires the same number of additional parameters. Since the improvement from each additional rank is non-increasing, repeatedly selecting the largest available improvement is globally optimal [1]. Our multiplier search with greedy completion implements this allocation rule. With width pruning, groups may retain different total widths, making the parameter cost of each additional rank unequal across groups. Our procedure is then approximate, and we evaluate its optimality gap against globally optimal integer solutions. We have added a paragraph to Appendix A.5 that establishes the equal-cost optimality guarantee using [1] and provides a computable upper bound on the objective gap for unequal costs.
We also solve Obj. 10 to global optimality using an integer-programming solver and compare the resulting optimal allocations with those produced by our method. On DeepSeekMoE-16B-Base at 60% compression, factorization-only allocation achieves zero objective gap and matches the exact solver's ranks in all 27 layers. For full GS-MoE (), the relative objective gap is at most in every layer, and the allocations differ by at most three ranks per group. These results support the accuracy of our solver for Obj. 10 in the evaluated setting.
[1] Federgruen, Awi, and Henri Groenevelt. "The greedy procedure for resource allocation problems: Necessary and sufficient conditions for optimality." Operations Research 34.6 (1986): 909-918.
Rebuttals 1 / 6
We thank the reviewer for their constructive feedbacks. Our responses are provided below.
Compression ratio and architecture. Whether the full pipeline helps at a given compression ratio and architecture is best predicted by the expert width dff relative to the model dimension d. Let kN denote the average number of neurons per expert that width-only pruning retains at a given compression ratio. While kN≥d, width pruning does not reduce the rank of the expert projections, so shared subspaces add little; once kN<d, pruning also removes rank, and sharing subspaces across experts lets GS-MoE retain more neurons under the same budget. Fine-grained DeepSeek and Qwen (dff≈0.7d) are below d from the outset, and GS-MoE's gain over width-only pruning grows with compression, from within 0.6 percentage points at mild ratios to +2.5 (DeepSeek) and +2.0 (Qwen) percentage points at 80% (Table R1).
For coarse-grained Mixtral (dff=3.5d), kN falls below d only beyond roughly 69% compression; GS-MoE matches width-only pruning through 60% and improves average accuracy by +1.5 and +1.8 percentage points at 70% and 80%. At 60% and 80% on DeepSeek, these gains hold across three calibration seeds, whose variation (at most 0.2 percentage points) is well below the gains (Table 4). The full derivation is given in our general response and App. C.3.
Table R1. Width retained per expert relative to d by pure width pruning (kN/d) and GS-MoE's gain over pure width pruning (ΔAvg., pts).
Calibration dataset. Calibrating on a perplexity corpus lowers perplexity on that corpus (WikiText 7.82, C4 15.19) but reduces average accuracy by 2.2 to 5.4 pts relative to Alpaca, the instruction data we use throughout (Table R2). This also explains D²-MoE's lower WikiText perplexity, since it calibrates on WikiText.
Table R2. GS-MoE on DeepSeekMoE-16B-Base at 60% compression (G=8, ν=0.55) calibrated on 512 samples from each dataset. Avg. ↑ is the average benchmark accuracy.