acceptodds
Under review as a conference paper at ICLR 2027

GS-MoE: Training-Free MoE Compression via Groupwise Subspace Sharing and Pruning

Abstract

Mixture-of-Experts (MoE) models reduce computation through sparse activation, but storing their expert parameters remains a major memory bottleneck. We introduce GS-MoE, a training-free compression framework that addresses redundancy across and within experts through two compression axes: shared-subspace rank and expert intermediate width. Groupwise low-rank representations capture shared structure across experts while retaining expert-specific projections, and Nystr\"om-based neuron compression reduces intermediate width within each expert. For fixed shared subspaces, we derive per-expert retained-energy gains from the reconstruction objective and use these gains to guide expert grouping. A gradient-free layer-importance score guides the allocation of rank and width budgets across layers, with non-uniform ranks across groups and intermediate widths across experts. Experiments show that compression along both axes achieves higher accuracy than compression along either axis alone. On DeepSeek-MoE-16B-Base, at a 60% reduction in routed-expert parameters, GS-MoE retains over 92% of the uncompressed model's average reasoning performance, compared with 73% for D-MoE, while achieving 30% lower out-of-domain perplexity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, Model Compression, Structured Pruning, Factorization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cbWNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of compressing large Mixture-of-Experts (MoE) models and proposes GS-MoE, which partitions experts into groups and constructs shared subspaces. Building upon factorization and low-rank projection, the method further reduces the intermediate width of experts, thereby decreasing both the parameter count and the intermediate layer dimensions. The main contribution lies in integrating group-shared subspace factorization and Nyström-based width reduction into a unified MoE compression pipeline, together with the construction and derivation of the corresponding projection matrices. For validation, the paper compares the original models, factorization-only compression, Nyström-only compression, and the complete GS-MoE method on DeepSeekMoE-16B-Base and Qwen models. The evaluation includes perplexity on Wiki and PTB, as well as performance on OpenB, ARC-e, WinoG, HellaS, ARC-c, PIQA, and MathQA. The results show that GS-MoE generally outperforms factorization-only compression and achieves performance close to that of width reduction alone, suggesting that the proposed method can effectively compress MoE architectures with relatively limited performance degradation.

Critical strengths

The most critical strength of this paper lies in its proposal of a unified compression framework for Mixture-of-Experts (MoE) models. The framework compresses shared representations through group-shared subspace and rank reduction, while also reducing the intermediate width of individual experts. Furthermore, it leverages calibration statistics to allocate compression budgets at the layer, group, and expert levels without requiring any post-compression training.

Critical weaknesses

The full GS-MoE method does not consistently demonstrate clear advantages over simpler compression approaches. For example, in Table 6, under a 20% compression ratio on DeepSeekMoE-16B-Base, GS-MoE achieves an average score of 0.552, which is lower than the 0.558 achieved by the Nyström-only width reduction method. Its Wiki and PTB perplexity results are also slightly worse than those of Nyström. These results suggest that the additional benefits of incorporating factorization and shared subspaces are not sufficiently clear. The core allocation of groups, ranks, and widths relies on calibration surrogates and a heuristic multiplier search. As the paper explicitly acknowledges, this approach is primarily motivated by continuous resource allocation and does not guarantee optimality in the resulting discrete allocation.

Questions for authors
  1. In the subset comparisons, GS-MoE does not always clearly outperform the Nyström-only or width-reduction approaches. Could the authors clarify under which compression ratios, model architectures, and calibration datasets the full pipeline consistently provides additional benefits? How sensitive are these improvements to settings such as βscale, rmin/rmax, and calibration set size?
  2. Does the bisection and multiplier search used for discrete group-rank allocation provide any theoretical guarantees of approximate optimality with respect to the objective function? Alternatively, could the authors provide a sensitivity analysis to assess the stability of the allocation strategy?
6 comments
AuthorsOpenReview ↗

Rebuttals 1 / 6

We thank the reviewer for their constructive feedbacks. Our responses are provided below.

Q.1 Part A In the subset comparisons, GS-MoE does not always clearly outperform the Nyström-only or width-reduction approaches. Could the authors clarify under which compression ratios, model architectures, and calibration datasets the full pipeline consistently provides additional benefits?

Compression ratio and architecture. Whether the full pipeline helps at a given compression ratio and architecture is best predicted by the expert width dffd_{ff} relative to the model dimension dd. Let kNk_N denote the average number of neurons per expert that width-only pruning retains at a given compression ratio. While kN≥dk_N\ge d, width pruning does not reduce the rank of the expert projections, so shared subspaces add little; once kN<dk_N<d, pruning also removes rank, and sharing subspaces across experts lets GS-MoE retain more neurons under the same budget. Fine-grained DeepSeek and Qwen (dff≈0.7dd_{ff}\approx0.7d) are below dd from the outset, and GS-MoE's gain over width-only pruning grows with compression, from within 0.6 percentage points at mild ratios to +2.5+2.5 (DeepSeek) and +2.0+2.0 (Qwen) percentage points at 80% (Table R1).

For coarse-grained Mixtral (dff=3.5dd_{ff}=3.5d), kNk_N falls below dd only beyond roughly 69% compression; GS-MoE matches width-only pruning through 60% and improves average accuracy by +1.5+1.5 and +1.8+1.8 percentage points at 70% and 80%. At 60% and 80% on DeepSeek, these gains hold across three calibration seeds, whose variation (at most 0.2 percentage points) is well below the gains (Table 4). The full derivation is given in our general response and App. C.3.

Table R1. Width retained per expert relative to dd by pure width pruning (kN/dk_N/d) and GS-MoE's gain over pure width pruning (ΔAvg., pts).

ModelRatiok_N/dΔAvg. ↑
DeepSeekMoE-16B-Base (E=64, d_ff/d=0.69)20%0.54−0.6
40%0.39+0.4
60%0.24+1.6
80%0.08+2.5
Qwen2-57B-A14B (E=64, d_ff/d=0.71)40%0.38−0.1
60%0.22+0.8
80%0.05+2.0
Mixtral-8×7B (E=8, d_ff/d=3.5)20%2.78−0.2
40%2.05−0.2
60%1.330.0
70%0.96+1.5
80%0.60+1.8

Calibration dataset. Calibrating on a perplexity corpus lowers perplexity on that corpus (WikiText 7.82, C4 15.19) but reduces average accuracy by 2.2 to 5.4 pts relative to Alpaca, the instruction data we use throughout (Table R2). This also explains D²-MoE's lower WikiText perplexity, since it calibrates on WikiText.

Calibration setWiki. ↓C4 ↓PTB ↓Avg. ↑
Alpaca (default)17.5521.7122.210.503
WikiText7.8220.4117.050.449
C413.5715.1916.130.481

Table R2. GS-MoE on DeepSeekMoE-16B-Base at 60% compression (G=8G{=}8, ν=0.55\nu{=}0.55) calibrated on 512 samples from each dataset. Avg. ↑ is the average benchmark accuracy.

AuthorsOpenReview ↗

Rebuttals 2 / 6

Q.1 Part B How sensitive are these improvements to settings such as βscale, rmin/rmax, and calibration set size?

Calibration set size. GS-MoE degrades gradually as the calibration set shrinks (Table R3). Mean perplexity and average accuracy both improve monotonically with more samples. Cutting from 512 to 256 samples raises mean perplexity by 0.15 and lowers accuracy by 0.4 pts; cutting to 128 raises it by 0.38 and lowers accuracy by 0.6 pts. Even with 128 samples, GS-MoE stays ahead of Nyström-only calibrated on the full 512 (mean perplexity 20.87 vs. 24.28, accuracy 0.497 vs. 0.486, Table 1).

SamplesWiki. ↓C4 ↓PTB ↓Mean ↓Avg. ↑
12817.5722.3022.7420.870.497
25617.3422.0722.5120.640.499
512 (default)17.5521.7122.2120.490.503

Table R3. Calibration-size sensitivity, DeepSeekMoE-16B-Base, 60% compression, Alpaca samples of 2048 tokens. Mean: WikiText/C4/PTB.

Group Rank Bounds. The bounds rarely constrain group size. In the full pipeline at our 60% setting on DeepSeek, none of the 216 groups sits at rminr_{min} or rmaxr_{max}; the closest group is one rank above rminr_{min}, and every group is at least 303 ranks below rmaxr_{max}. Because VgV_g is concave, relaxing a bound that no group touches leaves the multiplier solution unchanged. Tightening does bind at rmin=0.5,rℓr_{min} = 0.5\\,r_{\ell} and would raise 12 groups. Table R4.

r_minr_maxGroups raisedWiki. ↓C4 ↓PTB ↓Mean ↓
0.35 r_ℓ (default)2.5 r_ℓ (default)–17.4821.9622.1820.54
0.01 r_ℓ (none)d (none)017.4821.9622.1720.54
0.50 r_ℓd (none)1217.5621.9922.1720.57
0.70 r_ℓd (none)2617.5122.0522.2020.59

Table R4. Rank-bound sensitivity, DeepSeekMoE-16B-Base, 60% compression.

Beta scale. Generally, our method does not employ a tunable βscale\beta_{scale} parameter; instead, βscale\beta_{scale} is the fraction of up pre-activation energy retained by neuron selection, computed from calibration statistics. To test DeepSeek-MoE's sensitivity to βscale\beta_{scale}, we scale this by a multiplier ranging from 0.7 to 1.7. Scaling it by 0.7 to 1.7 changes mean perplexity by at most 0.28, and every setting remains well below Nyström-only compression (24.28) (Table R5).

β_scale multiplier0.70.851.01.151.31.7
Wiki. ↓18.1217.3617.5517.6717.4217.91
C4 ↓21.7621.8721.7121.9221.8021.98
PTB ↓22.4522.2822.2122.4022.2022.35
Mean ↓20.7720.5020.4920.6620.4720.75

Table R5. βscale\beta_{scale} sensitivity, DeepSeekMoE-16B-Base, 60% compression (G=8G{=}8, ν=0.55\nu{=}0.55); 1.0 is the default.

AuthorsOpenReview ↗

Rebuttals 3 / 6

Q.2 Part A Does the bisection and multiplier search used for discrete group-rank allocation provide any theoretical guarantees of approximate optimality with respect to the objective function?

Optimality. When each group contains the same number of experts and all experts have the same intermediate width, increasing any group’s rank by one requires the same number of additional parameters. Since the improvement from each additional rank is non-increasing, repeatedly selecting the largest available improvement is globally optimal [1]. Our multiplier search with greedy completion implements this allocation rule. With width pruning, groups may retain different total widths, making the parameter cost of each additional rank unequal across groups. Our procedure is then approximate, and we evaluate its optimality gap against globally optimal integer solutions. We have added a paragraph to Appendix A.5 that establishes the equal-cost optimality guarantee using [1] and provides a computable upper bound on the objective gap for unequal costs.

We also solve Obj. 10 to global optimality using an integer-programming solver and compare the resulting optimal allocations with those produced by our method. On DeepSeekMoE-16B-Base at 60% compression, factorization-only allocation achieves zero objective gap and matches the exact solver's ranks in all 27 layers. For full GS-MoE (ν=0.55\nu=0.55), the relative objective gap is at most 9×10−69\times10^{-6} in every layer, and the allocations differ by at most three ranks per group. These results support the accuracy of our solver for Obj. 10 in the evaluated setting.

[1] Federgruen, Awi, and Henri Groenevelt. "The greedy procedure for resource allocation problems: Necessary and sufficient conditions for optimality." Operations Research 34.6 (1986): 909-918.

Reviewer_LQjhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GS-MoE compresses MoE experts through group-shared low-rank projections and Nyström-based intermediate-width reduction. Calibration statistics guide expert grouping and nonuniform allocation of layer budgets, group ranks, and expert widths, without recovery training. Experiments cover DeepSeek-MoE-16B, Qwen2-57B-A14B, and Mixtral-8x7B, with language-modeling, zero-shot accuracy, and throughput evaluations. The strongest gains occur on fine-grained models at high compression.

Critical strengths
  1. The formulation provides a coherent way to control shared representation capacity and expert width separately. Reconstruction-based grouping and explicit parameter accounting make this more principled than simply applying a uniform factorization; the limitations of the up-projection surrogate and post-pruning reconstruction are clearly acknowledged (Sections 3.3–3.5; Appendix A).

  2. The fine-grained-model results show useful compression without recovery training. At 60% compression on DeepSeek, average task accuracy is 0.50 versus D2-MoE's 0.41, and PTB perplexity improves from 27.73 to 22.21 (Table 1). The factorization ablation also supports the grouping mechanism: guided assignment reduces WikiText perplexity from 29.49 to 25.23 (Table 2).

Critical weaknesses
  1. The added value of the complete framework is less established than the headline comparisons suggest. In Table 6, width-only Nyström compression nearly matches the full method in most settings: at 60% compression, Qwen averages 0.586 versus 0.594, and Mixtral reaches 0.502 with either method. DeepSeek shows meaningful complementarity, especially on PTB, but the guided-grouping ablation tests only factorization without width reduction. Thus, its contribution to the final combined system remains insufficiently isolated.

  2. The comparative evaluation omits relevant alternatives. TENP, a structured expert-neuron pruning method discussed in Section 2.1, is absent from the experiments; Table 1 also lacks D2-MoE at 60% compression on Qwen. These gaps limit the evidence for the claimed advantage over existing compression methods, particularly at the aggressive budgets emphasized by the paper.

Questions for authors

How were the model-specific values of G and ν in Table 4 selected, what validation data were used, and what are the layer-allocation temperatures ε_w and ε_r? A reproducible selection protocol would help determine whether the reported configurations transfer beyond the evaluated datasets.

Reviewer_SDYbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

GS-MoE proposes a post-training, training-free compression framework that jointly compresses parameters along two distinct structural axes: cross-expert groupwise shared-subspace rank and intra-expert intermediate neuron width. Instead of constraining all layer experts to a single shared subspace or relying solely on neuron pruning, GS-MoE clusters experts into balanced groups sharing input/output projection bases while preserving expert-specific mappings. It uses Nyström approximation to compress intermediate widths and derives closed-form error reductions to guide clustering and hierarchical capacity allocation. Mathematically, it derives closed-form reconstruction-error reductions that formulate an error-guided criterion for balanced medoid expert clustering. Furthermore, it establishes a three-tier capacity allocation scheme: layer-wise budgets via normalized gate-weighted output norms, group ranks via eigenvalue distributions, and expert widths via ridge-leverage scores. Empirically, GS-MoE demonstrates superior fidelity on fine-grained MoE models (DeepSeek-MoE-16B, Qwen2-57B-A14B), retaining 90% reasoning accuracy and superior perplexity at 60% compression where prior baselines collapse.

Critical strengths
  • GS-MoE couples group-shared subspaces with intra-expert intermediate width pruning. This parameterization explicitly decouples shared representation from individual expert transformations. This structural design enables the method to preserve high capacity and model fidelity under aggressive parameter budgets up to 60%.
  • GS-MoE directly derives the pairwise expert dissimilarity matrix from the linear reconstruction objective. The balanced medoid clustering with exact Jonker-Volgenant assignment ensures that assigned groups maximize collective subspace alignment under strict equal-size constraints. This grouping mechanism directly mitigates the throughput degradation of sparse routing by reducing the number of distinct active subspaces activated per token by approximately 19%.
  • The framework introduces a mathematically consistent, gradient-free resource allocation scheme across layers, groups, and individual experts. This systematic capacity assignment prevents arbitrary parameter allocation, achieving empirical performance gains over uniform baselines without requiring backpropagation, fine-tuning, or extensive search.
Critical weaknesses
  • The empirical benefits of GS-MoE heavily depend on model architecture and fine-grained granularity. On coarse-grained models like Mixtral-8x7B (8 experts, large intermediate dimension), GS-MoE performs worse than D2-MoE at 60% compression (Avg. 0.50 vs 0.52; WikiText PPL 15.94 vs 6.46), and the full pipeline fails to outperform pure width pruning. The authors should explicitly characterize the architectural failure boundaries.
  • The optimization steps are locally decoupled approximations. Once up-projections are compressed, actual intermediate hidden activations change, invalidating the optimality conditions of the down-projection and neuron leverage scores.
  • The paper misses a direct empirical comparison with MoLAE, which shares the closest conceptual formulation of group-shared latent projections. Furthermore, runtime benchmarks on B200 GPUs show that increasing group count G from 1 to 8 decreases throughput by 16.2% on DeepSeek-MoE, without benchmarking the actual generation speed of uncompressed models or isolating wall-clock trade-offs at identical perplexity.
Questions for authors
  • According to the ablation data in Table 6, at 60% compression on DeepSeek-MoE, full GS-MoE yields only a modest +1.6 percentage point gain in average reasoning accuracy over pure width pruning (Nyström), while on Qwen2 the gain is only +0.8, and on Mixtral it provides 0.0 gain. Given that groupwise subspace factorization introduces significant mathematical complexity, surrogate approximations, and runtime kernel overhead, can you demonstrate that this marginal gain is statistically significant across multiple calibration seeds, and why should practitioners adopt the full pipeline rather than pure width pruning?
  • Suggesting adding a sensitivity boundary analysis regarding expert count E versus intermediate width d_ff, providing guidelines on when subspace factorization ceases to provide gains over width pruning.
  • GS-MoE lacks global optimality guarantees because local reconstruction ignores router re-weighting and error interactions between compressed experts. Moreover, the down-projection assumes static intermediate activations h_e that are altered by compressed up-projections. How severe is this activation shift in practice?
  • MoLAE shares a nearly identical reparameterization using group-shared latent bases. Why was MoLAE omitted from the comparative evaluation in Table 1?
  • Table 3 demonstrate that increasing group count G from 1 to 8 on DeepSeek-MoE-16B reduces inference throughput by 16.2% (from 239.06 to 200.22 tokens/s) due to GPU kernel fragmentation and scattered memory access. However, Figure 4a shows that achieving lower perplexity strictly requires larger G(G=8). How should practitioners choose G to achieve a favorable wall-clock speedup while maintaining model quality?
6 comments
AuthorsOpenReview ↗

Rebuttals 1

We thank the reviewer for their constructive feedbacks. Our responses are provided below.

Q.1 Part A Can you demonstrate that this marginal gain is statistically significant across multiple calibration seeds, and why should practitioners adopt the full pipeline rather than pure width pruning?

We provide the requested ablation reporting three varied seeds over averaged perplexity (Wikitext/C4/PTB) and average benchmark accuracy at 60% and 80% compression ratios for DeepSeek-MoE.

Table R1. Sensitivity to calibration seed on DeepSeekMoE-16B-Base.

CompressionMethodSeed 42 PPL ↓Seed 42 Avg. ↑Seed 200 PPL ↓Seed 200 Avg. ↑Seed 1234 PPL ↓Seed 1234 Avg. ↑
60%Nyström24.280.48624.140.48824.270.487
GS-MoE (Ours)20.640.50020.420.50020.520.502
GS-MoE + refit (Ours)19.800.50719.620.50719.680.506
80%Nyström54.010.37453.970.37553.990.376
GS-MoE (Ours)47.200.40047.500.40047.130.400
GS-MoE + refit (Ours)42.460.41242.840.41342.430.412

We find that the advantage of shared subspaces combined with width pruning grows with compression. At our 60% compression setting, GS-MoE maintains a +1.4pt lead in benchmark accuracy over pure Nyström compression averaged across seeds. At 80% GS-MoE's averaged benchmark gains over pure Nyström are +2.5pts. We also provide a refitted variant (see W2-rebuttal and AppC.5) for GS-MoE, described in Eq (6′), which has a +3.7pt gain over pure Nyström compression at the 80% setting.

Q.1 Part B why should practitioners adopt the full pipeline rather than pure width pruning

Where other compression methods plateau at compression ratios beyond 40%, GS-MoE's dual-axis functionality permits complementarity as compression strength increases. For example, on DeepSeek-MoE, our refitted 80% compression setting matches the benchmark performance of D²-MoE's 60% compression setting. Many recent MoE models use a large number of narrow experts, which is precisely the regime where GS-MoE's gains over competing methods exist. Further, GS-MoE has lower perplexity at near-identical throughput against a parameter-matched Nyström compressed model (Table R4).

AuthorsOpenReview ↗

Rebuttals 2

W.1 The empirical benefits of GS-MoE heavily depend on model architecture and fine-grained granularity. On coarse-grained models like Mixtral-8x7B (8 experts, large intermediate dimension), GS-MoE performs worse than D2-MoE at 60% compression (Avg. 0.50 vs 0.52; WikiText PPL 15.94 vs 6.46), and the full pipeline fails to outperform pure width pruning. The authors should explicitly characterize the architectural failure boundaries.

Q.2 Suggesting adding a sensitivity boundary analysis regarding expert count E versus intermediate width d_ff, providing guidelines on when subspace factorization ceases to provide gains over width pruning.

A good indicator is the expert width dffd_{ff} relative to the model dimension dd, rather than expert count EE. Intuitively, width pruning is effective when experts are wide, while subspace sharing becomes more useful when further pruning would leave each expert too narrow. Let kNk_N denote the average number of neurons retained per expert by width-only pruning. When kN≥dk_N \ge d, reducing intermediate width does not force the expert's projection matrices below rank dd. When kN<dk_N < d, further width pruning also reduces the maximum possible rank of the projection matrices. GS-MoE offers an alternative trade-off: sharing subspaces across experts saves parameters, allowing more intermediate neurons to be retained under the same budget.

For a total-model compression ratio ρ\rho, width-only pruning retains an average width of

kN=(1−ρNNrouted)dff,k_N=\left(1- \rho \frac{N}{N_{\mathrm{routed}}}\right)d_{ff},

where NN and NroutedN_{\mathrm{routed}} are the total and routed-expert parameter counts. For wide experts (dff>dd_{ff}>d), the retained width falls below dd when

ρ>(1−ddff)NroutedN.\rho>\left(1-\frac{d}{d_{ff}}\right)\frac{N_{\mathrm{routed}}}{N}.

For Mixtral, this threshold is approximately 69% compression. Table R5 has results consistent with this interpretation, GS-MoE provides little additional accuracy benefit through 60% compression, where kN/d=1.33k_N/d=1.33, but improves average accuracy by 1.5 and 1.8 percentage points at 70% and 80%, where kN/d=0.96k_N/d=0.96 and 0.600.60, respectively.

DeepSeek and Qwen already have dff<dd_{ff}<d, so width pruning reduces their maximum possible projection rank from the outset. GS-MoE's gains emerge at lower compression ratios and generally increase with compression. Thus, kN/dk_N/d provides a useful indicator of when subspace sharing may help.

Regarding expert count EE, finer-grained models with 64 experts benefit from GS-MoE's additional shared subspaces, especially at higher compression ratios with group assignment on. However, for Mixtral, additional shared subspaces provide no accuracy improvements. This result is intuitive given the parameters savings are fewer due to its lower expert count and the added cost of an extra basis, removing from the available neuron width.

Table R2. Width retained per expert relative to dd by pure width pruning (kNk_N) and by GS-MoE (kk), number of groups GG, and GS-MoE's gain over pure width pruning (ΔAvg., pts).

ModelRatiok_N/dk/dGΔAvg. ↑
DeepSeekMoE-16B-Base (E=64, d_ff/d=0.69)20%0.540.571−0.6
40%0.390.472+0.4
60%0.240.388+1.6
80%0.080.2416+2.5
Qwen2-57B-A14B (E=64, d_ff/d=0.71)40%0.380.482−0.1
60%0.220.378+0.8
80%0.050.1916+2.0
Mixtral-8×7B (E=8, d_ff/d=3.5)20%2.782.871−0.2
40%2.052.221−0.2
60%1.331.5310.0
70%0.961.201+1.5
80%0.600.811+1.8
AuthorsOpenReview ↗

Rebuttals 3

W.2 The optimization steps are locally decoupled approximations. Once up-projections are compressed, actual intermediate hidden activations change, invalidating the optimality conditions of the down-projection and neuron leverage scores.

Indeed, we acknowledge this limitation in App A.3. We resolve this limitation by refitting the down projection on the compressed activations it receives from the factorized up/gate projections.

A_e^{\downarrow} =\arg\min_{A}\\,\mathbb{E}_e\big\\|(P_g^{\downarrow})^\top y_e-A\\,\hat h_e\big\\|_2^2 =B_e\\,\mathbb{E}_e\big[h_e\hat h_e^\top\big]\big(\mathbb{E}_e\big[\hat h_e\hat h_e^\top\big]\big)^{-1}. \tag{6'}

Eq. (6′) solves the same least-squares problem as Eq. (6), but replaces the original unfactorized activations ϕ(Se⊤W~e↑x)⊙Se⊤We↑x\phi\big(S_e^\top\tilde W_e^{\uparrow}x\big)\odot S_e^\top W_e^{\uparrow}x with the factorized activations h^e=ϕ(A~e↑(P~g↑)⊤x)⊙Ae↑(Pg↑)⊤x\hat h_e=\phi\big(\tilde A_e^{\uparrow}(\tilde P_g^{\uparrow})^\top x\big)\odot A_e^{\uparrow}(P_g^{\uparrow})^\top x.

CompressionMethodWiki. ↓C4 ↓PTB ↓Avg. ↑
60% (G=8)Nyström only18.9526.3727.490.487
GS-MoE, Eq. (6)17.6821.8622.010.502
GS-MoE, Eq. (6′)16.9820.9521.110.506
80% (G=16)Nyström only45.5558.5057.900.376
GS-MoE, Eq. (6)42.2048.7950.410.400
GS-MoE, Eq. (6′)39.4843.1144.700.412

Table R7. Experiment results.

Avg↑ is the average benchmark performance. At the 60% compression ratio Eq. (6′) out performs Eq. (6) mildly at a 0.4pt gap, but entering into extreme compression regimes, the gap between Eq. (6′) and Eq. (6) expands to 1.2pts.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Summary of Revisions

Changes in the revised paper are highlighted in red.

We thank all reviewers for their feedback. We provide a brief summary of the revisions. Each item lists the reviewer comments it addresses.

Main paper.

  • Table 1: added D²-MoE Qwen (60/80%) and DeepSeek (80%). GS-MoE rows at 60% and 80% compression for DeepSeek and Qwen2. (SDYb-W1, LQjh-W2)
  • Table 1: added GS-MoE rows Qwen (60/80%) and DeepSeek (80%). (SDYb-W1, LQjh-W2)
  • Table 2 and Sec. 4.2: added width reduction to Table 2 grouped-assignment / group rank allocation ablation and updated the corresponding ablation paragraph. (LQjh-W1, cbWN-W2)
  • Table 3 and Sec. 4.2: added uncompressed and Nyström-only baselines, mean perplexity, and G=16G=16 to the DeepSeekMoE-16B-Base throughput results and updated runtime efficiency paragraph. (SDYb-Q5, SDYb-W3b)
  • Table 3 and Sec. 4.2: added TD-MoE and MoE-SVD throughput for Mixtral-8x7B. (SDYb-Q5, SDYb-W3b)
  • Table 4: added sensitivity to calibration seed on DeepSeekMoE-16B-Base. (SDYb-Q1)
  • Sec. 3.3: added a pointer to the refitted down-projection variant (GS-MoE refit). (SDYb-W2)
  • Sec. 4.2: added a calibration-seed sensitivity paragraph with Table 4. (SDYb-Q1)
  • Sec. 4.2: added an "Operating regime and architectural boundaries" paragraph. (SDYb-Q2, SDYb-W1, LQjh-W1, cbWN-Q1, cbWN-W1)

Appendix.

  • App. A.3: added GS-MoE refit, which refits Ae↓A_e^{\downarrow} on activations from the compressed up/gate projections (Eq. (19)). (SDYb-W2, SDYb-Q3)
  • App. A.5: added a paragraph on the optimality of group-rank allocation under equal per-rank costs. (cbWN-Q2, cbWN-W2)
  • App. B: added a hyperparameter-selection paragraph. (LQjh-Q1)
  • Table 5: added the layer-allocation temperatures ϵw\epsilon_w and ϵr\epsilon_r. (LQjh-Q1)
  • Table 7: added Mixtral-8×7B rows at 70% and 80%, and DeepSeek and Qwen rows at 80%. (SDYb-W1, LQjh-W1, cbWN-W1)
  • App. C.3 (Table 8): rank vs. width allocation analysis at varied compression ratios and models. (SDYb-Q2, cbWN-Q1)
  • App. C.4 (Table 9): comparison with MoLAE. (SDYb-W3a/W4a, SDYb-Q4)
  • App. C.5 (Table 10): GS-MoE refit results. (SDYb-W2, SDYb-Q3)
  • App. C.5 (Fig. 5): added a figure of the per-expert activation shift δe\delta_e (Eq. (26)) across layers of DeepSeekMoE-16B-Base at 80% compression. (SDYb-Q3)
  • App. C.6 (Table 11): results over three calibration seeds. (SDYb-Q1)
  • App. C.7 (Table 12): subspace-rank stability under perturbed rank scores. (cbWN-Q2)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.