acceptodds
Under review as a conference paper at ICLR 2027

SLEP-MoEfication: Space-Level Expert Partition for Rich Information Preservation in Sparse Dense-to-MoE Conversion

Abstract

Sparse dense-to-MoE conversion methods cluster pre-trained dense feed-forward networks (FFNs) into selectively activated experts to avoid the high cost of pre-training Mixture-of-Experts (MoE) models from scratch. Calibration-based approaches guide expert construction using calibration dataset to maintain performance close to the pre-trained dense model at lower inference cost. However, consistent information preservation across the input datasets is limited by reliance on calibration-dependent representations. To address this limitation, we propose a space-level expert partition method for sparse dense-to-MoE conversion (SLEP-MoEfication). Specifically, we introduce a subspace coverage score that measures the fraction of dense weight energy captured by projection onto the subspace spanned by each expert's pre-trained neuron weights. We maximize the sum of subspace coverage scores to support information preservation while keeping pre-trained weight values fixed. Our method consistently outperforms the evaluated baselines across multiple models under a common training-free centroid-based router. Our empirical analyses reveal strong positive correlations between subspace coverage and information preservation, and between information preservation and task performance. These gains hold even when random partition outperforms other existing methods. Further analysis shows that our method improves the trade-off between expert activation sparsity and performance, reduces inference cost, and improves expert load balance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparse Dense-to-MoE, Moefication

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_n1tYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SLEP-MoEfication, a data-free method for converting pre-trained dense language models into sparse Mixture-of-Experts (MoE) models. The method partitions FFN neurons into experts by maximizing a subspace coverage objective, aiming to preserve the original model's information while reducing inference computation. A Sinkhorn-based optimization algorithm is used for expert partitioning, together with a training-free centroid router. Experiments on three language models show improvements over several existing partitioning methods, particularly on LLaMA-3 8B.

Critical strengths
  1. The paper addresses a problem of improving inference efficiency by converting pre-trained dense models into sparse MoE architectures.
  2. The proposed subspace coverage objective provides an interesting perspective on expert partitioning, supported by a spectral analysis in Section 5.3.
  3. The experiments cover three language models and include comparisons with existing methods, random partitioning, information preservation analysis, and load balancing evaluation.
Critical weaknesses
  1. The connection between subspace coverage and information preservation is insufficiently justified (Section 5.2). The objective only considers the input or gate projection, ignoring other FFN projections and nonlinearities.
  2. The data-free claim is questionable (Appendix C.4), as the subspace rank is selected using the same downstream benchmarks reported in the main experiments. Independent validation is needed.
  3. Performance gains are inconsistent across models (Table 1). Improvements are substantial on LLaMA-3 8B but marginal on Qwen2.5-7B. Further analysis is needed to explain these differences.
  4. The experiments lack statistical validation (Table 1). Multiple random seeds and standard deviations are needed to establish the reliability of the improvements.
  5. The information preservation analysis relies heavily on GMI correlations (Section 6.1). Direct FFN reconstruction errors or output similarity measurements would provide stronger evidence.
  6. The implementation optimizes normalized weights, while the theoretical objective uses original weights (Equation 6 and Appendix C.1). An ablation study is needed to justify this difference.
  7. The efficiency evaluation only reports FLOPs (Table 3). Actual inference latency, throughput, and conversion costs should also be evaluated.
Questions for authors

see weeknesses

7 comments
AuthorsOpenReview ↗

Official Authors’ Comments to Reviewer n1tY (1)

Thanks to Reviewer n1tY for the fruitful comments. We use W, and Q to denote weaknesses, and questions respectively. In the revised version, the corresponding question and weakness appear in green text.


Common Comments

SLEP-MoEfication is compatible with various routers. Our work does not propose an integrated MoEfication framework encompassing partitioning, routing, and other components. Instead, we focus on evaluating the proposed partition while keeping all other settings identical to those of the baselines. Accordingly, SLEP-MoEfication is compatible with various router designs.

This focus was not sufficiently clear in the original manuscript.


R3 W1

The connection between subspace coverage and information preservation is insufficiently justified (Appendix D.2 / E.4). The objective only considers the input or gate projection, ignoring other FFN projections and nonlinearities.

Answer of R3 W1

Our coverage criterion is positively associated with GMI measured after the full FFN computation. We agree that the original objective does not explicitly account for the other FFN projections and nonlinearities. Our GMI analysis evaluates features after the output/down projection. In gated FFNs, these features incorporate the nonlinear activation, gate–up multiplication, and down projection. This allows us to examine whether coverage remains associated with information preservation as measured by GMI after these operations.

To evaluate the connection with information preservation, we compute GMI using features after the down projection. In gated FFNs, these features incorporate the nonlinear activation, gate–up multiplication, and down projection. Thus, the GMI analysis evaluates representations that reflect all these operations.

\begin{array}{|c|c|c|c|c|c|c|} \hline \text{Model} & \text{metrics} & \text{Ours} & \text{CMoE} & \text{MoEfication} & \text{LLaMA-MoE v2} & \text{correlation} \\ \hline \text{LLaMA-3 8B} & \bar{\eta}_L & \textbf{1.606}(\pm0.001) & 1.575(\pm0.001) & 1.420(\pm0.001) & 1.521(\pm0.001) & 0.97 \\ \text{LLaMA-3 8B} & \text{GMI} & \textbf{23.84}(\pm0.000) & 23.76(\pm0.001) & 23.62(\pm0.001) & 23.73(\pm0.001) & 0.97 \\ \hline \text{OPT-2.7B} & \bar{\eta}_L & \textbf{3.995}(\pm0.000) & 3.709(\pm0.003) & 3.228(\pm0.000) & 3.566(\pm0.003) & 0.95 \\ \text{OPT-2.7B} & \text{GMI} & \textbf{16.71}(\pm0.001) & 16.35(\pm0.006) & 16.12(\pm0.002) & 16.39(\pm0.002) & 0.95 \\ \hline \text{Qwen-2.5 7B} & \bar{\eta}_L & \textbf{2.211}(\pm0.001) & 1.994(\pm0.002) & 1.730(\pm0.035) & 1.853(\pm0.000) & 0.95 \\ \text{Qwen-2.5 7B} & \text{GMI} & \textbf{17.82}(\pm0.001) & 17.69(\pm0.002) & 17.45(\pm0.001) & 17.65(\pm0.000) & 0.95 \\ \hline \end{array}

The table reports the mean and standard deviation of coverage and GMI across three partition seeds for each method and backbone. We compute the correlation across the resulting 12 converted model instances per backbone, with each instance contributing one coverage–GMI pair. The correlations are 0.97 for LLaMA-3 8B and 0.95 for both OPT-2.7B and Qwen-2.5 7B. Our method also achieves the highest mean GMI on all three backbones. These results provide empirical support for the relationship between coverage and information preservation within the scope of GMI. We clarify this interpretation and GMI’s restriction to Gaussian-style dependence in Appendix D.2 and E.4 of the revised paper.

To address the scope of the objective, we additionally apply the coverage objective to the gate, up, and down projections with activation-based routing. This setting allows separate grouping of weight column vectors within each projection while retaining their original indices. Results for this extension are provided in Section 6.1 Robustness across Router Types, and Appendix E. 2 of the revised paper.

AuthorsOpenReview ↗

Official Authors’ Comments to Reviewer n1tY (2)

R3 W2

The data-free claim is questionable (Appendix C.4), as the subspace rank is selected using the same downstream benchmarks reported in the main experiments. Independent validation is needed.

Answer of R3 W2

Our data-free claim applies to partition construction for a specified rank. This stage uses only pretrained weights without calibration data, while rank selection is data-dependent. We clarify this distinction in the revised manuscript. Validation-based rank selection preserves higher mean evaluation accuracy than the best-performing baselines on all three models. The original ranks were selected using six-task evaluation accuracy. To assess whether the reported gains depend on this choice, we conduct additional experiments using ARC-C and SciQ validation accuracy and held-out C4 perplexity. These validation data are separate from the final evaluation sets. Final accuracy results are means and standard deviations across three partition seeds.

1) ARC-C and SciQ validation accuracy. We use only these two tasks because the validation splits of BoolQ, PIQA, WinoGrande, and HellaSwag are already used for final evaluation in our setup. The table reports average validation accuracy.

\hline \text{Model} & \text{r=24} & \text{r=25} & \text{r=26} & \text{r=27} & \text{r=28} \\ \hline \text{LLaMA-3-8B} & 54.51 & 55.95 & \textbf{59.37} & 53.09 & 53.09 \\ \hline \text{OPT-2.7B} &57.93 & 55.95 & 57.97 & \textbf{58.55} & 57.70 \\ \hline \text{Qwen2.5-7B} & 74.59 & 74.39 & 73.53 & \textbf{74.60} & 73.31 \\ \hline \end{array}$$

This criterion selects ranks 26, 27, and 27 for LLaMA, OPT, and Qwen, respectively. LLaMA and Qwen retain their original ranks. OPT changes to rank 27 and achieves 61.56(±0.34)61.56(\pm0.34), exceeding the best baseline, MoEfication, at 61.11(±0.34)61.11(\pm0.34) (pp-value = 0.011).

2) C4 validation perplexity. We select the rank with the lowest held-out C4 perplexity without using accuracy on any of the six tasks.

\hline \text{Model} & \text{r=24} & \text{r=25} & \text{r=26} & \text{r=27} & \text{r=28} \\ \hline \text{LLaMA-3-8B} &30.79 & 26.74 & 25.03 & \textbf{24.74} & 30.28\\ \hline \text{OPT-2.7B} & 15.80 & 15.79 & \textbf{15.68} & 15.73 & 15.77 \\ \hline \text{Qwen2.5-7B} & 13.31 & 13.38 & \textbf{13.18} & 13.19 & 13.43 \\ \hline \end{array}$$

C4-based selection gives r=27, 26, and 26 for LLaMA-3 8B, OPT-2.7B, and Qwen-2.5 7B, respectively. The six-task evaluation results at the selected ranks are compared with the best-performing baselines below:

  • LLaMA-3 8B: CMoE achieves 53.88(±1.22)53.88(\pm1.22), while our method achieves 56.32(±1.63)56.32(\pm1.63) at r=27r=27 (pp-value is 0.0092).
  • OPT-2.7B: MoEfication achieves 61.11(±0.34)61.11(\pm0.34), while our method achieves 62.15(±0.45)62.15(\pm0.45) and pp-value is 0.0025.
  • Qwen-2.5 7B: CMoE achieves 75.33(±0.16)75.33(\pm0.16), while our method achieves 75.39(±0.09)75.39(\pm0.09) at r=26r=26 (pp-value is 0.3429).

These results retain statistically significant improvements on LLaMA and OPT and a numerical improvement on Qwen.

Both validation criteria preserve higher mean evaluation accuracy than the best-performing baselines. Under C4-based selection, the improvements remain statistically significant on LLaMA and OPT, while Qwen retains a numerical improvement. These results support the performance comparisons under validation-based rank selection.

To obtain a single rank per model in the additional validation analysis, we combine the two criteria using equally weighted z-scores.

AuthorsOpenReview ↗

Official Authors’ Comments to Reviewer n1tY (3)

R3 W4

(R3 W4) The experiments lack statistical validation (Table 1). Multiple random seeds and standard deviations are needed to establish the reliability of the improvements.

Answer of R3 W3 & R3 W4

Across three seeds, SLEP-MoEfication achieves higher mean accuracy on all three models, with statistically significant improvements on LLaMA and OPT. To assess variability across runs, we evaluate each method on each model using three random seeds. We report the mean and standard deviation for each task and the six-task average. These results have also been incorporated into Table 1 of the revised manuscript.

\hline \text{Model} & \text{Method} & \text{C.D.} & \text{BoolQ} & \text{PIQA} & \text{SciQ} & \text{WinoGrande} & \text{ARC-C} & \text{HellaSwag} & \text{Avg.} \\ \hline & \text{Dense} & - & 83.70 & 80.69 & 94.60 & 77.35 & 59.47 & 80.56 & 79.39 \\ & \text{Llama-MoE v2} & \text{O} & 38.67(\pm 0.98) & 51.54(\pm 1.13) & 25.80(\pm 3.94) & 49.12(\pm 0.39) & 24.60(\pm 1.58) & 26.19(\pm 0.49) & 35.99(\pm 0.55) \\ \text{LLaMA-3 8B} & \text{CMoE} & \text{O} & 56.27(\pm 3.76) & 63.18(\pm 2.41) & 73.43(\pm 2.62) & 54.85(\pm 2.38) & 27.67(\pm 0.91) & 47.85(\pm 3.84) & 53.88(\pm 1.22) \\ & \text{MoEfication} & \text{X} & 41.03(\pm 1.40) & 51.16(\pm 0.52) & 24.20(\pm 0.30) & 50.49(\pm 0.57) & 26.25(\pm 0.49) & 26.63(\pm 0.19) & 36.63(\pm 0.23) \\ & \text{Ours} & \text{X} & \textbf{56.89}(\pm 1.63) & \textbf{68.81}(\pm 1.59) & \textbf{78.40}(\pm 1.40) & \textbf{57.72}(\pm 1.31) & \textbf{35.13}(\pm 0.30) & \textbf{59.80}(\pm 0.42) & \textbf{59.46}(\pm 0.89) \\ \hline & \text{Dense} & - & 61.87 & 74.76 & 80.60 & 61.33 & 34.30 & 59.73 & 62.10 \\ & \text{Llama-MoE v2} & \text{O} & 57.30(\pm 3.96) & 72.98(\pm 0.64) & 76.03(\pm 0.81) & 60.96(\pm 0.16) & 33.87(\pm 0.09) & 57.17(\pm 0.55) & 59.72(\pm 0.66) \\ \text{OPT-2.7B} & \text{CMoE} & \text{O} & 59.01(\pm 0.46) & 72.71(\pm 0.25) & 78.93(\pm 0.59) & \textbf{61.35}(\pm 0.20) & 33.62(\pm 0.23) & 57.77(\pm 0.04) & 60.56(\pm 0.21) \\ & \text{MoEfication} & \text{X} & 58.33(\pm 1.47) & \textbf{73.87}(\pm 0.08) & \textbf{80.90}(\pm 0.36) & 61.12(\pm 0.46) & 34.27(\pm 0.42) & 58.18(\pm 0.16) & 61.11(\pm 0.34) \\ & \text{Ours} & \text{X} & \textbf{64.89}(\pm 0.57) & 73.81(\pm 0.58) & 80.07(\pm 0.80) & 61.17(\pm 1.17) & \textbf{34.44}(\pm 0.71) & \textbf{58.49}(\pm 0.26) & \textbf{62.15}(\pm 0.45) \\ \hline & \text{Dense} & - & 88.13 & 79.98 & 95.20 & 75.30 & 64.33 & 78.47 & 80.24 \\ & \text{Llama-MoE v2} & \text{O} & 69.48(\pm 2.42) & 67.50(\pm 2.04) & 90.63(\pm 0.60) & 57.75(\pm 1.21) & 39.14(\pm 2.01) & 56.46(\pm 0.32) & 63.49(\pm 0.85) \\ \text{Qwen-2.5 7B} & \text{CMoE} & \text{O} & \textbf{83.58}(\pm 0.48) & 77.51(\pm 0.55) & 93.17(\pm 0.60) & 68.32(\pm 1.18) & 55.23(\pm 0.64) & 74.16(\pm 0.11) & 75.33(\pm 0.16) \\ & \text{MoEfication} & \text{X} & 79.59(\pm 3.60) & 76.91(\pm 0.69) & 90.67(\pm 1.65) & 67.27(\pm 0.63) & 54.69(\pm 1.70) & \textbf{75.11}(\pm 0.16) & 74.04(\pm 0.69) \\ & \text{Ours} & \text{X} & 82.86(\pm 0.47) & \textbf{77.86}(\pm 0.16) & \textbf{94.03}(\pm 0.06) & \textbf{68.67}(\pm 0.96) & \textbf{56.51}(\pm 1.21) & 73.99(\pm 0.15) & \textbf{75.65}(\pm 0.18) \\ \hline \end{array}$$

Compared with the best-performing baseline on each model, SLEP-MoEfication improves the six-task average by 5.58 percentage points on LLaMA-3 8B, 1.04 points on OPT-2.7B, and 0.32 points on Qwen-2.5 7B. The improvements on LLaMA and OPT are statistically significant at the 0.05 significance level. On Qwen, the gain remains modest and does not reach statistical significance at this threshold (p=0.062p=0.062); we therefore report it as a numerical improvement.

Reviewer_QhAeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a calibration-free criterion for partitioning a dense FFN's neurons into equal-sized experts. For each expert, the top-r left singular vectors of its gate-weight slice span a subspace; the "subspace coverage score" measures how much dense FFN weight energy that subspace captures, and the objective maximizes the sum over experts. Optimization learns a cost matrix mapped to a soft assignment by entropic balanced Sinkhorn, with differentiable subspace iteration, annealed temperature, and min-cost-flow discretization. Weights are never updated, and routing uses MoEfication's training-free centroid rule. Experiments cover OPT-2.7B, LLaMA-3 8B, and Qwen2.5-7B at 320/448/592 experts per layer.

Critical strengths
  • The subspace criterion is a concrete methodological contribution with a spectral interpretation (p. 5, Section 5.3, Eqs. (8)–(10)). Table 2 (p. 7) reports higher final-partition coverage, and Figure 5 (p. 8) shows gains over random grouping on all three backbones. These measurements substantiate the quality of the actual discrete partitions.

  • Table 1 (p. 6) reports a substantial LLaMA improvement over the strongest compared conversion baseline, 60.16 versus 52.09, and an OPT improvement from 61.08 to 62.18, approximately matching the dense reference of 62.10. These are meaningful benefits. LLaMA remains below its dense reference of 79.39; Qwen is approximately tied at 75.84 versus 75.79, with reversals already acknowledged at other sparsities (pp. 24–25, Figure 9).

Critical weaknesses
  • The link from soft optimization to the quality of the final hard partition remains insufficiently established (pp. 20–21, Appendix C.1, Eqs. (31)–(32), Table 5). Uniform assignment can maximize the ideal soft objective when exact top-r eigenspaces are used; this does not invalidate the implemented heuristic, which uses nonuniform initialization, approximate subspace iteration, annealing, and discretization. Nevertheless, final-partition comparisons do not isolate improvement from optimizing beyond the initialization. The implementation also explicitly optimizes a column-normalized surrogate, whereas the main criterion uses the original weights and Figure 6 diagnoses full 32-dimensional subspaces rather than the optimization ranks of 26/27 (p. 5, Eqs. (5)–(6); p. 8). These disclosed differences make direct evidence connecting the optimized surrogate to final hard-partition quality consequential to the method’s justification.

  • Rank-selection independence is unresolved, and the sensitivity analysis makes this consequential (p. 21, Table 5; p. 22, Appendix C.4/Figure 8). The adopted ranks coincide with the best six-task averages among the tested candidates, with a relatively sharp local peak for LLaMA. The paper does not clearly distinguish prior or held-out selection from selection using the final reported benchmarks. This limits confidence in how robust the gains are to the selection procedure, especially the near-tie on Qwen. It does not establish test leakage, a particular amount of tuning gain, or the loss of the substantial LLaMA improvement.

Questions for authors
  • Can existing logs or a focused comparison isolate the contribution of soft optimization using the same initialization and discretization, with soft/hard coverage before and after optimization (pp. 20–21)? Please identify whether the measured coverage uses raw or normalized weights and which retained rank is used. This would show whether improving the implemented surrogate improves the final partition under the stated criterion.

  • How were the Table 5 ranks selected relative to Figure 8 evaluation (pp. 21–22)? Please identify the splits and timing, distinguishing post-hoc sensitivity analysis from parameter selection. If the reported benchmark scores guided selection, a result with an independently selected or fixed rank would help establish the effect on the principal comparisons.

5 comments
AuthorsOpenReview ↗

Official Authors’ Comments to Reviewer QhAe (1)

Thanks to Reviewer QhAe for the fruitful comments. We use W, and Q to denote weaknesses, and questions respectively. In the revised version, the corresponding question and weakness appear in blue text.


Common Comments

SLEP-MoEfication is compatible with various routers. Our work does not propose an integrated MoEfication framework encompassing partitioning, routing, and other components. Instead, we focus on evaluating the proposed partition while keeping all other settings identical to those of the baselines. Accordingly, SLEP-MoEfication is compatible with various router designs.

This focus was not sufficiently clear in the original manuscript.

AuthorsOpenReview ↗

Official Authors’ Comments to Reviewer QhAe (2)

R2 W1-1

The link from soft optimization to the quality of the final hard partition remains insufficiently established (pp. 20–21, Appendix C.1, Eqs. (31)–(32), Table 5). Uniform assignment can maximize the ideal soft objective when exact top-r eigenspaces are used; this does not invalidate the implemented heuristic, which uses nonuniform initialization, approximate subspace iteration, annealing, and discretization. Nevertheless, final-partition comparisons do not isolate improvement from optimizing beyond the initialization.

Answer of R2 W1-1

On the link between the soft objective and the final hard partition. We agree that the uniform plan maximizes the ideal soft objective. It even attains the upper bound E∑i≤rσi2(W~)/∥W~∥F2E\sum_{i\le r}\sigma_i^2(\tilde W)/\|\tilde W\|_F^2 given by Ky Fan's maximum principle. The uniform plan is not a partition, however. Every neuron belongs to every expert with weight 1/E1/E, so the plan is equivalent to the dense FFN and no expert can be selected from it. Its high value belongs to a state that cannot be used, not to a partition.

A hard assignment cannot be differentiated, so the optimization has to pass through a soft plan, and at high temperature this plan stays close to uniform. This is the standard behavior of deterministic annealing, where the assignment changes only below a critical temperature [1-2] , applied here to Sinkhorn as in softassign [3]. The soft values of the early stages therefore carry no information about the partition. At ϵ=1\epsilon=1, two random partitions and our final partition of OPT 2.7B layer 0 all give the same value of 23.304.

The partition is formed at low temperature, where the plan is close to one-hot and the soft value tracks the hard one. In this regime the optimizer increases the objective, and at the end of the run the soft and hard values agree to within 0.08 for all three models (Table 20). For OPT 2.7B the hard value stays at the level of the random initialization through the high-temperature stages, rises only in the low-temperature stages, and ends above random balanced partitions (11.93 vs 11.76).

  • [1] K. Rose. Statistical mechanics and phase transitions in clustering. Physical Review Letters, 1990.
  • [2] K. Rose. Deterministic annealing for clustering, compression, classification, regression, and related optimization problems. Proceedings of the IEEE, 86(11): 1998.
  • [3] S. Gold. A graduated assignment algorithm for graph matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1996.

Sinkhorn-based partition optimization improves average accuracy over initialization alone on all three models. For each seed, we start from the same initial assignment and compare the hard partitions obtained before and after optimization. The initialization-only condition directly discretizes the initial assignment without applying Sinkhorn-based optimization. Both conditions use the same discretization procedure, router, sparsity, rank, and all other evaluation settings. The table reports the mean and standard deviation across three partition seeds.

\hline \text{Model} & \text{Method} & \text{BoolQ} & \text{PIQA} & \text{SciQ} & \text{WinoGrande} & \text{ARC-C} & \text{HellaSwag} & \text{Avg.} \\ \hline \text{LLaMA-3 8B} & \text{Ours (init.)} & 45.86(\pm 2.46) & 63.47(\pm 1.18) & 65.00(\pm 0.46) & 54.17(\pm 0.95) & 26.34(\pm 1.31) & 48.93(\pm 0.41) & 50.63(\pm 0.73) \\ & \text{Ours} & \textbf{56.89}(\pm 1.63) & \textbf{68.81}(\pm 1.59) & \textbf{78.40}(\pm 1.40) & \textbf{57.72}(\pm 1.31) & \textbf{35.13}(\pm 0.30) & \textbf{59.80}(\pm 0.42) & \textbf{59.46}(\pm 0.89) \\ \hline \text{OPT-2.7B} & \text{Ours (init.)} & 58.69(\pm 2.43) & 73.20(\pm 0.08) & 77.63(\pm 2.27) & 60.12(\pm 1.03) & 33.93(\pm 0.18) & 57.92(\pm 0.18) & 60.25(\pm 0.60) \\ & \text{Ours} & \textbf{64.89}(\pm 0.57) & \textbf{73.81}(\pm 0.58) & \textbf{80.07}(\pm 0.80) & \textbf{61.17}(\pm 1.17) & \textbf{34.44}(\pm 0.71) & \textbf{58.49}(\pm 0.26) & \textbf{62.15}(\pm 0.45) \\ \hline \text{Qwen-2.5 7B} & \text{Ours (init.)} & 82.14(\pm 0.74) & 77.37(\pm 0.77) & 93.63(\pm 1.21) & \textbf{69.17}(\pm 1.26) & 55.32(\pm 0.99) & 73.43(\pm 0.65) & 75.18(\pm 0.56) \\ & \text{Ours} & \textbf{82.86}(\pm 0.47) & \textbf{77.86}(\pm 0.16) & \textbf{94.03}(\pm 0.06) & 68.67(\pm 0.96) & \textbf{56.51}(\pm 1.21) & \textbf{73.99}(\pm 0.15) & \textbf{75.65}(\pm 0.18) \\ \hline \end{array}$$

The six-task average improves after optimization on all three models, with the largest improvement on LLaMA. Although task-level gains vary, this controlled comparison provides direct empirical evidence that the implemented optimization procedure improves the downstream performance of the final hard partition beyond initialization alone.

More details are in Section 6.2 Effect of Sinkhorn-Based Partition Optimization and Appendix E. 7 of the revised paper.

AuthorsOpenReview ↗

Official Authors’ Comments to Reviewer QhAe (3)

R2 W1-2

The implementation also explicitly optimizes a column-normalized surrogate, whereas 1) the main criterion uses the original weights and 2) Figure 6 diagnoses full 32-dimensional subspaces rather than the optimization ranks of 26/27 (p. 5, Eqs. (5)–(6); p. 8). These disclosed differences make direct evidence connecting the optimized surrogate to final hard-partition quality consequential to the method’s justification.

Answer of R2 W1-2

SLEP-MoEfication retains its advantage when final hard partitions are evaluated using the original weights and the ranks used for optimization. In both tables, standard deviations are computed across three partition seeds for each method and model.

1) Effect of column normalization. We compare SLEP-MoEfication with and without column normalization. For both variants, we compute η\eta using the original, unnormalized weights and the same evaluation procedure. We evaluate the optimization ranks of r=26r=26 for LLaMA-3 8B and OPT-2.7B and r=27r=27 for Qwen-2.5 7B. We also evaluate r=32r=32. The table below shows that column normalization maintains or improves mean η\eta across all three models at both rank settings. These results support the effectiveness of the column-normalized surrogate in producing high-quality hard partitions evaluated under the original weights.

\begin{array}{|c|c|c|c|} \hline \text{Model} & \text{Method} & \eta (r=26/27) & \eta (r=32) \\ \hline \text{LLama3-8B} & \text{Ours (w/o norm)} & \textbf{6.16}(\pm1.81) & 7.20(\pm1.77) \\ \text{LLama3-8B} & \text{Ours} & \textbf{6.16}(\pm1.98) & \textbf{7.21}(\pm2.20) \\ \hline \text{OPT-2.7B} & \text{Ours (w/o norm)} & 11.32(\pm1.55) & 12.63(\pm2.11) \\ \text{OPT-2.7B} & \text{Ours} & \textbf{11.50}(\pm1.77) & \textbf{12.78}(\pm0.49) \\ \hline \text{Qwen-2.5 7B} & \text{Ours (w/o norm)} & 11.24(\pm18.13) & 12.84(\pm22.01) \\ \text{Qwen-2.5 7B} & \text{Ours} & \textbf{11.74}(\pm7.39) & \textbf{13.11}(\pm1.75) \\ \hline \end{array}

2) Comparison at the optimization ranks. Figure 6 evaluates full 32-dimensional subspaces. To directly assess final hard-partition quality at the ranks used for optimization, we additionally evaluate η\eta at r=26r=26 for LLaMA-3 8B and OPT-2.7B and r=27r=27 for Qwen-2.5 7B. All methods are evaluated on their final hard partitions using the original, unnormalized weights. As shown below, SLEP-MoEfication with column normalization achieves higher mean η\eta than the baselines on all three models. These results show that the advantage observed in Figure 6 also holds at the actual optimization ranks, providing empirical support for the quality of the hard partitions produced by our column-normalized optimization. We have added these results to the ablation study in **Appendix E.5, Table 14 ** of the revised manuscript.

\begin{array}{|c|c|c|c|} \hline \text{Method} & \text{LLaMA3-8B (r=26)} & \text{OPT-2 7B (r=26)} & \text{Qwen-2.5 7B(r=27)} \\ \hline \text{LLaMA-MoE v2} & 5.82(\pm4.00) & 10.18(\pm8.15) & 9.75(\pm2.65) \\ \text{CMoE} & 6.03(\pm2.66) & 10.60(\pm9.25) & 10.51(\pm18.32) \\ \text{MoEfication} & 5.39(\pm2.41) & 9.03(\pm4.88) & 8.90(\pm38.26) \\ \text{Ours (w/o norm)} & \textbf{6.16}(\pm1.81) & 11.32(\pm1.55) & 11.24(\pm18.13) \\ \text{Ours} & \textbf{6.16}(\pm1.98) & \textbf{11.50}(\pm1.77) & \textbf{11.74}(\pm7.39) \\ \hline \end{array}

Reviewer_YT6VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a way to split the FFN neurons of a pre-trained dense LLM into equal-sized experts without using any data. Each expert's neurons span a subspace, and the "subspace coverage score" measures how much of the dense input-projection energy, ||U_i^T F||_F^2 / ||F||_F^2, falls into that subspace. The partition is chosen to maximize the sum of these scores, using a balanced Sinkhorn relaxation followed by an exact assignment with capacity constraints. The authors also show that the objective weights each dense eigen-direction by its eigenvalue. With the same training-free centroid router for all methods, the approach gets the best average accuracy over MoEfication, LLaMA-MoE v2 and CMoE on six commonsense and QA benchmarks for LLaMA-3 8B, OPT-2.7B and Qwen2.5-7B. The paper also reports correlations between coverage, Gaussian mutual information (GMI) and accuracy, plus FLOPs and load balance.

Critical strengths
  • The method needs no data and adds no parameters. Experts are built only from the pre-trained weights, so there is no dependence on a calibration set.
  • The objective is simple and well defined, and the spectral form (Eq. 9) makes it easy to interpret.
  • The analysis covers more than accuracy: Grassmann alignment, GMI under distribution shift, different sparsity levels, FLOPs at matched accuracy, and load balance.
  • Comparing against a random partition is a good sanity check that this line of work often leaves out.
Critical weaknesses
  1. The main operating point is not very practical. The main results use only 25% expert activation sparsity, so 75% of the experts are still active. Even then, LLaMA-3 8B drops from 79.39 to 60.16 average accuracy, a loss of about 19 points. At this level the compute savings are small and the accuracy loss is large. The paper doesn't show a setting where the converted model is actually usable, for example after some continued training or router fine-tuning, which is how LLaMA-MoE v2 and CMoE are normally used.

  2. The shared router may be unfair to the baselines. All methods have to use the MoEfication centroid router, and shared experts are removed. On LLaMA-3 8B, LLaMA-MoE v2 and MoEfication fall to around chance (35-37% average, SciQ about 25%), which is far below what those methods report in their own papers. To me this looks more like their partitions not fitting this router than their partitions being bad. Please also report each baseline with its own routing, and with a small amount of router training. The claim is about partition quality, but the current setup mixes partition quality with how well each partition suits the chosen router.

  3. The rank is tuned on the test benchmarks. Appendix C.4 picks the subspace rank r (26 or 27) by the best average accuracy on the same six tasks used for evaluation. Given how small some of the margins are (next point), this matters.

  4. The margins are small and there are no error bars. On Qwen2.5-7B the gain over CMoE is 0.05 points (75.84 vs 75.79). On OPT-2.7B the method is only about 1 point above MoEfication. There are no standard deviations over seeds, even though the Sinkhorn initialization is random. The only large gap is on LLaMA-3 8B, which is exactly where the baselines collapse (point 2).

  5. The evidence for the mechanism is weak. The coverage-GMI correlations in Table 2 are Pearson correlations over just four methods per model, so they don't say much. The per-expert GMI differences are tiny compared to their standard deviations (e.g. 23.84 ± 1.45 vs 23.76 ± 2.47). GMI itself only captures linear, Gaussian-style dependence, which the authors acknowledge.

  6. The objective and the evaluation are narrow. For gated FFNs, the objective only looks at the gate projection and ignores the up and down projections, even though they also shape each expert's output. All benchmarks are log-likelihood multiple choice. There is no perplexity, no generation or knowledge-heavy task (e.g. MMLU, GSM8K), and no measured latency. The FLOPs are theoretical and leave out router and gather/scatter costs.

  7. Some related baselines are missing. Training-free activation sparsity methods such as Deja Vu, CATS and TEAL also try to skip FFN computation without retraining, and they often reach much higher sparsity with less degradation. They should at least be discussed, and ideally compared.

Questions for authors
  1. How do the baselines do with their own routers, and how do all methods compare after a small amount of router training or continued pre-training?
  2. Could you choose r on a held-out validation set (e.g. C4 perplexity) instead of the evaluation tasks, and report the numbers you get?
  3. Please report mean ± std over several partition seeds for Table 1.
  4. What happens at more realistic sparsity levels, say 50-75%? How does the method compare to training-free activation sparsity methods at the same FLOPs?
  5. Does adding the up and down projections to the coverage objective change the results?
  6. Could you report perplexity, at least one generation benchmark, and actual wall-clock latency?
8 comments
AuthorsOpenReview ↗

Official Authors’ Comments to Reviewer YT6V (1)

Thanks to Reviewer YT6V for the fruitful comments. We use W, and Q to denote weaknesses, and questions respectively. In the revised version, the corresponding question and weakness appear in red text.


Common Comments

SLEP-MoEfication is compatible with various routers. Our work does not propose an integrated MoEfication framework encompassing partitioning, routing, and other components. Instead, we focus on evaluating the proposed partition while keeping all other settings identical to those of the baselines. Accordingly, SLEP-MoEfication is compatible with various router designs.

This focus was not sufficiently clear in the original manuscript. We have clarified it in Section 6.1 Robustness across Router Types of the revised manuscript.


R1 W1-1

  1. The main operating point is not very practical. The main results use only 25% expert activation sparsity, so 75% of the experts are still active. 2) Even then, LLaMA-3 8B drops from 79.39 to 60.16 average accuracy, a loss of about 19 points. At this level the compute savings are small and the accuracy loss is large.

Answer of R1 W1-1

We agree that the original centroid-router setting causes a substantial accuracy drop on LLaMA-3 8B. As discussed in the Common Comments, our contribution focuses on partitioning and is compatible with various routers. Our main experiments use MoEfication's centroid router to compare partitions under the same routing conditions.

Our partition achieves higher accuracy than TEAL at sparsity levels from 0.50 to 0.75 with activation-based routing. We denote this configuration as Ours$^\star$. Whereas TEAL selects individual weight column vectors, our method selects groups of two column vectors. We retain this grouping because singleton groups eliminate the partitioning decision addressed by our method. Both methods use activations for selection. The table reports average accuracy (%) across the six evaluation datasets.

\hline \text{Model (Dense)} & \text{Method} & 0.50 & 0.55 & 0.60 & 0.65 & 0.70 & 0.75 \\ \hline \text{LLaMA-3 8B (79.39)} & \text{TEAL} & 77.41 & 76.28 & 74.47 & 71.98 & 67.90 & 58.25 \\ & \text{Ours}^{\star} & \mathbf{78.07} & \mathbf{77.74} & \mathbf{76.36} & \mathbf{75.64} & \mathbf{73.70} & \mathbf{71.29} \\ \hline \text{Qwen-2.5 7B (80.24)} & \text{TEAL} & 78.75 & 78.42 & 77.42 & 75.87 & 73.46 & 69.17 \\ & \text{Ours}^{\star} & \mathbf{79.19} & \mathbf{79.12} & \mathbf{78.68} & \mathbf{77.80} & \mathbf{76.48} & \mathbf{75.10} \\ \hline \end{array}$$

Our method outperforms TEAL on both models across all evaluated sparsity levels, with larger improvements as sparsity increases. These results show that our partition can retain substantially more accuracy at higher sparsity when combined with activation-based routing. Results at higher sparsity under the original centroid-router setting are provided in Appendix E.8 (Table 21) of the revised paper.

We also report measured inference efficiency under the centroid-router setting. The following results use Qwen-2.5 7B at 50% expert activation sparsity on a single GPU, measured on a converted model from a single partition seed. The reported latency includes router computation and expert selection. This configuration differs from the activation-based configuration evaluated above.

\hline \text{Method} & \text{Acc. (\%)} & \text{Latency (ms/token)} \downarrow & \text{Throughput (tok/s)} \uparrow \\ \hline \text{Dense} & 80.24 & 23.00 & 43.48 \\ \hline \text{SLEP-MoEfication} & \mathbf{63.63} & \mathbf{19.16} & \mathbf{52.19} \\ \text{CMoE} & 58.60 & 19.32 & 51.76 \\ \text{MoEfication} & 59.26 & 19.33 & 51.73 \\ \text{LLaMA-MoE v2} & 43.89 & 19.26 & 51.92 \\ \hline \end{array}$$

Our method achieves approximately 1.20× faster inference than the dense model and higher accuracy than the converted baselines with comparable latency and throughput. The accuracy gap from the dense model remains under this configuration. These measurements document the accuracy–runtime trade-off of the centroid-router setting, while the activation-based results above demonstrate improved accuracy retention at higher sparsity. Detailed in Appendix E.10, and conversion costs in Appendix E.9 (Table 22), and results for LLaMA-3 8B and OPT-2.7B are provided in Appendix E. 10 (Tables 23-24) of the revised paper.

AuthorsOpenReview ↗

Official Authors’ Comments to Reviewer YT6V (2)

R1 W1-2

The paper doesn't show a setting where the converted model is actually usable, for example after some continued training or router fine-tuning, which is how LLaMA-MoE v2 and CMoE are normally used.

Answer of R1 W1-2

Our partition achieves higher accuracy on Qwen and comparable accuracy on OPT after training. We follow each baseline's original training procedure: router fine-tuning for LLaMA-MoE v2, and router fine-tuning and expert fine-tuning (continued training) for CMoE. We use the following notation:

  • LLaMA-MoE v2$^\dagger$: LLaMA-MoE v2 partition + LLaMA-MoE v2 router.
  • CMoE$^\ddagger$: CMoE partition + CMoE router.
  • Ours$^\dagger$ and Ours$^\ddagger$: our partition + the LLaMA-MoE v2 and CMoE routers, respectively.

For each router, we apply the same training procedure to the baseline partition and our partition. All converted models are evaluated at 25% expert activation sparsity.

\begin{array}{|c|c|c|c|} \hline \text{Method} & \text{LLaMA-3 8B} & \text{OPT-2.7B} & \text{Qwen-2.5 7B} \\ \hline \text{Dense} & 79.39 & 62.10 & 80.24 \\ \hline \text{Llama-MoE v2}^\dagger & \textbf{66.10} & 35.09 & 68.00 \\ \text{Ours}^\dagger & 63.69 & \textbf{35.17} & \textbf{70.06} \\ \hline \text{CMoE}^\ddagger & \textbf{54.06} & 55.01 & 69.21 \\ \text{Ours}^\ddagger & 53.46 & \textbf{56.83} & \textbf{70.45} \\ \hline \end{array}

As shown in the table, our partition achieves higher average accuracy on Qwen-2.5 7B under both router settings and comparable accuracy on OPT-2.7B, while accuracy on LLaMA-3 8B remains below the corresponding baselines. We acknowledge that a gap from the dense model remains under these training budgets. Further experimental details are provided in Section 6.1 Robustness across Router Types, Table 3 (Types 2 and 3), and Appendix E. 1 of the revised manuscript.

AuthorsOpenReview ↗

Official Authors’ Comments to Reviewer YT6V (3)

R1 W2

The shared router may be unfair to the baselines. All methods have to use the MoEfication centroid router, and shared experts are removed. On LLaMA-3 8B, LLaMA-MoE v2 and MoEfication fall to around chance (35-37% average, SciQ about 25%), which is far below what those methods report in their own papers. To me this looks more like their partitions not fitting this router than their partitions being bad. Please also report each baseline with its own routing, and with a small amount of router training. The claim is about partition quality, but the current setup mixes partition quality with how well each partition suits the chosen router.

Answer of R1 W2

SLEP-MoEfication achieves higher average accuracy than the baselines even when they retain their original routers and shared experts. To address the concern about the shared-router evaluation, we evaluate LLaMA-MoE v2$^\ast$ and CMoE$^\ast$ with their native routers and shared experts, following their corresponding router fine-tuning or continued-training procedures.

For comparison with these native configurations, Ours$^\ast$ combines our partition with a router using mean and standard deviation statistics instead of the mean-only centroid router, without shared experts.

All experiments use 25% expert activation sparsity. The tables below report the mean and standard deviation of accuracy across three seeds.

\hline \text{Model} & \text{Method} & \text{BoolQ} & \text{PIQA} & \text{SciQ} & \text{WinoGrande} & \text{ARC-C} & \text{HellaSwag} & \text{avg.} \\ \hline & \text{Llama-MoE v2}^\ast & \textbf{70.08}(\pm 0.81) & 73.10(\pm 1.25) & \textbf{90.83}(\pm 1.37) & 66.11(\pm 0.53) & 41.10(\pm 0.32) & 66.33(\pm 0.10) & 67.93(\pm 0.12) \\ \text{LlaMA-3 8B} & \text{CMoE}^\ast & 67.38(\pm 2.08) & 72.42(\pm 0.41) & 89.73(\pm 1.77) & \textbf{68.03}(\pm 0.39) & 46.39(\pm 2.78) & 69.10(\pm 1.24) & 68.84(\pm 0.95) \\ & \text{Ours}^\ast & 69.25(\pm 1.61) & \textbf{75.77}(\pm 0.99) & 89.57(\pm 1.10) & 67.93(\pm 0.56) & \textbf{46.99}(\pm 1.46) & \textbf{72.15}(\pm 0.26) & \textbf{70.27}(\pm 0.59) \\ \hline & \text{Llama-MoE v2}^\ast & 60.55(\pm 2.10) & 73.52(\pm 0.27) & 79.37(\pm 0.21) & 60.93(\pm 0.99) & \textbf{35.41}(\pm 0.39) & 57.59(\pm 0.21) & 61.23(\pm 0.52) \\ \text{OPT-2.7B} & \text{CMoE}^\ast & 61.94(\pm 0.97) & 70.40(\pm 0.25) & 79.33(\pm 1.62) & 58.35(\pm 1.42) & 31.57(\pm 0.81) & 51.85(\pm 0.40) & 58.91(\pm 0.36) \\ & \text{Ours}^\ast & \textbf{64.43}(\pm 1.22) & \textbf{73.70}(\pm 0.11) & \textbf{80.30}(\pm 0.80) & \textbf{61.19}(\pm 0.95) & 34.87(\pm 0.82) & \textbf{58.40}(\pm 0.32) & \textbf{62.15}(\pm 0.19) \\ \hline & \text{LLama-MoE v2}^\ast & \textbf{82.62}(\pm 2.36) & 73.45(\pm 0.38) & \textbf{94.23}(\pm 0.42) & 66.17(\pm 1.83) & 46.87(\pm 1.66) & 65.67(\pm 0.39) & 71.50(\pm 0.43) \\ \text{Qwen-2.5 7B} & \text{CMoE}^\ast & 81.47(\pm 3.48) & 73.50(\pm 0.36) & 92.73(\pm 1.46) & 68.11(\pm 0.93) & 49.69(\pm 1.91) & 68.18(\pm 0.47) & 72.28(\pm 0.60) \\ & \text{Ours}^\ast & 82.42(\pm 0.97) & \textbf{78.06}(\pm 0.61) & \textbf{94.23}(\pm 0.81) & \textbf{69.61}(\pm 1.06) & \textbf{57.59}(\pm 1.19) & \textbf{74.35}(\pm 0.18) & \textbf{76.04}(\pm 0.41) \\ \hline \end{array}$$

Our method achieves the highest average accuracy on all three models. Together with the activation-based routing results in W1-1 and the comparisons using the same router and training procedure in W1-2, these results support the effectiveness of our partition across multiple routing designs. Further experimental details are provided in Appendix E.1, Table 10 of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.