- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SLEP-MoEfication, a data-free method for converting pre-trained dense language models into sparse Mixture-of-Experts (MoE) models. The method partitions FFN neurons into experts by maximizing a subspace coverage objective, aiming to preserve the original model's information while reducing inference computation. A Sinkhorn-based optimization algorithm is used for expert partitioning, together with a training-free centroid router. Experiments on three language models show improvements over several existing partitioning methods, particularly on LLaMA-3 8B.
Critical strengths
- The paper addresses a problem of improving inference efficiency by converting pre-trained dense models into sparse MoE architectures.
- The proposed subspace coverage objective provides an interesting perspective on expert partitioning, supported by a spectral analysis in Section 5.3.
- The experiments cover three language models and include comparisons with existing methods, random partitioning, information preservation analysis, and load balancing evaluation.
Critical weaknesses
- The connection between subspace coverage and information preservation is insufficiently justified (Section 5.2). The objective only considers the input or gate projection, ignoring other FFN projections and nonlinearities.
- The data-free claim is questionable (Appendix C.4), as the subspace rank is selected using the same downstream benchmarks reported in the main experiments. Independent validation is needed.
- Performance gains are inconsistent across models (Table 1). Improvements are substantial on LLaMA-3 8B but marginal on Qwen2.5-7B. Further analysis is needed to explain these differences.
- The experiments lack statistical validation (Table 1). Multiple random seeds and standard deviations are needed to establish the reliability of the improvements.
- The information preservation analysis relies heavily on GMI correlations (Section 6.1). Direct FFN reconstruction errors or output similarity measurements would provide stronger evidence.
- The implementation optimizes normalized weights, while the theoretical objective uses original weights (Equation 6 and Appendix C.1). An ablation study is needed to justify this difference.
- The efficiency evaluation only reports FLOPs (Table 3). Actual inference latency, throughput, and conversion costs should also be evaluated.
Questions for authors
see weeknesses
7 comments
Official Authors’ Comments to Reviewer n1tY (2)
R3 W2
The data-free claim is questionable (Appendix C.4), as the subspace rank is selected using the same downstream benchmarks reported in the main experiments. Independent validation is needed.
Answer of R3 W2
Our data-free claim applies to partition construction for a specified rank. This stage uses only pretrained weights without calibration data, while rank selection is data-dependent. We clarify this distinction in the revised manuscript. Validation-based rank selection preserves higher mean evaluation accuracy than the best-performing baselines on all three models. The original ranks were selected using six-task evaluation accuracy. To assess whether the reported gains depend on this choice, we conduct additional experiments using ARC-C and SciQ validation accuracy and held-out C4 perplexity. These validation data are separate from the final evaluation sets. Final accuracy results are means and standard deviations across three partition seeds.
1) ARC-C and SciQ validation accuracy. We use only these two tasks because the validation splits of BoolQ, PIQA, WinoGrande, and HellaSwag are already used for final evaluation in our setup. The table reports average validation accuracy.
\hline \text{Model} & \text{r=24} & \text{r=25} & \text{r=26} & \text{r=27} & \text{r=28} \\ \hline \text{LLaMA-3-8B} & 54.51 & 55.95 & \textbf{59.37} & 53.09 & 53.09 \\ \hline \text{OPT-2.7B} &57.93 & 55.95 & 57.97 & \textbf{58.55} & 57.70 \\ \hline \text{Qwen2.5-7B} & 74.59 & 74.39 & 73.53 & \textbf{74.60} & 73.31 \\ \hline \end{array}$$
This criterion selects ranks 26, 27, and 27 for LLaMA, OPT, and Qwen, respectively. LLaMA and Qwen retain their original ranks. OPT changes to rank 27 and achieves , exceeding the best baseline, MoEfication, at (-value = 0.011).
2) C4 validation perplexity. We select the rank with the lowest held-out C4 perplexity without using accuracy on any of the six tasks.
\hline \text{Model} & \text{r=24} & \text{r=25} & \text{r=26} & \text{r=27} & \text{r=28} \\ \hline \text{LLaMA-3-8B} &30.79 & 26.74 & 25.03 & \textbf{24.74} & 30.28\\ \hline \text{OPT-2.7B} & 15.80 & 15.79 & \textbf{15.68} & 15.73 & 15.77 \\ \hline \text{Qwen2.5-7B} & 13.31 & 13.38 & \textbf{13.18} & 13.19 & 13.43 \\ \hline \end{array}$$
C4-based selection gives r=27, 26, and 26 for LLaMA-3 8B, OPT-2.7B, and Qwen-2.5 7B, respectively. The six-task evaluation results at the selected ranks are compared with the best-performing baselines below:
- LLaMA-3 8B: CMoE achieves , while our method achieves at (-value is 0.0092).
- OPT-2.7B: MoEfication achieves , while our method achieves and -value is 0.0025.
- Qwen-2.5 7B: CMoE achieves , while our method achieves at (-value is 0.3429).
These results retain statistically significant improvements on LLaMA and OPT and a numerical improvement on Qwen.
Both validation criteria preserve higher mean evaluation accuracy than the best-performing baselines. Under C4-based selection, the improvements remain statistically significant on LLaMA and OPT, while Qwen retains a numerical improvement. These results support the performance comparisons under validation-based rank selection.
To obtain a single rank per model in the additional validation analysis, we combine the two criteria using equally weighted z-scores.
Official Authors’ Comments to Reviewer n1tY (3)
R3 W4
(R3 W4) The experiments lack statistical validation (Table 1). Multiple random seeds and standard deviations are needed to establish the reliability of the improvements.
Answer of R3 W3 & R3 W4
Across three seeds, SLEP-MoEfication achieves higher mean accuracy on all three models, with statistically significant improvements on LLaMA and OPT. To assess variability across runs, we evaluate each method on each model using three random seeds. We report the mean and standard deviation for each task and the six-task average. These results have also been incorporated into Table 1 of the revised manuscript.
\hline \text{Model} & \text{Method} & \text{C.D.} & \text{BoolQ} & \text{PIQA} & \text{SciQ} & \text{WinoGrande} & \text{ARC-C} & \text{HellaSwag} & \text{Avg.} \\ \hline & \text{Dense} & - & 83.70 & 80.69 & 94.60 & 77.35 & 59.47 & 80.56 & 79.39 \\ & \text{Llama-MoE v2} & \text{O} & 38.67(\pm 0.98) & 51.54(\pm 1.13) & 25.80(\pm 3.94) & 49.12(\pm 0.39) & 24.60(\pm 1.58) & 26.19(\pm 0.49) & 35.99(\pm 0.55) \\ \text{LLaMA-3 8B} & \text{CMoE} & \text{O} & 56.27(\pm 3.76) & 63.18(\pm 2.41) & 73.43(\pm 2.62) & 54.85(\pm 2.38) & 27.67(\pm 0.91) & 47.85(\pm 3.84) & 53.88(\pm 1.22) \\ & \text{MoEfication} & \text{X} & 41.03(\pm 1.40) & 51.16(\pm 0.52) & 24.20(\pm 0.30) & 50.49(\pm 0.57) & 26.25(\pm 0.49) & 26.63(\pm 0.19) & 36.63(\pm 0.23) \\ & \text{Ours} & \text{X} & \textbf{56.89}(\pm 1.63) & \textbf{68.81}(\pm 1.59) & \textbf{78.40}(\pm 1.40) & \textbf{57.72}(\pm 1.31) & \textbf{35.13}(\pm 0.30) & \textbf{59.80}(\pm 0.42) & \textbf{59.46}(\pm 0.89) \\ \hline & \text{Dense} & - & 61.87 & 74.76 & 80.60 & 61.33 & 34.30 & 59.73 & 62.10 \\ & \text{Llama-MoE v2} & \text{O} & 57.30(\pm 3.96) & 72.98(\pm 0.64) & 76.03(\pm 0.81) & 60.96(\pm 0.16) & 33.87(\pm 0.09) & 57.17(\pm 0.55) & 59.72(\pm 0.66) \\ \text{OPT-2.7B} & \text{CMoE} & \text{O} & 59.01(\pm 0.46) & 72.71(\pm 0.25) & 78.93(\pm 0.59) & \textbf{61.35}(\pm 0.20) & 33.62(\pm 0.23) & 57.77(\pm 0.04) & 60.56(\pm 0.21) \\ & \text{MoEfication} & \text{X} & 58.33(\pm 1.47) & \textbf{73.87}(\pm 0.08) & \textbf{80.90}(\pm 0.36) & 61.12(\pm 0.46) & 34.27(\pm 0.42) & 58.18(\pm 0.16) & 61.11(\pm 0.34) \\ & \text{Ours} & \text{X} & \textbf{64.89}(\pm 0.57) & 73.81(\pm 0.58) & 80.07(\pm 0.80) & 61.17(\pm 1.17) & \textbf{34.44}(\pm 0.71) & \textbf{58.49}(\pm 0.26) & \textbf{62.15}(\pm 0.45) \\ \hline & \text{Dense} & - & 88.13 & 79.98 & 95.20 & 75.30 & 64.33 & 78.47 & 80.24 \\ & \text{Llama-MoE v2} & \text{O} & 69.48(\pm 2.42) & 67.50(\pm 2.04) & 90.63(\pm 0.60) & 57.75(\pm 1.21) & 39.14(\pm 2.01) & 56.46(\pm 0.32) & 63.49(\pm 0.85) \\ \text{Qwen-2.5 7B} & \text{CMoE} & \text{O} & \textbf{83.58}(\pm 0.48) & 77.51(\pm 0.55) & 93.17(\pm 0.60) & 68.32(\pm 1.18) & 55.23(\pm 0.64) & 74.16(\pm 0.11) & 75.33(\pm 0.16) \\ & \text{MoEfication} & \text{X} & 79.59(\pm 3.60) & 76.91(\pm 0.69) & 90.67(\pm 1.65) & 67.27(\pm 0.63) & 54.69(\pm 1.70) & \textbf{75.11}(\pm 0.16) & 74.04(\pm 0.69) \\ & \text{Ours} & \text{X} & 82.86(\pm 0.47) & \textbf{77.86}(\pm 0.16) & \textbf{94.03}(\pm 0.06) & \textbf{68.67}(\pm 0.96) & \textbf{56.51}(\pm 1.21) & 73.99(\pm 0.15) & \textbf{75.65}(\pm 0.18) \\ \hline \end{array}$$
Compared with the best-performing baseline on each model, SLEP-MoEfication improves the six-task average by 5.58 percentage points on LLaMA-3 8B, 1.04 points on OPT-2.7B, and 0.32 points on Qwen-2.5 7B. The improvements on LLaMA and OPT are statistically significant at the 0.05 significance level. On Qwen, the gain remains modest and does not reach statistical significance at this threshold (); we therefore report it as a numerical improvement.
Official Authors’ Comments to Reviewer n1tY (1)
Common Comments
R3 W1
Answer of R3 W1