- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper converts dense FFNs into MoE layers by running NMF on activation magnitudes from calibration data. One factor decides how neurons are grouped into experts, and the other is predicted per token and used for routing. Only the router is trained; the original weights stay frozen. On LLaMA-3-8B with a quarter of the experts active, the method beats CMoE and gives a modest prefill speedup.
Critical strengths
-
Using the two factors of one decomposition for partitioning and for routing is an elegant idea, and the ablation shows both parts matter.
-
The speedups come from actually running the experts rather than from FLOP estimates, and the authors are honest that decoding doesn't get faster.
Critical weaknesses
-
I couldn't figure out how the KL and CE losses actually train the router. The paper says the top-k experts are selected and that these losses update the predictor, but it never writes out the FFN output used in training, and I don't see gate weights applied to the expert outputs anywhere. If the selected experts' outputs are just summed, the loss doesn't depend on the router scores unless the selected set changes, so KL and CE give no gradient and only the z-loss and balance loss would train the router. Maybe there is a weighting or a straight-through trick that just isn't mentioned, but since the router is the only trained part, this needs to be clear.
-
The accuracy loss is large for the speedup. The average drops from about 78 to 58 for a 1.1-1.2x prefill gain. Some tasks are near chance (WinoGrande, ARC-C), and BoolQ is below the majority-class baseline. Calling this "competitive" in the abstract feels like a stretch. Only one sparsity level is tested, so I don't know whether a less aggressive setting would give a better trade-off. I'd also like to see a standard learned router on the same expert partition, since the alternative in the ablation is an analytical router. Finally, the conversion cost isn't fully reported; the token count covers only the last alignment stage.
Questions for authors
-
What exactly is the forward computation during alignment training, and how do the KL/CE gradients reach the router? Does training differ from inference?
-
What losses, how many steps and what data are used for predictor initialization and warmup? Where does the alignment data come from, and does it overlap with the calibration data? What are the total conversion time and memory?
-
How does a standard learned router do on the same partition? Have you tried a higher active ratio, say 40-50%?
5 comments
Q2. Training configurations, data, and conversion cost
The router optimization consists of three stages:
1. Coefficient predictor pretraining. Each layer uses a 4096-to-256-to-128 MLP to predict NMF coefficients obtained through non-negative least squares (NNLS), which reconstructs activation features using the fixed NMF dictionary. The training objective is
where measures relative squared activation reconstruction error, and measures the squared distance between normalized predicted and NNLS coefficient vectors. This stage reuses the same 32,768 calibration tokens used for NMF fitting. Training uses a batch size of 512 and an average of 20.66 epochs per layer.
2. Expert-distribution warmup. We further train the coefficient predictor to match the expert distributions derived from offline NMF coefficients using
where is the NMF-derived target expert distribution and is the predicted expert distribution. This stage reuses the same 32,768 calibration tokens, with eight epochs, batch size 2,048, and 128 optimizer steps per layer. The NMF factors, expert partition, and initial factor-to-expert mapping remain fixed.
3. Model-level alignment. We optimize the coefficient predictors and mapping adapters for 128 steps using 512 sequences of length 256 (131,072 input-token exposures). The objective is
Here, KL distillation uses the frozen dense model as teacher, while CE optimizes next-token prediction. The auxiliary terms regularize router logits and expert utilization. Alignment improves the five-task average accuracy by approximately 2.45 percentage points relative to the pretrained and warmed-up router.
All data are sampled from Dolmino-Mix. NMF calibration and model-level alignment use document-disjoint splits, while coefficient predictor pretraining and expert-distribution warmup reuse the original NMF calibration data.
We have also profiled the complete conversion pipeline on an Intel Core i7-13700KF and a single RTX 4090:
- Activation factorization (activation collection, NMF fitting, and NNLS encoding): 26 min
- Expert construction: 4 min
- Router initialization (predictor pretraining and warmup): 8 min
- Model-level alignment: 7 min
- Total conversion time: 45 min
The recorded peak GPU memory usage and CPU resident set size (RSS) were 21.45 GiB and 8.31 GiB, respectively. Importantly, the 0.13M input-token exposure count refers only to the final model-level alignment stage, whereas the reported conversion time accounts for the entire pipeline.
Full training configurations and conversion costs are provided in Appendices A.4 and A.7.
Q3. Standard learned router and higher activation ratios
Standard learned router. To provide a fair comparison, we evaluate two direct MLP router baselines using the same NMF expert partition, with 112 balanced physical experts and Top-28 routing.
A direct MLP trained from random initialization may be disadvantaged relative to our NMF-pretrained router due to the absence of supervised initialization. We therefore include both a supervised direct MLP and a randomly initialized direct MLP.
Both use a 4096-to-256-to-112 architecture. The supervised variant learns expert-level activation distributions from the same calibration data, receiving approximately 950K pretraining token exposures per layer, compared with approximately 939K for our NMF-guided initialization. The randomly initialized variant skips pretraining. All three methods use the same 128-step model-level alignment budget.
The five-task average accuracies are:
- NMF-guided MLP (Ours): 57.93%
- Supervised Direct MLP: 55.82%
- Randomly Initialized Direct MLP: 50.38%
Our NMF-guided router outperforms the supervised direct MLP by 2.11 percentage points under the same expert partition and alignment budget. The randomly initialized baseline further illustrates the importance of supervised router initialization. Detailed results are provided in Section 5.7 and Appendix A.5.
Higher active FFN ratios. To investigate the quality–efficiency trade-off, we additionally evaluate our method at 50% active FFN capacity, compared with the original 25% setting.
The following table reports the five-task average accuracy under the 50% FFN sparsity setting:
| Method | Avg. Accuracy (%) |
|---|---|
| Dense (100% FFN) | 78.11 |
| Wanda (Unstructured) | 74.59 |
| R-Sparse | 72.38 |
| LLM-Pruner | 53.83 |
| SliceGPT | 52.37 |
| CMoE | 59.47 |
| CMoE (Finetuned) | 64.58 |
| Ours (50% Active FFN) | 70.37 |
We also measure end-to-end prefill performance at 50% active FFN capacity on a single NVIDIA RTX 4090:
| Seq. Length | Dense Latency (ms) | Ours Latency (ms) | Speedup |
|---|---|---|---|
| 512 | 59.58 | 54.70 | 1.08 |
| 1024 | 142.34 | 128.23 | 1.11 |
| 2048 | 347.00 | 312.61 | 1.11 |
Increasing the active FFN capacity from 25% to 50% improves our average accuracy from 57.93% to 70.37%, reducing the gap to the original dense model while retaining measured end-to-end prefill acceleration.
These results further demonstrate the flexibility of our NMF-guided dense-to-MoE conversion framework. By using the same factorized activation representation for both expert construction and routing, our method enables different expert activation budgets while preserving the original dense-model weights. The measured prefill speedups also confirm that the resulting physical experts can provide practical inference benefits, with further opportunities for implementation-level optimization.
We thank the reviewer for the detailed feedback and for recognizing the value of our unified NMF-based expert construction and routing framework, as well as the measured inference speedups from physical expert execution. In response to the reviewer's concerns, we have clarified the router training mechanism, provided a complete accounting of conversion costs, added controlled comparisons with standard learned routers, and extended our evaluation to higher FFN activation budgets. We address each question below.
Q1. How do KL and CE gradients reach the router?
We use a straight-through estimator (STE) for hard Top-k expert selection. Given the hard selection mask rthard and differentiable expert distribution pt, we define
rtSTE=rthard+pt−sg(pt)
where sg denotes stop-gradient. The sparse FFN output is
F^(zt)=Wdown[h(zt)⊙(MrtSTE)],
where h(zt) is the original SwiGLU intermediate activation and M is the fixed neuron-to-expert assignment matrix.
The forward pass uses unweighted hard Top-k selection. Selected expert outputs are summed without multiplying them by routing probabilities. However, during backpropagation, STE allows the KL and CE gradients to flow through pt to the coefficient predictor and the factor-to-expert mapping adapter.
Training uses a masked FFN implementation to support STE gradients, whereas inference executes only the selected physical experts. Both have the same hard-selection forward computation. The original FFN weights and expert assignments remain frozen.
These details and the explicit forward equation have been added to Section 4.3.