acceptodds
Under review as a conference paper at ICLR 2027

Activation Factorization for Dense-to-MoE Conversion

Abstract

Converting pretrained dense language models into Mixture-of-Experts (MoE) architectures can enable structured sparse execution without pretraining a new model from scratch. The quality of the converted model, however, depends on how neuron activation structure is represented for expert construction and rout- ing. We propose a dense-to-MoE conversion framework based on non-negative matrix factorization (NMF) of FFN activation magnitudes. The factorization provides complementary neuron-side factor loadings and token-side factor co- efficients. We use the neuron-side representation to construct balanced physi- cal experts, while a lightweight predictor estimates token coefficients and maps them to expert-level routing scores through the same factorized activation struc- ture. The original dense-model parameters remain frozen, with only lightweight routing modules adapted after conversion. On LLaMA-3-8B with 25% active FFN capacity, our method achieves an average accuracy of 57.93% across five downstream benchmarks, exceeding the average accuracy of all evaluated CMoE configurations. Controlled ablations show that both NMF-based expert construc- tion and NMF-guided routing contribute to the final performance. On a single NVIDIA RTX 4090, the resulting physical expert execution achieves 1.13–1.22× end-to-end prefill speedup across sequence lengths from 512 to 2048. These re- sults show that activation factorization provides an effective representation for low-cost dense-to-MoE conversion with structured sparse execution

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Mixture of Experts, Efficient Inference, Model Compression, Sparse Neural Networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_JKcBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper converts dense FFNs into MoE layers by running NMF on activation magnitudes from calibration data. One factor decides how neurons are grouped into experts, and the other is predicted per token and used for routing. Only the router is trained; the original weights stay frozen. On LLaMA-3-8B with a quarter of the experts active, the method beats CMoE and gives a modest prefill speedup.

Critical strengths
  1. Using the two factors of one decomposition for partitioning and for routing is an elegant idea, and the ablation shows both parts matter.

  2. The speedups come from actually running the experts rather than from FLOP estimates, and the authors are honest that decoding doesn't get faster.

Critical weaknesses
  1. I couldn't figure out how the KL and CE losses actually train the router. The paper says the top-k experts are selected and that these losses update the predictor, but it never writes out the FFN output used in training, and I don't see gate weights applied to the expert outputs anywhere. If the selected experts' outputs are just summed, the loss doesn't depend on the router scores unless the selected set changes, so KL and CE give no gradient and only the z-loss and balance loss would train the router. Maybe there is a weighting or a straight-through trick that just isn't mentioned, but since the router is the only trained part, this needs to be clear.

  2. The accuracy loss is large for the speedup. The average drops from about 78 to 58 for a 1.1-1.2x prefill gain. Some tasks are near chance (WinoGrande, ARC-C), and BoolQ is below the majority-class baseline. Calling this "competitive" in the abstract feels like a stretch. Only one sparsity level is tested, so I don't know whether a less aggressive setting would give a better trade-off. I'd also like to see a standard learned router on the same expert partition, since the alternative in the ablation is an analytical router. Finally, the conversion cost isn't fully reported; the token count covers only the last alignment stage.

Questions for authors
  1. What exactly is the forward computation during alignment training, and how do the KL/CE gradients reach the router? Does training differ from inference?

  2. What losses, how many steps and what data are used for predictor initialization and warmup? Where does the alignment data come from, and does it overlap with the calibration data? What are the total conversion time and memory?

  3. How does a standard learned router do on the same partition? Have you tried a higher active ratio, say 40-50%?

5 comments

We thank the reviewer for the detailed feedback and for recognizing the value of our unified NMF-based expert construction and routing framework, as well as the measured inference speedups from physical expert execution. In response to the reviewer's concerns, we have clarified the router training mechanism, provided a complete accounting of conversion costs, added controlled comparisons with standard learned routers, and extended our evaluation to higher FFN activation budgets. We address each question below.

Q1. How do KL and CE gradients reach the router?

We use a straight-through estimator (STE) for hard Top-k expert selection. Given the hard selection mask rthardr_t^{hard} and differentiable expert distribution ptp_t, we define

rtSTE=rthard+pt−sg⁡(pt)r_t^{STE}=r_t^{hard}+p_t-\operatorname{sg}(p_t)

where sg⁡\operatorname{sg} denotes stop-gradient. The sparse FFN output is

F^(zt)=Wdown[h(zt)⊙(MrtSTE)],\hat{F}(z_t)=W_{down}\left[h(z_t)\odot(Mr_t^{STE})\right],

where h(zt)h(z_t) is the original SwiGLU intermediate activation and MM is the fixed neuron-to-expert assignment matrix.

The forward pass uses unweighted hard Top-k selection. Selected expert outputs are summed without multiplying them by routing probabilities. However, during backpropagation, STE allows the KL and CE gradients to flow through ptp_t to the coefficient predictor and the factor-to-expert mapping adapter.

Training uses a masked FFN implementation to support STE gradients, whereas inference executes only the selected physical experts. Both have the same hard-selection forward computation. The original FFN weights and expert assignments remain frozen.

These details and the explicit forward equation have been added to Section 4.3.

Q2. Training configurations, data, and conversion cost

The router optimization consists of three stages:

1. Coefficient predictor pretraining. Each layer uses a 4096-to-256-to-128 MLP to predict NMF coefficients obtained through non-negative least squares (NNLS), which reconstructs activation features using the fixed NMF dictionary. The training objective is

Lpred=Lrecon+0.1LcoeffL_{pred} = L_{recon} + 0.1L_{coeff}

where LreconL_{recon} measures relative squared activation reconstruction error, and LcoeffL_{coeff} measures the squared distance between normalized predicted and NNLS coefficient vectors. This stage reuses the same 32,768 calibration tokens used for NMF fitting. Training uses a batch size of 512 and an average of 20.66 epochs per layer.

2. Expert-distribution warmup. We further train the coefficient predictor to match the expert distributions derived from offline NMF coefficients using

Lwarmup=KL(pt∗∣∣pt)L_{warmup} = KL(p_t^* || p_t)

where pt∗p_t^* is the NMF-derived target expert distribution and ptp_t is the predicted expert distribution. This stage reuses the same 32,768 calibration tokens, with eight epochs, batch size 2,048, and 128 optimizer steps per layer. The NMF factors, expert partition, and initial factor-to-expert mapping remain fixed.

3. Model-level alignment. We optimize the coefficient predictors and mapping adapters for 128 steps using 512 sequences of length 256 (131,072 input-token exposures). The objective is

L=2LKL+LCE+0.001Lz+0.01LbalanceL = 2L_{KL} + L_{CE} + 0.001L_z + 0.01L_{balance}

Here, KL distillation uses the frozen dense model as teacher, while CE optimizes next-token prediction. The auxiliary terms regularize router logits and expert utilization. Alignment improves the five-task average accuracy by approximately 2.45 percentage points relative to the pretrained and warmed-up router.

All data are sampled from Dolmino-Mix. NMF calibration and model-level alignment use document-disjoint splits, while coefficient predictor pretraining and expert-distribution warmup reuse the original NMF calibration data.

We have also profiled the complete conversion pipeline on an Intel Core i7-13700KF and a single RTX 4090:

  • Activation factorization (activation collection, NMF fitting, and NNLS encoding): 26 min
  • Expert construction: 4 min
  • Router initialization (predictor pretraining and warmup): 8 min
  • Model-level alignment: 7 min
  • Total conversion time: 45 min

The recorded peak GPU memory usage and CPU resident set size (RSS) were 21.45 GiB and 8.31 GiB, respectively. Importantly, the 0.13M input-token exposure count refers only to the final model-level alignment stage, whereas the reported conversion time accounts for the entire pipeline.

Full training configurations and conversion costs are provided in Appendices A.4 and A.7.

Q3. Standard learned router and higher activation ratios

Standard learned router. To provide a fair comparison, we evaluate two direct MLP router baselines using the same NMF expert partition, with 112 balanced physical experts and Top-28 routing.

A direct MLP trained from random initialization may be disadvantaged relative to our NMF-pretrained router due to the absence of supervised initialization. We therefore include both a supervised direct MLP and a randomly initialized direct MLP.

Both use a 4096-to-256-to-112 architecture. The supervised variant learns expert-level activation distributions from the same calibration data, receiving approximately 950K pretraining token exposures per layer, compared with approximately 939K for our NMF-guided initialization. The randomly initialized variant skips pretraining. All three methods use the same 128-step model-level alignment budget.

The five-task average accuracies are:

  • NMF-guided MLP (Ours): 57.93%
  • Supervised Direct MLP: 55.82%
  • Randomly Initialized Direct MLP: 50.38%

Our NMF-guided router outperforms the supervised direct MLP by 2.11 percentage points under the same expert partition and alignment budget. The randomly initialized baseline further illustrates the importance of supervised router initialization. Detailed results are provided in Section 5.7 and Appendix A.5.

Higher active FFN ratios. To investigate the quality–efficiency trade-off, we additionally evaluate our method at 50% active FFN capacity, compared with the original 25% setting.

The following table reports the five-task average accuracy under the 50% FFN sparsity setting:

MethodAvg. Accuracy (%)
Dense (100% FFN)78.11
Wanda (Unstructured)74.59
R-Sparse72.38
LLM-Pruner53.83
SliceGPT52.37
CMoE59.47
CMoE (Finetuned)64.58
Ours (50% Active FFN)70.37

We also measure end-to-end prefill performance at 50% active FFN capacity on a single NVIDIA RTX 4090:

Seq. LengthDense Latency (ms)Ours Latency (ms)Speedup
51259.5854.701.08 ×\times
1024142.34128.231.11 ×\times
2048347.00312.611.11 ×\times

Increasing the active FFN capacity from 25% to 50% improves our average accuracy from 57.93% to 70.37%, reducing the gap to the original dense model while retaining measured end-to-end prefill acceleration.

These results further demonstrate the flexibility of our NMF-guided dense-to-MoE conversion framework. By using the same factorized activation representation for both expert construction and routing, our method enables different expert activation budgets while preserving the original dense-model weights. The measured prefill speedups also confirm that the resulting physical experts can provide practical inference benefits, with further opportunities for implementation-level optimization.

Reviewer_7VkvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a dense-to-MoE conversion method based on non-negative matrix factorization (NMF) of FFN activation magnitudes collected from a small calibration set. The neuron-side factor loadings are used to cluster FFN neurons into balanced, non-overlapping experts, while a lightweight predictor estimates token-side factor coefficients and maps them to expert scores for top-k routing. The original model parameters remain frozen, with only the routing modules aligned using approximately 0.13M tokens. On LLaMA-3-8B with 25% active FFN capacity, the method achieves 57.93% average accuracy across five benchmarks, outperforming the evaluated CMoE and pruning baselines on average. Ablation studies show that both NMF-based expert construction and routing contribute to performance. The method also achieves 1.13–1.22× end-to-end prefill speedup on a single RTX 4090.

Critical strengths
  1. Simple, unified formulation. A single non-negative factorization of FFN activations provides both the neuron-side representation for expert construction and the token-side signal for routing. Controlled ablations show that both components contribute (−5.63 and −3.83 points when replaced).
  2. Low conversion cost with structured execution. Original parameters remain frozen and router alignment uses only about 0.13M tokens, while the balanced physical experts yield 1.13–1.22$\times$ end-to-end prefill speedups. The authors transparently report that decoding is not accelerated.
Critical weaknesses
  1. Substantial quality degradation and questionable baselines. The converted model trails the dense model by about 20 percentage points (57.93% vs. 78.11%), performs near chance on ARC-C and WinoGrande, falls below the approximately 62% majority-class baseline on BoolQ, and underperforms training-free R-Sparse (59.36%), although the latter uses finer-grained sparsity. Some pruning baseline results are also unusually low: LLM-Pruner scores 29.14% on BoolQ, below even the weaker constant-prediction baseline (~37.8%), and far below chance on PIQA and WinoGrande. The authors should clarify the evaluation setup and how LLM-Pruner was adapted to LLaMA-3-8B's grouped-query attention. Since several pruning baselines perform near chance at this sparsity level, outperforming them provides limited evidence of competitive model quality.

  2. Missing ablations, comparisons, and limited evaluation scope. The proposed routing mechanism is compared only with CMoE's representative-neuron routing. A learned router with comparable capacity and training budget is needed to determine whether the gains arise specifically from NMF-guided routing or from router training itself. Closely related methods are also missing from the comparisons, including ToMoE (TMLR 2026), which converts frozen dense models into dynamically routed expert structures, and ExpertWeaver (ICML 2026), which constructs experts from GLU activation patterns. Results cover only one model and five tasks, without multi-seed estimates or sensitivity analyses of key hyperparameters (NMF rank, KsK_s, expert size, and calibration size). Finally, decoding throughput remains below that of the dense model (0.90–0.99× in the evaluated settings), whereas methods such as TEAL (ICLR 2025) demonstrate decoding speedups under different sparsity and execution regimes.

Questions for authors
  1. Baseline validity. LLM-Pruner reports unusually low accuracy on BoolQ (29.14%), PIQA (36.51%), and WinoGrande (36.81%), all below simple guessing or constant-prediction baselines. Given that LLaMA-3-8B uses grouped-query attention (GQA), how was LLM-Pruner adapted to this architecture? Could the authors also clarify whether pruning was restricted to FFN layers and specify the evaluation metrics (acc vs. acc_norm), few-shot settings, and harness configuration used for all baselines?

  2. Isolating the contribution of NMF. While Table 4 provides component ablations, how does NMF-guided routing compare with a standard learned MLP router under matched capacity and training budgets? Would alternative representations, such as SVD, achieve similar results? These comparisons would help establish whether the gains arise specifically from NMF or from learned routing and low-rank representations more generally.

  3. Comparable sparsity. Could the authors report the overall active parameter ratio, including non-FFN components such as attention and embeddings, in addition to the reported 25% active FFN capacity? This would enable more meaningful comparisons with related dense-to-MoE conversion methods such as ToMoE, which reports model-level active parameter ratios.

13 comments

We thank the reviewer for the detailed and constructive feedback, particularly the suggestions regarding baseline validity, the contribution of NMF-guided routing, and comparable sparsity budgets. In response, we have corrected and clarified the baseline evaluations, added controlled learned-router comparisons and additional dense-to-MoE benchmarks, and extended our evaluation to higher FFN activation budgets.

Q1. Baseline Validity and Evaluation Setup

Corrected LLM-Pruner Results. Following the reviewer's observation, we re-examined and re-evaluated LLM-Pruner under our FFN-only pruning configuration. The corrected results are:

BenchmarkOriginal (%)Corrected (%)
BoolQ29.1443.43
SciQ43.5839.60
PIQA36.5152.72
ARC-Challenge17.9419.37
WinoGrande36.8149.33
Average32.8040.89

These corrected values replace the previously reported results. We also have verified the remaining baseline results.

Baseline Configurations. To ensure a consistent comparison, we adapted the baselines to operate exclusively on the FFN modules of LLaMA-3-8B, keeping attention (including GQA), embeddings, and all other components unchanged.

  • LLM-Pruner: First-order Taylor-based structured pruning across all 32 FFN layers, jointly removing corresponding gate/up output channels and down input channels.
  • Wanda: Row-wise unstructured pruning of FFN projection weights based on activation-aware importance scores.
  • R-Sparse: FFN-only activation sparsification with low-rank compensation, configured to approximate the target FFN computation budget.
  • SliceGPT: PCA-based compression of FFN input/output projection dimensions, with projection overhead included in the computation budget.

These configurations follow a common FFN-only modification scope, with sparsity budgets aligned according to each method's compression mechanism. We will include the detailed configurations in the revised manuscript.

Evaluation Protocol. All methods were evaluated using lm-evaluation-harness v0.3.0, reporting acc. The few-shot settings were BoolQ (32), SciQ (0), PIQA (0), WinoGrande (5), and ARC-Challenge (25).

Q2. Isolating the Contribution of NMF

1. Comparison with Standard Learned MLP Routers

We added two direct MLP baselines using the same NMF-derived expert partition and Top-28 execution budget.

The Supervised Direct MLP (4096→256→112) predicts expert scores directly, supervised by aggregated sparsified FFN activation magnitudes. Both methods use the same 32,768 calibration tokens, comparable initialization training budgets (950K vs. 939K layer-token exposures), and identical 128-step model-level alignment. We also evaluate a randomly initialized direct MLP trained only through model-level alignment.

Routing MethodAverage Accuracy (%)
NMF-guided MLP (Ours)57.93
Direct MLP (Supervised)55.82
Direct MLP (Scratch)50.38

Our NMF-guided MLP outperforms the supervised direct MLP by 2.11 percentage points, demonstrating the benefit of factorization-guided routing beyond standard learned expert-score prediction.

Detailed training configurations and per-task results are provided in Appendix A.5.

2. Comparison with ExpertWeaver and ToMoE

We additionally evaluated our method under the 25% FFN sparsity setting and five benchmark configurations reported by ExpertWeaver.

BenchmarkExpertWeaver (Published)Ours
MMLU (5-shot)47.0050.37
HellaSwag (10-shot)69.8074.72
ARC-Easy (0-shot)64.4070.23
ARC-Challenge (25-shot)44.3045.26
PIQA (0-shot)76.3077.42
Average60.3663.60

Our reported five-task average is 63.60%, compared with 60.36% calculated from ExpertWeaver’s published task scores.

The ExpertWeaver results are taken from its published Table 2, as an official implementation was not publicly available. Our evaluation uses acc_norm for HellaSwag and acc for the remaining tasks; ExpertWeaver's table does not explicitly specify the metric for each task. ExpertWeaver uses training-free routing, whereas our method includes lightweight router training.

ToMoE differs from our FFN-only setting by jointly optimizing expert construction and routing through dynamic structural pruning of both attention and FFN layers. Its reported model-level active parameter budgets and evaluation protocols also differ from ours, making a direct comparison under matched conditions inappropriate. We therefore discuss ToMoE in the revised Related Work section rather than including its published results in the main comparison table.

3. Alternative Factorizations

Our controlled router comparison establishes the contribution of NMF-guided routing relative to a standard learned MLP. Exploring alternative matrix factorizations, such as SVD, within the same expert construction and routing framework is an interesting direction for future work.

Q3. Comparable Sparsity

We additionally report the model-level active parameter ratios, accounting for all non-FFN components and the routing modules.

LLaMA-3-8B contains approximately 8.03B parameters, including 5.64B FFN parameters. Our framework introduces an additional 35.07M routing parameters.

ConfigurationActive FFN CapacityModel-level Active Parameters
Ours (Top-28)25%47.58%
Ours (Top-56)50%65.05%

These ratios include the unchanged attention, embeddings, language-model head, and other non-FFN components, together with the additional routing parameters. The denominator includes both the original model and the routing parameters.

Model Quality and Quality–Efficiency Trade-off

To further evaluate model quality beyond the aggressive 25% active FFN setting, we extended our experiments to 50% active FFN capacity, including additional comparisons with CMoE and sparsity baselines.

MethodAverage Accuracy (%)
Dense (100% FFN)78.11
Wanda (Unstructured)74.59
R-Sparse72.38
Ours (50% Active FFN)70.37
CMoE (Finetuned)64.58
CMoE59.47
LLM-Pruner53.83
SliceGPT52.37

Increasing the active FFN capacity from 25% to 50% improves our average accuracy from 57.93% to 70.37% (+12.44 percentage points), reducing the gap to the original dense model from 20.18 to 7.74 percentage points.

We also measure end-to-end prefill performance at 50% active FFN capacity on a single NVIDIA RTX 4090:

Seq. LengthDense Latency (ms)Ours Latency (ms)Speedup
51259.5854.701.08 ×\times
1024142.34128.231.11 ×\times
2048347.00312.611.11 ×\times

These results demonstrate that our NMF-guided dense-to-MoE framework maintains competitive model quality at a moderate FFN computation budget while providing measurable inference acceleration.

Evaluation Scope and Decoding Performance

We extended our evaluation to two expert activation budgets: 25% and 50% active FFN capacity. At 50% active FFN capacity, our method achieves 70.37% average accuracy with approximately 1.10× end-to-end prefill speedup, complementing the 25% configuration.

For the learned-router comparison, we evaluated the randomly initialized direct MLP across five random seeds, with detailed results reported in Appendix A.5.

Our current evaluation uses LLaMA-3-8B as the backbone. We will include additional sensitivity experiments on NMF rank, expert size, and calibration set size in the revised manuscript to further examine the robustness of our framework under different configurations.

Regarding decoding performance, our current implementation achieves 0.90–0.99× throughput relative to the dense model, while providing measurable end-to-end prefill acceleration. Our primary contribution is an NMF-guided dense-to-MoE conversion framework that constructs independently executable physical experts with lightweight router training. The observed prefill speedups demonstrate the practical benefits of this conversion, while optimizing routing, dispatch, and expert execution for low-batch decoding represents a separate opportunity for system-level improvements.

Reviewer_qhaHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for converting pretrained dense language-model FFNs into sparsely activated physical experts. It applies non-negative matrix factorization to calibration activation magnitudes, using neuron-side factor loadings to construct balanced experts and token-side coefficients to supervise a lightweight routing predictor. A precomputed factor-to-expert mapping permits routing without reconstructing the full activation vector. The original model parameters remain frozen during subsequent router alignment. On LLaMA-3-8B with 25% active FFN capacity, the method achieves 57.93% average accuracy across five benchmarks, exceeding the strongest evaluated CMoE configuration by 3.46 percentage points. Component ablations support contributions from both expert construction and routing. The implementation achieves 1.13-1.22x end-to-end prefill speedup, while small-batch decoding remains approximately comparable to, or slower than, dense execution.

Critical strengths
  1. Coherent representation for construction and routing. Deriving both components from the same activation factorization provides a clear connection between offline neuron organization and online token-dependent selection.
  2. Executable expert structure. Balanced, non-overlapping experts offer a concrete route to structured sparse execution. Precomputing the factor-to-expert mapping also avoids reconstructing all neuron activity during inference.
  3. Useful component ablations. Table 4 holds the backbone, 112-expert structure, Top-28 budget, and router-training budget fixed. NMF construction improves average accuracy from 52.30% to 57.93% with NMF routing held fixed. NMF routing improves accuracy from 54.10% to 57.93% with the NMF partition fixed. These are substantive positive results, not effects attributable merely to a different expert count or nominal training budget.
  4. Measured runtime results and transparent limitations. End-to-end GPU measurements strengthen the paper beyond arithmetic savings alone. Reporting the decoding results and acknowledging their limitations is particularly helpful.
Critical weaknesses

1 Sparse FFN output and router training gradient Sections 3.2 and 4.3 define expert scores and hard Top-k selection, but do not define how selected expert outputs are combined. Appendix A.4 then uses language-model KL and cross-entropy losses to optimize routing modules while the original parameters remain frozen. If routing scores determine only discrete expert indices, ordinary differentiation through those indices supplies no task gradient to the scores. Score-dependent output weights, a straight-through estimator, or another training mechanism could resolve this, but the manuscript needs to specify which is used. The exact forward equation should specify selected-expert aggregation, any gate normalization or scaling, and training/inference differences. This is an important clarification request. The manuscript does not show that its implementation uses an invalid unweighted hard-selection rule, so that hypothetical implementation should not be assumed in the assessment. 2 Baseline provenance and comparison scope Table 4 already establishes useful within-framework component effects under matched structure and training budget. The remaining comparison concern is narrower: Table 1 compares the proposed 28/112 configuration with CMoE configurations using 2/8 or 4/16 experts, alongside different adaptation regimes. This limits how precisely the headline cross-method gain can be attributed to the factorized representation alone. The paper should identify which baseline results were reimplemented, the calibration and adaptation settings, and the exact active-capacity calculation. These details would make the cross-method comparison reproducible. A parameter-matched direct MLP router is a useful additional diagnostic, rather than a prerequisite for recognizing the contribution supported by Table 4. The CMoE reimplementation also needs explicit provenance and settings: its published experiments use different backbones and benchmark protocols, and its architecture includes shared experts. The active-capacity accounting should include all shared and routed computation. [1] 3 Total conversion cost The approximately 0.13M token exposures refer to final model-level alignment. Appendix A.4 additionally describes coefficient-predictor training and expert-distribution warmup, without specifying their training exposures, batch sizes, or optimization steps. Consequently, the comparison with approximately 4M CMoE LoRA exposures characterizes selected adaptation stages rather than total conversion cost. Reporting all training stages, NMF and clustering time, teacher computation, and CPU/GPU hours would make the efficiency claim assessable. 4 Runtime reproducibility The paper should specify the expert kernel and software versions, prefill batch size, dense baseline optimizations, warmup and synchronization procedures, and timing repetitions with variability. The six layer/input pairs used for FFN timing should also be identified. These details matter for interpreting speedups of 1.13-1.22x.

Questions for authors

1 Sparse FFN output and router training gradient Sections 3.2 and 4.3 define expert scores and hard Top-k selection, but do not define how selected expert outputs are combined. Appendix A.4 then uses language-model KL and cross-entropy losses to optimize routing modules while the original parameters remain frozen. If routing scores determine only discrete expert indices, ordinary differentiation through those indices supplies no task gradient to the scores. Score-dependent output weights, a straight-through estimator, or another training mechanism could resolve this, but the manuscript needs to specify which is used. The exact forward equation should specify selected-expert aggregation, any gate normalization or scaling, and training/inference differences. This is an important clarification request. The manuscript does not show that its implementation uses an invalid unweighted hard-selection rule, so that hypothetical implementation should not be assumed in the assessment. 2 Baseline provenance and comparison scope Table 4 already establishes useful within-framework component effects under matched structure and training budget. The remaining comparison concern is narrower: Table 1 compares the proposed 28/112 configuration with CMoE configurations using 2/8 or 4/16 experts, alongside different adaptation regimes. This limits how precisely the headline cross-method gain can be attributed to the factorized representation alone. The paper should identify which baseline results were reimplemented, the calibration and adaptation settings, and the exact active-capacity calculation. These details would make the cross-method comparison reproducible. A parameter-matched direct MLP router is a useful additional diagnostic, rather than a prerequisite for recognizing the contribution supported by Table 4. The CMoE reimplementation also needs explicit provenance and settings: its published experiments use different backbones and benchmark protocols, and its architecture includes shared experts. The active-capacity accounting should include all shared and routed computation. [1] 3 Total conversion cost The approximately 0.13M token exposures refer to final model-level alignment. Appendix A.4 additionally describes coefficient-predictor training and expert-distribution warmup, without specifying their training exposures, batch sizes, or optimization steps. Consequently, the comparison with approximately 4M CMoE LoRA exposures characterizes selected adaptation stages rather than total conversion cost. Reporting all training stages, NMF and clustering time, teacher computation, and CPU/GPU hours would make the efficiency claim assessable. 4 Runtime reproducibility The paper should specify the expert kernel and software versions, prefill batch size, dense baseline optimizations, warmup and synchronization procedures, and timing repetitions with variability. The six layer/input pairs used for FFN timing should also be identified. These details matter for interpreting speedups of 1.13-1.22x.

7 comments

We thank the reviewer for the constructive feedback and for recognizing the contributions of our NMF-guided expert construction and routing, executable physical expert structure, and controlled ablation studies. We have further clarified the router training mechanism, baseline configurations, complete conversion cost, and runtime evaluation protocol to address the reviewer's concerns.

Q1. Sparse FFN Output and Router Training Gradient

Our method uses hard Top-k expert selection without score-dependent weighting of expert outputs. The selected experts' contributions are summed directly, preserving the original FFN computations for the selected neurons.

During model-level alignment, we apply a Straight-Through Estimator (STE) to propagate gradients through the discrete routing decisions:

rtSTE=rthard+pt−sg(pt)r_t^{STE} = r_t^{hard} + p_t - sg(p_t)

where rthardr_t^{hard} is the binary Top-k selection mask, ptp_t is the differentiable distribution derived from the expert scores, and sgsg denotes stop-gradient.

The sparse FFN output during alignment is:

Ftrain(zt)=Wd[h(zt)⊙(MrtSTE)]F_{train}(z_t) = W_d [h(z_t) \odot (M r_t^{STE})]

where h(zt)h(z_t) denotes the original SwiGLU intermediate activations and MM maps FFN neurons to physical experts.

In the forward pass, rtSTEr_t^{STE} equals rthardr_t^{hard}, so only the selected experts contribute to the FFN output, without additional gate normalization or scaling.

In the backward pass, STE allows the cross-entropy and teacher KL-divergence gradients to flow through the expert scores to the coefficient predictor and factor-to-expert mapping adapter. The original model weights and expert assignments remain frozen.

During training, we use the masked FFN computation to enable gradient propagation. At inference, we execute only the selected physical experts. Both implementations produce the same sparse FFN output mathematically, with inference avoiding computation over inactive neurons.

We have clarified the sparse FFN forward equation and STE-based gradient propagation in Section 4.3 and Appendix A.4.

Q2. Baseline Provenance and Comparison Scope

CMoE Configuration. We reimplemented CMoE on LLaMA-3-8B following its official implementation and configurations, using the same Dolmino-Mix data source as our method for expert construction and LoRA fine-tuning. Following the official configuration, expert construction uses 8 calibration sequences of 2,048 tokens, while LoRA adaptation uses 2,048 sequences of 2,048 tokens for one epoch (approximately 4.19M token exposures), with LoRA rank 8 and alpha 32. The LoRA implementation adapts both attention and FFN projection modules, following the official implementation. Both the training-free and LoRA-adapted configurations were evaluated under our common evaluation protocol.

CMoE includes always-active shared experts and conditionally activated routed experts. Our active FFN capacity calculation accounts for both:

ConfigurationShared ExpertsActive Routed ExpertsTotal Active FFN
CMoE E8112/8 (25%)
CMoE E16224/16 (25%)
Ours E11202828/112 (25%)

At 25% active FFN capacity, our method achieves 57.93% average accuracy, compared with 54.47% for the strongest LoRA-adapted CMoE configuration, an improvement of 3.46 percentage points.

Controlled Router Comparison. We additionally introduced a supervised direct MLP router with the same expert partition, Top-28 execution budget, and comparable initialization training budget. Our NMF-guided MLP achieves 57.93%, compared with 55.82% for the supervised direct MLP, demonstrating the contribution of NMF-guided routing under a fixed expert structure.

Detailed router configurations, training budgets, and per-task results are provided in Appendix A.5.

For the pruning and activation-sparsity baselines, we used FFN-only adaptations on the same LLaMA-3-8B backbone, keeping attention and other non-FFN components unchanged. All methods were evaluated using lm-evaluation-harness v0.3.0 with consistent few-shot settings and the acc metric.

Q3. Total Conversion Cost

Complete Conversion Time. We measured the entire dense-to-MoE conversion pipeline for LLaMA-3-8B on a system equipped with an Intel Core i7-13700KF CPU and a single NVIDIA RTX 4090 GPU.

Conversion StageWall-clock Time
Activation Collection, NMF, and NNLS26 min
Expert Construction4 min
Router Initialization8 min
Model-level Alignment7 min
Total Conversion45 min

The reported 45-minute total includes all four stages, including dense-teacher forward computation during model-level alignment.

Training Budget. Our conversion involves three lightweight learning stages:

  1. Coefficient Predictor Pretraining: Each layer reuses the 32,768 NMF calibration tokens, with a batch size of 512 and an average of 20.66 epochs, corresponding to approximately 677K layer-token exposures per layer.

  2. Expert-distribution Warmup: We reuse the same 32,768 calibration tokens for eight epochs, with a batch size of 2,048 and 128 optimizer steps per layer, corresponding to 262K additional layer-token exposures per layer.

  3. Model-level Alignment: We use a separate, document-disjoint dataset of 131,072 input tokens, with 128 optimizer steps. The dense teacher is evaluated without gradient tracking, while only the routing modules are optimized.

The first two stages therefore involve approximately 939K layer-token exposures per layer, while the final model-level alignment uses 0.13M model-level input-token exposures. These are distinct accounting units.

Comparison with CMoE. Our previously reported 0.13M tokens refer specifically to final model-level alignment, whereas CMoE's approximately 4M token exposures refer to its LoRA adaptation stage. The complete conversion cost of our method is captured by the 45-minute end-to-end measurement, including activation factorization, expert construction, router initialization, and alignment.

Detailed optimization settings are provided in Appendix A.4, and the updated full conversion time breakdown is reported in Appendix A.7.

Reviewer_8RgSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper converts dense FFNs into physical experts using an NMF representation of activation magnitudes. Neuron-side factor loadings are clustered into balanced experts, while a small predictor estimates token-side coefficients and maps them to expert scores. Original model weights remain frozen. On LLaMA-3-8B at 25% active FFN capacity, the method improves average accuracy over the evaluated CMoE variants and achieves 1.13–1.22x end-to-end prefill speedup.

Critical strengths
  1. Using the two sides of the factorization for construction and routing is a sensible design. Table 4 provides useful evidence at a fixed 112-expert, Top-28 configuration: replacing NMF construction lowers average accuracy by 5.63 points, and replacing NMF-guided routing lowers it by 3.83 points. These controls make a better case for the representation than the main comparison with coarser CMoE configurations alone.

  2. The runtime evaluation measures actual end-to-end prefill performance and includes routing and dispatch overheads. The authors also report the unfavorable decoding results rather than extrapolating the arithmetic reduction into a general speedup claim. This gives a reasonably clear picture of where the implementation is useful.

Critical weaknesses
  1. I could not reconstruct the forward computation used during model-level router alignment. Section 4.3 defines expert scores and hard Top-k selection, while Appendix A.4 optimizes the predictors and mapping adapters using language-model CE and KL losses. However, the paper never specifies whether selected expert outputs are weighted by router scores or whether a straight-through estimator or relaxation is used. If outputs are simply summed after discrete selection, those losses cannot differentiate through the selected indices. This may be an omitted implementation detail, but it is essential to understanding and reproducing the training procedure.

  2. The low-cost comparison does not account for the full conversion pipeline. The advertised 0.13M exposures cover only the final model-level alignment. Before that, the method fits NMF separately for every layer, trains coefficient predictors, and performs an expert-distribution warmup. Appendix A.4 gives learning rates but no update counts for the first two predictor-training stages, and no complete conversion time is reported. Consequently, the comparison with CMoE's 4M-token adaptation does not yet establish a conversion-cost advantage. Since low-cost conversion is a central motivation, this affects my overall recommendation.

Questions for authors
  1. Could you write the sparse FFN output equation used during alignment and explain exactly how CE/KL gradients reach f_theta and Delta? Are router-dependent output weights or a gradient estimator involved, and does inference use the same forward computation? Reporting accuracy immediately before and after model-level alignment would also clarify its contribution.

  2. What is the complete conversion budget, including NMF fitting, coefficient regression, expert-distribution warmup, and final alignment? Please include training steps or epochs and token exposures for each learning stage, together with CPU/GPU time and hardware. Does the claimed cost advantage over CMoE remain under this accounting?

4 comments

We thank the reviewer for the constructive feedback and for recognizing the contributions of our NMF-based expert construction and routing. We have clarified the sparse FFN computation, expanded the router training details, and added a complete accounting of the conversion pipeline.

Q1. Sparse FFN computation and router gradients

We use a straight-through estimator (STE) to propagate gradients through hard Top-k expert selection. Specifically, given the binary Top-k mask rthardr_t^{hard} and a differentiable routing distribution ptp_t derived from expert scores, we define

rtSTE=rthard+pt−sg(pt)r_t^{STE} = r_t^{hard} + p_t - sg(p_t)

where sgsg denotes stop-gradient. The sparse FFN output is

Fsparse(zt)=Wd[h(zt)⊙(MrtSTE)]F_{sparse}(z_t) = W_d [h(z_t) \odot (M r_t^{STE})]

where h(zt)h(z_t) is the original SwiGLU intermediate activation and MM is the fixed neuron-to-expert assignment matrix.

The forward pass uses unweighted hard Top-k selection where selected expert outputs are summed without multiplying them by routing probabilities. During backpropagation, STE allows the KL and CE gradients to reach the coefficient predictor fθf_\theta and the factor-to-expert mapping adapter Δ\Delta. The original model weights, NMF factors, and expert assignments remain frozen.

Training uses a masked FFN implementation to support STE gradients, whereas inference executes only the selected physical experts. Both follow the same hard-selection forward computation.

We also evaluated the model immediately before and after model-level alignment. The five-task average accuracy improves from approximately 55.48% to 57.93%, demonstrating the additional contribution of model-level router adaptation beyond initialization and warmup.

The forward computation and gradient mechanism have been clarified in Section 4.3, with additional training details in Appendix A.4.

Q2. Complete conversion budget and comparison with CMoE

We now provide the training configurations for all three router optimization stages.

1. Coefficient predictor pretraining. Each layer uses a 4096-to-256-to-128 MLP to predict coefficients obtained through non-negative least squares (NNLS) over the fixed NMF dictionary. The objective is

Lpred=Lrecon+0.1LcoeffL_{pred} = L_{recon} + 0.1 L_{coeff}

where the two terms measure relative activation reconstruction error and normalized coefficient matching, respectively. This stage reuses the same 32,768 tokens used for NMF fitting, with batch size 512 and an average of 20.66 epochs per layer (approximately 677K layer-token exposures).

2. Expert-distribution warmup. We further train the predictor to match NMF-derived expert distributions using

Lwarmup=KL(pt∗∣∣pt)L_{warmup} = KL(p_t^* || p_t)

where pt∗p_t^* is the target expert distribution. Warmup reuses the same 32,768 calibration tokens, with batch size 2,048 and eight epochs (128 optimizer steps and 262K token exposures per layer).

3. Model-level alignment. We optimize the predictors and mapping adapters using

L=2LKL+LCE+0.001Lz+0.01LbalanceL = 2L_{KL} + L_{CE} + 0.001L_z + 0.01L_{balance}

with the frozen dense model as the KL teacher. Alignment uses 512 sequences of length 256, corresponding to 131,072 input-token exposures over 128 optimizer steps.

All data are sampled from Dolmino-Mix. Calibration and model-level alignment use document-disjoint splits, while predictor pretraining and warmup reuse the NMF calibration data. The first two stages' exposures are counted per FFN layer, whereas the final alignment processes the complete model.

We have additionally measured the complete conversion pipeline on an Intel Core i7-13700KF and a single NVIDIA RTX 4090:

Conversion StageWall-clock Time
Activation factorization (collection, NMF, NNLS)26 min
Expert construction4 min
Router initialization (pretraining and warmup)8 min
Model-level alignment7 min
Total conversion45 min

Comparison with CMoE. CMoE reports 4.5 minutes for analytical expert construction and 46 minutes for end-to-end conversion including LoRA fine-tuning with approximately 4M tokens on LLaMA-2-7B. Our complete conversion takes 45 minutes on LLaMA-3-8B using a single RTX 4090, with only lightweight routing modules optimized.

The reported end-to-end times are comparable in magnitude, although they are not hardware-matched measurements, as CMoE does not specify the hardware configuration for its timing results. Our updated results establish the full conversion cost of our framework and clarify that the previously reported 0.13M tokens refer specifically to model-level alignment, not the entire conversion pipeline.

The detailed optimization configurations and complete cost breakdown are now provided in Appendices A.4 and A.7.

Dear Reviewer 8RgS,

We thank you again for the thoughtful and constructive feedback. We have carefully considered the concerns raised in the review and provided detailed responses, clarifications, and additional evidence where applicable. As the discussion period is approaching its end, we would appreciate any further feedback on whether our responses adequately address your main concerns, or if any questions remain. We would be happy to provide further clarification or make necessary revisions during the remaining discussion period.

Kind Regards,

The Authors

Dear Reviewer 8RgS,

We would like to briefly follow up on our rebuttal responses as the discussion period draws to a close.

We would greatly appreciate it if you could take a moment to review our clarifications and let us know whether any of your concerns remain unresolved. We remain available to address any outstanding questions promptly.

Thank you again for your time and consideration.

Kind regards,
The Authors

Comments on the paper

1 comment

Final Rebuttal Summary: Clarifications, Corrections, and Additional Evidence

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their constructive feedback. Throughout the rebuttal, we have carefully addressed the methodological, experimental, and reproducibility concerns through manuscript revisions, implementation corrections, and additional experiments. We summarize the key updates below.

1. Methodological Soundness and Conversion Cost

We clarified the exact sparse FFN forward computation and STE-based gradient propagation through hard Top-k expert selection. Training and inference use mathematically equivalent hard-selection forward computations, with the original dense-model weights frozen.

We also provided complete training configurations and measured the entire conversion pipeline at 45 minutes on a single RTX 4090, including NMF fitting, expert construction, router initialization, and model-level alignment. The previously reported 0.13M tokens refer specifically to final alignment, not the full conversion budget.

2. Baseline Corrections and Fair Comparisons

We identified and corrected an FFN channel-ordering issue in our LLM-Pruner implementation, reran pruning from the original dense checkpoint, and verified the corrected results. We further clarified CMoE's shared/routed expert accounting, audited the remaining baseline implementations, and specified matched training budgets for our controlled ablations.

3. Additional Evidence for NMF-Guided Expert Construction and Routing

Under the same 112-expert, Top-28 configuration, our controlled ablations show that NMF-guided expert construction and routing contribute 5.63 and 3.83 percentage points, respectively.

We additionally compared against a supervised direct MLP router using five training seeds. Our method consistently outperforms this baseline (57.93 ± 0.06% vs. 55.84 ± 0.07%). We also conducted NMF rank sensitivity experiments with five independent conversion runs per rank, confirming stable results and diminishing improvements beyond rank 128.

4. Extended Quality–Efficiency Evaluation

At 25% active FFN capacity, our method achieves 57.93% average accuracy, exceeding the strongest evaluated CMoE configuration by 3.46 percentage points, with 1.13–1.22× measured end-to-end prefill speedup.

At 50% active FFN capacity, accuracy increases to 70.37%, compared with 64.58% for fine-tuned CMoE, while retaining 1.09–1.11× prefill speedup.

We acknowledge the remaining accuracy gap relative to the dense model, the higher accuracy of some fine-grained sparsification methods, and the limitations in small-batch decoding. Our claims are therefore explicitly scoped to NMF-guided construction and routing of independently executable physical experts under structured sparse-execution constraints, rather than universal superiority across deployment alternatives.

We appreciate the reviewers' engagement, including the explicit acknowledgment during the discussion that our additional evidence addressed the identified soundness concerns.

We hope that these substantive clarifications, corrections, and experimental additions will be considered in the overall assessment of the paper.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.