acceptodds
Under review as a conference paper at ICLR 2027

Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion

Abstract

Large language models impose substantial memory demands and arithmetic costs during inference. Dense-to-mixture-of-experts (MoE) conversion is a model optimization technique that aims to reduce these arithmetic costs. Dense-to-MoE conversion partitions pretrained feed-forward network (FFN) neurons into experts and selects a subset for each token. Leading methods fall into two categories: analytic methods produce a converted model quickly using heuristics, while learning methods produce a model much more slowly by optimizing via gradients. Both approaches, however, fix the same active expert count, or “utilization,” at every layer. We introduce Spectral-MoE, which learns to allocate a utilization budget heterogeneously across layers. Its name draws a metaphor between separating a dense FFN into experts and separating a light beam into its spectrum. Spectral-MoE consists of two distinct approaches: a static approach that uses a single per-layer scalar which is shared across tokens, and a dynamic approach which emits a per-token schedule. We demonstrate that both allocators improve performance over uniform on both the analytic-based and learning-based SOTA dense-to-MoE methods. At FFN utilization, dynamic allocation reduces WikiText-2 perplexity relative to uniform by , , and points on the learned base for Llama-2-7B, Llama-3-8B, and Qwen2.5-7B, respectively. The corresponding reductions on the analytic base are , , and points. On the learned base, dynamic allocation improves six-task mean accuracy over uniform by an average of , , and percentage points across the three models at FFN utilizations of , , and , respectively. These results establish per-layer expert utilization as a novel optimization axis in dense-to-MoE conversion.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, MoE, Dense-to-MoE, Optimization, Learning, Per-Layer Utilization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3NwoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Spectral-MoE learns how to distribute active experts across layers during dense-to-MoE conversion. The static version learns one count per layer, while the dynamic version revises each token's remaining allocation using intermediate hidden states. Both conserve the same total expert budget for every token. The method is evaluated on CMoE- and DOT-MoE-based conversions of three models, with dynamic allocation improving perplexity and downstream accuracy across several utilization levels.

Critical strengths
  1. The budget constraint makes the main comparison meaningful. Dynamic allocation cannot improve quality simply by activating more experts on difficult tokens: it must redistribute a fixed total across layers. The construction explicitly handles integer counts and remaining-layer feasibility, making this a useful extension of conventional uniform routing.

  2. The gains are reasonably consistent. Dynamic allocation improves both perplexity and mean accuracy across all nine learned-base model/utilization combinations. It also lowers perplexity when the analytic partition and router are frozen, which supports the usefulness of allocation beyond joint partition training alone. Although the strongest baseline comparison needs clarification, this consistency makes me lean toward acceptance.

Critical weaknesses
  1. The gap between the reproduced and published DOT-MoE baselines complicates the interpretation. On Qwen2.5-7B, uniform BoolQ accuracy is 43.7%, versus the published 64.3%; dynamic allocation reaches 65.2%. This single task accounts for much of the reported mean improvement on that model. Dynamic matches, rather than exceeds, published DOT-MoE's overall Qwen accuracy. The authors appropriately disclose the discrepancy, but do not explain it. I would like to know whether allocation addresses a genuine limitation of uniform routing or partly compensates for an implementation or optimization weakness.

  2. The need for token-specific revision is not yet well isolated. Figure 5 shows a strong average preference for later layers, but the paper does not test a fixed schedule obtained from the dynamic allocator's average counts while keeping its trained partition and router unchanged. The separately trained static allocator cannot answer this question because its optimization and resulting expert structure differ. This matters for deployment: a fixed nonuniform schedule could retain much of the benefit with simpler execution. The paper also gives no latency measurements showing the cost of sequential allocator revisions and variable expert counts.

Questions for authors
  1. What explains the particularly low uniform BoolQ result on Qwen? Does it recur across paired seeds, and have you checked whether the shared-expert modification or another implementation difference accounts for it? Resolving this would strengthen the comparison with published DOT-MoE.

  2. If you freeze a dynamic checkpoint's partition and router and replace its token-dependent allocation with a budget-preserving, rounded average schedule, how much of the quality improvement remains? This would directly test the value of adapting allocation to each token.

  3. What is the inference overhead relative to uniform and static allocation at the same utilization? Even a representative single-token and batched measurement would help establish whether the quality gain comes with a meaningful execution penalty.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3Nwo W1 / Q1

Thank you for your efforts in reviewing our work. Below are our responses to the concerns raised and questions asked.

W1/Q1

Repeatability. We ran three seeds for the dynamic, static, and uniform allocation approaches in order to address your question about whether the low uniform BoolQ result recurs across paired seeds. Due to time and compute constraints we chose to only run on Qwen2.5-7B, as that would also allow us to answer whether BoolQ recovery issues are systemic or just a result of the seed selected. Below are the results.

AllocationSeedPPLBoolQ (%)Avg. excl. BoolQ (%)Six-task avg. (%)
UniformC16.9466.060.061.0
A17.0163.960.461.0
B16.9243.760.357.6
StaticC15.8156.560.760.0
A15.8264.660.060.8
B15.9965.960.261.1
DynamicC14.1863.062.762.7
A14.2563.962.662.8
B14.2165.262.963.3

From this we can see that our dynamic allocation approach does show consistent benefit over multiple seeds when compared to static and uniform allocation. The improvement over uniform allocation ranges from 2.71 to 2.76 in perplexity and stays above 2 points on average when excluding BoolQ across the three seeds. Additionally, this shows that the low uniform BoolQ result you identified comes from seed variability rather than from the shared expert.

Shared expert. We investigated DOT-MoE-style conversions without the shared expert at various utilizations early on in our research, and we report them here in order to address whether adding the shared expert accounts for the gap to the published DOT-MoE results. Below are the results.

ModelAllocationShared expertPPLAvg. excl. BoolQ (%)Six-task avg. (%)
Llama-2-7BUniformYes11.5759.459.0
UniformNo12.1558.959.4
Qwen2.5-7BUniformYes16.7160.459.9
UniformNo18.7359.555.9

From this we can see that the shared expert lowers perplexity and improves the average excluding BoolQ by at least 0.5 points on both models. Also, the addition of the shared expert drastically improves the training stability in some of the lower utilization regimes we were exploring. Below are the results of the uniform conversions of Llama-2-7B that we trained at these utilizations, compared at step 2,518, the last step that all four runs reached.

FFN utilizationShared expertPPL
12.8%Yes16.83
No18.83
5.8%Yes30.88
No850.17

From this we can see that below some utilization the shared expert becomes pivotal for training stability, since both conversions train at 12.8% FFN utilization, while at 5.8% the conversion without the shared expert diverges and the one with it still reaches a perplexity of 30.88.

The results at 25% also show that the shared expert does not explain the gap to the published DOT-MoE results, since without it our uniform conversions still fall below the published averages excluding BoolQ, 60.5 on Llama-2-7B and 63.1 on Qwen2.5-7B. We address this gap directly in the DOT-MoE reproduction paragraph below.

We also investigated dynamic allocation without the shared expert early on in our research, and we report it here in order to address whether our improvement over uniform allocation depends on it. Below are the results.

ModelAllocationPPLAvg. excl. BoolQ (%)Six-task avg. (%)
Llama-2-7BUniform12.1558.959.4
Dynamic11.2659.860.6

From this we can see that our dynamic allocation approach still improves on uniform allocation without the shared expert, so its improvement does not depend on the shared expert.

DOT-MoE reproduction. Since no public implementation of DOT-MoE was available, we replicated it to the best of our ability based on the description and the hyperparameters provided in the paper. (The hyperparameters were sourced from Tables 6 and 7 of the DOT-MoE paper.) Despite our inability to exactly replicate the numbers found in the paper, we still are able to exceed the performance of the published DOT-MoE numbers (not our replication) on two of the three models tested.

We have edited Appendix D to include these extra seed experiments.

AuthorsOpenReview ↗

Response to Reviewer 3Nwo W2 / Q2 and Q3 (Part 1/2)

W2/Q2

Frozen mean schedule. We generated an average per-layer utilization from our dynamic model and used it as a schedule in order to address your question about how much of the quality improvement remains with an average schedule. The schedule averages the count of each layer over 131,072 Dolmino training tokens. Below are the results.

ModelFFN utilizationUniform PPLDynamic PPLFrozen mean PPL
Llama-2-7B12.5%15.9314.8817.03
25%11.6010.5713.40
50%8.277.349.47
Llama-3-8B12.5%27.6825.4628.99
25%19.4516.4620.51
50%12.189.6513.78
Qwen2.5-7B12.5%23.5721.1424.70
25%16.8714.2218.37
50%11.359.1812.06

From this we can see that none of the quality improvement remains with the frozen mean schedule, which is worse than uniform allocation at every model and utilization. This heavily implies that the token-specific utilizations learned by the dynamic approach have a significant effect on the overall accuracy.

We report the latency of sequential allocator revisions and variable expert counts in our response to Q3.

We have added an additional section to the appendix (Section H) where we discuss the frozen mean schedule.

Q3

Analytic model. Since we do not have the time to create dedicated kernels for our approach, we created an analytic model of the inference cost of uniform, static, and dynamic allocation with custom kernels in order to address your question about the inference overhead relative to uniform and static allocation. A token computes kℓk_\ell routed experts in layer ℓ\ell, and a decode step at batch size bb reads every routed expert that at least one of its tokens selects. Since these selections overlap, we count the routed experts per token, BB, and the expected routed experts per decode step, UU, for tokens that select at random among the MM routed experts of the model,

B=∑ℓkℓ,U=M(1−(1−BM)b),B = \sum_{\ell} k_\ell, \qquad U = M\Bigl(1 - \bigl(1 - \tfrac{B}{M}\bigr)^{b}\Bigr),

so U=BU = B at batch size 1 and U≈MU \approx M at batch size 256. The model treats a decode step as bound by memory bandwidth and a prefill as bound by arithmetic. It also accounts for the serial nature of the allocator by charging each of the HH allocating layers one extra kernel that runs before its experts,

Tdecode=W+bcv+Uw+Qβ+(K+H),τ,Tprefill=N,g+na+Bf+Fπ+Hτ,T_{\mathrm{decode}} = \frac{W + bcv + Uw + Q}{\beta} + (K + H)\\,\tau, \qquad T_{\mathrm{prefill}} = N\\,\frac{g + na + Bf + F}{\pi} + H\tau,

SymbolValueMeaning
M2,720Routed experts, 85 in each of the 32 layers
w, f3.15 MB, 3.15 MFLOPBytes and FLOPs per token of one routed expert
W, g4.68 GB, 4.42 GFLOPWeights and FLOPs per token outside routed experts
v, a0.524 MB, 0.262 MFLOPKV cache and attention FLOPs per context token
Q, F8.19 MB, 8.19 MFLOPAllocator bytes per decode step and FLOPs per token
H, K, τ31, 388, 1.66 μsAllocating layers, other kernels, and time per kernel
β, π2.36 TB/s, 716 TFLOP/sDense-model bandwidth and throughput on our H100
b, c–Batch size and context length
N, n–Prefill tokens and sequence length

Below are the times that the model predicts for three workloads.

WorkloadUniformStaticDynamic
Decode step, batch 1, context 2,048 (ms)3.983.984.03
Decode step, batch 256, context 256 (ms)20.8120.8120.86
Prefill, 16 sequences of 2,048 tokens (ms)323.7323.7324.1

From this we can see that with custom kernels dynamic allocation would run about as fast as uniform allocation, since even with the allocator running serially before the experts it adds only 55 μs to a decode step, or 1.4% at batch size 1.

AuthorsOpenReview ↗

Response to Reviewer 3Nwo Q3 (Part 2/2)

Q3 (continued)

These estimates use the constants of Llama-2-7B at 25% FFN utilization in bfloat16, whose 13.5 GB of weights split into 8.56 GB of routed experts, the 4.68 GB in WW, and 0.26 GB of input embeddings. The allocator adds 158,942 parameters, or 0.32 MB, and reading its 0.26 MB shared projection in each of the HH allocating layers accounts for almost all of QQ. Uniform and static allocation have Q=F=H=0Q = F = H = 0, and dynamic allocation keeps BB and UU unchanged, since it bounds the utilization of every token at the same budget of BB routed experts. The model is an estimate as it assumes custom kernels that reach the bandwidth and per-kernel time that we measured on the dense model. This is a reasonable assumption, since both values come from off-the-shelf software, the bandwidth from the cuBLAS matrix multiplications of the dense model, which reach only 70% of the 3.35 TB/s peak of the H100, and the per-kernel time from replaying the dense model as a CUDA graph.

Practical cost. We also measured the inference cost of uniform, static, and dynamic allocation in our current implementation, which runs every conversion in eager PyTorch on one NVIDIA H100 GPU through the Hugging Face implementation of Llama-2-7B at 25% FFN utilization, with our sparse FFN layer running the selected experts as grouped matrix multiplications. Batch size 32 is the largest that fits in memory with a 2,048-token context. Below are the results.

MeasurementUniformStaticDynamic
Prefill, 16 sequences of 2,048 tokens (ms)891.2892.9926.8
Decode step, batch 1, context 2,048 (ms)20.920.531.8
Decode step, batch 32, context 2,048 (ms)54.254.161.0
Decode step, batch 256, context 256 (ms)69.366.471.4
FFN arithmetic per token (GFLOP)2.212.212.21
Routing and allocation arithmetic per token (GFLOP)0.02230.02230.0305

From this we can see that the extra latency of dynamic allocation comes from dispatching its per-token expert counts from PyTorch, which at batch size 1 adds 670 kernel launches and 97 CPU-GPU synchronizations to a decode step but only 1.5 ms of GPU time. Additionally, this shows that dynamic allocation adds almost no arithmetic to uniform allocation, so custom kernels would remove most of its extra time, as the analytic model predicts. Our measured 20.9 ms for a uniform decode step at batch size 1 is far above the modeled 3.98 ms mainly because eager PyTorch launches every kernel from Python, which leaves the GPU idle for more than half of the step, and partly because of the many small unfused kernels of the Hugging Face implementation and slower weight streaming, which costs all three allocations about the same.

We have added an additional section to the appendix (Section G) where we discuss the latency effects and conversion cost of our approach compared to uniform.

Reviewer_mBwhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Spectral-MoE learns how to distribute expert activations across layers during dense-to-MoE conversion. It proposes static per-layer allocation and dynamic token-dependent allocation, both preserving an exact per-token budget through discrete selection and straight through gradient estimation. Integrated with CMoE and a reimplemented DOT-MoE, the method improves perplexity and downstream accuracy across three models, with dynamic allocation outperforming uniform allocation at all tested utilization levels on the learned base.

Critical strengths
  1. Learning per-layer expert counts complements partitioning and routing. Exact per-token budget conservation enables controlled comparisons without increasing total FFN activations.
  2. Dynamic allocation improves perplexity and average accuracy across all nine learned-base model budget combinations. Perplexity improvements on the analytic base further support its applicability.
Critical weaknesses
  1. The reimplemented DOT-MoE underperforms its published results. On Qwen2.5-7B, dynamic allocation improves accuracy from 57.6% to 63.3%, but only matches published DOT-MoE. Moreover, the limited seed study does not establish the variability of paired gains over uniform allocation.
  2. No latency or throughput measurements are reported. Repeated allocator evaluations and token-dependent expert counts may introduce overhead or reduce batching efficiency, so matched FFN utilization does not establish a better accuracy–runtime trade-off.
Questions for authors
  1. At matched expert and training-token budgets, how does dynamic allocation compare with predicting the entire token-dependent schedule at the first layer, without subsequent revisions?
  2. Appendix F selects temperature using perplexity, while Appendix C.1 states that all reported perplexities use the test split. Was a separate validation split used for selection? If not, do the conclusions hold under independent validation-based selection?
3 comments
AuthorsOpenReview ↗

Response to Reviewer mBwh W1 and W2 (Part 1/2)

Thank you for your efforts in reviewing our work. Below are our responses to the concerns raised and questions asked.

W1

Repeatability. We ran three seeds for the dynamic, static, and uniform allocation approaches in order to address your concerns with repeatability. Due to time and compute constraints we chose to only run on Qwen2.5-7B, as that would also allow us to answer whether BoolQ recovery issues are systemic or just a result of the seed selected. Below are the results.

AllocationSeedPPLBoolQ (%)Avg. excl. BoolQ (%)Six-task avg. (%)
UniformC16.9466.060.061.0
A17.0163.960.461.0
B16.9243.760.357.6
StaticC15.8156.560.760.0
A15.8264.660.060.8
B15.9965.960.261.1
DynamicC14.1863.062.762.7
A14.2563.962.662.8
B14.2165.262.963.3

From this we can see that our dynamic allocation approach does show consistent benefit over multiple seeds when compared to static and uniform allocation. The improvement over uniform allocation ranges from 2.71 to 2.76 in perplexity and stays above 2 points on average when excluding BoolQ across the three seeds. Additionally, this shows that the uniform accuracy of 57.6% that you identified is lowered by the BoolQ result of the specific seed selected.

DOT-MoE reproduction. Since no public implementation of DOT-MoE was available, we replicated it to the best of our ability based on the description and the hyperparameters provided in the paper. (The hyperparameters were sourced from Tables 6 and 7 of the DOT-MoE paper.) Despite our inability to exactly replicate the numbers found in the paper, we still are able to exceed the performance of the published DOT-MoE numbers (not our replication) on two of the three models tested.

We have edited Appendix D to include these extra seed experiments.

W2

Analytic model. Since we do not have the time to create dedicated kernels for our approach, we created an analytic model of the inference cost of uniform, static, and dynamic allocation with custom kernels in order to address your concerns with latency and throughput. A token computes kℓk_\ell routed experts in layer ℓ\ell, and a decode step at batch size bb reads every routed expert that at least one of its tokens selects. Since these selections overlap, we count the routed experts per token, BB, and the expected routed experts per decode step, UU, for tokens that select at random among the MM routed experts of the model,

B=∑ℓkℓ,U=M(1−(1−BM)b),B = \sum_{\ell} k_\ell, \qquad U = M\Bigl(1 - \bigl(1 - \tfrac{B}{M}\bigr)^{b}\Bigr),

so U=BU = B at batch size 1 and U≈MU \approx M at batch size 256. The model treats a decode step as bound by memory bandwidth and a prefill as bound by arithmetic. It also accounts for the serial nature of the allocator by charging each of the HH allocating layers one extra kernel that runs before its experts,

Tdecode=W+bcv+Uw+Qβ+(K+H),τ,Tprefill=N,g+na+Bf+Fπ+Hτ,T_{\mathrm{decode}} = \frac{W + bcv + Uw + Q}{\beta} + (K + H)\\,\tau, \qquad T_{\mathrm{prefill}} = N\\,\frac{g + na + Bf + F}{\pi} + H\tau,

SymbolValueMeaning
M2,720Routed experts, 85 in each of the 32 layers
w, f3.15 MB, 3.15 MFLOPBytes and FLOPs per token of one routed expert
W, g4.68 GB, 4.42 GFLOPWeights and FLOPs per token outside routed experts
v, a0.524 MB, 0.262 MFLOPKV cache and attention FLOPs per context token
Q, F8.19 MB, 8.19 MFLOPAllocator bytes per decode step and FLOPs per token
H, K, τ31, 388, 1.66 μsAllocating layers, other kernels, and time per kernel
β, π2.36 TB/s, 716 TFLOP/sDense-model bandwidth and throughput on our H100
b, c–Batch size and context length
N, n–Prefill tokens and sequence length

Below are the times that the model predicts for three workloads.

WorkloadUniformStaticDynamic
Decode step, batch 1, context 2,048 (ms)3.983.984.03
Decode step, batch 256, context 256 (ms)20.8120.8120.86
Prefill, 16 sequences of 2,048 tokens (ms)323.7323.7324.1

From this we can see that with custom kernels dynamic allocation would run about as fast as uniform allocation, since even with the allocator running serially before the experts it adds only 55 μs to a decode step, or 1.4% at batch size 1.

AuthorsOpenReview ↗

Response to mBwh W2 (Part 2/2) and Q1

W2 (continued)

These estimates use the constants of Llama-2-7B at 25% FFN utilization in bfloat16, whose 13.5 GB of weights split into 8.56 GB of routed experts, the 4.68 GB in WW, and 0.26 GB of input embeddings. The allocator adds 158,942 parameters, or 0.32 MB, and reading its 0.26 MB shared projection in each of the HH allocating layers accounts for almost all of QQ. Uniform and static allocation have Q=F=H=0Q = F = H = 0, and dynamic allocation keeps BB and UU unchanged, since it bounds the utilization of every token at the same budget of BB routed experts. The model is an estimate as it assumes custom kernels that reach the bandwidth and per-kernel time that we measured on the dense model. This is a reasonable assumption, since both values come from off-the-shelf software, the bandwidth from the cuBLAS matrix multiplications of the dense model, which reach only 70% of the 3.35 TB/s peak of the H100, and the per-kernel time from replaying the dense model as a CUDA graph.

Practical cost. We also measured the inference cost of uniform, static, and dynamic allocation in our current implementation, which runs every conversion in eager PyTorch on one NVIDIA H100 GPU through the Hugging Face implementation of Llama-2-7B at 25% FFN utilization, with our sparse FFN layer running the selected experts as grouped matrix multiplications. Batch size 32 is the largest that fits in memory with a 2,048-token context. Below are the results.

MeasurementUniformStaticDynamic
Prefill, 16 sequences of 2,048 tokens (ms)891.2892.9926.8
Decode step, batch 1, context 2,048 (ms)20.920.531.8
Decode step, batch 32, context 2,048 (ms)54.254.161.0
Decode step, batch 256, context 256 (ms)69.366.471.4
FFN arithmetic per token (GFLOP)2.212.212.21
Routing and allocation arithmetic per token (GFLOP)0.02230.02230.0305

From this we can see that the extra latency of dynamic allocation comes from dispatching its per-token expert counts from PyTorch, which at batch size 1 adds 670 kernel launches and 97 CPU-GPU synchronizations to a decode step but only 1.5 ms of GPU time. Additionally, this shows that dynamic allocation adds almost no arithmetic to uniform allocation, so custom kernels would remove most of its extra time, as the analytic model predicts. Our measured 20.9 ms for a uniform decode step at batch size 1 is far above the modeled 3.98 ms mainly because eager PyTorch launches every kernel from Python, which leaves the GPU idle for more than half of the step, and partly because of the many small unfused kernels of the Hugging Face implementation and slower weight streaming, which costs all three allocations about the same.

We have added an additional section to the appendix (Section G) where we discuss the latency effects and conversion cost of our approach compared to uniform.

Q1

No revision. We ran our dynamic allocation approach with the entire token-dependent schedule predicted at the first layer and no subsequent revisions, and with revision at every fourth layer, in order to address your question about this alternative. Due to time and compute constraints we chose to only run the no-revision and every-fourth-layer variants on Llama-2-7B at 25% FFN utilization, with the same expert budget, training tokens, and initialization as the paper. Below are the results.

AllocationAllocator headsPPLAvg. excl. BoolQ (%)Six-task avg. (%)
Uniform–11.6159.259.9
Static–11.5060.060.8
Dynamic, no revision110.9261.262.4
Dynamic, every 4th layer810.5561.161.7
Dynamic, every layer3110.5762.062.8

From this we can see that predicting the whole schedule at the first layer captures about two thirds of the perplexity reduction and that revision at every fourth layer recovers the rest, so we thank you for this question, and we will highlight minimizing the number of revision heads as a potential area for future work in the final version.

We have added an additional section to the appendix (Section H) where we discuss these results.

AuthorsOpenReview ↗

Response to Reviewer mBwh Q2

Q2

Model selection. We used the WikiText-2 test split as a validation set for hyperparameter and architectural tuning, and we never used the downstream accuracy benchmarks for selection. We re-scored every model of the paper on the WikiText-2 validation split, which our tuning never used and which therefore serves as a test set for our models, in order to address your question about validation-based selection. Below are the results.

ModelTest PPLValidation PPL
Dense model
Llama-2-7B5.475.58
Llama-3-8B6.146.38
Qwen2.5-7B6.857.34
Inclusion temperature, static allocation on Llama-2-7B
Uniform control16.3016.78
T = 0.2516.5716.89
T = 116.1516.62
T = 1.515.8916.57
Learned base at 25%, uniform / static / dynamic
Llama-2-7B11.61 / 11.50 / 10.5711.82 / 11.70 / 10.79
Llama-3-8B19.40 / 18.19 / 16.4420.24 / 18.78 / 17.10
Qwen2.5-7B16.92 / 15.99 / 14.2117.55 / 16.71 / 14.83
Analytic base at 25%, uniform / static / dynamic
Llama-2-7B23.14 / 21.07 / 18.0623.87 / 21.58 / 18.44
Llama-3-8B44.53 / 38.09 / 28.0146.59 / 39.73 / 29.05
Qwen2.5-7B44.05 / 39.86 / 28.9746.17 / 41.86 / 30.34

From this we can see that the conclusions hold under validation-based selection, since the validation split selects the same inclusion temperature, T = 1.5, and the ranking of uniform, static, and dynamic allocation is the same on both splits for every model.

We now report the WikiText-2 validation perplexity, which serves as our test set, as the headline perplexity in the main-body tables. We have edited Appendix C.1 and Appendix F to include these validation results, and Appendix C.1 now states that we used the test split for hyperparameter selection and architectural design.

Reviewer_JSCkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies allocation of active experts across layers during dense-to-MoE conversion. Spectral-MoE learns either a static layer schedule shared across tokens or a dynamic allocator that revises each token's remaining schedule using its current hidden state. Feasible count bounds, integer rounding, and final-layer remainder consumption conserve the total expert budget for every token; training uses soft inclusion functions and straight-through gradients. The method is integrated with CMoE-style analytic conversion and DOT-MoE-style learned conversion and evaluated on three 7–8B models at three utilization levels. Dynamic allocation consistently reduces perplexity relative to the authors' uniform implementations and improves six-task average accuracy in all nine learned-conversion model–budget combinations. The paper also examines continued partition turnover and router refitting after freezing the partition.

Critical strengths
  1. The method provides a concrete, composable dynamic allocator under a hard per-token budget. The feasible remainder bounds in Section 3.3 prevent current decisions from making later allocations infeasible, and the final layer consumes the remainder to ensure the exact total. This gives a clear resource constraint beyond an efficiency penalty that only indirectly controls average utilization. The allocator can also be combined with both analytic and learned partition/router construction.

  2. The quality improvements extend across models and budgets. In Table 1, learned dynamic allocation reduces PPL for all three models at 25% utilization. All nine model–utilization combinations in Table 2 improve both PPL and average task accuracy, and analytic conversion also improves PPL on all three models. These comparisons at matched realized budgets support the central finding that layer allocation is worth optimizing, even though the improvement over the strongest existing conversion systems remains less certain.

Critical weaknesses
  1. The headline improvement magnitudes depend on a relatively weak implementation baseline, with incomplete evidence of attribution and stability. In Table 1, Qwen2.5-7B uniform accuracy is 57.6 versus published DOT-MoE's 63.3, while dynamic allocation reaches exactly 63.3. BoolQ alone increases from 43.7 to 65.2, accounting for a substantial part of the mean improvement. The Llama results exceed the published numbers, but cross-paper settings are not paired, and Appendix D provides three dynamic seeds for only one model–budget setting rather than repeated paired uniform/static/dynamic runs. It is therefore unclear how much of the gain is a robust allocation advantage versus recovery from a weaker conversion implementation. Although a static allocator is included, matched-budget sensitivity-based or search-based allocation baselines, a frozen mean-dynamic schedule, and a dynamic allocator without revision heads would better establish the value of the more complex design.

  2. Practical efficiency benefits have not been measured. The main budget accounts for active experts, while dynamic allocation introduces per-layer decisions, selection work, and token-dependent expert counts. These can affect kernel utilization, batching, weight movement, and cache behavior. Approximately 159K additional parameters do not imply negligible serial scheduling overhead. The paper does not report end-to-end prefill/decode latency, throughput, memory use, or conversion GPU-hours. The results therefore establish improved quality at an activation budget but do not yet establish an improved quality–latency frontier. For the analytic base, allocator training also adds work beyond the recovered uniform checkpoint and should enter the cost comparison.

Questions for authors
  1. Baselines and repeatability: What explains the gap between the DOT-MoE reproduction and published results, and how much does adding a shared expert contribute? Could at least one representative setting include repeated paired uniform/static/dynamic comparisons, particularly checking whether the large Qwen BoolQ recovery is stable? Three dynamic-only seeds do not estimate the variance of the paired improvement.
  2. Allocation mechanism: How do a frozen mean schedule from the trained dynamic allocator, removal of intermediate revision heads, and a simple layer-sensitivity allocation perform at the same realized expert budget? These controls would distinguish layer heterogeneity, token conditioning, and sequential revision.
  3. System benefit: For specified hardware, batch sizes, sequence lengths, and inference implementation, what are the prefill and decode times of uniform/static/dynamic allocation? Please report actual FFN arithmetic, allocation/routing overhead, and total conversion GPU-hours so that quality gains can be assessed against practical cost.
  4. Model selection: Which data split was used to select the inclusion temperature in Appendix F? The training curves repeatedly report WikiText-2 test PPL; were these measurements used for hyperparameter selection? If so, is an independent PPL evaluation available to support the main comparisons?
4 comments
AuthorsOpenReview ↗

Response to Reviewer JSCk W1 / Q1

Thank you for your efforts in reviewing our work. Below are our responses to the concerns raised and questions asked.

W1/Q1

Repeatability. We ran three seeds for the dynamic, static, and uniform allocation approaches in order to address your concerns with repeatability. Due to time and compute constraints we chose to only run on Qwen2.5-7B, as that would also allow us to answer whether BoolQ recovery issues are systemic or just a result of the seed selected. Below are the results.

AllocationSeedPPLBoolQ (%)Avg. excl. BoolQ (%)Six-task avg. (%)
UniformC16.9466.060.061.0
A17.0163.960.461.0
B16.9243.760.357.6
StaticC15.8156.560.760.0
A15.8264.660.060.8
B15.9965.960.261.1
DynamicC14.1863.062.762.7
A14.2563.962.662.8
B14.2165.262.963.3

From this we can see that our dynamic allocation approach does show consistent benefit over multiple seeds when compared to static and uniform allocation. The improvement over uniform allocation ranges from 2.71 to 2.76 in perplexity and stays above 2 points on average when excluding BoolQ across the three seeds. Additionally, this shows that the issues you identified with BoolQ only occur with the specific seed selected.

Shared expert. We investigated DOT-MoE-style conversions without the shared expert at various utilizations early on in our research, and we report them here in order to address your question about how much adding the shared expert contributes. Below are the results.

ModelAllocationShared expertPPLAvg. excl. BoolQ (%)Six-task avg. (%)
Llama-2-7BUniformYes11.5759.459.0
UniformNo12.1558.959.4
Qwen2.5-7BUniformYes16.7160.459.9
UniformNo18.7359.555.9

From this we can see that the shared expert lowers perplexity and improves the average excluding BoolQ by at least 0.5 points on both models. Also, the addition of the shared expert drastically improves the training stability in some of the lower utilization regimes we were exploring. Below are the results of the uniform conversions of Llama-2-7B that we trained at these low utilizations.

FFN utilizationShared expertPPL
12.8%Yes16.83
No18.83
5.8%Yes30.88
No850.17

From this we can see that below some utilization the shared expert becomes pivotal for training stability, since both conversions train at 12.8% FFN utilization, while at 5.8% the conversion without the shared expert diverges and the one with it still reaches a perplexity of 30.88.

The results at 25% also show that the shared expert does not explain the gap to the published DOT-MoE results, since without it our uniform conversions still fall below the published averages excluding BoolQ, 60.5 on Llama-2-7B and 63.1 on Qwen2.5-7B. We address this gap directly in the DOT-MoE reproduction paragraph below.

We also investigated dynamic allocation without the shared expert early on in our research, and we report it here in order to address whether our improvement over uniform allocation depends on it. Below are the results.

ModelAllocationPPLAvg. excl. BoolQ (%)Six-task avg. (%)
Llama-2-7BUniform12.1558.959.4
Dynamic11.2659.860.6

From this we can see that our dynamic allocation approach still improves on uniform allocation without the shared expert, so its improvement does not depend on the shared expert.

DOT-MoE reproduction. Since no public implementation of DOT-MoE was available, we replicated it to the best of our ability based on the description and the hyperparameters provided in the paper. (The hyperparameters were sourced from Tables 6 and 7 of the DOT-MoE paper.) Despite our inability to exactly replicate the numbers found in the paper, we still are able to exceed the performance of the published DOT-MoE numbers (not our replication) on two of the three models tested.

We address your request for sensitivity-based and search-based allocation baselines, a frozen mean-dynamic schedule, and a dynamic allocator without revision heads in our response to W1/Q2.

We have edited Appendix D to include these extra seed experiments.

AuthorsOpenReview ↗

Response to Reviewer JSCk W1 / Q2

W1/Q2

Frozen mean schedule. We generated an average per-layer utilization from our dynamic model and used it as a schedule in order to address your question about a frozen mean schedule. The schedule averages the count of each layer over 131,072 Dolmino training tokens. Below are the results.

ModelFFN utilizationUniform PPLDynamic PPLFrozen mean PPL
Llama-2-7B12.5%15.9314.8817.03
25%11.6010.5713.40
50%8.277.349.47
Llama-3-8B12.5%27.6825.4628.99
25%19.4516.4620.51
50%12.189.6513.78
Qwen2.5-7B12.5%23.5721.1424.70
25%16.8714.2218.37
50%11.359.1812.06

From this we can see that the frozen mean schedule is worse than uniform allocation at every model and utilization. This heavily implies that the token-specific utilizations learned by the dynamic approach have a significant effect on the overall accuracy.

Revision heads. We ran our dynamic allocation approach with revision at every fourth layer and without revision in order to address your question about removing intermediate revision heads. Due to time and compute constraints we chose to only run the no-revision and every-fourth-layer variants on Llama-2-7B at 25% FFN utilization, with the same expert budget, training tokens, and initialization as the paper. Below are the results.

AllocationAllocator headsPPLAvg. excl. BoolQ (%)Six-task avg. (%)
Uniform–11.6159.259.9
Static–11.5060.060.8
Dynamic, no revision110.9261.262.4
Dynamic, every 4th layer810.5561.161.7
Dynamic, every layer3110.5762.062.8

From this we can see that sequential revision accounts for about a third of the perplexity reduction and that revision at every fourth layer already matches revision at every layer on perplexity, so we thank you for suggesting the removal of intermediate revision heads, and we will highlight minimizing their number as a potential area for future work in the final version.

Layer sensitivity. We investigated fixed schedules from layer sensitivity and from search on Llama-2-7B early on in our research, and we report one of each here in order to address your request for sensitivity-based and search-based allocation baselines. For the sensitivity-based schedule, we create a performance degradation model by applying the MoE carving algorithm to one layer at a time and measuring the change in perplexity caused by carving just that layer while every other layer stays dense. Adding the degradation of each layer at its utilization estimates the degradation of a whole configuration, and dynamic programming over this model finds the configuration with the lowest estimated degradation at 25% FFN utilization. For the search-based schedule, regularized evolution scores per-layer carving configurations by the perplexity of the carved model, and we recover its best configuration at 25% FFN utilization and a uniform one with the same LoRA fine-tuning. Below are the results.

AllocationUniform PPLPPLChange (%)
Dynamic programming12.4112.16−2.1
Regularized evolution13.1413.17+0.3
Static11.6111.50−1.0
Dynamic11.6110.57−9.0

From this we can see that our dynamic allocation approach improves perplexity far more than the fixed schedules from layer sensitivity and search, which is why we did not pursue such schedules. Since the fixed schedules come from early in our research, whose conversion setups differ from that of the paper, we compare each row with the uniform conversion of its own setup.

We have added an additional section to the appendix (Section H) where we discuss the frozen mean schedule, the no-revision and every-fourth-layer variants, and the fixed schedules from layer sensitivity and search.

AuthorsOpenReview ↗

Response to Reviewer JSCk W2 / Q3 (Part 1/2)

W2/Q3

Analytic model. Since we did not have the time to create dedicated kernels for our approach, we created an analytic model of the inference cost of uniform, static, and dynamic allocation with custom kernels in order to address your concerns regarding whether our approach actually demonstrates a performance improvement. A token computes kℓk_\ell routed experts in layer ℓ\ell, and a decode step at batch size bb reads every routed expert that at least one of its tokens selects. Since these selections overlap, we count the routed experts per token, BB, and the expected routed experts per decode step, UU, for tokens that select at random among the MM routed experts of the model,

B=∑ℓkℓ,U=M(1−(1−BM)b),B = \sum_{\ell} k_\ell, \qquad U = M\Bigl(1 - \bigl(1 - \tfrac{B}{M}\bigr)^{b}\Bigr),

so U=BU = B at batch size 1 and U≈MU \approx M at batch size 256. The model treats a decode step as bound by memory bandwidth and a prefill as bound by arithmetic. It also accounts for the serial nature of the allocator by charging each of the HH allocating layers one extra kernel that runs before its experts,

Tdecode=W+bcv+Uw+Qβ+(K+H),τ,Tprefill=N,g+na+Bf+Fπ+Hτ,T_{\mathrm{decode}} = \frac{W + bcv + Uw + Q}{\beta} + (K + H)\\,\tau, \qquad T_{\mathrm{prefill}} = N\\,\frac{g + na + Bf + F}{\pi} + H\tau,

SymbolValueMeaning
M2,720Routed experts, 85 in each of the 32 layers
w, f3.15 MB, 3.15 MFLOPBytes and FLOPs per token of one routed expert
W, g4.68 GB, 4.42 GFLOPWeights and FLOPs per token outside routed experts
v, a0.524 MB, 0.262 MFLOPKV cache and attention FLOPs per context token
Q, F8.19 MB, 8.19 MFLOPAllocator bytes per decode step and FLOPs per token
H, K, τ31, 388, 1.66 μsAllocating layers, other kernels, and time per kernel
β, π2.36 TB/s, 716 TFLOP/sDense-model bandwidth and throughput on our H100
b, c–Batch size and context length
N, n–Prefill tokens and sequence length

Below are the times that the model predicts for three workloads.

WorkloadUniformStaticDynamic
Decode step, batch 1, context 2,048 (ms)3.983.984.03
Decode step, batch 256, context 256 (ms)20.8120.8120.86
Prefill, 16 sequences of 2,048 tokens (ms)323.7323.7324.1

From this we can see that with custom kernels dynamic allocation would run about as fast as uniform allocation, since even with the allocator running serially before the experts it adds only 55 μs to a decode step, or 1.4% at batch size 1.

These estimates use the constants of Llama-2-7B at 25% FFN utilization in bfloat16, whose 13.5 GB of weights split into 8.56 GB of routed experts, the 4.68 GB in WW, and 0.26 GB of input embeddings. The allocator adds 158,942 parameters, or 0.32 MB, and reading its 0.26 MB shared projection in each of the HH allocating layers accounts for almost all of QQ. Uniform and static allocation have Q=F=H=0Q = F = H = 0, and dynamic allocation keeps BB and UU unchanged, since it bounds the utilization of every token at the same budget of BB routed experts. The model is an estimate as it assumes custom kernels that reach the bandwidth and per-kernel time that we measured on the dense model. This is a reasonable assumption, since both values come from off-the-shelf software, the bandwidth from the cuBLAS matrix multiplications of the dense model, which reach only 70% of the 3.35 TB/s peak of the H100, and the per-kernel time from replaying the dense model as a CUDA graph.

Practical cost. We also measured the inference cost of uniform, static, and dynamic allocation in our current implementation, which runs every conversion in eager PyTorch on one NVIDIA H100 GPU through the Hugging Face implementation of Llama-2-7B at 25% FFN utilization, with our sparse FFN layer running the selected experts as grouped matrix multiplications. Batch size 32 is the largest that fits in memory with a 2,048-token context. Below are the results.

MeasurementUniformStaticDynamic
Prefill, 16 sequences of 2,048 tokens (ms)891.2892.9926.8
Decode step, batch 1, context 2,048 (ms)20.920.531.8
Decode step, batch 32, context 2,048 (ms)54.254.161.0
Decode step, batch 256, context 256 (ms)69.366.471.4
FFN arithmetic per token (GFLOP)2.212.212.21
Routing and allocation arithmetic per token (GFLOP)0.02230.02230.0305
AuthorsOpenReview ↗

Response to Reviewer JSCk W2 / Q3 (Part 2/2) and Q4

W2/Q3 (continued)

From this we can see that the extra latency of dynamic allocation comes from dispatching its per-token expert counts from PyTorch, which at batch size 1 adds 670 kernel launches and 97 CPU-GPU synchronizations to a decode step but only 1.5 ms of GPU time. Additionally, this shows that dynamic allocation adds almost no arithmetic to uniform allocation, so custom kernels would remove most of its extra time, as the analytic model predicts. Our measured 20.9 ms for a uniform decode step at batch size 1 is far above the modeled 3.98 ms mainly because eager PyTorch launches every kernel from Python, which leaves the GPU idle for more than half of the step, and partly because of the many small unfused kernels of the Hugging Face implementation and slower weight streaming, which costs all three allocations about the same.

Conversion cost. We measured the additional conversion time of our dynamic allocation approach relative to DOT-MoE, which is uniform allocation on our learned base, in order to address your concern that allocator training adds work that should enter the cost comparison. Both conversions train for 3,500 steps on four H100 GPUs, and we count their GPU-hours at the steady-state time per step, including the periodic evaluations. Qwen2.5-7B takes about twice as long because its 152,064-token vocabulary makes the full-vocabulary distillation loss too large for two sequences per GPU, so it trains on one sequence per GPU with twice as many accumulation steps. Below are the results.

Llama-2-7BLlama-3-8BQwen2.5-7B
DOT-MoE conversion (GPU-hours)30.336.662.1
Additional time of dynamic allocation (GPU-hours)+3.6+3.5+0.4

From this we can see that our dynamic allocator adds only a few GPU-hours to a conversion. Importantly, this time does not come from the computation of the allocator, which on Llama-2-7B adds only F = 8.19 MFLOP per token, under 0.1% of the arithmetic of a training step, while every token keeps the same budget of B routed experts. It comes instead from dispatching the per-token expert counts in our implementation, so, as at inference, custom kernels would remove most of this time. Additionally, this shows that the allocator is only a one-time cost of training, since a model is converted once and then serves every inference afterwards.

We have also replaced the analytic rows of the main results table with conversions that train the allocator jointly with the LoRA recovery, and Appendix I now reports the allocator trained after the recovery.

We have added an additional section to the appendix (Section G) where we discuss the latency effects and conversion cost of our approach compared to uniform.

Q4

Model selection. We used the WikiText-2 test split as a validation set for hyperparameter and architectural tuning, and we never used the downstream accuracy benchmarks for selection. We re-scored every model of the paper on the WikiText-2 validation split, which our tuning never used and which therefore serves as a test set for our models, in order to address your concerns with potential overfitting. Below are the results.

ModelTest PPLValidation PPL
Dense model
Llama-2-7B5.475.58
Llama-3-8B6.146.38
Qwen2.5-7B6.857.34
Inclusion temperature, static allocation on Llama-2-7B
Uniform control16.3016.78
T = 0.2516.5716.89
T = 116.1516.62
T = 1.515.8916.57
Learned base at 25%, uniform / static / dynamic
Llama-2-7B11.61 / 11.50 / 10.5711.82 / 11.70 / 10.79
Llama-3-8B19.40 / 18.19 / 16.4420.24 / 18.78 / 17.10
Qwen2.5-7B16.92 / 15.99 / 14.2117.55 / 16.71 / 14.83
Analytic base at 25%, uniform / static / dynamic
Llama-2-7B23.14 / 21.07 / 18.0623.87 / 21.58 / 18.44
Llama-3-8B44.53 / 38.09 / 28.0146.59 / 39.73 / 29.05
Qwen2.5-7B44.05 / 39.86 / 28.9746.17 / 41.86 / 30.34

From this we can see that the validation split supports the main comparisons, since the ranking of uniform, static, and dynamic allocation is the same on both splits for every model.

We now report the WikiText-2 validation perplexity, which serves as our test set, as the headline perplexity in the main-body tables. We have edited Appendix C.1 and Appendix F to include these validation results, and Appendix C.1 now states that we used the test split for hyperparameter selection and architectural design.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Note on Joint Allocator Training for the Analytic Base

We would like to highlight one change to our analytic-base results outside of individual reviewer responses. In the submission, the analytic base trained the allocator sequentially, after CMoE's LoRA recovery had finished, which required a second training stage with its own full forward and backward passes. In the revision, the allocator trains jointly with the LoRA recovery, within the same 1,024 steps over the same tokens. The objection to sequential training was raised in one of the reviews, although we had already begun this work before the reviews were released since we wanted both learned and analytic approaches to train only from the gradients that the base conversion already computes, thus requiring no additional full gradient computations. Below are the results, where each value compares dynamic allocation with the uniform conversion trained in the same way, perplexity reductions are on the WikiText-2 validation split, and accuracy gains are in points.

ModelPPL reduction, sequential (%)PPL reduction, joint (%)Six-task avg. gain, sequentialSix-task avg. gain, joint
Llama-2-7B11.822.7+1.2+1.3
Llama-3-8B14.337.6+0.6+2.4
Qwen2.5-7B4.334.3+0.2+1.2

From this we can see that training the allocator jointly with the recovery increases its perplexity reduction over uniform allocation by two to eight times, and increases its accuracy gain on every model. Additionally, this shows that the allocator does not need a training stage of its own, since it trains within the token budget of the CMoE recovery.

Table 1 of the revised paper now reports the jointly trained analytic base, and Appendix I sets the sequential and joint conversions side by side and discusses this comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.