- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Spectral-MoE learns how to distribute active experts across layers during dense-to-MoE conversion. The static version learns one count per layer, while the dynamic version revises each token's remaining allocation using intermediate hidden states. Both conserve the same total expert budget for every token. The method is evaluated on CMoE- and DOT-MoE-based conversions of three models, with dynamic allocation improving perplexity and downstream accuracy across several utilization levels.
Critical strengths
-
The budget constraint makes the main comparison meaningful. Dynamic allocation cannot improve quality simply by activating more experts on difficult tokens: it must redistribute a fixed total across layers. The construction explicitly handles integer counts and remaining-layer feasibility, making this a useful extension of conventional uniform routing.
-
The gains are reasonably consistent. Dynamic allocation improves both perplexity and mean accuracy across all nine learned-base model/utilization combinations. It also lowers perplexity when the analytic partition and router are frozen, which supports the usefulness of allocation beyond joint partition training alone. Although the strongest baseline comparison needs clarification, this consistency makes me lean toward acceptance.
Critical weaknesses
-
The gap between the reproduced and published DOT-MoE baselines complicates the interpretation. On Qwen2.5-7B, uniform BoolQ accuracy is 43.7%, versus the published 64.3%; dynamic allocation reaches 65.2%. This single task accounts for much of the reported mean improvement on that model. Dynamic matches, rather than exceeds, published DOT-MoE's overall Qwen accuracy. The authors appropriately disclose the discrepancy, but do not explain it. I would like to know whether allocation addresses a genuine limitation of uniform routing or partly compensates for an implementation or optimization weakness.
-
The need for token-specific revision is not yet well isolated. Figure 5 shows a strong average preference for later layers, but the paper does not test a fixed schedule obtained from the dynamic allocator's average counts while keeping its trained partition and router unchanged. The separately trained static allocator cannot answer this question because its optimization and resulting expert structure differ. This matters for deployment: a fixed nonuniform schedule could retain much of the benefit with simpler execution. The paper also gives no latency measurements showing the cost of sequential allocator revisions and variable expert counts.
Questions for authors
-
What explains the particularly low uniform BoolQ result on Qwen? Does it recur across paired seeds, and have you checked whether the shared-expert modification or another implementation difference accounts for it? Resolving this would strengthen the comparison with published DOT-MoE.
-
If you freeze a dynamic checkpoint's partition and router and replace its token-dependent allocation with a budget-preserving, rounded average schedule, how much of the quality improvement remains? This would directly test the value of adapting allocation to each token.
-
What is the inference overhead relative to uniform and static allocation at the same utilization? Even a representative single-token and batched measurement would help establish whether the quality gain comes with a meaningful execution penalty.
3 comments
Response to Reviewer 3Nwo W2 / Q2 and Q3 (Part 1/2)
W2/Q2
Frozen mean schedule. We generated an average per-layer utilization from our dynamic model and used it as a schedule in order to address your question about how much of the quality improvement remains with an average schedule. The schedule averages the count of each layer over 131,072 Dolmino training tokens. Below are the results.
| Model | FFN utilization | Uniform PPL | Dynamic PPL | Frozen mean PPL |
|---|---|---|---|---|
| Llama-2-7B | 12.5% | 15.93 | 14.88 | 17.03 |
| 25% | 11.60 | 10.57 | 13.40 | |
| 50% | 8.27 | 7.34 | 9.47 | |
| Llama-3-8B | 12.5% | 27.68 | 25.46 | 28.99 |
| 25% | 19.45 | 16.46 | 20.51 | |
| 50% | 12.18 | 9.65 | 13.78 | |
| Qwen2.5-7B | 12.5% | 23.57 | 21.14 | 24.70 |
| 25% | 16.87 | 14.22 | 18.37 | |
| 50% | 11.35 | 9.18 | 12.06 |
From this we can see that none of the quality improvement remains with the frozen mean schedule, which is worse than uniform allocation at every model and utilization. This heavily implies that the token-specific utilizations learned by the dynamic approach have a significant effect on the overall accuracy.
We report the latency of sequential allocator revisions and variable expert counts in our response to Q3.
We have added an additional section to the appendix (Section H) where we discuss the frozen mean schedule.
Q3
Analytic model. Since we do not have the time to create dedicated kernels for our approach, we created an analytic model of the inference cost of uniform, static, and dynamic allocation with custom kernels in order to address your question about the inference overhead relative to uniform and static allocation. A token computes routed experts in layer , and a decode step at batch size reads every routed expert that at least one of its tokens selects. Since these selections overlap, we count the routed experts per token, , and the expected routed experts per decode step, , for tokens that select at random among the routed experts of the model,
so at batch size 1 and at batch size 256. The model treats a decode step as bound by memory bandwidth and a prefill as bound by arithmetic. It also accounts for the serial nature of the allocator by charging each of the allocating layers one extra kernel that runs before its experts,
| Symbol | Value | Meaning |
|---|---|---|
| M | 2,720 | Routed experts, 85 in each of the 32 layers |
| w, f | 3.15 MB, 3.15 MFLOP | Bytes and FLOPs per token of one routed expert |
| W, g | 4.68 GB, 4.42 GFLOP | Weights and FLOPs per token outside routed experts |
| v, a | 0.524 MB, 0.262 MFLOP | KV cache and attention FLOPs per context token |
| Q, F | 8.19 MB, 8.19 MFLOP | Allocator bytes per decode step and FLOPs per token |
| H, K, τ | 31, 388, 1.66 μs | Allocating layers, other kernels, and time per kernel |
| β, π | 2.36 TB/s, 716 TFLOP/s | Dense-model bandwidth and throughput on our H100 |
| b, c | – | Batch size and context length |
| N, n | – | Prefill tokens and sequence length |
Below are the times that the model predicts for three workloads.
| Workload | Uniform | Static | Dynamic |
|---|---|---|---|
| Decode step, batch 1, context 2,048 (ms) | 3.98 | 3.98 | 4.03 |
| Decode step, batch 256, context 256 (ms) | 20.81 | 20.81 | 20.86 |
| Prefill, 16 sequences of 2,048 tokens (ms) | 323.7 | 323.7 | 324.1 |
From this we can see that with custom kernels dynamic allocation would run about as fast as uniform allocation, since even with the allocator running serially before the experts it adds only 55 μs to a decode step, or 1.4% at batch size 1.
Response to Reviewer 3Nwo Q3 (Part 2/2)
Q3 (continued)
These estimates use the constants of Llama-2-7B at 25% FFN utilization in bfloat16, whose 13.5 GB of weights split into 8.56 GB of routed experts, the 4.68 GB in , and 0.26 GB of input embeddings. The allocator adds 158,942 parameters, or 0.32 MB, and reading its 0.26 MB shared projection in each of the allocating layers accounts for almost all of . Uniform and static allocation have , and dynamic allocation keeps and unchanged, since it bounds the utilization of every token at the same budget of routed experts. The model is an estimate as it assumes custom kernels that reach the bandwidth and per-kernel time that we measured on the dense model. This is a reasonable assumption, since both values come from off-the-shelf software, the bandwidth from the cuBLAS matrix multiplications of the dense model, which reach only 70% of the 3.35 TB/s peak of the H100, and the per-kernel time from replaying the dense model as a CUDA graph.
Practical cost. We also measured the inference cost of uniform, static, and dynamic allocation in our current implementation, which runs every conversion in eager PyTorch on one NVIDIA H100 GPU through the Hugging Face implementation of Llama-2-7B at 25% FFN utilization, with our sparse FFN layer running the selected experts as grouped matrix multiplications. Batch size 32 is the largest that fits in memory with a 2,048-token context. Below are the results.
| Measurement | Uniform | Static | Dynamic |
|---|---|---|---|
| Prefill, 16 sequences of 2,048 tokens (ms) | 891.2 | 892.9 | 926.8 |
| Decode step, batch 1, context 2,048 (ms) | 20.9 | 20.5 | 31.8 |
| Decode step, batch 32, context 2,048 (ms) | 54.2 | 54.1 | 61.0 |
| Decode step, batch 256, context 256 (ms) | 69.3 | 66.4 | 71.4 |
| FFN arithmetic per token (GFLOP) | 2.21 | 2.21 | 2.21 |
| Routing and allocation arithmetic per token (GFLOP) | 0.0223 | 0.0223 | 0.0305 |
From this we can see that the extra latency of dynamic allocation comes from dispatching its per-token expert counts from PyTorch, which at batch size 1 adds 670 kernel launches and 97 CPU-GPU synchronizations to a decode step but only 1.5 ms of GPU time. Additionally, this shows that dynamic allocation adds almost no arithmetic to uniform allocation, so custom kernels would remove most of its extra time, as the analytic model predicts. Our measured 20.9 ms for a uniform decode step at batch size 1 is far above the modeled 3.98 ms mainly because eager PyTorch launches every kernel from Python, which leaves the GPU idle for more than half of the step, and partly because of the many small unfused kernels of the Hugging Face implementation and slower weight streaming, which costs all three allocations about the same.
We have added an additional section to the appendix (Section G) where we discuss the latency effects and conversion cost of our approach compared to uniform.
Response to Reviewer 3Nwo W1 / Q1
Thank you for your efforts in reviewing our work. Below are our responses to the concerns raised and questions asked.
W1/Q1
Repeatability. We ran three seeds for the dynamic, static, and uniform allocation approaches in order to address your question about whether the low uniform BoolQ result recurs across paired seeds. Due to time and compute constraints we chose to only run on Qwen2.5-7B, as that would also allow us to answer whether BoolQ recovery issues are systemic or just a result of the seed selected. Below are the results.
From this we can see that our dynamic allocation approach does show consistent benefit over multiple seeds when compared to static and uniform allocation. The improvement over uniform allocation ranges from 2.71 to 2.76 in perplexity and stays above 2 points on average when excluding BoolQ across the three seeds. Additionally, this shows that the low uniform BoolQ result you identified comes from seed variability rather than from the shared expert.
Shared expert. We investigated DOT-MoE-style conversions without the shared expert at various utilizations early on in our research, and we report them here in order to address whether adding the shared expert accounts for the gap to the published DOT-MoE results. Below are the results.
From this we can see that the shared expert lowers perplexity and improves the average excluding BoolQ by at least 0.5 points on both models. Also, the addition of the shared expert drastically improves the training stability in some of the lower utilization regimes we were exploring. Below are the results of the uniform conversions of Llama-2-7B that we trained at these utilizations, compared at step 2,518, the last step that all four runs reached.
From this we can see that below some utilization the shared expert becomes pivotal for training stability, since both conversions train at 12.8% FFN utilization, while at 5.8% the conversion without the shared expert diverges and the one with it still reaches a perplexity of 30.88.
The results at 25% also show that the shared expert does not explain the gap to the published DOT-MoE results, since without it our uniform conversions still fall below the published averages excluding BoolQ, 60.5 on Llama-2-7B and 63.1 on Qwen2.5-7B. We address this gap directly in the DOT-MoE reproduction paragraph below.
We also investigated dynamic allocation without the shared expert early on in our research, and we report it here in order to address whether our improvement over uniform allocation depends on it. Below are the results.
From this we can see that our dynamic allocation approach still improves on uniform allocation without the shared expert, so its improvement does not depend on the shared expert.
DOT-MoE reproduction. Since no public implementation of DOT-MoE was available, we replicated it to the best of our ability based on the description and the hyperparameters provided in the paper. (The hyperparameters were sourced from Tables 6 and 7 of the DOT-MoE paper.) Despite our inability to exactly replicate the numbers found in the paper, we still are able to exceed the performance of the published DOT-MoE numbers (not our replication) on two of the three models tested.
We have edited Appendix D to include these extra seed experiments.