- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies inference acceleration for mixture-of-experts diffusion language models on memory constrained devices. It identifies that processing multiple tokens simultaneously can require loading a large combined set of expert weights. The proposed method selects a fixed-size expert pool based on the current router scores at each MoE layer and denoising step, then performs token wise top-k routing within that pool. It further exploits similarities in expert selections across denoising iterations to encourage expert reuse. Experiments on DiffusionGemma and SDAR evaluate the impact on accuracy, while simulations estimate speedups under the specified hardware and memory assumptions.
Critical strengths
-
The paper identifies that MoE expert weights can impose substantial GPU memory demands in dLLMs: the union of experts required at each denoising step can be large, and expert selections exhibit overlap across denoising steps.
-
The paper is clearly written and easy to follow.
Critical weaknesses
The method lacks sufficient validation on real devices, as the reported speedups are derived entirely from simulation. Sensitivity analysis for some parameters, such as the expert pool size, is also insufficient. In addition, the accuracy and speedup evaluations use different configurations, making it difficult to assess the quality–efficiency trade-off consistently.
Questions for authors
-
Could the authors provide measurements and analysis on real hardware?
-
Could the authors provide a sensitivity analysis under non-ideal overlap between computation and data transfer, as well as varying bandwidth conditions?
3 comments
Response 2/2
Reviewer ypLX — response 2/2
Q2: Sensitivity to non-ideal overlap and bandwidth
Could the authors provide a sensitivity analysis under non-ideal overlap between computation and data transfer, as well as varying bandwidth conditions?
Answer:
We thank the reviewer for this excellent suggestion. We have added both analyses and also evaluated faster flash together with non-ideal overlap. Removing ideal overlap reduces the projected speedup but preserves a substantial benefit on both models.
Non-ideal and partial overlap. We compare the original MoE term,
with the non-ideal variant requested by the reviewer,
This variant assumes that loading experts from flash does not overlap with compute/DRAM work, so the two times are added. Both variants add the sequential costs of the modeled non-MoE components. We also interpolate between these endpoints with an overlap fraction : writing and , we use . Here represents ideal overlap and represents no overlap between flash transfers and compute/DRAM work.
Amortized speedup averaged over A18 / A19 Pro / MTK D9400, with , int4 weights. Native routing is evaluated under the same overlap assumption in each column.
| Model | Method | N | Ideal, α=1 | α=0.75 | α=0.5 | α=0.25 | Non-ideal, α=0 |
|---|---|---|---|---|---|---|---|
| DiffusionGemma | PPB | 32 | 5.77× | 5.54× | 5.34× | 5.15× | 4.98× |
| SB max | 32 | 4.67× | 4.54× | 4.41× | 4.30× | 4.19× | |
| PPB | 48 | 2.03× | 2.01× | 2.00× | 1.98× | 1.97× | |
| SB max | 48 | 1.90× | 1.89× | 1.87× | 1.86× | 1.85× | |
| SDAR | PPB | 32 | 9.09× | 8.16× | 7.43× | 6.84× | 6.34× |
| SB max | 32 | 5.13× | 4.83× | 4.58× | 4.35× | 4.16× |
At , PPB retains 4.98× speedup on DiffusionGemma and 6.34× on SDAR with no overlap between flash transfers and compute/DRAM work, approximately 86% and 70% of its ideal-overlap speedup. SB max retains 4.19× and 4.16×. The ordering of the methods is preserved throughout the sweep. The non-ideal formulation also follows the copy-then-execute order of our A100 runtime, where the paired roofline analysis closely predicts the measured seconds saved, as discussed in Q1.
Flash bandwidth. The method reduces flash reads, so its gain shrinks as flash gets faster and grows as it gets slower. We varied effective flash bandwidth from 1 to 8 GB/s while holding A18 compute throughput and DRAM bandwidth fixed and using .
Amortized speedup versus effective flash read bandwidth, with ideal overlap.
| Model | Method | 1 GB/s | 2 GB/s | 2.98 GB/s | 4 GB/s | 6 GB/s | 8 GB/s |
|---|---|---|---|---|---|---|---|
| DiffusionGemma | PPB | 6.7× | 6.1× | 5.7× | 5.3× | 4.7× | 4.2× |
| SB max | 5.2× | 4.9× | 4.6× | 4.4× | 4.0× | 3.6× | |
| SDAR | PPB | 9.8× | 9.3× | 8.9× | 8.5× | 6.1× | 4.6× |
| SB max | 5.4× | 5.2× | 5.1× | 4.9× | 4.7× | 4.4× |
At 8 GB/s, twice the fastest peak flash read bandwidth in Table 1, the methods retain 3.6–4.6× modeled speedup. This is consistent with the source of the benefit: faster flash reduces the cost of each avoided transfer, while a smaller expert working set still removes most of the transfers.
Faster flash and non-ideal overlap together. At 8 GB/s with no overlap between flash transfers and compute/DRAM work, PPB still achieves 3.27× on DiffusionGemma and 3.78× on SDAR; SB max achieves 2.96× and 2.90×. Thus, the advantage persists when both assumptions are changed together. Appendix H.1 and H.4 provide the bandwidth and overlap analyses, with the physical-cache, cold-start, and eviction-policy studies in H.2, H.3, and H.5.
We hope that these additional measurements and sensitivity analyses address the reviewer's concerns and clarify the conditions under which expert working-set compression translates into acceleration. We would be happy to clarify any remaining questions.
Thank you for your rebuttal, as well as for conducting the additional experiments. Your responses have satisfactorily addressed my concerns, and I have updated my review score accordingly.
Response 1/2
We thank the reviewer for their constructive assessment and helpful suggestions. We appreciate the request for stronger hardware validation and a clearer assessment of the quality–efficiency trade-off. In response, we have added a real-hardware study with paired roofline predictions, extended the expert-pool and physical-cache sweeps, and evaluated sensitivity to non-ideal overlap and bandwidth. The hardware study is presented in the global response and Appendix F; the additional sensitivity analyses are in Appendix H. We summarize the findings relevant to each concern below.
W1
We address hardware validation in Q1 below and the overlap and bandwidth assumptions in Q2. Here we clarify the expert-pool sensitivity and the correspondence between accuracy and efficiency.
Expert-pool size and physical cache capacity.
We have additionally swept the physical cache independently of the routing budget, with C∈{16,24,32,40,48,64,96}. The following subset illustrates that the advantage persists when the cache does not hold exactly 32 experts.
Amortized speedup at N=32, LRU, averaged over A18 / A19 Pro / MTK D9400.
Thus, the benefit extends to cache capacities both below and above the routing budget. Larger caches also benefit native routing, which explains why the relative gain eventually decreases as native expert demand fits more readily. Our existing evaluation includes accuracy ablations over N∈{16,32,48,64,96,128} in Tables 4 and 10, comparisons of sum, max, and state-balanced max selection in Tables 5 and 6, and temporal-reuse ablations in Appendix E. Newly added Appendix H additionally compares LRU, LFU, and offline-optimal Belady eviction and evaluates alternative cold-start accounting. Together, these results characterize how the benefit depends on the available memory and cache behavior.
Consistency of accuracy and efficiency evaluations. We agree that their correspondence should be explicit. Table 3 reports accuracy alongside per-step and amortized speedup for each routing policy and pool size. The amortized metric accounts for the method's observed denoising-step overhead, so a reduction in per-step expert demand is not presented without its generation cost. Benchmark accuracy and timing traces are collected in separate evaluation runs; the mobile speedups are estimates under the stated hardware and precision assumptions.
The additional hardware study provides a direct comparison of measured denoising time and roofline predictions on the same runs, using each method's actual expert demand, cache misses, and denoising forwards. For the hardware sensitivity sweeps, we likewise hold the recorded routing traces and denoising work fixed and vary only the cache or cost-model assumptions. These sweeps therefore isolate hardware sensitivity without changing the routing policy or its generated outputs. The int4 setting in the mobile simulator specifies weight-storage and transfer costs; it does not establish the accuracy of an int4 deployment, whose quantization effects remain a limitation discussed in the global response.
Q1: Measurements and analysis on real hardware
Answer: We have conducted an additional study on two A100-SXM4-80GB GPUs with a per-layer LRU expert cache of C=32 on the compute GPU and the full expert bank on the second GPU. Cache misses trigger real NVLink transfers before expert execution. This enforces a bounded resident-weight budget and reduces peak allocated memory to approximately 21 GB for DiffusionGemma and 19 GB for SDAR, compared with 54 GB and 61 GB with the full models resident.
The method produces substantial acceleration in this real runtime. With N=32, measured denoising GPU-time speedups are:
The matching roofline analysis predicts the absolute time saved to within approximately 4%: pruning removes 151–157 s of denoising time on DiffusionGemma and 336–361 s on SDAR. Expert-bank traffic falls by 7.3–10.0× and 7.1–13.7×, respectively. The measured speedup ratios are lower than the roofline ratios because framework and other runtime work outside the model form a larger fraction of the shorter pruned runs. This shared work largely cancels when comparing seconds saved.
The global response and Appendix F provide the setup, absolute timings, memory and traffic measurements, and the paired ideal- and non-ideal-overlap predictions, including a TEAM comparison, establishing the benefit in a memory-capped GPU deployment.