acceptodds
Under review as a conference paper at ICLR 2027

Diffusion MoE LLMs for Memory Constrained Deployment

Abstract

Diffusion large language models (dLLMs) generate multiple tokens in parallel through iterative refinement, offering an alternative to sequential autoregressive decoding. Many recent dLLMs adopt Mixture-of-Experts (MoE) backbones, in which each token activates only a small subset of experts. However, when many tokens are processed simultaneously, their combined routes can span much the expert bank. This is particularly costly on memory-constrained devices, where the full expert bank cannot remain in DRAM and non-resident expert weights must be transferred from flash during inference. We propose dynamic just-in-time expert pruning, a training-free method that restricts routing at each layer and denoising step to a compact expert set selected from the model's current router scores. We further introduce an approach that favors experts selected in the preceding iteration, reducing expert-pool turnover and associated memory traffic. Experiments on DiffusionGemma 26B-A4B and SDAR-30B-A3B show that restricting the selected expert working set to 32 of 128 experts preserves benchmark performance within of the full baseline while substantially reducing expert-weight demand. Trace-driven simulations estimate up to relative modeled speedup on the evaluated mobile hardware configurations.

Primary area:
generative models
Keywords:
diffusion llms, mixture of experts

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ypLXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies inference acceleration for mixture-of-experts diffusion language models on memory constrained devices. It identifies that processing multiple tokens simultaneously can require loading a large combined set of expert weights. The proposed method selects a fixed-size expert pool based on the current router scores at each MoE layer and denoising step, then performs token wise top-k routing within that pool. It further exploits similarities in expert selections across denoising iterations to encourage expert reuse. Experiments on DiffusionGemma and SDAR evaluate the impact on accuracy, while simulations estimate speedups under the specified hardware and memory assumptions.

Critical strengths
  1. The paper identifies that MoE expert weights can impose substantial GPU memory demands in dLLMs: the union of experts required at each denoising step can be large, and expert selections exhibit overlap across denoising steps.

  2. The paper is clearly written and easy to follow.

Critical weaknesses

The method lacks sufficient validation on real devices, as the reported speedups are derived entirely from simulation. Sensitivity analysis for some parameters, such as the expert pool size, is also insufficient. In addition, the accuracy and speedup evaluations use different configurations, making it difficult to assess the quality–efficiency trade-off consistently.

Questions for authors
  1. Could the authors provide measurements and analysis on real hardware?

  2. Could the authors provide a sensitivity analysis under non-ideal overlap between computation and data transfer, as well as varying bandwidth conditions?

3 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for their constructive assessment and helpful suggestions. We appreciate the request for stronger hardware validation and a clearer assessment of the quality–efficiency trade-off. In response, we have added a real-hardware study with paired roofline predictions, extended the expert-pool and physical-cache sweeps, and evaluated sensitivity to non-ideal overlap and bandwidth. The hardware study is presented in the global response and Appendix F; the additional sensitivity analyses are in Appendix H. We summarize the findings relevant to each concern below.

W1

We address hardware validation in Q1 below and the overlap and bandwidth assumptions in Q2. Here we clarify the expert-pool sensitivity and the correspondence between accuracy and efficiency.

Expert-pool size and physical cache capacity.

We have additionally swept the physical cache independently of the routing budget, with C∈{16,24,32,40,48,64,96}C\in\{16,24,32,40,48,64,96\}. The following subset illustrates that the advantage persists when the cache does not hold exactly 32 experts.

Amortized speedup at N=32N=32, LRU, averaged over A18 / A19 Pro / MTK D9400.

ModelMethodC=24C=32C=40
DiffusionGemmaPPB3.5×5.8×5.0×
SB max3.3×4.7×4.7×
SDARPPB3.5×9.1×7.8×
SB max2.8×5.2×5.7×

Thus, the benefit extends to cache capacities both below and above the routing budget. Larger caches also benefit native routing, which explains why the relative gain eventually decreases as native expert demand fits more readily. Our existing evaluation includes accuracy ablations over N∈{16,32,48,64,96,128}N\in\{16,32,48,64,96,128\} in Tables 4 and 10, comparisons of sum, max, and state-balanced max selection in Tables 5 and 6, and temporal-reuse ablations in Appendix E. Newly added Appendix H additionally compares LRU, LFU, and offline-optimal Belady eviction and evaluates alternative cold-start accounting. Together, these results characterize how the benefit depends on the available memory and cache behavior.

Consistency of accuracy and efficiency evaluations. We agree that their correspondence should be explicit. Table 3 reports accuracy alongside per-step and amortized speedup for each routing policy and pool size. The amortized metric accounts for the method's observed denoising-step overhead, so a reduction in per-step expert demand is not presented without its generation cost. Benchmark accuracy and timing traces are collected in separate evaluation runs; the mobile speedups are estimates under the stated hardware and precision assumptions.

The additional hardware study provides a direct comparison of measured denoising time and roofline predictions on the same runs, using each method's actual expert demand, cache misses, and denoising forwards. For the hardware sensitivity sweeps, we likewise hold the recorded routing traces and denoising work fixed and vary only the cache or cost-model assumptions. These sweeps therefore isolate hardware sensitivity without changing the routing policy or its generated outputs. The int4 setting in the mobile simulator specifies weight-storage and transfer costs; it does not establish the accuracy of an int4 deployment, whose quantization effects remain a limitation discussed in the global response.

Q1: Measurements and analysis on real hardware

Could the authors provide measurements and analysis on real hardware?

Answer: We have conducted an additional study on two A100-SXM4-80GB GPUs with a per-layer LRU expert cache of C=32C=32 on the compute GPU and the full expert bank on the second GPU. Cache misses trigger real NVLink transfers before expert execution. This enforces a bounded resident-weight budget and reduces peak allocated memory to approximately 21 GB for DiffusionGemma and 19 GB for SDAR, compared with 54 GB and 61 GB with the full models resident.

The method produces substantial acceleration in this real runtime. With N=32N=32, measured denoising GPU-time speedups are:

ModelSB maxPPB
DiffusionGemma2.74×2.94×
SDAR3.40×4.12×

The matching roofline analysis predicts the absolute time saved to within approximately 4%: pruning removes 151–157 s of denoising time on DiffusionGemma and 336–361 s on SDAR. Expert-bank traffic falls by 7.3–10.0× and 7.1–13.7×, respectively. The measured speedup ratios are lower than the roofline ratios because framework and other runtime work outside the model form a larger fraction of the shorter pruned runs. This shared work largely cancels when comparing seconds saved.

The global response and Appendix F provide the setup, absolute timings, memory and traffic measurements, and the paired ideal- and non-ideal-overlap predictions, including a TEAM comparison, establishing the benefit in a memory-capped GPU deployment.

AuthorsOpenReview ↗

Response 2/2

Reviewer ypLX — response 2/2

Q2: Sensitivity to non-ideal overlap and bandwidth

Could the authors provide a sensitivity analysis under non-ideal overlap between computation and data transfer, as well as varying bandwidth conditions?

Answer:

We thank the reviewer for this excellent suggestion. We have added both analyses and also evaluated faster flash together with non-ideal overlap. Removing ideal overlap reduces the projected speedup but preserves a substantial benefit on both models.

Non-ideal and partial overlap. We compare the original MoE term,

TMoEideal=max⁡(FΘcompute,BDRAMβDRAM,Bflashβflash),T_{\mathrm{MoE}}^{\mathrm{ideal}}=\max\left(\frac{F}{\Theta_{\mathrm{compute}}},\frac{B_{\mathrm{DRAM}}}{\beta_{\mathrm{DRAM}}},\frac{B_{\mathrm{flash}}}{\beta_{\mathrm{flash}}}\right),

with the non-ideal variant requested by the reviewer,

TMoEnonideal=max⁡(FΘcompute,BDRAMβDRAM)+Bflashβflash.T_{\mathrm{MoE}}^{\mathrm{nonideal}}=\max\left(\frac{F}{\Theta_{\mathrm{compute}}},\frac{B_{\mathrm{DRAM}}}{\beta_{\mathrm{DRAM}}}\right)+\frac{B_{\mathrm{flash}}}{\beta_{\mathrm{flash}}}.

This variant assumes that loading experts from flash does not overlap with compute/DRAM work, so the two times are added. Both variants add the sequential costs of the modeled non-MoE components. We also interpolate between these endpoints with an overlap fraction α\alpha: writing Tlocal=max⁡(F/Θcompute,BDRAM/βDRAM)T_{\mathrm{local}}=\max(F/\Theta_{\mathrm{compute}},B_{\mathrm{DRAM}}/\beta_{\mathrm{DRAM}}) and Tflash=Bflash/βflashT_{\mathrm{flash}}=B_{\mathrm{flash}}/\beta_{\mathrm{flash}}, we use TMoE(α)=max⁡(Tlocal,Tflash)+(1−α)min⁡(Tlocal,Tflash)T_{\mathrm{MoE}}(\alpha)=\max(T_{\mathrm{local}},T_{\mathrm{flash}})+(1-\alpha)\min(T_{\mathrm{local}},T_{\mathrm{flash}}). Here α=1\alpha=1 represents ideal overlap and α=0\alpha=0 represents no overlap between flash transfers and compute/DRAM work.

Amortized speedup averaged over A18 / A19 Pro / MTK D9400, with C=32C=32, int4 weights. Native routing is evaluated under the same overlap assumption in each column.

ModelMethodNIdeal, α=1α=0.75α=0.5α=0.25Non-ideal, α=0
DiffusionGemmaPPB325.77×5.54×5.34×5.15×4.98×
SB max324.67×4.54×4.41×4.30×4.19×
PPB482.03×2.01×2.00×1.98×1.97×
SB max481.90×1.89×1.87×1.86×1.85×
SDARPPB329.09×8.16×7.43×6.84×6.34×
SB max325.13×4.83×4.58×4.35×4.16×

At N=32N=32, PPB retains 4.98× speedup on DiffusionGemma and 6.34× on SDAR with no overlap between flash transfers and compute/DRAM work, approximately 86% and 70% of its ideal-overlap speedup. SB max retains 4.19× and 4.16×. The ordering of the methods is preserved throughout the sweep. The non-ideal formulation also follows the copy-then-execute order of our A100 runtime, where the paired roofline analysis closely predicts the measured seconds saved, as discussed in Q1.

Flash bandwidth. The method reduces flash reads, so its gain shrinks as flash gets faster and grows as it gets slower. We varied effective flash bandwidth from 1 to 8 GB/s while holding A18 compute throughput and DRAM bandwidth fixed and using N=C=32N=C=32.

Amortized speedup versus effective flash read bandwidth, with ideal overlap.

ModelMethod1 GB/s2 GB/s2.98 GB/s4 GB/s6 GB/s8 GB/s
DiffusionGemmaPPB6.7×6.1×5.7×5.3×4.7×4.2×
SB max5.2×4.9×4.6×4.4×4.0×3.6×
SDARPPB9.8×9.3×8.9×8.5×6.1×4.6×
SB max5.4×5.2×5.1×4.9×4.7×4.4×

At 8 GB/s, twice the fastest peak flash read bandwidth in Table 1, the methods retain 3.6–4.6× modeled speedup. This is consistent with the source of the benefit: faster flash reduces the cost of each avoided transfer, while a smaller expert working set still removes most of the transfers.

Faster flash and non-ideal overlap together. At 8 GB/s with no overlap between flash transfers and compute/DRAM work, PPB still achieves 3.27× on DiffusionGemma and 3.78× on SDAR; SB max achieves 2.96× and 2.90×. Thus, the advantage persists when both assumptions are changed together. Appendix H.1 and H.4 provide the bandwidth and overlap analyses, with the physical-cache, cold-start, and eviction-policy studies in H.2, H.3, and H.5.

We hope that these additional measurements and sensitivity analyses address the reviewer's concerns and clarify the conditions under which expert working-set compression translates into acceleration. We would be happy to clarify any remaining questions.

Thank you for your rebuttal, as well as for conducting the additional experiments. Your responses have satisfactorily addressed my concerns, and I have updated my review score accordingly.

Reviewer_1FYmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies Diffusion MoE LLM inference on memory-constrained devices. Although each token activates only a small subset of experts, parallel routing across multiple tokens can produce a large expert working set whose weights exceed the available cache capacity. Changes in this set across denoising iterations can trigger additional weight transfers. To address these issues, the paper proposes dynamic just-in-time expert pruning to bound the expert working set, together with previous-pool bias to reduce its turnover across iterations. Experiments show that a compact expert pool largely preserves benchmark accuracy while reducing weight demand, and trace-driven simulations estimate substantial speedups under the evaluated deployment assumptions.

Critical strengths
  1. The paper is clearly written and easy to follow. It is well structured and uses concise equations and illustrative diagrams to explain expert-pool selection and reuse across denoising steps, making the core ideas and the role of each design choice easy to understand.
  2. The accuracy evaluation and ablation studies are thorough. The results support the method’s effectiveness and illustrate the trade-off between reducing weight demand and preserving model quality.
Critical weaknesses
  1. The motivation needs more evidence. The paper argues that parallel token routing can enlarge the expert working set and increase weight transfers. However, the resulting bottleneck is not clearly shown. Although Table 2 reports the average number of unique experts activated, I still cannot tell how much memory their weights need or how often the expert set changes. The resulting transfer costs across different models and devices are also unclear.
  2. The acceleration results lack validation on real devices. The reported speedups are based on trace-driven simulations. These simulations assume that the weights of 32 experts fit in DRAM. Without measurements on the target devices, it remains unclear whether the simulation settings accurately reflect actual execution costs.
Questions for authors
  1. Could you provide a memory breakdown showing that caching 32 experts per layer fits within the stated DRAM budget on each evaluated device?
  2. Could you report end-to-end latency measurements on at least one target device? This would provide stronger evidence of the proposed method’s effectiveness in practice.
4 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for the helpful questions and for finding the paper well structured and the evaluation thorough. We added memory and traffic analyses (Appendices G–H) and real hardware measurements (Appendix F). We address W1–W2 here and Q1–Q2 in the next message.

W1: Memory demand, expert turnover, and transfer cost

Memory. An int4 expert occupies 3.0 MB in DiffusionGemma and 2.4 MB in SDAR. The full expert banks (11.4 and 14.5 GB) exceed the weight budget of all target devices. Keeping the native working sets resident (69.2 and 56.2 experts per layer-step) would require 7.7 and 7.1 GB including non-expert weights. Our C=32 cache instead uses 4.4–4.5 GB of resident weights, roughly half the DRAM of the 8 GB A18/A19 Pro, reserving the rest for runtime and system allocations. Native routing therefore requests 1.8–2.2× as many experts as this cache can hold. Q1 below gives the full memory breakdown.

Temporal overlap helps only when experts remain cached. In DiffusionGemma, consecutive native steps overlap strongly (Jaccard 0.83), introducing only 7.7 new experts per layer-step (Appendix G). Yet a working set of approximately 69 experts cannot stay resident in a 32-expert cache: even experts reused on the next step may already have been evicted and must be loaded again. A slowly changing set can therefore still generate substantial traffic. Bounding the routing pool to N≤C makes reuse possible; PPB (previous-pool bias) then favors experts from the previous pool to reduce turnover further.

The matched C=32 ablation in Appendix H.5 makes both effects visible: native routing, state-balanced max (SB), and PPB incur 42.1, 7.1, and 5.7 misses per layer-step, respectively. Fitting the pool in cache removes most misses; stabilizing it gives PPB a further 20% reduction relative to SB. Increasing PPB's pool to N=48>C raises misses to 19.5. Thus, temporal overlap becomes useful when the routing pool fits the available memory; overlap alone does not ensure low transfer cost.

Transfer costs across models and devices. Appendix G translates the routing traces into int4 flash traffic and modeled transfer time. Per MATH500 request, PPB reduces DiffusionGemma's traffic from 250.7 to 34.1 GB. At 85% of peak flash bandwidth, this corresponds to 84.3→11.5 s on A18/A19 Pro and 73.8→10.0 s on Dimensity 9400. SDAR's traffic falls from 1478.5 to 142.8 GB (10.4×). FLOPs and DRAM traffic per forward remain similar across policies; the native flash term accounts for 96–99% of modeled time. These estimates identify expert-weight transfer as the main bottleneck and show how reducing it benefits devices with different bandwidths. Q2 complements them with measured expert-bank memory transfers.

W2: Hardware validation and the C=32 assumption

We added a runtime on two A100 GPUs with 16-bit weights: the compute GPU holds a per-layer C=32 LRU expert cache, and misses trigger actual NVLink copies from the second GPU's expert bank. This makes the setup memory-capped. At N=32, SB/PPB achieve measured denoising GPU-busy-time speedups of 2.74×/2.94× on DiffusionGemma and 3.40×/4.12× on SDAR. The paired roofline predicts the absolute time saved within 5% across both models, both methods, and both overlap assumptions (Appendix F). These results support the predicted savings with real transfers, kernels, and cache management.

AuthorsOpenReview ↗

Response 2/2

Q1: Memory breakdown for C=32

Could you provide a memory breakdown showing that caching 32 experts per layer fits within the stated DRAM budget on each evaluated device?

Certainly! With int4 weights, caching 32 experts per layer across all layers requires 2.85 GB for DiffusionGemma and 3.62 GB for SDAR. The table below shows how expert-weight storage scales with cache capacity.

Experts per layerGemma (GB)SDAR (GB)
161.431.81
322.853.62
484.285.44
645.717.25
968.5610.87
12811.4214.50

Each expert has three projection matrices (gate, up, and down), totaling 3 × H × I parameters, where H is the hidden size and I is the expert intermediate size. Int4 uses 0.5 bytes per parameter, so storage across all layers is C × L × 3 × H × I × 0.5 bytes. DiffusionGemma uses H=2816, I=704, and L=30; SDAR uses H=2048, I=768, and L=48. Values are calculated from these dimensions and rounded to two decimal places, using decimal GB (10^9 bytes).

Adding non-expert weights (1.49 GB for DiffusionGemma and 0.77 GB for SDAR) gives resident-weight totals of 4.35 and 4.39 GB at C=32. Including one fp16 quantization scale per 128 weights adds 3.125% to weight storage, giving resident-weight totals of 4.48 and 4.53 GB.

For KV-cache and buffer accounting, we assume a 4096-token context, batch size 1, and bf16 activations. The following breakdown uses C=32 and includes the fp16 quantization scales in resident weights. All values are in GB; totals are computed before rounding.

ComponentGemma (GB)SDAR (GB)
Resident weights, including quantization scales4.484.53
KV cache at 4096 tokens0.290.40
Logits/probability buffers0.270.02
Accounted memory5.054.95
Remaining on A18 / A19 Pro (8 GB)2.953.05
Remaining on Dimensity 9400 (16 GB)10.9511.05

At our assumed context length, the KV cache and counted buffers are estimated to occupy only 0.56 GB for Gemma and 0.42 GB for SDAR. Thus, after allocating roughly half of the 8 GB device memory to resident weights, these components fit with approximately 3 GB remaining for other runtime and system allocations. Kernel workspaces, prefill temporaries, allocator overhead, and OS memory are not individually accounted for; these figures establish an analytical memory budget rather than a measured peak-memory guarantee. Expert-weight storage itself is independent of context length.

Q2: Measured latency and expert transfers

The new hardware study measures denoising on a memory-capped A100 runtime (Appendix F). Total denoising time falls from 238.3 to 86.9/81.1 s on DiffusionGemma and from 476.7 to 140.4/115.7 s on SDAR for SB/PPB. These are GPU-busy times including kernels, expert transfers, and the language-model head; host idle gaps, prefill, block-commit, and encoder passes are excluded. End-to-end latency on a target mobile device remains unmeasured.

To address the transfer-cost concern directly, we include measured expert-bank memory transfers below (Appendix F.4, Table 18). Traffic is measured cache misses times the 16-bit expert size (11.9 MB for Gemma, 9.4 MB for SDAR). SB denotes state-balanced max and PPB adds previous-pool bias; both use N=C=32.

Model / methodTotal GBGB/forwardTraffic reduction
Gemma native11,68613.7—
Gemma SB1,5941.97.3×
Gemma PPB1,1671.410.0×
SDAR native27,28011.7—
SDAR SB3,8691.87.1×
SDAR PPB1,9850.913.7×

Totals include each method's denoising forwards. At the measured bank bandwidth, avoided-transfer time agrees with measured denoising savings within 5%, connecting the acceleration to fewer expert loads. The GPU bank stands in for flash; these measurements validate savings in this runtime. Mobile end-to-end validation additionally requires a target runtime with efficient int4 MoE kernels and suitable quantization, as discussed in the shared response.

We hope that our answers address the reviewer's concerns and questions. We would be happy to clarify any remaining questions.

Dear Reviewer 1FYm,

Thank you for your review. We would like to briefly follow up before the discussion period ends. You asked for a memory breakdown showing that C=32 fits the device DRAM budgets, and for latency measurements. We added both. With int4 weights, C=32 needs 4.5 GB of resident weights (about 5 GB including KV cache and buffers), which leaves about 3 GB on an 8 GB device. On a memory-capped A100 runtime, measured expert-bank traffic falls by 7-14× and denoising time by 2.7-4.1×. Details are in our responses and Appendices F-H.

With the discussion period ending soon, could you let us know whether this addresses your concerns, or whether anything else would help? Thank you.

Thank you for the additional analyses and measurements. The new evidence is clearly presented and addresses some of my concerns regarding memory breakdown. However, the hardware evaluation is limited to A100 GPUs, and end-to-end latency has not been measured on any real mobile device. Therefore, the practical speedup in on-device deployment remains unverified. I maintain my original score.

Reviewer_aC2rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free method for deploying diffusion MoE language models under limited memory. It restricts each layer-step to a shared expert pool selected from current router scores, and uses previous-pool bias to reduce expert turnover across denoising iterations. Experiments on DiffusionGemma and SDAR examine accuracy, expert activation, and cache misses. Trace-driven simulations report substantial modeled speedups under mobile-memory assumptions.

Critical strengths
  1. The paper identifies a practical bottleneck in diffusion MoE inference. Even with sparse token-level routing, parallel denoising can activate a large union of experts, which is costly when expert weights must be loaded from flash.
  2. The method is simple and training-free. A shared expert pool explicitly bounds active experts, while previous-pool bias reduces turnover without replacing the original router or retraining the backbone.
  3. The evaluation covers two diffusion mechanisms and several reasoning and coding tasks. At N = 32, Table 2 reports 31.7 to 58.2% fewer activated experts across the two models, with generally small accuracy changes, and the pruning-objective ablations are useful.
  4. The paper separately studies working-set size, state-balanced selection, and temporal reuse. Table 3 also accounts for changed denoising steps when translating routing traces into modeled speedups, rather than relying only on expert-count reductions.
Critical weaknesses
  1. The headline speedups are estimates from a trace-driven simulator, not measured end-to-end inference latency. The reported 5.8× and 9.1× gains depend on assumed 4-bit weights, DRAM and flash bandwidth, and cache capacity. It remains unclear whether these gains hold on the stated mobile devices once runtime overhead is included.
  2. The comparison with TEAM is incomplete for a deployment-oriented claim. The paper evaluates only DCD+LAC and disables TEAM's speculative exploration branch. Table 3 therefore does not establish how the proposed method compares with the full system in end-to-end speed and quality.
  3. The simulated speedup is highly sensitive to the assumed expert cache size. Table 3 shows that increasing N from 32 to 64 reduces the amortized gain from 5.8× to 1.3× on DiffusionGemma and from 9.1× to 1.2× on SDAR. The paper does not sufficiently establish how robust the claimed advantage is to other memory budgets and cache policies.
  4. Some of the claimed quality improvements are difficult to assess statistically. For example, SDAR GPQA rises from 31.31 to 37.88 in Table 2, but SDAR is evaluated using greedy decoding and no uncertainty interval is reported for this difference. The evidence is stronger for preserving accuracy than for improving it.
Questions for authors
  1. Could the authors measure end-to-end latency, peak memory, and flash traffic on at least one actual memory-constrained device? A comparison with the simulator's prediction for the same configuration would be especially helpful.
  2. Could the authors compare with the full TEAM inference pipeline, including SEH, at matched hardware and report both generation speed and accuracy?
  3. How do the projected speedups change with different expert-cache budgets, flash bandwidths, and cold-start costs? In particular, how much of the N = 32 advantage remains when the cache does not fit exactly 32 experts?
  4. Could the authors report uncertainty or repeated-run results for the SDAR benchmark differences, especially GPQA, and distinguish accuracy preservation from apparent improvements?
4 comments
AuthorsOpenReview ↗

Response 1/3

We thank the reviewer for their thorough and constructive assessment and helpful questions and suggestions. We have incorporated the additional experiments into the revised manuscript. This reply covers hardware validation, full TEAM, and accuracy uncertainty; the next two address cache robustness and bandwidth/cold-start sensitivity.

W1 & Q1: Hardware measurements

We added a memory-capped A100 runtime with a per-layer C=32 expert cache and actual transfers from a second GPU. Peak allocated memory falls from 54 to 21 GB for DiffusionGemma and from 61 to 19 GB for SDAR. State-balanced max (SB) and previous-pool bias (PPB) achieve measured denoising speedups of 2.74×/2.94× and 3.40×/4.12×, respectively; expert-bank transfers fall by 7.3–10.0× and 7.1–13.7×. The paired roofline predicts the absolute time saved within 5%. The shared response and Appendix F give the timings, traffic, and scope: these are denoising GPU-busy times, with target-mobile end-to-end validation remaining open.

W2 & Q2: Full TEAM, including SEH

We have extended Table 2 with full TEAM (DCD+LAC+SEH), reporting accuracy and distinct experts per layer-step, including rejected speculative branches. Including SEH preserves the conclusion: our method achieves competitive accuracy with a smaller expert footprint.

Fewer denoising steps needed does not mean less expert computation. In the matched SDAR static ablation, SEH reduces denoising calls by 51.4% but increases distinct experts per forward by 30–41% and total token-expert assignments by 90.8%. SEH trades sequential depth for additional expert workload, so its benefit depends on execution and memory costs.

The shared response also reports measured TEAM timings under the same memory-capped A100 setup. On SDAR, DCD+LAC and full TEAM achieve 2.03× and 2.11× speedup, versus PPB's 4.12×; on DiffusionGemma, the corresponding values are 1.17×, 0.89×, and 2.94×. Table 2 separately reports benchmark accuracy,and the revised comparison includes SEH in both the hardware study and the quality evaluation.

W4 & Q4: SDAR uncertainty and accuracy preservation

In our study, we have used the recommended SDAR setup from the original SDAR paper [1], where the authors explicitly recommend using greedy decoding setup for SDAR-Chat models. Nevertheless, we have re-run evaluations 3 times for all datasets as requested and witnessed 0.00 std for the considered benchmarks, as greedy decoding is deterministic. However, we recognise the reviewer's concern, and for GPQA, the only multiple-choice benchmark in our evaluation, we follow the existing practice of permuting the order of the answer options across runs [2, 3]. Since a different order of the options changes the prompt, it also changes the greedy generation, and the accuracy varies across runs. We have updated Table 2 with the mean and standard deviation over 3 runs with permuted answer options for all SDAR methods (including newly requested TEAM with SEH), and the pattern still holds. In contrast, our evaluations on DiffusionGemma model, which has a recommended sampling generation setup, have been averaged across 3-10 seeds depending on the benchmark with standard deviations reported. Nevertheless, in the paper, we have updated the claims to preserving the quality.

[1] SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation, arxiv:2510.06303. [2] Large Language Models Are Not Robust Multiple Choice Selectors. arxiv:2309.03882, ICLR 2024. [3] Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. arxiv:2308.11483, Findings of NAACL 2024.

AuthorsOpenReview ↗

Response 2/3

W3: Routing budget, physical cache capacity, and eviction policy

Per-step and amortized gain for N=32 vs N=64. We thank the reviewer for this question. We would like to respectfully note that the key quantity for interpreting Table 3 is how NN compares with the number of unique experts that native routing uses. As Table 2 reports, native routing on MATH500 activates on average 69.17 unique experts for DiffusionGemma and 56.53 for SDAR per layer-step. N=32N=32 is far below this native working set, and expert-weight misses fall from 42.1 to 5.7 (DiffusionGemma) and from 26.4 to 2.4 (SDAR) per layer-step. N=64N=64, in contrast, is almost the native working set: the layers still use 58.7 and 51.8 experts per layer-step, so the constraint rarely binds, expert demand barely changes (31.1 and 21.9 misses per layer-step), and only a small speedup is expected. The difference between the two rows therefore reflects how strongly each budget compresses the native working set, rather than a high sensitivity of the method: the speedup tracks the reduction in expert demand. We have added N=48N=48 to Table 3 as an intermediate budget, and as expected, the gain lies between N=32N=32 and N=64N=64 and follows the reduction in expert demand.

The smaller gain at N=64N=64 thus reflects the models’ native average expert usage and the reduced opportunity for pruning, rather than evidence of instability in the method. We have made this relationship clearer in the revised paper.

Varying physical memory independently. Appendix H.2, Table 23 contains the full sweep over C∈{16,24,32,40,48,64,96} and routing budgets N=32,48,64. The compact excerpt below fixes N=32 and shows that gains persist on either side of C=32, while larger caches also help native routing. Gemma denotes DiffusionGemma; SB is state-balanced max and PPB adds previous-pool bias.

Amortized speedup, averaged over A18 / A19 Pro / Dimensity 9400.

Model / methodC=24C=32C=40C=64
Gemma PPB3.5×5.8×5.0×2.9×
Gemma SB3.3×4.7×4.7×2.8×
SDAR PPB3.5×9.1×7.8×2.5×
SDAR SB2.8×5.2×5.7×2.1×

PPB's gain peaks when the cache matches its routing budget: N=C=48 gives 3.2×/4.0× and N=C=64 gives 1.9×/1.7× on Gemma/SDAR. Choosing N therefore requires considering both available memory and the quality–efficiency trade-off. At large C, native routing fits more readily and the relative gain shrinks. Each additional cache slot across all layers costs about 0.09/0.11 GB at int4; C=64 requires approximately 7.4/8.3 GB including non-expert weights and quantization scales. On an 8 GB device this leaves little or no room for runtime and system allocations. The largest caches characterize sensitivity beyond the practical budget of those devices.

Eviction policy. Appendix H.5, Table 26 contains the full policy comparison, including N=48. Below we retain native routing and both methods at N=32.

Each cell: misses per layer-step / amortized speedup. C=32; int4; speedups averaged over the same three devices, against native routing under the corresponding eviction policy. Belady is an offline oracle-like optimal reference.

Model / methodLRULFUBelady
Gemma native42.1 / 1.0×42.0 / 1.0×41.5 / 1.0×
Gemma PPB5.7 / 5.8×7.3 / 4.5×5.3 / 6.0×
Gemma SB7.1 / 4.7×8.1 / 4.2×6.0 / 5.2×
SDAR native26.4 / 1.0×26.3 / 1.0×25.4 / 1.0×
SDAR PPB2.4 / 9.1×6.0 / 4.1×2.3 / 9.6×
SDAR SB4.2 / 5.2×6.7 / 3.4×3.2 / 6.4×

Even Belady reduces native misses by only 1–4%, because the working set exceeds capacity. For PPB, LRU approaches Belady's speedup; LFU performs worse, consistent with a working set that changes across denoising steps. At N=48>C, policy differences narrow as capacity pressure increases. The benefit therefore persists across policies, although eviction strategy affects its magnitude.

The quality side is covered by the N∈{16,32,48,64,96,128} accuracy sweeps in Tables 4 and 10, the sum/max/state-balanced selection comparisons in Tables 5–6, and Appendix E's temporal-reuse ablations of accuracy, misses, and denoising overhead. Together with Appendix H, these show how quality and efficiency depend on routing budget, selection strategy, physical memory, and cache behavior.

AuthorsOpenReview ↗

Response 3/3

Q3: Cache budgets, bandwidth, and cold starts

We thank the reviewer for this question. Appendix H.1–H.3 adds the requested analysis. The preceding reply addresses physical cache capacity independently of N: PPB at N=32 retains 3.5× on both models at C=24 and 5.0×/7.8× on DiffusionGemma/SDAR at C=40. Below we examine bandwidth and cold-start costs.

Flash bandwidth. We varied effective flash bandwidth from 1 to 8 GB/s on the Table 3 traces, fixing A18 compute and DRAM rates and using N=C=32 with ideal overlap. The excerpt below retains the endpoints and nominal A18 bandwidth; Appendix H.1, Table 22 gives all six settings. Gemma denotes DiffusionGemma; SB is state-balanced max and PPB adds previous-pool bias.

Amortized speedup versus effective flash read bandwidth.

Model / method1 GB/s2.98 GB/s8 GB/s
Gemma PPB6.7×5.7×4.2×
Gemma SB5.2×4.6×3.6×
SDAR PPB9.8×8.9×4.6×
SDAR SB5.4×5.1×4.4×

Faster flash makes each avoided transfer cheaper, so the relative gain decreases. At 8 GB/s, twice the fastest peak flash bandwidth in Table 1, the methods still retain 3.6–4.6× speedup. The gain falls below 2× only above approximately 18–23 GB/s effective flash bandwidth, depending on model and method. The benefit also persists when bandwidth and cache size vary together: at C=24, Gemma PPB changes from 3.7× at 1 GB/s to 3.0× at 8 GB/s. At C=40, PPB stays within 3.6–5.8× on Gemma and 3.2–8.6× on SDAR across this bandwidth range.

Cold-start costs. The main simulation includes the initial expert loads. The cold-canvas reference resets the cache at every canvas (Gemma, 256 tokens) or block (SDAR, 32 tokens), charging the fill like any other step. These fills account for 56%/38% of PPB's remaining misses on Gemma/SDAR, compared with 14%/5% for native routing. Appendix H.3, Table 24 compares three alternatives, applying the same accounting to native routing and each method:

  • Cold canvas + preload: reset at each canvas/block, with initial expert weights already in DRAM and charged at DRAM cost.
  • Warm request: retain residency across canvases/blocks, resetting and charging the fill once per request.
  • Warm request + preload: retain residency within each request, with initial request-level weights already in DRAM and charged at DRAM cost.

Each cell: misses per layer-step / amortized speedup. N=C=32; speedups averaged over A18 / A19 Pro / Dimensity 9400. Native routing has unit speedup under each assumption.

Model / methodCold canvasCold + preloadWarm requestWarm + preload
Gemma native42.1 / 1.0×36.3 / 1.0×40.0 / 1.0×38.9 / 1.0×
Gemma PPB5.7 / 5.8×2.5 / 8.4×4.1 / 6.7×3.5 / 7.3×
Gemma SB7.1 / 4.7×4.0 / 6.3×5.6 / 5.4×5.0 / 5.7×
SDAR native26.4 / 1.0×25.1 / 1.0×25.9 / 1.0×25.8 / 1.0×
SDAR PPB2.4 / 9.1×1.5 / 12.7×2.0 / 10.9×1.9 / 11.1×
SDAR SB4.2 / 5.2×3.3 / 6.2×3.8 / 5.7×3.7 / 5.7×

All three alternatives improve speedup in this ablation, making the per-canvas reset conservative among the tested assumptions. Retaining the cache within a request raises PPB from 5.8× to 6.7× on Gemma and from 9.1× to 10.9× on SDAR. This also holds away from C=32: warm-request PPB reaches 3.7×/7.0× at C=24/40 on Gemma (versus 3.5×/5.0× with per-canvas reset), and 3.6×/9.8× on SDAR (versus 3.5×/7.8×).

In absolute terms, filling C=32 once transfers 2.85/3.62 GB and takes a modeled 0.92/1.17 s on Gemma/SDAR at int4. This is about 8%/2% of PPB's modeled request time (11.8/54.3 s). A persistent cache pays the fill once per request, or once per session if residency is retained between requests.

We hope these measurements and sensitivity analyses address the reviewer's concerns. We would be happy to clarify any remaining questions.

Dear Reviewer aC2r,

Thank you again for your detailed review. We would like to briefly follow up before the discussion period ends. Your main concerns were that the speedups come from a simulator only, that full TEAM (with SEH) was missing, and that the results might be sensitive to cache size. In our rebuttal we addressed each one:

  • Hardware: a memory-capped two-A100 runtime with real expert transfers shows measured denoising speedups of 2.7–4.1×. The roofline predicts the absolute time saved to within 5%.
  • Full TEAM: it is now in Table 2 and in the hardware study. It reaches 2.0–2.1× on SDAR, against 4.1× for our method.
  • Cache sensitivity: we swept cache capacity, eviction policy, bandwidth and cold-start accounting (Appendix H). The benefit persists throughout.
  • Uncertainty: SDAR GPQA now reports mean and std over 3 runs with permuted answer options.

Since the discussion period closes soon, we would be grateful to know whether these responses resolve your concerns, or whether anything is still unclear that we can address in time. Thank you.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global response

We thank reviewers aC2r, 1FYm, and ypLX for requesting hardware validation and sensitivity analyses. We added a memory-capped runtime and paired roofline predictions on the same runs (Appendix F). Measured denoising accelerates by 2.7–4.1×, and the roofline predicts the absolute time saved by our methods to within 5%.

Hardware and measurement scope. We use two A100-SXM4-80GB GPUs with 16-bit weights. The compute GPU holds non-expert weights and a per-layer LRU cache of C=32 experts. This makes the setup memory-capped: resident weights are limited to 18.9 GB of DiffusionGemma's 51.6 GB and 18.8 GB of SDAR's 61.1 GB. The second GPU holds the expert bank, standing in for flash; misses trigger NVLink transfers before execution. Peak allocated memory falls from approximately 54 to 21 GB for DiffusionGemma and from 61 to 19 GB for SDAR. Expert-bank bandwidth is measured directly; modeled compute and HBM rates are 40% of peak, consistent with profiled kernel efficiencies (Appendix F.1).

Times cover denoising GPU-busy time, including kernels, transfers, and the language-model head. Below, Gemma denotes DiffusionGemma; SB is state-balanced max and PPB adds previous-pool bias, both at N=32.

Measured speedups and seconds saved. The ideal roofline overlaps expert transfers with compute/HBM work; the non-ideal variant adds transfers sequentially, following the runtime's execution order.

Model / methodMeasured speedupSeconds saved: measured / non-ideal / ideal
Gemma SB2.74×151.4 / 151.2 / 149.4
Gemma PPB2.94×157.3 / 156.9 / 155.6
SDAR SB3.40×336.3 / 351.4 / 345.7
SDAR PPB4.12×361.0 / 376.9 / 366.9

All eight savings predictions agree within 5%. The paired analysis uses measured misses and forwards to test how accurately the cost model translates expert movement into time. Appendix F (Tables 16–17) gives absolute timings and predicted speedups.

Absolute savings and runtime costs. Measured savings include framework and routing-policy work. Shared runtime work largely cancels in the Gemma savings comparison; cancellation is less complete for SDAR. The absolute savings quantify the work removed by pruning, while the speedup ratio also reflects the remaining runtime costs (Appendix F.3).

Measured expert-bank memory transfers support this mechanism. Total transfers fall by 7.3–10.0× on Gemma and 7.1–13.7× on SDAR. Per-forward traffic falls from 13.7 to 1.4–1.9 GB and from 11.7 to 0.9–1.8 GB, respectively. Avoided-transfer time agrees with measured savings within 4% (Appendix F.4).

Full TEAM comparison. SDAR, same A100 setup and five matched prompts; savings are measured / non-ideal / ideal roofline, in seconds.

MethodMeasured time (s)SpeedupSeconds saved
TEAM DCD+LAC235.32.03×241.4 / 269.3 / 259.5
TEAM+SEH226.22.11×250.5 / 286.6 / 262.8

Gemma TEAM / TEAM+SEH take 203.8 / 268.5 s (1.17× / 0.89×), saving 34.5 / −30.1 s overall versus 43.8 / 19.2 s predicted without overlap. Transfer savings are 41.8 / 51.3 s versus 44.6 / 55.7 s predicted; TEAM+SEH's speculative work outweighs the transfer savings (Appendix F.5).

We have also added Appendix H covering ablations in cache capacity, eviction, cold starts, bandwidth, and non-ideal overlap.

We hope that the provided measurements on the real memory-capped hardware together with the matching simulations address the reviewers' concerns about the roofline setup. At the same time, analytical costs and roofline models are a common instrument for studying memory-bound LLM and MoE inference, for example [1–4].

The provided results show that the roofline predicts the time our method saves to within 5%. A faithful end-to-end measurement on a mobile device is, however, not straightforward and requires a considerable engineering effort, as it depends on several components that are largely independent of the routing method studied in the paper. It requires access to a suitable prototyping device and a mature on-device inference framework, since framework overhead directly shapes the measured ratio. It also requires efficient int4 MoE kernels and a high-quality int4 quantization scheme of the model: for diffusion LLMs, quantization errors may lower token confidence and lead to additional denoising steps, which would change the measured latency for reasons unrelated to expert routing.

[1] Efficiently Scaling Transformer Inference (arXiv:2211.05102), MLSys'23

[2] FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU (arXiv:2303.06865), ICML'23

[3] LLM in a flash: Efficient Large Language Model Inference with Limited Memory (arXiv:2312.11514), ACL'24

[4] MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs (arXiv:2411.11217)

Dear Reviewers,

The authors have submitted their rebuttal. Please take a look at their responses and actively participate in the discussion, especially regarding any remaining concerns or disagreements. Please consider updating your evaluations if appropriate.

Thank you for your time and effort!

Best regards, Area Chair

Dear Reviewers and Area Chair,

We thank all reviewers for their feedback. The main shared concern was the lack of hardware validation and sensitivity analysis for the simulated speedups. In response we added:

  • a memory-capped two-A100 runtime with real expert transfers (2.7-4.1× measured denoising speedup, with roofline-predicted time saved within 5%),
  • full TEAM (with SEH) results,
  • sensitivity sweeps over cache capacity, eviction policy, bandwidth, overlap, and cold starts,
  • a memory breakdown for the mobile budget, and uncertainty estimates for SDAR on GPQA.

We thank Reviewer ypLX for their engagement and updated score. We have posted individual follow-ups to Reviewers aC2r and 1FYm, and remain happy to answer any further questions before the deadline.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.