acceptodds
Under review as a conference paper at ICLR 2027

TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offload

Abstract

Diffusion Large Language Models (dLLMs) have emerged as a competitive alternative to autoregressive (AR) models, offering better hardware utilization and bidirectional context through parallel block-level decoding. However, as dLLMs continue to scale up with mixture-of-experts (MoE) architectures, their deployment on resource-constrained devices remains an open challenge. Existing AR-based methods often incur either prohibitive I/O overhead or significant compute bottlenecks. In this work, we propose TIDE, a novel resource-efficient inference system that leverages the temporal stability of expert activations during the diffusion process within the block. Specifically, we leverage the temporal stability of expert activations during the diffusion process within the block and introduce an interval-based expert refresh strategy that updates the expert placement in an I/O-aware fashion. To ensure optimal performance, we formulate the inference scheduling as a mathematical programming problem, solving for the optimal interval that minimizes I/O traffic and CPU computation. Most importantly, TIDE is a lossless optimization that requires no model training, providing a "free lunch" acceleration for dLLM inference. In a single GPU-CPU system, we demonstrate that TIDE achieves up to 1.4× and 1.5× throughput improvements over prior baselines on LLaDA2.0-mini and LLaDA2.0-flash models, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion LLM, Mixture-of-experts inference, Expert offloading

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sDTFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TIDE, a system for expert offloading in diffusion MoE inference. It exploits similar expert demand across adjacent denoising steps to periodically update GPU-resident experts based on usage counts. Expert placement remains fixed between updates, while nonresident experts continue to run on the CPU. A cost model selects the refresh interval to balance expert migration and CPU computation. Experiments on two LLaDA2.0 models show that TIDE achieves higher throughput than both baselines across all tested configurations.

Critical strengths
  1. TIDE uses expert demand patterns in diffusion MoE to periodically update GPU-resident experts, balancing weight migration and CPU computation costs while keeping the original routing unchanged and requiring no retraining.

  2. The results are fairly consistent across two models, different generation lengths, and GPU expert budgets.

Critical weaknesses

Equation (7) omits the I/O cost coefficient C_IO from Equation (5) without explaining whether the objective was normalized. The paper also does not clearly define f(τ) or explain how it is fitted. Table 2 reports neither the selected refresh intervals nor a comparison with a tuned fixed-interval baseline, making it hard to judge the additional benefit of the cost model over periodic refresh. The paper reports only throughput on MBPP, without accuracy or output agreement results to support its losslessness claim.

Questions for authors
  1. Is Equation (7) normalized by C_IO? How is f(τ) determined? Please report the selected refresh intervals and compare performance with a tuned fixed-interval baseline to assess the cost model's additional benefit.
  2. Can the authors provide output agreement results under the same decoding settings, along with MBPP accuracy, to support the losslessness claim?
3 comments

Dear Reviewer sDTF,

Thank you for your valuable and constructive comments. We have corrected the objective and added the CPU-cost formulation and fitting procedure to the appendix.

W1 / Q1. I/O coefficient and CPU cost function.

We restored CI/OC^{\mathrm{I/O}} in Eq. (7), so the revised objective includes both timing coefficients explicitly.

We clarify the functional form of f(τ)f(\tau) and explain how its coefficients are obtained here. For refresh interval τ\tau and expert-placement drift parameter dd, the CPU workload relative to full refresh is modeled by

f(τ)=1+β(τ−1)+γ[(1−d)−(1−d)τ],β,γ≥0.f(\tau)=1+\beta(\tau-1)+\gamma\bigl[(1-d)-(1-d)^\tau\bigr],\qquad \beta,\gamma\geq0.

Each term has a specific role:

  • Baseline 11. Even full refresh leaves CPU work when the GPU cannot hold every demanded expert. This term retains that baseline workload, whose time scale is absorbed into CCPUC^{\mathrm{CPU}}.
  • Linear term β(τ−1)\beta(\tau-1). This captures additional CPU work that continues to grow as placement ages. For example, if additional CPU assignments grow approximately linearly with placement age, averaging ages 0,…,τ−10,\ldots,\tau-1 gives a contribution proportional to (τ−1)/2(\tau-1)/2; the factor 1/21/2 is absorbed into β\beta. This is an empirical approximation over the calibrated interval range.
  • Mismatch term γ[(1−d)−(1−d)τ]\gamma[(1-d)-(1-d)^\tau]. Under the constant-drift approximation, accumulated mismatch is 1−(1−d)τ1-(1-d)^\tau. Subtracting the one-step baseline dd gives (1−d)−(1−d)τ(1-d)-(1-d)^\tau. This term captures a saturating contribution to additional CPU assignments as GPU placement becomes less aligned with current expert demand; γ\gamma determines its magnitude.

Both additional terms vanish at τ=1\tau=1, giving f(1)=1f(1)=1. Nonnegative β\beta and γ\gamma make this workload response nondecreasing. Their combination allows both saturating mismatch and sustained workload growth.

Substituting f(τ)f(\tau) into the CPU-computation term gives the full cost model for one sparse layer over TT denoising steps:

Lat^(τ)=CI/OBTτ[1−(1−d)τ]+CCPUBT[1+β(τ−1)+γ((1−d)−(1−d)τ)].\widehat{\mathrm{Lat}}(\tau)=C^{\mathrm{I/O}}\frac{BT}{\tau}[1-(1-d)^\tau]+C^{\mathrm{CPU}}BT[1+\beta(\tau-1)+\gamma((1-d)-(1-d)^\tau)].

τ∗∈arg⁡min⁡τ∈{1,2,…,T−1}  Lat^(τ),\tau^*\in\underset{\tau\in\{1,2,\ldots,T-1\}}{\arg\min}\;\widehat{\mathrm{Lat}}(\tau),

where BB is the GPU expert budget per sparse layer, CI/OC^{\mathrm{I/O}} and CCPUC^{\mathrm{CPU}} are configuration-dependent timing coefficients.

We fit the model offline using profiling runs at several refresh intervals under a fixed model, GPU expert budget, and decoding configuration. We collect CPU token–expert assignment counts, expert replacement counts, and separate CPU-computation and migration times, normalized by the denoising steps actually executed. First, we fit an effective dd to migration counts with the full-refresh count anchored. With dd fixed, we jointly fit nonnegative β\beta and γ\gamma by least squares to CPU assignment counts divided by their full-refresh value. We then freeze the count curves and separately regress CPU time on predicted assignments and migration time on predicted replacements, both through the origin, converting the fitted scales to CCPUC^{\mathrm{CPU}} and CI/OC^{\mathrm{I/O}} using the objective's normalization. Finally, we fix all coefficients and select τ\tau by minimizing the estimated latency.

W1 / Q1. Selected intervals and the fixed-interval baseline.

We compare the throughput performance of (1) the interval selected by minimizing the fitted cost model with (2) the oracle interval found by exhaustively measuring all candidates. The optimal interval derived from the cost model achieve 89-96% of the best measured throughput.

ModelGPU Expert BudgetTIDE Throughput (tok/s)Oracle hroughput (tok/s)Throughput Ratio
LLaDA2.0-mini642.112.2494%
LLaDA2.0-mini1282.362.6489%
LLaDA2.0-flash321.251.3990%
LLaDA2.0-flash641.731.8196%

Q2. Accuracy and output agreement.

We would like to highlight that TIDE is strictly lossless by design. TIDE does not modify the MoE router selection, model weights, or the parallel decoding mechanism. Specifically, as shown in Algorithm 1, at every denoising step tt, every token xx is still routed to and processed by its exact set of selected top-kk experts Ex\mathcal{E}_x. Because the mathematical operations performed for every tokenare identical regardless of whether an expert resides on the CPU or GPU (and regardless of the refresh interval τ\tau), TIDE is mathematically equivalent to standard full-GPU inference and the baselines.

We have added accuracy experiments on MBPP and GSM8k. TIDE achieves almost identical model performance with GPU-only baselines.

ModelDatasetNormal executionTIDE
LLaDA2.0-miniMBPP81.6781.26
LLaDA2.0-flashMBPP88.1188.55
LLaDA2.0-miniGSM8K94.0694.07
LLaDA2.0-flashGSM8K96.2096.72

Thank you for your rebuttal, as well as for conducting the additional experiments. Your responses have satisfactorily addressed my concerns, and I have updated my review score accordingly.

Thank you for taking the time to read our response and for your constructive feedback! We are glad that our clarifications and additional experiments addressed your concerns.

Reviewer_BAN2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the efficient deployment of MoE dLLMs on memory-constrained hardware. It proposes TIDE, a training-free and lossless approach that optimizes expert weight placement and migration between the CPU and GPU. he key observation is that expert activation patterns exhibit strong temporal locality across adjacent denoising steps. TIDE moves high-demand experts to the GPU only at designated refresh steps and keeps their placement unchanged between refreshes, while executing experts not resident on the GPU on the CPU. The authors further develop a cost model for weight transfer and CPU computation and use offline profiling to select the refresh interval. Experimental results demonstrate the effectiveness of TIDE in improving MoE dLLM inference efficiency.

Critical strengths
  1. The paper is well organized and well motivated. The analysis shows that expert activation patterns are highly similar across adjacent denoising steps and exhibit temporal locality over multiple steps. These observations motivate the periodic refresh of expert placement.
  2. The expert placement model is clear and helps explain the method. The paper first analyzes how the number of active experts, GPU expert capacity, and expert computation and placement across the CPU and GPU affect inference latency. It then models the choice of refresh frequency as a trade-off between weight transfer and CPU computation costs, making the rationale and optimization objective easy to follow.
Critical weaknesses
  1. Section 4.1 describes Fiddler as a baseline with static expert placement. However, Fiddler also supports dynamically choosing between CPU execution and transferring weights to the GPU for execution based on the number of tokens assigned to each expert. This raises concerns about whether the baseline configuration fully reflects Fiddler’s capabilities and whether the comparison is fair.
  2. The experiments use only the sanitized MBPP dataset, and both evaluated models belong to the LLaDA2.0 family. Since TIDE’s performance gains depend on temporal locality in expert activations across denoising steps, differences in routing distributions and how quickly they change across tasks may affect the effectiveness of the refresh strategy.
Questions for authors
  1. Could you evaluate TIDE on additional datasets or models to assess whether its performance gains generalize?
  2. How does TIDE perform under tighter GPU memory constraints? Could you evaluate it on GPUs with less memory or with smaller GPU expert budgets?
  3. What is the time cost of offline profiling?
  4. How does the number of CPU cores used for expert computation affect TIDE’s throughput?
1 comment

Dear Reviewer BAN2,

Thank you for your valuable and constructive comments. We clarify the baseline configuration and provide the additional evaluation below.

W1. Fiddler dynamic execution rule.

We thank the reviewer for pointing out Fiddler's dynamic execution rule. We clarify that in our single-request block-decoding regime (batch_size = 1, block size = 32), Fiddler's behavior is bounded by two extremes, both of which are directly represented by our evaluated baselines (Fiddler and Mixtral-Offload):

  1. Why Fiddler naturally operates in static placement mode in our setup: Fiddler dynamically transfers an offloaded expert to the GPU only when the number of tokens ss routed to that expert in a single step exceeds a threshold where \text{cpu_lat}(s) > \text{gpu_lat}(s) + \text{transfer_lat}. While dLLM decoding processes a block of tokens per step at batch_size = 1, LLaDA2.0-mini and LLaDA2.0-flash employ fine-grained MoE layers with 256 experts (top-k=8\text{top-}k = 8). Consequently, an expert receives only a few tokens per step, where CPU execution latency is usually smaller than transferring the full expert weights over PCIe. Thus, Fiddler's dynamic rule naturally selects CPU execution without weight transfer.
  2. Even if a block of tokens were to trigger Fiddler's weight migration threshold, because adjacent diffusion steps exhibit high routing similarity, triggering dynamic migration on a block would cause Fiddler to repeatedly transfer offloaded experts at every denoising step, which is equivalent to our Mixtral-Offload baseline.

By evaluating both Fiddler (static expert placement, τ=∞\tau = \infty) and Mixtral-Offload (every-step GPU weight migration on misses, τ=1\tau = 1), our experiments capture both extremes of existing AR offloading strategies and highlight why TIDE's intermediate refresh interval (1<τ<∞1 < \tau < \infty) is essential for dLLMs.

Q1. Additional datasets and models.

To validate the generalizatio of our approach, we have added additional evaluation on GSM8K dataset, with results shown as below:

ModelGeneration lengthGPU expert budgetGPU memory constraintFiddlerMixtral-OffloadTIDE
LLaDA2.0-mini2566410 GB1.881.722.25
LLaDA2.0-mini25612818 GB2.002.082.39
LLaDA2.0-mini10246410 GB1.791.732.11
LLaDA2.0-mini102412818 GB1.921.982.32
LLaDA2.0-flash2563230 GB1.111.091.36
LLaDA2.0-flash2566455 GB1.131.291.63
LLaDA2.0-flash10243230 GB0.981.071.48
LLaDA2.0-flash10246455 GB1.191.211.67

TIDE achieves consistent speedup against prior SOTA methods, providing 1.15-1.38x speedups.

Q2. Tighter memory constraints.

The middle one of Figure 5 showcases the impact of GPU memory constraint on throughput performance. TIDE maintains the highest throughput across different configurations. We also report the detailed results as below.

GPU expert budgetGPU memory constraintFiddlerMixtral-OffloadTIDE
164 GB1.261.121.46
326 GB1.651.371.74
6410 GB1.791.451.89
9614 GB1.821.772.15
12818 GB1.801.912.44

Q3. Offline profiling cost.

We report the offline profiling costs as below The substantial offline work is to collect CPU token-expert assignment counts, expert replacement counts, and their component times at the calibration intervals. The same traces are then reused to fit the cost-model parameters and select the refresh interval. We report the offline profiling costs for LLaDA2.0-mini on A40 GPU over candidate intervals 1–31, calibration uses τ∈{1,2,4,8,16,31}\tau \in \{1,2,4,8,16,31\}. Each run evaluates 50 MBPP prompts, with generation budget 512, block length 32, and confidence threshold 0.95.

GPU expert budgetProfiling Time (hours)
6419.36
9615.49
1288.89
1606.34
1925.13
2243.64

Q4. CPU core count.

Reducing the CPU cores available for expert computation lowers TIDE's throughput, because tokens assigned to CPU-resident experts take longer to process. This makes CPU execution a larger bottleneck, particularly when the GPU expert budget is small. We have provided results below for reference.

CPU CoresThroughput (token/s)
120.97
241.49
482.39
Reviewer_mejnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TIDE, a method for speeding up MoE diffusion language models when GPU memory is limited. It uses the similarity of expert usage across nearby denoising steps to reduce expert transfers between CPU and GPU. A cost model selects how often to update the experts stored on the GPU. The method keeps model weights and routing unchanged. Experiments on LLaDA2.0-mini and LLaDA2.0-flash show higher throughput than the tested baselines.

Critical strengths

The method is simple and practical. TIDE reduces unnecessary expert transfers while keeping all required expert computations. It needs no extra training and addresses a clear problem in running large models with limited GPU memory.

The experiments show consistent speed gains. TIDE improves throughput across two model sizes, different generation lengths, and different GPU expert budgets. Further tests with different block sizes and confidence thresholds also support its practical value.

Critical weaknesses

The cost model needs stronger validation. The paper does not clearly explain the form of the CPU cost function f(τ)f(\tau) or how it is fitted. It also does not compare predicted and measured latency. Comparing only against updates at every step and random intervals does not show whether the model selects an interval close to the best measured choice.

Questions for authors

On inputs not used for offline profiling, how does the throughput achieved with the selected interval compare with the best throughput found by testing all candidate intervals? This would help verify the value of the cost model.

3 comments

Dear Reviewer mejn,

Thank you for your valuable and constructive comments. We have added the previously omitted functional form, its motivation, and the coefficient-fitting procedure to the appendix.

W1. CPU cost function and fitting.

We clarify the functional form of f(τ)f(\tau) and explain how its coefficients are obtained here. For refresh interval τ\tau and expert-placement drift parameter dd, the CPU workload relative to full refresh is modeled by

f(τ)=1+β(τ−1)+γ[(1−d)−(1−d)τ],β,γ≥0.f(\tau)=1+\beta(\tau-1)+\gamma\bigl[(1-d)-(1-d)^\tau\bigr],\qquad \beta,\gamma\geq0.

Each term has a specific role:

  • Baseline 11. Even full refresh leaves CPU work when the GPU cannot hold every demanded expert. This term retains that baseline workload, whose time scale is absorbed into CCPUC^{\mathrm{CPU}}.
  • Linear term β(τ−1)\beta(\tau-1). This captures additional CPU work that continues to grow as placement ages. For example, if additional CPU assignments grow approximately linearly with placement age, averaging ages 0,…,τ−10,\ldots,\tau-1 gives a contribution proportional to (τ−1)/2(\tau-1)/2; the factor 1/21/2 is absorbed into β\beta. This is an empirical approximation over the calibrated interval range.
  • Mismatch term γ[(1−d)−(1−d)τ]\gamma[(1-d)-(1-d)^\tau]. Under the constant-drift approximation, accumulated mismatch is 1−(1−d)τ1-(1-d)^\tau. Subtracting the one-step baseline dd gives (1−d)−(1−d)τ(1-d)-(1-d)^\tau. This term captures a saturating contribution to additional CPU assignments as GPU placement becomes less aligned with current expert demand; γ\gamma determines its magnitude.

Both additional terms vanish at τ=1\tau=1, giving f(1)=1f(1)=1. Nonnegative β\beta and γ\gamma make this workload response nondecreasing. Their combination allows both saturating mismatch and sustained workload growth.

Substituting f(τ)f(\tau) into the CPU-computation term gives the full cost model for one sparse layer over TT denoising steps:

Lat^(τ)=CI/OBTτ[1−(1−d)τ]+CCPUBT[1+β(τ−1)+γ((1−d)−(1−d)τ)].\widehat{\mathrm{Lat}}(\tau)=C^{\mathrm{I/O}}\frac{BT}{\tau}[1-(1-d)^\tau]+C^{\mathrm{CPU}}BT[1+\beta(\tau-1)+\gamma((1-d)-(1-d)^\tau)].

τ∗∈arg⁡min⁡τ∈{1,2,…,T−1}  Lat^(τ),\tau^*\in\underset{\tau\in\{1,2,\ldots,T-1\}}{\arg\min}\;\widehat{\mathrm{Lat}}(\tau),

where BB is the GPU expert budget per sparse layer, CI/OC^{\mathrm{I/O}} and CCPUC^{\mathrm{CPU}} are configuration-dependent timing coefficients.

We fit the model offline using profiling runs at several refresh intervals under a fixed model, GPU expert budget, and decoding configuration. We collect CPU token–expert assignment counts, expert replacement counts, and separate CPU-computation and migration times, normalized by the denoising steps actually executed. First, we fit an effective dd to migration counts with the full-refresh count anchored. With dd fixed, we jointly fit nonnegative β\beta and γ\gamma by least squares to CPU assignment counts divided by their full-refresh value. We then freeze the count curves and separately regress CPU time on predicted assignments and migration time on predicted replacements, both through the origin, converting the fitted scales to CCPUC^{\mathrm{CPU}} and CI/OC^{\mathrm{I/O}} using the objective's normalization. Finally, we fix all coefficients and select τ\tau by minimizing the estimated latency.

Q1. Selected interval versus exhaustive search.

We compare the throughput performance of (1) the interval selected by minimizing the fitted cost model with (2) the oracle interval found by exhaustively measuring all candidates. The optimal interval derived from the cost model achieve 89-96% of the best measured throughput.

ModelGPU Expert BudgetTIDE Throughput (tok/s)Oracle hroughput (tok/s)Throughput Ratio
LLaDA2.0-mini642.112.2494%
LLaDA2.0-mini1282.362.6489%
LLaDA2.0-flash321.251.3990%
LLaDA2.0-flash641.731.8196%

Thank you for clarifying the CPU cost function and its offline fitting procedure, and for adding the exhaustive comparison. Across the four configurations, the model-selected refresh interval achieves 89–96% of the best measured throughput. This gives me more confidence that the cost model can select a useful interval in practice. The rebuttal strengthens the empirical case for TIDE, and I maintain my overall rating.

Thank you for taking the time to review our response and for your constructive feedback! We are glad that the cost-function clarification and exhaustive comparison gave you greater confidence in TIDE's practical value, and we appreciate your continued support for our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear ACs, PCs, and Reviewers,

We sincerely thank all reviewers for their constructive feedback and for engaging with our rebuttal. We are encouraged to have received positive evaluations from all reviewers, and are especially pleased that Reviewer sDTF found their concerns satisfactorily addressed and raised their score, and that Reviewer mejn noted the rebuttal strengthened the empirical case for TIDE.

To assist the Area Chair and Reviewers, we summarize the key clarifications, new experiments, and manuscript updates made during the discussion period:

  • Cost Model Formulation & Fitting (Reviewers mejn, sDTF): Restored CI/OC_{\text{I/O}} in Eq. (7) and added the explicit functional form of the CPU cost function f(τ)f(\tau), its theoretical motivation, and the offline coefficient-fitting procedure to the appendix.
  • Cost Model Validation vs. Oracle Search (Reviewers mejn, sDTF): Compared the throughput performance of (1) the interval selected by minimizing the fitted cost model with (2) the oracle interval found by exhaustively measuring all candidates. The optimal interval derived from the cost model achieve 89-96% of the best measured throughput.
  • Generalization to Additional Datasets (Reviewer BAN2): Added throughput evaluations on GSM8K across LLaDA2.0-mini and LLaDA2.0-flash (generation lengths 256 and 1024), where TIDE consistently achieves 1.15–1.38× speedups over Fiddler and Mixtral-Offload.
  • Losslessness & Accuracy Verification (Reviewer sDTF): Clarified mathematical equivalence to full-GPU execution and added accuracy benchmarks on both MBPP and GSM8K, confirming virtually identical accuracy to normal execution.
  • Baseline Clarification (Reviewer BAN2): Clarified how Fiddler and Mixtral-Offload bound AR-based offloading strategies under single-request block decoding (τ=∞\tau = \infty vs. τ=1\tau = 1).
  • System Sensitivity (Reviewer BAN2): Reported additional analyses on tighter GPU memory budgets (down to 4 GB / 16 experts), offline profiling costs, and CPU core scaling (12–48 cores).

Thank you again for your time and thoughtful review of our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.