- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents TIDE, a system for expert offloading in diffusion MoE inference. It exploits similar expert demand across adjacent denoising steps to periodically update GPU-resident experts based on usage counts. Expert placement remains fixed between updates, while nonresident experts continue to run on the CPU. A cost model selects the refresh interval to balance expert migration and CPU computation. Experiments on two LLaDA2.0 models show that TIDE achieves higher throughput than both baselines across all tested configurations.
Critical strengths
-
TIDE uses expert demand patterns in diffusion MoE to periodically update GPU-resident experts, balancing weight migration and CPU computation costs while keeping the original routing unchanged and requiring no retraining.
-
The results are fairly consistent across two models, different generation lengths, and GPU expert budgets.
Critical weaknesses
Equation (7) omits the I/O cost coefficient C_IO from Equation (5) without explaining whether the objective was normalized. The paper also does not clearly define f(τ) or explain how it is fitted. Table 2 reports neither the selected refresh intervals nor a comparison with a tuned fixed-interval baseline, making it hard to judge the additional benefit of the cost model over periodic refresh. The paper reports only throughput on MBPP, without accuracy or output agreement results to support its losslessness claim.
Questions for authors
- Is Equation (7) normalized by C_IO? How is f(τ) determined? Please report the selected refresh intervals and compare performance with a tuned fixed-interval baseline to assess the cost model's additional benefit.
- Can the authors provide output agreement results under the same decoding settings, along with MBPP accuracy, to support the losslessness claim?
3 comments
Thank you for your rebuttal, as well as for conducting the additional experiments. Your responses have satisfactorily addressed my concerns, and I have updated my review score accordingly.
Thank you for taking the time to read our response and for your constructive feedback! We are glad that our clarifications and additional experiments addressed your concerns.
Dear Reviewer sDTF,
Thank you for your valuable and constructive comments. We have corrected the objective and added the CPU-cost formulation and fitting procedure to the appendix.
W1 / Q1. I/O coefficient and CPU cost function.
We restored CI/O in Eq. (7), so the revised objective includes both timing coefficients explicitly.
We clarify the functional form of f(τ) and explain how its coefficients are obtained here. For refresh interval τ and expert-placement drift parameter d, the CPU workload relative to full refresh is modeled by
f(τ)=1+β(τ−1)+γ[(1−d)−(1−d)τ],β,γ≥0.
Each term has a specific role:
Both additional terms vanish at τ=1, giving f(1)=1. Nonnegative β and γ make this workload response nondecreasing. Their combination allows both saturating mismatch and sustained workload growth.
Substituting f(τ) into the CPU-computation term gives the full cost model for one sparse layer over T denoising steps:
Lat(τ)=CI/OτBT[1−(1−d)τ]+CCPUBT[1+β(τ−1)+γ((1−d)−(1−d)τ)].
τ∗∈τ∈{1,2,…,T−1}argminLat(τ),
where B is the GPU expert budget per sparse layer, CI/O and CCPU are configuration-dependent timing coefficients.
We fit the model offline using profiling runs at several refresh intervals under a fixed model, GPU expert budget, and decoding configuration. We collect CPU token–expert assignment counts, expert replacement counts, and separate CPU-computation and migration times, normalized by the denoising steps actually executed. First, we fit an effective d to migration counts with the full-refresh count anchored. With d fixed, we jointly fit nonnegative β and γ by least squares to CPU assignment counts divided by their full-refresh value. We then freeze the count curves and separately regress CPU time on predicted assignments and migration time on predicted replacements, both through the origin, converting the fitted scales to CCPU and CI/O using the objective's normalization. Finally, we fix all coefficients and select τ by minimizing the estimated latency.
W1 / Q1. Selected intervals and the fixed-interval baseline.
We compare the throughput performance of (1) the interval selected by minimizing the fitted cost model with (2) the oracle interval found by exhaustively measuring all candidates. The optimal interval derived from the cost model achieve 89-96% of the best measured throughput.
Q2. Accuracy and output agreement.
We would like to highlight that TIDE is strictly lossless by design. TIDE does not modify the MoE router selection, model weights, or the parallel decoding mechanism. Specifically, as shown in Algorithm 1, at every denoising step t, every token x is still routed to and processed by its exact set of selected top-k experts Ex. Because the mathematical operations performed for every tokenare identical regardless of whether an expert resides on the CPU or GPU (and regardless of the refresh interval τ), TIDE is mathematically equivalent to standard full-GPU inference and the baselines.
We have added accuracy experiments on MBPP and GSM8k. TIDE achieves almost identical model performance with GPU-only baselines.