acceptodds
Under review as a conference paper at ICLR 2027

Beyond Forward Counts: Caching in MoE Diffusion Models

Abstract

Caching reduces repeated token computation in diffusion language models, but its serving benefit in offloaded mixture-of-experts (MoE) models depends on how reuse propagates through expert execution and weight movement. These operations occur at different granularities: routed query positions are grouped into expert calls, while expert residency determines which weights must move. Consequently, a large reduction in processed query positions need not produce a comparable reduction in expert calls or transferred bytes, and forward count alone cannot characterize serving efficiency. We introduce TRACE-MoE, an execution-aware evaluation framework that jointly measures query visits, forward evaluations, expert calls, transferred bytes, and request latency to connect token reuse with its downstream execution costs. Family-matched controls and paired timing compare cache policies under identical requests and runtime configurations, while independent diagnostic runs reconcile the measured work counters. Workload-preserving controls provide a timing reference, and fixed-forward-count comparisons reveal changes in work within denoising steps. Controlled executions of Fast-dLLM and dKV-Cache on LLaDA2 mini checkpoints compare cache policies with their corresponding family controls at a common 64-token generation budget. These measured configurations include lower latency with more forward evaluations, with paired savings of up to 59.0% when forward count increases. TRACE-MoE makes the relationships among token reuse, expert demand, weight movement, and latency explicit, providing a practical basis for evaluating cache policies under matched execution conditions.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Diffusion Language Models, Mixture-of-Experts, Expert Offloading, KV Caching, Efficient Inference, Performance Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CYkoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how caching affects the actual serving cost of mixture-of-experts (MoE) diffusion language models. The main observation is that reducing the number of forward evaluations does not necessarily mean that inference becomes cheaper. Cached execution may process fewer query positions and transfer fewer expert weights, even when it performs the same number of forwards or, in some cases, more forwards.

To study this problem, the authors introduce TRACE-MoE, an evaluation framework that tracks five quantities: query visits, forward evaluations, expert calls, transferred bytes, and end-to-end latency. The framework compares cache policies against controls from the same cache family and uses paired timing and separate diagnostic runs to check the measurements.

The experiments evaluate Fast-dLLM and dKV-Cache on LLaDA2.0-mini and LLaDA2.1-mini. One notable result is that Fast-dLLM dual caching reduces latency by 46.7% on the M20/GSM8K setting, even though the number of forward evaluations increases from 29 to 36. In the same setting, query visits decrease by 73.6%, expert calls by 48.5%, and transferred bytes by 68.3%. The paper also includes fixed-forward-count experiments and supplementary analyses of expert residency, cache capacity, and execution under different workloads.

Overall, the paper argues that evaluating caching by forward count alone is insufficient, and that a more complete picture requires measuring the work performed within each forward, the expert weights that must be loaded, and the resulting request latency.

Critical strengths
  1. The paper asks a practical and relevant question. The distinction between forward count and actual serving cost is important for MoE diffusion inference. The paper makes this concrete through the M20/GSM8K experiment: forward evaluations increase from 29 to 36, yet latency falls by 46.7%. The accompanying reductions in query visits, expert calls, and transferred bytes help explain why counting forwards alone can give a misleading picture of efficiency.

  2. The evaluation design includes useful controls. The authors compare caching policies against controls from the same cache family, keep request inputs and runtime settings matched, reverse the timing order between repeats, and use separate diagnostic runs to check the work counters. The Refresh-1 and Refresh-4 comparison is also helpful. Refresh-1 provides a reference where the recorded workload remains essentially unchanged, while Refresh-4 reduces execution work at the same forward count. These comparisons make the results easier to interpret than a simple comparison of latency numbers alone.

Critical weaknesses
  1. The experimental evidence is still too limited to support the broader claims confidently. The primary timing protocol uses only two paired repeats per mode. Reversing the execution order is a sensible precaution, but two repeats are not enough to characterize runtime variability or establish how stable the reported savings are. The main evaluation also covers a narrow set of model–task configurations, using two LLaDA mini checkpoints and a single A800 setup with a fixed expert-residency configuration. The appendix adds useful experiments on other inputs, cache capacities, hardware, and execution settings. However, several of these additional experiments use fixed traces or different residency policies, so they do not fully replace broader live evaluations of the central caching comparisons. This matters because the paper presents TRACE-MoE as a practical evaluation framework, not merely as an explanation of a few particular measurements. More evidence is needed to establish how consistently its findings hold across requests, models, and serving configurations.

  2. The quality evaluation is too small to establish that the speedups are useful in practice, and the framework’s broader contribution remains insufficiently demonstrated. The paper reports task correctness at 64 and 512 tokens for the inputs used in the main comparison. However, this is a very small sample for drawing conclusions about quality-preserving acceleration. Appendix B reports that, for M20/GSM8K at 64 tokens, Fast dual is correct while the other configurations, including its no-cache control, are incorrect; at 512 tokens, all the corresponding configurations are correct. These results do not invalidate the latency measurements, but they show that correctness can depend on the generation budget and cache policy. A common token budget alone does not establish comparable output quality. The paper does not yet clearly establish what TRACE-MoE enables that existing runtime counters and profiling approaches cannot provide, or demonstrate that it is a reusable framework beyond the particular implementation studied here. A clearer distinction from prior evaluation approaches, together with broader validation and an accessible implementation or measurement artifact, would make the contribution more convincing.

Questions for authors
  1. How stable are the reported latency savings? The primary timing experiments use only two paired repeats. Could the authors provide the per-repeat results and additional runs, together with a measure of variability or confidence intervals? It would be helpful to know whether the reported reductions remain stable across repeated executions.

  2. Do the speedups hold when output quality is evaluated on a broader set of requests? Could the authors evaluate task correctness on a more representative sample of GSM8K and HumanEval inputs, at both 64 and 512 tokens? In particular, how should readers interpret the M20/GSM8K result where Fast dual is correct at 64 tokens but its matched no-cache control is not? Additional quality measurements would help clarify when the reported latency reductions translate into useful acceleration.

Reviewer_gDz6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TRACE-MoE, an execution-aware evaluation framework for analyzing KV cachine in MoE-based dLLMs. The central claim is that reductions in token-level computation do not necessarily translate into proportional reductions in expert computation, weight transfers, or end-to-end inference latency. TRACE-MoE jointly measures query visits, forward evaluations, expert calls, transferred bytes, and request latency, using family-matched controls, paried timing, and independent diagnostic runs. The evaluation of existing caching methods (Fast-dLLM and dKV-Cache) on LLaDA2.0-mini and LLaDA2.1-mini show that query-level computation, expert calls, and weight transfers can exhibit different reduction patterns, and that lower latency can be achieved even with an increased number of forward evaluations.

Critical strengths
  1. The paper identifies a meaningful and practically relaveant characterisctic of MoE-based dLLM execution: existing KV caching methods designed for dense dLLMs do not necessarily translate to reduced token-level computation into fewer expert loads or proportional end-to-end sppedup. In particular, forward evaluation counts measure how often the model runs, but not how much work each run performs. Highlighting this gap between token-level reuse and expert-levle execution costs motivates a more comprehensive evaluation of caching strategies in MoE-based dLLMs.
Critical weaknesses
  1. The proposed evaluation methodlogy has limited methodological novelty. The metrics used are already commonly considered in MoE inference and serving systems, and the paper does not sufficiently demonstrate what new analytical capability TRACE-MoE provides beyond jointly profiling existing metrics. Moreover, the evaluated caching methods were not designed for MoE models, so it is unsurprising that token-level savings do not translate proportionally; the results are largely intuitive and do not imply new design principles for MoE-specific caching.

  2. The results do not establish how much of the observed latency saving is attributable to reduced attention work, expert computation, or exposed host-to-device weight transfer time, since KV reuse changes attention work at the same time and the main experitments report only whole-request latency. It also remains unclear whether these relationships persist for longer generations or on different hardware. This limits the reach of the empirical findings rather than validity of the reported comparisons.

Questions for authors
  1. What fraction of the latency change in the main cache-policy comparisions comes from attention computation, expert computation, and exposed host-to-device transfer time? Can you provide component timings or matched runs with all experts resident?

  2. When a cache mode changes the number of forward evaluations, how different are its generated tokens and expert routes from those of its family-matched control? This would clarify how much the change in expert calls and transferred bytes reflects skipped query processing versus a changed decoding trajectory.

  3. Could you identify any non-trivial findings that are not readily predictable from existing knowledge of MoE inference, and show how they translate into actionable design principles for MoE-specific dLLM caching? Would evaluating a caching method designed for MoE-based dLLMs reveal additional insights from TRACE-MoE?

  4. Can the proposed metrics be used to pinpiont the specific bottlenecks responsible for the limited speedups of certain caching configurations, or to predict when a reduction in query visits will translate into meaningful end-to-end sendup?

Reviewer_PofDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TRACE-MoE, an evaluation framework for caching in offloaded mixture-of-experts diffusion language models. It connects token reuse and forward evaluations to expert execution, weight transfers, and request latency through controlled comparisons of caching policies.

Critical strengths

None

Critical weaknesses
  1. The evaluation isn't complete, for example, missing the Comparisons with strong AR models. Accelerated autoregressive systems would help place the results in context and assess their implications for efficient LLM serving. For exmaple, if well-optimized caching can accelerate AR well and better than DLM, it is less convincing to justify the diffusion-model caching policies.

  2. The study is restricted to offloaded MoE diffusion models, and it is unclear how far the findings extend across deployment settings. Local inference and server serving have different operating requirements: local use often involves small batches and prioritizes request latency, while server workloads must also account for batching and throughput. The paper needs clearer evidence of where its conclusions apply and where they do not.

Overall, I recommend rejection because the practical significance and applicability across serving settings remain insufficiently established.

Questions for authors

How do the evaluated systems compare with accelerated autoregressive inference at comparable output quality? Which conclusions are specific to offloaded MoE diffusion models? How do the latency benefits change between small-batch local inference and server workloads?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.