- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how caching affects the actual serving cost of mixture-of-experts (MoE) diffusion language models. The main observation is that reducing the number of forward evaluations does not necessarily mean that inference becomes cheaper. Cached execution may process fewer query positions and transfer fewer expert weights, even when it performs the same number of forwards or, in some cases, more forwards.
To study this problem, the authors introduce TRACE-MoE, an evaluation framework that tracks five quantities: query visits, forward evaluations, expert calls, transferred bytes, and end-to-end latency. The framework compares cache policies against controls from the same cache family and uses paired timing and separate diagnostic runs to check the measurements.
The experiments evaluate Fast-dLLM and dKV-Cache on LLaDA2.0-mini and LLaDA2.1-mini. One notable result is that Fast-dLLM dual caching reduces latency by 46.7% on the M20/GSM8K setting, even though the number of forward evaluations increases from 29 to 36. In the same setting, query visits decrease by 73.6%, expert calls by 48.5%, and transferred bytes by 68.3%. The paper also includes fixed-forward-count experiments and supplementary analyses of expert residency, cache capacity, and execution under different workloads.
Overall, the paper argues that evaluating caching by forward count alone is insufficient, and that a more complete picture requires measuring the work performed within each forward, the expert weights that must be loaded, and the resulting request latency.
Critical strengths
-
The paper asks a practical and relevant question. The distinction between forward count and actual serving cost is important for MoE diffusion inference. The paper makes this concrete through the M20/GSM8K experiment: forward evaluations increase from 29 to 36, yet latency falls by 46.7%. The accompanying reductions in query visits, expert calls, and transferred bytes help explain why counting forwards alone can give a misleading picture of efficiency.
-
The evaluation design includes useful controls. The authors compare caching policies against controls from the same cache family, keep request inputs and runtime settings matched, reverse the timing order between repeats, and use separate diagnostic runs to check the work counters. The Refresh-1 and Refresh-4 comparison is also helpful. Refresh-1 provides a reference where the recorded workload remains essentially unchanged, while Refresh-4 reduces execution work at the same forward count. These comparisons make the results easier to interpret than a simple comparison of latency numbers alone.
Critical weaknesses
-
The experimental evidence is still too limited to support the broader claims confidently. The primary timing protocol uses only two paired repeats per mode. Reversing the execution order is a sensible precaution, but two repeats are not enough to characterize runtime variability or establish how stable the reported savings are. The main evaluation also covers a narrow set of model–task configurations, using two LLaDA mini checkpoints and a single A800 setup with a fixed expert-residency configuration. The appendix adds useful experiments on other inputs, cache capacities, hardware, and execution settings. However, several of these additional experiments use fixed traces or different residency policies, so they do not fully replace broader live evaluations of the central caching comparisons. This matters because the paper presents TRACE-MoE as a practical evaluation framework, not merely as an explanation of a few particular measurements. More evidence is needed to establish how consistently its findings hold across requests, models, and serving configurations.
-
The quality evaluation is too small to establish that the speedups are useful in practice, and the framework’s broader contribution remains insufficiently demonstrated. The paper reports task correctness at 64 and 512 tokens for the inputs used in the main comparison. However, this is a very small sample for drawing conclusions about quality-preserving acceleration. Appendix B reports that, for M20/GSM8K at 64 tokens, Fast dual is correct while the other configurations, including its no-cache control, are incorrect; at 512 tokens, all the corresponding configurations are correct. These results do not invalidate the latency measurements, but they show that correctness can depend on the generation budget and cache policy. A common token budget alone does not establish comparable output quality. The paper does not yet clearly establish what TRACE-MoE enables that existing runtime counters and profiling approaches cannot provide, or demonstrate that it is a reusable framework beyond the particular implementation studied here. A clearer distinction from prior evaluation approaches, together with broader validation and an accessible implementation or measurement artifact, would make the contribution more convincing.
Questions for authors
-
How stable are the reported latency savings? The primary timing experiments use only two paired repeats. Could the authors provide the per-repeat results and additional runs, together with a measure of variability or confidence intervals? It would be helpful to know whether the reported reductions remain stable across repeated executions.
-
Do the speedups hold when output quality is evaluated on a broader set of requests? Could the authors evaluate task correctness on a more representative sample of GSM8K and HumanEval inputs, at both 64 and 512 tokens? In particular, how should readers interpret the M20/GSM8K result where Fast dual is correct at 64 tokens but its matched no-cache control is not? Additional quality measurements would help clarify when the reported latency reductions translate into useful acceleration.