- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether expert prefetching improves offloaded diffusion MoE inference. The authors find that stable routing across denoising steps can limit its benefit: many experts predicted for the next step are already retained by the GPU cache. They therefore use per-layer LRU caching with asynchronous demand loading, overlapping transfers with computation without speculative prefetching. Experiments on LLaDA-2.0-mini and LLaDA-2.0-flash across GPUs, cache capacities, and tasks show throughput gains over the evaluated prefetching baseline, particularly when the cache can retain most of a denoising step’s expert working set.
Critical strengths
(1) The paper addresses a practical bottleneck in offloaded diffusion MoE inference: the latency of transferring expert weights from CPU to GPU. Its main insight is that predictable routing does not guarantee useful prefetching, because many predicted experts are already cached. Accounting for cache residency and redundant transfers gives a useful perspective on when prefetching is worth its cost. (2) The evaluation does a good job of separating the effects of prefetching and asynchronous demand loading. The factorial comparison distinguishes predictive placement from transfer overlap, while cache-miss counts, an oracle predictor, and bandwidth throttling help explain the observed performance differences. Tests across models, GPUs, and cache capacities support the main result, with additional task-level experiments providing broader, though single-run, checks.
Critical weaknesses
(1) The novelty of this paper is limited, as it mainly combines established techniques such as LRU caching and asynchronous demand loading. While the finding that high routing predictability does not necessarily translate into prefetching benefits is interesting, prior studies on expert caching and offloading have already highlighted the importance of routing locality and cache residency. (2) The evaluation of alternative prefetching strategies could be more comprehensive. Although the authors compare Last-step Prefetching and a TIDE-inspired Interval Refresh policy, the additional predictor variants in Appendix A.3 are evaluated only with synchronous loading on an earlier engine revision. Moreover, the evaluated prefetching pipeline introduces synchronization overhead that is not separately quantified. (3) Section 5 claims that the proposed approach is "no worse" when the cache capacity falls far below the working set. However, Table 2 reports a statistically insignificant difference at cap/W = 0.43, with a 95% confidence interval of [−1.16%, +2.63%]. This result does not establish non-inferiority.
Questions for authors
(1) Could the authors clarify what new insights this work provides beyond existing studies on routing locality and expert caching? In particular, how does the observation that predictable routing does not necessarily make prefetching beneficial advance the understanding established by prior work? (2)Could the authors provide additional evidence that the advantage of no-prefetching persists under alternative practical prefetching strategies evaluated within the same asynchronous execution framework? Additionally, how much of the observed throughput difference is attributable to synchronization overhead in the current prefetching pipeline?
2 comments
Response to Reviewer Csz2 - Continue
How much of the difference is synchronization?
We measured the time spent waiting for speculative copies and ran additional controls to test whether those waits explain the throughput gap.
| Measurement or comparison | Configuration | Result |
|---|---|---|
| Time waiting for speculative copies | Original last-step, capacity 128 | 4.4 s (2.8% of runtime) |
| Time waiting for speculative copies | Original last-step, capacity 64 | 18.4 s (4.5% of runtime) |
| No prefetch vs. null prefetch | Capacity 128, 43 paired rounds | +1.2% [−1.0, +3.4] |
| Null prefetch vs. original last-step | Capacity 128, 43 paired rounds | +4.1% [+1.6, +6.6] |
| Lean prefetch without blanket wait vs. lean prefetch | Capacity 128, 16 paired rounds | +2.3% [−1.2, +5.7] |
Wait times are summed measurements from individual instrumented runs. Throughput comparisons are paired mean percentage differences with 95% confidence intervals. A positive difference means the first configuration is faster.
The null prefetcher runs the original CPU-side prediction, bookkeeping, and synchronization but does not copy predicted experts to the GPU. Its comparison with no prefetch shows no statistically resolved overhead from that host-side path. Enabling speculative transfers produces an additional measured difference, which can also reflect changes in cache residency. Removing the blanket wait from the lean prefetcher does not produce a statistically resolved throughput improvement. Thus, synchronization alone does not explain the observed gap, but these controls cannot assign a precise share of the gap to waiting versus transfer contention or cache effects. We have made this limitation explicit in Sec. 4.3 (Lines 416–424), Sec. 4.4, Table 4, discussion (Lines 433–458), Appendix A.11, and Table 10 (Lines 963–983).
Small-cache interpretation
We agree that the original phrase “no worse” was too strong, since a statistically insignificant difference does not establish non-inferiority. At capacity 32 (cap/W ≈ 0.43), LLaDA-2.0-flash on H200 shows a no-prefetch advantage of +0.73% [−1.16, +2.63], while LLaDA-2.0-mini on A5000 shows −1.3% [−3.3, +0.6]. Against the optimized prefetcher, the latter becomes −2.0% [−3.2, −0.9] over three paired rounds, suggesting that prefetching can help at small capacities, although the sample size is limited.
These results do not establish a no-prefetch advantage far below the working-set size. We have removed “no worse” and clarified this capacity-dependent limitation in the Abstract (Lines 29–32), Figure 2 caption (Lines 361–368), Sec. 4.4, Table 4 (Lines 433–448), Sec. 4.5 (Lines 460–471), and Conclusion (Lines 534–537).
Response to Reviewer Csz2
We thank the reviewer for identifying the distinction between routing predictability and useful cache movement. We address the novelty, alternative prefetching policies, and the small-cache interpretation.
We agree that routing locality, expert caching, and asynchronous loading are established. What is missing from routing-locality analysis is whether a predictable future expert is also a useful expert to move. A predicted expert may already be resident under LRU. Conversely, a remaining cache miss may be avoidable by better replacement but not identifiable from the previous denoising step. Thus, routing overlap and cache hit rate alone cannot determine the incremental value of prefetching.
We distinguish two quantities using the same recorded trace: the fraction of LRU misses that optimal offline replacement (Belady) could avoid, and the fraction involving experts routed one step earlier that a last-step predictor could potentially identify.
Both percentages are fractions of LRU misses at that capacity. They do not directly measure the intersection of the two groups.
These quantities move in opposite directions across cache sizes. At capacity 128, almost half of LRU misses are avoidable with future-aware replacement, so the result is not simply that the cache is already optimal. However, only 2% involve previous-step experts. At capacity 32, previous-step experts appear in 75% of LRU misses, but Belady reduces the miss count by only 7%. The result identifies a distinction that routing-locality measurements miss: opportunity to improve residency and information available to a predictor are different constraints. Neither percentage by itself proves a limit on prefetching, which can also hide transfer latency. We therefore test practical predictors against the same asynchronous LRU execution, rather than attributing gains from asynchronous loading to prediction. This is the analytical and experimental contribution, not a new cache algorithm. Revised manuscript: Sec. 1 (Lines 74–77, 85–100); Sec. 4.3 (Lines 320–323, 371–373) and Table 3 (Lines 379–389).
We ported a TIDE-inspired interval-refresh predictor into the same engine and used the same eager, asynchronous execution settings for all policies.
Two interleaved rounds per setting. These are observed throughput values, not statistically conclusive pairwise comparisons.
None of the tested policies was faster in these runs. The key fairness check is the optimized last-step prefetcher. After reducing speculative copies and removing extra device synchronization, the no-prefetch advantage falls to +0.8% [−1.8, +3.4] over 27 pooled paired rounds at capacity 128. This interval establishes neither equivalence nor a no-prefetch advantage. The lean implementation closes most of the original gap, demonstrating that implementation overhead matters, but it has not demonstrated a throughput benefit over asynchronous LRU. Our conclusion concerns the incremental value of the evaluated predictors under a shared execution framework, not a claim that all prefetching strategies must lose. We have included these comparisons and this qualification in Sec. 4.4 (Lines 427–458), Table 4 (Lines 433–448), Sec. 4.5 (Lines 483–508), Appendix A.3, and Table 6 (Lines 801–827).