- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TEMPER is a distributed MoE inference system for memory-constraint multi-GPU servers, where expert parallelism and expert offloading need to be enabled at the same time. It proposes three techniques: destination planning before activation transfer, enabling token communication overlap with expert load; prediction-based replication and cost-aware token assignment; windowed layer-major execution across dependency-ready scheduling turns to increase expert reuse.
The evaluation uses bf16-version qwen3-30b-a3b on a 4 * 3080ti server, with short MMLU prompts and ~4k-length LongBench prompts. Compared with vllm-ep, vllm-dp, and ktransformers, the paper reports 76.8-82.0% lower mean e2e latency. It also reports normalized-load and offline comparisons, cumulative ablations, and a 2.13× increase in expert invocations per physical weight load under the same cache capacity.
Critical strengths
1. A coherent design for the coupled communication-offloading problem. The paper treats expert loading, token placement, and execution order as interacting decisions. Destination-aware communication can reduce activation transfers, replica-aware assignment accounts for both remote execution and one-time loading costs, and windowed execution creates opportunities to reuse the resulting resident experts. The distinction between designated execution locations and current GPU residency is particularly useful in an offloaded setting. These mechanisms form a plausible integrated design rather than a collection of unrelated optimizations.
2. Direct evidence for improved weight resue, with trade-offs. Figure 1(c) compares turn-major and windowed layer-major execution using the same workload and cache capacity, reporting 2.13× more token–expert invocations per physical load. This directly supports the central reuse mechanism. The paper also includes identical-request, identical-arrival comparisons and fixed output lengths, rather than relying exclusively on system-specific normalized loads.
Critical weaknesses
1. The incremental contribution over existing offloading and multi-batch techniques is not convinciingly established.
Existing work like MoE-Lightning already investigated offloade MoE inference through CPU-GPU-I/O pipelining, and Klotski develops expert-aware multi-batch scheduling and prefetching to reduce pipeline stalls. Thus, TEMPER should emphasize what it contributes beyond expert offloading, prefetching, and multi-batch weight reusing. The currrent evaluation does not provide a sufficient strong comparison with existing works.
The appendix b.1 specifies layer-granularity PrefetchOffloader for the vllm baselines, yet a persisitent, cross-layer expert cache for TEMPER. The vllm-EP baseline allso uses allgather/reduce-scatter transport, while TEMPER organizes transfers by destination.
Table 1 partially addresses this through cumulative ablations, but “Native” is not specified precisely enough to determine whether cache implementation and residency policy are already matched. Moreover, “Communication,” which provides the largest incremental gain, combines transfer coalescing and dispatch/loading overlap. Thus, neither the dominant source of improvement nor the incremental value beyond a well-implemented expert-offloading baseline is sufficiently isolated.
2. The evaluation does not sufficiently establish robustness or the useful latency operating range
The experiments cover one model, one hardware configuration, and two prompt-length regimes. More importantly, each online run has only 16 measured requests, and Figure 4—the shared-arrival comparison supporting the headline latency reduction—uses a single run per configuration. Its whiskers are within-run request-level min–max ranges, not uncertainty estimates across independent executions. Warmup cache and placement state are retained. For a stateful caching and scheduling system, this leaves substantial uncertainty about sensitivity to request order, initialization, and longer-running traffic.
The latency trade-off also requires clearer characterization. Offline LongBench shows a 9.8–10.0× increase in mean TTFT, although end-to-end latency decreases. This is not a correctness problem, and the paper explicitly optimizes mean request completion latency rather than imposing a TTFT constraint. Nevertheless, the result shows that the chosen schedule can exchange a large initial delay for more efficient subsequent generation. With fixed window settings and limited workload coverage, the paper does not yet establish when this exchange remains beneficial as output length, concurrency, or cache capacity changes.
The reported gains may be real for the measured traces, but they do not yet provide a robust characterization of the proposed system’s advantage.
Questions for authors
Q1. About baselines and contribution attribution.
What exactly does “Native” in Table 1 include? In particular, does it already use TEMPER’s persistent expert-granularity cache, eviction policy, memory budget, and admission limits? What changes when “Communication” is enabled beyond transfer coalescing and asynchronous loading? A comparison that keeps an expert-granularity cache and resource budget fixed while isolating communication overlap, coalescing, and windowed scheduling would directly affect my assessment. Please also explain which benefit is not already obtained by an ordinary prefetching and layer-major multi-batch baseline, given the closest related work.
Q2. About robustness and latency trade-off.
Can the shared-arrival comparison be repeated with more measured requests and independent request/arrival sequences, reporting run-level uncertainty and stating how cache and placement state are initialized? Within the existing platform, a focused comparison of TTFT, TPOT, and end-to-end latency at a few shared offered loads or output lengths would be particularly informative. How does the windowed scheduler behave when insufficient work is ready to form the intended cohort, and what prevents waiting for reuse opportunities from dominating the request’s completion time?
Q3. About replication.
Table 2 reports few new H2D replica copies, while Table 1 shows essentially no MMLU gain and an 11.3% LongBench gain from enabling replication. I do not interpret low copy counts as proof that replication is ineffective, since designated locations and resident weights are distinct. However, how many subsequent invocations actually execute at the additional locations, and how much exposed loading time or critical-rank execution time do they save beyond predictive prefetching?
For accepted placement updates, how well does estimated net benefit predict measured benefit? Appendix B.1 notes that the single-expert copy estimate excludes full distributed directory-commit latency; please clarify how that additional cost is accounted for in admission or hidden during execution. This would help establish whether the cost-based replication policy contributes reliably, rather than only in the reported LongBench configuration.