acceptodds
Under review as a conference paper at ICLR 2027

TEMPER: Amortising Data-Movement Overhead for Distributed MoE Inference on Memory-Constrained GPUs

Abstract

Mixture-of-Experts (MoE) models scale the capacity of large language models through sparse per-token computation, making them attractive for on-premises deployment on memory-constrained GPU servers. In practice, however, serving these large models requires expert parallelism together with weight offloading, which shifts the inference bottleneck from computation to data movement. Cross-GPU token exchange and host-to-device expert transfers dominate the execution timeline, and their exposed costs are further amplified by imbalanced routing and limited expert reuse. In this work, we present TEMPER, a distributed MoE inference system that amortises data-movement overhead along two complementary axes, reducing the exposed cost of each transfer and increasing the computation each transfer serves. TEMPER first determines token destinations before payload transfer, and overlaps token exchange with expert loading. Progressive expert replication then adds execution locations for hot experts, while replica-aware dispatch redistributes their token load. Finally, windowed layer-major execution reuses each fetched or replicated expert across several scheduling steps within a window of adjacent layers, and overlaps next-layer routing with in-flight expert transfers across steps. On Qwen3-30B-A3B with four PCIe-connected RTX 3080 Ti GPUs, TEMPER reduces mean end-to-end latency by 76.8-82.0% relative to three baselines under shared absolute arrival rates. Across system-specific normalized arrival rates, the reduction is 60.7-79.5%.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts Inference, Expert Parallelism, Expert Offloading

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GbsrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TEMPER is a distributed MoE inference system for memory-constraint multi-GPU servers, where expert parallelism and expert offloading need to be enabled at the same time. It proposes three techniques: destination planning before activation transfer, enabling token communication overlap with expert load; prediction-based replication and cost-aware token assignment; windowed layer-major execution across dependency-ready scheduling turns to increase expert reuse.

The evaluation uses bf16-version qwen3-30b-a3b on a 4 * 3080ti server, with short MMLU prompts and ~4k-length LongBench prompts. Compared with vllm-ep, vllm-dp, and ktransformers, the paper reports 76.8-82.0% lower mean e2e latency. It also reports normalized-load and offline comparisons, cumulative ablations, and a 2.13× increase in expert invocations per physical weight load under the same cache capacity.

Critical strengths

1. A coherent design for the coupled communication-offloading problem. The paper treats expert loading, token placement, and execution order as interacting decisions. Destination-aware communication can reduce activation transfers, replica-aware assignment accounts for both remote execution and one-time loading costs, and windowed execution creates opportunities to reuse the resulting resident experts. The distinction between designated execution locations and current GPU residency is particularly useful in an offloaded setting. These mechanisms form a plausible integrated design rather than a collection of unrelated optimizations.

2. Direct evidence for improved weight resue, with trade-offs. Figure 1(c) compares turn-major and windowed layer-major execution using the same workload and cache capacity, reporting 2.13× more token–expert invocations per physical load. This directly supports the central reuse mechanism. The paper also includes identical-request, identical-arrival comparisons and fixed output lengths, rather than relying exclusively on system-specific normalized loads.

Critical weaknesses

1. The incremental contribution over existing offloading and multi-batch techniques is not convinciingly established.

Existing work like MoE-Lightning already investigated offloade MoE inference through CPU-GPU-I/O pipelining, and Klotski develops expert-aware multi-batch scheduling and prefetching to reduce pipeline stalls. Thus, TEMPER should emphasize what it contributes beyond expert offloading, prefetching, and multi-batch weight reusing. The currrent evaluation does not provide a sufficient strong comparison with existing works.

The appendix b.1 specifies layer-granularity PrefetchOffloader for the vllm baselines, yet a persisitent, cross-layer expert cache for TEMPER. The vllm-EP baseline allso uses allgather/reduce-scatter transport, while TEMPER organizes transfers by destination.

Table 1 partially addresses this through cumulative ablations, but “Native” is not specified precisely enough to determine whether cache implementation and residency policy are already matched. Moreover, “Communication,” which provides the largest incremental gain, combines transfer coalescing and dispatch/loading overlap. Thus, neither the dominant source of improvement nor the incremental value beyond a well-implemented expert-offloading baseline is sufficiently isolated.

2. The evaluation does not sufficiently establish robustness or the useful latency operating range

The experiments cover one model, one hardware configuration, and two prompt-length regimes. More importantly, each online run has only 16 measured requests, and Figure 4—the shared-arrival comparison supporting the headline latency reduction—uses a single run per configuration. Its whiskers are within-run request-level min–max ranges, not uncertainty estimates across independent executions. Warmup cache and placement state are retained. For a stateful caching and scheduling system, this leaves substantial uncertainty about sensitivity to request order, initialization, and longer-running traffic.

The latency trade-off also requires clearer characterization. Offline LongBench shows a 9.8–10.0× increase in mean TTFT, although end-to-end latency decreases. This is not a correctness problem, and the paper explicitly optimizes mean request completion latency rather than imposing a TTFT constraint. Nevertheless, the result shows that the chosen schedule can exchange a large initial delay for more efficient subsequent generation. With fixed window settings and limited workload coverage, the paper does not yet establish when this exchange remains beneficial as output length, concurrency, or cache capacity changes.

The reported gains may be real for the measured traces, but they do not yet provide a robust characterization of the proposed system’s advantage.

Questions for authors

Q1. About baselines and contribution attribution.

What exactly does “Native” in Table 1 include? In particular, does it already use TEMPER’s persistent expert-granularity cache, eviction policy, memory budget, and admission limits? What changes when “Communication” is enabled beyond transfer coalescing and asynchronous loading? A comparison that keeps an expert-granularity cache and resource budget fixed while isolating communication overlap, coalescing, and windowed scheduling would directly affect my assessment. Please also explain which benefit is not already obtained by an ordinary prefetching and layer-major multi-batch baseline, given the closest related work.

Q2. About robustness and latency trade-off.

Can the shared-arrival comparison be repeated with more measured requests and independent request/arrival sequences, reporting run-level uncertainty and stating how cache and placement state are initialized? Within the existing platform, a focused comparison of TTFT, TPOT, and end-to-end latency at a few shared offered loads or output lengths would be particularly informative. How does the windowed scheduler behave when insufficient work is ready to form the intended cohort, and what prevents waiting for reuse opportunities from dominating the request’s completion time?

Q3. About replication.

Table 2 reports few new H2D replica copies, while Table 1 shows essentially no MMLU gain and an 11.3% LongBench gain from enabling replication. I do not interpret low copy counts as proof that replication is ineffective, since designated locations and resident weights are distinct. However, how many subsequent invocations actually execute at the additional locations, and how much exposed loading time or critical-rank execution time do they save beyond predictive prefetching?

For accepted placement updates, how well does estimated net benefit predict measured benefit? Appendix B.1 notes that the single-expert copy estimate excludes full distributed directory-commit latency; please clarify how that additional cost is accounted for in admission or hidden during execution. This would help establish whether the cost-based replication policy contributes reliably, rather than only in the reported LongBench configuration.

Reviewer_EppAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TEMPER targets distributed MoE inference when expert weights exceed GPU memory. It overlaps token exchange with expert loading, replicates hot experts for load balancing, and uses windowed layer-major execution to reuse weights across scheduling turns. On Qwen3-30B-A3B with four RTX 3080 Ti GPUs, it reports 76.8–82.0% lower mean end-to-end latency than vLLM and KTransformers baselines under shared arrival traces. Ablations attribute most gains to communication changes and execution reordering, with replication providing an additional benefit on LongBench.

Critical strengths
  1. The system addresses a practical deployment constraint. It jointly handles inter-GPU token exchange and host-to-GPU weight transfers on GPUs that cannot hold the full model.

  2. Communication and reuse are coordinated effectively. Destination-level aggregation reduces repeated token transfers, while layer-major execution increases measured expert reuse by 2.13×.

  3. The latency gains are substantial under identical arrivals. The shared-trace comparison strengthens the evidence beyond system-specific normalized loads, and the paper openly reports the first-token latency trade-off.

Critical weaknesses
  1. The advance over closely related schedulers needs clearer justification. For instance, MoE-Lightning and Klotski already amortize offloading through multi-batch execution and transfer overlap. Lina addresses distributed MoE scheduling and load balancing. TEMPER combines these directions under offloaded EP, but does not clearly establish which scheduling mechanism makes this combination technically distinct (MoE-Lightning, Klotski, Lina).

  2. Dispatch does not model its own communication savings. Equation (4) charges remote cost per expert invocation, but Section 4.1 transfers one activation per token–destination pair. Assigning another selected expert to an already-used destination can therefore be cheaper than the model predicts. This disconnect may undermine the benefit of destination coalescing.

  3. Section 4.3 explains execution within a cohort but leaves cohort formation and adaptation to sparse arrivals unclear. Fixed windows improve reuse while potentially delaying requests. The roughly 10× higher offline LongBench TTFT shows that this trade-off is substantial.

  4. Evaluation covers one model and one hardware setup, and the shared-rate result uses a single 16-request run. The closest offloading schedulers are absent from the comparisons. Replication adds no MMLU throughput and only three new H2D copies on LongBench, leaving this contribution weakly tested.

Questions for authors
  1. What scheduling capability distinguishes TEMPER from extending MoE-Lightning or Klotski with EP and replica-aware dispatch?

  2. How accurately does the replica-admission model predict realized savings after directory coordination, cache eviction, and actual dispatch?

  3. Why does dispatch charge communication per invocation rather than per token–destination pair?

  4. How are cohorts formed under sparse arrivals, and what limits waiting time or adapts W and k?

  5. Do the gains hold over longer shared-arrival traces and against the closest offloading schedulers?

Reviewer_1RHhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TEMPER accelerates distributed MoE inference on memory-constrained GPUs by overlapping token communication with expert loading, selectively replicating experts, and reordering execution to reuse loaded weights. On Qwen3-30B-A3B with four RTX 3080 Ti GPUs, it reports substantial latency reductions over vLLM and KTransformers baselines.

Critical strengths

The system coordinates communication, weight movement, and reuse around a relevant deployment constraint. The ablations demonstrate substantial benefits from communication overlap and windowed execution.

Shared-arrival comparisons complement normalized-load experiments, and the paper explicitly reports the trade-off between generation efficiency and first-token latency.

Critical weaknesses

Evaluation is narrow: one model, one GPU configuration, and only 16 requests per online run. The shared-rate results use a single run, providing limited evidence about sustained throughput and tail latency.

Comparisons omit closely related offloading systems discussed in the paper, such as MoE-Lightning and Klotski. Meanwhile, replication adds no measurable throughput on MMLU and 11.3% on LongBench, leaving its significance less established than the other components.

Questions for authors

Do the gains persist on longer, repeated shared-arrival traces, including p95/p99 TTFT and end-to-end latency?

Can you compare against a strong expert-reuse baseline and explain when replication provides benefits beyond prefetching and windowed execution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.