acceptodds
Under review as a conference paper at ICLR 2027

SEP: Speculative Expert Prefetching for Low-Latency Mixture-of-Experts Inference

Abstract

Mixture-of-experts (MoE) decode is memory-bandwidth bound: each token generation step must stream expert weight matrices from HBM before the expert GEMM can execute. We introduce Speculative Expert Prefetching (SEP), which predicts each token's active experts from the pre-attention hidden state and prefetches expert weights overlapped with attention, routing, and multi-GPU communication. SEP rests on a hardware-algorithm co-design insight, that on-package cache holds more experts than the router's per-GPU top-, enabling overprovisioned prefetching that relaxes the prediction objective from exact expert prediction to per-GPU coverage. When this overprovisioning is large, spare cache slots tolerate mispredictions and a lightweight predictor suffices, while tight cache capacity demands higher accuracy. A second limit is temporal, as the predictor and the expert weight transfer must together complete before the expert GEMM begins. We evaluate 18 predictor architectures across four modern MoE models, demonstrating strong out-of-distribution generalization across diverse task categories. Our best predictor achieves - accuracy in predicting experts, translating to - decode-GEMM speedups at batch 1 on production MoE kernels.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Speculative Expert Prefetching, MoE, Cache Prefetching, Expert Predictors, Low-Latency Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ze2gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at prefetching MoE expert weights from HBM into on-package cache during small-batch decode. It focuses on the AMD MI355X, where each XCD has a private L2 and all dies share a 256 MB MALL. SEP trains a small predictor per layer that reads the pre-attention hidden state and scores the experts each GPU owns. It then issues load-only prefetches for the top NpfN_{\text{pf}} of them while attention, the collective, and routing are still running.

The authors argue that two budgets govern the design. The first is how many experts fit in a given cache tier (Table 2). This motivates prefetching more experts than the router selects and measuring coverage on the busiest GPU. The second is how much time is available before the expert GEMM starts, Tpredictor+Ttransfer≤ΔtT_{\text{predictor}} + T_{\text{transfer}} \le \Delta t (Eq. 1). They compare 18 predictor architectures on four MoE models and report 98.5–100% coverage on 22 held-out datasets. On AITER kernels they measure 1.3–1.8× decode-GEMM speedups at batch 1.

Critical strengths
  1. The space and time budgets in Section 2.2 are the part of the paper I found most useful. Tables 2 and 4 give per-model numbers rather than a general argument. They also surface a result I would not have guessed: for Mixtral, the transfer alone takes 115% of Δt\Delta t at the cache-fit budget, so predictor capacity has to be given up to keep coverage.
  2. Measuring coverage on the busiest GPU instead of exact top-kk accuracy makes sense under expert parallelism (Section 3.4, Figure 6). It also explains why a linear predictor gets within two points of the best SiLU-MLP on GPT-OSS (96.9% vs. 98.5%, Figure 4). The experiments are well controlled. Figure 5 includes both an oracle and a random-expert control, Table 5 reports seed variance, and the authors are upfront about cases where prediction adds little, such as Mixtral at batch 8.
  3. The implementation details in Section 3.5 reflect real engineering effort. Three details here are easy to get wrong: routing prefetches through the GEMM’s XCD swizzle, keeping the XCD mapping consistent across streams, and choosing cache modifiers so the lines are retained. Building on AITER kernels also makes the GEMM-level numbers believable.
Critical weaknesses
  1. My main concern is that none of the speedups are end to end. Section 4 measures the MoE GEMM alone on one GPU. Yet the premise of SEP is that the prefetch runs alongside attention and the collective, which also read from HBM and can evict prefetched lines, as Section 3.5 acknowledges. Even with perfect overlap, the paper’s own estimate that experts take 30–70% of decode time (Section 2.1) caps a 1.46× GEMM speedup at roughly 1.10–1.28× overall. I could not find a TPOT number anywhere.
  2. The operating points behind the headline numbers don’t match the reported accuracies. The 1.78× Mixtral result uses the linear predictor at Npf=2N_{\text{pf}}=2, but Table 3 only gives Mixtral accuracy at Npf=3N_{\text{pf}}=3, and Figure 7 suggests linear coverage is closer to 80–90% at N=2N=2. The Qwen3-Next numbers also differ between Tables 3 and 5. Table 3 includes a DeepSeek L2 point even though Table 2 says no DeepSeek expert fits in L2, and coverage at batch 8 is never given.
  3. I’m not sure busiest-GPU coverage is the right proxy for latency, and the cost side is left unaccounted. The layer finishes when the slowest GPU finishes. A GPU with one missed expert can easily be slower than a busier one whose experts all hit in cache, and the paper doesn’t check this. Every GPU also prefetches NpfN_{\text{pf}} experts and appears to run the full predictor, so total HBM traffic and energy across the node probably go up, but this is never measured.
  4. The methodological novelty is modest: the pre-attention predictor and its best losses come from Zhu et al. (2025), and staging weights into cache during communication comes from PRESERVE. A more serious gap is that the paper neither cites nor compares against Libra (Yang et al., ICLR 2026), which predicts expert activations without any training by running the next layer’s gating function on the current layer’s hidden states. Although Libra was built for load balancing rather than cache prefetching, its predictor applies directly here, costs almost nothing, and gives a full layer of lookahead that could relieve exactly the time constraint limiting Mixtral. SEP’s coverage and speedup should be reported against it.
Questions for authors
  1. Could you report end-to-end TPOT in vLLM with and without SEP at batch 1 and 8? It would also help to see attention and all-reduce times measured while the prefetch is running concurrently.
  2. What coverage do the predictors achieve at the exact operating points used for the Figure 5 headlines? Please also explain the Qwen3-Next discrepancy between Tables 3 and 5, and what the DeepSeek L2 entry in Table 3 represents.
  3. How does SEP’s predictor compare with Libra’s training-free lookahead gating (Yang et al., ICLR 2026) and with previous-token routing reuse, in both coverage and speedup?
  4. How much total HBM traffic does SEP add per decode step across the node, counting predictor weights and all prefetches? In a full EP=8 run, does busiest-GPU coverage actually track the slowest GPU’s MoE latency?
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer ze2g

We thank the reviewer for the feedback and for recognizing SEP's hardware-algorithm co-design contributions. We provide additional results and clarifications below.

1. Contention and layer speedups

Figure 5 measures GEMM acceleration, while Table 4 evaluates whether prediction and prefetching fit within the overlap window. To address resource contention, we conducted concurrent-execution experiments using production AMD AITER kernels.

Concurrent operationsBatch 1 slowdownBatch 8 slowdown
GPT-OSS: Predictor scoring + attention≤1%≤1%
GPT-OSS: Prefetch + all-reduce≤3%≤2%
GPT-OSS: Prefetch + normalization≤3%≤1%
GPT-OSS: Prefetch + router0–2%—
DeepSeek: Predictor + MLA attention2%2%

Low-batch inference often underutilizes available CUs, allowing SEP's predictors and prefetch kernels to execute on otherwise idle compute resources. Underutilization also creates opportunities to exploit available memory bandwidth. Multi-GPU communication involves interconnect transfers and synchronization, rather than exclusively consuming HBM bandwidth. These different resource demands enable concurrent execution with limited contention.

Full-model TPOT depends on attention architecture, context length, and communication overhead, making SEP's contribution difficult to isolate end-to-end. Our experiments show up to ~7% lower per-layer latency, depending on the layer and per-GPU expert selection. As a control, prefetching incorrect experts instead increases layer latency by 0.8–2.5%.

These results support SEP's low-contention overlap. We note that kernel dispatch and scheduling latency are hardware-dependent and should also be considered. We will include the measurements and methodology.

2. Operating-point clarification

Table 3 reports cache-capacity operating points, whereas Figure 5 additionally enforces the timing budget. Mixtral's 1.78× speedup uses the linear predictor at Npf=2, rather than SiLU-MLP at Npf=3, demonstrating the importance of jointly considering accuracy and latency. We will clarify these configurations and report their coverage.

Table 3 uses the final checkpoint evaluated on all 22 held-out datasets, while Table 5 uses an earlier predictor. We will update Table 5; the reported latency results have been verified.

For DeepSeek, although its 44 MB experts exceed aggregate L2 capacity (32 MB), portions can still reside in L2, with MALL providing additional capacity. Our latency experiments account for this distinction, which we will clarify.

3. Busiest-GPU coverage versus execution latency

Busiest-GPU coverage measures prediction quality, not execution latency. We simulate all eight ranks individually on MI355X using production AITER kernels. For baseline and SEP, we independently take the maximum latency across ranks. Thus, reported speedups account for the slowest GPU, which need not be the busiest. We will clarify this in Section 4.

4. Cross-layer prediction

We thank the reviewer for highlighting Libra and will discuss it in Section 6.2. We already investigate cross-layer prediction through FATE (Fang et al., 2025a). Our lightweight linear predictor achieves 82.5% recall@4, versus 78.7% for the substantially larger FATE-style cross-layer MLP, consistent with Zhu et al. (2025).

Libra reuses the next layer's router on an earlier hidden state without additional training. Although training-free, it requires another projection with arithmetic complexity comparable to our linear predictor. Earlier lookahead also increases the risk of cache eviction by intervening kernels. We plan to investigate contention from earlier prefetching.

Methodological novelty

We acknowledge that expert prediction is established prior work. However, Zhu et al. (2025) evaluate limited architectures and settings without systematically examining generalization or hardware deployment constraints. SEP evaluates 18 architectures, four MoE models, and 22 datasets, characterizing how predictor accuracy, complexity, latency, cache capacity, and prefetch budgets interact. We demonstrate that predictor selection is hardware-dependent.

PRESERVE addresses HBM-to-cache prefetching for deterministically known tensors through graph-level scheduling, but not dynamically routed MoE experts. SEP develops the necessary kernel-level mechanisms: production GEMM-matched tiling, physical XCD placement across streams, prefetch ordering, and cache-retention control.

Our central contribution is the joint hardware-algorithm co-design: selecting predictors, determining feasible prefetch budgets under cache and timing constraints, and retaining weights for subsequent GEMMs. Neither existing prediction methods nor PRESERVE addresses these challenges together. SEP therefore represents a substantial ML systems contribution beyond combining prior techniques.

We thank the reviewer for the thoughtful observations and would appreciate reconsideration of the score.

Thank you for the response. The concurrency measurements and the clarification on per-rank max latency address part of my concerns. Two points remain:

  1. end-to-end comparison: The end-to-end benefit remains unclear. A TPOT comparison with SEP on and off in vLLM or SGLang, at batch 1 and 8, would be sufficient.
  2. lookahead prediction baseline: My question is whether Libra’s training-free, one-layer lookahead can relax timing limit, given that you estimate its cost is similar to the linear predictor. Previous-token routing reuse, a zero-cost baseline, is also not addressed. Please report coverage and speedup for both.

I would be willing to raise my score if they are resolved.

AuthorsOpenReview ↗

Author Response to Reviewer ze2g

We thank the reviewer for the two remaining points. Both are addressed below.

1. End-to-end decoding performance

To address this, we implemented the GPT-OSS decoding pipeline using the same AITER kernels as vLLM and RCCL for communication. We implemented both a baseline without prefetching and with SEP on eight MI355X GPUs, running all 36 layers. Our baseline has a decoding latency within 5% of vLLM.

We run 22 consecutive decoding steps using expert selections recorded from real decoding traces. Each layer has its own weights and KV caches. There is no artificial cache priming or flushing between layers or tokens. This allows us to account for both resource contention and cache interference across layers.

We observe that SEP provides approximately 7% lower TPOT at batch 1 and 5.1–5.5% at batch 8. Our additional experiments with QKV weight restoration, router tuning, and CU allocation suggest that there is still room for improvement.

Note that the fraction of time spent in MoE GEMMs is smaller at batch 8, which can explain the lower overall speedup despite higher GEMM speedups. We expect the benefit of SEP to increase for models where expert GEMMs are more expensive and as attention and communication kernels are further optimized.

We chose to implement the pipeline outside vLLM because of several issues we encountered with concurrent kernel execution. In ROCm, different HIP streams can be assigned to the same hardware queue, resulting in sequential execution. We also encountered issues with CU masks during graph replay and additional synchronization overhead between streams.

By implementing the pipeline ourselves, we can control these resources while still using the same production kernels and communication library. We believe these experiments answer the reviewer's question about the decoding performance of SEP. We expect the vLLM integration to be feasible and plan to work on it for the published version.

2. Lookahead and previous-token reuse

We re-evaluated Libra’s accuracy, then measured the speedup of issuing the prefetch when that lookahead would issue it. We evaluated Libra on prompts drawn at random from our dataset. Its coverage is 88%, compared with 96.9% for our linear predictor. The cost is comparable, but the answer is available one layer earlier, and that is the wrong time to prefetch.

The speedup measurement below uses perfect prediction, so the gap is only from eviction. Relative to our launch with the linear predictor, a prefetch keeps:

When the prefetch startsSpeedup kept vs SEP
Start of the layer through its attention91-92%
Right after the previous layer’s GEMMs90%
One layer ahead61-65%

Even if the experts were predicted perfectly, prefetching at any time earlier than our proposed launch is still slower: 90-92% of the speedup just before that launch, and 61-65% one layer ahead, because of cache eviction. Consequently, Libra is strictly slower than SEP, as it combines this eviction ceiling with lower prediction coverage. Our method, with the linear predictor, is faster even than a perfect prefetch issued any earlier.

Previous-token reuse, on the same layer, is not a workable baseline. On GPT-OSS at the paper’s operating point (EP=8, 2 experts cached per GPU), its accuracy is 26.4%. At that accuracy the prefetched set misses most of the experts that actually run, yielding virtually no speedup over the unprefetched baseline.

We thank the reviewer again for the feedback and hope that these new experiments address the concerns.

Reviewer_pseYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

MoE LLMs are known for its large memory footprint. At inference time, each expert needs a separate execution, and the HBM bandwidth can become a major bottleneck even if all model weights already reside in the GPU HBM. To mitigate the bandwidth congestion when the batch size is small, the authors propose Speculative Expert Prefetching (SEP), a method that predict the expert weights before the router with the pre-attention token embedding, and then prefetch the predicted expert into the L2 Cache or MALL. The authors train 18 predictor configurations per model on a 42-dataset corpus and evaluate the zero-shot OOD generalization score across four MoE models and reported a high OOD coverage on cache fitting. The authors further report a better batch-1 MoE decode GEMM speedup.

Critical strengths
  • Software-hardware co-design. The authors propose an algorithm grounded in hardware characteristics.
  • Broad predictor evaluation. The study compares 18 predictor configurations across four MoE models and evaluates generalization on held-out datasets.
  • The manuscript is clearly written in most places and is well organized (see caveats below). Figures 1 and 2 clearly illustrate the overall idea.
Critical weaknesses

Soundness of Major experiments:

  • Instead of performing an end-to-end experiment, the authors proposed to use isolated MoE GEMM microbenchmarks as the major experiment to justify the benefit of the proposed algorithm. No TPOT, no throughput. And the choice of microbenchmark can be problematic for the following reasons:
    • (a) The prefetch pass, IIUC is loaded before starting the MoE GEMM timer. Suppose the cases where the Transfer is longer than the Attn + Router + Comm, in this case such additional latency will be generally ignored. And in Table 4 there do exist such cases on Mixtral model.
    • (b) The main stream may also need L2 cache to execute the Attn/Router/Comm. How can we know that the L2 cache is safe enough to use for prefetching? Won't they compete for the bandwidth? Attn itself is also a bandwidth-constrained kernel, no?
    • (c) How important is the kernel speedup? The TPOT improvement can be very limited and this goes untested.
    • (d) The predictor itself also occupies memory and computation, but the cost of the predictor got ignored. With h=2048, on some models the predictor itself is larger than an expert.

Incorrect depiction about the prior literature. The authors mentioned that prior works "target single-GPU offload setups", which is basically not true -- the very initial works and most of the works cited are focusing on multi-GPU cases already.

No code has been provided alongside the paper.

Section 3.5 contains many technical terms specific to AMD GPUs. For readers unfamiliar with AMD GPU architecture, who I believe constitute a large proportion of the ICLR audience, this section is difficult to follow, and no additional explanation is provided.

Questions for authors

See above for major questions regarding the experiments

  • Line 073 claims that "Cache capacity limits overprovisioning, reframing prediction from exact top-k matching to per-GPU coverage and ultimately setting the prefetcher's training objective." However, I do not see how the training objective depends on the expert-parallelism degree or the number of GPUs.
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer pseY

We thank the reviewer for the constructive feedback. We acknowledge areas for improvement but would also like to clarify several concerns below.

1. Contention and experimental methodology

We disagree that predictor and prefetch overheads are ignored. Eq. 1 requires both operations to fit within the overlap window, and Table 4 reports their measured times. For Mixtral, the more accurate SiLU-MLP cannot meet this constraint, so we deploy the linear predictor. Both latencies are explicitly considered when selecting deployable configurations. On Qwen3-Next, an h=2048 predictor exceeds a single 6 MB expert, but only one predictor is stored per layer rather than per expert, and our linear predictor is considerably smaller.

We overlap prefetching only with router/COMM, not attention, to minimize contention. We will clarify this in Section 3.

However, we agree that concurrent resource contention is crucial and must also be considered.

We conducted additional concurrent-execution experiments using production AMD AITER kernels to evaluate contention during SEP's overlap window.

Concurrent operationsBatch 1 slowdownBatch 8 slowdown
GPT-OSS: Predictor scoring + attention≤1%≤1%
GPT-OSS: Prefetch + all-reduce≤3%≤2%
GPT-OSS: Prefetch + normalization≤3%≤1%
GPT-OSS: Prefetch + router0–2%—
DeepSeek: Predictor + MLA attention2%2%

The limited contention can be explained by several hardware characteristics. First, low-batch inference kernels often underutilize available CUs, allowing SEP's lightweight predictors (both linear and SiLU-MLP) and prefetch kernels to execute on otherwise idle compute resources. Underutilized CUs also lead to low effective bandwidth, enabling BW to be used for prefetching. Second, multi-GPU communication involves interconnect transfers and synchronization, rather than exclusively competing for the HBM bandwidth used by expert-weight prefetching. Although these operations can share memory-system resources, their different bottlenecks create opportunities for concurrent execution.

Full-model TPOT depends on attention architecture, context length, and communication overhead, making it difficult to isolate SEP's contribution in an end-to-end measurement. Our experiments show up to ~7% lower per-layer latency, with benefits depending on the layer and the number of experts selected per GPU per layer. As a control, if we always prefetch the wrong experts, it results in 0.8-2.5% higher layer latency.

These results support low-contention overlap. Hardware-dependent dispatch and scheduling latency should also be considered. We will include the measurements and methodology.

2. Prior work

We agree that prior expert-offloading systems are not restricted to single-GPU deployments and will correct this wording. However, SEP addresses a different problem. Prior MoE expert-prefetching systems primarily transfer weights from CPU memory or storage into GPU HBM. SEP targets multi-GPU deployments where experts already reside in HBM, prefetching weights into on-package L2/MALL.

To our knowledge, SEP is the first system addressing speculative HBM-to-on-package-cache expert prefetching for multi-GPU MoE inference. PRESERVE explores HBM-to-cache prefetching for known tensors, but not MoE models. SEP introduces the necessary prediction and hardware mechanisms, including GEMM-matched tiling, physical XCD placement across streams, prefetch ordering, and cache-retention control. These are distinct challenges from existing expert-offloading systems.

3. Implementation details and reproducibility

We agree that Section 3.5 needs clearer AMD terminology and will add explanations and implementation details in the appendix. Evaluating SEP on AMD does not make the method AMD-specific. Our implementation targets this architecture, but the approach applies to other accelerators with suitable memory hierarchies, including NVIDIA GPUs. Cache capacities and hardware-specific optimizations might differ, but SEP's joint prediction, prefetch-budgeting, and overlap principles remain applicable.

4. Training objective

The training loss does not depend on EP. We use existing objectives, including KL divergence and weighted BCE. Instead, EP affects deployment: each GPU owns a subset of experts, and its cache capacity determines how many can be prefetched.

This changes how prediction quality is evaluated. Global top-k accuracy misses the benefit of overprovisioned per-GPU prefetching. We therefore use per-GPU coverage at the selected (EP, Npf) operating point. Our contribution is the joint selection of predictor architecture and prefetch budget under hardware constraints, not an EP-dependent training loss. We will clarify Line 73.

We thank the reviewer again. We hope these clarifications and results address the concerns. We believe SEP makes a substantial ML systems contribution and would appreciate reconsideration of the score.

Reviewer_1CwbOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Speculative Expert Prefetching (SEP) to reduce the memory-access latency of MoE decoding when all expert weights reside in GPU HBM. SEP predicts expert selections from each layer’s pre-attention hidden state and prefetches likely expert weights into L2 or last-level cache while attention, communication, and routing execute. Its central design principle is to jointly consider cache capacity, prediction coverage, and the time available for prediction and weight transfer.

The paper evaluates 18 predictor configurations across four MoE models, using a diverse training corpus and held-out datasets. It reports approximately 98.5–100% expert coverage at selected cache-capacity operating points and 1.3–1.8× isolated expert-GEMM speedups at batch size 1 on AMD MI355X hardware.

Critical strengths
  1. A relevant deployment problem with a clearly identified memory hierarchy: The paper focuses on HBM-to-cache prefetching for models whose experts already reside in GPU memory. This distinguishes its deployment setting from CPU-to-GPU expert offloading and addresses a practical bottleneck in low-batch MoE decoding.
  2. Useful analysis of the interaction between prediction quality and hardware constraints: The distinction between cache capacity and available overlap time is informative. Allowing additional prefetched experts makes coverage more relevant than exact expert-set prediction, while prediction latency competes with weight transfer for the same window. The Mixtral results provide a concrete example where a faster linear predictor enables a better deployable configuration than a more accurate MLP.
  3. Substantive implementation details beyond predictor design: The discussion of matching prefetch and compute tiling, aligning physical XCD placement across streams, ordering prefetches, and selecting cache-retention behavior addresses practical obstacles to obtaining cache hits. These details strengthen the systems contribution.
  4. Broad predictor evaluation and useful controls: The experiments cover four models with different attention architectures, multiple predictor families, held-out workloads, and per-layer behavior. Oracle and random-prefetch controls help distinguish the value of prediction from prefetching alone. The appendix also reports small variation across three predictor seeds, providing some evidence of reproducibility.
Critical weaknesses
  1. The central deployment benefit remains unverified end to end: The reported speedups isolate expert GEMMs under cold-cache and prefetched-cache conditions. The paper explicitly states this scope, but these measurements do not establish the reduction in full-model token latency. Predictors and prefetch operations can contend with attention and communication for bandwidth and execution resources, and prefetched weights may be evicted before use. Demonstrating that separately measured prediction and transfer times fit within an overlap window is useful, but insufficient to quantify these effects during concurrent execution. An integrated eight-GPU evaluation including all overheads is important for substantiating the practical benefit.
  2. Operating-point differences and reporting inconsistencies complicate interpretation: The headline coverage results do not always correspond to the configurations used for deployable speedups. For example, Mixtral’s 98.7% coverage uses a SiLU-MLP with three prefetched experts, while its batch-1 deployment requires a linear predictor with two. This distinction deserves greater prominence. In addition, Tables 3 and 5 report different Qwen3-Next coverage values despite both specifying four prefetched experts per GPU: 97.3% versus 96.02% for Linear, and 98.8% versus 98.37% for the best SiLU-MLP. The source of these differences is unclear.
  3. Some comparisons do not cleanly isolate the claimed source of improvement: Predictor families differ in both architecture and training objective, so the superiority of SiLU-MLP cannot be attributed to architecture alone. A controlled comparison using matched objectives would make the design recommendations more informative. The static-popularity comparison also needs clarification: it is described as selecting the top two experts per layer, whereas the main GPT-OSS configuration prefetches two experts per GPU. If these budgets differ, the reported accuracy gap would not isolate the benefit of input-dependent prediction.
  4. The generalization and deployment claims extend beyond the demonstrated evidence: The held-out corpus is a strength, but results are mostly aggregated rather than broken down by dataset or shift type. The relationship between training GovReport and evaluation GovReport-long also needs clarification before interpreting the evaluation as involving disjoint sources. Hardware experiments are limited to MI355X, and performance experiments cover batch sizes 1 and 8. Larger batches, continuous batching, and concurrent serving could substantially change expert working sets and cache behavior; the existing results already show that prediction provides little additional benefit over random prefetching for batch-8 Mixtral.
  5. Important reproducibility details are incomplete: The predictor descriptions and kernel implementation discussion are useful, but the paper does not fully specify training sample counts, optimization hyperparameters, or how candidate experts are aggregated across tokens for batched prefetching. Reporting absolute timing breakdowns and variability, alongside normalized overlap-window percentages, would also make the deployment analysis easier to reproduce.
Questions for authors
  1. Can you report end-to-end decode latency on the complete eight-GPU deployment, including prediction, prefetching, and concurrent resource contention? How do mean and tail token latencies compare with the unmodified serving baseline?
  2. What explains the Qwen3-Next differences between Tables 3 and 5? Can you report aggregate coverage at the exact predictor and prefetch-budget configurations used for each deployable speedup, particularly Mixtral at batch size 1?
  3. Does the static-popularity baseline receive the same per-GPU prefetch budget as SEP? Please clarify its expert-selection procedure and ensure the comparison uses the same coverage metric.
  4. How much of the SiLU-MLP advantage comes from the architecture versus weighted BCE? Could you provide a comparison with matched training objectives and specify the missing training and batched-candidate aggregation details?
  5. Are the training and evaluation documents in GovReport and GovReport-long disjoint? Can you provide coverage by dataset or shift category to show whether the aggregate results conceal difficult workloads?
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer 1Cwb

We thank the reviewer for the constructive feedback. We hope the following clarifications and additional analyses address the remaining concerns.

1. Contention and layer speedups

SEP targets memory-bound MoE GEMMs. Figure 5 measures GEMM acceleration, while Table 4 evaluates whether prediction and prefetching can fit within the overlap window.

We agree that resource contention must also be considered.

We conducted additional concurrent-execution experiments using production AITER kernels to quantify resource contention during SEP's overlap window.

Concurrent operationsBatch 1 slowdownBatch 8 slowdown
GPT-OSS: Predictor scoring + attention≤1%≤1%
GPT-OSS: Prefetch + all-reduce≤3%≤2%
GPT-OSS: Prefetch + normalization≤3%≤1%
GPT-OSS: Prefetch + router0–2%—
DeepSeek: Predictor + MLA attention2%2%

The limited contention can be explained by several hardware characteristics. First, low-batch inference kernels often underutilize available CUs, allowing SEP's lightweight predictors (both linear and SiLU-MLP) and prefetch kernels to execute on otherwise idle compute resources. Underutilized CUs also lead to low effective bandwidth, enabling BW to be used for prefetching. Second, multi-GPU communication involves interconnect transfers and synchronization, rather than exclusively competing for the HBM bandwidth used by expert-weight prefetching. Although these operations can share memory-system resources, their different bottlenecks create opportunities for concurrent execution.

Full-model TPOT depends on attention architecture, context length, and communication overhead, making it difficult to isolate SEP's contribution in an end-to-end measurement. Our experiments show up to ~7% lower per-layer latency, with benefits depending on the layer and the number of experts selected per GPU per layer. As a control, if we always prefetch the wrong experts, it results in 0.8-2.5% higher layer latency.

These results support the feasibility of overlapping SEP operations with the inference pipeline with limited contention. Kernel dispatch and scheduling latency may vary across hardware architectures and should be considered when implementing SEP. We will include the measurements and methodology.

2. Operating points and reporting inconsistencies

Table 3 reports cache-capacity operating points, while Figure 5 uses configurations that also satisfy the timing budget. For Mixtral, the 1.78× speedup uses the linear predictor at Npf=2 rather than the SiLU-MLP at Npf=3. We will clarify this distinction and report the corresponding coverage.

Table 3 reflects the final checkpoint evaluation across all 22 held-out datasets, whereas Table 5 used an earlier predictor. We will update Table 5 accordingly. We have verified that the reported latency results are correct.

3. Predictor comparisons and training objectives

Our experiments compare practical predictor configurations, including architectures and training objectives from prior work, rather than attributing improvements to architecture alone. Our main finding is that predictor selection must jointly consider prediction quality and hardware constraints. Mixtral illustrates this, as the faster linear predictor is preferable to the more accurate MLP under the available timing budget.

We also performed controlled loss ablations, keeping the architecture and training setup fixed, and the loss function has negligible effect on the accuracy.

We would also like to clarify that the static-popularity baseline uses the same per-GPU prefetch budget as SEP. Both prefetch two experts per GPU under EP=8 and use the same coverage metric. The baseline selects experts by their activation frequency, while SEP makes input-dependent predictions. We will correct the wording in Section 6.2.

4. Generalization and deployment scope

Our predictors are trained on 42 datasets and evaluated on 22 held-out datasets. We will include a breakdown by dataset or distribution-shift category in the appendix.

We use separate training and evaluation splits to avoid document overlap between GovReport and GovReport-long, and will clarify this procedure.

Our experiments focus on batch sizes 1 and 8, where memory-bound expert GEMMs are most relevant. We acknowledge that larger batches can reduce SEP's benefit, as demonstrated by Mixtral at batch 8. Although evaluated on AMD MI355X, SEP's underlying principles are applicable to other memory hierarchies, with hardware-specific implementations.

5. Reproducibility

We will expand the appendix with training sample counts, hyperparameters, and batched candidate-aggregation procedures to improve reproducibility.

We thank the reviewer again for the feedback. We hope these clarifications and additional results address the concerns and would appreciate reconsideration of the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of New Experiments

To the Area Chair and Reviewers,

We would like to briefly summarize the additional experiments we conducted in response to the reviews.

To address the end-to-end performance concerns raised by all three reviewers (1Cwb, pseY, ze2g), we implemented the complete 36-layer GPT-OSS decoding pipeline across eight MI355X GPUs using production AITER kernels and RCCL communication. We run 22 consecutive decoding steps without artificial cache priming or flushing. Our baseline decoding latency is within 5% of vLLM. Compared with this baseline, SEP reduces TPOT by approximately 7% at batch 1 and 5.1–5.5% at batch 8.

We also measured contention when running SEP's predictors and prefetch kernels concurrently with production attention, all-reduce, normalization, and routing kernels. Under the tested configurations, the slowdown of the overlapping kernels remains within 0–3% at both batch sizes. As a control, prefetching incorrect experts increases latency by 0.8–2.5%.

In response to ze2g, we evaluated Libra's training-free lookahead prediction and previous-token routing reuse. Libra achieves 88.0% coverage, compared with 96.9% for our linear predictor. Previous-token reuse achieves only 26.4% accuracy on GPT-OSS at EP=8 with two cached experts per GPU. We also measured the effect of earlier prefetching using perfect expert predictions to isolate cache eviction. Prefetching earlier in the same layer retains only 90–92% of SEP's speedup, while prefetching one layer ahead retains only 61–65%. These results show that prefetching earlier than our proposed approach leads to lower speedups, even with perfect expert prediction.

This comment summarizes the main new experimental results. Further details on the experimental setup, methodology, and measurements are provided in our individual responses to the reviewers. Those responses also address the remaining concerns, including operating-point inconsistencies, predictor comparisons, prior work, generalization, reproducibility, and novelty. We will incorporate these additional experiments and their methodology into the revised paper.

We thank the reviewers for their constructive feedback, which has helped us strengthen the paper. We hope that our additional experiments and clarifications address their concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.