- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper looks at prefetching MoE expert weights from HBM into on-package cache during small-batch decode. It focuses on the AMD MI355X, where each XCD has a private L2 and all dies share a 256 MB MALL. SEP trains a small predictor per layer that reads the pre-attention hidden state and scores the experts each GPU owns. It then issues load-only prefetches for the top of them while attention, the collective, and routing are still running.
The authors argue that two budgets govern the design. The first is how many experts fit in a given cache tier (Table 2). This motivates prefetching more experts than the router selects and measuring coverage on the busiest GPU. The second is how much time is available before the expert GEMM starts, (Eq. 1). They compare 18 predictor architectures on four MoE models and report 98.5–100% coverage on 22 held-out datasets. On AITER kernels they measure 1.3–1.8× decode-GEMM speedups at batch 1.
Critical strengths
- The space and time budgets in Section 2.2 are the part of the paper I found most useful. Tables 2 and 4 give per-model numbers rather than a general argument. They also surface a result I would not have guessed: for Mixtral, the transfer alone takes 115% of at the cache-fit budget, so predictor capacity has to be given up to keep coverage.
- Measuring coverage on the busiest GPU instead of exact top- accuracy makes sense under expert parallelism (Section 3.4, Figure 6). It also explains why a linear predictor gets within two points of the best SiLU-MLP on GPT-OSS (96.9% vs. 98.5%, Figure 4). The experiments are well controlled. Figure 5 includes both an oracle and a random-expert control, Table 5 reports seed variance, and the authors are upfront about cases where prediction adds little, such as Mixtral at batch 8.
- The implementation details in Section 3.5 reflect real engineering effort. Three details here are easy to get wrong: routing prefetches through the GEMM’s XCD swizzle, keeping the XCD mapping consistent across streams, and choosing cache modifiers so the lines are retained. Building on AITER kernels also makes the GEMM-level numbers believable.
Critical weaknesses
- My main concern is that none of the speedups are end to end. Section 4 measures the MoE GEMM alone on one GPU. Yet the premise of SEP is that the prefetch runs alongside attention and the collective, which also read from HBM and can evict prefetched lines, as Section 3.5 acknowledges. Even with perfect overlap, the paper’s own estimate that experts take 30–70% of decode time (Section 2.1) caps a 1.46× GEMM speedup at roughly 1.10–1.28× overall. I could not find a TPOT number anywhere.
- The operating points behind the headline numbers don’t match the reported accuracies. The 1.78× Mixtral result uses the linear predictor at , but Table 3 only gives Mixtral accuracy at , and Figure 7 suggests linear coverage is closer to 80–90% at . The Qwen3-Next numbers also differ between Tables 3 and 5. Table 3 includes a DeepSeek L2 point even though Table 2 says no DeepSeek expert fits in L2, and coverage at batch 8 is never given.
- I’m not sure busiest-GPU coverage is the right proxy for latency, and the cost side is left unaccounted. The layer finishes when the slowest GPU finishes. A GPU with one missed expert can easily be slower than a busier one whose experts all hit in cache, and the paper doesn’t check this. Every GPU also prefetches experts and appears to run the full predictor, so total HBM traffic and energy across the node probably go up, but this is never measured.
- The methodological novelty is modest: the pre-attention predictor and its best losses come from Zhu et al. (2025), and staging weights into cache during communication comes from PRESERVE. A more serious gap is that the paper neither cites nor compares against Libra (Yang et al., ICLR 2026), which predicts expert activations without any training by running the next layer’s gating function on the current layer’s hidden states. Although Libra was built for load balancing rather than cache prefetching, its predictor applies directly here, costs almost nothing, and gives a full layer of lookahead that could relieve exactly the time constraint limiting Mixtral. SEP’s coverage and speedup should be reported against it.
Questions for authors
- Could you report end-to-end TPOT in vLLM with and without SEP at batch 1 and 8? It would also help to see attention and all-reduce times measured while the prefetch is running concurrently.
- What coverage do the predictors achieve at the exact operating points used for the Figure 5 headlines? Please also explain the Qwen3-Next discrepancy between Tables 3 and 5, and what the DeepSeek L2 entry in Table 3 represents.
- How does SEP’s predictor compare with Libra’s training-free lookahead gating (Yang et al., ICLR 2026) and with previous-token routing reuse, in both coverage and speedup?
- How much total HBM traffic does SEP add per decode step across the node, counting predictor weights and all prefetches? In a full EP=8 run, does busiest-GPU coverage actually track the slowest GPU’s MoE latency?
3 comments
Thank you for the response. The concurrency measurements and the clarification on per-rank max latency address part of my concerns. Two points remain:
- end-to-end comparison: The end-to-end benefit remains unclear. A TPOT comparison with SEP on and off in vLLM or SGLang, at batch 1 and 8, would be sufficient.
- lookahead prediction baseline: My question is whether Libra’s training-free, one-layer lookahead can relax timing limit, given that you estimate its cost is similar to the linear predictor. Previous-token routing reuse, a zero-cost baseline, is also not addressed. Please report coverage and speedup for both.
I would be willing to raise my score if they are resolved.
Author Response to Reviewer ze2g
We thank the reviewer for the two remaining points. Both are addressed below.
1. End-to-end decoding performance
To address this, we implemented the GPT-OSS decoding pipeline using the same AITER kernels as vLLM and RCCL for communication. We implemented both a baseline without prefetching and with SEP on eight MI355X GPUs, running all 36 layers. Our baseline has a decoding latency within 5% of vLLM.
We run 22 consecutive decoding steps using expert selections recorded from real decoding traces. Each layer has its own weights and KV caches. There is no artificial cache priming or flushing between layers or tokens. This allows us to account for both resource contention and cache interference across layers.
We observe that SEP provides approximately 7% lower TPOT at batch 1 and 5.1–5.5% at batch 8. Our additional experiments with QKV weight restoration, router tuning, and CU allocation suggest that there is still room for improvement.
Note that the fraction of time spent in MoE GEMMs is smaller at batch 8, which can explain the lower overall speedup despite higher GEMM speedups. We expect the benefit of SEP to increase for models where expert GEMMs are more expensive and as attention and communication kernels are further optimized.
We chose to implement the pipeline outside vLLM because of several issues we encountered with concurrent kernel execution. In ROCm, different HIP streams can be assigned to the same hardware queue, resulting in sequential execution. We also encountered issues with CU masks during graph replay and additional synchronization overhead between streams.
By implementing the pipeline ourselves, we can control these resources while still using the same production kernels and communication library. We believe these experiments answer the reviewer's question about the decoding performance of SEP. We expect the vLLM integration to be feasible and plan to work on it for the published version.
2. Lookahead and previous-token reuse
We re-evaluated Libra’s accuracy, then measured the speedup of issuing the prefetch when that lookahead would issue it. We evaluated Libra on prompts drawn at random from our dataset. Its coverage is 88%, compared with 96.9% for our linear predictor. The cost is comparable, but the answer is available one layer earlier, and that is the wrong time to prefetch.
The speedup measurement below uses perfect prediction, so the gap is only from eviction. Relative to our launch with the linear predictor, a prefetch keeps:
| When the prefetch starts | Speedup kept vs SEP |
|---|---|
| Start of the layer through its attention | 91-92% |
| Right after the previous layer’s GEMMs | 90% |
| One layer ahead | 61-65% |
Even if the experts were predicted perfectly, prefetching at any time earlier than our proposed launch is still slower: 90-92% of the speedup just before that launch, and 61-65% one layer ahead, because of cache eviction. Consequently, Libra is strictly slower than SEP, as it combines this eviction ceiling with lower prediction coverage. Our method, with the linear predictor, is faster even than a perfect prefetch issued any earlier.
Previous-token reuse, on the same layer, is not a workable baseline. On GPT-OSS at the paper’s operating point (EP=8, 2 experts cached per GPU), its accuracy is 26.4%. At that accuracy the prefetched set misses most of the experts that actually run, yielding virtually no speedup over the unprefetched baseline.
We thank the reviewer again for the feedback and hope that these new experiments address the concerns.
Author Response to Reviewer ze2g
We thank the reviewer for the feedback and for recognizing SEP's hardware-algorithm co-design contributions. We provide additional results and clarifications below.
1. Contention and layer speedups
Figure 5 measures GEMM acceleration, while Table 4 evaluates whether prediction and prefetching fit within the overlap window. To address resource contention, we conducted concurrent-execution experiments using production AMD AITER kernels.
Low-batch inference often underutilizes available CUs, allowing SEP's predictors and prefetch kernels to execute on otherwise idle compute resources. Underutilization also creates opportunities to exploit available memory bandwidth. Multi-GPU communication involves interconnect transfers and synchronization, rather than exclusively consuming HBM bandwidth. These different resource demands enable concurrent execution with limited contention.
Full-model TPOT depends on attention architecture, context length, and communication overhead, making SEP's contribution difficult to isolate end-to-end. Our experiments show up to ~7% lower per-layer latency, depending on the layer and per-GPU expert selection. As a control, prefetching incorrect experts instead increases layer latency by 0.8–2.5%.
These results support SEP's low-contention overlap. We note that kernel dispatch and scheduling latency are hardware-dependent and should also be considered. We will include the measurements and methodology.
2. Operating-point clarification
Table 3 reports cache-capacity operating points, whereas Figure 5 additionally enforces the timing budget. Mixtral's 1.78× speedup uses the linear predictor at Npf=2, rather than SiLU-MLP at Npf=3, demonstrating the importance of jointly considering accuracy and latency. We will clarify these configurations and report their coverage.
Table 3 uses the final checkpoint evaluated on all 22 held-out datasets, while Table 5 uses an earlier predictor. We will update Table 5; the reported latency results have been verified.
For DeepSeek, although its 44 MB experts exceed aggregate L2 capacity (32 MB), portions can still reside in L2, with MALL providing additional capacity. Our latency experiments account for this distinction, which we will clarify.
3. Busiest-GPU coverage versus execution latency
Busiest-GPU coverage measures prediction quality, not execution latency. We simulate all eight ranks individually on MI355X using production AITER kernels. For baseline and SEP, we independently take the maximum latency across ranks. Thus, reported speedups account for the slowest GPU, which need not be the busiest. We will clarify this in Section 4.
4. Cross-layer prediction
We thank the reviewer for highlighting Libra and will discuss it in Section 6.2. We already investigate cross-layer prediction through FATE (Fang et al., 2025a). Our lightweight linear predictor achieves 82.5% recall@4, versus 78.7% for the substantially larger FATE-style cross-layer MLP, consistent with Zhu et al. (2025).
Libra reuses the next layer's router on an earlier hidden state without additional training. Although training-free, it requires another projection with arithmetic complexity comparable to our linear predictor. Earlier lookahead also increases the risk of cache eviction by intervening kernels. We plan to investigate contention from earlier prefetching.
Methodological novelty
We acknowledge that expert prediction is established prior work. However, Zhu et al. (2025) evaluate limited architectures and settings without systematically examining generalization or hardware deployment constraints. SEP evaluates 18 architectures, four MoE models, and 22 datasets, characterizing how predictor accuracy, complexity, latency, cache capacity, and prefetch budgets interact. We demonstrate that predictor selection is hardware-dependent.
PRESERVE addresses HBM-to-cache prefetching for deterministically known tensors through graph-level scheduling, but not dynamically routed MoE experts. SEP develops the necessary kernel-level mechanisms: production GEMM-matched tiling, physical XCD placement across streams, prefetch ordering, and cache-retention control.
Our central contribution is the joint hardware-algorithm co-design: selecting predictors, determining feasible prefetch budgets under cache and timing constraints, and retaining weights for subsequent GEMMs. Neither existing prediction methods nor PRESERVE addresses these challenges together. SEP therefore represents a substantial ML systems contribution beyond combining prior techniques.
We thank the reviewer for the thoughtful observations and would appreciate reconsideration of the score.