acceptodds
Under review as a conference paper at ICLR 2027

ASYNCEP: Intra-layer Expert Asynchrony for MoE Serving with Imbalance Attention

Abstract

Mixture-of-experts (MoE) serving commonly deploys data and expert parallelism (DEP): attention replicas run distinct request batches while routed experts are sharded across an expert-parallel (EP) group. During prefill, attention replicas finish dispatch at different times, but synchronous EP delays expert feed-forward network (FFN) computation until routed inputs from all replicas are ready. Request schedulers seek to balance load while reusing the key-value (KV) cache of shared prompt prefixes to avoid redundant prefill computation. These goals can conflict when a replica holding a matching prefix is already overloaded, leaving residual attention imbalance. We present ASYNCEP, a distributed execution engine for MoE prefill. ASYNCEP proposes three mechanisms. Asynchronous EP allows expert computation to start before tokens from all attention replicas are ready. streamFFN batches ready tokens to balance early execution with FFN computation efficiency. Opportunistic expert weight fetching (OEWF) allows a faster replica to fetch expert weights and execute unstarted work from other GPUs. We evaluate ASYNCEP on DeepSeek-V4-Flash, DeepSeek-V4-Pro, and GLM-5.3, and our results show that ASYNCEP achieves up to 1.48× speedup in p95 time-to-first-token (TTFT) and improves the inference throughput by up to 1.17×.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM serving, Mixture-of-Experts, expert parallelism, GPU systems

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_TZMdOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents AsyncEP, a distributed execution engine for MoE prefill that addresses synchronization stalls caused by imbalanced attention workloads across DP replicas. It consists of three mechanisms: asynchronous EP, which starts expert computation as soon as inputs are ready; streamFFN, which uses threshold-based batching and multiple CUDA streams to maintain FFN efficiency; and opportunistic expert weight fetching (OEWF), which allows idle GPUs to take over part of the pending FFN work. The system is implemented in SGLang and evaluated on three MoE models, two workloads, and five scheduling policies. The results show up to 1.48× speedup in p95 TTFT and 1.17× improvement in throughput.

Critical strengths

S1. The paper addresses a practical bottleneck in MoE serving. The authors provide measurements of synchronization overhead and convincingly explain why request scheduling alone cannot fully resolve attention imbalance.

S2. The proposed mechanisms form a coherent system design with a nontrivial implementation. The evaluation covers multiple models, workloads, and scheduling policies, with solid ablation results supporting the reported performance gains.

Critical weaknesses

W1. The overhead of OEWF could be analyzed in more detail. Fetching expert weights introduces additional communication and temporary memory usage, and the decision to reassign work relies on runtime estimates. While the paper reports positive results, it would be useful to see how much time is spent fetching weights and how often the reassignment is actually performed. This would help clarify when OEWF is most effective.

W2. The experiments are limited to a single eight-GPU B200 node with NVLink and a fixed DP8+EP8 configuration. It is unclear how sensitive the gains are to communication bandwidth or different parallelism settings. Also, the component ablation is performed on only one model and workload configuration. Some additional analysis would help establish how broadly the individual mechanisms apply.

Questions for authors

Q1. Could the authors provide more details on the runtime overhead of OEWF, including weight transfer latency, temporary memory consumption, and the percentage of fetch attempts that result in successful FFN reassignment?

Q2. How sensitive are the performance gains to the degree of attention imbalance and the available communication bandwidth? It would be interesting to understand under what conditions OEWF provides little or no benefit.

1 comment

Thank you for your careful review and thoughtful questions. Below, we address weight-transfer costs and reassignment (Weakness 1 / Question 1), sensitivity to imbalance and bandwidth (Question 2 / Weakness 2), and parallelism settings and component contributions (Weakness 2).

Weakness 1 / Question 1: Transfer time, memory overhead, and successful FFN reassignment. For DeepSeek-V4-Flash in our DP8+EP8 setup, the transfer overhead is 0.63 ms, with 427.82 MB of temporary weight-and-scale storage per helper. These costs depend on the expert-parallel configuration. In our analysis, 21.6% of the helpers' pending token–expert assignments on their selected donors are candidates whose reassignment is predicted to shorten donor–helper completion time if the weights are already available. Of these candidates, 52.8% are successfully reassigned.

Question 2 / Weakness 2: Sensitivity to attention imbalance and communication bandwidth. We evaluate both factors for DeepSeek-V4-Flash on AgentX with the cache-aware Preble policy at 5 req/s, varying one factor at a time. Both tables report the end-to-end p95 TTFT reduction of full AsyncEP relative to Sync.

As the synchronous attention-idle ratio increases from 10% to 40%, the TTFT reduction grows from 6.4% to 14.3%.

Sync attention idle (%)10.0020.0030.0040.00
p95 TTFT reduction vs. Sync6.4%11.4%14.0%14.3%

For bandwidth sensitivity, we evaluate different numbers of active NVLink links per GPU, which NVIDIA's --setLinkWidth interface can control [1]. Across 2–18 active links, the TTFT reduction increases from 6.2% to 12.1%.

Active NVLink links per GPU26101418
NVLink bandwidth (GB/s)100300500700900
p95 TTFT reduction vs. Sync6.2%9.1%10.6%11.5%12.1%

In these experiments, limiting NVLink bandwidth to 100 GB/s (two active links per GPU) leaves OEWF with negligible additional benefit. The 6.2% p95 TTFT reduction comes primarily from asynchronous EP and streamFFN.

Weakness 2: Sensitivity to parallelism settings. We extend the comparison from DP8+EP8 to DP16+EP16. The table below reports results for DeepSeek-V4-Flash on AgentX at 10 req/s, using two eight-GPU B200 nodes connected by 400 Gb/s InfiniBand. EP communication spans nodes, while OEWF remains within each helper's local NVLink domain. AsyncEP reduces p95 TTFT by 9.6–13.5% across the five policies.

PolicySync p95 TTFT (s)AsyncEP p95 TTFT (s)Reduction
Round-robin26.1123.4910.0%
Least-loaded17.0515.1211.3%
Prefix-only3.863.499.6%
DualMap3.573.1511.8%
Preble3.743.2413.5%

Weakness 2: Component ablation. We extend the component ablation to DeepSeek-V4-Pro and GLM-5.3 using AgentX under round-robin scheduling with DP8+EP8, at 2.5 and 2 req/s, respectively. All throughput values below are in ktokens/s, and percentages report cumulative improvements relative to synchronous EP.

ModelSynchronous EPAsynchronous EP+streamFFN+OEWF (full AsyncEP)
DeepSeek-V4-Pro158.30162.28 (+2.5%)168.43 (+6.4%)171.37 (+8.3%)
GLM-5.3114.48117.01 (+2.2%)123.48 (+7.9%)127.01 (+10.9%)

References

[1] NVIDIA. nvidia-smi — NVIDIA System Management Interface program. Official documentation, n.d.

Reviewer_Em8eOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper addresses idle time in data-and-expert-parallel (DEP) MoE prefill: attention replicas finish at different times, and synchronous expert parallelism holds all FFN work until the slowest replica has dispatched. On DeepSeek-V4-Flash in SGLang this waiting takes 28–29% of end-to-end runtime per replica, and Figure 2 argues that request schedulers cannot remove it without losing prefix-cache hits. AsyncEP extends DeepEP so that FFN work starts on tokens from replicas that have already dispatched, batches ready tokens with a profiled launch threshold and multi-stream execution (streamFFN), and lets replicas that finish early fetch expert weights from a slower replica and run part of its unstarted FFN work when a cost model predicts no slowdown (OEWF). On one 8×B200 node with DeepSeek-V4-Flash, DeepSeek-V4-Pro and GLM-5.3 and two agentic traces, p95 TTFT at the highest request rates falls by 6–32% (1.21× on average) and throughput rises by 6% on average (up to 1.17×) across five scheduling policies.

Critical strengths
  • The problem is quantified on current models and real agentic traces, and its trade-off with cache reuse is made explicit. Appendix A defines the long-tail ratio and measures 28.0–28.9% idle time, and Figure 2 places seven request-scheduling policies on a cache-hit versus attention-balance Pareto frontier. This makes the argument that scheduling alone cannot close the gap concrete, and the framing is useful beyond this system.
  • A complete implementation with a broad evaluation and a clean incremental ablation. The system is built into SGLang and DeepEP with per-pair lanes, readiness signals and a commit-and-fallback protocol (Appendix C). The evaluation spans three frontier MoE models, two traces and five policies with five runs each, and Figure 8 attributes the gains step by step (throughput 247.2, 255.5, 275.1 and 288.2 ktok/s for Sync, Async, +streamFFN and +OEWF), while streamFFN keeps 97.3% of synchronous FFN throughput.
  • OEWF is conservative by construction. A reassignment is accepted only if the predicted maximum completion time does not increase (Eq. 6), the non-degradation argument is stated with its assumptions (Appendix E), and the predictors' accuracy is measured (90.3% and 85.9%, Appendix D). Figure 9 shows per-layer latency reductions without reported regressions.
Critical weaknesses
  1. The gains are smallest under the scheduling policies an agentic deployment would use. On AgentX, p95 TTFT reductions at the highest rate are 19–32% under round-robin and least-loaded but 6.3–14.0% under prefix-only, DualMap and Preble, where throughput gains are 2.7–7.3% (Figures 6–7). The cache-aware policies also have far lower absolute TTFT (for DeepSeek-V4-Flash, about 4 s for prefix-only against about 13 s for least-loaded and over 20 s for round-robin), so they are the relevant operating point, while the headline averages pool all five policies. Reporting the best policy with and without AsyncEP, and the share of the measured idle time recovered under each policy, would state the practical benefit more accurately. The abstract also gives a 1.13× average p95 speedup where Section 4.2 gives 1.21×.
  2. The evaluation covers one node, where weight fetching is cheapest. All experiments run DP8+EP8 inside a single NVLink-connected 8×B200 node, and OEWF copies full expert weights over NVLink. Multi-node DEP prefill, where dispatch crosses RDMA, more replicas increase straggling, and weight fetches compete with token traffic on slower links, is not evaluated, and the cost model does not include inter-node bandwidth. Results at EP16 or EP32 across nodes, or the fetch bandwidth below which OEWF stops paying off, are needed to support the general claims about DEP prefill.
  3. Other ways of absorbing the same idle time are neither discussed nor compared. It is not stated whether the synchronous baseline uses two-micro-batch overlap, which the cited DeepSeek inference overview describes for prefill and which also fills one micro-batch's waiting with another's work. Attention–FFN disaggregation (MegaScale-Infer, cited only as background on DEP) removes the coupling altogether. Moving expert weights instead of tokens has precedents in FasterMoE's dynamic shadowing and in MoonEP's runtime redundant experts, which the paper cites only as a synchronous EP library. Discussing these and adding a micro-batch-overlap baseline would clarify what is specific to AsyncEP.
Questions for authors
  1. Clarification. Is two-batch overlap or other micro-batching enabled in Sync and in AsyncEP? If not, how does AsyncEP compare with Sync plus micro-batch overlap?
  2. Clarification. Which average does the abstract's 1.13× refer to, and what are the gains when each workload uses its best scheduling policy?
  3. Clarification. What fraction of the 28–29% idle time does each mechanism recover under each of the five policies?
4 comments

Thank you for your detailed review and constructive suggestions. Below, we discuss gains under cache-aware policies (Weakness 1 / Question 2), recovered idle time (Question 3), cross-node execution and bandwidth (Weakness 2), and comparisons with TBO and prior work (Weakness 3 / Question 1).

Weakness 1 / Question 2: Gains under the best scheduling policy and averaging scope. We included round-robin and least-loaded alongside cache-aware policies because they also have practical value in real deployments: they are the out-of-the-box DP load balancers in mainstream engines—vLLM routes across DP engines by running and waiting queue sizes [1], and SGLang's built-in --load-balance-method offers round_robin as well as request- and token-count balancing [2]—and DeepInfer's production deployment also assigns requests within each DP group to the rank with the lowest compute-quota load, while shared KV storage lets prefill engines reuse cached prefixes [3]. Moreover, on Tool&Agent, round-robin and least-loaded already come close to the best cache-aware policy at some evaluated request rates—for example, both are within 10% on GLM-5.3 at 3 req/s, with and without AsyncEP (Figure 6a).

We apologize for the unclear summary statistics in the original submission: the abstract's 1.13× is the arithmetic mean of Sync / AsyncEP p95 TTFT across all 180 operating points in Figure 6 (3 models × 2 workloads × 5 policies × 6 request rates), whereas §4.2's 1.21× averages only the 30 points at the highest evaluated request rate. Because a single pooled average obscures the policy differences on AgentX, we will revise the relevant manuscript sections during the rebuttal period to explicitly distinguish all-rate vs. highest-rate averages, report results separately by workload and policy family, and highlight gains under the best scheduling policy.

For replica-local KV caching on high-reuse workloads like AgentX, cache-aware policies indeed achieve much lower absolute TTFT and represent the primary operating point. Under the best scheduling policy for each model–workload setting (identical with and without AsyncEP, and always cache-aware), AsyncEP consistently reduces highest-rate p95 TTFT (Figure 6) and improves high-load throughput (Figure 7):

Tool&Agent — p95 TTFT (Figure 6)

ModelBest policyRate (req/s)Sync → AsyncEP (s)Reduction
DeepSeek-V4-FlashDualMap242.48 → 1.9620.9%
DeepSeek-V4-ProDualMap123.40 → 2.7020.6%
GLM-5.3DualMap65.65 → 4.6218.2%

AgentX — p95 TTFT (Figure 6)

ModelBest policyRate (req/s)Sync → AsyncEP (s)Reduction
DeepSeek-V4-FlashPreble63.36 → 2.8914.0%
DeepSeek-V4-ProDualMap35.45 → 5.027.9%
GLM-5.3DualMap2.412.65 → 11.806.7%

Throughput gains (Figure 7). These use the same policies as above, at the separately reported request rates below.

WorkloadModelRate (req/s)Throughput gain
Tool&AgentDeepSeek-V4-Flash20+6.3%
Tool&AgentDeepSeek-V4-Pro10+5.2%
Tool&AgentGLM-5.35+5.6%
AgentXDeepSeek-V4-Flash5+2.7%
AgentXDeepSeek-V4-Pro2.5+3.7%
AgentXGLM-5.32+6.3%

At these highest-rate operating points, AsyncEP reduces best-policy p95 TTFT by 18.2–20.9% on Tool&Agent and 6.7–14.0% on AgentX.

References

[1] vLLM Contributors. Data Parallel Deployment. vLLM official documentation, n.d.

[2] SGLang Team. Server Arguments. SGLang official documentation, n.d.

[3] DeepSeek-AI. DeepSeek V4.1 Flash Inference Deployment Q&A. DeepInfer official documentation, 2026.

Question 3: Idle time recovered by each mechanism. For DeepSeek-V4-Flash on AgentX at 5 req/s, the table below reports the fraction of synchronous attention idle recovered by asynchronous EP with streamFFN and after further enabling OEWF. Each row is normalized by that policy's own synchronous attention idle, and both columns report cumulative recovery. The first column combines asynchronous EP and streamFFN; it does not isolate their individual contributions.

PolicyAsync EP + streamFFNFull AsyncEP (+OEWF)
Round-robin76.2%87.9%
Least-loaded83.3%93.9%
Prefix-only87.1%94.3%
DualMap90.2%95.3%
Preble81.6%93.8%

Weakness 2: Cross-node evaluation and bandwidth sensitivity. Our original evaluation focused on intra-node prefill because keeping prefill DP/EP groups as small as GPU memory permits helps improve both data-parallel and expert-parallel load balance and reduce NVLink traffic (also noted in DeepInfer’s deployment guidance [3]). We agree that evaluation across RDMA-connected domains would strengthen the paper. AsyncEP also supports EP16 spanning multiple NVLink domains connected through RDMA. Cross-node communication follows DeepEP's default hybrid NVLink–RDMA path with node-level traffic aggregation [4]. OEWF restricts weight fetching and reassignment to the helper's local NVLink domain, avoiding additional expert-weight traffic over RDMA. For DeepSeek-V4-Flash on AgentX with Preble at 5 req/s and DP8+EP8, limiting NVLink bandwidth to 100 GB/s (two active links per GPU) leaves OEWF with negligible additional benefit. The table below compares AsyncEP with synchronous EP16 on DeepSeek-V4-Flash with the AgentX workload under all five scheduling policies at 10 req/s, using two eight-GPU B200 nodes (DP16+EP16) with 400 Gb/s InfiniBand NICs.

PolicySync p95 TTFT (s)AsyncEP p95 TTFT (s)Reduction
Round-robin26.1123.4910.0%
Least-loaded17.0515.1211.3%
Prefix-only3.863.499.6%
DualMap3.573.1511.8%
Preble3.743.2413.5%

References

[3] DeepSeek-AI. DeepSeek V4.1 Flash Inference Deployment Q&A. DeepInfer official documentation, 2026.

[4] Chenggang Zhao et al. DeepEP: An Efficient Expert-Parallel Communication Library. GitHub repository, 2025.

Weakness 3 / Question 1: TBO configuration and comparison. Our original evaluation used SGLang v0.5.20, with TBO disabled in both Sync and AsyncEP. PR #42891 [5] documents a DeepSeek-V4 prefill TBO failure and adds an NVIDIA regression test.

TBO and AsyncEP target different sources of inefficiency and are partly orthogonal. TBO overlaps one micro-batch's communication with another's computation. Under synchronous EP, each micro-batch still waits for all attention replicas before its FFN computation can start.

Our implementation is also compatible with TBO. The table below compares TBO, AsyncEP and their combination with the synchronous baseline for DeepSeek-V4-Flash on a single B200 node (DP8+EP8), using the AgentX workload with Preble at 5 req/s.

Configurationp95 TTFT (s)Change vs. Sync
Sync2.926—
Sync + TBO2.808-4.03%
AsyncEP2.583-11.7%
AsyncEP + TBO2.503-14.5%

AsyncEP reduces p95 TTFT by 8.0% relative to Sync+TBO, while enabling TBO on AsyncEP provides a further 3.10% reduction.

Weakness 3: Comparison with attention–FFN disaggregation and dynamic expert replication. Attention–FFN disaggregation facilitates asynchronous execution, but does not itself remove synchronization. MegaScale-Infer [6] (§8) reports pipeline bubbles from attention imbalance and mitigates them through batching toward a target execution time. Our key insight is that ready expert tokens can already sustain near-peak FFN throughput before all attention replicas finish dispatch. Waiting for the remaining replicas then adds synchronization delay with little improvement in compute efficiency. streamFFN exploits this sweet spot through a profiled launch threshold and multi-stream execution. We acknowledge FasterMoE [7] and MoonEP [8] as precedents for dynamic expert replication. MoonEP's end-to-end training benchmark [8] shows small regressions at low expert imbalance. OEWF uses attention readiness and execution progress to admit only reassignments predicted not to delay pair completion (Eq. 6), guaranteeing no increase in layer completion time under our timing and resource-protection assumptions.

References

[5] Cheng Wan. [Fix] Merge DeepSeek-V4 two-batch-overlap microbatches without a residual. SGLang GitHub pull request #42891, 2026.

[6] Ruidong Zhu et al. MegaScale-Infer: Efficient Mixture-of-Experts Model Serving with Disaggregated Expert Parallelism. SIGCOMM, 2025.

[7] Jiaao He et al. FasterMoE: Modeling and Optimizing Training of Large-Scale Dynamic Pre-Trained Models. PPoPP, 2022.

[8] Yutian Chen et al. MoonEP: A Perfectly Balanced Expert Parallelism Library via Dynamic Redundant Experts. GitHub repository, 2026. End-to-end training benchmark.

Thank you for the additional experiments. The best-policy results, the two-node DP16+EP16 run, and the comparison with two-batch overlap answer my questions. I keep my rating at 3 and have raised Soundness to 3.

Reviewer_Xt1EOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is based on the observation that some attention modules may take different amounts of time to complete, causing the downstream FFN to wait for an imbalanced duration and resulting in GPU idle time. To mitigate this issue, the paper proposes asynchronously scheduling FFN execution by allocating CUDA streams to buffer tokens asynchronously and ensure that the FFN starts in a timely manner. To compensate for the potential loss in FFN execution efficiency, the token buffer is controlled by a threshold, and multiple CUDA streams are used to compensate for the last batch of data filled into the GEMM. Experimental results on several SOTA MoE models show that this approach can improve MoE prefill latency by a margin of around 10–17%.

Critical strengths

S1: Addresses a real, though small, problem in MoE serving systems.

S2: Several proposed designs operate at a very fine-grained systems level, which is a good way to maximize the remaining GPU capacity in a critical workload.

S3: Tested several very recent MoE models on a leading GPU server. The datasets are also meaningful for agentic workflows.

Critical weaknesses

W1: MoE prefill may not necessarily be the key bottleneck in today’s coding-agent scenarios, given the high KV reuse ratios. The practical value of this work is real, but perhaps not significant.

W2: Even for this practical, though not necessarily significant, problem, the improvement is marginal.

W3: Using asynchronous, finer-grained GPU operations to compensate for small bubbles or idle periods in an MoE layer’s execution flow is not particularly novel. Similar techniques have been used in other MoE optimization scenarios, such as router + FFN execution. As a result, expert reviewers or MoE infrastructure engineers may learn little from this paper.

Questions for authors

The paper is well written, the experiments are quite complete, and I am generally satisfied with it.

2 comments

Thank you for your constructive feedback and positive assessment of the paper’s clarity and experimental evaluation. Below, we address prefill’s practical relevance (Weakness 1), performance gains (Weakness 2), and AsyncEP’s novelty (Weakness 3).

Weakness 1: MoE prefill may not be the key bottleneck in coding-agent scenarios. AsyncEP is a general-purpose MoE prefill engine for DEP serving, not a system specialized only for coding agents. Whenever uncached prompt lengths or batch compositions differ across DP attention replicas, synchronous EP converts uneven attention times into pre-FFN GPU stalls. We acknowledge that high prefix reuse can reduce prefill’s share of end-to-end latency. Across general LLM services and diverse agent traces, prefill can remain a substantial latency and GPU-cost driver:

  1. Production traffic and non-coding agent workloads can exhibit much lower prefix-cache hit rates. Compared with high-reuse coding-agent sessions, these workloads require uncached prefill work for a substantially larger fraction of input tokens.

    • Production LLM and non-coding agent traces: DeepSeek-V3/R1's live service reports a 56.3% KV-cache hit rate across 608B daily input tokens—leaving 266B tokens/day (43.7%) to fresh prefill [1]—and Mooncake's production traces [2] show 59% prefix reuse on Tool&Agent (§4, where AsyncEP cuts best-policy p95 TTFT by 18.2–20.9%) and 40% on Conversation.
    • Retrieval and memory-intensive agents: Retrieval pauses in search agents trigger KV evictions, yielding 7% prefix-cache hit rate in the vLLM-based baseline, rising to 51–65% with SearchAgent-X’s optimizations [3]. Likewise, query-dependent context ordering limits exact prefix reuse in RAG, agent memory (Mem0), and multi-agent workflows. ContextPilot reports prefix-cache hit rates increasing from 8.49% to 33.97% with SGLang/Qwen3-32B and from 10.7% to 43.2% with vLLM/Llama3.3-70B after context alignment and scheduling [4].
    • Prefill-only serving beyond agents: Recommendation, credit verification, and data labeling process tens of thousands of prompt tokens to emit one token, as described in PrefillOnly [5].
  2. Even in coding-agent clusters, prefill efficiency also matters on its own. As shown in industrial deployments, Red Hat and IBM's llm-d study [6] uses production coding-agent traces and measures a 91.4% prefill cache-hit rate in its routing experiment. The study notes that “for this workload, the cluster is primarily buying prefill capacity,” and its 64-GPU sizing experiments compare setups that give half or three quarters of the GPUs to prefill. DeepSeek's production DeepInfer system [7] likewise sizes its prefill capacity by the number of uncached input tokens, separately from decode. Faster prefill can therefore cut the GPUs needed to serve the same request rate within a TTFT target, even when prefill is not the largest part of request latency; how large that part is depends on the model, cache-hit rate, decode setup and output length.

References

[1] DeepSeek-AI. DeepSeek-V3/R1 Inference System Overview. Official technical documentation, 2025.

[2] Ruoyu Qin et al. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot. FAST, 2025.

[3] Tiannuo Yang et al. Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents. ICLR, 2026.

[4] Yinsicheng Jiang et al. ContextPilot: Fast Long-Context Inference via Context Reuse. MLSys, 2026.

[5] Kuntai Du et al. PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications. SOSP, 2025.

[6] Maroon Ayoub et al. Serving GLM-5.2 for Agentic Workloads on llm-d. llm-d blog, 2026.

[7] DeepSeek-AI. DeepSeek V4.1 Flash Inference Deployment Q&A. DeepInfer official documentation, 2026.

Weakness 2: Marginal improvement. These gains come on top of an already optimized serving stack. Under the best scheduling policy for each model–workload setting, which is always cache-aware, AsyncEP reduces p95 TTFT at the highest evaluated request rate by 18–21% on Tool&Agent and 7–14% on AgentX.

In an additional experiment, we measure the request rates supported at comparable p95 TTFT on DeepSeek-V4-Flash with Tool&Agent and DualMap. At approximately 2 s p95 TTFT, Sync serves 17 req/s with 1.98 s latency, while AsyncEP serves 24 req/s with 1.96 s latency—a 41.2% increase in request rate. AsyncEP achieves this with the same model, hardware and scheduling policy as the SGLang baseline, which uses DeepEP and DeepGEMM.

Weakness 3: Novelty of AsyncEP. AsyncEP's core innovation is breaking the intra-layer cross-replica synchronization barrier in colocated DEP prefill. While asynchronous execution and fine-grained GPU operations are not unique to AsyncEP, our core technical innovations lie in (1) identifying and exploiting the sweet spot for minimum FFN launch granularity (streamFFN), and (2) OEWF's asynchronous weight-fetching and reassignment policy with a provable non-degradation guarantee under our timing and resource-protection assumptions. Although prior work such as MoonEP [8] also explores fine-grained weight fetching, it relies on a greedy heuristic without such a guarantee; its published end-to-end training benchmark reports small regressions at low imbalance. MoonEP executes weight prefetching on the critical path after collective dispatch rather than asynchronously while straggler replicas are still computing attention.

Beyond execution mechanisms, another contribution is a detailed characterization of attention imbalance and how it interacts with prefix caching. Earlier work such as MegaScale-Infer [9] has noted this imbalance; our analysis explains how it arises and why cache-aware scheduling does not remove it. Even at the same token count, attention time varies with batch composition (Appendix B), and better attention balance usually means less cache reuse (Figure 2).

Furthermore, whereas systems such as MoonEP use router outputs after a synchronous barrier to balance FFN token loads across ranks with dynamic redundant experts, our scheduling explicitly accounts for when each GPU becomes available and how much work it has left. Asynchronous EP tracks readiness per source, and streamFFN batches ready tokens to the throughput sweet spot (ρ=0.90\rho = 0.90, Figure 4) with multi-stream execution to avoid the small, inefficient GEMMs caused by launching immediately (Async alone gains only +3.4% throughput in Figure 8 vs. +11.3% with streamFFN, retaining 97.3% of full-batch GEMM throughput). Meanwhile, OEWF uses live execution progress to pick donors still in attention, asynchronously prefetch weights in the background, and locate FFN work that has not started yet; once the weights arrive, it commits the move only if the predicted finish time of the slower of the donor and helper does not increase (Eq. 6).

References

[8] Yutian Chen et al. MoonEP: A Perfectly Balanced Expert Parallelism Library via Dynamic Redundant Experts. GitHub repository, 2026.

[9] Ruidong Zhu et al. MegaScale-Infer: Efficient Mixture-of-Experts Model Serving with Disaggregated Expert Parallelism. SIGCOMM, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.