acceptodds
Under review as a conference paper at ICLR 2027

FastEPD: In-Network Scheduling for Disaggregated LMM Inference

Abstract

Encode–Prefill–Decode (EPD) disaggregation has been adopted to improve the performance of Large Multimodal Model (LMM) services. It separates the encoding (E), prefill (P), and decoding (D) stages across dedicated GPU servers while generating multi-stage data traffic among different servers. These data flows are commonly transmitted by switches in data centers, where queuing management and scheduling mechanisms significantly affect the overall performance. However, existing EPD inference systems lack in-network optimization which is necessary for efficient flow identification and queue scheduling, given the limited network resources and complex traffic characteristics. In this work, we propose FastEPD, an in-network identification and scheduling framework to accelerate LMM EPD inference by reducing the service completion time (SCT) for network transmissions. It consists of two main components: 1) lightweight multidimensional flow identification, and 2) service-oriented multi-queue scheduling. Experiments on a DPDK prototype driven by real model workloads show that FastEPD outperforms canonical network schemes, improving the overall SCT by 57.67 - 79.09% over FIFO and 41.02 - 68.10% over RSS, respectively. Experiments on a two-tier fat-tree also demonstrate that FastEPD remains beneficial as the topology scales from 32 to 128 endpoints.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LMM Inference, Encode-Prefill-Decode (EPD) disaggregation, queue scheduling, lightweight identification, in-network framework

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_F6ZFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FastEPD, a network scheduling method for LLM inference with disaggregated EPD stages. It uses packet tags and 12 queues to classify traffic and allocate sending budgets based on queue backlogs. The authors evaluate it with a DPDK prototype and network simulations across different workloads and models.

Critical strengths
  1. FastEPD tackles network contention in disaggregated LLM inference, where scheduling can affect service completion time.
  2. The design is relatively lightweight, using packet metadata and queue-level scheduling without requiring major changes to the inference pipeline.
Critical weaknesses
  1. The goal is to use EPD service dependencies to reduce completion time, but scheduling mainly relies on a fixed order of communication types and queue backlogs to allocate budgets. Service IDs are introduced, but it's unclear whether the scheduler adjusts the order based on each service's progress and remaining dependencies.
  2. Although Service IDs and service-size classes are introduced, budget allocation still largely follows queue-backlog proportions, and it's unclear what extra value they bring over existing scheduling methods.
  3. Experiments mainly focus on network-side SCT, but don't show whether these gains lead to lower end-to-end latency in a real inference system. The actual speedup for LLM inference is still unclear.
  4. Related work is quite limited, with little in-depth comparison to closely related network scheduling methods. Most of the work focuses on network scheduling and system implementation, while the gains for real LLM inference haven't been fully validated. It may be a better fit for a networking or systems venue.
Questions for authors
  1. How does Service ID actually affect scheduling decisions beyond the fixed communication-type order and backlog-based budget allocation?
  2. What extra benefits do Service IDs and service-size classes bring over existing scheduling methods, especially since budget allocation largely follows queue-backlog proportions?
  3. Do the SCT improvements translate into lower end-to-end latency in a real LLM inference system?
1 comment
AuthorsOpenReview ↗

Response to Reviewer F6ZF

Response to W1/W2/Q1/Q2. Service-Aware Scheduling and Component Contributions

Thank you for identifying the need to clarify how Service IDs and service-size classes affect scheduling beyond fixed stage order and backlog-proportional allocation. The key insight of FastEPD's scheduling is service-unit-aware dequeue batching. Specifically, instead of solely controlling packet-service shares, FastEPD estimates the average packet demand per active service and allocates contiguous dequeue budgets accordingly. Combined with stage-aware queues, this mechanism aims to reduce fragmented transmission and accelerate the release of downstream inference stages. The specific roles are detailed as follows.

1. Role of Service IDs.

Service IDs provide an approximate count of batched services observed in each queue's measurement epoch. Together with packet counts, this yields estimated mean packet demand per service, p^c\hat p_c. FastEPD uses this estimate to convert a service-unit allocation into a contiguous packet dequeue budget:

uc∗=U(pc/p^c)∑j∈Cg(pj/p^j),b~c=uc∗p^c.u_c^{\ast}=\frac{U(p_c/\hat p_c)}{\sum_{j\in C_g}(p_j/\hat p_j)},\qquad \tilde b_c=u_c^{\ast}\hat p_c.

where pcp_c is the queue backlog and UU is the group-level service-unit budget. Thus, Service IDs do not change the fixed flow-type order, but they affect the estimated service count and the absolute dequeue quantum. Notably, these estimates are approximate without exact per-service completion guarantees.

2. Role of service-size classes and backlog allocation.

Within each EPD stage, separating short, medium, and long services prevents them from sharing a single FIFO queue. Short-first class visits offer smaller services earlier transmission opportunities, while positive budgets for nonempty classes limit the risk of long-service starvation. Backlog-based allocation is employed for ensuring the balance of different queues. Particularly, in the ideal continuous case (Appendix C), b~i/b~j=pi/pj\tilde b_i/\tilde b_j=p_i/p_j: relative packet shares depend on backlogs, but the absolute visit budgets and completion timing also depend on the service-unit estimates and UU. To analyze these roles, we propose two ablation versions. No-size removes service-size separation, and C-WFQ only considers queue-backlog proportions without service-ID information. The average SCT results are as follows, with specific configuration in Appendix H.

request class (SCT)C-WFQNo-sizeFastEPD
Short95,62494,07388,649
Medium92,59793,90785,806
Long83,36794,51360,055
Overall93,46694,07084,857

Response to W3/Q3. End-to-End Inference Performance

Thank you for highlighting the gap between network-side SCT and end-to-end inference performance. FastEPD aims to reduce communication-induced stalls along the EPD execution path. Accelerating blocking EP/PP/PD transfers can advance downstream computation or first-token delivery, while reducing repeated DD communication delays can improve token generation intervals.

To quantify these effects, we extend our evaluation with model-derived E/P/D computation costs and token-level execution dependencies. We measure the modeled TTFT, TPOT by simulating dependency-safe computational endpoint model, which allows computation and communication overlapping across services. The modeled TTFT/TPOT results are as follows.

request class (modeled TTFT)C-WFQPIASFastEPD
Short85,72086,95084,163
Medium85,68786,96684,235
Long85,59486,95183,802
Overall85,69886,95584,148
request class (modeled TPOT)C-WFQPIASFastEPD
Short79.17692,05073,799
Medium79.03992,09074,158
Long78,56192.17172,355
Overall79,07382.07473,763

Response to W4, Related Work and Relevance to ICLR

We appreciate the reviewer's concern and agree that the related-work discussion should be expanded. FastEPD targets a communication bottleneck arising specifically from EPD-disaggregated LMM inference, instead of proposing a general-purpose network scheduler as most networking/system works. Its contribution is to expose inference-stage dependencies to the network and exploit them through lightweight, service-aware packet scheduling, complementing application-level resource allocation and communication optimization. This direction is closely related to prior ICLR works on distributed LLM serving and communication efficiency, including HexGen-2 (ICLR 2025), Preble (ICLR 2025), and Semantic Parallelism (ICLR 2026). We will expand the discussion to distinguish FastEPD from these systems and from traditional flow/coflow scheduling approaches such as pFabric, Varys, and Aalo.

Reviewer_8zSSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FastEPD, an in-network scheduling framework designed to accelerate disaggregated Large Multimodal Model (LMM) inference. FastEPD combines lightweight flow identification with service-oriented multi-queue scheduling to reduce network-side service completion time (SCT). The authors implement FastEPD using a DPDK-based prototype and evaluate its performance through ns-3 simulations. Experimental results demonstrate that FastEPD can effectively reduce SCT.

Critical strengths
  • The paper addresses an important networking challenge in EPD-disaggregated LMM inference by considering communication dependencies across different inference stages and heterogeneous request sizes. Unlike existing studies that primarily focus on computational resource allocation and model placement, this work explores network-level scheduling optimization to improve data transmission efficiency. This provides a valuable perspective on improving the performance of disaggregated LMM inference.
  • FastEPD achieves lightweight flow identification by reusing existing packet-header fields and integrates a multi-queue scheduling mechanism, demonstrating practical implementation feasibility. The authors implement a DPDK-based prototype and conduct extensive evaluations using model-driven traffic workloads and ns-3 simulations across different request scales and network topologies. The experimental results demonstrate that FastEPD can effectively reduce network-side service completion time.
Critical weaknesses
  • The proposed multidimensional flow identification, multi-queue classification, and dynamic dequeue scheduling mechanisms are largely built upon existing network scheduling techniques. The technical novelty mainly lies in their integration and adaptation to the EPD inference scenario. Furthermore, the scheduling strategy does not explicitly formulate or optimize the complete communication dependency structure of inference requests. A more rigorous theoretical analysis of the effectiveness and optimality of the proposed scheduling strategy would help strengthen the technical contributions.
  • The evaluation primarily relies on model-generated communication traffic and focuses on network-side service completion time (SCT), without reporting key end-to-end inference metrics such as time to first token (TTFT) and time per output token (TPOT) in a real LMM serving system. Moreover, the performance improvement over RSS decreases to only 4.94% in the 128-node topology, while the improvement over C-WFQ is only 4.61% under the highest workload. These results suggest that the benefits of FastEPD may become less pronounced in larger-scale networks and under heavy traffic loads, raising concerns about its practical effectiveness in realistic deployment scenarios.
Questions for authors
  1. To what extent does the traffic generator accurately capture the communication characteristics of the evaluated models, particularly the KV-cache migration volume and repeated communication during autoregressive decoding in DeepSeek-V3? How sensitive are the reported improvements to these modeling assumptions?
  2. What are the key algorithmic advantages of FastEPD over existing scheduling methods that consider flow sizes and communication dependencies? Could the authors include stronger and more competitive baselines to better demonstrate the technical novelty of the proposed approach?
  3. Could the authors clarify how improvements in network-side SCT translate into end-to-end inference performance, such as TTFT, TPOT, and request throughput? In particular, why does the performance advantage over RSS diminish considerably in the 128-node topology?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 8zSS

Response to W1/Q2: Technical Novelty and Scheduling Effectiveness

Thank you for pointing out the major techniques in the FastEPD. We would clarify that the primary contribution of FastEPD is an endpoint-assisted, in-network scheduling framework tailored to the heterogeneous traffic and communication dependencies of EPD inference. Particularly, it does not mean to invent entirely new network scheduling primitives. We intentionally build upon established mechanisms to maintain low data-plane complexity and facilitate integration into programmable network environments.

Importantly, FastEPD is not merely a combination of existing techniques. Its three components form a coordinated design: endpoint-assisted identification exposes EPD-specific semantics, multidimensional queues organize flows by communication stage and service size, and service-unit-aware dequeue scheduling translates these semantics into adaptive transmission decisions. Compared with other existing scheduling methods considering flows sizes and dependencies, FastEPD estimates the average packet demand per active service and allocates contiguous dequeue budgets accordingly. This aims to reduce fragmented transmission across scheduling rounds and facilitate earlier completion of dependent communications.

To further verify the effectiveness of FastEPD, we add a new baseline: PIAS (NSDI 2015), which leverages multiple level feedback queues, and does not use FastEPD’s service-stage or size-aware budget allocation. For average SCT performance in 6-node network, the results are as follows. Specific configurations can be seen in Appendix H.

Request class (SCT)C-WFQPIASFastEPD
Short403,650450,450390,712
Medium402,628449,980390,645
Long399,927449,618383,408
Overall402,972450,226389,963

For the optimality of scheduling strategy, We agree that FastEPD does not explicitly solve the globally optimal scheduling problem over complete inference dependency DAGs. Instead, it exploits the common stage-wise dependencies of EPD inference through a lightweight heuristic. Specifically, for a computation stage kk of service ss, its actual starting time is constrained by both computation readiness and the completion of prerequisite communication flows:

\max\left(t_{s,k}^{\mathrm{ready}}, \max_{f\in\mathcal{P}_{s,k}}C_f\right),$$ where $\mathcal{P}_{s,k}$ denotes its prerequisite flows and $C_f$ their completion times. Therefore, reducing the completion time of blocking communication flows can advance downstream computation and reduce communication-induced idle periods. ### **Response to W2/Q3: End-to-End Inference Performance and Scalability** FastEPD exploits the dependencies of EPD inference to reduce the waiting time of downstream computation stages. Specifically, **Earlier completion of EP/PP/PD communication may improve the first-token critical path, while accelerating repeated DD communication can reduce inter-token delays**. We measure the modeled TTFT, TPOT by simulating *dependency-safe computational endpoint model*, which allows **computation and communication overlapping across services**. The simulated end-to-end results are as follows. Specific configurations are also in Appendix H. | request class (modeled TTFT) | C-WFQ | PIAS | FastEPD | |:--|--:|--:|--:| | Short | 85,720 | 86,950 | *84,163* | | Medium | 85,687 | 86,966 | *84,235* | | Long | 85,594 | 86,951 | *83,802* | | Overall | 85,698 | 86,955 | *84,148* | | request class (modeled TPOT) | C-WFQ | PIAS | FastEPD | |:--|--:|--:|--:| | Short | 79.176 | 92,050 | *73,799* | | Medium | 79.039 | 92,090 | *74,158* | | Long | 78,561 | 92.171 | *72,355* | | Overall | 79,073 | 82.074 | *73,763* | Besides, we acknowledge that FastEPD's SCT improvement over RSS decreases from 17.67% (32 nodes) to 4.94% (128 nodes). This trend may arise because **multi-hop contention, ECMP load distribution, and DCQCN congestion-control interactions increasingly influence end-to-end network completion time** beyond local queue scheduling. ### **Response to Q1: Communication Modeling Fidelity** We clarify that the original DeepSeek-V3 evaluation used an MHA-style approximation for KV-cache migration. We have updated it to account for Multi-head Latent Attention (MLA) in the revision. Besides, Our traffic generator already models repeated MoE All-to-All communication during decoding. We have evaluated DeepSeek-V3 using the corrected MLA model. Particularly, all results in the response are based on the updated settings. These results can establish that FastEPD's scheduling benefits persist under more accurate model-specific communication assumptions. **Compared with the MHA-modeld DeepSeek-V3, the advantage of SCT over short services have dinimished in MLA-based model**, as the compressed KV cache lead to less different data volume between long and short services.
Reviewer_ekuxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FastEPD, an in-network scheduling framework for disaggregated large multimodal model (LMM) inference. It targets the communication dependencies among encoding, prefill, and decoding stages, aiming to reduce network-side service completion time (SCT). The framework consists of two main components: lightweight multidimensional flow identification, which encodes service identity, flow type, and service size into an 8-bit scheduling tag; and service-oriented multi-queue scheduling, which organizes packets by communication stage and service size and dynamically adjusts dequeue budgets. The authors implement a DPDK prototype and evaluate FastEPD using model-derived traffic and real-world request traces. Experiments across multiple MoE backbones, traffic scales, and network topologies demonstrate the improvements over conventional network scheduling schemes in terms of SCT.

Critical strengths
  1. A meaningful and relatively novel problem perspective. The paper identifies an interesting network-level optimization opportunity in disaggregated LMM inference. Exploiting the dependencies among encoding, prefill, and decoding communication flows is well motivated and potentially valuable. Although the individual scheduling techniques are not particularly novel, their integration into an EPD-aware framework provides a useful system-level contribution.
  2. Relatively comprehensive experimental evaluation. The paper presents a reasonably broad evaluation, including a DPDK prototype, scheduling-policy ablations, processing overhead measurements, endpoint perturbation tests, scalability experiments, and results across multiple MoE backbones. The results consistently demonstrate improvements over the basic network baselines, and most experimental observations are reasonably explained.
Critical weaknesses
  1. Potential mismatch between the KV-cache traffic model and DeepSeek-V3's MLA architecture. The paper appears to use a generic KV-cache traffic formulation based on the model's hidden dimension, without clearly accounting for DeepSeek-V3's Multi-head Latent Attention (MLA), which substantially reduces the KV-cache footprint. Since the generated traffic volumes directly affect network contention and scheduling behavior, this discrepancy may influence the reported performance results. The paper should clarify whether the actual traffic generator accounts for compressed MLA KV caches and, if not, assess how this modeling assumption affects the conclusions.
  2. Insufficient isolation and generalization of the proposed components' contributions. Although C-WFQ and C-TP provide useful scheduling-policy ablations, both reuse FastEPD's identification and multi-queue mechanisms. The experiments therefore do not fully isolate the contributions of flow identification, queue classification, and adaptive scheduling. Furthermore, these more informative baselines are absent from the cross-model and larger-topology experiments, making it difficult to determine whether the benefits of FastEPD's specific scheduling policy generalize beyond the main prototype setting.
Questions for authors
  1. KV-cache traffic modeling. In Section 2.1, the PD-migration traffic volume is formulated using the model hidden dimension. How does the traffic generator account for DeepSeek-V3's MLA architecture and compressed KV-cache representation? Does the implementation use the stated generic formula, or does it apply a model-specific correction? If the former, could the authors evaluate the sensitivity of the main results to a realistic MLA-based KV-cache size?
  2. Component ablation and baseline coverage. Could the authors further isolate the contributions of flow-type identification, service-size classification, and adaptive dequeue scheduling? In addition, can C-WFQ and C-TP be included in the cross-model and larger-topology experiments to evaluate whether the scheduling-policy gains persist across different settings?
  3. Scalability of the 4-bit Service ID. The current design supports 16 Service ID values and reuses IDs after service retirement. How are IDs allocated and coordinated across endpoints under high concurrency? What happens when all 16 IDs are occupied by active services? Could the authors clarify the allocation scope and provide evidence that ID exhaustion does not introduce significant waiting or performance degradation?
  4. Compatibility of the 8-bit scheduling tag. The DPDK prototype embeds the scheduling tag in the IPv4 ToS field, which normally carries DSCP and ECN information. How does the design preserve compatibility with existing QoS and congestion-control mechanisms, especially the RoCEv2/DCQCN stack used in the ns-3 experiments? Does the simulation use separate metadata, and what would be required to support the tag in a realistic deployment?
1 comment
AuthorsOpenReview ↗

Response to Reviewer ekux

Respond to W1/Q1. MLA-aware KV-cache modeling.

Thank you for the constructive questions. We clarify that Sec. 2.1 used an MHA-style approximation for DeepSeek-V3's PD migration without accounting for its Multi-head Latent Attention (MLA) architecture. We correct the KV-cache migration model as follows:

VPD=B(Nm+Nt)Latt(dc+dR)τ+Vfirst,V_{PD}=B(N_m+N_t)L_{att}(d_c+d_R)\tau+V_{first},

where Latt=61L_{att}=61, dc=512d_c=512, dR=64d_R=64, and τ=2\tau=2 bytes. Thus, KV state is 70,272 bytes/input token, versus 1,662,976 in our original approximation—a 95.77% reduction in the modeled KV payload. The generator supports repeated decode All-to-All based on Azure GeneratedTokens. We have added experimental results with the corrected MLA-based Deepseek-V3 model with different scales (6 nodes and 128 modes, separately), which verify the performance of FastEPD. Particularly, all experimental results in the response are based on such settings. Full data can be seen in Appendix H.

Respond to W2/Q2. Component isolation and generalization.

We clarify that C-TP/C-WFQ retain the same 12 semantic queues and only isolate dequeue-policy changes. Since the flow identification can directly influence the multi-queue design and dequeue policy, we design No-size ablation which perserves service-ID and flow-type. In this case, the scheduler retains four stage queues but removes S/M/L separation. The scheduling policy will not differentiate short or long services, compared with FastEPD scheduling. The average SCT results are shown as follows in large topology of 128 nodes. Specific experimental settings are in Appendix H.

Request class (SCT)C-TPC-WFQNo-sizeFastEPD
Short86,83195,62494,07388,649
Medium79,74492,59793,90785,806
Long68,38083,36794,51360,055

Generally, FastEPD outperforms No-size version for different requests. Compared with C-TP in larger-topology, it shows that FastEPD only has better performance for long requests. The results also demonstrate the sensitivity between MLA-modeled and MHA-modeled KV cache. For MHA-modeled Deepseek-V3, KV cache accounts for 78% data volume of total communcation transmissions, while for the MLA-modeled, the KV cache only accounts for 7.3% data volume. Therefore, the distinctions between 'short' and 'long' services have diminished, resulting in relatively average performance of the average SCT metrics. However, we have measured the end-to-end metrics, the modeled TPOT as follows. Specific configurations are seen in Appendix H.

Request class (modeled TPOT)C-TPC-WFQNo-sizeFastEPD
Short34,46342,00440,10434,193
Medium35,24939,96039,96931,633
Long35,60032,32540,46613,810

The TPOT results show that in this case, FastEPD still remain advantage against ablation versions and baselines. The main reason is that TPOT mainly depends on the tail completion time of relatively short flows, i.e., the all-to-all traffic between decoding phase. The service-unit aware scheduling in FastEPD is more friendly to such traffic, compared with other scheduling strategy.

Respond to Q3. Four-bit Service-ID scalability.

We clarify that the current implementation adopts cyclic Service-ID allocation, IDi=i mod 16ID_i=i\bmod16, without waiting or blocking new services when all 16 IDs have been assigned. In this case, concurrently active services may share the same service-ID. Such collisions can underestimate the observed number of distinct services, inflating the estimated average packet count per service (p^c\hat p_c) and affecting dequeue-budget calculations. Nevertheless, Appendix C establishes that the ideal continuous packet budgets within a flow group satisfy:

b~ib~j=pipj,\frac{\tilde b_i}{\tilde b_j}=\frac{p_i}{p_j},

independently of p^c\hat p_c. Thus, the relative packet allocation remains backlog-proportional, although collisions may still affect absolute budgets, rounding behavior, and service completion timing. Figure 6 in the paper has exhibited the sensitivity of the service-ID width, showing the tradeoff results between ID collisions and ID overheads.

Respond to Q4. Compatibility of the 8-bit scheduling tag.

Our controlled DPDK prototype overlays the IPv4 ToS byte, as it does not implement RoCEv2/DCQCN. So such setting in DPDK experiments do not need to preserve standard DSCP/ECN semantics. The RoCEv2/DCQCN ns-3 model instead carries a separate 8-bit simulation tag. For realistic deployment, FastEPD requires an idle metadata carrier supported by both endpoints and switches, such as reserved header bits and IPv4 padding discussed in Appendix B. Alternatively, explicit encapsulation or protocol extensions may be necessary, potentially introducing additional header overhead.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.