- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FastEPD, a network scheduling method for LLM inference with disaggregated EPD stages. It uses packet tags and 12 queues to classify traffic and allocate sending budgets based on queue backlogs. The authors evaluate it with a DPDK prototype and network simulations across different workloads and models.
Critical strengths
- FastEPD tackles network contention in disaggregated LLM inference, where scheduling can affect service completion time.
- The design is relatively lightweight, using packet metadata and queue-level scheduling without requiring major changes to the inference pipeline.
Critical weaknesses
- The goal is to use EPD service dependencies to reduce completion time, but scheduling mainly relies on a fixed order of communication types and queue backlogs to allocate budgets. Service IDs are introduced, but it's unclear whether the scheduler adjusts the order based on each service's progress and remaining dependencies.
- Although Service IDs and service-size classes are introduced, budget allocation still largely follows queue-backlog proportions, and it's unclear what extra value they bring over existing scheduling methods.
- Experiments mainly focus on network-side SCT, but don't show whether these gains lead to lower end-to-end latency in a real inference system. The actual speedup for LLM inference is still unclear.
- Related work is quite limited, with little in-depth comparison to closely related network scheduling methods. Most of the work focuses on network scheduling and system implementation, while the gains for real LLM inference haven't been fully validated. It may be a better fit for a networking or systems venue.
Questions for authors
- How does Service ID actually affect scheduling decisions beyond the fixed communication-type order and backlog-based budget allocation?
- What extra benefits do Service IDs and service-size classes bring over existing scheduling methods, especially since budget allocation largely follows queue-backlog proportions?
- Do the SCT improvements translate into lower end-to-end latency in a real LLM inference system?
Response to Reviewer F6ZF
Response to W1/W2/Q1/Q2. Service-Aware Scheduling and Component Contributions
Thank you for identifying the need to clarify how Service IDs and service-size classes affect scheduling beyond fixed stage order and backlog-proportional allocation. The key insight of FastEPD's scheduling is service-unit-aware dequeue batching. Specifically, instead of solely controlling packet-service shares, FastEPD estimates the average packet demand per active service and allocates contiguous dequeue budgets accordingly. Combined with stage-aware queues, this mechanism aims to reduce fragmented transmission and accelerate the release of downstream inference stages. The specific roles are detailed as follows.
1. Role of Service IDs.
Service IDs provide an approximate count of batched services observed in each queue's measurement epoch. Together with packet counts, this yields estimated mean packet demand per service, p^c. FastEPD uses this estimate to convert a service-unit allocation into a contiguous packet dequeue budget:
uc∗=∑j∈Cg(pj/p^j)U(pc/p^c),b~c=uc∗p^c.
where pc is the queue backlog and U is the group-level service-unit budget. Thus, Service IDs do not change the fixed flow-type order, but they affect the estimated service count and the absolute dequeue quantum. Notably, these estimates are approximate without exact per-service completion guarantees.
2. Role of service-size classes and backlog allocation.
Within each EPD stage, separating short, medium, and long services prevents them from sharing a single FIFO queue. Short-first class visits offer smaller services earlier transmission opportunities, while positive budgets for nonempty classes limit the risk of long-service starvation. Backlog-based allocation is employed for ensuring the balance of different queues. Particularly, in the ideal continuous case (Appendix C), b~i/b~j=pi/pj: relative packet shares depend on backlogs, but the absolute visit budgets and completion timing also depend on the service-unit estimates and U. To analyze these roles, we propose two ablation versions. No-size removes service-size separation, and C-WFQ only considers queue-backlog proportions without service-ID information. The average SCT results are as follows, with specific configuration in Appendix H.
Response to W3/Q3. End-to-End Inference Performance
Thank you for highlighting the gap between network-side SCT and end-to-end inference performance. FastEPD aims to reduce communication-induced stalls along the EPD execution path. Accelerating blocking EP/PP/PD transfers can advance downstream computation or first-token delivery, while reducing repeated DD communication delays can improve token generation intervals.
To quantify these effects, we extend our evaluation with model-derived E/P/D computation costs and token-level execution dependencies. We measure the modeled TTFT, TPOT by simulating dependency-safe computational endpoint model, which allows computation and communication overlapping across services. The modeled TTFT/TPOT results are as follows.
Response to W4, Related Work and Relevance to ICLR
We appreciate the reviewer's concern and agree that the related-work discussion should be expanded. FastEPD targets a communication bottleneck arising specifically from EPD-disaggregated LMM inference, instead of proposing a general-purpose network scheduler as most networking/system works. Its contribution is to expose inference-stage dependencies to the network and exploit them through lightweight, service-aware packet scheduling, complementing application-level resource allocation and communication optimization. This direction is closely related to prior ICLR works on distributed LLM serving and communication efficiency, including HexGen-2 (ICLR 2025), Preble (ICLR 2025), and Semantic Parallelism (ICLR 2026). We will expand the discussion to distinguish FastEPD from these systems and from traditional flow/coflow scheduling approaches such as pFabric, Varys, and Aalo.