- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents HarmonyTransfer, which treats the APG (agent prefill group) as a scheduling unit for agentic workloads. APG refers to a group of requests which belong to the same fan-out stage and need to wait for one another before returning the results back to the agent. Currently, the LLM inference systems such as vLLM and LMCache still handle APG at a request-level, and neglect their inter-dependence. The author has proposed optimizations for mean APG completion time and APG deadline attainments and designed bounded-batch transfer, which splits KV cache transfer into batches of KV chunks for more flexible scheduling. The experiment results show that both APG completion time and deadline attainment can be improved significantly by 51.6% and 23% respectively.
Critical strengths
- The authors identified a meaningful design pitfall in the current LLM serving system that the request-level scheduling does not cater to the collective fan-out stages in agentic workloads. The use of abstraction is clear and reasonable.
- The authors have conducted experiments on the product-grade systems including vLLM and LMCache and achieved good improvement (up to 51.6%) for APG over different workloads.
- The paper includes additional experiments on bandwidth sensitivity, alternative deadline assignment, arrival-pattern robustness, and scheduling overhead measurements to provide full picture of the system.
Critical weaknesses
- this paper focuses on optimizing , however, the fan-out stage does not only include TTFT, but also decoding time for subsequent tokens. Ideally, we should measure the end-to-end time for the fan-out stage instead of just the collective prefill time and the authors failed to conduct such experiments. Moreover, this could be measured under PD-shared and PD-disaggregated setups.
- The APG abstraction is similar to the coflow scheduling policy, where completion is determined by the last member. Although the authors discuss differences from classical coflow scheduling in Appendix D.3, the paper does not convincingly establish why these differences necessitate the proposed algorithms or how much improvement they provide over appropriately adapted coflow scheduling methods.
- HarmonyTransfer-Mean ranks APGs using the sum of remaining transfer and prefill-compute demands across member requests. However, transfer and computation may overlap, and the completion time of an APG depends on their execution order rather than simply the sum of their durations
Questions for authors
- How does HarmonyTransfer compare against a simple group-level SJF policy?
2 comments
Response to Reviewer wwUQ (Part 2)
W2: Comparison with coflow scheduling methods
We illustrate our method's benefits over classical methods from both a theoretical perspective and testbed experiments.
- Theory: Differences to CoFlow. An APG has a coflow's completion semantics (it is done when its last member is done) but not its consumption semantics. A coflow's flows feed one consumer that needs all of them (e.g. a reducer needs its whole shuffle input), so a flow that finishes early is wasted and Varys's MADD deliberately slows short flows so that all of them finish together. However, in the LLM serving setting, APG's members can be independently consumed by prefill computation. As a result, early delivery of a flow (KVCache transfer) is not wasted: it can trigger computation that could overlap with transfer. The difference explains why coflow does not suit the agent serving setting: Proposition 2 (Appendix A.4 in revised manuscript) shows that delivering a group's members together, as MADD does, postpones every prefill to the last byte and can take twice as long as serving them sequentially, and the bound is tight.
- New Experiments comparing varying policies. Table 1 compares HarmonyTransfer with Group-SJF, the appropriately adapted coflow baseline (Varys's smallest-effective-bottleneck-first on one link, members served sequentially), and with Varys (SEBF + MADD), all with the same APGs, dispatcher, estimator and arrival plan on Qwen2.5-14B.
- Against the adapted baseline, HarmonyTransfer-Mean is 10.3%, 3.1% and 41.6% faster on DeepSearcher, LongAgent and DocQA: the coflow ordering is close on the transfer-dominated LongAgent, but leaves 41.6% on the compute-heavy DocQA because it ignores prefill demand.
- Varys as published adds MADD, which postpones every member's prefill to the group's last transfer; it is 6.5% slower than Group-SJF on LongAgent and within 3.5% elsewhere. This shows that coflow's consumption semantics fail in the LLM serving setting.
| Policy / Workload | DeepSearcher | LongAgent | DocQA |
|---|---|---|---|
| FIFO over APGs | 44.88 | 130.21 | 67.33 |
| Group-SJF (SEBF on one link, members sequential) | 36.04 | 37.14 | 39.61 |
| Varys (SEBF + MADD) | 34.77 | 39.55 | 39.73 |
| Makespan (Johnson) | 51.87 | 35.83 | 24.05 |
| HarmonyTransfer-Mean | 32.34 | 35.99 | 23.12 |
W3/Q1: Comparison with a makespan-aware ranking and a group-level SJF
We reuse the experiments from W2 to describe our method's benefits over the makespan-aware ranking and the simple group-level SJF.
W3: Makespan-aware ranking versus aggregate remaining demand.
- An APG completes at the makespan of its transfer and prefill pipeline (A.4), not at the sum of its member demands. For ranking, however, Lemma 1 (A.4) is the relevant result: with negligible prefill and all APGs available at time 0, serving APGs contiguously in increasing order of aggregate transfer minimizes mean completion. With prefill, HarmonyTransfer-Mean applies the same rule with the aggregate remaining transfer + prefill demand in place of transfer alone—a heuristic extension that orders APGs by their total demand, which is what the mean-completion objective rewards.
- Our experiments on Qwen2.5-14B compare the Makespan policy, which ranks APGs by their two-stage overlapped makespan for inter-APG scheduling.
- On LongAgent, Makespan is 0.4% better than HarmonyTransfer-Mean, while it is 4% and 60% slower on DocQA and DeepSearcher.
- It loses on DocQA and DeepSearcher because it doesn't track its total demand.
- LongAgent is the transfer-dominated case in which each member's prefill hides behind the transfers that follow it, so the makespan reduces to the aggregate transfer plus one tail prefill. As a result, makespan orders APGs much like HarmonyTransfer-Mean; that is why the two differ by 0.4% here.
Q1: Comparison with a simple group-level SJF.
- Our new experiments show that Group-SJF ranks APGs only by their transfer demands, but an APG completes only after its members' prefill computations are done, so ignoring compute demand misorders APGs whose prefill demand is large. Group-SJF takes 36.04/37.14/39.61 s on DeepSearcher/LongAgent/DocQA against 32.34/35.99/23.12 s for HarmonyTransfer-Mean. It performs well in the transfer-dominated LongAgent workload, but fails on the compute-heavy DocQA workload.
Response to Reviewer wwUQ (Part 1)
Thanks for your careful and constructive review. We address your concerns below. More details can be found in the revised manuscript.
W1: Fan-out stage Time
We now measure decode-inclusive fan-out completion on our testbed with Llama-3.1-8B under PD-shared setups (Table 2). We observe that decode adds 9.7--11.2 s per APG on LongAgent and 4.1--5.0 s on DocQA (8--22% of the mean stage time) and does not change policy rankings. HarmonyTransfer-Mean still outperforms other baselines in optimizing mean decode-inclusive fan-out completion by optimizing mean APG completion time. We leave PD-disaggregated experiments to future work. In a PD-disaggregated deployment, KV cache is shipped from the prefill instance to the decode instance over a separate P–D link. Our paper focuses on scheduling transfers from the L3 KV cache storage system to the prefill instance, which do not contend with that P–D link in common cases.