acceptodds
Under review as a conference paper at ICLR 2027

HarmonyTransfer: KVCache Transfer Scheduling for Efficient LLM Agent Serving

Abstract

LLM agent applications often contain fan-out stages whose completion depends on multiple requests. Such a stage reaches collective prefill completion only when every required request has returned its first token. Serving long-context fan-out stages efficiently requires moving the reusable KVCache from lower-tier storage back onto the GPU. That transfer runs over a link shared by every live stage. Existing systems, however, schedule KVCache transfers independently for each request, without accounting for the stage's joint completion semantics. We introduce the Agent Prefill Group (APG), a first-class abstraction representing the set of requests whose first-token completion jointly determines a stage's collective prefill completion, and present HarmonyTransfer, an APG-aware KVCache transfer scheduling system. HarmonyTransfer implements objective-specific APG scheduling policies for reducing mean completion time and improving deadline attainment. It dispatches KVCache transfers in bounded chunk-level batches, allowing the scheduler to make a new APG scheduling decision at each batch boundary rather than only at full-request transfer completion. Built on LMCache and vLLM, HarmonyTransfer reduces mean APG completion time by up to 51.6% over the best-performing baseline and improves APG deadline attainment by up to 23.0 percentage points in testbed experiments.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Inference, LLM Agents, KVCache Transfer, Hierarchical Caching, Scheduling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wwUQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HarmonyTransfer, which treats the APG (agent prefill group) as a scheduling unit for agentic workloads. APG refers to a group of requests which belong to the same fan-out stage and need to wait for one another before returning the results back to the agent. Currently, the LLM inference systems such as vLLM and LMCache still handle APG at a request-level, and neglect their inter-dependence. The author has proposed optimizations for mean APG completion time and APG deadline attainments and designed bounded-batch transfer, which splits KV cache transfer into batches of KV chunks for more flexible scheduling. The experiment results show that both APG completion time and deadline attainment can be improved significantly by 51.6% and 23% respectively.

Critical strengths
  1. The authors identified a meaningful design pitfall in the current LLM serving system that the request-level scheduling does not cater to the collective fan-out stages in agentic workloads. The use of abstraction is clear and reasonable.
  2. The authors have conducted experiments on the product-grade systems including vLLM and LMCache and achieved good improvement (up to 51.6%) for APG over different workloads.
  3. The paper includes additional experiments on bandwidth sensitivity, alternative deadline assignment, arrival-pattern robustness, and scheduling overhead measurements to provide full picture of the system.
Critical weaknesses
  1. this paper focuses on optimizing TAPG=max(TTTFT)T_{APG} = max(T_{TTFT}), however, the fan-out stage does not only include TTFT, but also decoding time for subsequent tokens. Ideally, we should measure the end-to-end time for the fan-out stage instead of just the collective prefill time and the authors failed to conduct such experiments. Moreover, this could be measured under PD-shared and PD-disaggregated setups.
  2. The APG abstraction is similar to the coflow scheduling policy, where completion is determined by the last member. Although the authors discuss differences from classical coflow scheduling in Appendix D.3, the paper does not convincingly establish why these differences necessitate the proposed algorithms or how much improvement they provide over appropriately adapted coflow scheduling methods.
  3. HarmonyTransfer-Mean ranks APGs using the sum of remaining transfer and prefill-compute demands across member requests. However, transfer and computation may overlap, and the completion time of an APG depends on their execution order rather than simply the sum of their durations
Questions for authors
  1. How does HarmonyTransfer compare against a simple group-level SJF policy?
2 comments
AuthorsOpenReview ↗

Response to Reviewer wwUQ (Part 1)

Thanks for your careful and constructive review. We address your concerns below. More details can be found in the revised manuscript.

W1: Fan-out stage Time

We now measure decode-inclusive fan-out completion on our testbed with Llama-3.1-8B under PD-shared setups (Table 2). We observe that decode adds 9.7--11.2 s per APG on LongAgent and 4.1--5.0 s on DocQA (8--22% of the mean stage time) and does not change policy rankings. HarmonyTransfer-Mean still outperforms other baselines in optimizing mean decode-inclusive fan-out completion by optimizing mean APG completion time. We leave PD-disaggregated experiments to future work. In a PD-disaggregated deployment, KV cache is shipped from the prefill instance to the decode instance over a separate P–D link. Our paper focuses on scheduling transfers from the L3 KV cache storage system to the prefill instance, which do not contend with that P–D link in common cases.

PolicyLongAgent APGLongAgent Fan-out stageDocQA APGDocQA Fan-out stage
FIFO over APGs106.83116.5640.9945.34
Request-SJF39.6150.3135.2739.37
Group-SJF41.0152.1620.1825.17
HarmonyTransfer-Mean38.5949.2415.6019.90
AuthorsOpenReview ↗

Response to Reviewer wwUQ (Part 2)

W2: Comparison with coflow scheduling methods

We illustrate our method's benefits over classical methods from both a theoretical perspective and testbed experiments.

  • Theory: Differences to CoFlow. An APG has a coflow's completion semantics (it is done when its last member is done) but not its consumption semantics. A coflow's flows feed one consumer that needs all of them (e.g. a reducer needs its whole shuffle input), so a flow that finishes early is wasted and Varys's MADD deliberately slows short flows so that all of them finish together. However, in the LLM serving setting, APG's members can be independently consumed by prefill computation. As a result, early delivery of a flow (KVCache transfer) is not wasted: it can trigger computation that could overlap with transfer. The difference explains why coflow does not suit the agent serving setting: Proposition 2 (Appendix A.4 in revised manuscript) shows that delivering a group's members together, as MADD does, postpones every prefill to the last byte and can take twice as long as serving them sequentially, and the bound is tight.
  • New Experiments comparing varying policies. Table 1 compares HarmonyTransfer with Group-SJF, the appropriately adapted coflow baseline (Varys's smallest-effective-bottleneck-first on one link, members served sequentially), and with Varys (SEBF + MADD), all with the same APGs, dispatcher, estimator and arrival plan on Qwen2.5-14B.
    • Against the adapted baseline, HarmonyTransfer-Mean is 10.3%, 3.1% and 41.6% faster on DeepSearcher, LongAgent and DocQA: the coflow ordering is close on the transfer-dominated LongAgent, but leaves 41.6% on the compute-heavy DocQA because it ignores prefill demand.
    • Varys as published adds MADD, which postpones every member's prefill to the group's last transfer; it is 6.5% slower than Group-SJF on LongAgent and within 3.5% elsewhere. This shows that coflow's consumption semantics fail in the LLM serving setting.
Policy / WorkloadDeepSearcherLongAgentDocQA
FIFO over APGs44.88130.2167.33
Group-SJF (SEBF on one link, members sequential)36.0437.1439.61
Varys (SEBF + MADD)34.7739.5539.73
Makespan (Johnson)51.8735.8324.05
HarmonyTransfer-Mean32.3435.9923.12

W3/Q1: Comparison with a makespan-aware ranking and a group-level SJF

We reuse the experiments from W2 to describe our method's benefits over the makespan-aware ranking and the simple group-level SJF.

W3: Makespan-aware ranking versus aggregate remaining demand.

  • An APG completes at the makespan of its transfer and prefill pipeline (A.4), not at the sum of its member demands. For ranking, however, Lemma 1 (A.4) is the relevant result: with negligible prefill and all APGs available at time 0, serving APGs contiguously in increasing order of aggregate transfer minimizes mean completion. With prefill, HarmonyTransfer-Mean applies the same rule with the aggregate remaining transfer + prefill demand in place of transfer alone—a heuristic extension that orders APGs by their total demand, which is what the mean-completion objective rewards.
  • Our experiments on Qwen2.5-14B compare the Makespan policy, which ranks APGs by their two-stage overlapped makespan for inter-APG scheduling.
    • On LongAgent, Makespan is 0.4% better than HarmonyTransfer-Mean, while it is 4% and 60% slower on DocQA and DeepSearcher.
    • It loses on DocQA and DeepSearcher because it doesn't track its total demand.
    • LongAgent is the transfer-dominated case in which each member's prefill hides behind the transfers that follow it, so the makespan reduces to the aggregate transfer plus one tail prefill. As a result, makespan orders APGs much like HarmonyTransfer-Mean; that is why the two differ by 0.4% here.

Q1: Comparison with a simple group-level SJF.

  • Our new experiments show that Group-SJF ranks APGs only by their transfer demands, but an APG completes only after its members' prefill computations are done, so ignoring compute demand misorders APGs whose prefill demand is large. Group-SJF takes 36.04/37.14/39.61 s on DeepSearcher/LongAgent/DocQA against 32.34/35.99/23.12 s for HarmonyTransfer-Mean. It performs well in the transfer-dominated LongAgent workload, but fails on the compute-heavy DocQA workload.
Reviewer_io9POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HarmonyTransfer schedules remote KV-cache transfers for groups of requests that jointly determine an agent stage’s prefill completion. Applications declare these Agent Prefill Groups (APGs), and the system combines bounded transfer batches with two policies: aggregate remaining-demand ordering for mean completion time, and a finish-if-next feasibility test followed by deadline ordering for deadline attainment. The implementation integrates LMCache and vLLM on a single inference node. Experiments replay three fan-out workloads with two models and report up to 51.6% lower mean APG completion time and 23 percentage points higher deadline attainment than the best evaluated baseline.

Critical strengths
  1. The scheduling objective matches a real application dependency. A fan-out stage cannot complete until its last required member completes, so reducing individual request latency can be the wrong objective. The APG interface exposes that dependency to the KV-transfer path, and the bounded-batch implementation makes changes in priority actionable. This is a useful systems integration for workloads with large reusable prefixes and a shared remote-cache link.

  2. The reported prefill-side gains come from reasonably controlled comparisons. Policies share the hardware, demand estimator, transfer dispatcher, and arrival plan, while FIFO is already group-aware. The detailed tables support the headline gains. The appendix also includes additional arrival patterns, uniform deadlines, bandwidth variation, and scheduler overhead. These controls make the scoped scheduling results credible; the advantage is not simply the result of giving the proposed method a better transfer implementation.

Critical weaknesses
  1. The evaluation does not isolate the advance over simple size-aware group scheduling. Group completion and feasibility-aware prioritization have a substantial coflow precedent, acknowledged in Section 6 and Appendix D.3; for example, Varys addresses both completion time and deadline objectives (Chowdhury et al., SIGCOMM 2014). The mean-completion baselines are group-aware FIFO and request-level SJF, leaving out the direct intermediate control: APG ordering by aggregate remaining transfer bytes, with the same dispatcher. Likewise, a transfer-only feasibility projection would help establish the value of the prefill-aware deadline policy. The batch-granularity ablation does not answer this question. Without these controls, it is difficult to tell how much the proposed demand model and overlap-aware projection add beyond applying established group scheduling to this setting.

  2. The connection to full agent-stage latency relies on an insufficiently validated decode model. The experiments stop at collective prefill completion. This is the full logical response for DeepSearcher’s one-token reranking, but not for LongAgent or DocQA. Appendix E estimates residual decode using a fixed 18.4 ms/token profile and argues that it cannot change policy rankings. That conclusion assumes the policies leave equivalent decode service conditions, which is not measured. There is also a concrete mismatch: Appendix E says concurrency 16 exceeds every relevant stage’s fan-out, whereas Table 3 reports a maximum DocQA width of 47. Thus, the prefill gains are demonstrated, but the broader application-latency implication remains uncertain.

Questions for authors
  1. Contribution attribution: How do both policies compare with versions that use only remaining transfer demand, while preserving APG membership, batching, and all other runtime settings? How much does the prefill term and the decreasing-prefill-time member ordering contribute?
  2. Application impact: Can you report decode-inclusive fan-out completion on LongAgent or DocQA under the same competing policies, and reconcile the concurrency-16 claim with DocQA’s maximum fan-out of 47? This need not require a large multi-node deployment; one representative measured condition would materially strengthen the argument.
  3. Projection validity: What is the error of the finish-if-next completion prediction under actual contention, including GPU queueing? The cross-validated isolated prefill fit is useful, but does not by itself validate the full completion projection. Does perturbing these estimates change feasibility decisions or deadline attainment substantially?
2 comments
AuthorsOpenReview ↗

Response to Reviewer io9P (Part 1)

Thanks for your careful and constructive review. We address your concerns below. More details can be found in the revised manuscript.

W1/Q1: Comparison with existing group-aware scheduling and Contribution attribution

We elaborate on the advantages of our method over existing group-aware scheduling methods and quantify its contribution attribution with new experiments.

Contribution attribution for mean APG completion time.

  • On DeepSearcher, LongAgent and DocQA, Group-SJF already accounts for 71%, 99% and 63% of the FIFO to HarmonyTransfer-Mean reduction; the prefill term (Group-SJF to HarmonyTransfer-Mean) supplies the remaining 29%, 1% and 37%. It is critical for compute-heavy workload (37% on DocQA) and is negligible for transfer-dominated workload (1% on LongAgent), because an APG completes only after its members' prefills.
  • Varys's finish-together mechanism postpones every member's prefill to the group's last transfer, which makes it 6.5% slower than Group-SJF on LongAgent: coflow's consumption semantics hurt in LLM serving.
  • The Makespan row uses Johnson's member order, which reduces to decreasing prefill time only when every member is transfer-dominated. With that member order fixed, ranking by the exact two-stage makespan is 0.4% better on LongAgent, where the two rankings nearly coincide, but 4% and 60% worse on DocQA and DeepSearcher, because it tracks a group's critical path rather than its total demand.
Mean Optimization Policy / WorkloadDeepSearcherLongAgentDocQA
FIFO over APGs44.88130.2167.33
Group-SJF (SEBF on one link, members sequential)36.0437.1439.61
Varys (SEBF + MADD)34.7739.5539.73
Makespan (Johnson)51.8735.8324.05
HarmonyTransfer-Mean32.3435.9923.12

Contribution attribution for APG deadline attainment. For the deadline policy, we add the baseline the reviewer asks for — a transfer-only feasibility projection — under the same APGs, dispatcher and estimator (Table 12, Appendix C.6; Llama-3.1-8B, LongAgent):

Deadline rule (same APGs, dispatcher, estimator)Deadline attainment
EDF over APGs, no projection19.0%
Transfer-only finish-if-next projection + demotion of infeasible APGs85.0%
HarmonyTransfer-DDL (transfer + prefill projection)87.0%

A transfer-only projection reaches 85%, so of the 68 points between EDF and HarmonyTransfer-DDL the finish-if-next classification is worth 66 (19% →\to 85%), and the prefill term adds the remaining 2 (85% →\to 87%).

AuthorsOpenReview ↗

Response to Reviewer io9P (Part 2)

W2/Q2: Impact on Fan-out Stage Completion

Your concern is right. Our old claim (concurrency 16 exceeds every relevant stage's fan-out) in the previous version's Appendix E (removed in the revision) is a mistake: with a DocQA stage of width 47 the profiled inter-token latency was incorrect there. We thank you for pointing out this mistake; we have removed the old residual decode estimation description based on the fixed 18.4 ms/token profile.

We now measure decode-inclusive fan-out completion on our testbed with Llama-3.1-8B (Table 2). We observe that decode adds 9.7--11.2 s per APG on LongAgent and 4.1--5.0 s on DocQA (8--22% of the mean stage time) and does not change policy rankings. HarmonyTransfer-Mean still outperforms all other baselines.

PolicyLongAgent APGLongAgent Fan-out stageDocQA APGDocQA Fan-out stage
FIFO over APGs106.83116.5640.9945.34
Request-SJF39.6150.3135.2739.37
Group-SJF41.0152.1620.1825.17
HarmonyTransfer-Mean38.5949.2415.6019.90

Q3: Projection validity

Scheduling Correctness of finish-if-next Projection

The finish-if-next projection is a classification, not a duration predictor: it charges an APG's own remaining work and the in-flight prefix but not the APGs served before it, so it is systematically optimistic. We therefore validate the decision it feeds under actual contention, including GPU queueing, rather than its absolute prediction error:

  • In the scheduling robustness experiment, only 2--4% of LongAgent's APGs and 1--2% of DocQA's APGs change their accept/downgrade decision at σ≤50%\sigma\le50\%. These perturbations therefore leave the projection's decisions essentially unchanged.

We investigate whether finish-if-next projection labels APGs correctly in testbed experiments. We found:

  • No APG that the projection labelled infeasible actually met its deadline, i.e. the projection never demoted an APG that would have succeeded. All disagreements are therefore one-sided: APGs admitted as feasible that then missed their deadlines under contention.
  • As the table shows, across two runs with different seeds, the projection label matches the realized outcome in most cases (88% and 77% on average).
Workload / SeedSeed ASeed B
LongAgent89%87%
DocQA80%73%
Reviewer_wFUkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies KV-cache transfer scheduling for LLM agent applications characterized by fan-out execution patterns. The authors identify a mismatch between conventional request-level KV-cache scheduling and the collective completion objectives of agent workloads. To address this limitation, they introduce the Agent Prefill Group (APG), which explicitly represents requests sharing a synchronization dependency, and propose HarmonyTransfer, an APG-aware scheduling framework integrated with LMCache and vLLM. Experiments across three agent workloads and two language models report reductions of up to 51.6% in mean APG completion time and improvements of up to 23 percentage points in deadline attainment relative to the best-performing baseline.

Critical strengths

The paper is well-written and addresses an important systems problem arising from the increasing adoption of multi-agent and parallel LLM workflows. Its central observation that request-level optimization can be misaligned with application-level synchronization objectives is well motivated and provides a useful perspective on KV-cache management. The APG abstraction is conceptually straightforward and captures an important scheduling dependency that conventional serving infrastructure may overlook. The distinction between optimizing average completion time and deadline attainment is meaningful, as these objectives have different practical implications for latency-sensitive agent applications. Implementing the proposed policies within LMCache and vLLM demonstrates engineering feasibility, while the evaluation across multiple workloads provides encouraging evidence that group-aware scheduling can outperform request-centric approaches.

Critical weaknesses

The algorithmic novelty and generality of the proposed scheduling policies are limited. Prioritizing groups according to aggregate remaining processing demand resembles classical shortest-remaining-processing-time and group scheduling heuristics, while deadline-aware prioritization is closely related to established scheduling approaches. The paper would benefit from a more rigorous characterization of what is fundamentally new beyond adapting these principles to KV-cache transfers. In addition, the formulation focuses on collective prefill completion, whereas practical agent workflows may exhibit more complex dependencies, including dynamically generated requests, conditional branches, overlapping computation and communication, and heterogeneous downstream execution costs. It is unclear how robust the approach remains when complete APG membership is unavailable at scheduling time. In addition, improvements in APG prefill completion do not necessarily translate into proportional reductions in end-to-end agent task latency.

Questions for authors
  1. Can the authors provide a formal scheduling formulation and characterize whether the proposed policies admit approximation guarantees or optimality results under simplified assumptions?
  2. How does HarmonyTransfer handle dynamically expanding APGs, incomplete knowledge of group membership, or agent workflows with nested synchronization dependencies?
  3. How sensitive are the scheduling decisions to errors in estimated remaining transfer demand and prefill computation time?
  4. Can the authors report end-to-end agent execution latency, rather than primarily collective prefill completion, to demonstrate application-level benefits?
3 comments
AuthorsOpenReview ↗

Response to Reviewer wFUk (Part 1)

Thanks for your careful and constructive review. We address your concerns below. More details (Proposition, Lemma, etc.) can be found in the revised manuscript.

W1/Q1: What's fundamentally new

  • We agree that ranking groups by remaining demand is close to typical group scheduling. But our main contribution is about how to schedule an execution pipeline including KVCache transfer and prefill computation.
  • Differences to CoFlow. An APG has a coflow's completion semantics (it is done when its last member is done) but not its consumption semantics: a coflow's flows feed one consumer, so an early flow is wasted, whereas an APG's members are consumed one at a time by their own prefills, so an early KVCache transfer is not wasted—it triggers computation that overlaps the transfers that follow. Proposition 2 (Appendix A.4) turns this into a bound: constraining an APG's transfers to finish together can make the APG complete up to twice as late as the best possible schedule, and the bound is tight.
  • New policy baselines on Qwen2.5-14B:
    • Group-SJF ranks APGs by transfer demand only, but an APG completes only after its members' prefills, so it misorders groups with large prefill demand: it performs well on the transfer-dominated LongAgent but fails on compute-heavy DocQA.
    • Varys's finish-together mechanism postpones every member's prefill to the group's last transfer and hurts it: existing coflow methods that follow coflow's consumption semantics fail in the LLM serving setting.
    • The Makespan row is 0.4% better on LongAgent but 4% and 60% worse on DocQA and DeepSearcher: it ranks a group's critical path rather than its total demand, and on the transfer-dominated LongAgent each member's prefill hides behind the following transfers, so the makespan reduces to the aggregate transfer plus one tail prefill and nearly coincides with HarmonyTransfer-Mean (§5.4).
Policy / WorkloadDeepSearcherLongAgentDocQA
FIFO over APGs44.88130.2167.33
Group-SJF (SEBF on one link, members sequential)36.0437.1439.61
Varys (SEBF + MADD)34.7739.5539.73
Makespan (Johnson)51.8735.8324.05
HarmonyTransfer-Mean32.3435.9923.12

New Experiments comparing varying policies for deadline attainment. For the deadline policy, we add a new baseline, a transfer-only feasibility projection, under the same APGs, dispatcher and estimator (Table 12, Appendix C.6; Llama-3.1-8B, LongAgent at 0.60 QPS):

Deadline rule (same APGs, dispatcher, estimator)Deadline attainment
EDF over APGs, no projection19.0%
Transfer-only finish-if-next projection + demotion of infeasible APGs85.0%
HarmonyTransfer-DDL (transfer + prefill projection)87.0%
  • On the transfer-dominated LongAgent workload, the transfer-only projection already recovers most of the gain (85%), which shows that the finish-if-next classification is the dominant factor for better performance. This baseline still uses our APG interface, bounded-batch dispatcher and estimator.

Q1: Formal Scheduling Formulation.

  • APG gg arrives at AgA_g with member set Rg\mathcal R_g; member rr needs ara_r seconds of link service followed by brb_r seconds of prefill, its prefill can start only after its transfer completes, and the GPU runs one prefill at a time. One shared L3→L2L3\to L2 link serves one transfer batch at a time, non-preemptively.
  • Optimization Goal: With FrF_r the return of rr's first token, Cg=max⁡r∈RgFrC_g=\max_{r\in\mathcal R_g}F_r, and the two objectives are 1∣G∣∑g(Cg−Ag)\frac{1}{|\mathcal G|}\sum_g(C_g-A_g) and 1∣G∣∑g1[Cg≤Dg]\frac{1}{|\mathcal G|}\sum_g\mathbf 1[C_g\le D_g].

Q1: Guarantees. The problem is NP-hard because it contains the two-machine flow shop with total completion time; online arrivals and deadlines only make it harder, so we claim no approximation guarantee for our online multi-APG policies.

  • Optimality under simplified assumptions (Appendix A.4): Lemma 1 — with negligible prefill and all APGs arriving at time 0, contiguous service in increasing order of remaining transfer minimizes mean completion; Proposition 1 — for one APG, Johnson's rule is optimal for member ordering over all schedules, including those sharing the link.
AuthorsOpenReview ↗

Response to Reviewer wFUk (Part 2)

Q2: Dynamically expanding APGs

For groups that grow while running, members can join a live APG. Its remaining demand is recomputed when a newly arrived member request becomes visible to the scheduler. We describe the interface in Appendix A.5: a stage is announced once, with the members known so far and the final member count. After that, later members can be submitted through the same job handle. In the new experiment (Table 10, Appendix C.4; DocQA, Llama-3.1-8B): we reveal only 50% of each APG's members at arrival and add the rest after 2000ms. HarmonyTransfer-Mean keeps its gain over Group-SJF. The scheduling mechanism still works well under this setting.

PolicyFull membership50% at arrival
Group-SJF14.3215.58
HarmonyTransfer-Mean10.7711.45

We next discuss how APG handles more complex agent dependency shapes.

  • Overlapping computation and communication are already captured by our formulation: a member's prefill starts as soon as its own transfer completes and overlaps the transfers that follow.
  • Nested synchronization dependencies and conditional branches. We can announce each fan-out stage as an independent APG, so a sub-agent's nested fan-out arrives as a separate group, and a branch that resolves late simply produces its requests (and their APG, or new members of a live one) when the condition is known. All these independent APGs can be handled by the online scheduler and the dynamic-membership interface above (Appendix A.5) with no change to the policies.
  • Heterogeneous downstream execution costs are deliberately outside the scope of our work. Our method can only schedule each member's transfer and prefill computation. Other downstream execution costs are out of our scheduling scope. Optimizing downstream execution or the whole workflow across stages remains future work.

Q3: Robustness to Demand Estimation Error

In order to estimate the impact of estimation errors on scheduling decisions, we perturb the estimation results observed by the scheduler. We inject multiplicative noise ϵ∼U(−σ,σ)\epsilon\sim U(-\sigma,\sigma) into both the transfer and the prefill estimates, with σ∈{10,20,30,50}%\sigma\in\{10,20,30,50\}\%: each member's two estimates are multiplied by their own factor, drawn independently across members and across the two terms (Prefill and Transfer).

Performance stays essentially flat as σ\sigma increases:

  • Mean completion moves by at most 11%, and deadline attainment by at most 4 percentage points (87% to 85--86% on LongAgent, 77% to 73--74% on DocQA). Only 2--4% of LongAgent's APGs and 1--2% of DocQA's APGs change their accept/downgrade decision at σ≤50%\sigma\le50\%.
  • Why: errors are drawn independently per member and partly cancel when aggregated: independent zero-mean errors of scale σ\sigma leave a deviation of about σ∑iwi2\sigma\sqrt{\sum_i w_i^{2}}, i.e. σ/n\sigma/\sqrt{n} in relative terms when members are comparable. Robustness therefore grows with APG size.
WorkloadPolicyσ=0\sigma=010%20%30%50%
LongAgentHT-Mean (s)18.7518.4918.5418.5918.46
LongAgentHT-DDL (attainment)87%86%85%86%85%
DocQAHT-Mean (s)8.568.038.447.627.83
DocQAHT-DDL (attainment)77%73%74%74%74%
AuthorsOpenReview ↗

Response to Reviewer wFUk (Part 3)

Q4: Agent Fan-out Stage E2E Completion Time

  • We conducted new experiments to demonstrate application-level benefits of our proposed methods. For LongAgent and DocQA, we now measure decode-inclusive fan-out stage end-to-end completion time under varying policies on the same testbed (Table 2 in the revised manuscript). The results show that decode adds 9.7--11.2 s per APG on LongAgent and 4.1--5.0 s on DocQA (8--22% of the mean stage time), i.e. a nearly additive tail that is the same for every policy. HarmonyTransfer works well in optimizing fan-out stage completion time: 49.2 s against 116.6 s for FIFO on LongAgent and 19.9 s against 45.3 s on DocQA (-58% and -56%).
  • Fan-out stage E2E completion time is critical for latency-sensitive user experience. A fan-out stage (parallel tool calls, gather-style sub-agent launches, reranking) produces no output until its last member returns, and the application cannot proceed before that moment. Users have to wait for the whole stage rather than each request. We leave full agent execution latency optimization for future work.
PolicyLongAgent APGLongAgent Fan-out stageDocQA APGDocQA Fan-out stage
FIFO over APGs106.83116.5640.9945.34
Request-SJF39.6150.3135.2739.37
Group-SJF41.0152.1620.1825.17
HarmonyTransfer-Mean38.5949.2415.6019.90

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.