acceptodds
Under review as a conference paper at ICLR 2027

GoodServe: Globally-coordinated Routing of Agentic LLM Inferences for High Goodput

Abstract

Agentic applications consume LLM inferences as complete outputs: a code patch or a database query is useless until its last token arrives, so what matters to the application is end-to-end latency against a deadline, and what matters to the operator is goodput, the number of inferences that finish within their deadlines. Operators usually serve these inferences on pools of heterogeneous GPUs, and every request must be routed to one instance on arrival. Existing routers are greedy: each request goes to the instance that would serve it fastest or is least loaded, which spends the strongest GPUs on requests that would have met their deadlines on weaker ones and leaves urgent requests waiting. Goodput instead calls for globally coordinated routing, in which a request may be placed sub-optimally for itself so that the pool meets more deadlines overall. Coordination needs two quantities a greedy router never estimates: how long each request will run, since its output length is unknown until it finishes, and how fast each instance currently is. GoodServe formulates routing as a global goodput problem whose latency model couples both, including prefix-cache reuse, and solves it in a predict-and-rectify manner: a mixture-of-experts predictor gated by task type estimates output length accurately and efficiently; an EMA-smoothed black-box estimator tracks each instance's queuing delay and per-token speed; a just-enough rule sends each request to the weakest instance that can still meet its deadline and rejects requests that no instance can serve in time; and periodic risk checks migrate endangered requests by token IDs, 7–15× cheaper than transferring KV cache. On a four-GPU heterogeneous testbed serving three agentic benchmarks, GoodServe attains the highest goodput at every SLO level, up to 31.0% over the best existing router, and its routing overhead stays at 5 ms per request at 10,000 requests per second across 512 simulated instances.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Model Serving, Agentic AI, Heterogeneous GPUs, SLO-aware Scheduling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_4N3SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GoodServe is a router for LLM inference requests with completion deadlines on heterogeneous GPU instances. It predicts output lengths, estimates instance performance, and selects the weakest instance expected to meet each request’s deadline. Requests predicted to miss their deadlines everywhere are rejected, while periodic checks migrate endangered requests using token IDs and destination recomputation. The latency model also accounts for prefix reuse. Experiments report improved aggregate goodput over several routing baselines. However, the distinction from prior SLO-aware serving, the isolated contribution of its routing policy, and its effectiveness under high prefix reuse remain insufficiently established by the current experimental evaluation.

Critical strengths
  1. Addresses a practical problem: meeting request deadlines across heterogeneous GPU instances.

  2. Combines demand prediction, instance-performance estimation, and runtime correction in a deployable routing framework.

  3. Includes end-to-end experiments, component ablations, and robustness tests under workload changes.

Critical weaknesses
  1. Clarify the contribution relative to existing goodput-aware systems. JITServe already allocates just enough serving bandwidth to satisfy SLOs and refines predictions during generation. Cauchy includes goodput-weighted routing across heterogeneous GPU groups. These systems operate at somewhat different layers, but their overlap with GoodServe deserves clearer analysis. GoodServe’s JITServe-inspired comparison uses a custom earliest-available-slot routing analogue, which does not establish superiority over the full prior method. Please explain what additional capability the proposed routing policy provides and evaluate stronger alternatives under comparable configurations.

  2. Separate routing improvements from supporting mechanisms. The component ablations demonstrate that prediction, admission control, and migration contribute to performance. However, they do not establish that the weakest-feasible-instance rule is responsible for the advantage over other placement policies. Please hold the predictor, resource estimates, admission test, and migration mechanism fixed while varying only the routing decision. Comparing weakest-feasible, fastest-feasible, cache-aware, and capacity-weighted placement under this common setup would directly quantify the routing contribution.

  3. Evaluate realistic workloads with higher prefix reuse. GoodServe includes prefix reuse in its latency model, but its experiments emulate caching and report a 29% hit rate. SMetric provides motivation for broader evaluation: it reports roughly 80% reuse potential under an idealized cache and a 64% local cache-hit rate in one evaluated deployment. These measurements use different workloads and configurations, so they are not directly comparable. Nevertheless, they indicate that substantial reuse is an important agentic-serving setting. Please evaluate real, capacity-constrained caches across higher reuse levels, report cache-hit rates and migration-induced recomputation, and compare with SMetric under matched conditions where feasible.

Questions for authors
  1. What is the main new contribution beyond the SLO-aware allocation and goodput-aware routing principles in JITServe and Cauchy?

  2. How much improvement remains when competing routing policies use identical prediction, admission, and migration mechanisms?

  3. Does GoodServe retain its advantage on high-reuse agentic workloads with real cache management, particularly against SMetric?

1 comment

Thanks for the thorough review and the pointers to related systems.

W1/Q1. Contribution beyond JITServe and Cauchy

Thanks, and you are right about JITServe. Our description followed an early version; its NSDI'26 version (§4.3) extends GMAX across replicas that may run at different speeds by creating KK per-replica copies of each request. We correct the related work and the appendix.

The difference lies in what is coordinated. JITServe's core mechanism is intra-instance: it shapes batches by giving each request just enough bandwidth. Its replica choice is per-request, in that a request goes to the replica where it ranks highest; it does not move running requests across replicas, and drops requests only after a waiting timeout. GoodServe works across instances and coordinates across requests: it uses each request's slack to place it on the least capable feasible instance, keeping fast instances for tighter requests, rejects a request at arrival when no instance can meet its deadline, and migrates at-risk requests across heterogeneous instances. JITServe also refines length predictions during generation, as we do, and we do not claim this as a difference. The two compose: JITServe can schedule within each instance that GoodServe routes to.

Instead of our earlier earliest-available-slot analogue, we now directly compare weakest-feasible with fastest-feasible while keeping all other GoodServe components fixed (Sec. 2 of the General Response). Both strategies are SLO-aware, and weakest-feasible achieves up to 9.3% higher goodput, demonstrating the independent benefit of our routing rule. The comparison table in Sec. 1 of the General Response also contrasts GoodServe with JITServe and the other systems.

Cauchy decides which heterogeneous GPU combinations to provision for prefill/decode pairs and dispatches across combinations by goodput-weighted round-robin, with cost efficiency as the objective. GoodServe takes the pool as given and decides per request, so the two are complementary: Cauchy can provision the pool on which GoodServe routes.

W2/Q2. Separating the Routing Rule

Thanks. We ran this comparison (Sec. 2 of the General Response) under an identical predictor, resource estimates, admission test, and migration. Weakest-feasible achieves higher goodput than fastest-feasible at every SLO scale, by up to 9.3% (2.82 vs. 2.58 RPS at SLO scale 3.0), and the gap widens as SLOs loosen. Of the other rules you name, cache-aware placement is represented by Preble, one of the baselines in our main comparison, and GoodServe already accounts for prefix hits through Hr,gH_{r,g} in its feasibility test (see W3/Q3). Capacity-weighted placement would pick a feasible instance in proportion to its capacity; like fastest-feasible, it ignores how much slack a request has, which is exactly what just-enough exploits.

W3/Q3. High Prefix Reuse and SMetric

Thanks. We first clarify why our hit rate is 29%. Our workload consists of single-turn requests with 460 prompt tokens on average, and 29% is the hit rate measured under our routing. SMetric's ~80% is the ideal reuse ratio of multi-turn coding-agent sessions under an infinite cache, where every turn resends the growing history; its v2 reports 82% for these sessions and about 50–70% for non-agentic workloads. The two numbers thus measure different quantities on different workloads.

We agree that high reuse is an important setting. GoodServe explicitly incorporates the prefix-cache hit length on each candidate instance into its routing decisions to accommodate the high prefix reuse characteristic of agentic workloads.

Locality and just-enough do not conflict in GoodServe, because reuse enters T(r,g)T(r,g) through Hr,gH_{r,g}: higher prefix reuse reduces prefill latency and can make weaker instances feasible, so locality is traded against keeping fast instances free in the same unit, latency. While SMetric focuses on balancing cache reuse and instance load in homogeneous clusters, GoodServe additionally accounts for differences in serving capabilities across heterogeneous instances. SMetric also relies on a cluster-wide KV tier (Mooncake and LMCache over RDMA), which our testbed lacks. We acknowledge the limited evaluation of high-reuse workloads and leave a comprehensive comparison under matched conditions to future work.

We hope these changes address your concerns, and would be grateful if you would reconsider your score.

Reviewer_q13MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GoodServe, a scheduling system for agentic LLM serving that aims to improve request-level goodput. GoodServe predicts output lengths using a TF-IDF/MLP-based mixture-of-experts model and estimates backend latency using historical measurements. It then uses these estimates, together with cache information, to guide request admission, routing, and migration across serving instances. The results show up to 31.0% higher goodput than the best evaluated baseline.

Critical strengths
  1. Comprehensive experimental evaluation. The paper provides extensive experiments across different models, serving instances, and deadline settings.
  2. Clear problem formulation. The paper clearly defines request-level goodput based on end-to-end deadlines and considers practical factors such as queueing delays, output lengths, and prefix-cache reuse.
Critical weaknesses
  1. The paper focuses on request-level goodput, but this metric may not fully reflect the performance experienced by users of agentic applications. In practice, users care more about the completion time of an entire workflow than that of individual LLM requests. For example, faster SQL generation does not necessarily lead to faster completion of a database-related task. The authors should discuss this limitation or evaluate whether the improvements in request-level goodput translate into better workflow-level performance.
  2. The ablation studies in Fig. 7 examine prediction, admission control, and migration, but do not evaluate the weakest-feasible routing policy itself. A comparison with other routing strategies, such as fastest-feasible routing, would help clarify the contribution of this design choice.
Questions for authors
  1. How are per-call deadlines related to workflow-level budgets, and what evidence shows that request-level goodput gains improve end-to-end workflow performance?
  2. With prediction, resource estimation, admission control, caching, and migration held fixed, how does weakest-feasible routing compare with fastest-feasible routing in goodput?
1 comment

Thanks for the positive assessment of our formulation and evaluation.

W1/Q1. Request-level Goodput vs. Workflow-level Performance

Thanks. We first clarify our scope: GoodServe is a serving-layer router for agentic inferences, and an agentic inference is not a workflow. The agent framework owns the workflow, including its control flow, tool calls, and how its time budget is spent, and it sends each LLM call to the serving layer with a deadline. This separation matches how agent frameworks and serving systems are deployed today, and lets the serving layer improve every workflow without knowing its structure.

Per-call deadlines come from the workflow budget. A framework splits the budget into per-call deadlines; HexGen-Flow, for example, assigns each call the remaining budget scaled by that call's share of the remaining expected compute, and recomputes this after every call. Under any such split, a workflow meets its budget if each of its calls meets its deadline, so per-call goodput is a conservative proxy for workflow completion, and raising it raises the fraction of workflows whose calls all finish in time. GoodServe accepts an arbitrary per-request DrD_r and thus composes with any splitting policy. Splitting also gives different calls different slack, which is exactly what just-enough routing exploits.

On the SQL example, we agree that a faster SQL generation does not necessarily speed up the database task. GoodServe does not try to make calls faster than needed: it makes each call meet its deadline and spends the fast capacity it saves on other calls, and a call finishing earlier than its deadline gains nothing in our objective.

We note as a limitation that a call missing its deadline does not always delay its workflow (e.g., when a later step has spare time), so per-call goodput can be conservative, and that choosing the budget split is outside the scope of a serving-layer router. This discussion and the budget-splitting interface will be added to Sec. 2 and the limitations.

W2/Q2. Weakest-feasible vs. Fastest-feasible with Everything Else Fixed

Thanks. We ran exactly this comparison (Sec. 2 of the General Response): prediction, resource estimation, admission control, cache information, and migration are held fixed and only the selection rule changes. Weakest-feasible achieves higher goodput than fastest-feasible at every SLO scale, by up to 9.3% (2.82 vs. 2.58 RPS at SLO scale 3.0), and the gap widens as SLOs loosen.

We hope this addresses your concerns, and would be grateful if you would reconsider your score.

Reviewer_a4YgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GoodServe, an SLO-aware routing method for agentic LLM serving on heterogeneous GPUs. It predicts the request length using a lightweight MoE model and estimates the latency through an EMA-based estimator. The system routes the requests to different GPUs based on the predictions, combined with the token-ID transfer method for request correctness. The authors conduct the experiments on 4 heterogeneous GPUs to show goodput improvements over existing methods.

Critical strengths
  1. The paper indicates that the problem of scheduling agentic task requests to heterogeneous GPU clusters is a significant problem. Reaching goodput in these systems will provide huge benefits.
  2. The proposed system combines output-length prediction, resource estimation, SLO-aware routing, and runtime migration into a practical implementation built on vLLM.
Critical weaknesses
  1. The claimed global goodput routing design is not sufficiently justified. The framework routes each new request individually to the weakest GPU that is predicted to meet its deadline. This is still a greedy heuristic, and there is no ablation comparing it with fastest-feasible routing under the same predictor and migration. Also, the paper’s own results don’t support its claim that the routing principle carries most of the gain. In Appendix A.3, just-enough routing with the history-based predictor only reaches 2.23 RPS, lower than the best baseline (2.27), and the predictor is the largest contributor in the ablation (−33.9%).
  2. The comparison with baselines is not fair enough to support the reported gains. I notice that only GoodServe rejects requests that are predicted to miss their deadlines, while the baselines have no SLO-based admission control, and this component alone contributes 7.7% goodput. The authors construct the only SLO-aware baseline and reach 1.79 RPS, even lower than the best SLO-unaware baseline (2.27). Also, the testbed has only one instance per GPU type, and the main experiments do not report the arrival rate. So it is still unclear how much of the 31.0% improvement comes from the routing design.
  3. The novelty of the framework is limited. The main components of GoodServe have been explored in prior works. Aladdin [1] and TetriInfer [2] place requests based on predicted output length, and Aladdin already uses a best-fit rule that sends each request to the most loaded worker that can still meet its SLOs, which is very close to the just-enough rule. Mooncake [3] rejects requests early, and Llumnix [4] migrates requests across instances. However, Aladdin and TetriInfer are not cited, and Mooncake’s early rejection is not mentioned. Also, HexGen-Flow [5], which the paper only cites for the SLO setting, already schedules agentic Text-to-SQL requests with end-to-end deadlines on heterogeneous GPU clusters. So the claim of being the first routing work for agentic inference with TTLT as the primary metric is questionable.
  4. The selected workload and the serving performance model don’t capture the complexity of agentic serving. I notice that the requests are single-turn prompts replayed with Mooncake timestamps, which don’t form closed-loop agentic execution with reasoning, tool calls, and growing context. The average prompt is only 460 tokens (Sec. 5.1), although Sec. 3.3 argues that agentic inferences have long contexts. Also, Eq. 2 treats q_g, p_g, and d_g as constants. It misses the slowdown that a new request brings to the co-located requests, as well as prefill/decode interference and KV cache pressure. The re-prefill cost of token-ID migration is not clearly quantified either. So it is still unclear whether the gains hold under real agentic workloads.

[1] Aladdin: Joint Placement and Scaling for SLO-Aware LLM Serving. arXiv 2024.

[2] ShuffleInfer: Disaggregate LLM Inference for Mixed Downstream Workloads. ACM TACO 2025.

[3] Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. FAST 2025.

[4] Llumnix: Dynamic Scheduling for Large Language Model Serving. OSDI 2024.

[5] HexGen-Flow: Optimizing LLM Inference Request Scheduling for Agentic Text-to-SQL. arXiv 2025.

Questions for authors

No additional questions.

4 comments

Thanks for the detailed review, which identified several issues that we have now fixed. Due to the length limit, our response is split into two parts.

W1. Justification of the Routing Rule

Thanks. We ran the comparison you asked for (Sec. 2 of the General Response): only the selection rule changes, while the predictor, admission test, cache information, and migration stay identical. Weakest-feasible achieves higher goodput than fastest-feasible at every SLO scale, by up to 9.3% (2.82 vs. 2.58 RPS at SLO scale 3.0), demonstrating the independent contribution of our routing rule.

You are right that the 2.23 vs. 2.27 result in Appendix A.3 does not show that the rule alone beats the best baseline. We remove that argument, together with the corresponding claim in the Introduction, and measure the contribution of the rule directly with the isolation experiment instead.

We do not claim that prediction is secondary to routing; the two are complementary. Just-enough routing judges feasibility from the predicted completion time, so with a poor predictor it places requests on weak instances that cannot meet their deadlines. Handling uncertainty in output length and instance performance is a key challenge of end-to-end serving, and the large drop without our predictor (−33.9%) reflects exactly that: the predictor makes the coordination rule usable, and the isolation experiment shows what the rule adds on top of it.

We agree that just-enough is an online greedy heuristic and no longer claim global optimality. Unknown future arrivals and different instance occupancy durations can make either weakest-feasible or fastest-feasible suboptimal. However, under nested feasibility and fixed slots without departures, weakest-feasible is provably optimal while fastest-feasible is not.

W2. Fairness of the Comparison

Thanks. Admission control is part of our coordination design, since capacity spent on requests that cannot meet their deadlines is lost to goodput. To separate its effect, we compare GoodServe without admission control against all baselines (Llama-3.1-8B, goodput in RPS, same setting as the isolation experiment; the best baseline at each scale is taken from Fig. 6). Without admission control, GoodServe still outperforms the best baseline by 12.8–26.5% across the evaluated SLO scales. In the isolation experiment, every rule uses the same admission test.

SLO scaleGoodServeGoodServe w/o admissionBest baseline
1.01.121.080.92
2.02.382.291.81
3.02.822.642.34

The SLO-aware baseline at 1.79 was our own adaptation of JITServe's principle, and we agree it is not a strong baseline. It is replaced by the isolation variants, which are SLO-aware by construction and share all of GoodServe's other components.

For the arrival rate, our main experiments replay the publicly available Mooncake trace in Sec. 5.1, with an average request arrival rate ranging from 3.7 to 4.0 RPS.

On one instance per GPU type: the rule exploits heterogeneity in speed rather than the number of instances. Additional instances of a type are simply more candidates in Cr\mathcal C_r, and routing stays at 5 ms per request with 512 instances at 10k RPS (Sec. 5.3). Our four types span four GPU generations, comparable to prior heterogeneous-serving studies such as Mélange. We note the single-instance-per-type testbed as a limitation.

(Part 2 of 2; Part 1 answers W1–W2.)

W3. Novelty and Prior Work

Thanks for the pointers. The comparison table in Sec. 1 of the General Response summarizes the differences; we detail each system below.

Aladdin is closest in spirit: its best-fit rule places a request on the most loaded worker that still meets its SLOs. However, these SLOs bound TTFT and the average per-token generation time (i.e., TPOT), not the end-to-end latency of the request. Its workers are also homogeneous, so "most loaded" ranks occupancy rather than hardware capability. Its goal is to minimize the number of GPUs, and when no worker fits it adds a worker instead of rejecting; it does not migrate requests. In contrast, GoodServe targets end-to-end deadlines in a fixed heterogeneous pool, preserving faster instances for requests with tighter deadlines to maximize overall goodput.

TetriInfer uses predicted length to spread decode load (power-of-two choices, then least interference), without deadlines.

Mooncake rejects requests early when predicted TTFT/TBT load would violate its SLOs, to avoid overload. GoodServe rejects a request when no heterogeneous instance can meet its end-to-end deadline under the predicted output length, so that capacity goes to requests that can count toward goodput. We now credit Mooncake for early rejection.

Llumnix migrates requests across homogeneous instances via KV-cache transfer for load balancing, de-fragmentation, and priority. GoodServe instead migrates requests across heterogeneous instances when their deadlines are at risk, using token-ID transfer, which is more practical under limited network bandwidth.

HexGen-Flow: we agree that it already schedules agentic calls with end-to-end deadlines on heterogeneous GPUs, and remove our "first" claim. It differs from GoodServe in three ways. First, its dispatcher does not use the deadline: it scores instances by (1−α)β/tqueue−αtcomp(1-\alpha)\beta/t_{queue}-\alpha t_{comp}, with a single weight α\alpha that trades queueing against execution time and is tuned to minimize P95 latency, while deadlines only order the local queues. Minimizing tail latency is not the same as maximizing goodput: a request either meets its deadline or not, so finishing early gains nothing, and GoodServe instead spends each request's slack on keeping fast instances free for tighter requests. Second, it admits every request and does not migrate running ones: estimation errors are passed on as tighter budgets for later calls of the same query, and a slow instance is handled by timeout and retry. GoodServe rejects requests that no instance can serve in time and migrates running requests at risk. Third, it is designed for Text-to-SQL pipelines and estimates output length from the input length, whereas GoodServe predicts it from the request content for general agentic requests and refines the prediction during generation.

W4. Workload and Performance Model

Thanks. GoodServe serves agentic inferences, not entire agent workflows: tool calls, control flow, and workflow-level budget allocation are managed by the agent framework, while each LLM call reaches the serving layer with its own deadline.

We agree that the 460-token average prompt does not support our "long context" justification, and we remove it. Even so, token-ID migration is still necessary in a heterogeneous pool, because KV-cache layouts differ across instances (our V100 instance runs with TP=2, the others with TP=1), so KV must be re-sharded before reuse, and in our deployment moving it is also much slower than recomputing it, as quantified below.

qgq_g, pgp_g, and dgd_g are not constants: they are re-estimated online from observed latencies with an EMA (Sec. 3.2), so load-induced slowdown, including prefill/decode interference and KV pressure, is reflected in later decisions, and the runtime checks correct the remaining error. We add as a limitation that the slowdown a new request causes to co-located requests is not modeled at decision time.

We quantify the re-prefill cost with Fig. 9 (Sec. 5.3). Migrating a request by token IDs, including the re-prefill on the destination, takes 0.10–0.47 s for 500–3,000 tokens, while transferring its KV cache takes 1.2–3.8 s over the same range, i.e., 7.1–15.3× longer. The migration check charges the re-prefill time against the remaining deadline (see our response to Kk69, W3), so a request is moved only if it can still finish in time.

We hope these clarifications and new results address your concerns, and would be grateful if you would reconsider your score.

Thanks for the additional experiments and clarifications. The new comparisons address some of my earlier concerns. However, with several of the stronger claims on optimality and novelty withdrawn, I still find the remaining contribution somewhat limited.

I am also not fully convinced by the evaluation. Although the paper is motivated by agentic inference serving, the experiments still rely on independent benchmark requests with an average prompt length of only 460 tokens. The authors acknowledge this limitation, but do not provide longer-context experiments or evaluation using actual agent-generated inference traces. The results under more demanding concurrent serving conditions also remain limited, while the large-scale simulation only measures routing overhead.

I appreciate the authors' efforts in the rebuttal, but these remaining concerns are significant enough that I will maintain my original score.

Thank you for the follow-up and for acknowledging the new comparisons. We could not run new serving experiments in the remaining time, so we answer the evaluation concern with statistics from real agent traces and state precisely what our workload does and does not cover.

1. Prompt length in real agentic traces. We analyzed 20,009 trajectories (535K LLM calls) from the public SWE-agent-trajectories dataset (Nebius, CC-BY-4.0), produced by SWE-agent solving repository-level issues. You are right that the accumulated context is long: 11.7K–12.3K tokens per call on average (P90 ≈ 25K). However, about 95% of these prompt tokens are a prefix carried over from the previous call of the same trajectory (assuming append-only history). With prefix caching, which vLLM and SGLang both support, the tokens a call must actually prefill are its increment: 530–575 on average (median 239–299), and each call's output averages 104–109 tokens (median ≈ 80). The per-call compute exercised by our 460-token workload is therefore representative of these calls. What our workload does not exercise is the long reusable context itself, which makes cache affinity and memory footprint matter. GoodServe's latency model already prices this: in T(r,g)=qg+pg (Lin−Hr,g)+dg LoutT(r,g) = q_g + p_g\,(L_{\mathrm{in}} - H_{r,g}) + d_g\,L_{\mathrm{out}}, sending a call away from the instance that holds its context changes the prefill term from about 0.5K to about 12K tokens, and the router compares the two options in the same latency units. We have not yet replayed these traces on our testbed; this will be the first experiment added to the revision.

2. Concurrency and simulation. We agree that the high-load evidence is limited, and that the large-scale simulation only shows that routing overhead does not become a bottleneck (5 ms per decision at 512 instances and 10k RPS). Simulating goodput at that scale requires a validated model of batching interference, which we do not claim. The revision states this scope explicitly.

3. Contribution reiteration. The withdrawn statements were overclaims about the deployed system (global optimality and "first"). Our contribution itself is unchanged and has three substantial parts that only work together.

(i) Placement principle. On a heterogeneous pool with per-request end-to-end deadlines, each request should use the weakest instance that still meets its deadline, which keeps fast capacity for the requests that only it can save. As the comparison table in our rebuttal shows, best-fit routers (Aladdin, PolyServe) fit on load over a single GPU type, and routers that do see both heterogeneities (HexGen-Flow, HexAGenT) still place speed-first.

(ii) Estimation built for this rule. Because the rule spends each request's slack, an underestimated output length directly becomes a missed deadline. Our predictor is designed for this setting: its gate recognizes the implicit agentic task type, which dominates output length; it runs on the router for every request at 2.5 ms; and it re-predicts from the generated tokens during execution. It reduces error by 1.4× relative to an LLM-based predictor and by up to 3.8× relative to a history-based one, and an oracle predictor adds only 5.1% goodput on top of it. Unlike MoPE, which predicts once on arrival to score fairness inside one server, our predictor drives cross-instance feasibility checks and rectification. Instance capabilities are estimated as black boxes, without engine changes.

(iii) Rectification across heterogeneous engines. Requests at risk are migrated by token IDs, which is 7.1–15.3× faster than KV-cache migration and independent of the engines' KV layouts; removing migration costs 24.0% goodput.

We appreciate the detailed engagement.

Reviewer_Kk69OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GoodServe predicts request length and instance performance, assigns requests to the slowest instance expected to meet their deadlines, and migrates requests that become at risk. The goal is to improve goodput on heterogeneous GPU deployments while preserving faster instances for requests that need them.

Critical strengths
  • The scheduling intuition is sensible. Sending every feasible request to the fastest available GPU can consume capacity that would be more valuable for a later, tighter-deadline request. GoodServe turns this intuition into an implemented system with prediction and migration.

  • The evaluation includes component ablations, bursty traffic, and predictor transfer across models. These are useful checks, and the reported goodput improvements suggest that the overall system deserves further investigation. The attempt to account for heterogeneous hardware is also practically relevant.

Critical weaknesses
  • The global-optimality claim is not established. Weakest-feasible selection is an online heuristic. Future arrivals and queue interactions can change which assignment is best, and Equation 1 does not explicitly capture the queue dynamics or the implemented rejection rule. Please narrow the claim to the demonstrated heuristic, or state assumptions under which an optimality result can actually be proved.

  • The experiments do not isolate the routing rule. The current ablations show that prediction and migration matter, but do not establish the benefit of weakest-feasible placement itself. Please compare it with fastest-feasible routing under the same predictor, admission policy, cache information, and migration mechanism. This is the most direct way to test the paper's central scheduling argument.

  • Migration feasibility and timing need clarification. Does the check use the remaining deadline and include destination queueing, transfer, and re-prefill of already generated tokens? Figure 9 should state exactly which operations are timed. Results on 10 Gbps Ethernet support a conclusion for that setup, but do not establish that moving token IDs is generally faster than transferring KV states.

Questions for authors

See weaknesses.

1 comment

Thanks for the careful review, and for finding the scheduling intuition sensible and the robustness checks useful.

W1. Global Optimality

Thanks, we agree and no longer claim optimality. The revised Eq. 1 allows rejection, ∑gxr,g≤1\sum_g x_{r,g}\le 1, matching Alg. 1; it makes queuing depend on earlier placements, T(r,g)=qg(x<r,ar)+pg(Lrin−Hr,g)+dgLroutT(r,g)=q_g(\mathbf x_{<r},a_r)+p_g(L_r^{in}-H_{r,g})+d_g L_r^{out}; and it requires xrx_r to be decided at arrival time ara_r without knowledge of future requests. Just-enough is presented as an online heuristic for this problem.

In fact, no online rule is optimal. Take a fast instance SS (1 time unit per request) and a slow instance WW (2 units), one slot each; deadlines are absolute times. Request AA arrives at t=0t=0 with deadline 2 and is feasible on both. If A→WA\to W, then BB (t=1t=1, deadline 3) and CC (t=1.2t=1.2, deadline 2.5) arrive and only one of them can be served, whereas A→SA\to S, B→WB\to W, C→SC\to S serves all three. If instead A→SA\to S, then B′B' (t=0.1t=0.1, deadline 1.2) misses its deadline, whereas A→WA\to W, B′→SB'\to S serves both. Every online rule therefore loses on some arrival sequence; we add this example to the appendix.

We also add a proposition stating when weakest-feasible is optimal. Suppose instances are ordered by speed so that feasibility is nested (feasible on gg implies feasible on every instance at least as fast), each instance has a fixed number of slots, and requests do not leave within the horizon. Then weakest-feasible serves the maximum number of requests for every arrival order, whereas fastest-feasible can serve only half of the optimum, which is tight. The proof is short: let II be the fastest instance that has a free slot at the end; since requests do not leave, that slot was free throughout. Every rejected request, and every request placed on a faster instance, is feasible only on faster instances, since otherwise nestedness would make it feasible on II and weakest-feasible would have placed it on II or a slower instance. Any assignment therefore serves at most as many of these requests as the faster instances have slots, all of which weakest-feasible fills, and weakest-feasible also serves all the others. The counterexample shows that departures are what break optimality, because a slow instance holds a request longer. The proposition explains the design; the experiment in Sec. 2 of the General Response tests it on the real system.

W2. Isolating the Routing Rule

Thanks. We ran exactly this comparison (Sec. 2 of the General Response): only the selection rule changes, while prediction, admission, cache information, and migration stay identical. Weakest-feasible achieves higher goodput than fastest-feasible at every SLO scale, by up to 9.3% (2.82 vs. 2.58 RPS at SLO scale 3.0), demonstrating the independent contribution of our routing rule.

W3. Migration Feasibility and Timing

Thanks. Every τ=50\tau=50 iterations, GoodServe re-predicts the remaining output length L^rrem\hat L_r^{rem} and migrates a request whose current instance is predicted to miss DrD_r. A destination gg is feasible if qg+pg(Lrin+Lrgen−Hr,g)+dgL^rrem≤Dr−(t−ar)q_g + p_g(L_r^{in}+L_r^{gen}-H_{r,g}) + d_g\hat L_r^{rem} \le D_r-(t-a_r), i.e., the check uses the remaining deadline and charges destination queuing and the re-prefill of all generated tokens. Under queue-free conditions, Fig. 9 compares the KV-cache transfer latency against the combined token-ID transfer and destination re-prefill latency for the same sequence length.

We agree that the 10 Gbps result does not establish general superiority, and narrow the claim. Token-ID migration costs a re-prefill of about pdst(Lin+Lgen)p_{dst}(L^{in}+L^{gen}), while KV migration costs about skv(Lin+Lgen)/BWs_{kv}(L^{in}+L^{gen})/BW plus setup, where skvs_{kv} is the KV size per token (128 KiB for Llama-3.1-8B in FP16) and BWBW is the throughput the KV path achieves end to end. Token IDs are thus cheaper when BW<skv/pdstBW < s_{kv}/p_{dst}. This holds in our testbed, whose instances lack high-bandwidth interconnects (Fig. 9: 7.1–15.3× lower latency for token IDs), but may not with GPUDirect RDMA or NVLink, so we limit the claim accordingly. A second reason is specific to heterogeneous pools: KV layouts differ across instances (our V100 instance runs with TP=2, the others with TP=1), so KV must be re-sharded before reuse, whereas token IDs are layout-independent; pipelined KV migration as in Llumnix assumes identical layouts.

We hope these changes address your concerns, and would be grateful if you would reconsider your score.

Comments on the paper

2 comments

We thank all reviewers for their constructive feedback. Reviewers found the problem practical (Kk69, a4Yg, 4N3S), the scheduling intuition sensible (Kk69), the formulation clear (q13M), and the evaluation extensive (q13M, 4N3S). Three concerns are shared: isolating the routing rule, the core contribution relative to prior work, and the fairness of the comparison. We address them here and answer the remaining points in the individual responses.

1. Core Contribution

GoodServe takes the view of an agentic service provider, whose objective shifts in two ways, each raising a challenge that existing routers do not address.

The first shift is from TTFT/TPOT to end-to-end latency. An agentic inference is useful only when its whole output arrives before its deadline, so routing must reason about completion time, which depends on an unknown output length and on instance speeds that drift with load. This brings uncertainty, which GoodServe handles by predicting output length, estimating instance speed from black-box observations, and correcting wrong placements by deadline-triggered migration.

The second shift is from a single request to total goodput. What matters is how many requests in the pool meet their deadlines, not how fast any one of them finishes. This calls for coordination: GoodServe places each request on the least capable instance predicted to meet its deadline, keeping fast instances for tighter requests, and rejects requests that cannot meet their deadline anywhere, so capacity is not spent on work that cannot count.

Individual mechanisms such as length prediction, early rejection, and migration have precedents, which we now cite and compare below. Prior deadline-aware systems use the deadline to decide whether a request is served or in what order; GoodServe also uses it to decide which hardware serves it, and contains the risk this creates through prediction and rectification.

SystemE2E deadlineHeterogeneous GPUsLength predictionSlack-aware placementAdmission controlRuntime migration
Aladdin✗✗✓✓✗✗
TetriInfer✗✗✓✗✗✗
Mooncake✗✗✗✗✓✗
Llumnix✗✗✗✗✗✓
HexGen-Flow✓✓△✗✗✗
JITServe✓✓✓△✗✗
GoodServe✓✓✓✓✓✓

Slack-aware placement uses a request's slack to choose its instance. △: HexGen-Flow estimates output length from input length only; JITServe uses slack to allocate bandwidth within an instance, but sends each request to its most favorable replica. Cauchy (GPU provisioning) and SMetric (cache locality) address complementary problems; see our response to 4N3S.

2. Isolating the Routing Rule

Following all four reviewers, we hold the predictor, instance estimator, admission test, cache information, and migration trigger fixed, and change only how an instance is chosen from the feasible set Cr\mathcal{C}_r, for both placement and migration targets (Llama-3.1-8B, goodput in RPS).

SLO scaleWeakest-feasible (GoodServe)Fastest-feasible
1.01.121.11
1.51.631.59
2.02.382.28
2.52.702.51
3.02.822.58

The goodput gap widens as SLOs loosen, with weakest-feasible outperforming fastest-feasible by up to 9.3%. This is because larger feasible sets provide more opportunities to preserve fast instances for requests with tighter deadlines.

3. End-to-end Comparison without Admission Control

To show that the gain does not come from rejection, we also compare GoodServe without admission control against all baselines. It still outperforms the best baseline by 12.8–26.5% across SLO scales, and admission control adds a further 3.7–6.8% (details in our response to a4Yg).

4. Corrections

Our description of JITServe followed an early version; its NSDI'26 version extends scheduling across replicas of different speeds (see our response to 4N3S). We remove our claim of being the first routing work for agentic inference with TTLT as the primary metric, since HexGen-Flow already routes agentic Text-to-SQL calls with end-to-end deadlines on heterogeneous GPUs (see our response to a4Yg). Finally, we remove the "globally optimal" wording and present just-enough as an online heuristic with a provable special case (see our response to Kk69).

Summary of Revisions

Eq. 1 now allows rejection and states the online setting, and a new appendix adds the counterexample and the proposition; Sec. 5.1 reports the arrival rates; Sec. 5.3 and the appendix add the routing-rule comparison and the results without admission control; the related work is expanded, with the comparison table above in a new appendix, and corrects our descriptions of JITServe and HexGen-Flow; Sec. 3.3 and Sec. 5.3 clarify the migration check and what Fig. 9 measures; and a new limitations paragraph covers co-located interference, workflow budget splitting, the one-instance-per-type testbed, and high-reuse workloads.

Follow-up: why the gain of weakest-feasible selection is modest on our testbed, and when it is larger

Sec. 2 of the General Response shows that, with everything else fixed, weakest-feasible selection beats fastest-feasible selection by 0.9–9.3%. Here we explain this magnitude.

Why the gain is small at tight SLOs. The rule matters only when a request has more than one feasible instance. At SLO scale 1.0, most requests can meet their deadlines only on the fastest instances, so both rules pick the same instance and differ by 0.9%. As deadlines loosen, feasible sets grow, weakest-feasible selection keeps fast instances free for the requests that need them, and the gain rises steadily to 9.3% at scale 3.0. At scales 2.5 and 3.0, swapping only the rule removes about half of GoodServe's advantage over the best baseline (0.19 of 0.35 RPS and 0.24 of 0.48 RPS), more than removing admission control does at scale 3.0 (0.18 RPS).

Why our testbed understates it. Our evaluation gives all requests one deadline budget, so they differ in urgency only through their lengths, and the testbed has one instance per GPU type. The rule gains most when requests of different urgency compete for fast instances. In the extreme, under the conditions of Proposition 1 (Appendix A.3), fastest-feasible selection can serve only half as many requests as weakest-feasible selection, which is optimal there.

Simulation. To quantify the gain between these extremes, Appendix A.3 now includes a simulation with a stylized continuous-batching model. As noted in our reply to Reviewer a4Yg, we do not claim a validated model of batching interference; the simulation only compares the two rules under identical assumptions. Gain of weakest- over fastest-feasible goodput (mean of five seeds; a load of 1 is the arrival rate that the four instances sustain at a batch of 16 each):

DeadlinesLoad 0.61.01.62.02.5
One budget for all requests, 1× (as in Sec. 5)2.4%4.2%5.6%6.3%6.8%
One budget for all requests, 3× (as in Sec. 5)3.6%4.2%4.3%4.9%5.4%
Each request at 3× its own latency0.0%0.0%0.4%−0.1%−0.2%
25% of requests at 1×, 75% at 3×26.3%26.0%4.5%0.8%0.5%
50% of requests at 1×, 50% at 3×58.1%45.0%31.5%18.0%3.8%

With one budget for all requests, the simulated gain (2–7%) is close to the testbed. When requests of different urgency share the pool, such as interactive and background agents, the gain reaches 26% and 58% at moderate load; it shrinks under heavy overload, where most requests are rejected under either rule, and vanishes when every request has the same slack relative to its own size.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.