- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GoodServe is a router for LLM inference requests with completion deadlines on heterogeneous GPU instances. It predicts output lengths, estimates instance performance, and selects the weakest instance expected to meet each request’s deadline. Requests predicted to miss their deadlines everywhere are rejected, while periodic checks migrate endangered requests using token IDs and destination recomputation. The latency model also accounts for prefix reuse. Experiments report improved aggregate goodput over several routing baselines. However, the distinction from prior SLO-aware serving, the isolated contribution of its routing policy, and its effectiveness under high prefix reuse remain insufficiently established by the current experimental evaluation.
Critical strengths
-
Addresses a practical problem: meeting request deadlines across heterogeneous GPU instances.
-
Combines demand prediction, instance-performance estimation, and runtime correction in a deployable routing framework.
-
Includes end-to-end experiments, component ablations, and robustness tests under workload changes.
Critical weaknesses
-
Clarify the contribution relative to existing goodput-aware systems. JITServe already allocates just enough serving bandwidth to satisfy SLOs and refines predictions during generation. Cauchy includes goodput-weighted routing across heterogeneous GPU groups. These systems operate at somewhat different layers, but their overlap with GoodServe deserves clearer analysis. GoodServe’s JITServe-inspired comparison uses a custom earliest-available-slot routing analogue, which does not establish superiority over the full prior method. Please explain what additional capability the proposed routing policy provides and evaluate stronger alternatives under comparable configurations.
-
Separate routing improvements from supporting mechanisms. The component ablations demonstrate that prediction, admission control, and migration contribute to performance. However, they do not establish that the weakest-feasible-instance rule is responsible for the advantage over other placement policies. Please hold the predictor, resource estimates, admission test, and migration mechanism fixed while varying only the routing decision. Comparing weakest-feasible, fastest-feasible, cache-aware, and capacity-weighted placement under this common setup would directly quantify the routing contribution.
-
Evaluate realistic workloads with higher prefix reuse. GoodServe includes prefix reuse in its latency model, but its experiments emulate caching and report a 29% hit rate. SMetric provides motivation for broader evaluation: it reports roughly 80% reuse potential under an idealized cache and a 64% local cache-hit rate in one evaluated deployment. These measurements use different workloads and configurations, so they are not directly comparable. Nevertheless, they indicate that substantial reuse is an important agentic-serving setting. Please evaluate real, capacity-constrained caches across higher reuse levels, report cache-hit rates and migration-induced recomputation, and compare with SMetric under matched conditions where feasible.
Questions for authors
-
What is the main new contribution beyond the SLO-aware allocation and goodput-aware routing principles in JITServe and Cauchy?
-
How much improvement remains when competing routing policies use identical prediction, admission, and migration mechanisms?
-
Does GoodServe retain its advantage on high-reuse agentic workloads with real cache management, particularly against SMetric?
Thanks for the thorough review and the pointers to related systems.
W1/Q1. Contribution beyond JITServe and Cauchy
Thanks, and you are right about JITServe. Our description followed an early version; its NSDI'26 version (§4.3) extends GMAX across replicas that may run at different speeds by creating K per-replica copies of each request. We correct the related work and the appendix.
The difference lies in what is coordinated. JITServe's core mechanism is intra-instance: it shapes batches by giving each request just enough bandwidth. Its replica choice is per-request, in that a request goes to the replica where it ranks highest; it does not move running requests across replicas, and drops requests only after a waiting timeout. GoodServe works across instances and coordinates across requests: it uses each request's slack to place it on the least capable feasible instance, keeping fast instances for tighter requests, rejects a request at arrival when no instance can meet its deadline, and migrates at-risk requests across heterogeneous instances. JITServe also refines length predictions during generation, as we do, and we do not claim this as a difference. The two compose: JITServe can schedule within each instance that GoodServe routes to.
Instead of our earlier earliest-available-slot analogue, we now directly compare weakest-feasible with fastest-feasible while keeping all other GoodServe components fixed (Sec. 2 of the General Response). Both strategies are SLO-aware, and weakest-feasible achieves up to 9.3% higher goodput, demonstrating the independent benefit of our routing rule. The comparison table in Sec. 1 of the General Response also contrasts GoodServe with JITServe and the other systems.
Cauchy decides which heterogeneous GPU combinations to provision for prefill/decode pairs and dispatches across combinations by goodput-weighted round-robin, with cost efficiency as the objective. GoodServe takes the pool as given and decides per request, so the two are complementary: Cauchy can provision the pool on which GoodServe routes.
W2/Q2. Separating the Routing Rule
Thanks. We ran this comparison (Sec. 2 of the General Response) under an identical predictor, resource estimates, admission test, and migration. Weakest-feasible achieves higher goodput than fastest-feasible at every SLO scale, by up to 9.3% (2.82 vs. 2.58 RPS at SLO scale 3.0), and the gap widens as SLOs loosen. Of the other rules you name, cache-aware placement is represented by Preble, one of the baselines in our main comparison, and GoodServe already accounts for prefix hits through Hr,g in its feasibility test (see W3/Q3). Capacity-weighted placement would pick a feasible instance in proportion to its capacity; like fastest-feasible, it ignores how much slack a request has, which is exactly what just-enough exploits.
W3/Q3. High Prefix Reuse and SMetric
Thanks. We first clarify why our hit rate is 29%. Our workload consists of single-turn requests with 460 prompt tokens on average, and 29% is the hit rate measured under our routing. SMetric's ~80% is the ideal reuse ratio of multi-turn coding-agent sessions under an infinite cache, where every turn resends the growing history; its v2 reports 82% for these sessions and about 50–70% for non-agentic workloads. The two numbers thus measure different quantities on different workloads.
We agree that high reuse is an important setting. GoodServe explicitly incorporates the prefix-cache hit length on each candidate instance into its routing decisions to accommodate the high prefix reuse characteristic of agentic workloads.
Locality and just-enough do not conflict in GoodServe, because reuse enters T(r,g) through Hr,g: higher prefix reuse reduces prefill latency and can make weaker instances feasible, so locality is traded against keeping fast instances free in the same unit, latency. While SMetric focuses on balancing cache reuse and instance load in homogeneous clusters, GoodServe additionally accounts for differences in serving capabilities across heterogeneous instances. SMetric also relies on a cluster-wide KV tier (Mooncake and LMCache over RDMA), which our testbed lacks. We acknowledge the limited evaluation of high-reuse workloads and leave a comprehensive comparison under matched conditions to future work.
We hope these changes address your concerns, and would be grateful if you would reconsider your score.