GoodServe: Globally-coordinated Routing of Agentic LLM Inferences for High Goodput

Agentic applications consume LLM inferences as complete outputs: a code patch or a database query is useless until its last token arrives, so what matters to the application is end-to-end latency against a deadline, and what matters to the operator is goodput, the number of inferences that finish within their deadlines. Operators usually serve these inferences on pools of heterogeneous GPUs, and every request must be routed to one instance on arrival. Existing routers are greedy: each request goes to the instance that would serve it fastest or is least loaded, which spends the strongest GPUs on requests that would have met their deadlines on weaker ones and leaves urgent requests waiting. Goodput instead calls for globally coordinated routing, in which a request may be placed sub-optimally for itself so that the pool meets more deadlines overall. Coordination needs two quantities a greedy router never estimates: how long each request will run, since its output length is unknown until it finishes, and how fast each instance currently is. GoodServe formulates routing as a global goodput problem whose latency model couples both, including prefix-cache reuse, and solves it in a predict-and-rectify manner: a mixture-of-experts predictor gated by task type estimates output length accurately and efficiently; an EMA-smoothed black-box estimator tracks each instance's queuing delay and per-token speed; a just-enough rule sends each request to the weakest instance that can still meet its deadline and rejects requests that no instance can serve in time; and periodic risk checks migrate endangered requests by token IDs, 7–15× cheaper than transferring KV cache. On a four-GPU heterogeneous testbed serving three agentic benchmarks, GoodServe attains the highest goodput at every SLO level, up to 31.0% over the best existing router, and its routing overhead stays at 5 ms per request at 10,000 requests per second across 512 simulated instances.