COAST: From Support-Limited Retrieval Policies to Timely Execution on Shared RAG Stacks
Abstract
A budgeted RAG router commits twice: it proposes a retrieval configuration from logged evidence and then decides whether that work is still timely in the live shared stack. Conflating these decisions leaves a blind spot because logging support and average resource charge do not encode the queue and accelerator contention present at execution. COAST (Coverage-constrained Offline Allocation and Serving-time Tail control) turns this boundary into an explicit propose–admit interface. Its doubly robust proposal policy masks conditionally unsupported actions and selects a validation-feasible expected-charge point; its serving layer intersects that proposal with calibrated, monotone quantile-latency and violation-risk tests, while sustained drift changes proposals without bypassing admission. In matched Stack A runs that hold the SupportDR proposal policy and downstream stack fixed, adding admission changes EM/F1 from 51.2/65.3 to 51.3/65.4, while P99 moves from 304 to 252 ms and retrieval-budget violations from 1.4% to 0.42%. Across 22,328 Natural Questions, HotpotQA, and TriviaQA requests and five repetitions, full COAST reaches 51.4/65.6 EM/F1—paired differences of +0.3 [0.0,0.6]/+0.4 [0.1,0.7] against Dense+RR—while reducing P95/P99 from 278/412 to 156/248 ms, limiting violations to 0.4%, and staying within the predeclared request-attributed service-charge budget. At 2 load, its P95 increases by 11.2% versus 38.4% for Dense+RR; after replacing both generator and reranker, adding admission to SupportDR moves P95/P99 from 221/338 to 194/304 ms, and full COAST reaches 188/288 ms. These results make live admissibility a distinct design target for RAG: logged support determines what can be proposed responsibly, whereas request-state risk determines what should execute now.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.