RouterShift: Evaluating LLM-Routing Under Non-Stationary Traffic

Deployed LLM routers must keep choosing a good model as conditions change: models are constantly updated, the query context drifts, and serving latency rises under load. Standard routing benchmarks assume none of this happens. We present RouterShift, a benchmark that treats routing as an ordered stream with controlled, labeled changes in model quality, availability, query composition, and latency. We investigate twelve scenarios over $74,287$ queries and seven LLMs, across sixteen routing policies and three pool sizes ($k=3,5,7$). Our findings are as follows: i) offline routers lead both the regret and the latency rankings, but not on language and topic shifts; ii) contextual routers, which consume query embeddings, beat their non-contextual counterparts whether online or offline (post-shift regret $.254$ against $.326$ at $k=7$); iii) topic and language changes are the hardest shifts because they break the learned routing mapping. To speed recovery after a shift, we develop an adaptation layer (AdaptiveShift) that shortens an online router's memory or biases a frozen router's scores when the chosen model's reward drops, removing up to $11.8$ regret per $1,000$ steps. We release construction code, stream specifications, router implementations, and evaluation code that rebuild the corpus from public datasets.