acceptodds
Under review as a conference paper at ICLR 2027

RouterShift: Evaluating LLM-Routing Under Non-Stationary Traffic

Abstract

Deployed LLM routers must keep choosing a good model as conditions change: models are constantly updated, the query context drifts, and serving latency rises under load. Standard routing benchmarks assume none of this happens. We present RouterShift, a benchmark that treats routing as an ordered stream with controlled, labeled changes in model quality, availability, query composition, and latency. We investigate twelve scenarios over queries and seven LLMs, across sixteen routing policies and three pool sizes (). Our findings are as follows: i) offline routers lead both the regret and the latency rankings, but not on language and topic shifts; ii) contextual routers, which consume query embeddings, beat their non-contextual counterparts whether online or offline (post-shift regret against at ); iii) topic and language changes are the hardest shifts because they break the learned routing mapping. To speed recovery after a shift, we develop an adaptation layer (AdaptiveShift) that shortens an online router's memory or biases a frozen router's scores when the chosen model's reward drops, removing up to regret per steps. We release construction code, stream specifications, router implementations, and evaluation code that rebuild the corpus from public datasets.

Primary area:
datasets and benchmarks
Keywords:
LLM routing, non-stationarity, benchmark, distribution shift, multi-armed bandits, online learning, change detection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_if1iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RouterShift is a benchmark for evaluating LLM routers with various kinds of shifts in consideration, such as query topic, model degradation, outages, and more. It further proposes AdaptiveShift, which is a simple wrapper that detects such shift-induced performance drops and improves router performance by reducing trust in stale history info.

Critical strengths
  • Shift is a very reflective environment that routers will face.
  • A broad coverage of shifts.
  • Transparent limitation section and a very reasonable constraints: this kind of shift problem can only be artificial + have accuracy signals, or real production + LLM judging.
Critical weaknesses
  • There exists prior work that has already studied many of the shifts that happen in this work (e.g ParetoBandit, mentioned by the authors). But there is no comparison showing why RouterShift's evaluation is capable of showing signals that are not already shown in ParetoBandit and other prior work that studied similar problems.
  • Most of the shifts studied here are model/system-side (e.g. degradation, latency, outage). It seems like these kinds of shifts can be simply studied by running the same static benchmark multiple times under different conditions, so I am not seeing the value of the proposed RouterShift beyond standardizing this evaluation. This comes back to the previous point, which is whether RouterShift is a better standardization than these prior attempts (intuitive adaptations of existing benchmarks and prior work studying similar problems), which remain unanswered.
  • It seems like the only query-side shifts studied are topic and language shifts, which seem relatively narrow and can also be done by running the same static benchmark as multiple sub-benchmarks.
  • Although disclosed, some types of shifts studied here are very roughly simulated. E.g. model degradation is literally done by flipping previously correct outcomes to incorrect ones, which is not reflective of real-world model degradation. A better approach might be downgrading to a smaller model / quantized model / same model but with less reasoning effort, so that we can really test whether a router is capable of being aware of such shifts and making online adjustments.
  • AdaptiveShift's performance looks mixed and often incremental with improvements, though I understand this is more of a benchmark paper so I am not putting much weight here.
  • Routers are often deployed to improve the cost–accuracy tradeoff, but this work does not evaluate that tradeoff. I am not too sure if I am reading it right, but the reported “cost” of this paper is just output token count, which ignores the cost differences between models, and there is no cost–accuracy frontier or fixed-cost comparison.

I am generally very open to authors improving the paper during rebuttal. My main concerns are the lack of comparison with prior evaluations and the realism of some of the simulated shifts. These may require fairly substantial additional work, so I am not sure if it is feasible to do right during rebuttal. If the authors can first provide a concrete plan for how they intend to address these concerns, I would try my best to provide quick feedback on whether I think the proposed additions would meaningfully address them, before sending the authors to invest significant effort in implementation.

Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Plan for rebuttal - Part 1/2.

Many thanks for the careful reading and for the offer to give feedback on a plan before implementation, which we gladly take up. Below we describe, for each weakness (W), what we plan to add. We would be grateful for a quick indication of whether these additions address the concerns, in particular on the two questions in the last paragraph. We split this answer into two responses due to word limit constraints.

Comparison with prior evaluations (W1, W2). The reviewer asks what RouterShift shows that ParetoBandit, other prior work, or repeated runs of a static benchmark do not. ParetoBandit evaluates its own router, together with its ablations, on a three-model portfolio under three shifts: a price drop of one model, a silent quality degradation of another, and the cold-start onboarding of a fourth model. RouterShift uses similar shifts but does not limit itself to them, since production traffic exhibits more. It adds gradual, recurring, transient, and concurrent schedules, changes in query composition (topic and language), latency changes, and judge recalibration, and it evaluates sixteen policies from four families on seven models in three pool sizes, over streams of up to 10,000 steps. We note that ParetoBandit also simulates quality degradation by lowering the stored reward of the affected model, so the real-degradation scenarios described under W4 would be new with respect to both.

We plan two additions. First, a comparison table with ParetoBandit, RouteNLP, RouterBench, LLMRouterBench, RouterArena, and TwinRouterBench, listing for each the shift factors covered, whether onsets are labelled, the number and families of policies, online or offline evaluation, latency, multilingual traffic, and release status. Second, a protocol comparison on the same data and the same sixteen policies under three protocols: (i) a static re-run, as the reviewer suggests, with pre-shift and post-shift conditions evaluated as separate i.i.d. benchmarks; (ii) the ParetoBandit protocol, replaying its three shifts in their three-phase structure on our matrix and scoring them with its oracle-relative quality; and (iii) RouterShift streams. We will report the ranking under each protocol, the Kendall τ between protocols, and the conclusions supported only by (iii).

We expect, and will state, that for frozen offline routers a static re-run of the post-shift condition is close to equivalent to the stream protocol. The value of the stream protocol concentrates on adaptive routers and on the transition period. A static re-run discards the pre-shift history, whereas for online routers the cost of a shift is largely stale history, as shown by the memory-horizon interaction (Appendix E), the detection delays (Table 10), and the dependence on gradual versus hard transitions and on recurring patterns (Figure 5). None of these quantities is defined in a static re-run. Beyond these measurements, the standardisation provides labelled onsets, magnitudes and affected arms, fixed seeds and stream specifications, and one interface for all policies, so that results remain comparable across papers.

Query-side shifts (W3). We agree that topic and language shifts cover only part of the query side. Using the existing labels, without new generations, we can add difficulty drift (traffic moving from easy to hard queries via d(x)), the arrival of an unseen task (a dataset withheld from offline training entering mid-stream), task-category ramps (e.g., a growing share of tool-use or SQL queries), and a prompt-length shift. We would welcome the reviewer's view on which of these would be most informative.

Realism of simulated degradation (W4). On a stratified subset of about 10k queries covering all nine task categories, drawn from the rows with complete labels, we will generate and label outcomes, with the same prompts and judge, for real degraded variants of existing arms: a smaller sibling (Qwen3-235B-A22B to Qwen3-30B-A3B-Instruct-2507 for a mild and Qwen3-4B-Instruct-2507 for a strong degradation, both from the same release, with the same tokenizer and chat template, so that only model capacity changes), lower reasoning effort (gpt-oss-20b from medium, its default, to low, which is also faster and therefore yields a degradation with a genuine accuracy–latency trade-off), and quantisation (Nemotron-Nano-9B-v2 from full precision to 4-bit). We target arms that the routers select frequently, as degrading a rarely selected arm produces little regret signal.

AuthorsOpenReview ↗

Plan for rebuttal - Part 2/2.

Cost–accuracy trade-off (W6). The reviewer's reading is correct: the current cost column is the output token count. We will replace it with price-weighted cost over input and output tokens, and add cost–accuracy curves before and after each shift, with the oracle curve and a random-mixture baseline, together with the accuracy at a fixed budget and the cost needed to match the best single model, as also requested by Reviewer DNse. Once cost is price-weighted, we will also add a price-shift scenario, in which one arm's per-token price changes mid-stream, so that RouterShift also covers the price drift evaluated by ParetoBandit.

Feasibility. All additions except W4 reuse the stored outcome matrix, and we expect to report their results during the discussion period. For W4, we will report results on the subset during the discussion period and extend them to the full corpus for the final version. In particular, we would appreciate the reviewer's view on two points: whether the protocol comparison under W1/W2 is the kind of evidence the reviewer had in mind, and which of the degradation variants under W4 the reviewer would find most convincing.

I thank the authors for proposing a detailed plan and regret that I couldn't get back to you sooner. I want to be upfront that I find some of your responses quite hard to read, e.g.

On a stratified subset of about 10k queries covering all nine task categories, drawn from the rows with complete labels, we will generate and label outcomes, with the same prompts and judge, for real degraded variants of existing arms

This seems to be saying something like "we will sample 10k questions across all nine categories, collect answers from genuinely degraded models, and judge them the same way."

We expect, and will state, that for frozen offline routers a static re-run of the post-shift condition is close to equivalent to the stream protocol. The value of the stream protocol concentrates on adaptive routers and on the transition period. A static re-run discards the pre-shift history, whereas for online routers the cost of a shift is largely stale history, as shown by the memory-horizon interaction (Appendix E), the detection delays (Table 10), and the dependence on gradual versus hard transitions and on recurring patterns (Figure 5).

This seems to be saying that routers that don't learn during use will not see much difference between streaming and running static benchmarks, while streaming helps study how learning routers adjust to changes.

Such difficulties mean I have to set aside chunks of time to understand and then reply. I don't want to speculate whether this response is heavily AI-generated, but I do hope the authors can put significantly more effort into producing more presentable rebuttal responses that are easy to read (as above), concise (e.g. you possibly don't need to explain that a model with lower reasoning effort would be "faster and therefore yields a degradation with a genuine accuracy–latency trade-off"), and nicely structured (right now, the plan, details, and commentary are often intertwined).

With that, I find the proposed plan to be on the right track, but I am not sure your Feasibility section's claim that all additions except W4 can reuse the stored outcome matrix covers the comparisons I intended. I expect to see RouterShift compared against ParetoBandit and the other static benchmarks you presented, with evaluation results from multiple runs under different conditions, not just differences in their coverage. My motivation is to confirm that RouterShift is able to discover meaningful signals that other benchmarks can't, not just that it is constructed differently. For instance, if I can just run the same static benchmark twice to observe signals of model degradation, I don't think I'd need RouterShift. Similarly, if I can just run tasks on different topics sequentially, I don't think RouterShift's topic-shift task is too meaningful. The same goes for RouterShift vs ParetoBandit: RouterShift might cover more types of shifts, but do they really reflect in the evaluation result?

I am also not sure I agree that a static rerun must discard the pre-shift history for an adaptive router: couldn't we preserve its learned state between runs? It seems like the streaming benchmark is just a standardization of a static benchmark with a particular problem order. I don't disagree that standardization would help, but unless there is evidence indicating such standardization requires meaningful design and engineering effort to result in useful benchmark signal, the proposed benchmark may remain a toy.

I feel like these are quite significant but meaningful asks. I am not sure it is possible to address them well during the rebuttal, and generally, with such significant updates, I'd recommend resubmission so that the presentation can be more polished and cohesive, instead of a bunch of patches. But I will carefully check any updates the authors post, provided they are presented in a much more reader-friendly way.

Reviewer_DNseOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce Routershift, a method for benchmarking routing methods that account for the natural distrubtion shifts that occur when dealing with LLM routing: namely tasks can update, models can drop out or be updated, and even judge labels can change. The authors point out that recent works such as ParetoBandit and RouteNLP attempt robustness to this phenomenon, but only benchmark their own routers so some kind of unified approach is needed. This is the main contribution of the paper: A corpus containing 74,287 labeled query records run against several candidate models + a structured set of controlled changes to the stored LLM outcomes the router will have to handle. The main validation is the evaluation of four groups of routing policies on Routershift, finding that offline methods generally outperform, and that results are sensitive to reward weighting.

Critical strengths

The main strength of the paper is the thoroughness of the types of routing distribution shift considered, and their comprehensive application to validating prior work in the area. The authors are correct in pointing out the number of methods that attempt this problem without some kind of universal benchmarking.

Critical weaknesses

The main sort of "hard to fix" weakness I see is that the distribution shift evaluation feels fairly independent of the corpus of questions the authors provide (maybe aside from the fact that their corpus includes multiple languages). I think a potential fair ask of the paper is more of a focus on shift methods, less on corpus and maybe applying their methods to other routing datasets to check for consistency for example. On the other hand it is sort of a strength if some of the methods are generally applicable.

An easier to fix weakness is a lack of focus on routing specific metrics. In router terms i think rolling accuracy and regret are hard to interpret, i think what we generally care about is the routing curve that traces a cost trade off, and measures for how much cost we can match or beat the best model. Additionally its nice to compare this to the oracle curve and a sort of random baseline curve that just assigns probability weightings to each model based on their cost/performance and the desired tradeoff.

Questions for authors
  • Can we see more routing specific metrics?
  • How specific are these methods to your corpus?
2 comments
AuthorsOpenReview ↗

Authors' response - Part 1/2.

Many thanks for the careful reading and for the constructive suggestions. We have run the analyses that the two questions (Q) ask for, and we report the results below together with the related weaknesses (W). We will present the outcome through two comments, due to limited amount of words in the answer boxes. The experiments used two datasets other than ours: RouterBench [1] (18,000 queries, 11 models, dollar cost as the second objective) and a new corpus of 10,000 queries from eight of our datasets, answered by Qwen3-32B and two small models and with latency modelled from per-token times measured on one GPU. They cover 17 routers: our policies except GraphRouter, with BaRP and Avenger as simplified re-implementations, plus two offline routers that we added, Ridge-acc and MLP-acc, which predict each model's probability of being correct.

Corpus specificity of the shift methods (Q2, W1). We agree that the shift layer is largely independent of the corpus, yet this is intended by design: it operates on any query-model outcome, optionally with latency. To test consistency, as the reviewer suggests, we applied five scenarios (silent regression, capability decay, recurring regression, transient outage, and judge recalibration) to RouterBench with pools of 3, 5 and 7 models, and to the new corpus with 3 models. Each scenario was swept over five intensities (0.05 to 0.60) and five onset shift positions, with ten stream orderings per condition. Post-shift accuracy at the standard point (intensity 0.25, onset at 0.4 of the stream), averaged over the five scenarios, is as follows.

                                      RouterBench            New corpus
                                      k=3    k=5    k=7      k=3
Shift-aware oracle                    0.832  0.885  0.902    0.859
Online family                         0.646  0.642  0.636    0.708
Offline family (as in the paper)      0.601  0.571  0.588    0.653
Offline, trained for accuracy         0.665  0.689  0.697    0.726
References (Random, StaticBest)       0.586  0.596  0.585    0.689

Two findings of the paper apply here: i) the online family is ahead of the paper's offline family and of the references in all the runs for the experiment. Notably, the oracle rises with the pool size while the online family does not, thus showing a gap to the oracle that widens: this behavior can be treated as a future opportunity in research, as expressed in Table 1. The inversion of the family ranking between correctness and latency-aware metrics under a latency shift is also shared from our previous experiments in the paper, with dollar cost in place of latency on RouterBench.

Interestingly, in the stationary setup from RouterBench the online advantage over the offline family does not grow with the pool size (+0.045, +0.071 and +0.048 for 3, 5 and 7 models). Without a shift, the online family leads the paper's offline family by 0.107 to 0.135 on RouterBench and by 0.133 on the new corpus, which is more than after a shift (0.045 to 0.071, and 0.055). This lead is therefore not caused by the shift. It comes from the training objective: our offline routers are trained on the fastest-correct target (Section 5.1), whereas the online routers are rewarded for correctness. With an accuracy comparison (following the same setup in RouterBench) the results favour the latter. Offline routers trained for accuracy (Ridge-acc and MLP-acc) are ahead of the online family by 0.02 to 0.07, with or without a shift. Measured against them, the part of the gap that the shift causes is positive but small: +0.007 to +0.013 on average over the five scenarios, +0.02 to +0.03 under a silent regression, +0.01 to +0.02 under capability decay, and close to zero for recurring regressions, transient outages and judge recalibration.

Thus, a comparison on a stationary benchmark, or after a shift but without a control, attributes to adaptation a difference that the routers' objectives produce. RouterShift separates these two cases, with the separation showing that adaptation pays off mainly after silent regressions and gradual decay. We will add the no-shift control and the accuracy-trained offline routers to the benchmark, report this decomposition in Section 5 and restate the comparisons between online and offline routers in terms of the part caused by the shift. We will also report, for each shift, the rank correlation between router rankings on our matrix and on RouterBench. The language scenario cannot be transferred, as RouterBench contains no multilingual queries.

[1] Hu, Qitian Jason, et al. "RouterBench: A Benchmark for Multi-LLM Routing System." arXiv:2403.12031 (2024).

AuthorsOpenReview ↗

Authors' response - Part 2/2.

Routing-specific metrics (Q1, W2). We agree that rolling accuracy and regret alone are hard to interpret in routing terms. We computed cost–quality curves on RouterBench, which has per-model dollar prices, for the 3-model pool (GPT-4, GPT-3.5 and Mistral-7B) and on three windows: before the onset, the first 20% after it, and the last 25% of the stream. Online routers trace their curve through the cost weight of the reward, while the offline BaRP router is retrained at each weight. We summarise each curve by its area gain over the random-mixture baseline (the AIQC of RouterBench [1]) and by the cost needed to reach 95% of the best model's pre-shift accuracy. The two shifts are a silent quality regression of GPT-4 (flip probability 0.25) and a threefold price increase of GPT-4, ramped over 30% of the stream. Means over 10 orderings are as follows.

                                  Area gain over mixture     $ per 1k queries to 95%
                                  before  early   late       before  early   late
Quality regression of GPT-4
  Cost-aware oracle               +0.18   +0.22   +0.21      0.16    0.14    0.14
  BaRP (offline)                  +0.02   -0.03   -0.03      1.21    never   never
  ParetoBandit-style (online)     -0.01   -0.05   -0.06      1.20    never   never
  CostTS (online)                 -0.02   -0.03   -0.03      2.83    never   never
Price of GPT-4 x3
  Cost-aware oracle               +0.18   +0.21   +0.21      0.16    0.14    0.14
  BaRP (offline)                  +0.02   +0.03   +0.01      1.21    1.84    3.38
  ParetoBandit-style (online)     -0.01   -0.03   -0.08      1.20    2.29    4.02
  Random mixture                   0       0       0         3.27    5.63    9.70

Before a shift, the learned offline router lies above the random-mixture baseline, while the online routers lie slightly below it. After the quality regression, all learned routers fall to or below the baseline, and none reaches 95% of the pre-shift quality, whereas the oracle's gain over the baseline grows from 0.18 to 0.22. After the price increase, the offline router keeps its gain, but the cost it needs to reach the target nearly triples. Thus, in this setting, the cost–quality view shows what accuracy and regret do not: after a quality regression a better trade-off exists, but no router recovers it. We will add these curves to the main text as the primary routing-quality summary, together with the 7-model results and with the accuracy at a fixed budget. We will retain post-shift regret, as the standard measure in the non-stationary bandit literature.

Finally, we will couple this analysis with a set of graphical plots related to the experimental results shown in the aforementioned table. In particular each plot includes the single-model points, the random-mixture baseline that the reviewer describes (the upper convex hull of the single-model points, which is the best that a probabilistic mix of models can achieve), and a cost-aware oracle that, for each query, picks the model maximising correctness minus a price times cost, swept over the price.

[1] Hu, Qitian Jason, et al. "RouterBench: A Benchmark for Multi-LLM Routing System." arXiv:2403.12031 (2024).

Reviewer_KidROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Authors aim to examine the effect of time varying query distribution, model performance and generation latency on routing decisions. They introduce a benchmark for evaluating LLM routers under changing conditions during test time. A change detector AdaptiveShift is also introduced. The improvements in performance are modest, and primarily existing routing approaches are examined without any fundamental contribution.

Critical strengths
  1. I appreciate the problem setting which examines time varying effects such as query distribution change, model latencies etc.
  2. The experimental setting is reproducible.
Critical weaknesses

There are significant and major weaknesses to the presented work.

  1. The online joint reward for accuracy and latency is completely arbitrary, and lacks justification and connection to examined metrics such as predicting fastest correct model. The half/half weight in the joint objective is not explained and justified.
  2. How is latency even measured? there is not clear consensus in experiments or appendix explanation on how latency is measured, how is the hardware/API latency measured, is the deployment even compared across models? what is the examined latency here? is it Time to first token (TTFT), or end-to-end generation latency? it is not specified.
  3. Some of the motivation such as model performance degradation over time is artificial, and lacks any clear explanation. In what scenarios does a given language model's performance degrade over time? that is something purely simulated by adding errors to simulated response scores in the presented experiments.
  4. What new routing algorithm does the paper propose? all examined versions are already existing solutions, they are simply used as wrappers in the experiments.
  5. there is significant gap between the motivation and presented evidence.
Questions for authors

All questions mentioned above. This paper requires major restructuring and thought before it can be considered for ICLR main tracks.

4 comments
AuthorsOpenReview ↗

Authors' response - Part 1/2.

Many thanks for the careful reading. We address the reviewer's concerns below, starting from the scope of the contribution, since it bears on the remaining points. We split the answer in two comments to further deep-dive into the clarifications, due to a wording limit constraint.

Scope of the contribution (reviewer summary and W4). We agree that RouterShift does not propose a new routing algorithm, however we do not claim one. The paper is submitted to the datasets and benchmarks area, and its contribution is the evaluation resource: a 74,287-query outcome matrix over seven LLMs with per-call latency, twelve shift scenarios with labelled factor, onset, magnitude and affected arms, and an interface that scores any router exposing select and update operations. Evaluating existing routers under a common protocol is therefore the purpose of the paper rather than a limitation of it. AdaptiveShift is a wrapper by design: Algorithm 1 is the "AdaptiveShift wrapper for a TS router", and Section 6 introduces it as a response that "retains the base selection rule". Its role is to show that the labelled shifts can isolate where an adaptation mechanism helps and where it hurts; for example, the MLP instantiation improves on five of six shift families but loses on language shift (Table 9). We report these mixed results deliberately, as a benchmark is more informative when it also exposes failures.

The benchmark also exposes behaviours that a stationary benchmark cannot show. Under a language shift, the frozen routers' gap to the oracle grows from 0.144 to 0.255 although no model changes, whereas the online routers move only from 0.074 to 0.096 (Section 5.3). Adding arms raises the oracle, but regret grows faster than the router-free headroom (+0.047 and +0.049 against +0.031, Table 1). The better forgetting rule between SW-UCB and ε-greedy reverses with shift magnitude and stream length (Appendix E). We will make this framing explicit in the Introduction and present Section 6 as a reference adaptation baseline.

Joint reward and the 0.5 weight (reviewer W1). The reviewer asks how the joint reward relates to the fastest-correct target and why the two terms are weighted equally. L-DCG requires knowing, for every query, which arms are correct and how fast each of them is. An online router never observes this under bandit feedback (Section 3.1), since it sees only the outcome of the chosen arm, and therefore needs a scalar reward that can be computed from that arm alone; Equation 6 is this surrogate. Offline routers, which train under full observability, are trained on the fastest-correct target directly (Section 5.1).

The 0.5 weight is not arbitrary. With weight w on correctness and normalised latency in [0, 1], a correct arm receives at least w and an incorrect arm at most 1 − w. For w ≥ 0.5, every correct arm is therefore ranked above every incorrect arm on a given query, and among correct arms the faster one ranks higher, which is the ordering that L-DCG rewards. w = 0.5 is the boundary case, i.e., the largest latency weight that never ranks an incorrect answer above a correct one. We will add this argument to Section 4.4, together with a sweep over w∈0.25,0.5,0.75,0.9,1.0w \in {0.25, 0.5, 0.75, 0.9, 1.0} reporting L-DCG and accuracy, which also traces each online router's accuracy–latency trade-off.

Latency measurement (reviewer W2). We agree that the definition is not explicit in the main text, but the serving details are currently only in Appendix K. Latency is the end-to-end wall-clock time per request, measured on the client side from submission to the return of the last generated token, in a separate pass without concurrent load, recorded once per query–model cell. This is the quantity a routing decision affects, since correctness is judged on the complete answer. Time to first token was also logged, as shown in the code provided. We will add a dedicated latency-measurement paragraph to Section 4 with the exact definition and per-arm latency distributions.

AuthorsOpenReview ↗

Authors' response - Part 2/2.

Realism of model degradation (reviewer W3). The reviewer asks in which scenarios a model's performance degrades over time. Degradation behind a stable endpoint is documented in practice: Chen et al. [1] measured substantial behaviour changes in GPT-3.5 and GPT-4 between March and June 2023 under the same API names. Then, providers update models behind stable aliases; serving-stack changes have caused publicly acknowledged quality regressions [2]. Finally, the same open-weight model can score differently across inference providers [3]. A router experiences all of these as one arm's reward dropping without notification, which is our silent-regression scenario.

We agree that our current instantiation is simulated, as stored outcomes are flipped either uniformly or with difficulty-conditioned probability δ(1−d(x))\delta(1- d(x)). This choice gives a known onset and magnitude and an exact regret reference, but, as also raised by Reviewer if1i, it should be complemented with real degradations. We will therefore add scenarios in which the affected arm's post-onset outcomes and latencies are the stored outcomes of a real degraded variant on the same queries (a smaller sibling model, a quantised checkpoint, or lower reasoning effort), and report whether router rankings under synthetic flips agree with those under real degradation. This directly tests whether the synthetic shifts are a valid proxy. In general we preferred sticking to the synthetic degradations to make RouterShift also a stress-testing routing tool, not only a benchmark.

[1] Chen, Lingjiao, Matei Zaharia, and James Zou. "How is ChatGPT's behavior changing over time?." arXiv preprint arXiv:2307.09009 (2023).

[2] Anthropic. "A postmortem of three recent issues." Anthropic Engineering, September 2025. https://www.anthropic.com/research/a-postmortem-of-three-recent-issues

[3] Moonshot AI. "K2 Vendor Verifier." GitHub repository, 2025. https://www.kimi.com/en/blog/kimi-vendor-verifier

Motivation and evidence (reviewer W5). To make the link between motivation and experiments explicit, we will add a table mapping each of the twelve scenarios to the deployment event it represents (checkpoint update, silent serving regression, outage, model onboarding, traffic drift, load-induced slowdown) and to the evidence for that event.

Presentation. We will restructure the results section around a single summary table of findings, move the per-policy grids to the Appendix, and simplify the notation in Section 3. We would be grateful if the reviewer could point to the parts that were hardest to follow, so that we can prioritise them.

Thank you authors for your response. I shall maintain my score.

Reviewer_if1iOpenReview ↗

Speaking as one reviewer to another

If you want to maintain your score, it should be basic courtesy to explain where the authors' rebuttal falls short. This is especially the case for a score of 1.

It is unprofessional and, in my opinion, borderline disrespectful of the authors' time and effort to reply to a full rebuttal with something as lacking in substance as "thank you ... I shall maintain my score", as you can literally copy-paste this response to any paper.

I don't disagree that this paper has its shortcomings, and the authors' rebuttal is also not the easiest read, but some level of care and substantive engagement should be shown.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.