- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RouterShift is a benchmark for evaluating LLM routers with various kinds of shifts in consideration, such as query topic, model degradation, outages, and more. It further proposes AdaptiveShift, which is a simple wrapper that detects such shift-induced performance drops and improves router performance by reducing trust in stale history info.
Critical strengths
- Shift is a very reflective environment that routers will face.
- A broad coverage of shifts.
- Transparent limitation section and a very reasonable constraints: this kind of shift problem can only be artificial + have accuracy signals, or real production + LLM judging.
Critical weaknesses
- There exists prior work that has already studied many of the shifts that happen in this work (e.g ParetoBandit, mentioned by the authors). But there is no comparison showing why RouterShift's evaluation is capable of showing signals that are not already shown in ParetoBandit and other prior work that studied similar problems.
- Most of the shifts studied here are model/system-side (e.g. degradation, latency, outage). It seems like these kinds of shifts can be simply studied by running the same static benchmark multiple times under different conditions, so I am not seeing the value of the proposed RouterShift beyond standardizing this evaluation. This comes back to the previous point, which is whether RouterShift is a better standardization than these prior attempts (intuitive adaptations of existing benchmarks and prior work studying similar problems), which remain unanswered.
- It seems like the only query-side shifts studied are topic and language shifts, which seem relatively narrow and can also be done by running the same static benchmark as multiple sub-benchmarks.
- Although disclosed, some types of shifts studied here are very roughly simulated. E.g. model degradation is literally done by flipping previously correct outcomes to incorrect ones, which is not reflective of real-world model degradation. A better approach might be downgrading to a smaller model / quantized model / same model but with less reasoning effort, so that we can really test whether a router is capable of being aware of such shifts and making online adjustments.
- AdaptiveShift's performance looks mixed and often incremental with improvements, though I understand this is more of a benchmark paper so I am not putting much weight here.
- Routers are often deployed to improve the cost–accuracy tradeoff, but this work does not evaluate that tradeoff. I am not too sure if I am reading it right, but the reported “cost” of this paper is just output token count, which ignores the cost differences between models, and there is no cost–accuracy frontier or fixed-cost comparison.
I am generally very open to authors improving the paper during rebuttal. My main concerns are the lack of comparison with prior evaluations and the realism of some of the simulated shifts. These may require fairly substantial additional work, so I am not sure if it is feasible to do right during rebuttal. If the authors can first provide a concrete plan for how they intend to address these concerns, I would try my best to provide quick feedback on whether I think the proposed additions would meaningfully address them, before sending the authors to invest significant effort in implementation.
Questions for authors
See weaknesses.
5 comments
Plan for rebuttal - Part 2/2.
Cost–accuracy trade-off (W6). The reviewer's reading is correct: the current cost column is the output token count. We will replace it with price-weighted cost over input and output tokens, and add cost–accuracy curves before and after each shift, with the oracle curve and a random-mixture baseline, together with the accuracy at a fixed budget and the cost needed to match the best single model, as also requested by Reviewer DNse. Once cost is price-weighted, we will also add a price-shift scenario, in which one arm's per-token price changes mid-stream, so that RouterShift also covers the price drift evaluated by ParetoBandit.
Feasibility. All additions except W4 reuse the stored outcome matrix, and we expect to report their results during the discussion period. For W4, we will report results on the subset during the discussion period and extend them to the full corpus for the final version. In particular, we would appreciate the reviewer's view on two points: whether the protocol comparison under W1/W2 is the kind of evidence the reviewer had in mind, and which of the degradation variants under W4 the reviewer would find most convincing.
I thank the authors for proposing a detailed plan and regret that I couldn't get back to you sooner. I want to be upfront that I find some of your responses quite hard to read, e.g.
On a stratified subset of about 10k queries covering all nine task categories, drawn from the rows with complete labels, we will generate and label outcomes, with the same prompts and judge, for real degraded variants of existing arms
This seems to be saying something like "we will sample 10k questions across all nine categories, collect answers from genuinely degraded models, and judge them the same way."
We expect, and will state, that for frozen offline routers a static re-run of the post-shift condition is close to equivalent to the stream protocol. The value of the stream protocol concentrates on adaptive routers and on the transition period. A static re-run discards the pre-shift history, whereas for online routers the cost of a shift is largely stale history, as shown by the memory-horizon interaction (Appendix E), the detection delays (Table 10), and the dependence on gradual versus hard transitions and on recurring patterns (Figure 5).
This seems to be saying that routers that don't learn during use will not see much difference between streaming and running static benchmarks, while streaming helps study how learning routers adjust to changes.
Such difficulties mean I have to set aside chunks of time to understand and then reply. I don't want to speculate whether this response is heavily AI-generated, but I do hope the authors can put significantly more effort into producing more presentable rebuttal responses that are easy to read (as above), concise (e.g. you possibly don't need to explain that a model with lower reasoning effort would be "faster and therefore yields a degradation with a genuine accuracy–latency trade-off"), and nicely structured (right now, the plan, details, and commentary are often intertwined).
With that, I find the proposed plan to be on the right track, but I am not sure your Feasibility section's claim that all additions except W4 can reuse the stored outcome matrix covers the comparisons I intended. I expect to see RouterShift compared against ParetoBandit and the other static benchmarks you presented, with evaluation results from multiple runs under different conditions, not just differences in their coverage. My motivation is to confirm that RouterShift is able to discover meaningful signals that other benchmarks can't, not just that it is constructed differently. For instance, if I can just run the same static benchmark twice to observe signals of model degradation, I don't think I'd need RouterShift. Similarly, if I can just run tasks on different topics sequentially, I don't think RouterShift's topic-shift task is too meaningful. The same goes for RouterShift vs ParetoBandit: RouterShift might cover more types of shifts, but do they really reflect in the evaluation result?
I am also not sure I agree that a static rerun must discard the pre-shift history for an adaptive router: couldn't we preserve its learned state between runs? It seems like the streaming benchmark is just a standardization of a static benchmark with a particular problem order. I don't disagree that standardization would help, but unless there is evidence indicating such standardization requires meaningful design and engineering effort to result in useful benchmark signal, the proposed benchmark may remain a toy.
I feel like these are quite significant but meaningful asks. I am not sure it is possible to address them well during the rebuttal, and generally, with such significant updates, I'd recommend resubmission so that the presentation can be more polished and cohesive, instead of a bunch of patches. But I will carefully check any updates the authors post, provided they are presented in a much more reader-friendly way.
Plan for rebuttal - Part 1/2.
Many thanks for the careful reading and for the offer to give feedback on a plan before implementation, which we gladly take up. Below we describe, for each weakness (W), what we plan to add. We would be grateful for a quick indication of whether these additions address the concerns, in particular on the two questions in the last paragraph. We split this answer into two responses due to word limit constraints.
Comparison with prior evaluations (W1, W2). The reviewer asks what RouterShift shows that ParetoBandit, other prior work, or repeated runs of a static benchmark do not. ParetoBandit evaluates its own router, together with its ablations, on a three-model portfolio under three shifts: a price drop of one model, a silent quality degradation of another, and the cold-start onboarding of a fourth model. RouterShift uses similar shifts but does not limit itself to them, since production traffic exhibits more. It adds gradual, recurring, transient, and concurrent schedules, changes in query composition (topic and language), latency changes, and judge recalibration, and it evaluates sixteen policies from four families on seven models in three pool sizes, over streams of up to 10,000 steps. We note that ParetoBandit also simulates quality degradation by lowering the stored reward of the affected model, so the real-degradation scenarios described under W4 would be new with respect to both.
We plan two additions. First, a comparison table with ParetoBandit, RouteNLP, RouterBench, LLMRouterBench, RouterArena, and TwinRouterBench, listing for each the shift factors covered, whether onsets are labelled, the number and families of policies, online or offline evaluation, latency, multilingual traffic, and release status. Second, a protocol comparison on the same data and the same sixteen policies under three protocols: (i) a static re-run, as the reviewer suggests, with pre-shift and post-shift conditions evaluated as separate i.i.d. benchmarks; (ii) the ParetoBandit protocol, replaying its three shifts in their three-phase structure on our matrix and scoring them with its oracle-relative quality; and (iii) RouterShift streams. We will report the ranking under each protocol, the Kendall τ between protocols, and the conclusions supported only by (iii).
We expect, and will state, that for frozen offline routers a static re-run of the post-shift condition is close to equivalent to the stream protocol. The value of the stream protocol concentrates on adaptive routers and on the transition period. A static re-run discards the pre-shift history, whereas for online routers the cost of a shift is largely stale history, as shown by the memory-horizon interaction (Appendix E), the detection delays (Table 10), and the dependence on gradual versus hard transitions and on recurring patterns (Figure 5). None of these quantities is defined in a static re-run. Beyond these measurements, the standardisation provides labelled onsets, magnitudes and affected arms, fixed seeds and stream specifications, and one interface for all policies, so that results remain comparable across papers.
Query-side shifts (W3). We agree that topic and language shifts cover only part of the query side. Using the existing labels, without new generations, we can add difficulty drift (traffic moving from easy to hard queries via d(x)), the arrival of an unseen task (a dataset withheld from offline training entering mid-stream), task-category ramps (e.g., a growing share of tool-use or SQL queries), and a prompt-length shift. We would welcome the reviewer's view on which of these would be most informative.
Realism of simulated degradation (W4). On a stratified subset of about 10k queries covering all nine task categories, drawn from the rows with complete labels, we will generate and label outcomes, with the same prompts and judge, for real degraded variants of existing arms: a smaller sibling (Qwen3-235B-A22B to Qwen3-30B-A3B-Instruct-2507 for a mild and Qwen3-4B-Instruct-2507 for a strong degradation, both from the same release, with the same tokenizer and chat template, so that only model capacity changes), lower reasoning effort (gpt-oss-20b from medium, its default, to low, which is also faster and therefore yields a degradation with a genuine accuracy–latency trade-off), and quantisation (Nemotron-Nano-9B-v2 from full precision to 4-bit). We target arms that the routers select frequently, as degrading a rarely selected arm produces little regret signal.