- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper highlights an important issue in existing routing research: improvements in average performance do not necessarily reflect genuine state-dependent routing value, and may hide local performance regressions. To address this issue, the authors propose RankShift, a dual-reference framework for identifying routing value and safely deploying routing policies under delayed feedback. The paper provides theoretical analyses of the identification problem and selective safety, along with extensive experiments across LLM routing, finance, news recommendation, and commerce.
Critical strengths
-
The paper identifies an important and often overlooked issue in routing evaluation, distinguishing genuine state-dependent routing value from improvements driven by stronger expert selection.
-
The theoretical analysis is generally rigorous and well-developed, covering identification, delayed feedback, and selective safety guarantees.
Critical weaknesses
-
Limited methodological and theoretical novelty. Although the problem is interesting and the theoretical analysis is fairly rigorous, the proposed method and some theoretical results appear rather straightforward. RankShift essentially applies two standard lower confidence bound tests against fixed reference policies, followed by a fallback decision. This is closely related to existing ideas in safe policy improvement and selective prediction, without introducing a substantially new learning or routing mechanism. Similarly, Theorem 1 mainly shows that observing only one reference cannot identify the improvement over another unobserved reference, while Theorem 2 uses a simple indistinguishability construction for delayed feedback. These results are formally correct, but the insights seem somewhat obvious, and it is unclear what fundamentally new understanding they provide beyond existing work.
-
Insufficient empirical evidence for the necessity and effectiveness of the dual-reference mechanism. In Table 3, the state-independent gate alone and the dual-reference mean gate produce exactly the same results, authorizing 19 out of 35 units with identical value capture. Moreover, in LLMRouterBench and Amazon-M2, the state-independent reference coincides with the deployed anchor, making the two gates effectively equivalent. Although the authors provide an additional 420-cell experiment to demonstrate the necessity of the anchor gate, this analysis lacks familywise multiple-testing adjustment, as acknowledged in the appendix. More systematic experiments with distinct and strong state-independent references are needed to establish when the dual-reference design actually provides additional benefits over a single-reference approach.
-
Limited generalization guarantees and practical applicability. RankShift relies on predefined state partitions and historical validation outcomes to determine where a router should be authorized, rather than learning fine-grained query-specific routing advantages. Its effectiveness therefore depends on whether the relative performance within each state remains stable between validation and deployment, which may not hold under distribution shifts or changes in query composition. Although the paper provides theoretical safety guarantees, these require restrictive dependence and drift assumptions that are not fully verified in the real-world experiments. Furthermore, most experiments rely on offline paired outcomes, while practical deployment may require additional shadow calls to obtain such feedback, introducing potentially substantial costs.
Questions for authors
Here I list the questions I am most concerned about. Please see the Weaknesses section for more details.
-
What is the main methodological novelty of RankShift beyond applying standard confidence-based tests and fallback decisions? In particular, how do the theoretical results provide new insights compared with existing safe policy improvement and delayed-feedback literature?
-
Given that Gate 2 alone achieves the same results as the dual-reference mean gate in Table 3, can the authors provide stronger empirical evidence that both references are necessary in practice? How does RankShift compare with simpler single-reference authorization or validation-based filtering methods?
-
How well does RankShift generalize when the query distribution or the relative performance of experts changes between validation and deployment? Can the authors provide additional experiments with finer-grained states or distribution shifts to evaluate this limitation?
-
How significant is the cost of collecting paired feedback in practical deployment, especially when the expert pool is large? Does RankShift still provide meaningful net benefits when these costs are fully considered?
I would be willing to raise my score if the authors can adequately address these concerns, as I believe the problem studied in this paper is important and worth investigating.