acceptodds
Under review as a conference paper at ICLR 2027

RankShift: Dual-Reference Identification of Routing Value under Delayed Feedback

Abstract

Aggregate routing gains do not identify routing value. A positive gain may come from matching experts to states or simply from calling a globally stronger expert more often. It may also hide local losses against the deployed model. We therefore define operational routing value relative to two fixed references. The deployed anchor tests current-system improvement, and a same-budget state-independent policy tests value beyond a registered blind allocation. A single comparison need not determine both properties. RANKSHIFT authorizes a fixed proposal only when matured paired outcomes give positive lower bounds against both. Under explicit dependence and drift conditions, we describe the delay and support required for this decision. On a held-out language-model routing benchmark, one validation-frozen rule authorizes three of twelve proposal learners at the primary seed without retraining. Their gains are +0.0224, +0.0227, and +0.0237 with no observed local fallback regression. The other nine remain on the anchor. An independent routing trace plus delayed replay and three non-LLM domains test the same mechanism.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Routing, Routing Evaluation, Partial Identification, Selective Prediction, Safe Decision Making, Delayed Feedback, Adaptive Systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9sdUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper highlights an important issue in existing routing research: improvements in average performance do not necessarily reflect genuine state-dependent routing value, and may hide local performance regressions. To address this issue, the authors propose RankShift, a dual-reference framework for identifying routing value and safely deploying routing policies under delayed feedback. The paper provides theoretical analyses of the identification problem and selective safety, along with extensive experiments across LLM routing, finance, news recommendation, and commerce.

Critical strengths
  1. The paper identifies an important and often overlooked issue in routing evaluation, distinguishing genuine state-dependent routing value from improvements driven by stronger expert selection.

  2. The theoretical analysis is generally rigorous and well-developed, covering identification, delayed feedback, and selective safety guarantees.

Critical weaknesses
  1. Limited methodological and theoretical novelty. Although the problem is interesting and the theoretical analysis is fairly rigorous, the proposed method and some theoretical results appear rather straightforward. RankShift essentially applies two standard lower confidence bound tests against fixed reference policies, followed by a fallback decision. This is closely related to existing ideas in safe policy improvement and selective prediction, without introducing a substantially new learning or routing mechanism. Similarly, Theorem 1 mainly shows that observing only one reference cannot identify the improvement over another unobserved reference, while Theorem 2 uses a simple indistinguishability construction for delayed feedback. These results are formally correct, but the insights seem somewhat obvious, and it is unclear what fundamentally new understanding they provide beyond existing work.

  2. Insufficient empirical evidence for the necessity and effectiveness of the dual-reference mechanism. In Table 3, the state-independent gate alone and the dual-reference mean gate produce exactly the same results, authorizing 19 out of 35 units with identical value capture. Moreover, in LLMRouterBench and Amazon-M2, the state-independent reference coincides with the deployed anchor, making the two gates effectively equivalent. Although the authors provide an additional 420-cell experiment to demonstrate the necessity of the anchor gate, this analysis lacks familywise multiple-testing adjustment, as acknowledged in the appendix. More systematic experiments with distinct and strong state-independent references are needed to establish when the dual-reference design actually provides additional benefits over a single-reference approach.

  3. Limited generalization guarantees and practical applicability. RankShift relies on predefined state partitions and historical validation outcomes to determine where a router should be authorized, rather than learning fine-grained query-specific routing advantages. Its effectiveness therefore depends on whether the relative performance within each state remains stable between validation and deployment, which may not hold under distribution shifts or changes in query composition. Although the paper provides theoretical safety guarantees, these require restrictive dependence and drift assumptions that are not fully verified in the real-world experiments. Furthermore, most experiments rely on offline paired outcomes, while practical deployment may require additional shadow calls to obtain such feedback, introducing potentially substantial costs.

Questions for authors

Here I list the questions I am most concerned about. Please see the Weaknesses section for more details.

  1. What is the main methodological novelty of RankShift beyond applying standard confidence-based tests and fallback decisions? In particular, how do the theoretical results provide new insights compared with existing safe policy improvement and delayed-feedback literature?

  2. Given that Gate 2 alone achieves the same results as the dual-reference mean gate in Table 3, can the authors provide stronger empirical evidence that both references are necessary in practice? How does RankShift compare with simpler single-reference authorization or validation-based filtering methods?

  3. How well does RankShift generalize when the query distribution or the relative performance of experts changes between validation and deployment? Can the authors provide additional experiments with finer-grained states or distribution shifts to evaluate this limitation?

  4. How significant is the cost of collecting paired feedback in practical deployment, especially when the expert pool is large? Does RankShift still provide meaningful net benefits when these costs are fully considered?

I would be willing to raise my score if the authors can adequately address these concerns, as I believe the problem studied in this paper is important and worth investigating.

Reviewer_VpfROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RankShift, an authorization layer for deploying a fixed routing proposal under delayed feedback. The paper argues that a positive aggregate routing gain does not reveal whether the gain comes from effective state-dependent allocation, increased use of a globally stronger expert, or improvements that coexist with local regressions relative to the deployed system.

RankShift therefore evaluates a candidate router against two fixed references: the deployed anchor and a state-independent policy with a matched call budget. Only matured paired outcomes enter the evaluation record, and a routing unit is authorized only when the lower confidence bounds against both references are positive. Unsupported units fall back to the anchor. The paper provides identification and conditional safety results, analyzes the mature support required for authorization, and evaluates the mechanism across LLM routing and several non-LLM domains. Overall, the experiments indicate that RankShift can retain supported parts of frozen routing proposals while falling back to the deployed system when the available evidence is insufficient.

Critical strengths
  1. The paper identifies an important ambiguity in router evaluation. Improvement over the deployed system and improvement over a same-budget state-independent allocation answer different questions. The dual-reference formulation provides a clear way to distinguish operational improvement from evidence that the request-dependent allocation itself is useful.

  2. RankShift can be composed with existing routers without retraining them. The wrapper experiment keeps proposal scores and training fixed, determines authorization from validation evidence, and evaluates the resulting decisions on held-out data. Treating proposals with insufficient evidence as explicit fallbacks, rather than positive results, is a sound and practically relevant choice.

  3. The delayed-feedback setting is handled carefully. The method uses only matured outcomes, fixes evaluation times in advance, and retains the anchor until sufficient evidence is available. The support-boundary analysis also gives a useful account of how feedback delay and increasingly complex state representations postpone authorization.

  4. The evaluation covers several routing mechanisms and application domains. The experiments span LLM routing, finance, news, and recommendation, with different utilities and feedback clocks. This provides evidence that the authorization principle is not tied to a particular proposal learner or task-specific metric.

Critical weaknesses
  1. The empirical contribution of the dual-reference mean rule is not cleanly isolated from the additional tail screen. In Table 3, the state-independent-only gate and the dual mean gate authorize exactly the same 19 of 35 development units. They also produce the same number of nonpositive activations and the same retained positive dual-reference value. The authorization count changes substantially only after adding the worst-family tail screen, which the paper explicitly describes as an empirical condition outside Theorem 3. Thus, this ablation clearly demonstrates the filtering value of Gate 2 and the additional conservatism of the tail screen, but it does not demonstrate an incremental empirical benefit from requiring both mean-gain gates on these units. Since the headline RankShift-Full results combine the two mean gates with the tail screen, their individual contributions remain difficult to separate.

  2. The stability of the headline learned-router qualification result is not sufficiently visible in the main paper. The primary CALIBRI comparison is reported for seed 2027. The paper states that five-seed sensitivity records are available for eleven proposal adapters, while RouterDC is evaluated only at the primary seed, but the main results do not show how frequently each proposal passes the complete authorization rule across seeds. This matters because authorization is a discrete threshold decision: a small change in the fitted proposal, split, or estimated lower bound can change a proposal from full fallback to qualification. Aggregate gains averaged over seeds would not fully address this concern; the relevant quantity is the stability of the pass/fail authorization decision itself.

Questions for authors
  1. Could the authors provide a matched ablation in which the same worst-family tail screen is applied to the anchor-only, state-independent-only, and dual-reference gates? Reporting anchor-only + tail, state-independent-only + tail, dual mean, and dual mean + tail on the same development units would clarify whether the observed effect comes from combining the two references or primarily from Gate 2 and the tail criterion.

  2. For each frozen proposal, how often does the complete RankShift rule qualify the proposal across the available seeds? A proposal-level table reporting the number of passing seeds, together with the corresponding fallback frequency, would make the stability of the primary qualification result easier to assess.

Reviewer_jpFuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RankShift adds an authorization layer to an existing router. It permits proposed switches only when matured, paired outcomes support positive cost-adjusted utility gains against both the deployed anchor and a same-budget, state-independent allocation. Otherwise, it retains the anchor. The comparisons distinguish improvement over the current system from value attributable to request-dependent model selection (Section 3, pp. 3–4; Algorithm 1, p. 6).

The paper develops an identification argument and conditional safety results, then evaluates frozen proposals, delayed replay, public routing traces, and non-LLM applications. Three of twelve learned proposals activate at the primary CALIBRI seed and improve their observed utility; the other nine fall back to the anchor (Table 13, p. 28).

Critical strengths

Useful evaluation distinction. Aggregate improvement over the deployed system does not necessarily establish that request-dependent allocation contributes value. The paper makes these separate objectives explicit (Section 3, pp. 3–4).

Controlled before/after comparison. Proposal scores remain fixed, authorization is determined on validation, and the resulting policy is evaluated on held-out data. The three successful CALIBRI wrappers improve observed cost-adjusted utility without retraining the proposals (Table 2, p. 8; Table 13, p. 28).

Breadth and transparency. The authors report all twelve cases and five-seed sensitivity for eleven adapters (Tables 12–13, p. 28), negative open-text results (Appendix E.1, p. 25), and a public 13-model experiment where the ungated proposal achieves higher utility (Table 19, p. 32). A separate replay charges shadow-call costs (Appendix E.4, Table 10, p. 26).

Explicit limitations. The paper distinguishes comparator-relative value from causal identification (Section 3, p. 4; Appendix D.1, pp. 22–23), and empirical intervals from the stronger conditional deployment guarantee (Section 4.3, p. 5; Appendix B.4, p. 18).

Critical weaknesses

My main concern is whether the observed improvement requires RankShift’s particular authorization rule, or whether simpler uses of the same validation outcomes could produce similar gains. The central comparison evaluates the same frozen router without and with RankShift (Table 13, p. 28). This validly measures the effect of adding the wrapper. However, the outcomes used to approve switches could also support ordinary adjustments: tuning a threshold for the utility objective, recalibrating scores, or disabling switches in query groups that perform poorly on validation. The comparison does not establish whether RankShift uses this information more effectively.

RankShift deliberately retains the anchor when either approval condition fails or remains uncertain (Algorithm 1, p. 6). This is a legitimate objective. The question is whether its favorable results reflect better selection, or could also be achieved by an ordinary router operating more conservatively. For example, StageRoute proposes 315 candidate calls in the primary CALIBRI experiment, while RankShift retains 211 and achieves higher observed cost-adjusted utility (Table 13, p. 28). This is a genuine benefit. However, a simpler baseline could also tune its routing threshold on the same validation data to retain fewer, higher-scoring requests. Its performance at approximately 211 calls would provide a more informative comparison than its original operating point alone.

Questions for authors

The reported held-out splits (Section 6, p. 7) and prompt-hash partitioning (Appendix E.6, p. 31) are useful safeguards. However, I could not see whether translations, paraphrases, and repeated generations of underlying problems remain grouped across training, authorization, and test partitions. CALIBRI’s bootstrap keeps multilingual rows of the same problem together (Appendix E.3, p. 26); this does not itself establish grouping during split construction. A brief problem-level audit would clarify this; no leakage has been demonstrated. Separately, source-order replay and chronological cross-domain protocols provide breadth (Appendix E.4, p. 26; Appendix F, pp. 35–36), but it remains useful to clarify whether fixed authorization decisions transfer to later periods or different query sources.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.