acceptodds
Under review as a conference paper at ICLR 2027

Dynamic LLM Routing: When to Admit a New Model?

Abstract

To leverage newly released Large Language Models (LLMs) cost-effectively, dynamic routing methods onboard new LLMs using observations of whether they answer the queries correctly (i.e., correctness observations). Such observations may mislead the routers to redirect queries from existing LLMs that would answer correctly to a new LLM that answers incorrectly. Consequently, this worsens the overall system accuracy. However, existing dynamic routing methods focus solely on router updating, leaving the deployment decision: *whether to admit a candidate LLM*, unaddressed. To bridge this gap, we propose the *Dual-Evidence Admission Filter (-Filter)*, an admission decision method that can operate on top of existing dynamic routing methods. -Filter combines *pool-history evidence* with *self-simulated evidence*. The former quantifies the reliability of predicted routing gains using residuals from pseudo-onboarding trials; the latter uses the candidate LLM's observations to assess whether redirecting queries to the candidate improves accuracy. These dual-channel evidences are then jointly used to determine whether to admit the candidate. Experiments across multiple benchmarks and routing methods demonstrate that -Filter reduces harmful model admissions while retaining beneficial routing updates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LM Routing, Dynamic Model Routing, New-Model Onboarding, Model Admission

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wh9rOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

E2-Filter aims to address the problem of dynamic model admission, where a model is added to a router with a small support set. E2-Filter will then determine whether this updated router is worth deploying over the old router.

Critical strengths
  • The layer of deciding whether a router is deployable seems relatively novel and worth studying.
Critical weaknesses
  • I am not exactly convinced by the problem setup, where the updated router is only calibrated on a small support set to onboard a new model. Routers are generally lightweight and supervised-trained, so the old data is already collected. The authors motivate this by saying repeated retraining is “computationally prohibitive” but the cited work does not seem to support that claim well. Without showing that retraining/calibrating on more data is actually impractical, the observation risks reducing to “if we onboard a model with very few examples, the router is unreliable”, which is expected.

  • The benefit of E2-Filter is concentrated in the small support size regime. Results from Figure 1(b) and Table 6 seem to suggest with around 256 samples the updated router would beat the old one. This raises the question of why we cannot simply collect a few hundred inference-only samples from a model we are seriously considering routing to.

  • The novelty relative to broader dynamic deployment/model-selection work is unclear. StageRoute studies a broader streaming deployment problem that appears to subsume the single model admission E2 faces. It is unclear why its uncertainty-aware deployment rule, or a simplified version of it, cannot be adapted to the single-candidate setting considered here.

  • The deployable baselines are mainly Always Admit and Predicted-Gain Admit, which are too simple. Natural baselines like confidence thresholds or bootstrap intervals should be included.

  • A nitpick: cost-efficiency is treated as a secondary extension rather than a core evaluation, despite being one of the main motivations for model routing.

Overall, I am skeptical of the realism of this proposed problem setup. I will need to see substantial evidence for why there is a practical need to admit a new model with just a few data samples. The baseline and evaluation axes will also need to be more extensive. Happy to change my mind if there is strong evidence supporting the problem setup and e2's advantage.

Questions for authors

None

9 comments
AuthorsOpenReview ↗

Author Response to Reviewer wh9r (Part 1/2)

We thank the reviewer for the comments. Below, we address each concern with new empirical evidence—including a personalized routing study demonstrating the few-shot regime—and comprehensive validation-tuned baseline comparisons.

W1. Why few-shot admission, when routers can be retrained?

Even with a lightweight router, frequent full retraining across dozens of models and candidate updates disrupts production deployment pipelines, motivating test-time onboarding without retraining [1].

At the same time, we agree that in standard task-specific routing, collecting a few hundred objective accuracy labels is often feasible. However, the few-shot regime is unavoidable in personalized routing, an increasingly active direction [2, 3]. Unlike objective correctness, personalized routing depends on each individual user's subjective preferences (e.g., preferred tone, verbosity, formatting, or reasoning style), which diverge significantly from global benchmark accuracy. Because labels must come from each user's own preference feedback—and asking a user to evaluate hundreds of responses for every newly released candidate model is impractical—per-user onboarding inherently operates in the small-kk regime.

To verify the practical value of personalized routing and illustrate this setting, we constructed a personalized routing benchmark over LLMRouterBench across 16 independent user environments corresponding to the 16 MBTI personas—a standard, structured persona taxonomy in LLM user simulation that captures orthogonal preferences in communication style, detail granularity, and reasoning structure [4, 5]. In each user environment, candidate model responses are evaluated for subjective user satisfaction (like vs. dislike) using two LLM judges (gemini-3.5-flash-lite and gemini-3.1-flash-lite) conditioned on that persona's profile. Using a kNN router with 100 initial training queries per user environment, we compare four routing policies by average preference satisfaction (% liked) across all 16 users:

  • Accuracy-optimal: Routes queries using a router trained solely on objective task-accuracy labels rather than subjective user preferences.
  • General: A single shared router trained on pooled preference feedback across all 16 users (population-average preference, ignoring individual differences).
  • Personal: A user-specific router trained exclusively on that individual user's own preference labels (100 queries per user).
  • Oracle: The per-query upper bound selecting the best-liked model available in the pool for each query.
Selection Policygemini-3.5-flash-litegemini-3.1-flash-lite
Accuracy-optimal27.1%46.4%
General26.9%54.5%
Personal41.3%61.1%
Oracle68.3%92.8%

These results highlight two key insights that directly motivate our problem setting:

  1. Individual user preferences diverge sharply from global accuracy and population averages: Both Accuracy-optimal (27.1% / 46.4%) and General (26.9% / 54.5%) lag substantially behind Personal routing (41.3% / 61.1%). Because different users favor substantially different models than the objective accuracy-optimal or population-average choice, models must be evaluated and onboarded per user rather than globally.
  2. Per-user onboarding is inherently few-shot: Maintaining separate routers per user means every new candidate model must be vetted against each user's personal preference using only a small handful of user-labelled interactions (kk). Applying E2E^2-Filter in this few-shot per-user onboarding setting—without any hyperparameter re-tuning—consistently improves onboarding AUDG over naive admission across routers and pool sizes. We will add this personalized routing benchmark and discussion to the revised paper as a concrete motivating application and foundation for future work on real user logs and active preference elicitation.

[1] Jitkrittum et al., UniRoute: Universal Model Routing for Efficient LLM Inference, ICLR 2025.

[2] Dai et al., PersonalizedRouter: Personalized LLM Routing via Graph-based User Preference Modeling, TMLR 2025.

[3] Xie et al., GMTRouter: Personalized LLM Router over Multi-turn User Interactions, 2025.

[4] Jiang et al., Evaluating and Inducing Personality in Pre-trained Language Models, NeurIPS 2023.

[5] Tseng et al., Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization, Findings of EMNLP 2024.

AuthorsOpenReview ↗

Author Response to Reviewer wh9r (Part 2/2)

W2. Why not collect ~256 observations and always admit?

As noted in the response to W1, collecting hundreds of user preference labels per candidate is unrealistic in personalized routing, making small-kk onboarding the natural operating regime. Even when larger support sets are available, more data does not make admission safe: at k=256k = 256, Always Admit still trails the incumbent router in 8/24 settings (and even at k=1024k = 1024 in several LOCUS settings), whereas E2E^2-Filter beats Always Admit in 20/24 settings. Rather than replacing data collection, E2E^2-Filter gates deployment safety at whatever kk is currently available, with admission rates naturally rising with kk; we will explicitly note that gains concentrate at small kk.

W3. Relation to StageRoute.

The decision problem is distinct between E2E^2-Filter and StageRoute. StageRoute optimizes online stream-level model routing from per-model rewards, whereas E2E^2-Filter makes a one-shot gating decision on whether a specific routing update improves over a fixed incumbent—depending on which queries are rerouted and whether they fix more errors than they introduce. Furthermore, we evaluated an adapted StageRoute UCB baseline. As shown below, E2E^2-Filter substantially outperforms StageRoute UCB, since optimism suits stream exploration rather than one-shot deployment. We will add this baseline and discussion to the revised paper.

W4. Stronger baselines: confidence thresholds and bootstrap intervals.

We implemented the requested confidence-threshold and bootstrap baselines—alongside the StageRoute UCB rule from W3—and evaluated every rule under a strict validation-to-test protocol (where all hyperparameters are tuned solely on validation queries) as well as the paper's original post hoc protocol:

  1. Threshold & UCB baselines: Including zero-threshold rules (Always Admit, Predicted-Gain Admit, Positive-Gain Admit using Eq. 3), validation-tuned threshold rules (Predicted-Gain threshold, Positive-part threshold), the sample-count rule (Budget threshold), and StageRoute UCB (which admits an update when the upper confidence bound μ^+β SE^\hat{\mu} + \beta\,\widehat{\text{SE}} on the cross-fitted paired gain is positive, with confidence width β\beta tuned on validation).
MethodVal → Test AUDGPost hoc (original protocol) AUDG
Always Admit−0.999−0.999
Positive-Gain Admit (Eq. 3)−0.999−0.999
Predicted-Gain Admit−0.446−0.449
Predicted-Gain threshold−0.002−0.002
Positive-part threshold (Eq. 3)−0.002−0.001
Budget threshold0.0910.086
StageRoute UCB0.1070.105
E2E^2-Filter0.1730.162
  1. Bootstrap baselines: We perform BB resamples with replacement over the kk labelled candidate support queries (evaluating on the fixed unlabelled validation or test queries) to compute either a bootstrap confidence interval on the predicted score differences (Predicted-Gain + Bootstrap) or a bootstrap lower bound on our dual-evidence statistic (E2E^2-Filter + Bootstrap).
Method (with Query-Level Bootstrap)Val → Test AUDGPost hoc (original protocol) AUDG
Predicted-Gain + Bootstrap−0.001−0.001
E2E^2-Filter + Bootstrap0.1900.182

Across all comparisons, E2E^2-Filter outperforms every baseline under both protocols: without our cross-fitted candidate evidence, zero-threshold rules incur severe losses while ungated thresholds and bootstrap stall near zero on noisy small-kk predictions. Inspired by your suggestion, pairing query-level bootstrap resampling with our own evidence (E2E^2-Filter + Bootstrap) pushes performance even higher—offering a strong drop-in upgrade when B×B\times resampling overhead is acceptable, while standard E2E^2-Filter already beats all baselines in a single closed-form pass. All of these baselines will be included in the revision.

W5. Cost should be a core evaluation.

We will move the cost-aware evaluation to the main paper.

Thank you for the reply. Unfortunately, I don't find this rebuttal particularly helpful in addressing my concerns.

In terms of whether the proposed scenario is realistic, the authors cite UniRoute, but I don't see evidence in this work that collecting a few hundred candidate observations and updating a router is prohibitive. In general, I don't believe the authors have provided strong evidence to support the idea that a model provider would be unwilling to collect a few hundred samples from a new model before deciding whether it should be deployed.

The authors then pivot to personalized routing—where users can have their own preferences for model quality—which is a much better justified scenario. However, I have several major concerns with this pivot:

  1. This is a substantial pivot: personalized routing is not the focus of, or even mentioned in, the original submission. Given the limited support for the original motivation, it now appears to carry much of the practical justification. This kind of pivot may be too substantial to incorporate during rebuttal.
  2. The added personalized routing results are very preliminary. The authors adopt an MBTI-based setting that neither of the two cited works (PersonalizedRouter and GMTRouter) uses, with almost no details on how such preferences are set up and judged. The displayed results use just one router and one training budget. This seems way too thin if personalized routing becomes the new focus. The two LLM judges also produce substantially different satisfaction rates, which warrants further explanation and validation.
  3. Most importantly, the displayed table seems to only show that tailoring a router to simulated user preferences improves satisfaction. It does not evaluate new-model admission or demonstrate E2-Filter’s benefit.
  4. I couldn't find results supporting the claim that “Applying E2-Filter in this few-shot per-user onboarding setting—without any hyperparameter re-tuning—consistently improves onboarding AUDG over naive admission across routers and pool sizes.” I couldn't locate any accompanying results or an updated PDF, so it is difficult to assess this otherwise significant claim.

My general impression here is that personalized routing is a good scenario for the proposed work, but its evaluation remains too rough and preliminary to show that the proposed method is competitive in this scenario.

Moving on to why one couldn't simply collect more samples, the authors' counterarguments are:

  • “at k=256, Always Admit still trails the incumbent router in 8/24 settings”
  • “E2-Filter beats Always Admit in 20/24 settings”

I acknowledge that these results qualify my initial reading of Figure 1(b) and Table 6. However, my concern is that the gaps here are very small. In the first comparison, Always Admit is worse than the old router by around 0.19 percentage points on average across those eight settings. In the second comparison, E2 beats Always Admit by around 0.21 percentage points on average across all 24 settings. I am not convinced that these gaps are practically meaningful, and without uncertainty estimates, it is difficult to tell whether they are distinguishable from noise.

Lastly, regarding StageRoute, I am not entirely sure how to interpret the discussion and results. My understanding is that StageRoute selects which models to deploy, collects feedback from using them, and then decides which models to retain at the next update. The scenario presented in this work is to collect candidate feedback from a new model, build an updated router, and then decide whether to deploy it. I am therefore a bit confused about what prevents adapting StageRoute to this setting. For example, why can't we give StageRoute the same candidate feedback used in E2, and then compare their decisions on whether to include the new model in deployment? I am not sure whether the reported StageRoute UCB baseline does this, and would appreciate more details.

For the above reasons, the rebuttal has not changed my initial impression. I will be happy to review any further updates from the authors, though I want to be honest that I don't expect my opinion to change substantially unless the authors can address the concerns about the proposed scenario highlighted above. Additionally, if further results are cited, I would appreciate accompanying table or section references. The authors response to W2 lacks these references, which makes the relevant results difficult to locate.

Reviewer_hrT8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a newly onboarded LLM should be admitted to an existing routing system. E²-Filter combines historical gain-estimation residuals with cross-fitted candidate observations to make this decision. Experiments cover three benchmarks and four routing methods.

Critical strengths

The paper identifies a useful distinction between constructing an updated router and deciding whether to deploy it. The filter reuses available correctness observations, and its evaluation across several routing methods supports its practical relevance.

Critical weaknesses
  1. The main results are selected on the test set. Appendix B.5 explicitly states that the reported configuration maximizes mean test AUDG over 270 configurations, without a separate validation set. Using one shared configuration across 24 settings does not remove this selection bias. Consequently, the reported gains do not establish performance on unseen admission decisions. This also weakens the distinction between E²-Filter and the baselines excluded from the deployable comparison for using test outcomes.

  2. The comparisons do not convincingly establish the value of dual evidence. Predicted-Gain Admit averages negative margins even on queries that would remain with the original router (Appendix B.4), although these queries contribute zero deployment gain under Eq. (2). It therefore tests a different quantity from the gain being optimized. Table 2 provides a more relevant comparison, but fusion improves AUDG over self-simulated ranking only from 0.150 to 0.162 percentage points. A comparison against independently tuned single-evidence admission rules is needed to establish whether fusion provides a meaningful benefit.

Questions for authors
  1. Does the advantage persist when all admission hyperparameters are selected on validation data and frozen before test evaluation?

  2. Under that protocol, how does E²-Filter compare with a threshold on the positive-part predicted gain in Eq. (3), and with separately tuned single-evidence rules?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer hrT8

Thank you for your comments. In response, we re-ran the full comparison under a strict validation-to-test protocol and expanded our baseline suite with both tuned thresholds and query-level bootstrap resampling.

  1. Threshold baselines: Including zero-threshold rules (Always Admit, Predicted-Gain Admit, Positive-Gain Admit using Eq. 3), validation-tuned threshold rules (Predicted-Gain threshold, Positive-part threshold), and the sample-count rule (Budget threshold).
MethodVal → Test AUDGPost hoc (original protocol) AUDG
Always Admit−0.999−0.999
Positive-Gain Admit (Eq. 3)−0.999−0.999
Predicted-Gain Admit−0.446−0.449
Predicted-Gain threshold−0.002−0.002
Positive-part threshold (Eq. 3)−0.002−0.001
Budget threshold0.0910.086
E2E^2-Filter0.1730.162
  1. Bootstrap baselines: We perform BB resamples with replacement over the kk labelled candidate support queries (evaluating on the fixed unlabelled validation or test queries) to compute either a bootstrap confidence interval on the predicted score differences (Predicted-Gain + Bootstrap) or a bootstrap lower bound on our dual-evidence statistic (E2E^2-Filter + Bootstrap).
Method (with Query-Level Bootstrap)Val → Test AUDGPost hoc (original protocol) AUDG
Predicted-Gain + Bootstrap−0.001−0.001
E2E^2-Filter + Bootstrap0.1900.182

Under matched resampling compute, E2E^2-Filter + Bootstrap clearly outperforms Predicted-Gain + Bootstrap, and further improves over single-pass E2E^2-Filter. This demonstrates that bootstrap resampling alone cannot rescue noisy raw gain estimates, whereas pairing resampling with our dual-evidence formulation extracts substantially richer uncertainty information from both historical pool residuals and cross-fitted candidate observations.

We will make these results our primary comparison and include both tables in the revision.

W2 / Q2. Value of dual evidence and independently tuned single-evidence rules.

We tuned each single-evidence rule and statistical variant independently on its own hyperparameter grid under both the validation-to-test and post hoc protocols across all 24 settings:

RuleVal → Test AUDGPost hoc AUDG
E2E^2-Filter0.1730.162
Pool-history evidence only0.1340.132
Self-simulated evidence only0.1640.151

Even when independently tuned on its own grid, dual-evidence fusion consistently outperforms both Pool-history evidence only and Self-simulated evidence only across both evaluation protocols, confirming that historical pool calibration and cross-fitted candidate evidence provide complementary signals rather than redundant information. We will update Table 2 with this independently tuned ablation in the revision.

Dear Reviewer hrT8,

Thank you again for your review. With the discussion period closing on November 18th, we wanted to check whether our response sufficiently addressed your concerns. If anything is still unclear, we would be glad to clarify before the deadline. Thank you for your time!

Reviewer_umLuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Dual-Evidence Admission outputs a binary decision whether to deploy a newly updated router or to retain the original router.
  • It solves the "Optmizer's Curse" (when there is a new candidate LLM and there are only a few observations, estimation noise can result in a suboptimal deployment)
  • Dual-Evidence Admission uses two approaches:
    • pool-history evidence - simulates trials to correct for estimation bias
    • self-simulated evidence - uses k-fold validation to model beneficial vs. harmful re-routings
  • Designed for dynamic model pools (where new models arrive with sparse data)
Critical strengths
  • The approach where the Dual-Evidence admission filter doesn't require additional queries for a candidate is novel and useful.
Critical weaknesses
  • The filter introduces another component in a likely complex multi-stage system (with its own set of hyperparameters that need to be tuned).
Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer umLu

Thank you for your thoughtful comments. Below, we address your concern regarding system complexity and hyperparameter tuning:

W1. Another component with its own hyperparameters in a multi-stage system.

In practice, E2E^2-Filter is designed as a lightweight plug-in that introduces minimal tuning burden and runtime overhead:

  • E2E^2-Filter uses a single shared hyperparameter configuration across all 24 experimental settings, requiring no per-candidate, per-router, or per-benchmark tuning. Furthermore, as shown in Fig. 4, AUDG remains stable across broad ranges of ϵ\epsilon and JJ, as well as τ∈[0.975,0.99]\tau \in [0.975, 0.99], confirming that performance does not hinge on delicate hyperparameter tuning.

  • E2E^2-Filter does not change the old router or its training process. It simply acts as a lightweight filter after the existing update step, making a binary deployment decision (admit the updated router or retain the old one). Moreover, it requires zero extra model calls, fully reusing the kk candidate observations already collected for onboarding alongside existing pool observations.

  • Across all 24 settings, the runtime cost of E2E^2-Filter is a tiny fraction of standard router updates:

RouterRouter update (s)Residual bank (s / episode)Pseudo-onboarding trial (ms)
kNN0.41–0.660.35–1.202.3–3.0
UniRoute-Cluster0.05–0.110.05–0.380.2–1.2
UniRoute-LearnedMap22–2950.04–0.310.3–1.0

Specifically, the residual bank is constructed only once per episode from existing pool observations and reused across all candidates and support draws, while each final admission decision takes just 0.044 ms. We will include these runtime measurements in the revision.

Thank you for the additional information. My score remains positive-leaning.

Thank you for your continued support and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.