acceptodds
Under review as a conference paper at ICLR 2027

Model Metropolis: A Digital-Twin Contextual Bandit for Difficulty-Aware LLM Routing

Abstract

Dynamic LLM routing has recently emerged as an effective way to balance model quality and cost across diverse queries. While real-world deployments increasingly demand adaptive routing, existing methods often rely on offline estimates and lack fine-grained evidence of model performance. We introduce Model Metropolis, a live digital-twin framework for difficulty-aware LLM routing. We leverage time-decayed evidence from observed outcomes to maintain capability- and difficulty-specific delivery states for each model. A contextual-bandit router then uses these states to adjust the quality requirement based on query difficulty and to balance estimated delivery against cost among evidence-supported models. Our experiments demonstrate three key findings. First, difficulty-aware routing substantially improves delivery on challenging queries while preserving cost efficiency. We demonstrate this on a production gateway of 30 models, where hard and expert delivery improve from 73% to 91% and from 69% to 85%, with hard-query routing reaching 91% delivery at 0.44x the all-strong cost. Second, we show that the learned routing state generalizes to held-out benchmarks when the calibration evidence covers the target task family, achieving competitive cost-delivery trade-offs across public coding and general-purpose benchmarks. Third, we demonstrate that the framework extends to both escalation control and pre-generation routing. A pool ceiling reduces gateway errors from 62% to 0% on unwinnable queries. Overall, our results establish that continuously observed, evidence-updated model states provide a practical basis for adaptive LLM routing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM routing, model routing, contextual bandits, online learning, difficulty-aware routing, adaptive inference, uncertainty estimation, cost-aware inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_azL2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to route queries among LLMs while balancing answer quality and cost. Model Metropolis maintains capability- and difficulty-specific success estimates using discounted Beta updates, then combines quality thresholds, evidence requirements, exploration, and cost-aware selection (lines 181–245). It also extends the framework to escalation stopping and activation-based pre-generation routing (lines 253–295). Evaluation includes a repeated-task online gateway experiment, held-out public benchmarks, and tests of the two extensions (lines 298–305, 1250–1304, 1306–1320, and 1350–1357). The reported hard-task gains are promising, while transfer results depend on calibration coverage (lines 1215–1231 and 1268–1285).

Critical strengths

S1. A practical approach to an important routing problem. Tracking actual success by task type and difficulty is a useful basis for model selection, particularly when aggregate model scores hide differences in reliability (lines 181–201). The reported hard-task delivery improves from 73% for Cost-min and 70% for the difficulty-agnostic bandit to 91% for the full method, with 91.2 ± 1.0% across three repeats (lines 1215–1231). These results provide encouraging evidence for the overall design, although the contribution of each component remains unclear.

S2. Held-out evaluation reveals when calibration transfers. The paper includes disjoint calibration/evaluation splits (lines 1250–1252). On the same 30 held-out LiveCodeBench hard problems, adding separate, task-family-matched calibration evidence raises delivery from 10.0% to 46.7% (lines 1268–1285). This is informative because it identifies calibration coverage as an important condition for transfer, while the mixed benchmark results make the limits visible (lines 1254–1263 and 1277–1285).

Critical weaknesses

W1. The experiments do not systematically isolate the proposed mechanisms. The central design combines difficulty-dependent thresholds, a mean-based quality gate, evidence requirements, and targeted exploration (lines 812–852). Although the difficulty-agnostic comparison is useful (lines 1216–1218), the component ablation in Table 12 reports only qualitative routing behavior, without delivery or cost measurements (lines 1242–1249). It therefore remains unclear how much each mechanism contributes beyond shifting more hard queries to strong models. This limits the evidence for the specific algorithmic contribution.

The evaluation protocols also leave a gap: the main learning results are reported after five rounds on the 115-task stream (lines 303–305 and 1179–1181), whereas the deterministic public-benchmark replay disables Thompson sampling and targeted exploration (lines 812–816). Repeated-task learning is a valid setting, but these experiments do not establish the full online policy’s performance on fresh requests. Quantitative component ablations and learning curves on an unseen task stream, with matched feedback and budgets, would directly address this concern.

W2. The cost and feedback assumptions limit the practical conclusions. The headline 0.44× relative cost uses the mean selected-model output-token unit price, rather than actual token usage (lines 854–863 and 1329–1343). It therefore does not establish an equivalent reduction in total spending. Later benchmarks use token-based costs (lines 861–863), but the main savings claim still needs accounting for input/output tokens, calibration, exploration, and retries. Comparisons at matched quality or matched total cost would make the practical benefit easier to assess.

The general-benchmark escalation experiment also uses an oracle verdict that is explicitly unavailable during normal inference (lines 1017–1019). This is useful as a controlled test, but it leaves the benefit under deployable feedback uncertain. Evaluation with an available verifier, including its errors and cost, is needed to support that broader claim. Together, these limitations weaken the evidence for the paper’s central quality–cost benefit in deployment.

Questions for authors

Q1. Which mechanisms drive the gains, and how are the learning results aggregated? Does the reported 91% hard delivery refer to the final round or another aggregation of the five rounds (lines 303–305 and 1179–1181)? Can you provide quantitative delivery/cost ablations for the components in Table 12, together with results on a fresh held-out stream using matched feedback and total budgets (lines 1242–1249)? These results would help establish what the full online policy contributes.

Q2. Does the cost advantage remain under complete accounting? What are the overall and hard-only costs when actual input/output tokens, calibration, exploration, and retries are included, and which costs are amortized (lines 854–863 and 1329–1343)? At matched delivery or matched total cost, does the claimed advantage remain? This would clarify what the 0.44× figure implies for practical savings.

Q3. What feedback supports escalation in normal deployment? For the general-benchmark setting, what usable verifier would replace the oracle verdict in Table 7, and how do its errors, availability, and cost affect the reported gains (lines 1017–1024)? Evidence under such feedback would materially change the assessment of the escalation extension’s practical value.

Reviewer_WzZxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Model Metropolis maintains capability- and difficulty-conditioned delivery estimates for an LLM pool using discounted Beta statistics. A router combines evidence requirements, difficulty-dependent quality thresholds, posterior sampling, and cost-sensitive selection. Extensions limit escalation when estimated pool reliability is low and use local-model activations for pre-generation routing. The paper reports results on a 30-model gateway and several public benchmarks, using both online experiments and offline routing replays.

Critical strengths

Routing heterogeneous models according to difficulty-conditioned delivery estimates is a relevant practical problem. The pool-growth experiment and pre-generation gate explore useful deployment questions and give the work potential value.

Critical weaknesses
  1. The cost and online-routing evaluations use different cost definitions and feedback protocols, which complicates comparison across the reported results. The reported 0.44× cost is a mean output-token unit price rather than realized expenditure (Appendix E/Table 18). Public-suite replay disables Thompson sampling and exploration (Appendix H), and Table 7 uses oracle verification. The SWE-bench result is restricted to oracle-file context and 45 mutually completed hard/expert instances, with no all-77 router result. The reported tables do not clearly document a matched RouterDC/GraphRouter comparison; specifying the corresponding protocol and results would help establish comparability. These protocols each provide useful observations, but it remains unclear how far they establish the advantage of the deployable online policy. Comparable realized costs and matched feedback assumptions would materially strengthen the evidence.

  2. The interpretation of the escalation statistic deserves a more careful justification. Equation (8) treats the maximum of Wilson lower confidence bounds as a pool ceiling. A low lower bound can reflect limited evidence rather than low achievable success, and marginal success rates do not determine another model’s chance of succeeding after the first model fails. The rule may still be a useful conservative heuristic. Presenting it in that role and comparing it with retry-cap or direct-to-frontier alternatives under the same workload would help establish when it is effective.

Questions for authors
  1. Do the cost–delivery conclusions hold when realized token costs, calibration/exploration expenditure, and all attempted requests are accounted for under the same protocol?

  2. Which results directly evaluate the online policy with deployable feedback, and where are the matched learned-router comparisons? Please distinguish these from deterministic replay and oracle-assisted results.

  3. Could you clarify whether the lower-bound rule is intended as a heuristic, and provide a matched comparison supporting its benefit? Please also specify the workload and error definition behind the 62%→0% result.

Reviewer_kzZnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Model Metropolis maintains estimates of each model’s success rate across capabilities and difficulty levels. These estimates are updated from observed outcomes and used by a Thompson-sampling router. The paper also introduces a rule for stopping escalation and an activation-based gate for a local model. Experiments include a 30-model gateway, held-out public benchmarks, and the addition of new models.

Critical strengths

The paper studies a useful practical problem: updating routing decisions as evidence about models becomes available. The evaluation includes several task families and a model-pool growth experiment. The results also show that routing performance depends on whether the calibration data covers the target task family, which is a useful finding.

Critical weaknesses

The paper shows that the router can learn from feedback, but it is unclear whether the proposed method learns better than a simpler online router. The public benchmark experiments turn off exploration and Thompson sampling, so they do not answer this question. A direct comparison under the same queries and feedback would help establish the benefit of the proposed method.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.