Model Metropolis: A Digital-Twin Contextual Bandit for Difficulty-Aware LLM Routing

Dynamic LLM routing has recently emerged as an effective way to balance model quality and cost across diverse queries. While real-world deployments increasingly demand adaptive routing, existing methods often rely on offline estimates and lack fine-grained evidence of model performance. We introduce Model Metropolis, a live digital-twin framework for difficulty-aware LLM routing. We leverage time-decayed evidence from observed outcomes to maintain capability- and difficulty-specific delivery states for each model. A contextual-bandit router then uses these states to adjust the quality requirement based on query difficulty and to balance estimated delivery against cost among evidence-supported models. Our experiments demonstrate three key findings. First, difficulty-aware routing substantially improves delivery on challenging queries while preserving cost efficiency. We demonstrate this on a production gateway of 30 models, where hard and expert delivery improve from 73% to 91% and from 69% to 85%, with hard-query routing reaching 91% delivery at 0.44x the all-strong cost. Second, we show that the learned routing state generalizes to held-out benchmarks when the calibration evidence covers the target task family, achieving competitive cost-delivery trade-offs across public coding and general-purpose benchmarks. Third, we demonstrate that the framework extends to both escalation control and pre-generation routing. A pool ceiling reduces gateway errors from 62% to 0% on unwinnable queries. Overall, our results establish that continuously observed, evidence-updated model states provide a practical basis for adaptive LLM routing.