acceptodds
Under review as a conference paper at ICLR 2027

Lens-Router: Progressive Fine-Grained Routing for Large Language Models

Abstract

The LLM ecosystem is undergoing rapid model updates and intense tiered competition, requiring LLM routers to support efficient model expansion and fine-grained routing. However, existing routing paradigms face a dilemma between scalability and fine-grained routing: finer-grained decisions require more historical supervision, making new model incorporation less scalable. Motivated by this dilemma, we propose Lens-Router, a progressive fine-grained router that abstracts reusable, cross-model resource demand features from queries. These features provide effective supervision for both capability filtering and realized cost prediction, enabling Lens-Router to efficiently incorporate new models and select the lowest realized cost model that can fulfill the query. Our experiments show that with only 700 historical supervision for model adaptation, Lens-Router improves routing accuracy while reducing realized cost by up to 82.8% compared with SOTA LLM routers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Routing, Resource Demand, Fine-Grained Model Selection, Cost-Aware Routing, Query-Conditioned Cost, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iFnBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Lens-Router learns a query representation through resource-demand supervision, filters candidate models by capability clusters, and then predicts pairwise costs to select a model. The encoder is trained using annotations from three LLMs, while model-specific routing uses 700 queries. Experiments across nine benchmarks and two model pools report improved quality–cost trade-offs over existing routers.

Critical strengths

The distinction between token price and query-dependent realized cost is useful and well motivated. The cost-ranking component also has concrete empirical support: in Table 5, replacing pairwise ranking with direct ranking increases cost from 0.13 to 0.21 at the same reported accuracy. Evaluating a pool of similarly priced models makes this contribution particularly relevant.

Critical weaknesses
  1. The justification for capability clustering does not establish that models within a cluster are interchangeable for a given query. Appendix A.1 reports overlapping accuracy intervals and frequent ranking reversals across benchmarks. However, ranking reversals can reflect model specialization, rather than unreliable capability differences. These are precisely the differences a router could exploit. Once capability filtering is complete, Lens-Router selects solely by cost, so this distinction matters directly to answer quality. The definition of cluster correctness yq,cy_{q,c} is also missing: whether one, most, or all members must answer correctly leads to very different supervision. Without measuring correctness disagreement within clusters and the quality lost through cost-only selection, the central decomposition remains insufficiently justified.

  2. The experiments do not isolate the benefit of the proposed resource-demand supervision. The 700-query adaptation budget follows encoder training on 59,577 queries with responses and annotations from three strong LLMs. Amortizing this investment is reasonable, but Table 4 mainly compares demand-supervised encoders with pretrained encoders; it does not distinguish the proposed 24-dimensional representation from the benefit of additional task-relevant supervision. Likewise, the quoted supervision counts for competing routers do not establish their requirements at a comparable quality–cost target. A comparison using the same source responses to train a simpler auxiliary representation, followed by equal-budget adaptation, would directly test the paper’s main claim.

Questions for authors
  1. How exactly are yq,cy_{q,c} and the inference-time capability cutoff defined? Among queries passed to a cluster, how often do its members disagree on correctness, and how often does cost ranking select an incorrect model when a correct candidate remains?

  2. With the same ModernBERT backbone and source-response data, how does demand supervision compare with directly predicting response length and model correctness before 700-query adaptation? Does the advantage persist for target models excluded from encoder construction?

Reviewer_U86AOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Lens-Router, a progressive fine-grained routing framework that aims to balance response quality and inference cost while efficiently adapting to newly introduced LLMs. The key idea is to learn a reusable, query-level resource demand representation rather than directly mapping queries to specific models. Based on this representation, Lens-Router first filters candidate models through capability-aware clustering and then performs pairwise cost ranking to select the lowest-cost model capable of handling the query. The authors also introduce a resource demand encoder trained on 24 dimensions to capture task requirements beyond conventional semantic features. The proposed framework is evaluated across nine benchmarks covering mathematics, knowledge, coding, and instruction following, with comparisons against several representative LLM routers. Experimental results show improvements in routing accuracy and substantial reductions in realized inference cost, with reported savings of up to 82.8% compared to a strong baseline. Notably, the framework requires only 700 historical queries per model for adaptation. Additional ablation studies support the effectiveness of the resource demand representation and progressive routing design.

Critical strengths
  1. The paper addresses an important and practical challenge in LLM serving: how to achieve fine-grained model selection while efficiently adapting to rapidly evolving model pools. I particularly appreciate the idea of learning reusable, cross-model resource demand representations instead of relying solely on query semantics or model-specific historical outcomes. The progressive design, which separates capability filtering from cost-aware pairwise ranking, is intuitive and well motivated. This provides a promising direction for improving the scalability and cost efficiency of LLM routing.
  2. The authors evaluate Lens-Router across nine benchmarks covering diverse task domains and compare it against several competitive routing baselines. The reported improvements in routing accuracy, together with substantial reductions in realized inference cost (up to 82.8% compared with a strong baseline), are encouraging. I also find the ability to incorporate new models using only 700 model-specific historical queries particularly valuable. The additional ablation studies further support the effectiveness of the proposed resource demand representation and progressive routing strategy. Overall, these results provide meaningful empirical evidence for the practical value of the proposed approach.
Critical weaknesses
  1. One of the main claims is that Lens-Router requires only 700 historical queries to incorporate a new model. However, the framework relies on a resource demand encoder pretrained using 59,577 queries annotated with the assistance of multiple frontier LLMs. While this initial investment can be shared across models, its data collection and annotation costs are not fully accounted for in the comparison with existing routers. I would like to see a more comprehensive evaluation of the overall adaptation cost, ideally under comparable supervision budgets. This is important because efficient model adaptation is one of the central motivations of the paper.
  2. The proposed framework relies on capability clustering and several empirical heuristics for cost-based model pruning, including a fixed 5× price-ratio threshold. Although these choices appear reasonable based on the reported experiments, it is unclear how robust they are when the model pool changes or when queries come from different distributions. Additionally, the evaluation primarily focuses on routing accuracy and realized inference cost, without sufficiently examining end-to-end routing latency and system overhead. Since practical deployment is an important motivation of this work, further evidence on these aspects would strengthen the paper's conclusions.
Questions for authors
  1. The paper highlights that Lens-Router requires only 700 historical queries per model, which is an attractive result. However, the resource demand encoder is trained using 59,577 queries with supervision from multiple frontier LLMs. Could the authors clarify the total data collection, annotation, and training costs of the framework? How does the overall cost compare with existing routers when the shared preprocessing overhead is taken into account? I think this would help better establish the claimed scalability advantage.
  2. A central assumption is that resource demand characteristics are largely shared across different LLMs. However, models can have substantially different reasoning mechanisms, architectures, and domain-specific strengths. Could the authors provide more evidence that the learned representation remains effective when adapting to models with significantly different capabilities or reasoning behaviors? In particular, how often does capability-based clustering incorrectly filter out a model that could successfully complete the query?
  3. The framework uses fixed price-ratio thresholds, particularly the 5× threshold for directly pruning expensive models. While the authors provide empirical motivation for this choice, I wonder whether these observations generalize across different model pools and workloads. Could the authors clarify how these thresholds were selected and whether the routing performance is sensitive to them? It would also be helpful to understand how often the pairwise cost predictions produce inconsistent rankings and how these inconsistencies affect the final model selection.
  4. The reported reductions in realized inference cost are impressive, but the routing process itself introduces additional computation, including resource demand encoding, capability prediction, and multiple pairwise comparisons. Could the authors report or clarify the routing latency and computational overhead relative to the baselines? I am particularly interested in whether the cost advantage remains substantial when the routing overhead is included, especially for short or inexpensive queries.
Reviewer_NWvHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Lens-Router, an LLM routing framework for selecting models from a candidate pool, aiming to balance answer quality, inference cost, and the overhead of adding new models. The method consists of three stages: it first trains a resource-demand encoder offline to represent query requirements, then groups models based on their capabilities and filters candidates according to the query's requirements, and finally uses pairwise cost predictions to select a lower-cost model. The authors evaluate Lens-Router on nine benchmarks and different model pools, reporting improved accuracy–cost trade-offs over existing routing methods with 700 historical queries per model for adaptation.

Critical strengths
  1. The paper focuses on an important and practical problem in LLM routing: balancing model accuracy, inference cost, and the overhead of adding new models. The proposed progressive routing framework is intuitive and well-structured.
  2. The paper provides extensive experiments across nine benchmarks and different model pools. The results show promising accuracy–cost trade-offs compared with existing methods, and the ablation studies help illustrate the contributions of different components.
Critical weaknesses
  1. The paper does not adequately justify why resource-demand representations can enable reliable model routing. Models may exhibit substantially different capabilities even for queries with similar resource requirements. The theoretical analysis assumes that resource demand captures routing-relevant information, while the ablation studies mainly demonstrate the benefits of demand-oriented fine-tuning without establishing whether the learned representation preserves the information needed for model-specific routing decisions.
  2. The paper relies on the questionable assumption that model capabilities can be reliably clustered for query-level routing. Model capabilities are inherently multidimensional and task-dependent, and models with similar overall performance may succeed on substantially different subsets of queries. Clustering such models may obscure the capability differences essential for routing. Moreover, the subsequent cost-ranking stage does not reassess individual model correctness, raising concerns about the reliability of the proposed progressive routing framework.
  3. The realized cost modeling is incomplete, considering only output and reasoning tokens while ignoring input-token costs. This omission may distort the relative cost ordering between models, especially for long-context queries or models with different input/output pricing structures, potentially leading to inaccurate routing decisions and overestimated cost savings.
  4. The experimental evaluation is incomplete. The baseline comparisons do not establish performance under matched supervision budgets and accuracy targets. The paper also lacks systematic robustness evaluations of the resource-demand encoder under temporal and distribution shifts, as well as sensitivity analyses of key design choices, such as capability clustering and price-based pruning thresholds.
  5. The paper contains several writing and presentation issues, such as grammatical errors (“these prediction” in Section 2 and “does not exceeds” in Section 3) and incorrect quotation marks (“‘’What is Fermat’s Last Theorem?’” in Section 3). More importantly, Figure 1(b) lacks clear explanations of the data sources, collection date, and criteria used to identify the highlighted capability–price clusters.
Questions for authors
  1. Is the learned resource-demand representation sufficient for model-specific routing? Models may exhibit substantially different capabilities even for queries with similar resource requirements. Although Table 4 compares pretrained and demand-fine-tuned encoders, it does not isolate the benefit of combining semantic and resource-demand information. Please provide controlled comparisons among semantic features, resource-demand features, and their combination under matched training settings.
  2. How are model capability clusters constructed, and can they reliably reflect query-level capabilities? Please clarify the clustering criteria and provide within-cluster correctness agreement and an evaluation of how clustering affects capability filtering and routing accuracy. This would help justify the reliability of the proposed capability-aware filtering.
  3. How can the fixed resource-demand encoder generalize to evolving models and queries? The paper targets a continuously evolving LLM ecosystem, yet relies on a static resource-demand encoder trained on historical data. Please provide temporal generalization and out-of-distribution evaluations, and clarify whether the encoder requires retraining as the model ecosystem evolves.
  4. Table 1 reports benchmark results as percentages, yet OmniScience includes negative scores. Are these metrics directly comparable, and how are they normalized or aggregated when computing average performance across benchmarks?
1 comment

Thank you for the response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.