acceptodds
Under review as a conference paper at ICLR 2027

Every Second Counts: Quality-Cost-Aware LLM Routing for Low-Latency Concurrent Multi-LLM Serving

Abstract

As the Large Language Model (LLM) ecosystem grows, LLM routing is emerging as an important approach to balancing response quality and inference cost. Since real-world LLM serving involves continuously arriving requests, backlog-induced latency and quality degradation can no longer be overlooked. However, existing methods optimize only quality and cost, largely because they omit backlog-induced latency. In addition, they directly select a model without an admission mechanism, leading to quality degradation. To address these limitations, we propose RTRouter, a quality-cost-aware LLM routing framework that couples workload-awareness with quality admission. Specifically, we represent LLMs' unfinished work backlog as workload and update it online to avoid missing backlog-induced latency. Furthermore, we adaptively adjust the degree of pruning based on the quality vector and prune candidate models to obtain the feasible set, avoiding quality degradation. Extensive experiments on comprehensive routing benchmarks demonstrate that RTRouter improves the quality-cost-latency trade-off and effectively eliminates quality degradation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM routing, multi-LLM serving, load-aware scheduling, quality-cost-latency tradeoff

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bXHLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces RTRouter, a technique for routing LLM inference requests balancing latency, quality, and cost. The approach is based on a multi-step process. First, a learned predictor that estimates a model's expected response quality along with the expected output length is used to filter for the most relevant models. Second, the subset is used to select the candidate model that balances cost, latency, and quality. The approach is evaluated over LLMRouterBench.

Critical strengths

The paper addresses a timely challenge and provides an integrated view on LLM request routing based on quality, latency, and cost criteria, all of which matter for practical applications. The paper provides detailed ablation studies of individual algorithm components, which offer the reader a good level of decomposition. From a deployment perspective, the router appears to be easy to operate with little overhead.

Critical weaknesses

I have several major concerns about the paper:

  • While the authors admit the limitation that RTRouter is evaluated on its own objective, I think, this is more than just a limitation. It undermines the main claim of the paper to be cost and quality aware. The circular dependency is never resolved in the ablations, let alone is there a theoretical analysis.
  • The paper lacks a discussion about guarantees the routing algorithm provides. The multi-stage process does filter candidates but it is unclear how the estimator's error rate affects the selection of candidates (and the optimality of the first stage pool). I think, the paper could benefit from an analysis through the lens of calibrated sets or via the Lyapunov drift-plus-penalty framework.
  • The latency considerations are somewhat superficial and assume impractical setups. The queue mechanism discussed in the paper assumes on request per second and no batching of requests. Modern inference services like vLLM use dynamic batching and efficiency/throughput grows as the batch size increases. Hence, the results discussed in the experiments section might be inflated.

Some other observations: First, the paper misses a notable amount of related work on LLM request routing, especially in the space of bandits, Lyapunov drift-plus-penalty optimization, and causal routing). Some of those works only address parts of the problem but deliver guarantees on routing optimality that could be useful for the appraoch discussed in this paper.

  • Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads. Patel et al., 2026.
  • BARouter: A Budget-adaptive Online LLM Router Framework. Zu et al., 2026, WWW.
  • PROTEUS: SLA-Aware Routing via Lagrangian RL for Multi-LLM Serving Systems. Bhatti et al., 2026.
  • MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees. Woisetschläger et al., 2025, NeurIPS.
  • Causal LLM routing: End-to-end regret minimization from observational data, Tsiourvas et al., 2025, NeurIPS.

Second, a minor remark given the cost of running large LLMs. The models discussed in the paper are overall very small. A frequently discussed scenario is the routing decision between frontier models and a (locally deployed) small model to balance cost and accuracy. Showing the router's performance under a large performance gap between models (size- or fine-tuning induced) might strengthen the paper's claims.

Questions for authors
  • I appears the adaptive gate mechanism does not depend on the load. u(q)u(q) only depends on the quality and never introduces load. How does it enter the decision making process?
  • The paper claims "adaptive thresholds are essential". A fixed δ\delta value shows better performance than the adaptive mechanism. How is that claim justified and what does the pareto frontier for δ\delta look like?
  • How much does each component of RTRouter contribute to the overall performance? The ablations show the performance of each isolated but do not explain why the combination leads to stronger performance overall.
  • How were the hyperparameters chosen (Φ\Phi, α\alpha, k, β\beta, etc.)?
Reviewer_vNAzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RTRouter, a framework for balancing quality, cost, and latency when routing continuously arriving requests across multiple LLMs. It combines offline quality and output-length prediction, query-dependent quality admission, and online tracking of predicted unfinished work. The router first filters candidates by predicted quality, then selects among them using quality, cost, and backlog-aware latency scores. Experiments on LLMRouterBench with 20 models and three simulated Poisson arrival rates report improved trade-offs over the evaluated baselines. Ablations examine the contributions of workload awareness and quality admission. Latency results are obtained from a prediction-based simulator.

Critical strengths
  1. The paper addresses a practically important limitation of quality-cost routing: concentrating requests on a few models can create substantial queues. Combining predicted-quality admission with workload-aware dispatch provides a clear framework for balancing response quality and congestion.

  2. The component ablations provide useful supporting evidence within the simulated setting. Workload awareness reduces predicted mean latency from 79.58 to 10.10 at unchanged reported routing accuracy, while quality admission improves accuracy from 0.667 to 0.813. These comparisons help establish the contributions of the two components.

Critical weaknesses
  1. The central latency claims are supported only by an unvalidated predicted-latency simulator. The same service-time estimates drive routing, workload updates, and reported mean/p99 latency, without independent measurements of realized service times. The evaluation therefore does not establish whether the gains persist under actual batching, concurrent execution, and prediction errors. This is consequential because low-latency concurrent serving is a central contribution of the paper.

  2. The rationale and incremental benefit of the adaptive admission mechanism are insufficiently established. In Section 4.3, the admission bandwidth depends only on frozen query-conditioned quality predictions, whereas Section 5.3 attributes its benefits to adaptation to real-time congestion. Furthermore, the characterization of RouteBalance's latency term as unbounded conflicts with its candidate-max normalization (Da and Kalyvianaki, 2026, Eq. 1). Comparing against only one fixed bandwidth does not establish an improved quality-cost-latency frontier over appropriately tuned alternatives.

Questions for authors
  1. Can the latency gains be validated using independently measured service times or a small live deployment? How sensitive are the results to service-time prediction errors and backend saturation?

  2. Does the admission bandwidth depend on workload or arrival rate in the implementation? Please reconcile Section 4.3 with the congestion-adaptation explanation in Section 5.3.

  3. Does RTRouter retain its advantage at matched quality when sweeping fixed admission bandwidths and RouteBalance's preference weights under the same prediction and evaluation setup?

Reviewer_c71ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In concurrent multi-LLM serving systems, the authors point out that traditional quality-cost-aware routers overlook queuing delays caused by request backlogs, while existing latency-dominated routing methods such as RouteBalance may sacrifice response quality to reduce latency. This paper proposes RTRouter, which uses a Quality Admission mechanism to construct a high-quality candidate model set for each query, reducing the risk of selecting low-quality models. Through Workload-Aware Routing, RTRouter selects a model from the candidate set by jointly considering predicted quality, inference cost, and current workload, aiming to maintain response quality while reducing cost and request latency.

Critical strengths
  • S1: Practical significance. The paper addresses an important problem in concurrent multi-LLM serving: how to maintain response quality while balancing inference cost and request latency.
  • S2: Quality-aware routing design. The combination of Quality Admission and Workload-Aware Routing is intuitive. Quality Admission restricts the selection of low-quality models, while Workload-Aware Routing considers cost and current workload when selecting a model from the candidate set, balancing response quality and request latency.
  • S3: Relatively comprehensive comparisons and ablation studies. The paper compares RTRouter with multiple existing LLM routing methods and conducts ablation studies to analyze the contributions of its main components.
Critical weaknesses

W1: Quality Admission may lead to localized overload and underutilization of resources.

RTRouter restricts the candidate model set through Quality Admission. When all backend models satisfying the quality constraints are overloaded, RTRouter can only continue routing requests to these models, potentially causing persistent queuing and high request completion latency. Furthermore, under skewed workloads, such as when a large number of requests have identical or highly overlapping Quality Admission candidate sets, some models may become severely overloaded while others remain idle for extended periods. The paper lacks theoretical analysis and experimental validation of such scenarios.

W2: Insufficient motivation and comparison with related work.

The paper argues that existing quality-cost-aware routers tend to concentrate workloads on a small number of backend models, while latency-dominated routing methods such as RouteBalance may sacrifice response quality to reduce latency. However, prior work, Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads (arXiv:2607.18253v1), adopts a routing strategy that first filters candidate models based on TTFT SLO constraints and then maximizes quality-cost utility, similarly considering the trade-off among latency, response quality, and inference cost. The paper lacks a discussion of this related work.

W3: Limited experimental scale and lack of validation on a real multi-model serving system.

The paper uses only 300 queries per experimental run and evaluates three request arrival rates (1, 2, and 8 requests/s), making it difficult to assess routing performance under dynamic workload variations in high-concurrency scenarios. More importantly, the reported Mean Latency and p99 Latency are based on predicted-latency simulation rather than end-to-end latency measurements from a real multi-model serving system. The current experiments are therefore insufficient to fully substantiate RTRouter's performance benefits in real-world high-concurrency environments.

Questions for authors
  • Q1: Please analyze RTRouter's performance under skewed workloads, particularly when a large number of requests have identical or highly overlapping Quality Admission candidate sets. Could this lead to localized overload and underutilization of the overall model pool? Please provide corresponding experimental validation.
  • Q2: Please provide a comparison with Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads, and experimentally validate the advantages of RTRouter.
  • Q3: Please supplement the evaluation with experiments on a real multi-model serving system, increase the number of evaluation queries, and cover a wider range of request arrival rates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.