- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces RTRouter, a technique for routing LLM inference requests balancing latency, quality, and cost. The approach is based on a multi-step process. First, a learned predictor that estimates a model's expected response quality along with the expected output length is used to filter for the most relevant models. Second, the subset is used to select the candidate model that balances cost, latency, and quality. The approach is evaluated over LLMRouterBench.
Critical strengths
The paper addresses a timely challenge and provides an integrated view on LLM request routing based on quality, latency, and cost criteria, all of which matter for practical applications. The paper provides detailed ablation studies of individual algorithm components, which offer the reader a good level of decomposition. From a deployment perspective, the router appears to be easy to operate with little overhead.
Critical weaknesses
I have several major concerns about the paper:
- While the authors admit the limitation that RTRouter is evaluated on its own objective, I think, this is more than just a limitation. It undermines the main claim of the paper to be cost and quality aware. The circular dependency is never resolved in the ablations, let alone is there a theoretical analysis.
- The paper lacks a discussion about guarantees the routing algorithm provides. The multi-stage process does filter candidates but it is unclear how the estimator's error rate affects the selection of candidates (and the optimality of the first stage pool). I think, the paper could benefit from an analysis through the lens of calibrated sets or via the Lyapunov drift-plus-penalty framework.
- The latency considerations are somewhat superficial and assume impractical setups. The queue mechanism discussed in the paper assumes on request per second and no batching of requests. Modern inference services like vLLM use dynamic batching and efficiency/throughput grows as the batch size increases. Hence, the results discussed in the experiments section might be inflated.
Some other observations: First, the paper misses a notable amount of related work on LLM request routing, especially in the space of bandits, Lyapunov drift-plus-penalty optimization, and causal routing). Some of those works only address parts of the problem but deliver guarantees on routing optimality that could be useful for the appraoch discussed in this paper.
- Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads. Patel et al., 2026.
- BARouter: A Budget-adaptive Online LLM Router Framework. Zu et al., 2026, WWW.
- PROTEUS: SLA-Aware Routing via Lagrangian RL for Multi-LLM Serving Systems. Bhatti et al., 2026.
- MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees. Woisetschläger et al., 2025, NeurIPS.
- Causal LLM routing: End-to-end regret minimization from observational data, Tsiourvas et al., 2025, NeurIPS.
Second, a minor remark given the cost of running large LLMs. The models discussed in the paper are overall very small. A frequently discussed scenario is the routing decision between frontier models and a (locally deployed) small model to balance cost and accuracy. Showing the router's performance under a large performance gap between models (size- or fine-tuning induced) might strengthen the paper's claims.
Questions for authors
- I appears the adaptive gate mechanism does not depend on the load. only depends on the quality and never introduces load. How does it enter the decision making process?
- The paper claims "adaptive thresholds are essential". A fixed value shows better performance than the adaptive mechanism. How is that claim justified and what does the pareto frontier for look like?
- How much does each component of RTRouter contribute to the overall performance? The ablations show the performance of each isolated but do not explain why the combination leads to stronger performance overall.
- How were the hyperparameters chosen (, , k, , etc.)?