- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors introduce an abstract formulation of model routing based on reverse auctions, and study the theoretical properties of the framework. In practice, my understanding is that the novelty is in fitting two models: a p^_m(x) that measures probability that model m is correct on x, and a "buyer acceptance model" that measures if the answer a is correct for , plus some kind of theoretically sound auction framework that makes use of this second model. Some experiments are ran on Routerbench.
Critical strengths
- The idea of a centralized routing platform that allows users cheaper access to models through an auction system is interesting and novel.
- The theoretical results are interesting
Critical weaknesses
- I think the point of the experiments is a bit incoherent with the rest of the paper. As best as I can tell the central thesis is something like: "If LLM companies have better estimates of their models accuracy, can a central platform get good routing estimates through an acceptance model and some kind of auction mechanism? " Where as the empirical work focuses on "can we route better on Routerbench?" I recognize some of their ideas are not really testable but now sure how valuable this section is
- Sort of ties in but Routerbench is sort of out of data dataset for doing this kind of experiment at this point.
Questions for authors
- Do you think its possible to show case some of the more central ideas on a real routing dataset at all?
3 comments
W2 & Q1 (Why RouterBench Is Suitable for Evaluating EA-RAM on Real Routing Data)
Thank you for raising this concern. We choose RouterBench because it provides the information needed to evaluate EA-RAM's core ideas using real LLM routing data, for two practical reasons:
- Standardized comparison with existing routers: RouterBench is widely used to evaluate LLM routing methods, including EmbedLLM, allowing us to directly compare EA-RAM with established centralized baselines under the same experimental conditions.
- Complete records required for auction evaluation: RouterBench provides much more than aggregate routing accuracy. For each query, it includes (i) the original task, covering six diverse benchmarks (MBPP, GSM8k, MMLU, ARC, HellaSwag, and WinoGrande); (ii) the generated responses from all 11 candidate LLMs; (iii) the correctness of each model's response; and (iv) token costs. These records allow us to determine what each possible winning provider would produce, whether its response would be correct, and how much it would cost. This makes it possible to train model-specific performance predictors, simulate auction allocations under different provider-side information settings, and directly measure routing quality, costs, and market welfare using actual LLM outcomes. Moreover, the recorded responses allow us to test noisy outcome verification with a real LLM-as-a-Judge (DeepSeek-V3.2, Section 5.2.3). Thus, RouterBench provides the detailed evidence needed to examine the practical behavior of EA-RAM.
Although RouterBench may not cover the latest generation of LLMs, it provides the comprehensive data needed to evaluate EA-RAM's core properties across diverse routing tasks. We will also discuss the potential value of extending our evaluation to newer models in the Limitations section.
We hope our response has addressed your concerns, and we will gladly and promptly respond to any further questions you may have.
Thank you again for your thoughtful review. We have posted our detailed response along with new experimental results and manuscript updates. When you have a moment, we would greatly appreciate hearing whether these additions address your concerns, and we remain happy to answer any follow-up questions.
We sincerely thank you for your positive assessment of our auction-based routing formulation and theoretical results. Below, we address your concerns in detail.
W1 (Clarifying Our Core Motivation and Experimental Alignment)
Thank you for this question. We agree that it is important to clarify how our experiments support the central ideas of EA-RAM. Our goal is not simply to improve routing accuracy on RouterBench, but to enable more effective LLM routing by using providers' private information through an incentive-aligned auction mechanism.
Specifically, our work is motivated by three limitations of existing centralized LLM routing:
These challenges motivate EA-RAM: instead of requiring the platform to estimate every model's performance, we let providers use their own information to predict acceptance probabilities and submit bids. Our auction mechanism incentivizes truthful reporting and ties payments to evaluation outcomes. In this way, the platform can benefit from providers' private information without requiring them to disclose their underlying data, while reducing the burden of centralized prediction.
Our experiments are designed to validate different aspects of this mechanism using two complementary approaches. Controlled simulations (Section 5.1) allow us to systematically examine how prediction and evaluation errors affect the mechanism under different conditions. Experiments on real routing data (Section 5.2) then evaluate whether the proposed design delivers practical benefits using actual LLM performance records, costs, and evaluation outcomes. Specifically:
Thus, our experiments are not simply intended to show better routing scores. The simulations examine how the mechanism works under controlled conditions, while the real-data experiments demonstrate its practical benefits in information utilization, robustness, and scalability. Together, they provide complementary evidence supporting the motivation and design of EA-RAM.