- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a procurement platform for LLM inference in which providers bid for a user’s task, and the platform routes queries through a reverse second-price auction combined with UCB-style elimination of providers below a user-defined quality threshold. Providers do not know their own per-task cost and are shown to be worst-case optimal when bidding their running cost estimate. The authors prove that routing concentrates on the cheapest qualified provider and that average payments converge to the second-lowest qualified cost, and support this with simulations on GSM8K and GPQA using Llama and Qwen models.
Critical strengths
Problem framing: Treating LLM routing as a procurement auction rather than price-taking over listed rates is a new and well-motivated angle; the target price is the natural Bertrand benchmark.
Clean mechanism: The cost-side optimism term enters both the allocation index and the critical payment, so that truthful bidding and the UCB sandwich (Lemma 5) hold simultaneously. The proofs are complete and easy to follow.
Critical weaknesses
The provider model reduces each provider to a stationary scalar quality and a scalar mean cost, with no latency, throughput, or availability, which existing routing platforms treat as first-class criteria. Bids and payments are per query, while the targeted market prices per token, so the “token price” framing in the abstract does not match the model.
The guarantees hold for a single fixed task distribution and must be relearned per task at considerable cost. Excluding a provider 0.037 below takes about 11,000 queries, and within the first pass over each benchmark 24% to 47% of queries are exploration steps paid at (Tables 8 and 26), borne entirely by the user. The 10%/71% margins are computed on non-exploration steps only, and the appendix baselines are absent from the main text: routing to the lowest listed price among the same estimated-qualified set pays less than the platform over the long run on three of the five rosters (Tables 7,16 and 25), so the net benefit to the user is not established.
Strategic behaviour is assumed away rather than analyzed. The bandit is stochastic, and provider incentives are settled by a worst-case argument that sets , reducing the dynamic game to a one-shot auction. This leaves out an incentive the mechanism itself creates: the critical payment includes , so a provider that deliberately loses rounds keeps small and is paid more when it next wins. The restriction that only the selected provider may revise its bid, and the absence of any experiment with non-truthful or model-switching providers, leave the strategic-arms setting the paper cites (Braverman et al., 2019; Cao et al., 2026) untested.
Questions for authors
-
Which step of the proof of Theorem 2 fails if non-selected providers are allowed to update their bids at every round? Is the restriction needed, or only convenient?
-
If a provider knows (or a lower bound on it), is the strategy of overbidding on non-exploration rounds to preserve a large ever profitable under the stated payment rule? A simulation with one such provider would help.
-
Can latency or throughput be folded into without breaking the analysis, and if so, what does the resulting scalar threshold mean to a user who wants hard quality guarantees but only soft latency preferences?
2 comments
Rebuttal (cont.)
[Optimal strategy under known T] Knowledge of the horizon , or a lower bound on it, does not guarantee that deliberately losing early rounds is profitable. The gain depends on competitors’ private costs and qualities. In Appendix G of the revised submission, we give a two-provider example in which overbidding is profitable for every , while respecting the original standing-bid update rule. In this example, delaying a win lets the competitor’s confidence radius shrink and its adjusted bid increase, raising the later payment. The deviator’s own confidence radius contribution at its win remains unchanged. More generally, note that the term in the payment is the same for a given number of previous selections, regardless of when the win occurs. This possibility is consistent with Theorem 2, which establishes optimality under a worst-case horizon criterion.
In addition, in Appendix G, we also bound the gain from any unilateral bidding deviation against competitors that bid their cost estimates, even if the deviator knows the horizon and competitors’ types. On the high-probability event , the cumulative additional estimated profit is at most , yielding a vanishing bound on the gain per query. Consequently, the corresponding expected-utility bound establishes that cost-estimate bidding is an approximate Nash equilibrium.
[Proof of Theorem 2] No step of the proof of Theorem 2 fails if non-selected providers are allowed to (simultaneously) update their bids at every round. Bidding the cost estimate still ensures that payments cover that estimate, and the argument that the current round may be the last remains valid. That said, please note that, at equilibrium, as long as a provider is not selected, the current bid remains optimal for the next round, since its cost estimates have not changed. The restriction is therefore only a convenience. In the revised submission, we have clarified this whenever we state Theorem 2.
References
- Morse, S. (2026, October 28). How we rebuilt provider routing for better, faster inference. OpenRouter. https://openrouter.ai/blog/announcements/how-we-rebuilt-provider-routing/
Rebuttal
We thank the reviewer for their thoughtful feedback. Below, we respond in detail to each of the points they raised. In the revised submission, we have highlighted changes in blue.
[Latency] We make no assumptions about the score r(x,y) used by the user to evaluate a model’s response beyond boundedness. Consequently, our theoretical results on the providers’ equilibrium bidding (Theorem 2) and user guarantees (Theorems 3 and 4) remain valid if the user includes other response metrics, such as latency or provider availability, as long as the resulting score is normalized to the interval [0,1]. For example, for queries with verifiable ground truth, one can define a generalized scalar score as an indicator of whether the model’s response is correct, minus a bounded term penalizing each additional second of latency. This captures how routing platforms such as OpenRouter trade off model quality against latency, throughput, and time to first token [1].
In Appendices E and F of the revised submission, in response to the reviewer’s comment, we have extended our platform and its theoretical guarantees to: (i) multidimensional scores and thresholds, which can encode multiple hard constraints on the mean values of different response metrics, for example, quality and latency, and (ii) a selection and payment rule that accounts for both cost and latency, and thus can encode soft constraints on latency. In extension (i), our theoretical guarantees retain the same form, whereas in extension (ii), we prove that, with high probability, all but a vanishing fraction of queries are routed to the provider offering the best cost and latency tradeoff among those meeting the quality threshold.
[Token price] In the revised abstract, we will clarify that our procurement platform sets prices per response rather than per token.
[Number of required queries] Any routing mechanism seeking to learn unknown provider qualities and costs requires observing a minimum number of responses to user queries, and therefore a sufficient number of routed queries, to provide statistical guarantees. In Theorems 3, 4, and 7, we establish finite-time guarantees on identifying the most competitive qualified provider, payment convergence, and unqualified selections.
[Number of exploration steps] While our experiments show that exploration can initially result in a relatively high number of exploration steps, these rapidly amortize as the platform learns. On the final pass, the fraction of queries assigned to the optimal qualified provider is over 96% of all queries. Because the reported 10%/71% margins are calculated over the final pass, which is heavily dominated by non-exploration steps, these values do not significantly change if computed over both step types, directly resolving the reviewer’s concern.
[Net benefit to the user] The goal of our platform is not to reduce user payments relative to routing to the provider with the lowest listed price (among the same estimated-qualified set). Instead, it is designed to guarantee that users pay task-specific competitive prices. This requires identifying the lowest-cost qualified provider and ensuring that average payments converge to the second-lowest mean cost among qualified providers. Whether this results in users paying higher or lower prices depends on how competitive the LLM market is for the specific user task, as we extensively demonstrate in our experiments in Section 5 and Appendices I, J, and K. We have clarified this in Section 5 of the revised submission.