acceptodds
Under review as a conference paper at ICLR 2027

Error-Aware Reverse Auction Mechanism for Large Language Model Routing

Abstract

Routing each query to a cost-effective large language model (LLM) is critical for balancing quality and cost, yet most routers rely on a centralized task center to predict model performance, creating an information-risk mismatch and a scalability bottleneck as the model pool grows. We formulate LLM routing as a market-based allocation problem among strategic providers and propose a routing paradigm that shifts ex-ante prediction to LLM providers via a reverse auction, where providers submit self-predicted acceptance probabilities and execution costs. To account for noisy provider predictions and center evaluations, we introduce the Error-Aware Reverse Auction Mechanism (EA-RAM), which explicitly models this Dual Error. We prove that, under a private-evaluation-belief structure, truthful effective-surplus reporting is incentive compatible in the reduced-form score space and individually rational under sellers' subjective beliefs, establish sufficient conditions for center rationality, and derive an explicit social-welfare loss bound. We further identify robustness effects: opposite-signed errors can cancel, vanishing-tail link functions (e.g., logistic) stabilize clear-cut cases via saturation, and extra noise smooths belief maps and reduces their maximal local sensitivity. Simulations and real-world benchmarks show that EA-RAM is robust to Dual Error and achieves a better cost–performance Pareto frontier than centralized baselines, with additional gains from provider-side local information, validating its practical effectiveness.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Mechanism Design, Auction, LLM Routing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9f1MOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce an abstract formulation of model routing based on reverse auctions, and study the theoretical properties of the framework. In practice, my understanding is that the novelty is in fitting two models: a p^_m(x) that measures probability that model m is correct on x, and a "buyer acceptance model" h(a,xm)h(a,x_m) that measures if the answer a is correct for xmx_m, plus some kind of theoretically sound auction framework that makes use of this second model. Some experiments are ran on Routerbench.

Critical strengths
  • The idea of a centralized routing platform that allows users cheaper access to models through an auction system is interesting and novel.
  • The theoretical results are interesting
Critical weaknesses
  • I think the point of the experiments is a bit incoherent with the rest of the paper. As best as I can tell the central thesis is something like: "If LLM companies have better estimates of their models accuracy, can a central platform get good routing estimates through an acceptance model and some kind of auction mechanism? " Where as the empirical work focuses on "can we route better on Routerbench?" I recognize some of their ideas are not really testable but now sure how valuable this section is
  • Sort of ties in but Routerbench is sort of out of data dataset for doing this kind of experiment at this point.
Questions for authors
  • Do you think its possible to show case some of the more central ideas on a real routing dataset at all?
3 comments

We sincerely thank you for your positive assessment of our auction-based routing formulation and theoretical results. Below, we address your concerns in detail.

W1 (Clarifying Our Core Motivation and Experimental Alignment)

Thank you for this question. We agree that it is important to clarify how our experiments support the central ideas of EA-RAM. Our goal is not simply to improve routing accuracy on RouterBench, but to enable more effective LLM routing by using providers' private information through an incentive-aligned auction mechanism.

Specifically, our work is motivated by three limitations of existing centralized LLM routing:

  1. Information mismatch: Providers naturally know more about their own models' capabilities and costs than a third-party routing platform.
  2. Risk–responsibility mismatch: The routing platform bears the risk of choosing an unsuitable model, while providers are typically paid for inference regardless of whether their outputs satisfy the user's needs.
  3. Scalability bottleneck: As new models enter the market, centralized routers must continuously profile new models or update their performance predictors.

These challenges motivate EA-RAM: instead of requiring the platform to estimate every model's performance, we let providers use their own information to predict acceptance probabilities and submit bids. Our auction mechanism incentivizes truthful reporting and ties payments to evaluation outcomes. In this way, the platform can benefit from providers' private information without requiring them to disclose their underlying data, while reducing the burden of centralized prediction.

Our experiments are designed to validate different aspects of this mechanism using two complementary approaches. Controlled simulations (Section 5.1) allow us to systematically examine how prediction and evaluation errors affect the mechanism under different conditions. Experiments on real routing data (Section 5.2) then evaluate whether the proposed design delivers practical benefits using actual LLM performance records, costs, and evaluation outcomes. Specifically:

  • Leveraging provider-side information (Section 5.2.2, Figure 6 & Table 3): We first show that base EA-RAM achieves competitive cost–performance results without additional private information (average AIQ 0.80959 vs. 0.79832 for EmbedLLM). More importantly, adding provider-side local information further improves average AIQ, demonstrating the potential value of information available locally to providers.
  • Handling imperfect predictions and evaluations (Sections 5.1 and 5.2.3, Figures 5 & 7): Our simulations examine how different prediction and evaluation errors affect welfare, while experiments using real routing data and a noisy LLM-as-a-Judge show that EA-RAM maintains lower welfare loss under imperfect evaluation. Together, these experiments connect our theoretical error analysis with practical routing outcomes.
  • Reducing routing overhead (Section 5.2.4, Figure 8 & Tables 4–5): We evaluate how the system scales as the number of candidate models increases from 3 to 11. EA-RAM requires no additional center-side predictor retraining and maintains nearly constant buyer-side computation time (~0.042s), while also quantifying the extra communication overhead introduced by decentralized bidding.

Thus, our experiments are not simply intended to show better routing scores. The simulations examine how the mechanism works under controlled conditions, while the real-data experiments demonstrate its practical benefits in information utilization, robustness, and scalability. Together, they provide complementary evidence supporting the motivation and design of EA-RAM.

W2 & Q1 (Why RouterBench Is Suitable for Evaluating EA-RAM on Real Routing Data)

Thank you for raising this concern. We choose RouterBench because it provides the information needed to evaluate EA-RAM's core ideas using real LLM routing data, for two practical reasons:

  • Standardized comparison with existing routers: RouterBench is widely used to evaluate LLM routing methods, including EmbedLLM, allowing us to directly compare EA-RAM with established centralized baselines under the same experimental conditions.
  • Complete records required for auction evaluation: RouterBench provides much more than aggregate routing accuracy. For each query, it includes (i) the original task, covering six diverse benchmarks (MBPP, GSM8k, MMLU, ARC, HellaSwag, and WinoGrande); (ii) the generated responses from all 11 candidate LLMs; (iii) the correctness of each model's response; and (iv) token costs. These records allow us to determine what each possible winning provider would produce, whether its response would be correct, and how much it would cost. This makes it possible to train model-specific performance predictors, simulate auction allocations under different provider-side information settings, and directly measure routing quality, costs, and market welfare using actual LLM outcomes. Moreover, the recorded responses allow us to test noisy outcome verification with a real LLM-as-a-Judge (DeepSeek-V3.2, Section 5.2.3). Thus, RouterBench provides the detailed evidence needed to examine the practical behavior of EA-RAM.

Although RouterBench may not cover the latest generation of LLMs, it provides the comprehensive data needed to evaluate EA-RAM's core properties across diverse routing tasks. We will also discuss the potential value of extending our evaluation to newer models in the Limitations section.

We hope our response has addressed your concerns, and we will gladly and promptly respond to any further questions you may have.

Thank you again for your thoughtful review. We have posted our detailed response along with new experimental results and manuscript updates. When you have a moment, we would greatly appreciate hearing whether these additions address your concerns, and we remain happy to answer any follow-up questions.

Reviewer_82V4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EA-RAM, a reverse-auction mechanism for LLM routing in which providers estimate their own task success probabilities and submit bids. The mechanism accounts for errors in both provider predictions and the buyer's evaluation. The authors establish incentive and welfare properties under specific assumptions and evaluate the approach using simulations and RouterBench.

Critical strengths

The paper explores an interesting alternative to centralized LLM routing by introducing strategic providers and market-based allocation. Considering uncertainty in both provider predictions and buyer evaluations is a meaningful extension of the standard execution-uncertainty setting. The experimental results are promising, particularly the cost–performance comparisons across multiple benchmarks.

Critical weaknesses
  1. Unclear theoretical novelty and restrictive assumptions.

The paper devotes substantial attention to theoretical properties, but the novelty relative to classical mechanism design remains unclear. The allocation and payment structure largely follow execution-contingent mechanisms such as Porter et al. (2008). The incentive-compatibility result relies on private evaluation beliefs, and the welfare bound follows from standard error and Lipschitz arguments. The additional robustness properties, including error cancellation and noise-induced smoothing, also appear mathematically straightforward.

More importantly, the assumptions are strong: providers are risk-neutral, tasks have publicly known binary values, the evaluator is fixed and committed, and sellers' beliefs about evaluation do not depend on competitors' bids. Their applicability to real LLM markets needs stronger justification. The paper does not clearly establish whether its primary contribution is a new mechanism-design result or an empirical routing framework.

  1. Limited realism and unclear source of empirical improvements.

The real-world experiments use offline RouterBench data and author-trained provider predictors rather than actual strategic providers. Consequently, they do not directly validate truthful bidding, payment incentives, or the proposed decentralized market.

It is also unclear how much of the improvement comes from the auction mechanism rather than the available performance estimates. A centralized router with access to the same provider predictions could make the same allocation. Furthermore, the scalability evaluation mainly measures buyer-side computation, while provider-side prediction and actual communication delays are not fully accounted for. These issues limit the evidence for the claimed practical benefits.

The presentation also needs improvement. Several figures contain small labels and legends (particularly Figures 5–8), and the plotting styles are inconsistent. Colored method names in captions further reduce readability without providing much benefit.

Questions for authors
  1. What is the main theoretical advance over Porter et al. (2008)? In particular, which results are specific to the proposed dual-error setting rather than direct consequences of existing execution-contingent mechanisms?

  2. How realistic are the private-evaluation-belief assumption, common task value, and execution-contingent payments for actual LLM providers? Can the authors clarify how the mechanism would operate when providers have correlated information, strategic behavior, and limited willingness to accept negative payments?

  3. Can the authors isolate the contribution of the auction mechanism from that of provider-side prediction? A comparison against centralized routing with the same provider estimates, together with a more complete end-to-end cost and latency analysis, would help substantiate the claimed benefits.

Porter, R., Ronen, A., Shoham, Y., and Tennenholtz, M. (2008). Fault Tolerant Mechanism Design.

Takahashi, H. (2018). Strategic Design under Uncertain Evaluations: Structural Analysis of Design-Build Auctions.

Rontogiannis, D., Artola Velasco, A., and Gomez Rodriguez, M. (2026). Learning the Cost of Reliable Inference.

6 comments

We sincerely thank you for your insightful assessment. Below, we address your concerns in detail.

W1, Q1, & Q2 (Theoretical Novelty, Advances over FTMD, and Modeling Assumptions)

1. Theoretical Novelty (W1 & Q1)

Thank you for this important question. Our work is motivated by a practical challenge in LLM routing: providers and the routing platform have different information about model capabilities, while neither prediction nor evaluation is perfect. To address this challenge, we develop EA-RAM, an auction mechanism tailored to this new setting. Its theoretical challenge arises because the information-alignment assumptions underlying FTMD no longer hold in LLM routing: providers predict their own performance, while the center rely on an imperfect evaluator to determine payment.

In FTMD, the provider's true success probability, evaluator acceptance probability, and provider belief coincide:

pi=hi=gi.p_i = h_i = g_i.

In our Dual-Error setting, they may differ:

\xrightarrow{\epsilon_{\mathrm{ante},i}} g_i.$$ Here, $p_i$ determines true welfare, $h_i$ determines the expected payment, and $g_i$ determines the provider's bidding incentives. **Our theoretical contribution is to establish which guarantees survive this separation and what conditions are needed for those that no longer hold automatically.** | Property | FTMD (Idealized Error-Free Setting) | EA-RAM under Dual Error | | :--- | :--- | :--- | | **Provider incentives** | Providers are assumed to know their true success probabilities, and task outcomes can be verified without error. Under these ideal conditions, truthful bidding is optimal. | Providers may have imperfect predictions and face an imperfect evaluator. Even so, bidding honestly based on their own belief about passing evaluation remains optimal (Theorem 4.2). When competitors' bids shift that belief, we also bound the benefit of strategic misreporting (Proposition 4.4). | | **Buyer protection** | The buyer knows whether the task actually succeeded and pays accordingly, so it does not lose money. | The evaluator may accept an incorrect answer, causing the buyer to pay for a failed task. We identify conditions that protect the buyer from losing money on average (Theorem 4.5). | | **Social welfare** | Providers' bids reflect their true success probabilities, so the auction selects the provider with the highest expected value after cost. | Predictions and evaluations can both be imperfect, so the selected provider may not be the welfare-optimal one. We bound the resulting welfare loss and show that the two errors can sometimes offset each other (Theorem 4.8 and Proposition 4.9). | Thus, EA-RAM is not a routine reapplication of FTMD, but a new mechanism under LLM routing's dual-sided estimation noise. #### 2. Practical Rationale for Modeling Assumptions (W1 & Q2) Your considerations are very thorough. In fact, we carefully considered the practical realism of these assumptions when designing the mechanism. Below, we combine brief practical examples with our theoretical results to illustrate how these assumptions can be naturally satisfied in real-world scenarios: - **Risk neutrality:** Risk neutrality is a standard assumption in execution-contingent task-allocation mechanisms[1]. It is also a reasonable approximation for commercial LLM providers handling large volumes of requests, where the financial uncertainty of a single query is small relative to their overall operating scale. Classical risk-aversion theory[2] further shows that the risk premium for small monetary risks is a second-order effect. Thus, **maximizing expected profit provides a reasonable approximation for providers' per-query decisions** in high-volume LLM markets. - **Publicly known task value ($V$):** In our setting, **$V$ can be a reference value announced by the routing platform before bidding**, based on the user's selected service tier or stated willingness to pay for a successfully completed task. Such public buyer-side valuations are well established in procurement mechanism design[1, 3]. For example, a user may select a service tier valued at \$V=\$0.02\$ per successful response, and the platform announces this value alongside the query.
  • Stable and committed evaluator: We only require the evaluator to remain unchanged within each auction round, not forever. This is a natural requirement for fairness and market stability. Just as auction rules should not change after bids are submitted, the platform should not change its evaluation criteria after providers have committed to their bids. Otherwise, providers could be evaluated and paid under rules different from those they originally agreed to, making the market unfair and undermining their trust and willingness to participate. This is consistent with the classical notion of credible commitment in auction design[4]. The platform can still upgrade its evaluator over time, as long as changes are announced before new bidding rounds. Providers can then collect feedback under the updated evaluator and adjust their local predictors accordingly. Thus, our assumption ensures fair and predictable market operation without preventing future improvements to the evaluator.
  • Correlated information from competitors' bids (HH) and strategic behavior: In Section 4.2 (Proposition 4.4), we explicitly model how a provider may strategically adjust its bid using correlated information inferred from competitors' bids HH, as well as the resulting impact. For example, when observing the runner-up bid HH induces a belief shift of degree δi\delta_i, Provider ii can certainly shade or inflate its bid accordingly. However, Proposition 4.4 proves that the extra expected utility gained from such strategic adjustment is strictly bounded by VδiV\delta_i. Thus, even when providers act strategically on correlated bid information, the impact remains limited and the mechanism stays stable.
  • Negative payments: In practice, this works via a prepaid deposit or credit account (standard in ad exchanges and cloud marketplaces) rather than asking providers to wire money after a failed task. Each provider maintains a reserve balance on the platform: winning and passing evaluation adds V−HV - H to the balance, while failing deducts HH (bounded by the account's reserve limit). Because participation yields positive expected profit over time (Theorem 4.3), the account balance grows in the long run despite occasional single-query deductions.
  • Execution-contingent payments for actual LLM providers: From a provider's perspective, outcome-contingent payment is reasonable and attractive for two practical reasons. First, in an open routing market, a buyer cannot verify a provider's claimed accuracy before execution; tying payment to verified outcomes serves as a natural SLA guarantee (similar to outcome-based AI services like Intercom's Fin) that allows capable providers to credibly distinguish themselves from low-quality competitors who would otherwise inflate their claims. Second, because providers bid selectively based on their own capability estimates gig_i, they mainly win tasks they are confident in passing (Tˉi≥H\bar T_i \ge H), allowing them to capture higher value-based rewards (V−HV - H) on their strengths rather than only flat token fees, while maintaining positive expected profit overall (Theorem 4.3).

[1] Porter, Ryan, et al. "Fault tolerant mechanism design." Artificial Intelligence 172.15 (2008): 1783-1799.

[2] Pratt, John W. "Risk aversion in the small and in the large." Uncertainty in economics. Academic Press, 1978. 59-79.

[3] Singer, Yaron. "Budget feasible mechanisms." 2010 IEEE 51st Annual Symposium on foundations of computer science. IEEE, 2010.

[4] Akbarpour, Mohammad, and Shengwu Li. "Credible auctions: A trilemma." Econometrica 88.2 (2020): 425-467.

W2 & Q3 (Empirical Realism, Source of Gains, and End-to-End Latency)

1. Validating strategic bidding and payment incentives (W2)

Thank you for highlighting the importance of validating strategic bidding and payment incentives. We agree that these are essential properties of an auction-based routing mechanism. Our theoretical analysis explicitly considers providers' strategic bidding behavior. Theorem 4.2 proves that truthful effective-surplus reporting s^i=Vgi−ci\hat s_i=Vg_i-c_i is a weakly dominant strategy under the private-evaluation-belief condition, meaning that providers cannot improve their subjective expected utility through strategic over-bidding or under-bidding. Theorem 4.3 further guarantees non-negative expected utility under truthful reporting, ensuring that providers have an incentive to participate. To complement these theoretical guarantees with empirical evidence, we additionally conduct a strategic-misreporting experiment on RouterBench to examine whether providers can benefit from deviating from truthful bidding:

Bid Deviation Δ\DeltaBidding BehaviorMBPP Win RateMBPP UtilityMBPP WelfareGSM8K Win RateGSM8K UtilityGSM8K WelfareMMLU Win RateMMLU UtilityMMLU Welfare
−0.4-0.4Severe Under-bid0.45%0.003670.79250.00%0.000000.67450.00%0.000000.8087
−0.2-0.2Moderate Under-bid0.72%0.004440.79250.00%0.000000.67450.58%0.001270.8099
−0.1-0.1Mild Under-bid0.94%0.004740.79270.04%0.000050.67452.60%0.004090.8126
0.00.0Truthful (EA-RAM)9.09%0.005480.79229.09%0.001790.67559.09%0.007120.8165
+0.1+0.1Mild Over-bid41.13%-0.001370.789552.38%-0.018600.660726.87%-0.003240.8045
+0.2+0.2Moderate Over-bid44.65%-0.006390.786972.08%-0.047290.635248.87%-0.037120.7721
+0.4+0.4Severe Over-bid47.20%-0.013500.784698.03%-0.122620.566187.64%-0.146200.6671
Bid Deviation Δ\DeltaBidding BehaviorARC Win RateARC UtilityARC WelfareHellaSwag Win RateHellaSwag UtilityHellaSwag WelfareWinoGrande Win RateWinoGrande UtilityWinoGrande Welfare
−0.4-0.4Severe Under-bid0.00%0.000000.93630.00%0.000000.89280.00%0.000000.8459
−0.2-0.2Moderate Under-bid0.00%0.000000.93630.00%0.000000.89280.00%0.000000.8459
−0.1-0.1Mild Under-bid0.00%0.000000.93630.00%0.000000.89289.09%0.009900.8552
0.00.0Truthful (EA-RAM)9.09%0.002380.94039.09%0.001970.89469.09%0.009900.8552
+0.1+0.1Mild Over-bid45.45%-0.016800.913136.36%-0.016060.87569.09%0.009900.8552
+0.2+0.2Moderate Over-bid63.64%-0.038430.889054.55%-0.044410.847027.27%-0.016710.8299
+0.4+0.4Severe Over-bid100.00%-0.147520.778790.91%-0.150270.7345100.00%-0.249640.6054

As the results confirm:

  • Over-bidding (Δ>0\Delta > 0) causes negative provider utility and degrades market welfare: Although inflating bids boosts win rates (up to 47.20%–100.00% at Δ=+0.4\Delta = +0.4), the extra wins occur on queries where the runner-up score HH exceeds the provider's true effective surplus (Tˉi<H≤s^i\bar T_i < H \le \hat s_i). Under EA-RAM's payment rule ri=Vμ~i−Hr_i = V\tilde\mu_i - H, winning these queries turns the deviating provider's expected utility negative across all six benchmarks and reduces overall market social welfare.
  • Under-bidding (Δ<0\Delta < 0) forfeits profitable queries: Shading bids downward shrinks win rates toward zero and forfeits positive expected profit.
  • Truthful bidding (Δ=0.0\Delta = 0.0) is optimal: Across all six RouterBench benchmarks, truthful reporting maximizes provider utility while maintaining the highest market social welfare.
Reviewer_K9PTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EA-RAM, a reverse-auction mechanism for LLM routing that incorporates errors in provider predictions and buyer evaluations. It establishes incentive-compatibility, individual-rationality, and welfare guarantees under specified assumptions. Experiments on RouterBench and simulations demonstrate competitive cost-quality trade-offs and robustness to evaluation noise.

Critical strengths
  1. An interesting market-based formulation of LLM routing that shifts prediction responsibilities to providers.

  2. A theoretically grounded mechanism with explicit guarantees and promising empirical results across multiple benchmarks.

Critical weaknesses
  1. The incentive guarantees rely on restrictive assumptions about providers' evaluation beliefs, whose validity in realistic markets remains unclear.

  2. The experiments simulate provider-side predictions rather than actual strategic bidding. Consequently, the practical effectiveness of the proposed auction mechanism is not convincingly established.

Questions for authors
  1. How sensitive are the incentive guarantees to correlated provider beliefs in realistic routing settings?

  2. Can the authors provide experiments with strategic or misreporting providers to validate incentive compatibility empirically?

3 comments

We sincerely thank you for your positive assessment of our market-based routing formulation, theoretical guarantees, and empirical results. Below, we address your concerns in detail.

W1 & Q1 (Evaluation-Belief Assumptions and Sensitivity to Correlated Provider Beliefs)

Thank you for raising this important question. We agree that correlations between providers' beliefs and competitors' bids matter in realistic markets. Our exact incentive guarantee only requires that competitors' bids do not change a provider's own expected acceptance probability (Theorem 4.2). Even if they do, we show that the resulting incentive loss is bounded (Proposition 4.4).

  • Why this assumption is reasonable (Theorem 4.2): Each provider knows its own model and can use a local predictor to estimate its acceptance probability. For example, provider ii may predict an 80% chance of passing evaluation. A competitor's higher bid may simply reflect its stronger model or lower cost, without changing provider ii's own 80% estimate. In this case, the provider's own prediction gig_i remains its best estimate, while competitors' bids only affect the threshold HH it must beat. This is captured by Ei[μ~i∣H,Ii]=gi\mathbb E_i[\tilde\mu_i\mid H,\mathcal I_i]=g_i. Since winning yields expected utility Vgi−ci−HVg_i-c_i-H, bidding s^i=Vgi−ci\hat s_i=Vg_i-c_i ensures that the provider wins profitable tasks and avoids unprofitable ones. Bidding higher risks losses, while bidding lower misses profitable opportunities. Thus, truthful bidding is the optimal strategy.
  • What if competitors' bids contain additional information (Proposition 4.4)? In realistic markets, competitors' bids may reveal information about shared task difficulty, meaning that a provider's expected acceptance probability conditional on the runner-up score HH could differ from its original estimate gig_i. Proposition 4.4 quantifies this difference through δi=sup⁡H∣Ei[μ~i∣H,Ii]−gi∣\delta_i=\sup_H|\mathbb E_i[\tilde\mu_i\mid H,\mathcal I_i]-g_i|. We prove that the potential expected utility gain from deviating from truthful bidding is bounded by VδiV\delta_i. Therefore, smaller belief shifts imply smaller potential incentive losses, with exact DSIC recovered when δi=0\delta_i=0.

W2 & Q2 (Validating Strategic Bidding and Empirical Incentive Compatibility)

We thank the reviewer for highlighting the need to validate strategic bidding behavior. The key point is that knowing a prediction may be wrong does not mean knowing how to correct it. If a provider knows its prediction is systematically biased, it can correct the error and form a more accurate acceptance belief. This should be viewed as improving the belief itself, rather than strategically adjusting the bid for a particular query. However, if the provider does not know the direction of its prediction error, deliberately raising or lowering its bid may occasionally help, but cannot improve its expected utility. Theorem 4.2 proves that, given the provider's current belief gig_i, truthful bidding s^i=Vgi−ci\hat s_i=Vg_i-c_i maximizes its subjective expected utility, since over-bidding risks winning unprofitable tasks, while under-bidding risks losing profitable ones. To validate this property empirically, we conduct an additional strategic-misreporting experiment on RouterBench, examining how deliberate over-bidding and under-bidding affect provider utility and market welfare.

Specifically, we fix V=1.0V = 1.0 across all six RouterBench benchmarks (with N=11N=11 candidate LLMs, their actual API costs cic_i, and their trained capability predictors gig_i). In each run, we pick one target LLM ii to unilaterally misreport its probability by Δ∈{−0.4,−0.2,−0.1,0.0,+0.1,+0.2,+0.4}\Delta \in \{-0.4, -0.2, -0.1, 0.0, +0.1, +0.2, +0.4\} via s^i(Δ)=Vclip⁡(gi+Δ,0,1)−ci\hat s_i(\Delta) = V\operatorname{clip}(g_i + \Delta, 0, 1) - c_i, while the other 10 LLMs bid truthfully (s^j=Vgj−cj\hat s_j = Vg_j - c_j). We repeat this procedure for each of the 11 LLMs in turn and report the target provider's win rate, expected utility (Vgi−ci−H)1{s^i(Δ)≥H}(Vg_i - c_i - H)\mathbf{1}\{\hat s_i(\Delta) \ge H\}, and realized market social welfare Vμi†−ci†V\mu_{i^\dagger} - c_{i^\dagger} (using RouterBench ground-truth correctness μi†∈{0,1}\mu_{i^\dagger} \in \{0,1\}) averaged across all 11 target LLMs on all six benchmarks:

Bid Deviation Δ\DeltaBidding BehaviorMBPP Win RateMBPP UtilityMBPP WelfareGSM8K Win RateGSM8K UtilityGSM8K WelfareMMLU Win RateMMLU UtilityMMLU Welfare
−0.4-0.4Severe Under-bid0.45%0.003670.79250.00%0.000000.67450.00%0.000000.8087
−0.2-0.2Moderate Under-bid0.72%0.004440.79250.00%0.000000.67450.58%0.001270.8099
−0.1-0.1Mild Under-bid0.94%0.004740.79270.04%0.000050.67452.60%0.004090.8126
0.00.0Truthful (EA-RAM)9.09%0.005480.79229.09%0.001790.67559.09%0.007120.8165
+0.1+0.1Mild Over-bid41.13%-0.001370.789552.38%-0.018600.660726.87%-0.003240.8045
+0.2+0.2Moderate Over-bid44.65%-0.006390.786972.08%-0.047290.635248.87%-0.037120.7721
+0.4+0.4Severe Over-bid47.20%-0.013500.784698.03%-0.122620.566187.64%-0.146200.6671
Bid Deviation Δ\DeltaBidding BehaviorARC Win RateARC UtilityARC WelfareHellaSwag Win RateHellaSwag UtilityHellaSwag WelfareWinoGrande Win RateWinoGrande UtilityWinoGrande Welfare
−0.4-0.4Severe Under-bid0.00%0.000000.93630.00%0.000000.89280.00%0.000000.8459
−0.2-0.2Moderate Under-bid0.00%0.000000.93630.00%0.000000.89280.00%0.000000.8459
−0.1-0.1Mild Under-bid0.00%0.000000.93630.00%0.000000.89289.09%0.009900.8552
0.00.0Truthful (EA-RAM)9.09%0.002380.94039.09%0.001970.89469.09%0.009900.8552
+0.1+0.1Mild Over-bid45.45%-0.016800.913136.36%-0.016060.87569.09%0.009900.8552
+0.2+0.2Moderate Over-bid63.64%-0.038430.889054.55%-0.044410.847027.27%-0.016710.8299
+0.4+0.4Severe Over-bid100.00%-0.147520.778790.91%-0.150270.7345100.00%-0.249640.6054

As the results confirm:

  • Over-bidding (Δ>0\Delta > 0) causes negative provider utility and degrades market welfare: Although inflating bids boosts win rates (up to 47.20%–100.00% at Δ=+0.4\Delta = +0.4), the extra wins occur on queries where the runner-up score HH exceeds the provider's true effective surplus (Tˉi<H≤s^i\bar T_i < H \le \hat s_i). Under EA-RAM's payment rule ri=Vμ~i−Hr_i = V\tilde\mu_i - H, winning these queries turns the deviating provider's expected utility negative across all six benchmarks and reduces overall market social welfare.
  • Under-bidding (Δ<0\Delta < 0) forfeits profitable queries: Shading bids downward shrinks win rates toward zero and forfeits positive expected profit.
  • Truthful bidding (Δ=0.0\Delta = 0.0) is optimal: Across all six RouterBench benchmarks, truthful reporting maximizes provider utility while maintaining the highest market social welfare.

We hope our response has addressed your concerns, and we will gladly and promptly respond to any further questions you may have.

Thank you again for your thoughtful review. We have posted our detailed response along with new experimental results and manuscript updates. When you have a moment, we would greatly appreciate hearing whether these additions address your concerns, and we remain happy to answer any follow-up questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.