acceptodds
Under review as a conference paper at ICLR 2027

Better Models Can Reduce Accuracy Under Strategic Pricing

Abstract

Recent progress in language models is assessed against fixed computation budgets, but provider prices may change when they compete for users. We show that a model can become more accurate at every computation level and can generate higher equilibrium revenue, yet users on a fixed budget may incur lower accuracy after providers change prices. We model the market of combining verifiable outputs from models priced independently by providers, and show that every pure-strategy equilibrium is characterized by the maximizers of , where is the negative log failure probability. This characterization allows us to identify an explicit family of improvements that reduces equilibrium accuracy on an open parameter region, even when each task is eventually solvable, and holds under sufficiently small strictly positive serving costs. We perform experiments with 610,560 Qwen and Llama generations on LiveCodeBench and MATH-500, comparing fixed vs. equilibrium prices. Non-thinking replacements have gains of 3.8 and 2.2 percentage points from checkpoints at fixed prices, but become losses of 1.7 and 1.6 points after repricing. Thinking replacements have gains that remain positive, but are reduced by 1.4 and 0.5 points respectively. Moreover, we bound profitable price deviations discuss the implications of these results for the shape of test-time scaling curves.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
language models, test-time scaling, strategic pricing, algorithmic game theory, model serving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BSuuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work studies a pricing game for verifiable, test-time compute where providers set per-call prices, and a budget-constrained buyer optimises accuracy by allocating calls across models.

Other than that, the central theoretical result characterises all pure strategy equilibria via the maximisers.]

At last, they prove that pointwise improvements to a models accuracy-compute curve can reduce users' equilibrium accuracy after providers optimally reprice.

Critical strengths

The equilibrium characterisation in terms of log compute marginal capability is general and actionable; it directly ties scaling curve shape to price-mediated outcomes.

Furthermore, the price-mediated reversal result is surprising and important, where a pointwise better scaling curve can yield lower user accuracy once providers reprice.

At last, robustness to small serving costs via an implicit function theorem-style argument and the constructive near-maximum revenue bounds are the technically solid additions.

Critical weaknesses

There seems to be some issue with the work; the assumptions of linear per-call pricing, a single representative buyer with a tight budget, and perfect validation simplify the marker but limit immediate applicability to real-world API pricing.

The pointwise reversal construction relies on a specific latent rate family; while instructive, it remains to be seen how often real scaling curve improvements fall into the tail constant regime that induces reversal.

Equilibrium existence requires finite peaks of H(x). Empirically, identifying tail behaviour from finite samples is hard; while noted, practical guidance for diagnosing non-attained super-exponentiality in the wild is limited.

Other than the above, empirical markets consider two providers; the theory allows m >= 2, but it would be valuable to test triadic or larger markets to probe whether there are stronger or weaknesses with more competition.

At last, the empirical best-response analysis operates on a discrete allocation grid; while the work provides gap bounds, flat H-peaks indicate potential fragility in quantity predictions, and results may depend on grid resolution.

Questions for authors

The things mentioned in the weaknesses raise some questions, such as:

  1. How sensitive are the empirical equilibria to stronger cross-model dependence structures beyond task-aligned multiplicative failures? Could such dependence materially change equilibrium prices or the presence and absence of reversal?

  2. In practical API markets, nonlinear pricing is common. Which parts of the H-peak characterisation extend to two-part tariffs or block pricing, and do you anticipate new equilibria that would change the reversal result?

  3. Does the monotone comparative condition guarantee no reversal under unique equilibria? Do you have empirical evidence on real scaling curves indicating whether g/f tends to be non-decreasing or not?

  4. Your sensitivity analysis shows call limit effects on the sign magnitude of reversal? Can you provide more guidance on how discretisation granularity interacts with flat H-peaks to produce fragile quantity predictions, and how practitioners should set grids to avoid artefacts?

I would or could raise the rating if the justification seems convincing enough.

3 comments
AuthorsOpenReview ↗

Response to Reviewer BSuu (1/3): Cross-Model Dependence and Nonlinear Pricing Contracts

Thank you for recognizing our log-compute marginal capability characterization as general and actionable, for highlighting the price-mediated accuracy reversal, and for valuing our serving-cost analysis and constructive revenue bounds. Guided by your feedback, we sharpened Section 4 to specify both the global peak-separation condition required alongside local curvature for peak-location stability and the finite-sample limits on identifying distant tail peaks. This makes clear why we use smooth curve fits only to initialize price search, and then verify candidate approximate equilibria on the integer call grid via exact continuous-price best-response sweeps under our 1% budget-normalized deviation threshold. We also stated the exact hypotheses of our monotone no-reversal condition in Section 3.4.

[Q1] Cross-model error dependence

Our empirical baseline directly accounts for cross-model error correlation driven by shared task difficulty across benchmarks (Kim et al., ICML 2025). All four primary Qwen comparisons (Table 1) evaluate joint failure probabilities task-by-task rather than assuming population-level independence, which avoids the 3.6 and 1.9 percentage-point joint-success errors produced by multiplying population marginal failure rates on coding and math (Table 5).

To stress-test stronger within-task error correlation on frontier outputs, we analyzed 279 archived LiveCodeBench tasks (Jain et al., ICLR 2025) with 10 graded generations per task and service, pairing June and October 2024 Claude 3.5 Sonnet checkpoints against GPT-4o. Holding each task's measured failure curves W1,W2W_1, W_2 fixed, we interpolated conditional within-task joint failure along the linear mixture path from conditional independence W1W2W_1 W_2 (ρ=0\rho = 0) to maximal positive error overlap min⁡(W1,W2)\min(W_1, W_2) (ρ=1\rho = 1), the Fréchet-Hoeffding upper bound at those task margins. Across all 121 call allocations on the 138-task split, shifting from ρ=0\rho = 0 to ρ=1\rho = 1 changed portfolio success by at most 0.080 percentage points for June Claude and 0.646 percentage points for October Claude, bounding the change in optimized buyer success at any fixed prices. Solving the two task splits (141 and 138 tasks) separately, each checkpoint and split admits a common price pair that remains a valid equilibrium across every ρ∈[0,1]\rho \in [0,1] along this path, with the equilibrium upgrade gain at those prices on the 138-task split shifting only from 4.79 to 4.35 percentage points between ρ=0\rho = 0 and ρ=1\rho = 1. Dependence does alter equilibrium structure when it removes cross-provider differentiation, for instance if two zero-cost providers share the same latent per-task success rate and generate candidates conditionally independently given that rate, collapsing joint failure to F(x1+x2)F(x_1 + x_2) so rival calls become perfect substitutes and Bertrand undercutting eliminates positive-price pure-strategy equilibria.

[W1/Q2] Nonlinear billing, block pricing, and two-part tariffs

Our HH-peak characterization extends directly to nonlinear metered schedules where provider jj chooses a scalar rate aj>0a_j > 0 on a fixed, strictly increasing billing schedule ajtj(xj)a_j t_j(x_j) with tj(0)=0t_j(0) = 0: re-indexing compute in billed units yj=tj(xj)y_j = t_j(x_j) recovers Theorem 1 in billed units whenever the transformed capability curves satisfy all of its regularity, concavity, and finite-peak hypotheses. Furthermore, for fixed power-law volume schedules tj(x)=xβjt_j(x) = x^{\beta_j} (βj≥1\beta_j \ge 1), log-compute marginal capability rescales to Hj(x)/βjH_j(x)/\beta_j, which preserves the peak location zjz_j, leaves physical equilibrium call quantities and accuracy reversals unchanged, and sets budget shares proportional to Mj/βjM_j/\beta_j (where Mj=max⁡xHj(x)M_j = \max_x H_j(x)). On an integer call grid, if C−jC_{-j} denotes spending on rivals, a fixed block schedule ajtj(kj)a_j t_j(k_j) preserves our threshold best-response structure with affordability cutoffs at aj=(B−C−j)/tj(kj)a_j = (B - C_{-j})/t_j(k_j) for kj>0k_j > 0 and tj(kj)>0t_j(k_j) > 0, whereas allowing providers to choose multi-tier block prices simultaneously expands the unilateral deviation space.

By contrast, unrestricted two-part tariffs (where each provider jointly chooses an upfront access fee Aj≥0A_j \ge 0 and a per-call price pj>0p_j > 0) fundamentally alter competition by allowing surplus extraction through the entry fee. In a zero-cost market where at least two providers have unbounded capability, no finite pure-strategy equilibrium with positive per-call prices exists: at least one unbounded provider must earn less than the total budget BB, and that provider can profitably deviate by setting its access fee slightly above its current revenue (still below BB) while slashing its per-call price enough that spending the remaining budget solely on its calls strictly dominates every rival-only bundle.

AuthorsOpenReview ↗

Response to Reviewer BSuu (2/3): Monotone Comparative Statics and Scaling Ratios in Published Post-Training

[Q3.1] When a monotone marginal ratio rules out reversal

Yes: under unique finite peaks, our monotone comparative condition strictly guarantees no accuracy reversal. In the independent zero-cost market of Proposition 5, let f=−log⁡Ff = -\log F and g=−log⁡Gg = -\log G denote negative log failure before and after the upgrade, with Hf(x)=xf′(x)H_f(x) = x f'(x). Crucially, the sufficient no-reversal condition governs the marginal ratio g′(x)/f′(x)g'(x)/f'(x) rather than the level ratio g(x)/f(x)g(x)/f(x). Under pointwise improvement g≥fg \ge f, unique finite peaks zf,zgz_f, z_g, and unchanged rival peaks, the identity Hg(x)=[g′(x)/f′(x)]Hf(x)H_g(x) = [g'(x)/f'(x)] H_f(x) implies Hg(x)<Hg(zf)H_g(x) < H_g(z_f) for all x<zfx < z_f whenever g′/f′g'/f' is nondecreasing. Peak uniqueness rules out downward quantity jumps across tied maxima (which would otherwise require a tie-breaking rule), forcing zg≥zfz_g \ge z_f and guaranteeing g(zg)≥g(zf)≥f(zf)g(z_g) \ge g(z_f) \ge f(z_f).

[W2/Q3.2] Empirical scaling curves after post-training

Empirically, real model upgrades do not universally exhibit nondecreasing ratios. In our primary Qwen markets, where buyer purchases and unilateral price-deviation checks are evaluated directly on measured task-aligned empirical success (using parametric fits only for initialization), we already observe two full equilibrium reversals and two attenuations (Table 1). To inspect how relative scaling ratios behave in other published post-training pipelines, we also computed measured level and secant ratios from two recent studies. Writing u(k)u(k) for measured pass@k accuracy and f(k)=−log⁡(1−u(k))f(k) = -\log(1 - u(k)) with f(0)=g(0)=0f(0) = g(0) = 0, the secant ratio [g(b)−g(a)]/[f(b)−f(a)][g(b) - g(a)]/[f(b) - f(a)] across consecutive reported call budgets equals the f′f'-weighted interval mean of g′/f′g'/f' for any smooth increasing interpolant:

  1. Diminishing ratios under RL post-training (MATH-500, Yue et al., NeurIPS 2025, Table 3): All six RL variants outperform the Qwen2.5-7B base model at both k=1k = 1 and k=256k = 256 while exhibiting sharply falling level ratios across that range. For GRPO, the level ratio g(k)/f(k)g(k)/f(k) drops from 3.22 at 1 call to 1.09 at 256 calls, and consecutive interval secants drop from 3.22 to 0.78, which is mathematically incompatible with a globally nondecreasing g′/f′g'/f' for any smooth interpolant through the reported pass@k points.
  2. Rising ratios under hard-test synthesis (LiveCodeBench-105, He et al., ICLR 2026, HardTestGen Table 3): Comparing Qwen3-4B-HT against its Qwen3-4B base at k=1k = 1 and k=10k = 10 shows the level ratio g(k)/f(k)g(k)/f(k) rising from 1.03 to 1.26 and consecutive interval secants rising from 1.03 to 1.60, a pattern compatible with an increasing marginal ratio across those two measured intervals.

Together with our Qwen experiments, these published benchmarks demonstrate that modern post-training produces both falling and rising empirical secant ratios across practical call budgets. Superiority at both ends of a reported pass@k range (such as k=1k = 1 and large kk) leaves wide variation in how capability scales across intermediate call budgets, underscoring why checking the full shape of the scaling gain is essential when assessing market outcomes.

AuthorsOpenReview ↗

Response to Reviewer BSuu (3/3): Practical Tail Diagnostics, Grid Granularity, Market Entry, and Relaxed Assumptions

[W3] Tail behavior and peaks at the call limit

Importantly, non-attainment of a finite peak of H(x)=xf′(x)H(x) = x f'(x) does not require exotic super-exponential tails: even standard power-law failure F(x)=(1+x/τ)−κF(x) = (1 + x/\tau)^{-\kappa} produces H(x)=κx/(τ+x)H(x) = \kappa x / (\tau + x), which increases strictly toward κ\kappa without attaining an interior peak. This gives practitioners a concrete diagnostic workflow on finite samples: (1) inspect log-doubling increments [f(2k)−f(k)]/log⁡2[f(2k) - f(k)]/\log 2 (which average HH over consecutive log-compute intervals) to check whether empirical HH turns down inside the observed horizon or rises into the cap, (2) flag boundary maximizers and widen the call horizon where feasible, and (3) report the near-optimal call span alongside Proposition 4's constructive unilateral revenue-gain bound M/H(z)M / H(z) (where MM upper-bounds HH over the admissible domain), which explicitly bounds the maximum profitable revenue ratio from any unilateral price deviation even when the peak is approached only asymptotically.

[W5/Q4] Flat peaks, grid spacing, and call caps

Flat HH-peaks and grid granularity play distinct roles in continuous approximations versus our discrete integer-call API game. In continuous approximations, under uniform HH-estimation error δ\delta, solver tolerance ξ\xi, local curvature H′′≤−λ<0H'' \le -\lambda < 0, and global peak separation (2δ+ξ2\delta + \xi below the outside gap), Lemma 6 bounds peak-location drift by 2(2δ+ξ)/λ\sqrt{2(2\delta + \xi)/\lambda}. Evaluating on a continuous mesh of step hh near a smooth interior maximum (∣H′′∣≤L|H''| \le L) adds at most Lh2/8L h^2 / 8 to peak-value loss, so practitioners should choose hh so that Lh2/8≪δL h^2 / 8 \ll \delta. By contrast, our simulated integer-call game keeps unit one-call spacing throughout: it scans every integer call pair (k1,k2)(k_1, k_2) up to the cap and checks continuous unilateral price deviations at every affordability threshold. Varying the call cap across 32, 64, and 128 expands the set of supported call quantities rather than changing mesh spacing, reducing boundary selections from 12.5% at 64 to 3.1% at 128 (Appendix G.3).

[W4] Competitor entry and three-provider markets

In the independent zero-cost benchmark of Theorem 1 (with unique or consistently selected unchanged rival peaks), adding an independent entrant with delivered capability Ventrant>0V_{\text{entrant}} > 0 multiplies the equilibrium success difference Unew−UoldU_{\text{new}} - U_{\text{old}} by exp⁡(−Ventrant)>0\exp(-V_{\text{entrant}}) > 0, attenuating magnitude while strictly preserving the sign of any reversal. Evaluating a three-provider market under our Figure 1 parameters (B=1B = 1, κ=0.2\kappa = 0.2, τ=1\tau = 1, focal upgrade from α=0.24\alpha = 0.24 to 0.36) confirms clear analytic equilibrium reversals both when the two rivals match the initial model (88.36% down to 86.67%) and when both rivals already match the upgraded model (84.72% down to 82.49%). Complementing this analytic triad, on the 138 held-out tasks of our 279-task matched LiveCodeBench archive across Claude 3.5 Sonnet, GPT-4o, and DeepSeek-V3, a fixed one-call-per-provider portfolio achieves 44.44% and 48.73% pass rates before and after Claude's upgrade, beating three calls to the hindsight-best single provider (39.63% and 43.71%).

[W1] Imperfect verification, serving costs, and flexible budgets

Finally, we separately reoptimized equilibrium prices under imperfect verification, positive serving costs, and buyer value for unspent budget (Table 2 and Appendix G.3): (1) under imperfect verification on LiveCodeBench (filtering candidates on public tests and scoring on hidden tests, with 75% of price profiles meeting the 1% deviation criterion), the net coding effect shifts by −0.6 percentage points relative to the primary perfect-verifier baseline, (2) positive serving costs at γ=0.001\gamma = 0.001 yield mean net equilibrium losses of −0.4 and −0.6 percentage points on coding and math (and −0.2 and −0.5 percentage points when costs scale with measured generation time), and (3) valuing unspent funds via vU−p⋅kv U - p \cdot k (spend≤B\text{spend} \le B) produces net changes of +0.6, +0.3, −0.1, −0.6, and −0.9 percentage points across v/B∈0.5,1,2,4,8v/B \in \\{0.5, 1, 2, 4, 8\\}, turning negative and deepening steadily across the higher task valuations.

Reviewer_ZNLXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies how the accuracy received by a user from an LLM changes when the LLM improves and the providers change their prices strategically. It focuses on a market where the providers choose their prices for computation and a user has a budget and allocates inference calls in a way to maximize the probability of receiving a correct answer. The main contribution is the analysis of the pure-strategy pricing equilibria in terms of the negative log failure probability of the providers. Using this, the authors construct a family of improvements that increase accuracy at each computation level but decrease the buyer's equilibrium accuracy. The authors empirically test this mechanism using multiple generations from Qwen and Llama models on two benchmarks. They show that repricing reduces the accuracy gains.

Critical strengths
  1. There is some theoretical contribution. In theorem 1, the authors show a relationship between the accuracy-compute curves and the equilibrium prices, revenues, and computation. In theorem 2, they show the counterintuitive result clearly. That an improvement in model accuract can decrease equilibrium user accuracy while increasing the provider's revenue.

  2. The experiments seem meaningful and use many generations across two benchmarks.

Critical weaknesses
  1. The paper is quite hard to follow.

  2. The central observation that improvements in model capabilities can reduce equilibrium accuracy under competition seems very related to Jagadeesan et al. (2023). The present paper has a different mechanism based on strategic prices and gives a different equilibrium characterization, but the insight is already established by the prior work. The novely of the present paper is on the specific model and theoretical characterization rather than on the counterintuitive result.

  3. The theoretic analysis is based on a buyer who always uses a fixed budget, linear prcing, independent provider failures, and the main charecterization assumes zero serving cost. The authors examine multiple extensions but it is unclear how the results generalize to realistic inference markets. Lastly. the empirical results show small reversals and they seem very sensitive to choices (call limits, value of unspent funds)

Questions for authors
  1. Given the similarities with Jagadeesan et al. (2023), what do you think is the main conceptual contribution beyond studying a different strategic mechanism?

  2. The empirical results sometimes change sign under different market configurations. How robust are the accuracy reversals? Espesially when buyers can keep unspent funds or call limits change.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZNLX (1/2): Conceptual Novelty Beyond Predictor Selection and Scaling-Curve Diagnostics

Thank you for recognizing how Theorems 1 and 2 connect test-time scaling curves to equilibrium prices, revenues, and user accuracy, and for valuing our multi-generation experiments across two benchmarks.

[W1] Revisions for manuscript clarity

We took your readability feedback seriously and substantially rewrote the abstract and all seven main-text sections to make the intuition, assumptions, and contributions accessible to a broad machine learning audience. For example, Section 3.1 now builds immediate intuition before proving Theorem 1 by showing from the buyer's first-order condition (fj′(xj)/pj=μf'_j(x_j)/p_j = \mu) that a provider's revenue pjxjp_j x_j equals Hj(xj)/μH_j(x_j)/\mu, making it transparent why competing for revenue aligns with maximizing Hj(x)=xfj′(x)H_j(x) = x f'_j(x), while Section 6 sharpens our contrast with prior work on strategic model competition.

[W2/Q1] Strategic pricing of repeated calls beyond predictor selection

While Jagadeesan et al. (NeurIPS 2023) study how competition shapes which classification rules providers choose to offer when users select a predictor, we study how providers price repeated test-time calls to deployed models. In our setting, even when an upgraded model strictly improves population accuracy at every positive compute level and makes no individual task harder, endogenous per-call repricing can reduce equilibrium user accuracy. Consequently, standard fixed-compute pass@k benchmarks are incomplete for inference markets: pointwise dominance across every fixed kk cannot predict even the sign of the user's accuracy change once providers reprice repeated inference calls.

Our framework also supplies the quantitative scaling-curve law that determines when an upgrade survives market repricing. When a budget-constrained buyer samples across independent providers and succeeds if any candidate is correct, failure probabilities multiply and negative log failure fj(x)=−log⁡Fj(x)f_j(x) = -\log F_j(x) adds across providers. Equilibrium outcomes then hinge on a single measurable curve, Hj(x)=xfj′(x)H_j(x) = x f'_j(x), capturing the capability gained per proportional increase in compute. In our separable zero-cost benchmark with unique finite peaks of HjH_j (Theorem 1), each provider's own curve pins down its equilibrium compute quantity, relative peak heights determine revenue shares, those curves together with the budget set unit prices, and aggregate delivered capability determines buyer success. When a focal provider upgrades (g≥fg \ge f) while rivals stay fixed, rival equilibrium quantities remain unchanged and the scaling geometry of gg versus ff governs user accuracy: (1) a pure compute speedup g(x)=f(sx)g(x) = f(sx) (s>1s > 1) scales the focal unit price by ss and leaves equilibrium accuracy unchanged, (2) a nondecreasing marginal ratio g′/f′g'/f' guarantees no accuracy loss, and (3) front-loaded upgrades (κ<αold<αnew<2κ\kappa < \alpha_{\text{old}} < \alpha_{\text{new}} < 2\kappa in Theorem 2) sharpen diminishing returns enough that the higher equilibrium unit price rations purchased calls and lowers overall buyer accuracy.

AuthorsOpenReview ↗

Response to Reviewer ZNLX (2/2): Structured Regimes Across Call Caps, Unspent Funds, Costs, and Correlated Errors

[W3.1/Q2] Robustness across checkpoints, call limits, and unspent budgets

Across all four checkpoint-by-benchmark markets in our controlled price simulations (Section 5.2, Table 1), all old and new price profiles satisfy the 1% normalized unilateral deviation criterion, and strategic repricing erodes the fixed-price gain in every single case. Specifically, non-thinking upgrades turn fixed-quote gains of +3.8 and +2.2 percentage points on coding and math into net equilibrium losses of −1.7 and −1.6 percentage points (total negative pricing swings of 5.5 and 3.8 percentage points), while thinking upgrades see fixed-quote gains narrow from +2.2 and +0.8 percentage points to +0.8 and +0.3 percentage points.

Varying the call cap and the value of unspent funds reveals consistent directional patterns across the tested market settings rather than erratic flips. In Section 5.3 (Table 2) and Appendix G.3, as we raise the integer call cap across 32, 64, and 128 in our task-aligned market (scanning every integer call pair up to the cap and checking unilateral price deviations at every affordability threshold), boundary selections fall from 12.5% at 64 to 3.1% at 128. Across those same cap increases, the domain-mean net accuracy change (averaging thinking and non-thinking upgrades) becomes consistently more negative in both domains, shifting from +0.2 to −0.5 and −0.8 percentage points on coding, and from −0.5 to −0.7 and −0.8 percentage points on math.

Similarly, giving unspent money direct value under the budget-constrained objective vU−p⋅kv U - p \cdot k (Appendix G.3, Table 8) lets buyers retain funds whenever extra calls buy too little additional probability of success, tracing a clear progression as v/Bv/B increases. Across the five tested task valuations v/B∈0.5,1,2,4,8v/B \in \\{0.5, 1, 2, 4, 8\\}, the observed aggregate net accuracy changes are +0.6, +0.3, −0.1, −0.6, and −0.9 percentage points: modest net gains appear only at the two lowest tested valuations, whereas across the three higher valuations, where the buyer objective places greater weight on solving the task, the net accuracy effect turns negative and deepens steadily.

[W3.2] Correlated failures and positive serving costs

Finally, we directly address cross-provider error correlation and positive serving costs beyond the independent zero-cost baseline of Theorem 1. All four primary empirical markets evaluate task-aligned joint success (multiplying conditional failure rates within each task before averaging across tasks) to capture correlated cross-model difficulty on shared benchmarks (Kim et al., ICML 2025), avoiding the 3.6 and 1.9 percentage-point joint-success errors produced by multiplying population marginal failure rates. For serving costs, Theorem 3 (Section 3.5) proves that strict equilibrium reversals persist locally at small positive serving costs under strict smooth peaks separated from endpoint limits, and our simulations with matched serving costs at γ=0.001\gamma = 0.001 yield mean net equilibrium losses of −0.4 percentage points on coding and −0.6 percentage points on math.

Thank you for your response. After considering your rebuttal, I have decided to maintain my original score.

Reviewer_QR7TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how improvements in a language model’s accuracy–compute curve interact with strategic pricing by inference providers. In the proposed framework, a budget-constrained buyer purchases repeated samples from independently priced providers and succeeds if any sample passes a perfect verifier. The authors evaluate this in simulated markets built from 610,560 generations, in which Qwen3-4B checkpoints are replaced while a Llama-3.1-8B competitor is held fixed. Across thinking and non-thinking upgrades on LiveCodeBench and MATH-500, repricing reduces the fixed-price gain in all four comparisons and turns both non-thinking gains into losses.

Critical strengths
  • The specific question is novel and timely. Test-time scaling and routing methods are typically evaluated at fixed compute or fixed prices, and the paper argues that endogenous pricing can flip the sign of an apparent improvement.
Critical weaknesses
  • The experimental setup might be considerably more simplified than the framing suggests. The presentation may lead readers to take the experiments as evidence about real inference markets, but several design choices make the simulated market largely hypothetical. All model services (Qwen3-4B/Llama-3.1-8B-Instruct) are open-weight models that any host can serve, but the experiments treat each checkpoint as sold by a single provider with its own pricing power. The experiments might be better framed as a simulation of a hypothetical market, or complemented by a discussion of closed API models, where single-provider pricing is realistic.

  • Real APIs charge per token, whereas the paper prices per candidate. Thinking outputs are typically substantially longer than non-thinking ones, and the old and new checkpoints may also differ in typical output length. Because a long candidate costs the same as a short one in the main zero-cost comparison, length differences never affect prices or purchases. It is unclear whether the analysis under per-token pricing might still support the reversal findings.

  • Each market includes only one competitor, Llama-3.1-8B-Instruct, which is likely much weaker than Qwen3-4B on math and coding. With a weak competitor, the buyer’s success is driven almost entirely by the Qwen provider, so the results may say little about markets with comparably capable rivals. The upgrades also come from a single model family and only two checkpoint pairs at the 4B scale. Additional markets, for example with a stronger competitor, a different model family, or more than two providers, would help establish whether the effect generalizes.

Questions for authors

Please refer to the Weaknesses part.

2 comments
AuthorsOpenReview ↗

Response to Reviewer QR7T (1/2): Market Framing, Open-Weight Hosting, and Per-Token Billing

Thank you for recognizing the novelty and timeliness of showing how endogenous provider pricing can reverse apparent test-time scaling gains, and for your constructive questions on proprietary API framing, open-weight hosting, and per-token billing.

[W1] Market simulation with open-weight checkpoints

Prompted by your comments, we clarified our experimental market setup in Section 5.1. Section 2 defines continuous computation units and linear unit prices in the theoretical model, and Section 5.1 now states directly that each empirical unit is one candidate generation under fixed decoding settings, using real outputs sampled offline from fixed open-weight checkpoints while solving each competing service's unit price within our market model.

Our experiments pair 610,560 measured model generations with equilibrium prices solved directly from strategic interaction in our pricing game. This design models proprietary single-vendor APIs, where each provider posts a unit price for its own differentiated endpoint and buyers allocate verifiable inference calls across distinct services (Olmedo et al., ICML 2026). Generating the underlying candidates from open-weight checkpoints serves purely as an experimental control to keep weights and decoding strictly fixed across repeated samples, avoiding silent updates on opaque commercial endpoints. By contrast, commodity open-weight hosting, where multiple vendors serve identical weights with interchangeable outputs at uniform cost, turns rival endpoints into perfect substitutes and triggers Bertrand undercutting toward marginal cost.

[W2] Per-token billing, output length, and generation cost

Under an expected spending budget and positive, allocation-invariant mean billable lengths LjL_j, strategic per-token billing yields the same equilibrium accuracy comparison as per-call pricing, even when thinking or upgraded checkpoints generate much longer outputs. Whenever each service jj freely sets its per-token rate tj>0t_j > 0, the induced expected per-candidate price pj=tjLjp_j = t_j L_j is a bijection on positive prices. Providers absorb systematic length shifts into their chosen per-token rates, leaving feasible call allocations, expected provider revenues, and the net equilibrium accuracy change across an upgrade identical to the per-call game.

Output length also affects provider serving costs, and our empirical reversal persists when costs scale with measured generation duration. Setting each checkpoint's serving cost proportional to its measured wall-clock generation time per candidate (including failed and truncated generations) at γ=0.001\gamma=0.001 yields mean net equilibrium accuracy changes of −0.2 percentage points on coding and −0.5 percentage points on math, closely matching the −0.4 and −0.6 percentage-point reversals observed under matched costs.

AuthorsOpenReview ↗

Response to Reviewer QR7T (2/2): Multi-Provider Markets, Stronger Rivals, and Frontier Model Families

[W3.1] Analytic markets with equally strong or stronger rivals

The equilibrium reversal is not an artifact of a weak competitor: it persists analytically when rivals are just as capable as the original service, and even when rivals already match the upgraded service. In Theorem 1, with negative log failure fj=−log⁡Fjf_j = -\log F_j and Hj(x)=xfj′(x)H_j(x) = x f'_j(x), unique interior peaks keep every unchanged rival's equilibrium call quantity fixed when one provider upgrades. Adding an independent rival with unchanged delivered capability V>0V > 0 multiplies the before-and-after equilibrium success difference by exp⁡(−V)\exp(-V), scaling the magnitude while strictly preserving the sign of any gain or reversal. Solving the analytical three-provider market under the parameters of Figure 1 (B=1B = 1, κ=0.2\kappa = 0.2, τ=1\tau = 1) as one seller upgrades from α=0.24\alpha = 0.24 to 0.36 confirms substantial reversals against both equal and stronger rivals:

Analytic three-provider marketUnchanged rivals (2 competitors)Equilibrium success (old to new)Net change
Equal initial capabilityBoth at α=0.24\alpha = 0.2488.36% to 86.67%-1.70 pp
Stronger initial capabilityBoth at α=0.36\alpha = 0.36 (matching upgraded seller)84.72% to 82.49%-2.23 pp

[W3.2] Empirical three-provider complementarity across frontier families

To verify multi-provider complementarity across distinct frontier model families of comparable strength, we analyzed the official LiveCodeBench archive (Jain et al., ICLR 2025), pairing the June and October 2024 releases of Claude 3.5 Sonnet with GPT-4o and DeepSeek-V3 across 279 matched tasks and 10 graded completions per checkpoint and task (11,160 completions total). On the 138 held-out tasks, all three services are closely matched in capability, with pass@1 rates of 33.33% (June Claude), 39.64% (October Claude), 30.36% (GPT-4o), and 35.87% (DeepSeek-V3). Under task-aligned evaluation, a diversified three-provider portfolio of one call per provider achieves 44.44% before and 48.73% after Claude's upgrade, substantially outperforming three calls to the single best provider even when chosen with hindsight on the test set (39.63% and 43.71%).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We warmly thank Reviewers QR7T, ZNLX, and BSuu for their constructive engagement with a research question that QR7T describes as "novel and timely." ZNLX notes that our experiments "use many generations across two benchmarks" and that in Theorem 2 we "show the counterintuitive result clearly." BSuu likewise finds our equilibrium characterization "general and actionable" and highlights its implication that "a pointwise better scaling curve can yield lower user accuracy once providers reprice."

This reversal arises when providers adjust their prices after an upgrade. In our independent zero-cost benchmark, the peak location of each provider's log-compute marginal capability determines the calls purchased from that provider, while relative peak heights determine providers' revenue shares. An upgrade whose gains arrive early can raise the provider's price and cut the number of attempts a fixed budget buys, enough to outweigh higher per-call quality and lower the probability of obtaining a correct answer. In simulations using measured generations on LiveCodeBench and MATH-500, equilibrium repricing erodes fixed-quote gains in all four comparisons and reverses the non-thinking upgrade in both domains.

In response to the clarity feedback from Reviewer ZNLX and the other reviewers, we have rewritten the abstract and main text with more readability and clarity. Furthermore, Section 5.1 now clearly explains which inputs are measured and which prices are solved for, and Section 7 develops per-token billing and independent competitor entry.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.