- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work studies a pricing game for verifiable, test-time compute where providers set per-call prices, and a budget-constrained buyer optimises accuracy by allocating calls across models.
Other than that, the central theoretical result characterises all pure strategy equilibria via the maximisers.]
At last, they prove that pointwise improvements to a models accuracy-compute curve can reduce users' equilibrium accuracy after providers optimally reprice.
Critical strengths
The equilibrium characterisation in terms of log compute marginal capability is general and actionable; it directly ties scaling curve shape to price-mediated outcomes.
Furthermore, the price-mediated reversal result is surprising and important, where a pointwise better scaling curve can yield lower user accuracy once providers reprice.
At last, robustness to small serving costs via an implicit function theorem-style argument and the constructive near-maximum revenue bounds are the technically solid additions.
Critical weaknesses
There seems to be some issue with the work; the assumptions of linear per-call pricing, a single representative buyer with a tight budget, and perfect validation simplify the marker but limit immediate applicability to real-world API pricing.
The pointwise reversal construction relies on a specific latent rate family; while instructive, it remains to be seen how often real scaling curve improvements fall into the tail constant regime that induces reversal.
Equilibrium existence requires finite peaks of H(x). Empirically, identifying tail behaviour from finite samples is hard; while noted, practical guidance for diagnosing non-attained super-exponentiality in the wild is limited.
Other than the above, empirical markets consider two providers; the theory allows m >= 2, but it would be valuable to test triadic or larger markets to probe whether there are stronger or weaknesses with more competition.
At last, the empirical best-response analysis operates on a discrete allocation grid; while the work provides gap bounds, flat H-peaks indicate potential fragility in quantity predictions, and results may depend on grid resolution.
Questions for authors
The things mentioned in the weaknesses raise some questions, such as:
-
How sensitive are the empirical equilibria to stronger cross-model dependence structures beyond task-aligned multiplicative failures? Could such dependence materially change equilibrium prices or the presence and absence of reversal?
-
In practical API markets, nonlinear pricing is common. Which parts of the H-peak characterisation extend to two-part tariffs or block pricing, and do you anticipate new equilibria that would change the reversal result?
-
Does the monotone comparative condition guarantee no reversal under unique equilibria? Do you have empirical evidence on real scaling curves indicating whether g/f tends to be non-decreasing or not?
-
Your sensitivity analysis shows call limit effects on the sign magnitude of reversal? Can you provide more guidance on how discretisation granularity interacts with flat H-peaks to produce fragile quantity predictions, and how practitioners should set grids to avoid artefacts?
I would or could raise the rating if the justification seems convincing enough.
3 comments
Response to Reviewer BSuu (2/3): Monotone Comparative Statics and Scaling Ratios in Published Post-Training
[Q3.1] When a monotone marginal ratio rules out reversal
Yes: under unique finite peaks, our monotone comparative condition strictly guarantees no accuracy reversal. In the independent zero-cost market of Proposition 5, let and denote negative log failure before and after the upgrade, with . Crucially, the sufficient no-reversal condition governs the marginal ratio rather than the level ratio . Under pointwise improvement , unique finite peaks , and unchanged rival peaks, the identity implies for all whenever is nondecreasing. Peak uniqueness rules out downward quantity jumps across tied maxima (which would otherwise require a tie-breaking rule), forcing and guaranteeing .
[W2/Q3.2] Empirical scaling curves after post-training
Empirically, real model upgrades do not universally exhibit nondecreasing ratios. In our primary Qwen markets, where buyer purchases and unilateral price-deviation checks are evaluated directly on measured task-aligned empirical success (using parametric fits only for initialization), we already observe two full equilibrium reversals and two attenuations (Table 1). To inspect how relative scaling ratios behave in other published post-training pipelines, we also computed measured level and secant ratios from two recent studies. Writing for measured pass@k accuracy and with , the secant ratio across consecutive reported call budgets equals the -weighted interval mean of for any smooth increasing interpolant:
- Diminishing ratios under RL post-training (MATH-500, Yue et al., NeurIPS 2025, Table 3): All six RL variants outperform the Qwen2.5-7B base model at both and while exhibiting sharply falling level ratios across that range. For GRPO, the level ratio drops from 3.22 at 1 call to 1.09 at 256 calls, and consecutive interval secants drop from 3.22 to 0.78, which is mathematically incompatible with a globally nondecreasing for any smooth interpolant through the reported pass@k points.
- Rising ratios under hard-test synthesis (LiveCodeBench-105, He et al., ICLR 2026, HardTestGen Table 3): Comparing Qwen3-4B-HT against its Qwen3-4B base at and shows the level ratio rising from 1.03 to 1.26 and consecutive interval secants rising from 1.03 to 1.60, a pattern compatible with an increasing marginal ratio across those two measured intervals.
Together with our Qwen experiments, these published benchmarks demonstrate that modern post-training produces both falling and rising empirical secant ratios across practical call budgets. Superiority at both ends of a reported pass@k range (such as and large ) leaves wide variation in how capability scales across intermediate call budgets, underscoring why checking the full shape of the scaling gain is essential when assessing market outcomes.
Response to Reviewer BSuu (3/3): Practical Tail Diagnostics, Grid Granularity, Market Entry, and Relaxed Assumptions
[W3] Tail behavior and peaks at the call limit
Importantly, non-attainment of a finite peak of does not require exotic super-exponential tails: even standard power-law failure produces , which increases strictly toward without attaining an interior peak. This gives practitioners a concrete diagnostic workflow on finite samples: (1) inspect log-doubling increments (which average over consecutive log-compute intervals) to check whether empirical turns down inside the observed horizon or rises into the cap, (2) flag boundary maximizers and widen the call horizon where feasible, and (3) report the near-optimal call span alongside Proposition 4's constructive unilateral revenue-gain bound (where upper-bounds over the admissible domain), which explicitly bounds the maximum profitable revenue ratio from any unilateral price deviation even when the peak is approached only asymptotically.
[W5/Q4] Flat peaks, grid spacing, and call caps
Flat -peaks and grid granularity play distinct roles in continuous approximations versus our discrete integer-call API game. In continuous approximations, under uniform -estimation error , solver tolerance , local curvature , and global peak separation ( below the outside gap), Lemma 6 bounds peak-location drift by . Evaluating on a continuous mesh of step near a smooth interior maximum () adds at most to peak-value loss, so practitioners should choose so that . By contrast, our simulated integer-call game keeps unit one-call spacing throughout: it scans every integer call pair up to the cap and checks continuous unilateral price deviations at every affordability threshold. Varying the call cap across 32, 64, and 128 expands the set of supported call quantities rather than changing mesh spacing, reducing boundary selections from 12.5% at 64 to 3.1% at 128 (Appendix G.3).
[W4] Competitor entry and three-provider markets
In the independent zero-cost benchmark of Theorem 1 (with unique or consistently selected unchanged rival peaks), adding an independent entrant with delivered capability multiplies the equilibrium success difference by , attenuating magnitude while strictly preserving the sign of any reversal. Evaluating a three-provider market under our Figure 1 parameters (, , , focal upgrade from to 0.36) confirms clear analytic equilibrium reversals both when the two rivals match the initial model (88.36% down to 86.67%) and when both rivals already match the upgraded model (84.72% down to 82.49%). Complementing this analytic triad, on the 138 held-out tasks of our 279-task matched LiveCodeBench archive across Claude 3.5 Sonnet, GPT-4o, and DeepSeek-V3, a fixed one-call-per-provider portfolio achieves 44.44% and 48.73% pass rates before and after Claude's upgrade, beating three calls to the hindsight-best single provider (39.63% and 43.71%).
[W1] Imperfect verification, serving costs, and flexible budgets
Finally, we separately reoptimized equilibrium prices under imperfect verification, positive serving costs, and buyer value for unspent budget (Table 2 and Appendix G.3): (1) under imperfect verification on LiveCodeBench (filtering candidates on public tests and scoring on hidden tests, with 75% of price profiles meeting the 1% deviation criterion), the net coding effect shifts by −0.6 percentage points relative to the primary perfect-verifier baseline, (2) positive serving costs at yield mean net equilibrium losses of −0.4 and −0.6 percentage points on coding and math (and −0.2 and −0.5 percentage points when costs scale with measured generation time), and (3) valuing unspent funds via () produces net changes of +0.6, +0.3, −0.1, −0.6, and −0.9 percentage points across , turning negative and deepening steadily across the higher task valuations.
Response to Reviewer BSuu (1/3): Cross-Model Dependence and Nonlinear Pricing Contracts
Thank you for recognizing our log-compute marginal capability characterization as general and actionable, for highlighting the price-mediated accuracy reversal, and for valuing our serving-cost analysis and constructive revenue bounds. Guided by your feedback, we sharpened Section 4 to specify both the global peak-separation condition required alongside local curvature for peak-location stability and the finite-sample limits on identifying distant tail peaks. This makes clear why we use smooth curve fits only to initialize price search, and then verify candidate approximate equilibria on the integer call grid via exact continuous-price best-response sweeps under our 1% budget-normalized deviation threshold. We also stated the exact hypotheses of our monotone no-reversal condition in Section 3.4.
[Q1] Cross-model error dependence
Our empirical baseline directly accounts for cross-model error correlation driven by shared task difficulty across benchmarks (Kim et al., ICML 2025). All four primary Qwen comparisons (Table 1) evaluate joint failure probabilities task-by-task rather than assuming population-level independence, which avoids the 3.6 and 1.9 percentage-point joint-success errors produced by multiplying population marginal failure rates on coding and math (Table 5).
To stress-test stronger within-task error correlation on frontier outputs, we analyzed 279 archived LiveCodeBench tasks (Jain et al., ICLR 2025) with 10 graded generations per task and service, pairing June and October 2024 Claude 3.5 Sonnet checkpoints against GPT-4o. Holding each task's measured failure curves W1,W2 fixed, we interpolated conditional within-task joint failure along the linear mixture path from conditional independence W1W2 (ρ=0) to maximal positive error overlap min(W1,W2) (ρ=1), the Fréchet-Hoeffding upper bound at those task margins. Across all 121 call allocations on the 138-task split, shifting from ρ=0 to ρ=1 changed portfolio success by at most 0.080 percentage points for June Claude and 0.646 percentage points for October Claude, bounding the change in optimized buyer success at any fixed prices. Solving the two task splits (141 and 138 tasks) separately, each checkpoint and split admits a common price pair that remains a valid equilibrium across every ρ∈[0,1] along this path, with the equilibrium upgrade gain at those prices on the 138-task split shifting only from 4.79 to 4.35 percentage points between ρ=0 and ρ=1. Dependence does alter equilibrium structure when it removes cross-provider differentiation, for instance if two zero-cost providers share the same latent per-task success rate and generate candidates conditionally independently given that rate, collapsing joint failure to F(x1+x2) so rival calls become perfect substitutes and Bertrand undercutting eliminates positive-price pure-strategy equilibria.
[W1/Q2] Nonlinear billing, block pricing, and two-part tariffs
Our H-peak characterization extends directly to nonlinear metered schedules where provider j chooses a scalar rate aj>0 on a fixed, strictly increasing billing schedule ajtj(xj) with tj(0)=0: re-indexing compute in billed units yj=tj(xj) recovers Theorem 1 in billed units whenever the transformed capability curves satisfy all of its regularity, concavity, and finite-peak hypotheses. Furthermore, for fixed power-law volume schedules tj(x)=xβj (βj≥1), log-compute marginal capability rescales to Hj(x)/βj, which preserves the peak location zj, leaves physical equilibrium call quantities and accuracy reversals unchanged, and sets budget shares proportional to Mj/βj (where Mj=maxxHj(x)). On an integer call grid, if C−j denotes spending on rivals, a fixed block schedule ajtj(kj) preserves our threshold best-response structure with affordability cutoffs at aj=(B−C−j)/tj(kj) for kj>0 and tj(kj)>0, whereas allowing providers to choose multi-tier block prices simultaneously expands the unilateral deviation space.
By contrast, unrestricted two-part tariffs (where each provider jointly chooses an upfront access fee Aj≥0 and a per-call price pj>0) fundamentally alter competition by allowing surplus extraction through the entry fee. In a zero-cost market where at least two providers have unbounded capability, no finite pure-strategy equilibrium with positive per-call prices exists: at least one unbounded provider must earn less than the total budget B, and that provider can profitably deviate by setting its access fee slightly above its current revenue (still below B) while slashing its per-call price enough that spending the remaining budget solely on its calls strictly dominates every rival-only bundle.