acceptodds
Under review as a conference paper at ICLR 2027

Calibration Cannot Move the Cost-Quality Frontier of an LLM Cascade

Abstract

Cascaded inference routes each query to the cheapest model that can answer it, escalating when a confidence score falls below a threshold. A large body of work treats miscalibrated confidence as the bottleneck for that decision and adds a recalibration step. We show this step cannot help. A strictly increasing recalibration map only reparametrizes the score, so it cannot change which queries a threshold accepts at a matched escalation rate, and the achievable cost–quality set is unchanged. This identity holds for arbitrarily many tiers and on every sample, not just on average (Theorem 1). We place it in a four-way taxonomy indexed by the σ-algebra the escalation rule can see. The frontier turns out to depend only on the base rate and the ROC curve; no calibration quantity appears in it (Theorem 2). We measure the invariance on 6,471 generations from three model tiers over 701 items of GSM8K, MATH-500, and MMLU-Pro. Temperature and Platt scaling move the swept frontier by exactly zero, in all 15 task × signal cells, while mean expected calibration error (ECE) falls by roughly half. The zero survives a 25× sweep of the opus:haiku price ratio. Discrimination is what moves the frontier instead, and it is close to saturated: six confidence signals are statistically indistinguishable, with AUROC between 0.761 and 0.767. A fused score improves AUROC significantly (+0.044, 95% CI [+0.007, +0.083]) but yields no resolvable frontier gain, a preregistered prediction we falsify. Calibration’s real benefit is narrower than the literature assumes, and one-sided. It lowers how often a label-free risk promise is broken, from 35.8% of cells raw to 13.5% after temperature scaling, without making the risk estimate itself any more accurate, and hitting a target escalation budget needs no calibration at all. Recalibration is a controllability device, not a capability device.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.