acceptodds
Withdrawn from ICLR 2027

Calibration Cannot Move the Cost-Quality Frontier of an LLM Cascade

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cascaded inference routes each query to the cheapest model that can answer it, escalating when a confidence score falls below a threshold. A large body of work treats miscalibrated confidence as the bottleneck for that decision and adds a recalibration step. We show this step cannot help. A strictly increasing recalibration map only reparametrizes the score, so it cannot change which queries a threshold accepts at a matched escalation rate, and the achievable cost–quality set is unchanged. This identity holds for arbitrarily many tiers and on every sample, not just on average (Theorem 1). We place it in a four-way taxonomy indexed by the σ-algebra the escalation rule can see. The frontier turns out to depend only on the base rate and the ROC curve; no calibration quantity appears in it (Theorem 2). We measure the invariance on 6,471 generations from three model tiers over 701 items of GSM8K, MATH-500, and MMLU-Pro. Temperature and Platt scaling move the swept frontier by exactly zero, in all 15 task × signal cells, while mean expected calibration error (ECE) falls by roughly half. The zero survives a 25× sweep of the opus:haiku price ratio. Discrimination is what moves the frontier instead, and it is close to saturated: six confidence signals are statistically indistinguishable, with AUROC between 0.761 and 0.767. A fused score improves AUROC significantly (+0.044, 95% CI [+0.007, +0.083]) but yields no resolvable frontier gain, a preregistered prediction we falsify. Calibration’s real benefit is narrower than the literature assumes, and one-sided. It lowers how often a label-free risk promise is broken, from 35.8% of cells raw to 13.5% after temperature scaling, without making the risk estimate itself any more accurate, and hitting a target escalation budget needs no calibration at all. Recalibration is a controllability device, not a capability device.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM cascades, calibration, selective prediction, uncertainty estimation, cost-quality tradeoffs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RNGtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper provides a theoretical analysis of cascaded inference with LLMs. While LLM cascades often apply confidence calibration before thresholding, the paper shows that a strictly increasing recalibration does not change the cost–quality frontier (Theorem 1), and that, under certain assumptions, the frontier depends only on the base rate p and the ROC curve (Theorem 2). It further shows that the benefit of calibration lies in enabling the selection of a threshold that meets a risk target without labels (Theorem 11). Experiments using Claude Haiku/Sonnet/Opus 4.5 on GSM8K, MATH-500, and MMLU-Pro yield results that support the theory.

Critical strengths
  • S1. This paper theoretically shows that, in LLM cascaded inference, strictly increasing recalibration (e.g., temperature and Platt scaling) does not change the cost–quality frontier. Cascaded inference is an important technique for reducing the inference cost of LLMs, and many methods apply calibration before the thresholding decision. This paper shows that the calibration used in these methods does not move the frontier, but instead is useful for threshold selection.

  • S2. In addition to the theory, the paper provides empirical validation. While temperature and Platt scaling improve ECE, the change in the frontier is exactly zero in every cell.

Critical weaknesses
  • W1. The novelty of the problem framing is limited. Prior work [1] has already pointed out that standard calibration is not well suited to the objective of cascaded inference, and that the value of escalation depends on the correctness of the higher-tier model. In image classification, [1] argued and empirically showed that calibration methods such as temperature scaling reflect only the correctness of the fast model and do not account for the correctness of the expensive model, and therefore do not necessarily improve the accuracy–cost trade-off of a cascade. The paper does not discuss its relationship to [1]. The paper positions overturning the common belief that “calibration is the bottleneck” as its main contribution; in light of [1], this positioning is overstated.

  • W2. The paper does not present a concrete method for improving the frontier. Compared with [1], which proposes a calibration method that accounts for the correctness of the higher-tier model and demonstrates an improved accuracy–cost trade-off, the practical implication of this paper remains limited to the negative conclusion that “calibration does not improve the frontier.”

[1] Enomoto, Shohei, and Takeharu Eda. "Learning to cascade: Confidence calibration for improving the accuracy and computational cost of cascade inference systems." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 8. 2021.

Questions for authors
  • Q1. (Related to W2) Based on the theoretical and empirical results presented in this paper, can the authors suggest concrete directions for improving the frontier of cascaded inference (e.g., signal design or calibration methods)?
Reviewer_q1D1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper argues that calibration of a confidence score does not influence the performance frontier of a cascade.

This claim is clearly correct; as soon as a cascade optimization algorithm sweeps over all possible confidence thresholds, the set of considered configurations is invariant under monotone transformation of the confidence score.

The paper tests this obvious point empirically. Since the work does not present a novel research advance, I recommend rejection of the paper.

Incidentally, the paper is likely written by AI, as judged from the inclusion of phrases such as "here calibration is load-bearing" and headings such as "Cost model, and a harness artifact we must disclose." After scrolling to the AI disclosure section, this suspicion is confirmed by the following quote: "an AI system performed essentially all of the research work in this paper under human direction."

Critical strengths
  • None
Critical weaknesses
  • Belaboring an obvious point with complicated mathematics, including sigma algebras
Questions for authors
  • Your own AI disclosure states that "an AI system performed essentially all of the research work in this paper under human direction." Please do not submit AI-generated research to an AI conference. We human reviewers are already swamped with our reviewing workloads, and AI-generated submissions without human input only clog the system and waste everyone's time.
Reviewer_f6vNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether confidence recalibration improves the cost–quality frontier of LLM cascades. It proves that strictly increasing recalibration preserves threshold decisions and therefore cannot change the achievable frontier. The authors also characterise the role of ranking, additional information, and calibration in routing decisions. Experiments on 701 items across GSM8K, MATH-500, and MMLU-Pro show that calibration reduces ECE without changing the frontier. Additional experiments examine discrimination, signal fusion, risk targeting, and inference costs.

Critical strengths
  • Rigorous theoretical treatment: The paper formalises the invariance of threshold-based cascades under strictly increasing recalibration, including multi-tier settings and discrete confidence scores.
  • Detailed empirical analysis: The experiments examine calibration, discrimination, fusion, and risk targeting, with statistical uncertainty and transparent reporting of negative results.
Critical weaknesses
  • Limited novelty: The central observation that monotonic recalibration preserves ranking is well established. Prior work already distinguishes score-based and rank-based threshold selection [a], and has studied cascade-specific calibration and cost–quality optimisation [b, c]. The paper does not sufficiently establish what fundamentally new understanding its theoretical extension provides.
  • Limited practical validation: The main zero-frontier-change experiment largely verifies what the theorem guarantees. More importantly, the five-sample Haiku configuration costs more than using Opus directly, and the reported cascade operating points are economically dominated by Opus alone. Also, the constant-cost assumption does not fully match Table 3, where policies at identical escalation rates have different measured costs. The experiments are further limited to one model family and 701 items.
Questions for authors
  1. How does the main theoretical contribution differ from established threshold-selection theory [a] and earlier cascade optimisation work [b, c]? What is the principal new insight beyond applying rank preservation to multi-tier cascades?
  2. Could the authors demonstrate their conclusions using an economically competitive cascade, such as single-sample Haiku routing? How do the constant-cost assumptions relate to the variable costs reported in Table 3?
  3. Could the authors compare calibrated risk targeting against conservative rank-based threshold selection at matched acceptance rates? This would help establish whether calibration provides a distinct practical advantage

References

[a] https://www.jmlr.org/papers/v13/hernandez-orallo12a.html

[b] https://ojs.aaai.org/index.php/AAAI/article/view/16900

[c] https://proceedings.mlr.press/v267/dekoninck25a.html

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.