- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper provides a theoretical analysis of cascaded inference with LLMs. While LLM cascades often apply confidence calibration before thresholding, the paper shows that a strictly increasing recalibration does not change the cost–quality frontier (Theorem 1), and that, under certain assumptions, the frontier depends only on the base rate p and the ROC curve (Theorem 2). It further shows that the benefit of calibration lies in enabling the selection of a threshold that meets a risk target without labels (Theorem 11). Experiments using Claude Haiku/Sonnet/Opus 4.5 on GSM8K, MATH-500, and MMLU-Pro yield results that support the theory.
Critical strengths
-
S1. This paper theoretically shows that, in LLM cascaded inference, strictly increasing recalibration (e.g., temperature and Platt scaling) does not change the cost–quality frontier. Cascaded inference is an important technique for reducing the inference cost of LLMs, and many methods apply calibration before the thresholding decision. This paper shows that the calibration used in these methods does not move the frontier, but instead is useful for threshold selection.
-
S2. In addition to the theory, the paper provides empirical validation. While temperature and Platt scaling improve ECE, the change in the frontier is exactly zero in every cell.
Critical weaknesses
-
W1. The novelty of the problem framing is limited. Prior work [1] has already pointed out that standard calibration is not well suited to the objective of cascaded inference, and that the value of escalation depends on the correctness of the higher-tier model. In image classification, [1] argued and empirically showed that calibration methods such as temperature scaling reflect only the correctness of the fast model and do not account for the correctness of the expensive model, and therefore do not necessarily improve the accuracy–cost trade-off of a cascade. The paper does not discuss its relationship to [1]. The paper positions overturning the common belief that “calibration is the bottleneck” as its main contribution; in light of [1], this positioning is overstated.
-
W2. The paper does not present a concrete method for improving the frontier. Compared with [1], which proposes a calibration method that accounts for the correctness of the higher-tier model and demonstrates an improved accuracy–cost trade-off, the practical implication of this paper remains limited to the negative conclusion that “calibration does not improve the frontier.”
[1] Enomoto, Shohei, and Takeharu Eda. "Learning to cascade: Confidence calibration for improving the accuracy and computational cost of cascade inference systems." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 8. 2021.
Questions for authors
- Q1. (Related to W2) Based on the theoretical and empirical results presented in this paper, can the authors suggest concrete directions for improving the frontier of cascaded inference (e.g., signal design or calibration methods)?