acceptodds
Under review as a conference paper at ICLR 2027

When Does Depth Help? Theory and Evidence on LLM Cascades

Abstract

LLM cascades, in which a cheap model defers to an expensive one on low-confidence queries, are widely used to reduce inference cost. Given a pool of models, a practitioner must decide how many models to include and where to set each deferral threshold. We derive first-order optimality conditions showing that, at an optimum, the ratio of expected accuracy gain to expected downstream cost is equal across deferral boundaries. A local search based on these conditions closely matches exhaustive search. We also derive an identity that decomposes the accuracy gain of score-based escalation over random escalation into two AUROC terms. Across five benchmarks and nine deferral scores, with model sequences and thresholds optimized from a pool of eight models, two-model cascades improve mean test-set accuracy over single-model selection by 2.1 to 8.2 percentage points. However, allowing more than two models does not improve mean test-set accuracy in 118 of 135 comparisons across scorers, datasets, and depth caps, and adds at most 0.43 percentage points. To understand the role of deferral scores in depth gains, we conduct counterfactual experiments with simulated confidence scores. When these scores have high AUROC and reflect only whether the current model answered correctly, allowing more than two models improves test-set accuracy on four of five benchmarks. However, these gains do not persist when the scores also reflect query difficulty shared across models, even at the same AUROC. These results suggest that gains from additional depth depend on how well the confidence score separates correct from incorrect answers for the current model compared with later models.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM, cascade, uncertainty

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FQimOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether increasing the number of models in an LLM cascade improves the accuracy–cost tradeoff. The authors derive optimality conditions for confidence thresholds, introduce an AUROC-based characterization of escalation utility, and provide empirical evidence that conventional confidence-based cascades rarely benefit from more than two models.

Critical strengths
  1. I think the motivation is quite clear and the question is interesting and relevant to the field.
  2. The distinction between predicting the current model's correctness and identifying beneficial escalation is interesting.
Critical weaknesses
  1. I am not fully convinced by the theoretical contribution. Theorem 3.1 gives optimality conditions for threshold selection, while Proposition 3.2 studies the benefit of confidence-based escalation. However, neither directly explains when adding more models is actually beneficial.

  2. I have some concerns about the generalizability of the empirical results. The benefit of deeper cascades strongly depends on the task distribution and the models being used. I am not sure what conclusions we can draw beyond the specific setting considered in the paper.

  3. The novelty compared to existing learning-to-defer work is not entirely clear to me. In particular, the idea that confidence in the current model does not necessarily indicate whether deferring is beneficial is already well established.

  4. I am not fully convinced that the synthetic experiments explain why deeper cascades provide limited improvements. Changing the confidence scores also changes their correlations across models, making it difficult to identify which factor is actually responsible for the results.

Questions for authors
  1. Could the authors clarify whether the limited gains from deeper cascades come mainly from the models making similar errors or from the confidence scores not being informative enough?

  2. Is there a more direct connection between Proposition 3.2 and the benefit of adding more models? I am not sure how the theoretical result explains the main empirical findings.

  3. Would the conclusions change if the deferral scores estimated the benefit of the entire downstream cascade rather than just the next model?

7 comments
AuthorsOpenReview ↗

Response to Reviewer FQim (1/2): Theory and depth (W1, Q1, Q2)

Thank you for the precise questions. Each of them led to a change in the paper.

W1, Q2. Connecting the theory to depth. Section 3.4 now derives a necessary condition for a three-model cascade to improve on two-model cascades, stated through the quantity in Proposition 3.2 (Equation 9, proved for any depth in Appendix A.6).

Consider a cascade (M1,M2,M3)(\mathcal{M}_1, \mathcal{M}_2, \mathcal{M}_3) in which a fraction tt of the queries reaching M2\mathcal{M}_2 escalate. We compare it with a mixture that sends a random fraction tt of queries to the cascade (M1,M3)(\mathcal{M}_1, \mathcal{M}_3) and the rest to (M1,M2)(\mathcal{M}_1, \mathcal{M}_2), each with the threshold that the three-model cascade applies to M1\mathcal{M}_1. The accuracy of the three-model cascade exceeds that of this mixture by a2(t)−aˉ2(t)a_2(t) - \bar a_2(t), the gain from using the score of M2\mathcal{M}_2 to choose which queries escalate. Its expected cost exceeds that of the mixture by a term κ(t)\kappa(t), because it also calls M2\mathcal{M}_2 on the queries that escalate. It can therefore improve on every mixture of two-model cascades at the same budget only if

a2(t)−aˉ2(t)>λκ(t),a_2(t) - \bar a_2(t) > \lambda \kappa(t),

where λ\lambda is the shadow price of the budget among those mixtures. The gain must exceed the value of the added cost.

By Proposition 3.2, the average of the left side over tt rises with the AUROC of the score of M2\mathcal{M}_2 for the correctness of M2\mathcal{M}_2 and falls with its AUROC for the correctness of M3\mathcal{M}_3. A score that reflects difficulty shared by both models raises both AUROCs, so the gain can be small even when it predicts the correctness of M2\mathcal{M}_2 well. This accounts for the main empirical finding. Section 4.4 tests both predictions of the condition by varying the two AUROCs separately. Raising own-correctness AUROC to 0.9 makes S3S_3 improve on S2S_2 on every benchmark, and shifting the same signal to difficulty shared across models removes the gain.

ScoresS3−S2S_3 - S_2 difference (pp, range across datasets)
Observed mean token negentropy−0.28-0.28 to +0.61+0.61
Synthetic, own-correctness AUROC 0.9+0.32+0.32 to +1.87+1.87
AUROC 0.9, signal shifted to shared difficulty−0.24-0.24 to −0.03-0.03

Q1. Similar errors or uninformative scores? Our results indicate that the limited gains from deeper cascades are attributable primarily to the confidence scores rather than to similarity in the models' errors. Two experiments hold every model's answers and costs fixed, which leaves the overlap in model errors unchanged, and vary only the scores (Section 4.4, Appendix C.9). All ranges below are across the five datasets. With the observed scores, the S3−S2S_3 - S_2 difference ranges from −0.28 to +0.61 pp. Raising own-correctness AUROC to 0.9 increases this range to +0.32 to +1.87 pp. Replacing each score with the model's own correctness label, a diagnostic with an AUROC of 1, increases it to +0.92 to +3.80 pp. These results show that the models' errors differ enough for cascades of up to three models to improve on two-model cascades, provided the scores can identify the queries that benefit from escalation, but the observed scores lack this ability. Consistent with the necessary condition in Equation 9 (Section 3.4), the gain at AUROC 0.9 disappears when the score signal reflects difficulty shared across models. The limiting factor is therefore the extent to which a score distinguishes a model's own errors from errors that the next model would also make.

Continued in another comment...

AuthorsOpenReview ↗

Response to Reviewer FQim (2/2): Controls, novelty and generalizability (W2, W3, W4, Q3)

W4. Score quality versus score correlation. The synthetic score experiments for own-correctness AUROC are not subject to this concern, because they hold the observed cross-model score correlation fixed while AUROC varies. However, we agree this was a potential confound in the shared-difficulty experiment, and we have since added an ablation to address it (Appendix C.10). Specifically, we group queries with the same correctness outcomes across all ten models, and within each group we shuffle each model's scores across queries, using a separate shuffle for each model. This leaves each score's relation to every model's correctness unchanged and makes different models' scores independent among queries with the same correctness outcomes. Across the five benchmarks, shifting the score signal to shared difficulty at AUROC 0.9 lowers the S3−S2S_3 - S_2 difference by 0.44 to 2.01 pp, and the shuffle changes this decline by at most 0.13 pp, suggesting that our finding is robust to correlation between models' scores beyond what correctness explains.

Q3. Scores for the entire downstream cascade. Thank you for the question. The theory already covers this case, since Vi+1V_{i+1} in Theorem 3.1 and U>iU_{>i} in Proposition 3.2 measure the quality of the answer returned by the entire remaining cascade. To test the question empirically, we added an experiment in which we train scores on the benefit of the entire downstream cascade (Appendix C.12). For a three-model cascade (M1,M2,M3)(\mathcal{M}_1, \mathcal{M}_2, \mathcal{M}_3), the first-stage score predicts the correctness of the answer returned by the full (M2,M3)(\mathcal{M}_2, \mathcal{M}_3) cascade minus that of M1\mathcal{M}_1. Across the five benchmarks, the S3−S2S_3 - S_2 difference lies between −0.34 and +0.47 pp, within the range of the nine scorers in Table 12, and does not exceed that of a matched correctness-trained score on any benchmark (Table 33). Therefore, the conclusions do not change when the scores estimate the benefit of the entire downstream cascade.

W3. Novelty relative to learning to defer. We agree that the qualitative point is established in the learning to defer literature, and Section 2 now states this explicitly. We contribute an identity that quantifies the point for an arbitrary downstream cascade, a condition that turns it into a statement about depth, and, to our knowledge, the first comparison of depth classes in LLM cascades with jointly optimized sequences and thresholds.

W2. Generalizability. We added reasoning models of 501B and 975B total parameters and repeated every analysis with the ten-model pool. We also added a table of the pool (Table 1), which shows substantial variation in accuracy and cost across models. The five benchmarks cover mathematics, multiple-choice questions, factual recall and code generation. Across them, the accuracy of the best model ranges from 41% to 95% and that of the worst model from 4% to 55%. The S3−S2S_3 - S_2 difference is stable when any one model is removed via a leave-one-out ablation (−0.33 to +0.68, Appendix C.4) and under nine alternative price spacings (−0.31 to +0.75, Section 4.4). The same pattern holds for cascade routing (Dekoninck et al., 2025), a relaxed policy class that chooses the next model per query, where allowing a third model changes accuracy by −0.07 to +0.22 pp (Table 3). The paper lists the pool, scores and benchmarks as limitations. Beyond this setting, the condition above and Section 4.4 indicate when the conclusion should change.

We hope these changes address your concerns. If they do, we would be grateful if you would consider updating your score, and we are glad to clarify further.

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for your review. As the discussion period closes on November 18, we wanted to follow up on our response. In response to your concerns, the revised manuscript adds the following:

  • A necessary condition connecting Proposition 3.2 to the benefit of a third model (Section 3.4, Equation 9)
  • A diagnostic that replaces each score with the model's own correctness label (AUROC of 1), separating the role of the scores from the overlap in model errors (Appendix C.9)
  • An ablation that removes correlation between models' scores beyond what correctness explains (Appendix C.10)
  • Scores trained on the benefit of the entire downstream cascade (Appendix C.12)
  • A discussion of our contribution relative to the learning-to-defer literature (Section 2)
  • Inclusion of two larger reasoning models (501B and 975B) and a leave-one-model-out analysis of dependence on the 10-model pool (Appendix C.4)

If any of your concerns remain, we would be glad to address them before the discussion closes.

Reviewer_sL7XOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates whether deeper cascades yield better performance. The authors provide aggregate empirical results across several benchmarks, concluding that 2-model cascades outperform both the best single models as well as deeper cascades. The paper presents an exploration on synthetic data to explain under what conditions deeper cascades raise performance.

As far as contributions go, I'm concerned that the conclusions of this paper are heavily influenced by specific choices of models and tasks in their benchmarks, and are not universally valid as suggested by the paper.

Critical strengths
  • The main claim that two-model cascades outperform both single best models and deeper cascades, if it can be substantiated with more evidence, and a more transparent evaluation
Critical weaknesses
  • The evaluation lacks clear perspective on the magnitudes of cost/accuracy differences between models; the set of evaluated models and tasks may be limited: I'm concerned that the results are strongly dependent on chosen set of models under evaluation and their cost/accuracy profiles on the limited set of tasks, rather than representing universal findings on cascades
Questions for authors
  • You report cascade gain with a single metric averaged over cost budgets; this obfuscates the cost-accuracy trade-off for different cost budgets. Specifically, the single best model should always be chosen when the cost budget is high enough; the cheapest model should always be chosen when the cost budget is low enough. Since your analysis doesn't clearly hit these two extreme limits, it's unclear if your evaluation regime covers the full range of cascade behavior.
  • Related to the above point, your selection of models leans heavily towards small models. In general, the benefit of a cascade stage depends on the degree to which the accuracy gain from escalation outweighs the additional cost incurred from escalation. Thus, whether an additional stage yields a benefit can depend strongly on the diversity of accuracy/cost tiers within the models under consideration. I highly recommend including a table in the main body of the manuscript showing each model's individual cost and accuracy.
  • Could you include a greater diversity of models in your evaluation? Specifically, I would like to see larger models being evaluated (e.g. models with 500B or 1T parameters). It would also be interesting to explicitly include reasoning LLMs in the evaluation, since non-reasoning models are substantially out-of-date in 2026. For reasoning cascades, consider including answer generation speed / latency as one of the cascade performance metrics, rather than focusing only on budget.
  • I'm not sure the tasks are sufficiently diverse as far as overall model accuracy, and differences in accuracy between models, are concerned. Could you display these clearly in a table in the main body of the manuscript?
  • Have you considered using as confidence metric the log-prob of a single token (e.g., MaxProb for a classification, or p(true) on a text generation problem)? These single-token based confidence metrics typically perform better than sequence-wide token confidence scores.

References:

4 comments
AuthorsOpenReview ↗

Larger reasoning models, per-model table, and budget-level results

Thank you for the concrete requests. We implemented each of them, and we believe they improved the paper considerably.

Q3. Larger and reasoning models. We added Beam-1.0 (501B total parameters) and Inkling (975B), both reasoning models, and reran every analysis with ten models. The pool also contains DeepSeek-V3 (671B) and two smaller reasoning models, GPT-oss-20B and MiniMax-M2.7. Beam-1.0 or Inkling is the most accurate model on four of the five benchmarks.

Q2, Q4. Cost and accuracy of every model. Table 1 in the main text now reports parameters, reasoning status, accuracy and mean cost for all ten models on all five benchmarks. Within a benchmark, the highest value exceeds the lowest by a factor of 1.6 to 11 for accuracy and 140 to 993 for cost.

W1. Dependence on the pool. With the two larger reasoning models added, our findings hold. Across all nine deferral scores and five benchmarks, two-model cascades improve on the best single model, while deeper cascades add little beyond two models (Table 2).

To test whether this result depends on the composition of the model pool, we ran a leave-one-model-out analysis (Appendix C.4). For each of the ten models, we removed that model and repeated the full cascade search on the remaining nine. Across all ten exclusions and five benchmarks, the S3 − S2 difference lies between −0.33 and +0.68 pp, suggesting robustness of our results to removal of any single model. We also tested sensitivity to model prices by replacing the observed prices with nine alternative cost spacings of the intermediate models and arrived a simliar findings. Section 4.4 also characterizes the conditions under which deeper cascades do help, providing guidance to practitioners on when the finding applies to their own setting.

Q1. Budget averaging and the extremes. Wee agree that the average concealed structure, and Section 4.3 now reports results by budget position. Pooled over scores and datasets, S2 improves on S1 by more than 0.5 pp at 83% of positions and S3 improves on S2 by that margin at only 11% of positions (Appendix C.3).

The budget grid spans both limits you describe. It starts at the cost of the cheapest model, where every cascade class selects that model, and ends at the cost of the most accurate model, where that model becomes affordable. The revision makes these endpoints explicit and names the model that sets the upper endpoint on each benchmark (Appendix B.1). At this upper limit, a cascade can still exceed the most accurate model, because cheaper models answer some queries correctly that the most accurate model misses.

Q5. Single-token and answer-level confidence. We computed the minimum and mean log-probability over only the final-answer tokens on MATH, TriviaQA and MMLU. On MMLU this is the probability of the selected option letter, which is MaxProb. On MATH and TriviaQA these scores discriminate better than the other deferral scores, as you expected, and they strengthen two-model cascades. This own-correctness AUROC improvement is not large enough to yield gains from deeper cascades. Appendix C.2.3 reports the full results. As stated in the footnote in Section 4.1, we do not consider P(True), self-consistency, verbalized confidence or LLM-as-a-Judge, because each requires additional generation calls, which add to the cost that cascading aims to reduce. Section 4.4 instead simulates scores up to AUROC 0.9 and shows when such scores make depth useful.

Latency. We measure cost in dollars rather than latency, because API latency depends on rate limits, throttling and server load. These factors lie outside our control and are difficult to standardize across providers and models. Section 4.1 and the limitations now state this. As a proxy for latency, we report the mean number of model calls per query for each cascade class in Appendix C.3.

Q6. Covariance formula. Proposition 3.2 extends Theorem 3 of https://arxiv.org/pdf/2507.03834, and Section 3.4 now cites the theorem and makes the connection explicit. Written for accuracy among queries reaching stage ii, with DtD_t the indicator that a query is among the fraction tt with the lowest scores, that theorem gives

ai(t)−aˉi(t)=pi(Cov(Dt,U>i)−Cov(Dt,Ui)),a_i(t) - \bar{a}_i(t) = p_i (\mathrm{Cov}(D_t, U_{>i}) - \mathrm{Cov}(D_t, U_i)),

which is the first step of our proof (Appendix A.5). Averaging over tt and using Cov(ρ,Y)=π(1−π)(AUROC−1/2)\mathrm{Cov}(\rho, Y) = \pi(1 - \pi)(\mathrm{AUROC} - 1/2) for binary YY and rank ρ\rho gives Proposition 3.2. Our form generalizes the theorem to an arbitrary stage and an arbitrary downstream cascade, and expresses the result through AUROC. While that paper notes that the downstream covariance is much smaller in practice, our shared-difficulty experiments indicate that it is the term that matters for depth.

We hope the added models, tables and budget-level results address the soundness concern. If they do, we would be grateful if you would consider updating your score. We are glad to provide anything further.

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for your review. As the discussion period closes on November 18, we wanted to follow up on our response. In response to your concerns, the revised manuscript adds the following:

  • Two larger reasoning models (501B and 975B), increasing the pool size to ten models
  • A leave-one-model-out analysis of dependence on the model pool (Appendix C.4)
  • A table of each model's accuracy and cost on every benchmark (Table 1)
  • Results by budget position (Appendix C.3)
  • Scores computed over the final-answer tokens (Appendix C.2.3)
  • An explicit connection to the covariance formula you referenced (Section 3.4)

If any of your concerns remain, we would be glad to address them before the discussion closes.

Thanks to the authors for their revisions.

The paper benefits from adding the larger and reasoning-based models.

I remain unconvinced by the ratio of the paper's claims to the clarity of the evaluation. Below are a few points I would encourage updating to make the paper clearer. These are not meant as a recipe, but rather as pointers to explain my holistic judgment:

  • Be clearer on how the results come about. Specifically, what are the winning cascades for each benchmark? Is the optimal 2-model cascade a subset of the optimal 3-model cascade? More broadly, one conceptual issue is that a 3-model cascade is capable of performing at least as well as a 2-model cascade, so the negative performance deltas clearly arise from variance/overfitting. Your method of selecting the top model combination within larger pools for k≥3k \geq 3 would amplify the winner’s curse. I would start with a fixed list of models and construct shorter cascades by shortening the list.
  • Evaluate the actual performance differences between cascades more clearly. For example, considering a single model at increasing budget values is misleading because a single model’s costs can never go up. When comparing a single model to a 2-model cascade at different budgets, the single model’s replacement at intermediate budgets should be a method that randomly switches from the single model to a bigger model, because this properly reflects performance gains from using higher budgets. More generally, I would not evaluate accuracy at fixed budget, since budget is no guarantee of actual realized cost, so it confounds performance evaluation with the training algorithm’s performance at hitting the budget. Instead, evaluate accuracy at the same actual realized cost.
  • Maintain greater transparency of your results for k>3k > 3. Your paper's results are already valuable for k≤3k \leq 3. Going beyond k=3k=3 presents issues you are currently not sufficiently addressing. Specifically, the combinatorial number of threshold combinations explodes quickly as you increase kk. Your paper does not clearly lay out the methodology for k>3k>3.
  • Provide code. This is especially important in a technically subtle area like performance evaluation of cascades.
AuthorsOpenReview ↗

New results and clarifications on selected cascades, randomized baseline, realized costs, and search procedure

Thank you for the follow-up and for acknowledging the value of the expanded model pool. We agree that the evaluation should make the selected cascades, their realized costs and the search procedure easier to inspect.

1. Selected cascades and overfitting. The table shows the most frequently selected pair and triple on each benchmark for mean token negentropy.

BenchmarkPairTriple
MMLUGPT-4o-mini → BeamGPT-4o-mini → GPT-oss-20B → Beam
TriviaQAGPT-4o-mini → Llama-3.3-70BGPT-4o-mini → Llama-3.3-70B → GPT-4o
MATHBeam → InklingLlama-3.3-70B → Beam → Inkling
SimpleQAGPT-4o → InklingGPT-4o → Beam → Inkling
LiveCodeBenchGPT-oss-20B → BeamGPT-oss-20B → Llama-3.3-70B → Beam

The selected pair is contained in the selected triple in 86.4%, 99.0%, 98.0%, 99.8% and 95.2% of cases, respectively, for this scorer. The selected triple therefore usually adds one model to the selected pair. We will add these frequencies for all nine scores to the appendix.

We agree that the negative test-set differences reflect overfitting. However, overfitting does not explain the limited gains from a third model. On calibration data, where selection can only favor the deeper class, S3 improves on S2 by +0.05 to +1.01 pp across benchmarks. The gain available from a third model is therefore small before any loss from generalization. With stronger scores, the same selection procedure yields test-set gains of +0.32 to +1.87 pp at an own-correctness AUROC of 0.9 and +0.92 to +3.80 pp with correctness labels as scores, which have an AUROC of 1 (Section 4.4, Appendix C.9), indicating that the scores limit the gain.

2. Randomized baseline and realized costs. We address these two points separately.

Randomized baseline. Table 2 compares each cascade with the most accurate single model affordable at the same budget. Random switching between models is a different and stronger baseline. Like a cascade, it has a parameter, the switching probability, that can be tuned on calibration data to spend the full budget, but it chooses which queries go to the larger model at random. The comparison therefore measures what the cascade gains over choosing escalation queries at random. Proposition 3.2 uses a similar reference, measuring a cascade's gain over escalating a random fraction of queries. We computed this baseline and compared it with the two-model cascades from Table 2 at the same realized test cost. The cascade is more accurate in 29 of 45 score-benchmark combinations (26 of 45 at the original budgets), with a mean difference of +0.67 pp.

Realized costs. Table 2 follows the budget-constrained problem in Section 3.3, in which the budget is fixed in advance and the policy is selected on calibration data, as it would be in deployment. Since realized test cost can differ from the budget, we repeated the depth comparison at the same realized test cost. S3 is more accurate than S2 in 14 of 45 score-benchmark combinations, with a maximum gain of +0.24 pp and a mean difference of −0.31 pp. The limited gains from a third stage therefore persist. We will add both comparisons to the appendix.

3. Search at greater depths. The search is exhaustive up to a depth cap of four. For each split, it covers all 45 pairs, 120 triples and 210 four-model sequences in cost order. For each sequence, the candidate thresholds at each stage are the distinct calibration scores, and we evaluate every threshold combination that satisfies the budget. Combinations that exceed the budget are skipped, which does not change the optimum. We verified the implementation against brute-force enumeration on small instances.

The five-model results use an additional restriction, introduced when the pool grew from eight to ten models, that standalone calibration accuracy must increase along each five-model sequence in cost order. Threshold search is exact within those sequences, and all shallower policies remain eligible. With the original eight models, the five-model search was unrestricted and gave the same conclusion. We will state this restriction more clearly and distinguish those results from the unrestricted search through depth four.

4. Code. The code is included in the submission's supplementary material. We will add a pointer in the main text so that it is easier to find.

We appreciate these concrete suggestions. They identify presentation and evaluation changes that will make the evidence easier to assess, and we hope the clarification and additions will support reconsideration of your assessment.

Reviewer_hvt1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies specifically the depth of LLM cascades, where multiple models are queried in sequence in order to reduce inference cost. The paper focuses on a theoretical understanding of how to set the ratio of expected accuracy gain to expected downstream cost at each deferral threshold, derives the corresponding first order optimality conditions, and conducts extensive experiments validating these conditions and comparing cascades of different depths.

Critical strengths
  • This paper aims to discover the effect of depth in cascaded models, which is an important and underexplored problem in efficient ML serving.
  • This paper has extensive experiments under a pool of 8 models and on 5 benchmarks, with 9 deferral scores, 50 splits, 500 budget positions, exhaustive threshold search up to depth five, and counterfactual experiments on both confidence scores and cost structure.
Critical weaknesses
  • The policy class is restrictive, which limits what the depth comparison can establish. Sequences must be cost ordered, each threshold τ_i is a scalar independent of earlier scores, and the sequence is fixed offline rather than chosen adaptively per query. Where to place a stage or an early exit is not independent of depth, so comparing the optimized classes S_2 through S_5 within this class does not settle whether depth helps for adaptive or router cascade hybrid policies. The limitations section acknowledges this in a single sentence, which is thin given that it qualifies the headline claim.

  • Theorem 3.1 is stated without its assumptions (in the main text). The regularity conditions (continuous and strictly positive joint density, continuity of m_i, V_{i+1}, W_{i+1} in the thresholds, differentiation under the expectation, and the regular conditional distribution needed to condition on the measure zero event s_i = τ_i) appear only in Appendix A.1.

  • Missing related work. The related work coverage is narrow. Early exit and cascaded inference is a large area, and the related work section covers only the recent LLM routing and cascading literature. A substantial body of earlier work on early exit models is missing, including architecture specific designs for different model families (for example, early exit variants of Vision, BERT and GPT style models, which differ in where exits are placed and how the exit decision is made) and the systems literature on early exits in serving, such as Apparate (Dai et al., 2024).

  • No comparison against existing cascade methods. FrugalGPT, AutoMix, C3PO, and Dekoninck et al. are cited but none is run on the same pool and benchmarks. AutoMix and C3PO in particular report gains from exactly the multi stage depth this paper finds unhelpful, and the discrepancy is not examined. Since every reported number comes from the authors’ own pipeline with no external anchor, it is difficult to judge whether the results are consistent within other models in the literature.

Questions for authors

T-TAMER (Yang et al., 2026) also provides a theoretical framework for cascaded models and characterizes optimal stopping and model selection under cost quality trade-offs. How do the first order conditions in Theorem 3.1 and the depth conclusions in Section 4.3 relate to the characterization there?

Yang, Y., Zhang, R., Morgenstern, J., & Xu, H. (2026). T-TAMER: Provably taming trade-offs in ML serving. International Conference on Learning Representations, 38725–38758.

3 comments
AuthorsOpenReview ↗

Relaxed policy classes, external baseline, and stated assumptions

Response to Reviewer hvt1

Thank you for the careful review. Each weakness you raised led to a change in the paper.

W1. Policy class. We agree that one limitations sentence did not sufficiently address this point. The revision replaces it with experiments on the relaxations you name.

RelaxationWhat it relaxesThree-model vs. two-model cap (pp)
Cascade routing (Dekoninck et al., 2025)Model order and stopping rule jointly−0.07 to +0.22
History-dependent thresholdsThreshold rule only−0.09 to +0.71
Recall (Yang et al., 2026)Answer selection (highest-scoring answer among the models called)−0.21 to +0.35

Under each relaxation, a third model adds little beyond two, suggesting that the conclusions of Section 4.3 do not depend on the restrictions of the policy class. Cascade routing is now in the main text (Section 4.3, Table 3), and the other two relaxations are in Appendix C.6.

W4. External anchor. The cascade routing results come from the Dekoninck et al., 2025 released implementation (Appendix C.5). Our deferral scores also follow prior work (Section 4.1). Minimum token probability is the Chow-Quantile rule of Gupta et al. (2024) at α = 0, and length-normalized probability is equivalent to their Chow-Average rule. The benefit-trained scores fit the Diff-01 target of Jitkrittum et al. (2023). The correctness-trained embedding score gives a FrugalGPT-style cascade, with the same policy class and logistic regression on frozen embeddings in place of fine-tuned DistilBERT. Our gains from two-model cascades agree with those reported by FrugalGPT, AutoMix and Gupta et al.

AutoMix and C3PO. We examined this discrepancy, and Section 2 now states the relevant difference for each. We believe the results are compatible. AutoMix compares its three-model system with FrugalGPT and with baselines built from fixed model pairs, whereas our S2 selects the best pair from ten models at each budget. Its signal is self-verification sampled several times, which requires extra generation calls. We scope our deferral scores to signals that require no additional generations, because additional calls add to the cost that cascading aims to reduce. For the same reason, we do not use P(True) or verbalized confidence (Section 4.1). Section 4.4 shows that stronger scores can make depth useful, so a benefit from depth under such a signal is consistent with our results. C3PO evaluates three-model and four-model cascades and reports no two-model comparison, so it does not measure the gain beyond two models. It also notes that more stages fit training data better but may generalize worse, which matches the overfitting we observe.

Q. T-TAMER. T-TAMER models early exit and cascading as costly exploration in which querying a model reveals its loss, with an objective that adds loss and latency. It proves that no policy without recall can attain a bounded approximation to the offline optimum, and that with recall an index policy is optimal among online policies and computable in polynomial time. Theorem 3.1 covers a complementary setting. The cascade observes a confidence score in place of the loss, scores and correctness may be arbitrarily dependent, and cost enters as a budget constraint. Our conditions characterize the best thresholds for a fixed sequence without recall, which is compatible with their result on the gap to the offline optimum. Since T-TAMER shows that a bounded approximation to the offline optimum requires recall, Section 2 now cites this result, and we evaluate a variant with recall in our cascades (Appendix C.6). With recall, a third model changes accuracy by −0.21 to +0.35 pp, so the conclusions of Section 4.3 carry over.

W2. Assumptions. Assumption 3.1 now states the regularity conditions in the main text, directly before Theorem 3.1.

W3. Related work. Section 2 has a new paragraph on early exit and cascaded inference. It covers earlier cascades in vision and NLP, exits in vision models, BERT-style encoders and autoregressive language models, and serving systems such as Apparate.

We hope these changes address your concerns. If they do, we would be grateful if you would consider updating your score. We are happy to clarify anything further.

I thank the authors for their revisions, but remain unconvinced by the claim that two submodels outperform an arbitrary number of submodels, as Reviewer sL7X pointed out. The claim’s scope remains unclear: diminishing marginal returns do not rule out benefits from additional submodels in downstream applications. The mathematical model also relies on restrictive assumptions that limit its practical relevance.

Based on these facts, I maintain my current score.

AuthorsOpenReview ↗

Clarifying the scope of our claim

Thank you for the follow-up. We respond briefly to each point.

Scope of the claim. We agree that additional models can help, and the paper characterizes when. Equation 9 gives a necessary condition for a three-model cascade to improve on mixtures of two-model cascades at the same budget. Compared with such a mixture, the three-model cascade uses the middle model's score to select which queries escalate, and it also pays for the middle model on those queries. The accuracy this selection adds over escalating a random set of the same size must exceed the value of that extra cost. By Proposition 3.2, this accuracy gain, averaged across thresholds, increases with the score's AUROC for the current model's correctness and decreases with its AUROC for downstream correctness. Section 4.4 tests this relationship with synthetic scores. Raising own-correctness AUROC to 0.9 yields gains of +0.32 to +1.87 pp from a third model, and shifting the same signal to difficulty shared across models removes these gains. With the nine observed deferral scores, a third model improves on two at a small share of budget positions (Table 2, Appendix C.3). We do not claim that two models outperform an arbitrary number of models.

Model pool. In response to Reviewer sL7X, the revision adds two larger reasoning models and a leave-one-model-out analysis. The S3 − S2 difference stays between −0.33 and +0.68 pp across all ten exclusions (Appendix C.4).

Assumptions. The depth comparison in Section 4.3 uses exhaustive search over sequences and thresholds, so it does not rely on the regularity conditions of Theorem 3.1. The revision also tests relaxations of the policy-class restrictions raised in your review. Allowing a third model changes accuracy by −0.07 to +0.22 pp under cascade routing, −0.09 to +0.71 pp under history-dependent thresholds, and −0.21 to +0.35 pp under recall (Table 3, Appendix C.6). If a specific assumption remains a concern, we would be glad to address it.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: summary of revisions

We thank the reviewers for their careful and constructive reviews. The reviews asked whether our finding depends on the model pool, whether it depends on the policy class, and whether the theory explains it. We summarize below how the revision addresses each of these questions, and we respond to the remaining points in the individual responses.

1. Model pool. We added two reasoning models with 501B and 975B total parameters, and our findings hold with the ten-model pool across the nine deferral scores and five benchmarks (Table 2). Table 1 in the main text now reports parameters, reasoning status, accuracy and mean cost for every model on every benchmark, illustrating the substantial model diversity in our pool. To further evaluate whether the result depends on the composition of the pool, we ran a leave-one-model-out analysis. Across all ten exclusions and five benchmarks, the S3 − S2 difference lies between −0.33 and +0.68 pp (Appendix C.4).

2. Policy class. To test whether the result depends on the restrictions of our policy class, we ran the cascade routing approach of (Dekoninck et al., 2025), which chooses the next model per query, on the same ten models and five benchmarks. Allowing a third model changes the cascade-router's accuracy by −0.07 to +0.22 pp across datasets (Section 4.3, Table 3), suggesting that our results hold for this more general policy class. We additionally evaluate two other relaxations (history-dependent thresholds and recall, Appendix C.6) and find that our headline conclusion remains unchanged.

3. Clarifying the theory's connection to depth. Section 3.4 now derives a necessary condition for a three-model cascade to improve on two-model cascades, stated through the quantity in Proposition 3.2 (Equation 9, proved for any depth in Appendix A.6). The condition accounts for the main empirical finding. A score that reflects difficulty shared across models also predicts the next model's correctness, so the gain from escalation can be small even when the score predicts the current model's correctness well.

4. Clarifying what limits depth. Our results indicate that the limited gains from deeper cascades are attributable primarily to the confidence scores rather than to similarity in the models' errors. The synthetic score experiments in Section 4.4 hold every model's answers and costs fixed, which leaves the overlap in model errors unchanged, and vary only the scores. With an own-correctness AUROC of 0.9, the S3 − S2 difference ranges from +0.32 to +1.87 pp across the five benchmarks. The revision adds a diagnostic that replaces each score with the model's own correctness label, which has an AUROC of 1, and this increases the range to +0.92 to +3.80 pp (Appendix C.9). When we keep the own-correctness AUROC at 0.9 but shift the score signal to difficulty shared across models, the S3 − S2 difference falls to between −0.24 and −0.03 pp, as the condition in Equation 9 predicts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.