- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether increasing the number of models in an LLM cascade improves the accuracy–cost tradeoff. The authors derive optimality conditions for confidence thresholds, introduce an AUROC-based characterization of escalation utility, and provide empirical evidence that conventional confidence-based cascades rarely benefit from more than two models.
Critical strengths
- I think the motivation is quite clear and the question is interesting and relevant to the field.
- The distinction between predicting the current model's correctness and identifying beneficial escalation is interesting.
Critical weaknesses
-
I am not fully convinced by the theoretical contribution. Theorem 3.1 gives optimality conditions for threshold selection, while Proposition 3.2 studies the benefit of confidence-based escalation. However, neither directly explains when adding more models is actually beneficial.
-
I have some concerns about the generalizability of the empirical results. The benefit of deeper cascades strongly depends on the task distribution and the models being used. I am not sure what conclusions we can draw beyond the specific setting considered in the paper.
-
The novelty compared to existing learning-to-defer work is not entirely clear to me. In particular, the idea that confidence in the current model does not necessarily indicate whether deferring is beneficial is already well established.
-
I am not fully convinced that the synthetic experiments explain why deeper cascades provide limited improvements. Changing the confidence scores also changes their correlations across models, making it difficult to identify which factor is actually responsible for the results.
Questions for authors
-
Could the authors clarify whether the limited gains from deeper cascades come mainly from the models making similar errors or from the confidence scores not being informative enough?
-
Is there a more direct connection between Proposition 3.2 and the benefit of adding more models? I am not sure how the theoretical result explains the main empirical findings.
-
Would the conclusions change if the deferral scores estimated the benefit of the entire downstream cascade rather than just the next model?
7 comments
Response to Reviewer FQim (2/2): Controls, novelty and generalizability (W2, W3, W4, Q3)
W4. Score quality versus score correlation. The synthetic score experiments for own-correctness AUROC are not subject to this concern, because they hold the observed cross-model score correlation fixed while AUROC varies. However, we agree this was a potential confound in the shared-difficulty experiment, and we have since added an ablation to address it (Appendix C.10). Specifically, we group queries with the same correctness outcomes across all ten models, and within each group we shuffle each model's scores across queries, using a separate shuffle for each model. This leaves each score's relation to every model's correctness unchanged and makes different models' scores independent among queries with the same correctness outcomes. Across the five benchmarks, shifting the score signal to shared difficulty at AUROC 0.9 lowers the difference by 0.44 to 2.01 pp, and the shuffle changes this decline by at most 0.13 pp, suggesting that our finding is robust to correlation between models' scores beyond what correctness explains.
Q3. Scores for the entire downstream cascade. Thank you for the question. The theory already covers this case, since in Theorem 3.1 and in Proposition 3.2 measure the quality of the answer returned by the entire remaining cascade. To test the question empirically, we added an experiment in which we train scores on the benefit of the entire downstream cascade (Appendix C.12). For a three-model cascade , the first-stage score predicts the correctness of the answer returned by the full cascade minus that of . Across the five benchmarks, the difference lies between −0.34 and +0.47 pp, within the range of the nine scorers in Table 12, and does not exceed that of a matched correctness-trained score on any benchmark (Table 33). Therefore, the conclusions do not change when the scores estimate the benefit of the entire downstream cascade.
W3. Novelty relative to learning to defer. We agree that the qualitative point is established in the learning to defer literature, and Section 2 now states this explicitly. We contribute an identity that quantifies the point for an arbitrary downstream cascade, a condition that turns it into a statement about depth, and, to our knowledge, the first comparison of depth classes in LLM cascades with jointly optimized sequences and thresholds.
W2. Generalizability. We added reasoning models of 501B and 975B total parameters and repeated every analysis with the ten-model pool. We also added a table of the pool (Table 1), which shows substantial variation in accuracy and cost across models. The five benchmarks cover mathematics, multiple-choice questions, factual recall and code generation. Across them, the accuracy of the best model ranges from 41% to 95% and that of the worst model from 4% to 55%. The difference is stable when any one model is removed via a leave-one-out ablation (−0.33 to +0.68, Appendix C.4) and under nine alternative price spacings (−0.31 to +0.75, Section 4.4). The same pattern holds for cascade routing (Dekoninck et al., 2025), a relaxed policy class that chooses the next model per query, where allowing a third model changes accuracy by −0.07 to +0.22 pp (Table 3). The paper lists the pool, scores and benchmarks as limitations. Beyond this setting, the condition above and Section 4.4 indicate when the conclusion should change.
We hope these changes address your concerns. If they do, we would be grateful if you would consider updating your score, and we are glad to clarify further.
Follow-up on our response
Thank you again for your review. As the discussion period closes on November 18, we wanted to follow up on our response. In response to your concerns, the revised manuscript adds the following:
- A necessary condition connecting Proposition 3.2 to the benefit of a third model (Section 3.4, Equation 9)
- A diagnostic that replaces each score with the model's own correctness label (AUROC of 1), separating the role of the scores from the overlap in model errors (Appendix C.9)
- An ablation that removes correlation between models' scores beyond what correctness explains (Appendix C.10)
- Scores trained on the benefit of the entire downstream cascade (Appendix C.12)
- A discussion of our contribution relative to the learning-to-defer literature (Section 2)
- Inclusion of two larger reasoning models (501B and 975B) and a leave-one-model-out analysis of dependence on the 10-model pool (Appendix C.4)
If any of your concerns remain, we would be glad to address them before the discussion closes.
Response to Reviewer FQim (1/2): Theory and depth (W1, Q1, Q2)
Thank you for the precise questions. Each of them led to a change in the paper.
W1, Q2. Connecting the theory to depth. Section 3.4 now derives a necessary condition for a three-model cascade to improve on two-model cascades, stated through the quantity in Proposition 3.2 (Equation 9, proved for any depth in Appendix A.6).
Consider a cascade (M1,M2,M3) in which a fraction t of the queries reaching M2 escalate. We compare it with a mixture that sends a random fraction t of queries to the cascade (M1,M3) and the rest to (M1,M2), each with the threshold that the three-model cascade applies to M1. The accuracy of the three-model cascade exceeds that of this mixture by a2(t)−aˉ2(t), the gain from using the score of M2 to choose which queries escalate. Its expected cost exceeds that of the mixture by a term κ(t), because it also calls M2 on the queries that escalate. It can therefore improve on every mixture of two-model cascades at the same budget only if
a2(t)−aˉ2(t)>λκ(t),
where λ is the shadow price of the budget among those mixtures. The gain must exceed the value of the added cost.
By Proposition 3.2, the average of the left side over t rises with the AUROC of the score of M2 for the correctness of M2 and falls with its AUROC for the correctness of M3. A score that reflects difficulty shared by both models raises both AUROCs, so the gain can be small even when it predicts the correctness of M2 well. This accounts for the main empirical finding. Section 4.4 tests both predictions of the condition by varying the two AUROCs separately. Raising own-correctness AUROC to 0.9 makes S3 improve on S2 on every benchmark, and shifting the same signal to difficulty shared across models removes the gain.
Q1. Similar errors or uninformative scores? Our results indicate that the limited gains from deeper cascades are attributable primarily to the confidence scores rather than to similarity in the models' errors. Two experiments hold every model's answers and costs fixed, which leaves the overlap in model errors unchanged, and vary only the scores (Section 4.4, Appendix C.9). All ranges below are across the five datasets. With the observed scores, the S3−S2 difference ranges from −0.28 to +0.61 pp. Raising own-correctness AUROC to 0.9 increases this range to +0.32 to +1.87 pp. Replacing each score with the model's own correctness label, a diagnostic with an AUROC of 1, increases it to +0.92 to +3.80 pp. These results show that the models' errors differ enough for cascades of up to three models to improve on two-model cascades, provided the scores can identify the queries that benefit from escalation, but the observed scores lack this ability. Consistent with the necessary condition in Equation 9 (Section 3.4), the gain at AUROC 0.9 disappears when the score signal reflects difficulty shared across models. The limiting factor is therefore the extent to which a score distinguishes a model's own errors from errors that the next model would also make.
Continued in another comment...