acceptodds
Under review as a conference paper at ICLR 2027

CRISP: Conformal Routing and Interruption with Statistical Protection for LLM Cascades

Abstract

Reliable deployment of large language model (LLM) cascades requires deciding both which model to query and whether its realized output should be returned. We introduce CRISP (Conformal Routing and Interruption with Statistical Protection), which calibrates prompt- and output-side scores against labeled model failures. Under exchangeability, CRISP gives finite-sample, model- and task-conditional bounds on two events: an actually failing model entering the eligible set and an incorrect output passing the stopping gate. These are fixed-model failure-pass guarantees, not a bound of the same level on selective error or on the adaptively selected portfolio. Across seven open-weight models and four reasoning benchmarks, CRISP attains 81.5% accuracy and 99.5% coverage at a 0.0794 relative token-cost proxy on a fixed 200-example test split. Together, the formulation and evaluation show how failure-conditional calibration makes both routing and stopping decisions explicit and auditable in heterogeneous LLM cascades.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, LLM cascades, model routing, uncertainty quantification, cost-efficient inference, adaptive inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_x1jEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Crisp is a conformal controller that routes and cascades a given prompt across models of varying sizes and families. It operates using two gates: a routing gate that decides whether a model is eligible to answer a prompt, and a stopping gate that decides whether a generated response should be accepted or escalated to another model. The paper defines a prompt statistic on a given input prompt r(x) and an output statistic on the model's response to the prompt sk(x,y^)s_{k}(x, \hat{y}). Both are based on EG-NLL, which computes the mean negative log-likelihood over the fraction fraction \rho of the tokens with the highest predictive entropy. For each model k and task t, the datapoints where the model answered incorrectly are constructed as a failure set Fk,tF_{k,t}. r(x) and sk(x,y^)s_{k}(x, \hat{y}) are computed over this failure set and are used as reference distribution. Given a new prompt or output with score v, CRISP computes a lower-tail conformal rank $ \widehat p_b(v; v_{1:m}) = \frac{ 1 + \sum_{j=1}^{m} \mathbf{1}[v_j \le v] }{ m+1 } $ A low rank implies that it is more favorable than the previously observed failures, making it more likely to pass the corresponding gate. The routing gate uses the rank of r(x) to determine whether a model should be considered, while the stopping gate uses the rank of sk(x,y^)s_{k}(x, \hat{y}) to determine whether the generated answer should be returned or the query should be escalated. Under exchangeability of calibration and test failures, the paper proves that the probability that a failing model passes either the routing gate or the stopping gate is bounded by the chosen alpha α\alpha, but these are failure-pass guarantees, not guarantees on accuracy, selective risk, or the final adaptive cascade.

CRISP was evaluated using 7 models and 4 datasets (1) 200 calibration, 200 test samples yielding 81.5% accuracy (2) 400 calibration and 400 test samples yield 86% accuracy . Compared against UCR, FrugalGPT, and RouterBench-based baselines, CRISP reports the best accuracy-cost trade-off and the lowest proxy cost at its reported operating point. The main conceptual contribution is calibrating against observed failures rather than raw confidence. A key limitation, however, is that calibration is performed only on observed failures, so the quality of the conformal ranking depends heavily on whether those calibration failures are representative of future failure modes. The hand-crafted prompt and output statistics are weakly justified.

Critical strengths
  1. The paper introduces a clean failure-centric view of routing: instead of calibrating against confidence or correctness on all examples, it calibrates against a model's observed failures, which is a more natural target for reliability-oriented routing.
  2. The finite-sample analysis is practically useful. The paper makes explicit how the number of calibration failures limits the attainable conformal thresholds and highlights when a conformal gate becomes infeasible due to insufficient failure data.
  3. The use of conformal ranks instead of raw confidence values is appealing because decision thresholds are defined relative to observed failures rather than absolute score magnitudes, which are often poorly calibrated and difficult to compare across models.
Critical weaknesses
  1. Limited evidence that the stopping gate is useful. Table 3(a) suggests that the stopping gate contributes only a marginal improvement over the routing-only variant despite incurring additional model calls and cost. Since the stopping gate is a key contribution of the paper, the authors should clearly demonstrate its efficacy in identifying and rejecting incorrect outputs at a sufficient rate to justify its added complexity. As presented, the results do not clearly establish that the stopping stage contributes meaningfully beyond routing.

  2. The proposed scoring functions rely on a small number of hand-crafted features and heuristics whose relationship to model failure is not well justified. For example, prompt length, digit counts, and code indicators are only rough proxies for difficulty, and there is little reason to expect them to generalize across tasks, domains, or model families. A short proof-based question may be substantially harder than a long arithmetic problem, yet r(x) may assign the opposite ordering. The paper does not provide empirical details of accuracy, the false positives, false negatives of the the input and output statistics. Nor does it provide sufficient evidence that these statistics capture failure likelihood beyond the specific benchmarks used. It also does not compare the proposed statistics against strong failure-prediction baselines such as entropy, confidence margins, self-consistency (e.g., agreement across 32/64 generations), or learned difficulty predictors.

  3. Tables 2 and 9 mainly show which model was ultimately selected but do not characterize the difficulty of individual examples or whether cheaper models could also have solved them. A more informative analysis would categorize examples by the smallest model capable of solving them (e.g., easy: 3B, medium: 8B, hard: 32B, very hard: 72B). The current results reveal that routing is heavily dominated by Qwen-32B, suggesting that the effective portfolio may be much smaller than the advertised seven-model ensemble.

  4. CRISP's guarantees are only as representative as its failure reference sets. If calibration failures are concentrated in a subset of failure modes or disproportionately difficult examples, then moderately difficult or qualitatively different failures may receive favorable ranks and pass the gate despite being genuine errors. The method controls ranks relative to observed failures, not relative to all possible failures. Consequently, unseen failure modes, shifted distributions, or subtle errors may not be adequately protected.

  5. The paper reports 200 or 400 calibration examples overall but does not report how many calibration failures each model-task pair actually contributes. Since conformal calibration is performed only on failures, the relevant sample size is mk,tm_{k,t}mk,t​, not the overall calibration set size. Without reporting these values, it is impossible to determine which routing and stopping gates were meaningfully calibrated and which operated with very limited failure data.

  6. The experiments use only 200/200 or 400/400 calibration-test examples spread across four domains. Given seven models and task-conditional conformal calibration, these sample sizes appear too small to robustly characterize routing behavior, estimate failure distributions, or evaluate the stability of the reported gains. Repeated random splits and larger evaluation sets would be needed to establish robustness.

  7. The method assumes that the task type ttt is known when constructing task-specific failure sets and calibrating ranks. In realistic deployments, task labels would need to be predicted by a classifier. Misclassification could route examples to the wrong calibration distribution, potentially invalidating the conformal ranking procedure. The paper does not evaluate robustness to task-classification errors.

  8. The primary contribution is statistical control of failure-pass events, yet the paper does not report the actual held-out failure-pass rates for every model-task pair. Without an audit of these quantities, it is difficult to verify that the implementation empirically satisfies the guarantees that motivate the method.

Questions for authors
  1. Did you consider other prompt and output statistic formulations?
  2. During the experiments, how many times was the Dolphin model ever chosen? If not, You can try Muse Glimmer 30B, Gemma 4 12B Unified etc. instead
  3. It is not clear if you actually ran all the baselines or used their numbers from previously published works. Can you clarify the exact setup in Section 4.3. Which implementations were you able to reproduce? Also, what does RouterBench imply here? It would also be good to evaluate CRISP on RouterBench benchmark to see how it fares
Reviewer_synoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a conformal method for routing prompts between different models. Gates are calibrated based on model failures. From this the method gives finite-sample bounds for how often a wrong answer gets through. They empirically test on seven open-weight models and four benchmark suites.

Critical strengths

The paper provides a sensible mechanism for calibrated cascading. The paper is upfront about the extent of their guarantees.

Critical weaknesses

The paper provides limited novelty. The theory uses standard split conformal and is very close to existing works in the literature such as CP-Router, Conformal Arbitrage, Cascaded Selective Evaluation and other papers cited in those works. The methodology in this paper feels like a subset of those contributions as far as I can tell.

Questions for authors

How would you distinguish your work from the literature more clearly?

Reviewer_ySTWOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a hybrid routing/cascading strategy that splits the mathematical job of model selection into two halves: a routing function r(x) measuring prompt uncertainty, and a stopping function s(\hat{y}, x) measuring output uncertainty.

The authors do not claim higher performance compared to the other methods the authors measure against. Quote: "[Figure 2] reports the available within-artifact comparison rather than establishing dominance over these systems."

On the whole, the paper is hard to understand. The language is consistently non-standard, and it smacks of AI-generated writing.

Overall, the paper comes across more as a case study/project write-up than a scientific contribution expanding the boundaries of our knowledge.

Critical strengths
  • Splitting the model selection decision into a prompt-dependent component r(x) and an output-dependent component s(y,x) is interesting.
  • Evaluating uncertainty by considering the high-entropy tokens is an interesting approach, though not particular novel and I am almost certain has been proposed elsewhere in the literature.
Critical weaknesses
  • The listed contributions in the introduction, taken at face value, would not justify acceptance to ICLR (not novel enough).
  • Superiority over existing methods is neither claimed nor apparent.
  • The paper is generally written in a convoluted style using non-standard terminology; strong signs on unedited AI writing make the reader question whether there is a human intellect behind the paper (examples: "failure-pass events", "within-artifact comparison", "explicit analysis of rank resolution").
Questions for authors
  • Could you clarify Figure 1d) on how CRISP works? The arrows pointing in different directions leave the overall control flow unclear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.