- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Crisp is a conformal controller that routes and cascades a given prompt across models of varying sizes and families. It operates using two gates: a routing gate that decides whether a model is eligible to answer a prompt, and a stopping gate that decides whether a generated response should be accepted or escalated to another model. The paper defines a prompt statistic on a given input prompt r(x) and an output statistic on the model's response to the prompt . Both are based on EG-NLL, which computes the mean negative log-likelihood over the fraction fraction \rho of the tokens with the highest predictive entropy. For each model k and task t, the datapoints where the model answered incorrectly are constructed as a failure set . r(x) and are computed over this failure set and are used as reference distribution. Given a new prompt or output with score v, CRISP computes a lower-tail conformal rank $ \widehat p_b(v; v_{1:m}) = \frac{ 1 + \sum_{j=1}^{m} \mathbf{1}[v_j \le v] }{ m+1 } $ A low rank implies that it is more favorable than the previously observed failures, making it more likely to pass the corresponding gate. The routing gate uses the rank of r(x) to determine whether a model should be considered, while the stopping gate uses the rank of to determine whether the generated answer should be returned or the query should be escalated. Under exchangeability of calibration and test failures, the paper proves that the probability that a failing model passes either the routing gate or the stopping gate is bounded by the chosen alpha , but these are failure-pass guarantees, not guarantees on accuracy, selective risk, or the final adaptive cascade.
CRISP was evaluated using 7 models and 4 datasets (1) 200 calibration, 200 test samples yielding 81.5% accuracy (2) 400 calibration and 400 test samples yield 86% accuracy . Compared against UCR, FrugalGPT, and RouterBench-based baselines, CRISP reports the best accuracy-cost trade-off and the lowest proxy cost at its reported operating point. The main conceptual contribution is calibrating against observed failures rather than raw confidence. A key limitation, however, is that calibration is performed only on observed failures, so the quality of the conformal ranking depends heavily on whether those calibration failures are representative of future failure modes. The hand-crafted prompt and output statistics are weakly justified.
Critical strengths
- The paper introduces a clean failure-centric view of routing: instead of calibrating against confidence or correctness on all examples, it calibrates against a model's observed failures, which is a more natural target for reliability-oriented routing.
- The finite-sample analysis is practically useful. The paper makes explicit how the number of calibration failures limits the attainable conformal thresholds and highlights when a conformal gate becomes infeasible due to insufficient failure data.
- The use of conformal ranks instead of raw confidence values is appealing because decision thresholds are defined relative to observed failures rather than absolute score magnitudes, which are often poorly calibrated and difficult to compare across models.
Critical weaknesses
-
Limited evidence that the stopping gate is useful. Table 3(a) suggests that the stopping gate contributes only a marginal improvement over the routing-only variant despite incurring additional model calls and cost. Since the stopping gate is a key contribution of the paper, the authors should clearly demonstrate its efficacy in identifying and rejecting incorrect outputs at a sufficient rate to justify its added complexity. As presented, the results do not clearly establish that the stopping stage contributes meaningfully beyond routing.
-
The proposed scoring functions rely on a small number of hand-crafted features and heuristics whose relationship to model failure is not well justified. For example, prompt length, digit counts, and code indicators are only rough proxies for difficulty, and there is little reason to expect them to generalize across tasks, domains, or model families. A short proof-based question may be substantially harder than a long arithmetic problem, yet r(x) may assign the opposite ordering. The paper does not provide empirical details of accuracy, the false positives, false negatives of the the input and output statistics. Nor does it provide sufficient evidence that these statistics capture failure likelihood beyond the specific benchmarks used. It also does not compare the proposed statistics against strong failure-prediction baselines such as entropy, confidence margins, self-consistency (e.g., agreement across 32/64 generations), or learned difficulty predictors.
-
Tables 2 and 9 mainly show which model was ultimately selected but do not characterize the difficulty of individual examples or whether cheaper models could also have solved them. A more informative analysis would categorize examples by the smallest model capable of solving them (e.g., easy: 3B, medium: 8B, hard: 32B, very hard: 72B). The current results reveal that routing is heavily dominated by Qwen-32B, suggesting that the effective portfolio may be much smaller than the advertised seven-model ensemble.
-
CRISP's guarantees are only as representative as its failure reference sets. If calibration failures are concentrated in a subset of failure modes or disproportionately difficult examples, then moderately difficult or qualitatively different failures may receive favorable ranks and pass the gate despite being genuine errors. The method controls ranks relative to observed failures, not relative to all possible failures. Consequently, unseen failure modes, shifted distributions, or subtle errors may not be adequately protected.
-
The paper reports 200 or 400 calibration examples overall but does not report how many calibration failures each model-task pair actually contributes. Since conformal calibration is performed only on failures, the relevant sample size is mk,tm_{k,t}mk,t, not the overall calibration set size. Without reporting these values, it is impossible to determine which routing and stopping gates were meaningfully calibrated and which operated with very limited failure data.
-
The experiments use only 200/200 or 400/400 calibration-test examples spread across four domains. Given seven models and task-conditional conformal calibration, these sample sizes appear too small to robustly characterize routing behavior, estimate failure distributions, or evaluate the stability of the reported gains. Repeated random splits and larger evaluation sets would be needed to establish robustness.
-
The method assumes that the task type ttt is known when constructing task-specific failure sets and calibrating ranks. In realistic deployments, task labels would need to be predicted by a classifier. Misclassification could route examples to the wrong calibration distribution, potentially invalidating the conformal ranking procedure. The paper does not evaluate robustness to task-classification errors.
-
The primary contribution is statistical control of failure-pass events, yet the paper does not report the actual held-out failure-pass rates for every model-task pair. Without an audit of these quantities, it is difficult to verify that the implementation empirically satisfies the guarantees that motivate the method.
Questions for authors
- Did you consider other prompt and output statistic formulations?
- During the experiments, how many times was the Dolphin model ever chosen? If not, You can try Muse Glimmer 30B, Gemma 4 12B Unified etc. instead
- It is not clear if you actually ran all the baselines or used their numbers from previously published works. Can you clarify the exact setup in Section 4.3. Which implementations were you able to reproduce? Also, what does RouterBench imply here? It would also be good to evaluate CRISP on RouterBench benchmark to see how it fares