acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Escalation Signals for Optimal Language Model Routing

Abstract

Cascading a small language model into a large one requires a signal that predicts, before the large model is called, whether the call is needed. We study semantic entropy over sampled answers as that signal, and evaluate it on three tasks of increasing openness: templated arithmetic, GSM8K reasoning, and text-to-SQL. On GSM8K with a Gemma-3 1B/12B pair, semantic entropy predicts the small model's own failures at AUROC , beats a difficulty-calibrated baseline by , and converts into routing gains of up to accuracy over random escalation at matched cost. On synthetically generated chained arithmetic it adds nothing over a regular expression on the question text: a single-family benchmark makes difficulty a free surface feature of the question, so sampling reveals no more than the question already gives away. On text-to-SQL (Gemma-3 4B/12B, ), semantic entropy scores only AUROC pooled, below a free difficulty baseline () – but it is undefined, not weak, on the of items where the samples agree; on the remaining it scores . Splitting on this free-to-compute distinction and calibrating each half before recombining reaches a held-out AUROC of , our strongest result. A retrieval alternative – caching large-model outcomes and querying by embedding similarity – trades signal for cost: it appears to predict large-model success on arithmetic (), but a probe shows this is again difficulty lookup (), and it collapses to chance on GSM8K (), where the same probe returns . It costs always-large against for live sampling at its tuned budget. Framed as prediction rather than diagnosis, semantic entropy earns its keep only where the difficulty of a query cannot be read off the question itself. We report the diagnostics that produced these outcomes – a difficulty baseline, an oracle-headroom check, per-generation cost accounting, a retrieval-difficulty probe, and a regime split – against a measured AUROC noise floor of at these sample sizes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language models, semantic entropy, semantic routing, model and inference optimization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_oJNCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces semantic entropy for the cascading decision: run a small model several times, use semantic clustering to determine its consistency, and threshold that to see whether a bigger model needs to be run on the task. The authors evaluate their approach on several benchmarks, finding that it sometimes does not outperform a difficulty-decision baseline, frequently spends too much queries on the small model to determine consistency, but can sometimes slightly outperform the baseline.

Critical strengths

None

Critical weaknesses

The experiments presented in this paper are outdated and do not show a clear gain for the method. In particular:

  • The related work section is not sufficient to thoroughly describe a field that has a lot of work taking place in it. The paper also shows negligence when citing other works. For instance, Mallen et al. only occurs as a citation after their benchmark "PopQA" has already been featured prominently and the paper behind GSM8k is not cited (despite featuring even more prominently).
  • Several included subsections carry no value. For instance, section 4.7 only talks about why the benchmark did not prove suitable for the project, which has no additional value.
  • Writing, in particular in the experimental section, is very poor. As an example, L147-149 does not even write full English sentences.
  • The authors propose a method that is extremely similar to already proposed methods, with the only difference that they requery the same model instead of querying different models. Thus, at most the value of the work would be very incremental.
  • The experimental section is unclear, with no experiment providing a thorough and interesting analysis. Results are described unclearly and do not give an indication of why the method is useful.
  • The authors claim that the method itself is not a contribution, but the evaluation methodology. However, the experiments are poorly written, and do not provide value for the community.
  • The paper seems almost entirely LLM-written. Not a problem by itself, but it seems that no attempt was made to improve upon its output in any significant way.
Questions for authors

None

Reviewer_2rsyOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Using a cheap router to predict whether a given query needs larger language models is necessary to save cost and semantic entropy is a popular signal for prediction. This paper builds a pipeline to test: can semantic entropy reliably predict whether escalating a query to a large model would improve the answer? It distinguishes three evaluation targets: beneficial escalation, small-model failure, and large-model success. Through experiments on arithmetic, GSM8K, text-to-SQL, and additional settings, it examines how target choice, difficulty baselines, sampling costs, retrieval effects, and regime-specific calibration affect the interpretation of escalation signals. Its main claimed contribution is an evaluation methodology rather than a new routing signal.

Critical strengths
  1. The paper addresses a practically important problem: reducing the cost of large-model inference by identifying queries that can be handled by smaller models.
  2. Several observations are useful, particularly the distinction between detecting small-model errors and predicting whether escalation would help, as well as the need to account for the cost of repeated sampling. Reporting negative results and diagnostic controls is also valuable.
Critical weaknesses
  1. The paper needs a clearer central narrative and a more coherent presentation of its methodological contribution.
  • The practical motivation for routing is clear, but the motivation and structure of the proposed evaluation methodology are not sufficiently developed. The introduction announces six controls without clearly laying out what they are, which evaluation problems they address, and how the subsequent experiments support them. Although these controls are discussed later, an explicit framework at the outset would make the contribution much easier to understand.
  • Section 3 mixes existing escalation signals, evaluation targets, cost assumptions, and baselines, without sufficiently distinguishing established components from the authors’ methodological contributions and experimental protocol.
  • The results contain useful local observations, but their relationship to a unified argument is difficult to follow. The presentation often reads as a sequence of experimental iterations and corrections rather than a synthesis of what has been learned. It remains unclear which findings are specific to the evaluated settings and which provide actionable guidance for designing a routing system. Since methodological insights are the paper’s main claimed contribution, this lack of synthesis substantially limits its informativeness and practical usefulness.
  1. Several aspects of the experimental protocol need clarification before the strongest conclusions can be assessed reliably. In the text-to-SQL study, the final improvement combines regime splitting, signal selection, and supervised calibration. It is not sufficiently clear whether the baselines receive the same calibration data and evaluation protocol, making it difficult to isolate the contribution of the proposed combination. Table 7 also considers only 954 executable items out of 1,200 queries; the treatment of the remaining queries and the resulting full-query performance should be clarified.
Questions for authors

See Critical Weaknesses.

Reviewer_AuVFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies small-to-large LLM cascades, where a query is answered by a small model and escalated to a large model only when needed. It asks whether the semantic entropy of k samples from the small model, clustered by meaning (NLI entailment, or execution results for text-to-SQL), can predict before the large model is called whether escalation is worthwhile. The authors claim that semantic entropy reliably detects the small model’s own errors but only indirectly predicts whether the large model would fix them. They also argue that it is blind when the small model is consistently wrong (all samples agree), where token confidence must take over, and that it saves cost only with a small sample budget. Beyond the signal, they frame their main contribution as six evaluation controls. Validation uses Gemma-3 model pairs on four tasks. Entropy helps on GSM8K, ties with a simple difficulty feature on synthetic arithmetic, and works on text-to-SQL only after per-regime calibration; a retrieval router turns out to be a difficulty lookup.

Critical strengths

The paper reports its negative results candidly, and the retrieval-difficulty probe (Section 4.5) is a simple, reusable check for whether a nearest-neighbor router is looking up difficulty.

Critical weaknesses
  1. The contribution is unclear and overstated. The paper does not state a clear research question or conclusion. The abstract and conclusion present semantic-entropy numbers (0.871, +0.093, 0.907) as the main results, while the introduction says the contribution is the evaluation methodology. The "strongest result" of 0.907 relies on label-based calibration and is no longer the zero-label signal under study. The two "portable" findings in the conclusion ("cost and signal quality are separable barriers"; "a signal's behavior depends on benchmark structure") are too general to be testable.

  2. The experimental design does not support the claims. Each task uses a different model pair (1B/12B, 4B/12B, 270M/4B), all from one family, and the positive result rests on about 400 GSM8K questions. There is no experimental setup section (prompts, answer extraction, grading and sampling parameters are not described), and no comparison against any existing router, so the paper cannot show that its controls change any conclusion in practice.

Questions for authors
  1. What is the research question, and what conclusion is the experimental program designed to establish? The abstract and conclusion lead with semantic-entropy results, while the introduction says the contribution is the evaluation methodology.
  2. Do the findings generalize? The only positive result comes from one benchmark (GSM8K) and one model family (Gemma-3), and it uses only 250 + 150 of the 1,319 GSM8K test questions, with no description of how they were selected. Why was the full test set not used? Do the GSM8K results, and the claim that the six controls change conclusions, hold on the full test set, with at least one other model family (e.g., Qwen or Llama), and on a second open-ended reasoning benchmark? Without this, the general methodological claims rest on a very narrow experimental base.
  3. What is new relative to existing consistency-based cascades? Have the six controls been shown to change the conclusions of any published routing study?
  4. How does the method compare with existing routers and with simple majority-vote agreement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.