acceptodds
Withdrawn from ICLR 2027

Triage: Token-Level Language Model Cascades with Utility-Aware entropy Calibration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Large language models (LLMs) have achieved strong performance on complex reasoning tasks, but their growing scale leads to substantial inference costs. Language model cascades mitigate this issue by first using a small language model (SLM) for low-cost generation and selectively invoking a large language model (LLM), aiming to achieve a trade-off between model performance and inference cost. However, existing cascade systems mostly operate at the sequence level and rely on SLM uncertainty as the deferral signal, leading to coarse-grained decisions and a mismatch between uncertainty and deferral utility. In this paper, we propose Triage, a token-level cascade framework with utility-aware entropy calibration (UEC). Triage performs token-level deferral decisions during autoregressive generation and jointly considers the prediction outcomes of the SLM and LLM to calibrate SLM entropy, so that it no longer only reflects model uncertainty, but also better captures token-level deferral utility. Experiments on two model pairs from different families and scales across four reasoning datasets show that Triage achieves a superior cost–performance trade-off over various cascade baselines while preserving the downstream task performance of the SLM.

Primary area:
generative models
Keywords:
Large language model, Model cascade

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3f1qOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Triage, a token-level cascade in which a small model generates text and hands individual high-entropy tokens to a large model. It also proposes utility-aware entropy calibration, which fine-tunes the small model to raise its entropy where only the large model is correct and lower it where both models are wrong

Critical strengths

The idea that deferral should depend on whether the large model can actually fix the error, not only on the small model’s uncertainty, is sensible and clearly motivated by the joint error analysis.

Critical weaknesses
  • [Most important]The paper completely ignores speculative decoding and closely related small-large model decoding methods, which is a major omission for a paper on token-level collaboration between a small and a large model. Speculative decoding, https://arxiv.org/abs/2211.17192 and https://arxiv.org/abs/2302.01318, already uses a small model to generate tokens and a large model to step in, while preserving the large model’s output quality. Big Little Decoder, https://arxiv.org/abs/2302.07863, is almost the same as Triage’s inference procedure: the small model generates tokens and falls back to the large model whenever its confidence drops below a threshold, with an additional rollback step. Faster Cascades via Speculative Decoding, https://arxiv.org/abs/2405.19261, directly combines cascades with speculative decoding at the token level. None of these are cited or compared, so the token-level cascade itself cannot be counted as a contribution, and the latency comparison is missing its most important baselines.
  • The remaining new component, utility-aware entropy calibration, brings only a small gain. Compared with the same token-level cascade without calibration, the average AUC-DF improves only from 0.4821 to 0.5015 on Llama and from 0.5714 to 0.5800 on Qwen, and Triage is not the best method on Qwen GSM8K or Qwen MMLU. Table 1 reports no variance across runs, so it is unclear whether these small differences are meaningful.
  • The method requires task-specific training of both models, including supervised fine-tuning and a calibration stage that uses ground-truth answers and large-model predictions, while many cascade and speculative decoding methods need no training. The experiments are also limited to models of 8B parameters or smaller, so it is unclear whether the approach remains useful with stronger and larger models.
Questions for authors
  • How does Triage compare with speculative decoding and Big Little Decoder in terms of latency and accuracy, given that speculative decoding can reach the large model’s accuracy with reduced latency?
  • Can you report variance across runs for Table 1, to show whether the gain from calibration over the uncalibrated token-level cascade is significant?
  • Does a calibrated small model trained on one task still work as a good deferral signal on other tasks, or must calibration be repeated for every task?
Reviewer_pCyGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Triage decides token by token when to call a larger language model. It fine-tunes the smaller model to assign higher entropy to errors the larger model can correct, and lower entropy where calling it would not help. Experiments with two model pairs on four datasets show a better average trade-off between task performance and large-model usage than the tested baselines.

Critical strengths

1.The method targets unnecessary large-model calls by calibrating entropy to reflect the benefit of deferral, without adding a separate routing head. 2.The inference procedure is simple, using calibrated entropy for token-level decisions without an additional routing head. 3.The method addresses a practical issue: uncertainty in the small model does not always mean that the larger model can help. Distinguishing these cases can reduce unnecessary calls. 4.The ablations support the contributions of both token-level deferral and UEC. Entropy distributions and useful-deferral precision provide further insight into how the method works.

Critical weaknesses

1.The main new element is the entropy objective. Token-level collaboration and joint-correctness routing labels already exist, so the distinction from Co-LLM and Gatekeeper needs to be clearer. 2.UEC changes the distribution used for generation, which may affect the small model’s capabilities. The Llama–GSM8K results are reassuring, but do not rule out trade-offs elsewhere. 3.UEC changes both the small model’s generation ability and its deferral behavior, but the experiments do not separate their contributions. On Llama–GSM8K, standalone accuracy improves while UDP gains are inconsistent, leaving the contribution of better routing to AUC-DF unclear.

Questions for authors

1.How well do the A/B/C labels predict the benefit of deferral under prefixes generated by the mixed system? Could valid alternative continuations be mislabeled as errors because they do not match the reference token? 2. Can the authors compare continued SFT, LAL_A-only training, and full UEC from the same checkpoint under identical training settings? This would help separate the effects of additional training from those of entropy calibration. 3. Could the authors compare UEC with continued SFT, LAL_A-only training, and a separate routing head using the same utility labels, under matched starting checkpoints and training settings, to clarify where the gains come from?

Reviewer_e5DmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper combines a small and a large LM token by token, training the small model’s entropy to signal when calling the larger model would help. It reports better accuracy-cost tradeoffs across two model pairs and four benchmarks

Critical strengths
  • Simple idea with a clear motivation: uncertainty alone doesn’t tell you whether the larger model can actually help
  • Useful ablations and comparisons, including actual latency measurements and a control for longer SFT training
Critical weaknesses
  • My main concern is how much the entropy story depends on the domains tested. The initial analysis is entirely on math, and the extended-generation benchmarks are math/code. [1] clearly shows how numeric tokens can drive apparent benefits of entropy-based selection, so this is a concrete bias. Code also has strongly constrained syntax and token choices, which makes it important to check what entropy is actually capturing there. MMLU doesn't really resolve this because since it only requires one answer letter to be generated. Even with learned calibration, I'd want a non-math/code long-form benchmark and a breakdown by token type.
  • A different next token doesn't imply a wrong continuation. Both models are evaluated against one reference under the ground-truth prefix, but the small model might simply take another valid path. This makes the "useful" versus "useless" deferral labels questionable: matching the reference token isn't the same as improving the final answer. The downstream gains are still meaningful, but the token-level analysis doesn't establish the utility they claim.
  • The random baseline needs clarification. The "curves" are straight lines (e.g. Fig 3), and the Llama/GSM8K random AUC exactly matches the midpoint of the two standalone accuracies. It looks like interpolation but random token-level switching should actually be run because it changes subsequent prefixes. Multiple routing seeds and confidence intervals would also help judge how much different methods really work.
  • I think the inference-cost ablation is still incomplete. They retain the LLM cache but each intervention requires processing the missing prefix (i.e. tokens produced by the small model). Deferral rate therefore doesn't directly measure saved computation. Figure 4 helps but average seconds per sample don't tell us how this behaves with batching, concurrent requests, throughput, or latency spikes when the LLM takes over.

Minor:

  • the authors sweep the entropy threshold, but it's unclear how they select it on held-out data to meet a deployment budget
  • CoLLM is already a learned-deferral baseline, but a simple routing head trained with the same utility labels would be a cleaner way to test whether modifying vocabulary entropy is necessary.
Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.