- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Triage, a token-level cascade in which a small model generates text and hands individual high-entropy tokens to a large model. It also proposes utility-aware entropy calibration, which fine-tunes the small model to raise its entropy where only the large model is correct and lower it where both models are wrong
Critical strengths
The idea that deferral should depend on whether the large model can actually fix the error, not only on the small model’s uncertainty, is sensible and clearly motivated by the joint error analysis.
Critical weaknesses
- [Most important]The paper completely ignores speculative decoding and closely related small-large model decoding methods, which is a major omission for a paper on token-level collaboration between a small and a large model. Speculative decoding, https://arxiv.org/abs/2211.17192 and https://arxiv.org/abs/2302.01318, already uses a small model to generate tokens and a large model to step in, while preserving the large model’s output quality. Big Little Decoder, https://arxiv.org/abs/2302.07863, is almost the same as Triage’s inference procedure: the small model generates tokens and falls back to the large model whenever its confidence drops below a threshold, with an additional rollback step. Faster Cascades via Speculative Decoding, https://arxiv.org/abs/2405.19261, directly combines cascades with speculative decoding at the token level. None of these are cited or compared, so the token-level cascade itself cannot be counted as a contribution, and the latency comparison is missing its most important baselines.
- The remaining new component, utility-aware entropy calibration, brings only a small gain. Compared with the same token-level cascade without calibration, the average AUC-DF improves only from 0.4821 to 0.5015 on Llama and from 0.5714 to 0.5800 on Qwen, and Triage is not the best method on Qwen GSM8K or Qwen MMLU. Table 1 reports no variance across runs, so it is unclear whether these small differences are meaningful.
- The method requires task-specific training of both models, including supervised fine-tuning and a calibration stage that uses ground-truth answers and large-model predictions, while many cascade and speculative decoding methods need no training. The experiments are also limited to models of 8B parameters or smaller, so it is unclear whether the approach remains useful with stronger and larger models.
Questions for authors
- How does Triage compare with speculative decoding and Big Little Decoder in terms of latency and accuracy, given that speculative decoding can reach the large model’s accuracy with reduced latency?
- Can you report variance across runs for Table 1, to show whether the gain from calibration over the uncalibrated token-level cascade is significant?
- Does a calibrated small model trained on one task still work as a good deferral signal on other tasks, or must calibration be repeated for every task?