- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper measures token-level sufficient compute by identifying the cheapest model that reproduces each reference token under teacher forcing. It studies compute concentration, cross-family agreement, continuation behavior, and oracle-guided inference allocation.
Critical strengths
S1. The cross-family analysis provides useful evidence of shared reproduction difficulty beyond the evaluated surprisal controls.
S2. The continuation experiments clearly distinguish local token reproduction from reliable generation. The robustness checks and transparent discussion of limitations are appreciated.
Critical weaknesses
W1. The measurement requires known reference prefixes, while the allocation experiments additionally require advance token-agreement information. These are unavailable during ordinary inference. An offline measurement study can still be valuable, but I am not yet convinced that the findings provide sufficiently new explanatory insights to compensate for this limited applicability.
W2. Limited novelty. Duo-LLM[1] already studies oracle compute allocation and token difficulty; FS-GEN[2] analyzes sparse large-model intervention and prediction uncertainty; and FlexiDepth[3] examines token-dependent computational demand. The cross-family measurement extends this literature, but the conceptual advance beyond these established observations appears incremental.
[1]. Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Models
[2]. Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding
[3]. Adaptive Layer-skipping in Pre-trained LLMs
Questions for authors
Standard speculative decoding already exploits small/large-model agreement without advance reference access. Although the paper compares oracle-guided drafting against fixed and adaptive drafting, what additional efficiency benefit can the proposed measurement enable at comparable output quality when oracle information is unavailable? Alternatively, what new understanding does it provide beyond established speculative-acceptance behavior?
4 comments
Thank you for the clarification and additional experiments. I also remain unconvinced that the additional findings establish a sufficiently substantial conceptual advance over prior work on token difficulty and adaptive computation. I therefore maintain my original rating.
We thank the reviewer for reading our response and for the reply. We would like to understand the remaining concern precisely, so that we can address it in the revision.
The review cites Duo-LLM, FS-GEN and FlexiDepth as prior work on token difficulty and adaptive computation. As far as we can tell, none of them reports the following results from our paper and response:
- a per-token compute requirement measured across model families, against verified-correct answers, in FLOPs, and the resulting bound, 44-77x below 72B decoding on MATH-500;
- that this cost is highly concentrated: 10% of tokens carry 64-80% of the FLOPs (Fig. 3);
- that independently trained families agree on which tokens are expensive, beyond surprisal controls (Table 1);
- that 77% of the tokens a 0.5B drafter rejects are reproduced by a model of ≤ 7B, and only ~1% of tokens need ≥ 32B;
- that confidence-based routing misroutes the expensive tokens (≥ 14B) 1.9-3.5x more often than cheap ones;
- that across scale, connectives and step openers are expensive while numbers and math symbols are cheap, which is the opposite of FlexiDepth's within-model finding;
- that 37-59% of drafter misses fall at the target's near-ties and do not affect accuracy.
Could the reviewer point us to the work, or the specific result, that already establishes any of these? If so, we will gladly cite it and revise our claims. If the concern is instead the size of the advance rather than prior work, it would help us to know what result the reviewer would have needed to see.
Thank you for the clarification. I agree that the cited work does not establish the specific empirical findings you highlight. My score raise from 1 to 2. What practical implications do you see for these findings, particularly in real-world inference settings where oracle information is unavailable?
Response to Reviewer 8HM1
We thank the reviewer for the careful review and for recognizing the cross-family evidence and the continuation analysis. The closing question offers two ways the paper could justify itself: an efficiency benefit without the oracle, or understanding beyond speculative acceptance. We now provide both, with new results below. We first clarify the paper's goal, since it shapes how the oracle should be read.
The paper is a measurement study rather than a decoding method. It asks how much compute each token of a correct answer actually needs, and how far current serving methods are from that amount. In such a study, the oracle defines the quantity being measured. Belady's MIN is never deployed as a cache policy, and the Shannon limit is not a code, yet both serve as the reference that practical methods are measured against. To our knowledge, no prior work quantifies this distance for LLM decoding. We find that the bound is 44-77x below 72B decoding on MATH-500. We will make this framing clearer in the abstract and add this gap table to the main text.
(a) Benefit without the oracle
The map can be learned. We trained a rule on GSM8K maps only and tested it on MATH-500 and HumanEval, using only signals available at inference time:
This cuts wrong delegations 2.8x (MATH-500) and 4.3x (HumanEval). The rule reads whether a 1.5B model agrees with the 0.5B, and it calls the 1.5B on only 29% of steps. Run live on all 500 MATH-500 problems, with every cost charged:
This improves routing at equal quality; it does not replace lossless speculative decoding.
(b) Understanding beyond speculative acceptance
Sufficient compute and acceptance are related but different quantities. An acceptance rate is one number for one drafter-target pair: it tells whether the drafter failed, not what the token needed. The map answers the second, which adds three things acceptance cannot show:
W1: Requires a reference
We agree the reference is unavailable at inference; this is inherent to any bound: a quantity computed only from information a deployed system has would be a policy, not a bound. Duo-LLM [1], which the reviewer cites, takes the same view of its own oracle, which "requires access to ground truth labels, making it impractical for real-world deployment" (§5). In addition, (a) shows that the map yields a rule that needs no reference.
W2: Novelty vs [1] [2] [3]
We agree these works establish that tokens differ in difficulty, and we build on them. To our knowledge, none measures what a token needs across model families, against a verified answer, in FLOPs.
The distinction matters because findings can differ. [3] finds that computation tokens need more depth. Across scale, we find the opposite: connectives (3.34x their share of excess compute) and step openers (3.02x) are expensive, while numbers (0.23x) and math symbols (0.39x) are cheap. What is expensive within one model is not always what is expensive across models, and a cross-model measurement is needed to see this. We will add this comparison to the related-work section.