acceptodds
Under review as a conference paper at ICLR 2027

What Does a Token Cost? A Mixture-of-Agents Measurement of Sufficient Per-Token Compute

Abstract

Large language models spend the same amount of computation on every token they generate, regardless of how difficult each token is to produce. Methods such as speculative decoding and model routing are built on the premise that much of this computation is unnecessary, yet the computation an individual token actually requires has not been measured. We measure it through a Mixture-of-Agents (MoA) lens: a panel of fifteen language models of increasing capacity, drawn from three families, in which every agent attempts to reproduce a reference sequence token by token, conditioned on the correct preceding tokens. We define the inference cost of the smallest agent that succeeds as the token's sufficient compute, which upper-bounds what the token requires. On three core benchmarks, a 0.5B agent reproduces 92–95% of reference tokens. Across Qwen, OLMo, and R1-distilled panels, the most expensive 10% account for 64–80% of estimated FLOPs. On all 500 MATH-500 problems, the MoA-derived map helps model routing reduce projected latency from 7.59 to 5.12 seconds while slightly improving accuracy, relative to the best confidence-routing baseline. The MoA-map helps drafting use 32.6% fewer draft tokens and approximately 20% lower projected latency than fixed-window drafting at similar accuracy. These comparisons reveal remaining allocation headroom, motivating controllers that exploit sufficient-compute structure.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Efficiency, Mixture of Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8HM1OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper measures token-level sufficient compute by identifying the cheapest model that reproduces each reference token under teacher forcing. It studies compute concentration, cross-family agreement, continuation behavior, and oracle-guided inference allocation.

Critical strengths

S1. The cross-family analysis provides useful evidence of shared reproduction difficulty beyond the evaluated surprisal controls.

S2. The continuation experiments clearly distinguish local token reproduction from reliable generation. The robustness checks and transparent discussion of limitations are appreciated.

Critical weaknesses

W1. The measurement requires known reference prefixes, while the allocation experiments additionally require advance token-agreement information. These are unavailable during ordinary inference. An offline measurement study can still be valuable, but I am not yet convinced that the findings provide sufficiently new explanatory insights to compensate for this limited applicability.

W2. Limited novelty. Duo-LLM[1] already studies oracle compute allocation and token difficulty; FS-GEN[2] analyzes sparse large-model intervention and prediction uncertainty; and FlexiDepth[3] examines token-dependent computational demand. The cross-family measurement extends this literature, but the conceptual advance beyond these established observations appears incremental.

[1]. Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Models

[2]. Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding

[3]. Adaptive Layer-skipping in Pre-trained LLMs

Questions for authors

Standard speculative decoding already exploits small/large-model agreement without advance reference access. Although the paper compares oracle-guided drafting against fixed and adaptive drafting, what additional efficiency benefit can the proposed measurement enable at comparable output quality when oracle information is unavailable? Alternatively, what new understanding does it provide beyond established speculative-acceptance behavior?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8HM1

We thank the reviewer for the careful review and for recognizing the cross-family evidence and the continuation analysis. The closing question offers two ways the paper could justify itself: an efficiency benefit without the oracle, or understanding beyond speculative acceptance. We now provide both, with new results below. We first clarify the paper's goal, since it shapes how the oracle should be read.

The paper is a measurement study rather than a decoding method. It asks how much compute each token of a correct answer actually needs, and how far current serving methods are from that amount. In such a study, the oracle defines the quantity being measured. Belady's MIN is never deployed as a cache policy, and the Shannon limit is not a code, yet both serve as the reference that practical methods are measured against. To our knowledge, no prior work quantifies this distance for LLM decoding. We find that the bound is 44-77x below 72B decoding on MATH-500. We will make this framing clearer in the abstract and add this gap table to the main text.

(a) Benefit without the oracle

The map can be learned. We trained a rule on GSM8K maps only and tested it on MATH-500 and HumanEval, using only signals available at inference time:

Predictor of "0.5B suffices"MATH-500 AUCHumanEval AUC
Confidence (speculative / cascade signal)0.9560.960
Learned from our maps0.9760.983

This cuts wrong delegations 2.8x (MATH-500) and 4.3x (HumanEval). The rule reads whether a 1.5B model agrees with the 0.5B, and it calls the 1.5B on only 29% of steps. Run live on all 500 MATH-500 problems, with every cost charged:

14B targetAcc.Target callss / item
Confidence routing67.21725.65
Learned rule⟦67.8⟧⟦91⟧⟦4.8⟧

This improves routing at equal quality; it does not replace lossless speculative decoding.

(b) Understanding beyond speculative acceptance

Sufficient compute and acceptance are related but different quantities. An acceptance rate is one number for one drafter-target pair: it tells whether the drafter failed, not what the token needed. The map answers the second, which adds three things acceptance cannot show:

  • Most rejections are cheap. 77% of the tokens the 0.5B gets wrong are reproduced by a model of ≤ 7B; only ~1% of tokens need ≥ 32B. Acceptance treats all of them as "needs the target".
  • The remaining limit is the target's own near-ties. 37-59% of drafter misses (3 families) sit at the target's lowest-margin positions, where misses are predictable at only AUC 0.67 (vs 0.96 elsewhere). EAGLE-3, which reads the target's features, hits the same wall. Breaking these ties at random changes accuracy by −0.2 pts [−2.4, +1.8]. So a large share of the misses that remain are near-arbitrary choices that drafters, including EAGLE-3, do not learn, and that do not affect correctness. An aggregate acceptance rate cannot separate these from learnable misses; a token-level bound can.
  • The signal lies between models, not inside one. A probe on the 0.5B's hidden states does worse than confidence (0.926). Agreement across models recovers it, which is our Table 1 finding put to work.

W1: Requires a reference

We agree the reference is unavailable at inference; this is inherent to any bound: a quantity computed only from information a deployed system has would be a policy, not a bound. Duo-LLM [1], which the reviewer cites, takes the same view of its own oracle, which "requires access to ground truth labels, making it impractical for real-world deployment" (§5). In addition, (a) shows that the map yields a rule that needs no reference.

W2: Novelty vs [1] [2] [3]

We agree these works establish that tokens differ in difficulty, and we build on them. To our knowledge, none measures what a token needs across model families, against a verified answer, in FLOPs.

Duo-LLMFS-GENFlexiDepthOurs
Reference verified correct✗✗✗✓
Cross-family, per token✗✗✗✓ (4 families)
Signal beyond uncertainty tested✗✗ (visual only)✗✓, and it transfers
FLOP-weighted cost ladder✗✗layers only✓

The distinction matters because findings can differ. [3] finds that computation tokens need more depth. Across scale, we find the opposite: connectives (3.34x their share of excess compute) and step openers (3.02x) are expensive, while numbers (0.23x) and math symbols (0.39x) are cheap. What is expensive within one model is not always what is expensive across models, and a cross-model measurement is needed to see this. We will add this comparison to the related-work section.

Thank you for the clarification and additional experiments. I also remain unconvinced that the additional findings establish a sufficiently substantial conceptual advance over prior work on token difficulty and adaptive computation. I therefore maintain my original rating.

We thank the reviewer for reading our response and for the reply. We would like to understand the remaining concern precisely, so that we can address it in the revision.

The review cites Duo-LLM, FS-GEN and FlexiDepth as prior work on token difficulty and adaptive computation. As far as we can tell, none of them reports the following results from our paper and response:

  1. a per-token compute requirement measured across model families, against verified-correct answers, in FLOPs, and the resulting bound, 44-77x below 72B decoding on MATH-500;
  2. that this cost is highly concentrated: 10% of tokens carry 64-80% of the FLOPs (Fig. 3);
  3. that independently trained families agree on which tokens are expensive, beyond surprisal controls (Table 1);
  4. that 77% of the tokens a 0.5B drafter rejects are reproduced by a model of ≤ 7B, and only ~1% of tokens need ≥ 32B;
  5. that confidence-based routing misroutes the expensive tokens (≥ 14B) 1.9-3.5x more often than cheap ones;
  6. that across scale, connectives and step openers are expensive while numbers and math symbols are cheap, which is the opposite of FlexiDepth's within-model finding;
  7. that 37-59% of drafter misses fall at the target's near-ties and do not affect accuracy.

Could the reviewer point us to the work, or the specific result, that already establishes any of these? If so, we will gladly cite it and revise our claims. If the concern is instead the size of the advance rather than prior work, it would help us to know what result the reviewer would have needed to see.

Thank you for the clarification. I agree that the cited work does not establish the specific empirical findings you highlight. My score raise from 1 to 2. What practical implications do you see for these findings, particularly in real-world inference settings where oracle information is unavailable?

Reviewer_jvQYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a methodology to measure the "cost" of each decode token, by using LLMs with varying parameter sizes. By setting up mixture of agent-style multi-LLM minimum compute model finder, the authors were able to find the smallest model for each decode token, given that they have prior information up to the token right before the current one. Also, the authors provided simple headroom analysis by comparing their method (with an assumption that the router has prior knowledge of what the next token would be) with simple threshold-based router.

The main contribution I think is the novel approach of using multiple LLMs to estimate the compute cost. This approach answers the question: how expensive each token is to match the result of very big LLMs (e.g. Qwen 72B).

The validation for this work is inherently tricky, because in some sense the authors are proposing a new metric for each token. But for headroom, accuracy and latency were given to compare the new mixture of agent with existing LLMs.

Critical strengths

I really like the logical steps the authors came to the research question and the approach they took to tackle it. The cost is no doubt one of the major concerns of LLM nowadays. And to decompose and mitigate the cost issue, the authors focused on how to reduce the cost of decoding each token. And using multiple LLMs with varying size for indirectly "measuring" the cost was novel and intuitive: model cascades, speculative decoding, and mixture of depth all tried varying model sizes (or choosing fraction within same model) to reduce decode cost, but to my understanding they did not try to actually measure the per token compute cost. The outcome of this approach is essentially a mapping: token->minimum LLM param size. And I think this result, even without going into the application of this cost map to routing, can be very interesting and lead to further research. Maybe this work can lead to answering questions like, which tokens are actually carrying "expensive" information, whether those correlate with human intuition, are expensive tokens correlate with the outliers observed in speculative decoding or mixture of depth methods, etc.

Critical weaknesses

Applying the result of the work into real-world scenarios (for example token router choosing cheapest model, as authors demonstrate in the paper) is nearly impossible (The authors definitely acknowledge this point in the discussion section to some degree, which I do appreciate).

The paper assumes that s_t (defined in p3) has all information for decoding the token at t, and s_t consists of a prompt and the reference tokens from start to t-1. And this entails very heavy assumption: they assume prefix cache (KV cache) is available (p4). Also, there is another heavy assumption: "...every admissible agent emits the same token y_t^star and reaches the same next state". The paper's suggested way to apply the method into real world makes sense with these two assumptions. However, those assumptions are nearly impossible to achieve. To put it in another way, the headroom suggested in the paper seems like impossible to perform hillclimb on.

If we lift up that assumption, we have to run all models in the panel every time when we decode a token, and save KV cache for each model. Then we are actually spending more compute, not less, to predict each token, and the amount of compute we spend is strictly greater than the biggest model in the panel. The other related methods in literature, such as speculative decoding, model cascades, mixture of depth, do not require these heavy assumptions.

Questions for authors
  • Please let me know your thoughts on the weakness description. Would love to know how you would resolve the issue.
  • p1 first paragraph last two sentences: could you elaborate a little more? You said "Their gains depend on how often this is possible", but isn't it the same for your method?
  • p6: Out of my own curiosity, were the top 10% expensive tokens looked semantically important by human inspection?
  • Related to point 1: how would you make oracle controller if you actually implement it in real world? Can it be cheaper than the expensive models like 72B? Maybe we can train a smaller model separately but wouldn't it cost significant compute to preserve accuracy?
  • p7 last paragraph: would be nice to polish a little. In terms of accuracy, obviously best performing one would be theta=1.00. I think what you wanted to say was, you drew the accuracy vs latency curve for the token-level cascade via tuning the threshold, and your MoA-map routing method result positions on the upper left of the curve, which shows that it can outperform the baseline, or something similar?

Changed my rating from 2 to 3 after discussion.

3 comments
AuthorsOpenReview ↗

Response to Reviewer jvQY

We thank the reviewer for the thoughtful review and for engaging with the idea itself. We agree that the token-to-cost map is the main object, and the questions the reviewer raises (which tokens carry expensive information, and how they relate to speculative-decoding outliers) are exactly the directions we hope it enables. The main concern is that the headroom rests on assumptions no real system meets. We realize our presentation may have made the selector look like a deployment recipe. The selector is the measuring instrument, not the system we propose to deploy, and seen this way, both conditions play a natural role. We will make this explicit in Section 3.

W1: "Every admissible agent reaches the same next state"

This follows from the definition rather than being an additional assumption. An agent is admissible only if it emits the reference token y*t (Eq. 5). Appending the same token to the same prefix gives the same state, so all admissible agents share J{t+1} (Eq. 7). Nothing is assumed about the agents beyond admissibility.

W1: Free KV cache, no switching cost

This idealization is what makes the quantity a bound. It charges every token the least cost it could possibly have, so any real system can only pay more. The gap between this idealized cost and what practice pays is exactly what the paper measures. We also account for the real costs. Our live runs (App. E, and the new run below) use real KV caches and charge the target's catch-up and every rejected draft.

W1: "Running every model costs more than the 72B"

The reviewer is right, and this is a one-time, offline measurement cost, like profiling a program before optimizing it. The panel never runs at inference. What runs is a cheap controller learned from the map (Q3).

Q3: A real controller, cheaper than the 72B?

Yes. We trained a controller on GSM8K maps and tested it on MATH-500 and HumanEval:

Predictor of "0.5B suffices"MATH-500 AUCHumanEval AUC
0.5B confidence0.9560.960
0.5B + 1.5B agreement (learned)0.9760.983

It makes 2.8x / 4.3x fewer wrong delegations and calls the 1.5B on only 29% of steps. Run live on all 500 MATH-500 problems:

14B targetAcc.Target callss / item
Confidence routing67.21725.65
Learned controller⟦67.8⟧⟦91⟧⟦4.8⟧

So the headroom can be hill-climbed: the map supplies the training labels, and the controller costs a small fraction of the target.

Q1: "Gains depend on how often this is possible; isn't that the same for you?"

Yes, and this is a good way to state our motivation. Cascades and speculative decoding all depend on how often a cheap model suffices, but they do not measure it directly. We measure it, token by token. ⟦P: we will make this explicit on p.1⟧

Q2: Are the expensive tokens semantically important?

Yes. They are the decision words of the solution, not its arithmetic. Measured as share of excess compute over share of tokens:

  • expensive: connectives (3.34), step openers (3.02), common words (2.73);
  • cheap: math symbols (0.39), rare words (0.32), numbers (0.23).

Rare words are cheap because they are copied names; common words are expensive because they are choices about how to proceed. Confidence errs on these expensive tokens 1.9-3.5x more often than on cheap ones. ⟦P: 3 annotated examples in the appendix⟧

Q4: p.7 wording

Thank you; we agree, and the reviewer's reading is the one we intended. The claim is that map routing lies up and to the left of the whole confidence frontier: it matches 72B-only accuracy (69.6 vs 69.2%) in 5.12 vs 19.03 s.

Thank you for your detailed response. The response resolved most, if not all, of the concerns. I do have a general follow up on wordings. I thought your end goal (direction) is to save compute, and by measuring them we can allocate resourse wisely without sacrifying the accuracy. With your response, I feel like this is not the case? So you measure the cost per token, but what you actually can do with this methodology is to reduce the latency, while keeping the compute similar or even more than just a big model. And when I said "impossible to hillclimb" I meant we cannot save compute. Am I correct?

I think the main cotribution of this paper, in terms of real-world deployement possibility, is that we can save latency by enabling per-block decoding. In some sense this is more like a latency hiding method. I think this was kind of too subtle to catch in the first read. Please update the wording, maybe add a sentence or two to clarify this (or let me know if you wrote this already and I have missed your point). In other words, the method is in some sense a way of replacing or helping speculative decoding with leanred router?

changed my rating from 2 to 3.

We sincerely thank the reviewer for the follow-up and for raising the score. The question is sharp, and the reviewer's reading is largely correct for today's serving mechanisms. We clarify where it holds and where the measurement points further.

Compute or latency?

The measurement is about compute; current mechanisms can only realize it as latency. Both statements are supported by our numbers (MATH-500, 72B target):

AccuracyLatency (s)TFLOPs
72B only69.219.03101.9
MoA-map routing69.65.1296.7
Adaptive drafting69.04.69140.5
MoA-map drafting69.43.99112.1
  • Latency: the map cuts it substantially, as the reviewer says.
  • Compute: routing saves only ~5%, and drafting spends more than 72B alone. The reason is structural: to stay ready, the large model must still process every token to keep its KV cache up to date, so it pays nearly full compute even when a small model chooses the token.
  • The bound: the tokens themselves require 44-77x less compute than 72B decoding.

So, to the reviewer's question: with today's mechanisms, yes, the compute savings cannot be realized, and this is precisely what our gap shows. It is not a limit of the tokens, which need far less, but of the state the large model has to maintain. We see this as one of the paper's most useful findings: it tells the field that better routers alone will not save compute, and that the open problem is reducing this state cost.

Is it a way to help speculative decoding with a learned router?

Yes, that is the most direct practical use. The map supplies token-level labels from which a controller can be trained, deciding when the small model's token suffices and when to defer, for routing or for drafting. Our results show both uses: with oracle map labels, drafting uses 32.6% fewer draft tokens and is faster than adaptive drafting (3.99 vs 4.69 s), and a rule learned from GSM8K maps transfers to MATH-500 and HumanEval without any oracle.

Wording

The reviewer did not miss it; we did not state it clearly enough. We will add the following to the introduction and discussion:

"Our bound is stated in compute. Current serving mechanisms keep the large model's KV cache up to date, so the savings they realize appear as latency rather than FLOPs: with the map, latency drops from 19.03 to 3.99-5.12 s while FLOPs stay at 96.7-112.1 T, against a bound 44-77x lower. Converting the bound into compute savings requires reducing this state cost, which we identify as the main open bottleneck."

Thank you again for helping us sharpen this point.

Reviewer_QFNWOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the sufficient compute for each token in a given trace, measured by the size of the smallest model that can reproduce the token given the prefix. On multiple benchmarks, the paper finds that smaller models like 0.5B can reproduce most of the tokens, and the most expensive tokens account for a large portion of the estimated FLOPs. Also, if one uses the ideal map derived in the paper, one can significantly reduce the latency compared to the previous routing baseline. Hence, this paper reveals there is a large potential for improvement in the allocation mechanism.

Critical strengths
  1. The presentation is clear to me. Figure 1 clearly illustrates the insight of the paper: a small model can reproduce most of the tokens in the correct traces. The definition of sufficient compute is clear and reasonable, and the experiment section clearly shows the performance difference between previous routing baselines and the ideal performance (MoA-map routing)

  2. The sufficient compute defined in this paper could potentially be a useful metric for evaluating the newly proposed routing mechanism.

Critical weaknesses
  1. Even if the paper clearly identifies that the previous routing approaches are not optimal in terms of sufficient compute, this observation is expected and not novel enough. The fact that previous routing mechanisms are not optimal is not surprising. The difficult part is to establish approaches to make the actual compute closer to the sufficient compute, and the paper does not make any attempt in this direction. For example, the paper does not investigate whether a routing rule can be learned from the MoA map on the validation set, and generalized to another test set.

  2. The observation that small models can reproduce most tokens has already been demonstrated in prior speculative decoding work [1]. Hence, the observation itself is not new.

[1]. Leviathan, Yaniv, Matan Kalman, and Yossi Matias. "Fast inference from transformers via speculative decoding." International conference on machine learning. PMLR, 2023.

Questions for authors
  1. Is it possible to learn a routing rule from the MoA map on the validation set, and then use this routing rule to generalize to another test set?
3 comments
AuthorsOpenReview ↗

Response to Reviewer QFNW

We thank the reviewer for the clear summary, for finding the definition clear and reasonable, and for a concrete, testable question. The answer is yes: a rule learned from the map on one dataset generalizes to another, and we report it below. We first address the concern that the observation is "expected".

We agree that routers being suboptimal is expected; how far from optimal they are, and where they fail, was not known. "Caches are not optimal" is also expected, yet measuring how far they are from Belady's MIN is what guided cache design. Our contribution is this kind of measurement: the per-token bound sits 44-77x below 72B decoding on MATH-500, and confidence routing fails mostly on the expensive tokens, misrouting tokens that need ≥ 14B 1.9-3.5x more often than cheap ones. To our knowledge, these quantities had not been measured before, and they indicate where improvement effort should go.

Q: Learn a routing rule on one set, generalize to another?

We ran exactly this protocol. Training uses GSM8K maps only; testing uses MATH-500 (math) and HumanEval (code). The rule sees only signals available at inference time.

Predictor of "0.5B suffices"MATH-500 AUCHumanEval AUC
Confidence routing0.9560.960
Learned from our maps0.9760.983

At 85% delegation, wrong delegations fall from 0.585% to 0.207% on MATH-500 (2.8x) and from 0.384% to 0.089% on HumanEval (4.3x). Run live on all 500 MATH-500 problems, with every cost charged:

14B targetAcc.Target callss / item
Confidence routing67.21725.65
Learned rule⟦67.8⟧⟦91⟧⟦4.8⟧

The map matters for this result in two ways:

  • It works because of our central finding. The rule reads whether a 1.5B model agrees with the 0.5B. That is useful only because independently trained models share which tokens are hard (Table 1).
  • The obvious alternative fails. A probe on the 0.5B's own hidden states does worse than confidence (AUC 0.926). The measurement is what identified the right signal.

Is the remaining gap just hindsight?

A natural follow-up is whether the remaining gap only reflects hindsight, so we measured it directly. 37-59% of drafter misses (3 families) sit at the target's near-ties, where misses are predictable at only AUC 0.67 (vs 0.96 elsewhere), even for EAGLE-3. Breaking these ties at random changes MATH-500 accuracy by −0.2 pts [−2.4, +1.8]. So hindsight accounts only for choices that do not affect correctness; the rest is learnable, as shown above.

W2: Leviathan et al. already show that small models agree

We agree, and we cite it as our starting point rather than our finding. Leviathan's α is one average for one pair; even a bigram drafter reaches α = 0.2 (their Table 3). An average cannot show what our map shows:

  • 77% of 0.5B rejections need only a model of ≤ 7B, and only ~1% of tokens need ≥ 32B;
  • 10% of tokens carry 64-80% of FLOPs (Fig. 3);
  • independent families agree on which tokens are hard (Table 1), which is what makes the rule above possible.

In short, "small models agree most of the time" is the premise. Which tokens they fail on, what those tokens cost, and whether that is predictable are the contribution. We will state this distinction explicitly in the introduction.

Thanks for the authors' response and the generalization experiment. Since the authors claim that the contribution is the measurement, could the authors further clarify why it is important to measure the gap and how the measurement result guides the practical experiment? I am concerned about whether the measurement itself is meaningful enough for an ICLR paper.

We thank the reviewer for the follow-up and for engaging with the generalization experiment. This is the central question, and we answer it concretely: the measurement already guided the practical experiment in our response, and it tells practitioners three things that no existing method or acceptance rate tells them.

Why measure the gap: it says where effort pays off

Without a bound, one cannot tell whether a method is near its limit or far from it, or which part of the problem is limiting. The measurement answers both:

  • How much room exists. The bound is 44-77x below 72B decoding on MATH-500 (32-64x on GSM8K, 57-69x on HumanEval). Current methods are far from the limit, not close to it.
  • Which part limits. Even a perfect map, used for routing, recovers only ~5% of that room. What blocks the rest is not knowing where tokens are cheap but the cost of handing over: when a small model continues on its own, accuracy drops by 9.44 pts within 64 tokens (Sec. 4.2). This points research at state and handover, not only at better routers.

How it guided the practical experiment

The generalization experiment the reviewer acknowledged was designed from the measurement. Each design choice came from a measured result:

Measured resultDesign decision it led to
Confidence misroutes expensive (≥ 14B) tokens 1.9-3.5x more oftenConfidence alone is not enough; a second signal is needed
A probe on the 0.5B's own hidden states does worse than confidence (AUC 0.926 vs 0.956)The missing signal is not inside the small model
Independent model families agree on which tokens are hard (Table 1)Use agreement between two cheap models as the signal
77% of 0.5B misses need only ≤ 7BA small second model (1.5B) suffices

The result: trained on GSM8K maps, the rule transfers to MATH-500 and HumanEval (AUC 0.976 / 0.983 vs 0.956 / 0.960) with 2.8-4.3x fewer wrong delegations. Without the measurement, we would not have known which signal to try.

It also says where to stop

The measurement shows that 37-59% of drafter misses sit at the target's near-ties, where misses are predictable at only AUC 0.67 (vs 0.96 elsewhere), even for EAGLE-3. Breaking these ties at random does not change accuracy (**−0.2 pts **). This tells practitioners that a large share of the remaining misses is not worth chasing: further drafter improvements there will not pay off. A token-level bound is what makes this line visible.

Is a measurement meaningful enough for ICLR?

We believe it is, and the community has repeatedly valued, and awarded, such measurements:

  • Understanding Deep Learning Requires Rethinking Generalization (ICLR 2017, best paper) proposed no method. It measured that networks can fit random labels, and that one measurement changed how the field reasons about generalization.
  • Are Emergent Abilities of Large Language Models a Mirage? (NeurIPS 2023, outstanding paper) proposed no model. It showed that how we measure determines what the field concludes about scaling.
  • The Lottery Ticket Hypothesis (ICLR 2019, best paper) found subnetworks that can only be identified in hindsight, after full training. Its value was in showing what is possible, and it shaped pruning research for years.
  • Do ImageNet Classifiers Generalize to ImageNet? (ICML 2019) proposed no method. It measured a gap, between benchmark accuracy and accuracy on fresh data, and that number became a standard reference point.
  • Closest to our setting, Deja Vu (ICML 2023) measured oracle contextual sparsity in LLMs: the same measure-then-exploit path as our generalization experiment.

Our paper is in the same spirit: it measures what decoding actually requires, which the reviewer noted "could potentially be a useful metric for evaluating the newly proposed routing mechanism." We agree, and the generalization experiment is a first example of that use.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion

We thank the AC and all reviewers for their time and engagement. We summarize the discussion below.

What the paper contributes

The paper measures how much compute each token of a verified-correct answer actually needs, by finding the cheapest model, across four model families, that reproduces it. This gives a bound against which serving methods can be measured. Reviewers found the approach "novel and intuitive" (jvQY), the definition "clear and reasonable" and "potentially a useful metric" (QFNW), and the cross-family evidence "useful" (8HM1).

Main concerns and how they were addressed

ConcernRaised byResponseStatus
The oracle and reference are unavailable at inference8HM1, jvQYThe oracle defines the bound, as in Belady's MIN. New: a rule learned from GSM8K maps, with no oracle, transfers to MATH-500 and HumanEval (AUC 0.976 / 0.983 vs 0.956 / 0.960 for confidence; 2.8-4.3x fewer wrong delegations)jvQY: "resolved most, if not all, of the concerns", rating 2 → 3
Can a rule learned from the map generalize?QFNWRan the reviewer's exact protocol: yes (above)Acknowledged by QFNW
Why measure the gap, and how does it guide practice?QFNW (follow-up)The bound is 44-77x below 72B decoding; it shows that current mechanisms are limited by the large model's state, not by knowing where tokens are cheap; each design choice of the generalization experiment came from a measured result; awarded ICLR/NeurIPS measurement papers as precedentAnswered in full in our follow-up reply
Does it save compute or latency?jvQYLatency today (19.03 → 3.99-5.12 s); compute savings are blocked by the large model's KV-cache state, which the gap quantifies. Wording addedClarified
Novelty vs Duo-LLM, FS-GEN, FlexiDepth8HM1None measures per-token compute across families against verified answers in FLOPs. Across scale, our finding is the opposite of FlexiDepth's within-model resultSee note below
"Small models agree" is known (Leviathan et al.)QFNWCited as our starting point. New: 77% of 0.5B rejections need only ≤ 7B; 10% of tokens carry 64-80% of FLOPsAddressed

New results added during the discussion

  1. Learned, oracle-free rule that transfers across datasets and domains (above). It works because model families share which tokens are hard (Table 1), and a probe on the small model alone does not (AUC 0.926).
  2. Where confidence fails: it misroutes tokens that need ≥ 14B 1.9-3.5x more often than cheap ones.
  3. The limit of drafting: 37-59% of drafter misses (3 families) fall at the target's near-ties, where they are predictable at only AUC 0.67, even for EAGLE-3; breaking these ties at random does not change accuracy (−0.2 pts).
  4. Type-level costs: connectives and step openers are expensive (3.34x, 3.02x their share), numbers and math symbols cheap (0.23x, 0.39x).

Note on Reviewer 8HM1

Reviewer 8HM1 maintained the rating, stating that the findings are not a sufficiently substantial advance over prior work on token difficulty and adaptive computation. We listed seven specific results and asked which prior work establishes any of them, so that we could cite it and revise. As of DATE, no specific prior work has been identified. We respect the reviewer's judgment and leave the assessment to the AC.

Planned revisions

  • Abstract and introduction: state the measurement framing and the compute-vs-latency distinction explicitly (jvQY).
  • Main text: a table of the bound vs current practice (QFNW, 8HM1).
  • New section: how the measurement guides practice, with the learned rule and the near-tie analysis (QFNW, 8HM1).
  • Related work: the comparison with Duo-LLM, FS-GEN and FlexiDepth, including the opposite finding (8HM1).
  • Fixed wording on p.1 and p.7, and annotated examples of expensive tokens (jvQY).

We thank the reviewers again; their questions led directly to these additions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.