acceptodds
Under review as a conference paper at ICLR 2027

QUOTA: Head-Sensitive Uncertainty for Budget-Interpretable Token Routing

Abstract

Inference-time compute scaling underpins state-of-the-art reasoning in large language models (LLMs), but fully autoregressive generation with large models carries prohibitive computational costs. While speculative decoding and collaborative inference mitigate this via small–large model collaboration, existing approaches suffer from coarse routing granularity, reliance on supervised training, or model-dependent uncertainty thresholds that lack interpretable resource semantics. To address this challenge, we introduce QUOTA, a training-free token-level routing framework that decouples where to deploy LLM compute from how much capacity to allocate. QUOTA ranks generation states using head-sensitive collision entropy and calibrates raw scores into percentiles via a prompt-only empirical cumulative distribution function (ECDF). Given a user-specified LLM token quota , states in the top- uncertainty tail receive one LLM token, after which routing is recomputed from the updated shared prefix, eliminating auxiliary training or routing labels. Evaluated across mathematical reasoning, scientific QA, and code generation tasks, QUOTA achieves superior quality-latency tradeoffs and precise budget fidelity: with the Qwen3-0.6B/32B model pair, it obtains a 48.55 macro-average pass@1, outperforming SpecReason by 2.90 percentage points with a 17-second reduction in per-question latency; on AIME25 with Qwen3-0.6B/8B, it delivers 42.50% accuracy using only 29% LLM output tokens, surpassing GlimpRouter and SpecReason under lower compute budgets. Across three model pairs and twelve operating points, QUOTA maintains a budget mean absolute error of 1.32 percentage points, offering fine-grained and auditable token-level collaborative inference for efficient high-quality LLM reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
token-level LLM routing, collaborative inference, budget-aware inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NDeNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

QUOTA is a token-level router for small-large model collaboration. At each decoding step it scores the SLM's next-token distribution with entropy, maps the score to a percentile using ECDF calibrated only on unlabeled prompts, and lets the LLM emit the next token when the percentile falls in the top-q tail of a user-specified LLM token quota q; after the one-token takeover, routing is recomputed from the updated prefix. Experiments report better quality-latency tradeoffs than previous methods

Critical strengths
  • The motivation is clear, supported by formalization and validated by ablation studies. The experimental setup with MR is convincing.
  • The training-free pipeline is complete and sample-efficient, offering strong ease of use.
Critical weaknesses
  • Some details are unclear. I do not understand how KV cache management is performed in the proposed framework, where the SLM executes most of the tokens and hands over to the LLM at certain positions. At this point, the LLM's KV cache is interrupted, and the tokens executed by the SLM have no cache. How is this situation handled? Does the LLM perform a prefill again here (for the tokens previously generated by the SLM)? Or are the SLM and LLM structurally coupled to ensure that the LLM also saves the KV cache at every token? Additionally, while this paper reports the time cost, the memory pressure brought by the two models and the two KV caches should also be reported in the main results (e.g., peak memory usage).
  • Cross-domain generalization of the calibration data. The paper relies on the calibration data to estimate the distribution, which may cause cross-domain distribution shift. For example, if the calibration is done on natural text (e.g. wikitext) but the inference is run on math data, can the LLM participation ratio still stay consistent with the requested budget? The overall model performance may also degrade due to inaccurate estimation of the positions where the LLM is invoked.
  • From the main results (Table 1), the proposed method does not show a clear advantage over GlimpRouter. The accuracy is only slightly higher while the latency is also higher. Can the authors clarify this?
Questions for authors

see Weaknesses.

Reviewer_iWsTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes QUOTA, a training-free token-level router for small–large model collaborative inference. At every generation position the SLM full next-token distribution is scored with collision entropy and claimed "head-sensitive": it responds to competition among the leading candidates and is insensitive to fragmentation of the low-probability vocabulary tail, unlike Shannon entropy. Raw scores are mapped to percentiles through an ECDF frozen offline from SLM trajectories over a few hundred unlabeled prompts. A user-specified quota q is then applied directly on the percentile scale: a state is routed when u_t ≥ 1 − q. A routed state receives exactly one LLM token, after which the SLM resumes from the updated shared prefix and routing is recomputed. No router training, routing labels, paired outputs, or LLM scoring calls are required.

Validation covers three model pairs (Qwen3-0.6B/32B, Qwen3-0.6B/8B, DeepSeek-R1-Distill-Qwen-1.5B/32B) on pooled AIME 2024/2025, GPQA-Diamond, and LiveCodeBench v6, against SpecReason, GlimpRouter, standalone endpoints, an intermediate Qwen3-8B reference, and a matched-budget random-placement control (QUOTA-MR). Headline claims: 48.55 macro pass@1 at 167 s/question for Qwen3-0.6B/32B (+2.90 pts and −17 s vs SpecReason); 42.50 on AIME25 with Qwen3-0.6B/8B at 29% LLM output tokens; and 1.32 pp mean absolute budget error across 12 model-pair × quota operating points. Ablations separate the ranking statistic (H1 vs H2) from the threshold parameterization (raw vs ECDF), calibration-set size (64–512 prompts), and takeover span (1/2/4 tokens).

Critical strengths

1. The matched-budget random-placement control (QUOTA-MR) is the right ablation and is rarely done. Most adaptive-routing papers establish only that spending more strong-model compute helps, which is uninformative. By holding the realized LLM token share fixed and randomizing where the takeovers land, Table 1 separates the value of the placement policy from the value of the budget itself (48.55 vs 28.97 macro pass@1). This experiment makes the paper's central claim, that collision entropy identifies consequential positions, causally interpretable, and it is why I take the method seriously despite the concerns below.

2. The quota interface is a genuinely useful deployment property, and it is validated honestly. Expressing the routing threshold in ECDF-percentile coordinates rather than as a raw entropy cutoff means one knob carries the same resource meaning across SLM checkpoints, LLM scales, and task domains. Figure 2 and §4.3 test this across three pairs and four quotas. For anyone operating such a system, "set q and get roughly q" is worth more than a point gain on a 30-problem benchmark, and the paper does treat the control interface as a first-class contribution.

Critical weaknesses

1. Against the strongest baseline, every reported margin is under one point, and the paper reports no variance anywhere. The abstract, introduction, and conclusion all benchmark QUOTA against SpecReason (+2.90 pts, −17 s). But Table 1's strongest baseline is GlimpRouter, and against it QUOTA is +0.83 on AIME, +0.42 on LCBv6, +0.76 on GPQA-Diamond, +0.67 on the macro-average — while being slower (167 s vs 157 s). Table 2 repeats the pattern at matched allocation: +1.67 pts on AIME25 for +6 s. So the paper is not a Pareto improvement over its closest competitor; it is a sub-point accuracy trade against latency, framed throughout against the weaker of the two baselines. This matters because the benchmarks are small — AIME25 is 30 problems, pooled AIME24/25 is 60, GPQA-Diamond is 198 — and accuracy is pass@1 from a single sample per problem per run, averaged over four runs. At p ≈ 0.4 on 30 problems the per-run standard error is roughly 9 pp, and averaging four runs on the same problems reduces only the sampling component, not the problem-set component. The 5-point H1-ECDF vs H2-ECDF gap in Table 3 is about 1.5 problems. No standard deviations, confidence intervals, or significance tests appear in any table. I cannot currently distinguish any of the paper's main comparisons from run-to-run noise, and that is decisive for my recommendation because every contribution claim rests on these margins.

2. The paper's core novel ingredient is never tested against the obvious one-line alternative. Appendix A.3 motivates collision entropy with an analysis showing that Shannon entropy's tail contribution grows as ε log m with tail support size m. The analysis is correct, but the standard remedy is to compute entropy over the truncated nucleus rather than the full vocabulary — and §4.1 states that decoding already runs at top-p = 0.95, so a truncated distribution is available at zero cost. Under truncation the diffuse tail that motivates H2 does not exist. Every H1 baseline in Table 3 and Figure A3 is untruncated, so the comparison does not test "head-sensitive vs not," it tests "truncated vs untruncated" with H2 standing in for truncation. Also, because the ECDF is monotone, the routing rule depends only on the ranking the score induces, so the natural head-sensitive competitors are the top-1 probability and the top-1/top-2 margin — both cheaper than H2 and both strictly head-only. Neither appears. Without top-p-truncated H1 and a margin baseline, I cannot tell whether collision entropy contributes anything beyond ignoring the tail, which is the paper's main conceptual claim.

Questions for authors
  1. Please report per-run standard deviations or bootstrap CIs for every number in Tables 1–4. With four complete runs these already exist. Which of the QUOTA-vs-GlimpRouter differences survive?
  2. What quota q produced the QUOTA row of Table 1, and what was the realized LLM output-token share? Table 2 reports this for the 8B pair but Table 1 does not, so the headline comparison against SpecReason and GlimpRouter is not visibly budget-matched. Were all methods in Table 1 run at comparable LLM allocation?
  3. Please add two baselines: (a) Shannon entropy computed over the top-p = 0.95 nucleus with ECDF calibration, and (b) top-1/top-2 probability margin with ECDF calibration. If H2-ECDF does not separate from these, how should the head-sensitivity claim be read?
  4. Latency is strongly sublinear in token savings and the overhead is never characterized. From Table 2: pure SLM 115 s, pure LLM 197 s, QUOTA 168 s at 29% LLM tokens; linear interpolation predicts ≈139 s, so ≈29 s (35% of the SLM–LLM span) is unexplained. How is the LLM's KV cache maintained over the SLM-emitted tokens between takeovers, and what is the per-takeover switching cost? A breakdown of latency into SLM decode, LLM decode, and switching overhead would substantially strengthen the efficiency claim.
Reviewer_Bp3LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents QUOTA, a training-free token-level routing method for small–large model collaboration. It uses collision entropy to identify uncertain tokens and ECDF calibration to control the fraction of tokens generated by the LLM. The routing decision is updated after each token. Experiments on math, scientific QA, and code generation show competitive accuracy and latency.

Critical strengths
  1. The paper addresses a practical problem: when to use a larger model during generation while keeping its usage under control.
  2. The method is simple and training-free, requiring no additional router or labeled data.
  3. The experiments cover several model pairs and reasoning tasks, with useful ablations and budget-matched comparisons.
Critical weaknesses
  1. The token quota does not directly reflect computational cost. The SLM still runs at every token position, and switching to the LLM may require additional context processing. The paper does not explain how KV caches are managed between the two models or quantify this overhead. It is therefore unclear how well the reported LLM token share reflects actual compute savings.
  2. The improvement over the strongest baseline is modest. In Table 1, QUOTA achieves better pass@1 versus GlimpRouter, but is slower. Without confidence intervals, it is hard to judge whether this accuracy gain is meaningful. The table also lacks LLM token usage for the competing methods, making the efficiency comparison less clear.
  3. The budget calibration may not generalize well. The ECDF is calibrated on SLM-generated trajectories, while inference uses prefixes generated by both models. This can change the uncertainty distribution and hence the actual LLM usage. The reported average budget error is encouraging, but per-task and per-request results are needed to assess how reliable the quota is.
  4. The benefit of collision entropy is not fully isolated. In Table 3, H2-ECDF uses 29% LLM tokens, compared with 27% for H1-ECDF. The accuracy difference may partly reflect this extra LLM usage. A comparison at exactly matched budgets, including simpler scores such as maximum probability or top-two margin, would better establish the value of collision entropy.
  5. Some figures have limited information density and provide little additional insight, such as Figure 2.
Questions for authors
  1. How are KV caches handled when switching between the SLM and LLM? Does the LLM need to process earlier SLM-generated tokens? What is the overhead of these operations?
  2. Could the authors report LLM token usage for all methods in Table 1, along with confidence intervals for accuracy and latency?
  3. How does budget accuracy change across tasks and individual requests? What happens when the inference distribution differs from the calibration data?
  4. Does collision entropy still outperform Shannon entropy and simpler uncertainty scores when the realized LLM token budget is held fixed?
  5. Appendix A.2 studies token-level disagreement between the two models. How well does the proposed uncertainty score predict cases where an LLM intervention actually improves correctness, rather than simply producing a different token?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.