acceptodds
Under review as a conference paper at ICLR 2027

The Outsized Vocabulary Tax: Selective-Vocabulary Decoding for Edge LLMs

13 views

Abstract

Modern small language models pair a compact backbone with a large vocabulary which incur an output projection cost of per decoded token, reaching up to 28% of total parameters and 20% of decode latency on edge hardware. Recent cluster-routed decoding reduces this to without modifying any weights, by scoring only the tokens in the few of clusters whose centroids best match a given hidden state. Existing methods cluster LM-head rows directly, but we show that tokens close in weight space are often not the tokens that similar hidden states select. We introduce (Calibrated Hidden-state Embedding Selection), a training-free method that represents each token by a count-adaptive shrinkage estimate on the unit sphere. This estimate interpolates between the mean direction of the hidden states that select the token and a prior given by its LM-head row. We further propose , a closed-form anisotropic centroid update that aligns clustering with inference-time inner-product routing. On Qwen2.5 and Gemma2 models, reaches 95% top-1 routing accuracy while scoring only 6-8% of the vocabulary and yields up to 3.2 LM-Head speedup on Apple M4 and Intel Xeon CPUs. Across multiple downstream benchmarks (e.g., GSM8K, MATH, TriviaQA), can closely match the full LM-Head performance while utilizing only 18-20% of total vocabulary.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vocabulary Compression, LLMs, Efficient Inference, Sparse Computation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PzHaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper addresses the cost of large lm head in small LLMs. It identifies a mismatch between lm head weight similarity and the hidden state geometry relevant to token selection, compounded by noisy calibration estimates for rare tokens. The authors propose CHES, a training-free method that constructs token representations by blending calibrated hidden state directions with normalized lm head weight through count adaptive shrinkage. They further introduce AVQ, a closed-form anisotropic centroid update designed to better align clustering with inner-product routing. These components enable selective vocabulary scoring without modifying model weights or permanently removing tokens.

Critical strengths
  1. A key strength of the paper is that CHES's handling of limited calibration data is reasonable and practically valuable: it adaptively blends between the hidden-state direction and the original output weights based on the number of observations of each token, allowing high-frequency tokens to exploit calibration information while retaining the weight prior for rare or unobserved tokens. The method requires no fine-tuning of the model and does not permanently delete vocabulary entries, thus offering good deployment compatibility.
Critical weaknesses
  1. The comparison baselines are insufficient. The main external method in the experiments is TKCS, lacking comparisons with related methods already discussed in the paper, such as L2S.

  2. There are duplicate references. Chen et al. (2019a) and Chen et al. (2019b) actually correspond to the same paper. There is a compilation error in the appendix citations. The positions of the table captions are not consistent.

Questions for authors
  1. The paper describes L2S as the closest work, but does not include it in the experiments. Are there evaluation results against L2S, or specific applicability limitations that can be explained?

  2. Has applying AVQ directly in the LM-head weight space, i.e., TKCS+AVQ, been evaluated? Can this comparison demonstrate that, under the same use of AVQ, the CHES representation still brings additional benefits?

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We would first like to thank you for your time and for recognizing our work practically valuable and offering good deployment compatibility. We would now like to address your concerns point-by-point as follows:


[Weakness 1 + Question 1] Comparison with Baselines are insufficient

Thank you for raising this. We respectfully want to provide a small clarification that besides TKCS, Table 1 also compares against CSV-Decode (Liu et al., 2025), the most recent training-free dynamic sub-vocabulary method. In our submission, we choose not to include L2S because:

L2S is a end-to-end training technique (tested only on small LSTMs with 25K vocabulary) unlike CHES which partitions the context (hidden-state) space and attaches a candidate token set to each region. L2S learns each cluster's candidate set from the exact top-5 predictions of the training contexts (Eq. 6–7), so a token that never appears in any Top-5 is never added to any candidate set and becomes unreachable at inference. For modern large vocabularies (150-300K), majority of tokens falls under this unreachable category.

During the rebuttal we reimplemented L2S on Qwen2.5-1.5B following its Algorithm 1. Using the exactly same calibration tokens (100K) from nemotron-cc v2 as used by CHES, we trained L2S and report the performance as follows:

MethodTrainingGSM8KMATHMBPP
1TKCSNo58.221.734.4
2L2S (reimplemented)Yes40.68.312.1
3CHESNo60.627.338.4

It can be clearly observed that L2S suffers significantly high even in comparison to TKCS for complex math and code based tasks. This can be attributed to training dependent paradigm of L2S where relevant tokens didn't participate in Top-K when trained with a general purpose nemotron-cc v2 corpus.


[Weakness 2] Need to polish and correct writing mistakes

We sincerely apologize for our mistakes. We promise to fix all pointed issues (broken references and macro conflicts) and proof-read the final draft with utmost carefulness.


[Question 2] Benefits of AVQ directly on the Weight Space (TKCS + AVQ)

Thank you for raising this great question which we missed during our ablations. AVQ is a plug-n-play technique and can be applied with TKCS as well. We ran TKCS+AVQ with the exactly same settings in Figure 4 for Qwen2.5-1.5B for fair comparison. We report the vocabulary coverage (%) needed to reach target Top-1 hit rate as follows:

Method80%95%99%99.5%
1TKCS3.116.341.560.3
2TKCS+AVQ (requested)2.410.835.454.1
3CHES1.66.728.841.3
4CHES+AVQ1.26.625.935.4

Clearly, AVQ also benefits TKCS significantly in reducing the vocabulary coverage required to reach a desired Top-1 Hit rate.

AuthorsOpenReview ↗

Authors Message to Reviewer PzHa

Dear Reviewer PzHa,

We thank you for your time to review our work and your constructive comments to improve it, and we really hope to have a further discussion with you to see if our response solves your concerns.

We have replied to the important points raised by you such as comparison with L2S baseline and TKCS+AVQ ablation in our rebuttal response. Since the author-reviewer discussion period has now reached towards its end, we will deeply appreciate if you could check our response to your review comments soon. This way, if you have further questions and comments, we can still reply before the author-reviewer discussion period ends.

We again thank you for your time and efforts.

Best, Authors

Your reply solved my problem. However, I think you should also update the revision of the PDF and incorporate the suggestions from each reviewer.

Reviewer_NV8cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper reduces LM-head cost in large-vocabulary small models. The paper introduces CHES (Calibrated Hidden-state Embedding Selection), a training-free method that represents each token by a count-adaptive shrinkage estimate on the unit sphere. The authors further propose AVQ, a closed-form anisotropic centroid update that aligns clustering with inference-time inner-product routing. The authors demonstrated that CHES+AVQ can closely match the full LM-Head performance while utilising only 18-20% of the total vocabulary across multiple downstream benchmarks.

Critical strengths
  • I believe this is a timely problem, and the motivation of the study is well explained.
  • No training, no weight changes, same inference path; Figure 5(a) shows more calibration data helps with shrinkage and hurts without it.
Critical weaknesses
  • Some Downstream results at the 5-10% budget collapse to zero; however, the paper lacks a proper explanation for it.
  • The claims of the paper are inconsistent.
  • The experiments are somewhat limited.
  • The paper requires further polishing.
Questions for authors

Here is my feedback on this paper below.

First of all, some Downstream results at the 5-10% budgets collapse to zero; however, the paper lacks a proper explanation for this. In Table 1, both CHES variants score 0.0 on MATH at 5% for Qwen2.5-1.5B (TKCS 12.5) and Gemma-2-2B (TKCS 11.7), and 0.0-0.2 on Gemma-2-2B MBPP at 5-10% (TKCS 7.4, 12.8). The authors do not mention this and claim the advantage is largest where TKCS is weakest, which is only true for GSM8K and TriviaQA. I would like to suggest that the authors provide the proper justification for the results that collapse to zero.

Secondly, the claims of the paper are inconsistent. The authors write near-lossless at 18-20% in the abstract, 10-15% in the contribution list, yet only 20% is actually supported in Section 5.1. I would like to suggest that the authors make the claims consistent and tone down the claims if needed to ensure that the claims and the presented experimental evidence match.

Third, the reported latency seems to present the head-only latency, instead of the end-to-end latency. I believe that it would be a great addition to present the end-to-end latency gain with and without the proposed methods so as to provide the full picture of the proposed methods' contributions to the readers and reviewers.

In addition, the authors seem to assume that a laptop and a server CPU with FP16 heads are edge devices. I believe that a measurement on a phone, embedded devices, or IoT devices with int4/int8 heads would strengthen the claim.

Furthermore, the authors did not provide proper justification why they chose particular hardware like Apple M4 and Intel Xeon CPUs when a large number of research has been conducted on GPUs. I would like to suggest that the authors discuss this point and provide some justification for their choice of the experimental hardware.

Lastly, the paper requires substantial polishing to resolve the writing issues. For example, there are many broken references (??), e.g., Appendix ?? and Section ?? in both the main content and the Appendix. Also, some equation starts with "Eq. equation", some other equation starts with "Eq.", I believe this should be consistent.

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We would first like to thank you for your time and identifying our work as timely, well-motivated, having a low-overhead design with no training and identical inference path. We would now like to address your concerns point-by-point as follows:


[Weakness 1] Downstream results at 5–10% budgets collapse to zero without explanation

Thank you for mentioning this (also raised by Reviewer nDrz). To summarize, as mentioned in Table 1, our proposed method (CHES and CHES+AVQ) show substantially low performance on MATH-500 (0% for Qwen2.5-1.5B and Gemma2-2B with 5% keep-fraction) and MBPP (0% on Gemma-2B). These results are an artifact of the evaluation pipeline used by lm-evaluation-harness (https://github.com/eleutherai/lm-evaluation-harness) for MATH and MBPP.

  1. The MATH scoring uses the math-verify package to extract and parse the final answer from each generation. Across all the samples where TKCS was successful on Qwen (12.5) and Gemma (11.7), our proposed method scored zero because of algebraic and syntactic formatting issues in the generated response which math-verify treats differently. Specifically, mathematically equivalent outputs are treated differently due to minor syntax variations: \−3+6i-3 + 6i\\ (TKCS) vs. \−3+6∗i-3+6*i\\ (Ours), inserting an explicit '*' multiplication sign; \12\frac{1}{2}\\ (TKCS) vs. \0.50.5\\ (Ours), using decimal instead of fraction format; and \75+15i\frac{7}{5}+\frac{1}{5}i\\ (TKCS) vs. \frac{7}{5}+\frac{1}{5}i. (Ours), omitting the math delimiters with a trailing dot.

  2. The MBPP scoring takes the generated text, cut at a special [DONE] and uses a HF code_eval metric to build a program of the generated code and runs it with a short timeout (about 3s by default), and the problem counts as passed only if it finishes without raising. For all the samples in MBPP where TKCS passed and Ours failed, we found that [DONE] was generated as [Done], differing only in letter casing (uppercase vs. lowercase). The default check in the evaluation pipeline perform a plain string match which fails for [Done] and evaluation assert raises NameError.

For the fairness in reporting our performance with respect to other baselines, we choose not to incorporate any post-processing of our results and report as it was generated by third-party evaluation framework. Note that such issues only appear at very aggressive compression regime such as 5-10% which is not practical for challenging MATH/Code tasks (potentially because calibration is done using nemotron-v2 dataset). These concerns disappear with vocabulary usage around 20% where our proposed method significantly outperforms other baselines. As suggested, we promise to incorporate this explanation in our camera-ready version.


[Weakness 2] Inconsistent claims wrt. performance across abstract (18-20%), and contribution list (10-15%)

Thank you for raising this and we fully agree with your concern about our mistake. Based on our further evaluation in addition to datasets presented in submitted draft, we found that 10-15% vocabulary usage is sufficient for General and Text-generation based tasks (such as MMLU-pro, BBH, TruthfulQA - results attached below) while require ~20-25% vocabulary for complex Math, Coding and Reasoning heavy tasks (MATH, MBPP, HumanEval). We promise to provide this clear claims which perfectly aligns with our experimental evidences attached in the final version.

Qwen2.5-1.5B
MMLU-ProBBHTruthfulQA
Dense20.643642.74
10% Vocab20.335.0642.28

[Weakness 3] Reported latency is limited to Head-only instead of End-to-End latency

Thank you for this suggestion. We report head-only latency as our primary metric on purpose because: (a) Our proposed method modifies only how the output projection is evaluated. The backbone, its weights and its kernels are identical with and without CHES. This evaluation setting permits us to have a controlled measurement of our contribution, and everything else in the decode step is held fixed. (b) The end-to-end latency suffers higher run-time variability that can come from thermal throttling, OS scheduling etc. influencing the measurements.

As you recommended, we measured E2E latency during the rebuttal phase and propose to include them in final version. For Qwen2.5-1.5B where head share of total parameter is ~15%, over a 4096 tokens generation length we empirically found 1.08x end-to-end speedup, while for Gemma-2-2B where the head share is 23% we found a 1.11x speedup measured on Apple M4 Silicon.


AuthorsOpenReview ↗

Rebuttal by Authors [Part 2]

[Weakness 4 + 5] Laptop and server CPU with FP16 heads as "edge" + Quantized Head + Choice of experimental H/W

We appreciate your concern and we would like to clarify that M4 chips also ship in Apple iPads, and our Xeon measurements at 1–4 threads (Figure 6c) are meant to emulate a constrained CPU budget rather than a server workload. Note that our proposed method is orthogonal to quantization where quantization reduces bytes per row, while CHES reduces how many rows are read and the two different mode of saving will add up. In addition, deployments often quantize the backbone more aggressively than the output head, because the head is accuracy-sensitive. This makes the head's share of decode cost larger, which strengthens the case for CHES.

The reason to choose our H/W settings instead of GPUs is motivated by the primary usecase of CHES - small language models with large vocabulary. Typical GPU inference involves multi-batch inference + gather over rows maps poorly onto tensor-core GEMMs. CHES is motivated for on-device inference settings with bandwidth-bound decoding, tight energy constraints, and batch-1 processing where CHES can shine.

To further address your concern, we empirically estimated the speedup of lm-head with 262k vocabulary on NVIDIA Jetson Orin NX (GCID: 38968081, BOARD: generic, EABI: aarch64) across 2048 tokens and report our performance as follows:

Vocabulary UsageMedian (us)Mean (us)Speedup
100% - Full Head5339.15845.61.0x
10%739.2712.67.22x
20%1419.11366.43.76x
30%2287.22147.22.33x

[Weakness 6] Need to polish and correct writing mistakes

We sincerely apologize for our mistakes and strongly agree with the need for polished writing in scientific manuscripts. We promise to fix all pointed issues (broken references and macro conflicts causing Eq. equation) and proof-read the final draft with utmost carefulness.

Reviewer_nDrzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at the cost of computing scores over the full vocabulary in language models. This cost is especially noticeable in models with small backbones. Previous methods use vocabulary pruning or clustering. Pruning requires fine-tuning, while clustering does not. The authors build on clustering, arguing that earlier work mainly focused on which clusters to select, rather than how to group tokens. Existing methods group tokens using their output weight directions. The authors instead look at the hidden states (those highest-scoring ones) and use their average direction. However, this estimate can be noisy for tokens rarely selected during calibration. CHES therefore mixes the hidden-state direction with the token’s weight direction and normalizes the result. Experiments show that this works better than using weight directions alone. This is the most interesting part to me. The authors also note that there is a mismatch between clustering and cluster selection, so they introduce AVQ. Ablation results show that adding AVQ further improves CHES.

Critical strengths

The idea of clustering tokens using the hidden states that select them is intuitive and well motivated. The routing, downstream, and hardware results provide convincing evidence of practical benefits. The method also has deployment potential because it requires no model-weight updates.

Critical weaknesses
  1. It seems that the anisotropic loss was already proposed by Gao et al. at ICML 2020, so claiming AVQ as a theoretical innovation is inappropriate.
  2. The authors mention using the Nemotron-CC-v2 High-Quality subset as the calibration dataset (about 20M tokens), but Figure 5 uses 100M calibration tokens.
Questions for authors
  1. Could you clarify which calibration dataset was used and whether it was consistent across experiments?
  2. Could you explain why the proposed method fails on the MATH and MBPP tasks under a 5% vocabulary budget?
  3. the paper still has “Appendix ??” “Section ??”
2 comments
AuthorsOpenReview ↗

Rebuttal By Authors

We would first like to thank you for your time and finding our work intuitive, well motivated, and deployment-friendly. We would now like to address your concerns point-by-point as follows:


[Weakness 1] Claiming AVQ as a theoretical innovation is inappropriate.

Thank you for pointing this out. We completely agree that anisotropic loss was introduced by Gao et al. at ICML 2020 and our exact contribution is deriving AVQ as a plug-n-play technique within conventional k-means euclidean residual minimization for cluster selection with a closed-form solution. To better position our contribution, we propose to make following change (line 100-103) in our camera-ready draft:

"Theoretical Contribution: We adapt the Anisotropic Quantization Loss (Guo et al. 2020) to centroid estimation for cluster-routed decoding and derive AVQ as a plug-n-play technique within conventional k-means euclidean residual minimization to align with the requirement of asymmetric residual orientation for inner-product at inference. We additionally show that CHES asymptotically reduces to TKCS and pure h-space clustering in the cold and hot-row limits."


[Weakness 2 and Question 1] Clarification regarding the calibration datasets

Thank you for raising this, and we apologize for the confusing wording. All our experiments use the Nemotron-CC-v2 High-Quality subset. Depending on the purpose of experiments, the token budget is varied and we clarify it as below:

UseBudgetWhere
1Building the cluster index (default)[100K] tokensFig. 4, Fig. 6, Fig. 7, Table 1
2Calibration-budget sweep100K to 100M tokensFig. 5a
3Frequency deciles only (not used to build any index)~20M-token held-out streamFig. 5b, Appendix B

We promise to add these details within Appendix with appropriate references in main draft.


[Question 2] Proposed Method Performance drop at 5% vocabulary usage for MATH and MBPP

Thank you for mentioning this (also raised by Reviewer NV8c). To summarize, as mentioned in Table 1, our proposed method (CHES and CHES+AVQ) show substantially low performance on MATH-500 (0% for Qwen2.5-1.5B and Gemma2-2B with 5% keep-fraction) and MBPP (0% on Gemma-2B). These results are an artifact of the evaluation pipeline used by lm-evaluation-harness (https://github.com/eleutherai/lm-evaluation-harness) for MATH and MBPP.

  1. The MATH scoring uses the math-verify package to extract and parse the final answer from each generation. Across all the samples where TKCS was successful on Qwen (12.5) and Gemma (11.7), our proposed method scored zero because of algebraic and syntactic formatting issues in the generated response which math-verify treats differently. Specifically, mathematically equivalent outputs are treated differently due to minor syntax variations: \−3+6i-3 + 6i\\ (TKCS) vs. \−3+6∗i-3+6*i\\ (Ours), inserting an explicit '*' multiplication sign; \12\frac{1}{2}\\ (TKCS) vs. \0.50.5\\ (Ours), using decimal instead of fraction format; and \75+15i\frac{7}{5}+\frac{1}{5}i\\ (TKCS) vs. \frac{7}{5}+\frac{1}{5}i. (Ours), omitting the math delimiters with a trailing dot.

  2. The MBPP scoring takes the generated text, cut at a special [DONE] and uses a HF code_eval metric to build a program of the generated code and runs it with a short timeout (about 3s by default), and the problem counts as passed only if it finishes without raising. For all the samples in MBPP where TKCS passed and Ours failed, we found that [DONE] was generated as [Done], differing only in letter casing (uppercase vs. lowercase). The default check in the evaluation pipeline perform a plain string match which fails for [Done] and evaluation assert raises NameError.

For the fairness in reporting our performance with respect to other baselines, we choose not to incorporate any post-processing of our results and report as it was generated by third-party evaluation framework. Note that such issues only appear at very aggressive compression regime such as 5-10% which is not practical for challenging MATH/Code tasks (potentially because calibration is done using nemotron-v2 dataset). These concerns disappear around with vocabulary usage around 20% where our proposed method significantly outperforms other baselines. As suggested, we promise to incorporate this explanation in our camera-ready version.


[Question 3] the paper still has "Appendix ??" "Section ??"

The missing section number at line 759 refers to Section 4.4. For the missing Appendix number, it is a typo error which requires deletion. We promise to correct all these mistakes as well as proofread sincerely in our camera-ready version.

Thank you for your answers! I am keeping my current score, which I feel is reasonable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.