- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Paper addresses the cost of large lm head in small LLMs. It identifies a mismatch between lm head weight similarity and the hidden state geometry relevant to token selection, compounded by noisy calibration estimates for rare tokens. The authors propose CHES, a training-free method that constructs token representations by blending calibrated hidden state directions with normalized lm head weight through count adaptive shrinkage. They further introduce AVQ, a closed-form anisotropic centroid update designed to better align clustering with inner-product routing. These components enable selective vocabulary scoring without modifying model weights or permanently removing tokens.
Critical strengths
- A key strength of the paper is that CHES's handling of limited calibration data is reasonable and practically valuable: it adaptively blends between the hidden-state direction and the original output weights based on the number of observations of each token, allowing high-frequency tokens to exploit calibration information while retaining the weight prior for rare or unobserved tokens. The method requires no fine-tuning of the model and does not permanently delete vocabulary entries, thus offering good deployment compatibility.
Critical weaknesses
-
The comparison baselines are insufficient. The main external method in the experiments is TKCS, lacking comparisons with related methods already discussed in the paper, such as L2S.
-
There are duplicate references. Chen et al. (2019a) and Chen et al. (2019b) actually correspond to the same paper. There is a compilation error in the appendix citations. The positions of the table captions are not consistent.
Questions for authors
-
The paper describes L2S as the closest work, but does not include it in the experiments. Are there evaluation results against L2S, or specific applicability limitations that can be explained?
-
Has applying AVQ directly in the LM-head weight space, i.e., TKCS+AVQ, been evaluated? Can this comparison demonstrate that, under the same use of AVQ, the CHES representation still brings additional benefits?
6 comments
Authors Message to Reviewer PzHa
Dear Reviewer PzHa,
We thank you for your time to review our work and your constructive comments to improve it, and we really hope to have a further discussion with you to see if our response solves your concerns.
We have replied to the important points raised by you such as comparison with L2S baseline and TKCS+AVQ ablation in our rebuttal response. Since the author-reviewer discussion period has now reached towards its end, we will deeply appreciate if you could check our response to your review comments soon. This way, if you have further questions and comments, we can still reply before the author-reviewer discussion period ends.
We again thank you for your time and efforts.
Best, Authors
Your reply solved my problem. However, I think you should also update the revision of the PDF and incorporate the suggestions from each reviewer.
Rebuttal by Authors
We would first like to thank you for your time and for recognizing our work practically valuable and offering good deployment compatibility. We would now like to address your concerns point-by-point as follows:
Thank you for raising this. We respectfully want to provide a small clarification that besides TKCS, Table 1 also compares against CSV-Decode (Liu et al., 2025), the most recent training-free dynamic sub-vocabulary method. In our submission, we choose not to include L2S because:
L2S is a end-to-end training technique (tested only on small LSTMs with 25K vocabulary) unlike CHES which partitions the context (hidden-state) space and attaches a candidate token set to each region. L2S learns each cluster's candidate set from the exact top-5 predictions of the training contexts (Eq. 6–7), so a token that never appears in any Top-5 is never added to any candidate set and becomes unreachable at inference. For modern large vocabularies (150-300K), majority of tokens falls under this unreachable category.
During the rebuttal we reimplemented L2S on Qwen2.5-1.5B following its Algorithm 1. Using the exactly same calibration tokens (100K) from nemotron-cc v2 as used by CHES, we trained L2S and report the performance as follows:
It can be clearly observed that L2S suffers significantly high even in comparison to TKCS for complex math and code based tasks. This can be attributed to training dependent paradigm of L2S where relevant tokens didn't participate in Top-K when trained with a general purpose nemotron-cc v2 corpus.
We sincerely apologize for our mistakes. We promise to fix all pointed issues (broken references and macro conflicts) and proof-read the final draft with utmost carefulness.
Thank you for raising this great question which we missed during our ablations. AVQ is a plug-n-play technique and can be applied with TKCS as well. We ran TKCS+AVQ with the exactly same settings in Figure 4 for Qwen2.5-1.5B for fair comparison. We report the vocabulary coverage (%) needed to reach target Top-1 hit rate as follows:
Clearly, AVQ also benefits TKCS significantly in reducing the vocabulary coverage required to reach a desired Top-1 Hit rate.