- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how to select a fixed output vocabulary offline while keeping the tokenizer unchanged. The method maximizes the expected log retained probability mass, which is equivalent to minimizing expected reverse KL from the restricted and renormalized teacher distribution to the original teacher. The authors show that the objective is monotone submodular and use a lazy greedy algorithm with a standard approximation guarantee. They also interpret frequency pruning as a first-order approximation, giving sufficient conditions for optimality and examples where it is suboptimal. Experiments cover synthetic mixtures, multilingual distributional fidelity and text continuation across several model families, and speculative decoding with EAGLE-2. The authors report improvements over frequency and magnitude pruning on a smoothed fidelity metric and continuation quality, as well as modest throughput gains over frequency pruning in several decoding settings.
Critical strengths
- The paper presents a theoretical analysis of static vocabulary pruning based on a reverse-KL formulation and submodularity. This helps explain when frequency-based selection can be suboptimal and motivates a selection rule that gives greater weight to poorly covered contexts.
Critical weaknesses
1.Although the paper compares with FR-Spec, comparisons with other relevant methods, such as VocabTrim and SpecVocab, are missing. The main throughput gains are reported on Arabic, while gains on other tested languages are smaller or tied. A controlled comparison with representative prior methods would help establish the practical value of the proposed selector.
Questions for authors
1.Could you compare your method with VocabTrim and representative dynamic vocabulary-selection methods, such as SpecVocab, NanoSpec, or DynaSpec(choose one), under matched experimental conditions? Please discuss the trade-offs in throughput, memory usage, training costs, etc to clarify when the proposed approach is preferable.