- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
They introduce a calibration-guided, training-free method PASS built on two complementary per-token signals: participation which measures the rate at which a token enters the model's top- at decode time; and substitution cost} which is a mass-weighted logit deficit incurred by replacing a token with its nearest surviving neighbour.
Critical strengths
-
This article is well-written, with interesting observations and a clear motivation. Lines 71-77 clearly describe the three main problems of the existing static vocabulary pruning methods.
-
The method proposed in this article is applicable to most LLMs and is independent of the model's structure, as they all have a large LM Head.
-
The article considered various model structures and conducted experiments on different model scales.
-
The ablation results demonstrate the effectiveness of the method. (line 462)
Critical weaknesses
My main concerns lie in the selection of the comparison methods and the test benchmarks.
- The comparison method only compared the static methods. However, the dynamic methods also need to be included in the comparison (if a fair comparison can be achieved).
[1] Fuadi, Mukhlish, Adhi Dharma Wibawa, and Surya Sumpeno. "Efficient Transformer Models via Language-Aware Frequency-Based Vocabulary Pruning." IEEE Access 14 (2026): 50993-51006.
[2] Zhang, Hanling, et al. "Vocabtailor: Dynamic vocabulary selection for downstream tasks in small language models." Findings of the Association for Computational Linguistics: ACL 2026. 2026.
- Regarding the selection of the benchmark dataset, the dataset chosen in Table 1 is too difficult. Most small models do not have significant mathematical power (GSM8K, MATH500, AIME), and these performance tests actually make it difficult to evaluate the effectiveness of the methods. I hope you can verify by referring to the mainstream benchmarks in the field of pruning. [3-4]
[3] Chen, Xinrui, et al. "A simple linear patch revives layer-pruned large language models." Advances in Neural Information Processing Systems 38 (2026): 154484-154509.
[4] Shopkhoev, Dmitriy, et al. "Replaceme: Network simplification via depth pruning and transformer block linearization." Advances in Neural Information Processing Systems 38 (2026): 19487-19517.
- Does the hardware configuration affect the effectiveness of the method? If possible, please explain how different hardware affects the speedup ratio. It would be better to provide some experiments as well.
I hope the author can address my questions during the subsequent discussion stage. I will then re-evaluate the quality of the article.
Questions for authors
-
I would like to know if the benchmark test was conducted on edge devices or on the GPU? Are there significant differences between these two?
-
Different edge devices such as those with different processors from the Apple series, or even processors from different brands, whether they have different operator (eg. sparse GEMM) support for the model's LM head, has led to different speeds? Is it the case that in certain situations, vocabulary pruning is less important or vice versa?
-
I would like to know how the actual speed test details were achieved, as this involves improvements in both the calculation and storage aspects. I hope to see a breakdown table of the time details.
3 comments
Rebuttal By Authors (Part 2)
[Weakness 3 + Question 2] Different processors and operator support (e.g., sparse GEMM) + when does vocabulary pruning matter less?
A key deployment benefit of PASS is that it requires no sparse operator support. The pruned head is a smaller dense matrix which is executed by the same dense GEMV kernel as the full head. Its speedup therefore depends only on the device's dense GEMV bandwidth and compute BUT not on sparse kernel availability which varies widely across H/W vendors (unlike unstructured or per-token dynamic sparsity).
Vocabulary pruning matters less when:
- (a) the head is a small share of per-token cost (larger models or smaller vocabularies as in Figure 1a);
- (b) decoding is compute-bound (large batch or prefill - Figure 1b shows the head's share of bytes falling with batch size);
- (c) the device's bandwidth is high relative to kernel overhead (B200 at ≥ 0.6 in Figure 8b).
[Question 3] Details corresponding to our measurements
Thank you for your question. All of latency measurements use batch size 1 and BF16 weights, with KV-cached autoregressive decoding at context length 2048 tokens. We use MLX on Apple M4, PyTorch + CUDA on NVIDIA B200, and PyTorch on the Intel Xeon 8559C (16 threads). For the deployment scenario, we prune the tokenizer and the pruned head is a smaller dense matrix executed by the same dense GEMV kernel as the full head with no masking or sparse kernels are involved. Our details for 0.5 keep-fraction is given as:
| Model | Head params (M), full → PASS | Weights read / token, FP16 (MB) | FLOPs / token (MFLOP) | Vocabulary storage, FP16 (MB) | |
|---|---|---|---|---|---|
| Qwen2.5-0.5B | 151,936 × 896 | 136.1 → 68.1 | 272.3 → 136.1 | 272.3 → 136.1 | 272.3 → 136.1 |
| Qwen2.5-1.5B | 151,936 × 1,536 | 233.4 → 116.7 | 466.7 → 233.4 | 466.7 → 233.4 | 466.7 → 233.4 |
| Gemma-2-2B | 256,000 × 2,304 | 589.8 → 294.9 | 1,179.6 → 589.8 | 1,179.6 → 589.8 | 1,179.6 → 589.8 |
All columns scale linearly with the head size where each parameter is read once (2 bytes in FP16) contributing 2 FLOPs (one multiply-add), and is stored once (2 bytes in FP16 as the matrix is shared between embedding and head for tied models). At batch 1 the LM-head GEMV performs about one FLOP per byte read which is far below any device's compute-to-bandwidth ratio. Therefore its latency is set by the bytes read rather than by compute and halving the head roughly halves its time on bandwidth-bound devices as in Figure 8.
I acknowledge that the additional experiments have substantially addressed my original concerns. Therefore, I will keep my score of 3 (Weak Acceptance).
Rebuttal By Authors (Part 1)
We would like to thank you for finding our submission well-written, with interesting observations and a clear motivation. We would now like to address your concerns point-by-point as follows:
Thank you for the suggested references. We would first like to clarify how the two works relate to PASS:
Faudi et. al (https://ieeexplore.ieee.org/document/11460136): This is a language-aware frequency-based vocabulary pruning which is very similar to our comparison baseline VT (https://arxiv.org/abs/2305.15020) included in the paper draft.
VocabTailor (https://arxiv.org/abs/2508.15229): It is a hybrid static-dynamic vocabulary selection strategy for LM Head where for each input it loads the LM-head rows for the unique input tokens and combines them with a small static task-specific token set. A fair comparison can't be achieved because: (1) VocabTailor's static set is profiled per downstream task which is not deployment friendly, where as PASS's S is task-agnostic and fixed once on a general-purpose calibration set; (2) VocabTailor's head changes per input, so the full head must remain available all-the-time with memory overhead and the active head is rebuilt per request. PASS instead ships one smaller dense matrix with no per-request work.
We respectfully like to state that dynamic pruning techniques do not form a fair baseline for PASS because:
Thank you for raising the concern. We would respectfully like to explain the reason behind our original choice. Static head pruning affects a model only through the tokens it can emit, so free-form generation (math, code, open-domain QA) is where pruning damage appears. Log-likelihood multiple-choice tasks can mask it, as we see in our evaluation with MMLU, which is identical for every method in Table 1. We choose the tasks based on this and used % PPL Retention as a generic metric to capture language modeling capabilities of the model.
We would like to clarify that the quality results (Table 1) do not depend on hardware. PASS changes which rows in the LM-Head will be used for the softmax, so we evaluate quality on GPUs. We provide our latency (Figure 8) measurements on three platforms that span different regimes:
Apple M4 (MLX) having unified memory with batch-1 decode bandwidth-bound, so the head speedup closely tracks 1/kf (≈1.75–1.9× speedup at kf=0.5).
NVIDIA B200 which have very high HBM bandwidth making the head GEMV short, so fixed kernel-launch overheads dominate. We therefore observe that gains vanish at kf ≥ 0.6 for the smaller Qwen heads.
Intel Xeon 8559C (16 threads) batch-1 decoding is again bandwidth-bound, so the speedup follows 1/kf at moderate budgets.