acceptodds
Under review as a conference paper at ICLR 2027

Calibration-Guided One-Shot Vocabulary Pruning For Edge LLMs

Abstract

Modern language models often employ a large vocabulary to cover multiple languages and modalities, which makes their deployment expensive. To address this challenge, we study static vocabulary-head pruning that permanently restricts the head to a survival subset chosen once, offline, with no post-pruning fine-tuning. The de-facto criterion for this choice is token frequency, and we first establish why frequency is the wrong signal along three axes: a participation-frequency mismatch, blindness to substitutability, and the loss of rare-but-critical tokens. We introduce (Participation And Substitution Selection), a calibration-guided, training-free method built on two complementary per-token signals: participation which measures the rate at which a token enters the model's top- at decode time; and substitution cost} which is a mass-weighted logit deficit incurred by replacing a token with its nearest surviving neighbour. We then propose a two-stage selector that first uses participation to tag tokens as keep, prune or uncertain decisions, and then uses substitution cost to rank only the remaining uncertain tokens. reduces the decode-time head cost from to per token and at a keep fraction, retains nearly entire of full-head quality on multiple evaluation benchmarks as GSM8K, TrivaQA, MATH-500, etc.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vocabulary Compression, LLMs, Efficient Inference, Sparse Computation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_z2v4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

They introduce a calibration-guided, training-free method PASS built on two complementary per-token signals: participation which measures the rate at which a token enters the model's top- at decode time; and substitution cost} which is a mass-weighted logit deficit incurred by replacing a token with its nearest surviving neighbour.

Critical strengths
  1. This article is well-written, with interesting observations and a clear motivation. Lines 71-77 clearly describe the three main problems of the existing static vocabulary pruning methods.

  2. The method proposed in this article is applicable to most LLMs and is independent of the model's structure, as they all have a large LM Head.

  3. The article considered various model structures and conducted experiments on different model scales.

  4. The ablation results demonstrate the effectiveness of the method. (line 462)

Critical weaknesses

My main concerns lie in the selection of the comparison methods and the test benchmarks.

  1. The comparison method only compared the static methods. However, the dynamic methods also need to be included in the comparison (if a fair comparison can be achieved).

[1] Fuadi, Mukhlish, Adhi Dharma Wibawa, and Surya Sumpeno. "Efficient Transformer Models via Language-Aware Frequency-Based Vocabulary Pruning." IEEE Access 14 (2026): 50993-51006.

[2] Zhang, Hanling, et al. "Vocabtailor: Dynamic vocabulary selection for downstream tasks in small language models." Findings of the Association for Computational Linguistics: ACL 2026. 2026.

  1. Regarding the selection of the benchmark dataset, the dataset chosen in Table 1 is too difficult. Most small models do not have significant mathematical power (GSM8K, MATH500, AIME), and these performance tests actually make it difficult to evaluate the effectiveness of the methods. I hope you can verify by referring to the mainstream benchmarks in the field of pruning. [3-4]

[3] Chen, Xinrui, et al. "A simple linear patch revives layer-pruned large language models." Advances in Neural Information Processing Systems 38 (2026): 154484-154509.

[4] Shopkhoev, Dmitriy, et al. "Replaceme: Network simplification via depth pruning and transformer block linearization." Advances in Neural Information Processing Systems 38 (2026): 19487-19517.

  1. Does the hardware configuration affect the effectiveness of the method? If possible, please explain how different hardware affects the speedup ratio. It would be better to provide some experiments as well.

I hope the author can address my questions during the subsequent discussion stage. I will then re-evaluate the quality of the article.

Questions for authors
  1. I would like to know if the benchmark test was conducted on edge devices or on the GPU? Are there significant differences between these two?

  2. Different edge devices such as those with different processors from the Apple series, or even processors from different brands, whether they have different operator (eg. sparse GEMM) support for the model's LM head, has led to different speeds? Is it the case that in certain situations, vocabulary pruning is less important or vice versa?

  3. I would like to know how the actual speed test details were achieved, as this involves improvements in both the calculation and storage aspects. I hope to see a breakdown table of the time details.

3 comments
AuthorsOpenReview ↗

Rebuttal By Authors (Part 1)

We would like to thank you for finding our submission well-written, with interesting observations and a clear motivation. We would now like to address your concerns point-by-point as follows:


[Weakness 1] Comparison with Dynamic Pruning Methods

Thank you for the suggested references. We would first like to clarify how the two works relate to PASS:

  1. Faudi et. al (https://ieeexplore.ieee.org/document/11460136): This is a language-aware frequency-based vocabulary pruning which is very similar to our comparison baseline VT (https://arxiv.org/abs/2305.15020) included in the paper draft.

  2. VocabTailor (https://arxiv.org/abs/2508.15229): It is a hybrid static-dynamic vocabulary selection strategy for LM Head where for each input it loads the LM-head rows for the unique input tokens and combines them with a small static task-specific token set. A fair comparison can't be achieved because: (1) VocabTailor's static set is profiled per downstream task which is not deployment friendly, where as PASS's SS is task-agnostic and fixed once on a general-purpose calibration set; (2) VocabTailor's head changes per input, so the full head must remain available all-the-time with memory overhead and the active head is rebuilt per request. PASS instead ships one smaller dense matrix with no per-request work.

We respectfully like to state that dynamic pruning techniques do not form a fair baseline for PASS because:

  • (a) they do not offer any realistic memory savings (critical for edge devices) with large tokenizers as entire Head need to be stored.
  • (b) they can't directly utilize the highly optimized GEMV kernels and requires developing H/W dependent kernels to support Gather operation (key component in dynamic methods).
  • (c) they do not support batching and fallback to full LM-Head estimation in worst case because different input batch tokens might require completely different LM-Head rows.

[Weakness 2] Benchmark Selection in Table 1 is too difficult.

Thank you for raising the concern. We would respectfully like to explain the reason behind our original choice. Static head pruning affects a model only through the tokens it can emit, so free-form generation (math, code, open-domain QA) is where pruning damage appears. Log-likelihood multiple-choice tasks can mask it, as we see in our evaluation with MMLU, which is identical for every method in Table 1. We choose the tasks based on this and used % PPL Retention as a generic metric to capture language modeling capabilities of the model.


[Weakness 3 + Question 1] Were benchmarks run on edge devices or GPU? Do they differ?

We would like to clarify that the quality results (Table 1) do not depend on hardware. PASS changes which rows in the LM-Head will be used for the softmax, so we evaluate quality on GPUs. We provide our latency (Figure 8) measurements on three platforms that span different regimes:

  1. Apple M4 (MLX) having unified memory with batch-1 decode bandwidth-bound, so the head speedup closely tracks 1/kf1/k_f (≈1.75–1.9× speedup at kfk_f=0.5).

  2. NVIDIA B200 which have very high HBM bandwidth making the head GEMV short, so fixed kernel-launch overheads dominate. We therefore observe that gains vanish at kfk_f ≥ 0.6 for the smaller Qwen heads.

  3. Intel Xeon 8559C (16 threads) batch-1 decoding is again bandwidth-bound, so the speedup follows 1/kf1/k_f at moderate budgets.

AuthorsOpenReview ↗

Rebuttal By Authors (Part 2)

[Weakness 3 + Question 2] Different processors and operator support (e.g., sparse GEMM) + when does vocabulary pruning matter less?

A key deployment benefit of PASS is that it requires no sparse operator support. The pruned head is a smaller dense matrix WS∈R∣S∣×dW_S \in \mathbb{R}^{|S| \times d} which is executed by the same dense GEMV kernel as the full head. Its speedup therefore depends only on the device's dense GEMV bandwidth and compute BUT not on sparse kernel availability which varies widely across H/W vendors (unlike unstructured or per-token dynamic sparsity).

Vocabulary pruning matters less when:

  • (a) the head is a small share of per-token cost (larger models or smaller vocabularies as in Figure 1a);
  • (b) decoding is compute-bound (large batch or prefill - Figure 1b shows the head's share of bytes falling with batch size);
  • (c) the device's bandwidth is high relative to kernel overhead (B200 at kfk_f ≥ 0.6 in Figure 8b).

[Question 3] Details corresponding to our measurements

Thank you for your question. All of latency measurements use batch size 1 and BF16 weights, with KV-cached autoregressive decoding at context length 2048 tokens. We use MLX on Apple M4, PyTorch + CUDA on NVIDIA B200, and PyTorch on the Intel Xeon 8559C (16 threads). For the deployment scenario, we prune the tokenizer and the pruned head is a smaller dense matrix WS∈R∣S∣×dW_S \in \mathbb{R}^{|S| \times d} executed by the same dense GEMV kernel as the full head with no masking or sparse kernels are involved. Our details for 0.5 keep-fraction is given as:

ModelV×dV \times dHead params (M), full → PASSWeights read / token, FP16 (MB)FLOPs / token (MFLOP)Vocabulary storage, FP16 (MB)
Qwen2.5-0.5B151,936 × 896136.1 → 68.1272.3 → 136.1272.3 → 136.1272.3 → 136.1
Qwen2.5-1.5B151,936 × 1,536233.4 → 116.7466.7 → 233.4466.7 → 233.4466.7 → 233.4
Gemma-2-2B256,000 × 2,304589.8 → 294.91,179.6 → 589.81,179.6 → 589.81,179.6 → 589.8

All columns scale linearly with the head size where each parameter is read once (2 bytes in FP16) contributing 2 FLOPs (one multiply-add), and is stored once (2 bytes in FP16 as the matrix is shared between embedding and head for tied models). At batch 1 the LM-head GEMV performs about one FLOP per byte read which is far below any device's compute-to-bandwidth ratio. Therefore its latency is set by the bytes read rather than by compute and halving the head roughly halves its time on bandwidth-bound devices as in Figure 8.

I acknowledge that the additional experiments have substantially addressed my original concerns. Therefore, I will keep my score of 3 (Weak Acceptance).

Reviewer_BQXqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PASS, a calibration-guided, training-free static vocabulary-head pruning framework designed to mitigate the computational and memory bandwidth bottlenecks caused by large output heads in edge LLMs. A two-stage selector classifies tokens into confidence bands, retaining confident ones via participation and ranking the ambiguous middle band via substitution cost. Experiments across multiple models and benchmarks demonstrate promising memory and latency reductions while maintaining performance at a 50% keep fraction.

Critical strengths
  1. Addresses an important and timely deployment bottleneck, enabling direct utilization of existing dense matrix-vector kernels without per-token dynamic overhead.

  2. Top-KK participation aligns better with model behavior during decoding than surface-level corpus frequency, effectively addressing failure modes such as participation mismatch and rare-but-critical token deletion.

  3. Reports compelling downstream quality retention at a 50% keep fraction alongside notable hardware speedups.

Critical weaknesses

1.Figures 6 and 7 rely on "PPL retention," yet lack a rigorous mathematical definition. Given that deleted tokens yield zero probability mass, standard perplexity over sequences containing deleted targets is undefined. Clarification on handling re-tokenization, dropped targets, or probability floors is missing.

2.Equation 8 couples probability mass with logit gaps to nearest neighbors, but lacks direct baseline comparisons against simpler alternatives. Furthermore, the lack of cycle-prevention mechanisms in nearest-neighbor substitution chains when multiple interdependent tokens are pruned raises methodological concerns.

3.MMLU evaluations are artifacts of fixed multiple-choice option scoring, failing to reflect general vocabulary degradation. Baseline comparisons require clearer disclosure regarding special token protection rules and calibration budgets.

Questions for authors
  1. What is the exact mathematical definition of "PPL retention," and how are deleted target tokens handled during perplexity evaluation?

  2. Why does Equation 8 outperform simpler baselines, and how are circular nearest-neighbor dependencies handled?

  3. Once special tokens are uniformly protected, what is the exact performance gap between the frequency baseline and PASS?

  4. How are total memory footprint, peak memory, and end-to-end latency affected when accounting for input-output weight sharing in checkpoints like Qwen2.5?

5 comments
AuthorsOpenReview ↗

Rebuttal By Authors (Part 1)

We would first like to thank you for your time and for identifying our work as addressing an important and timely deployment bottleneck that directly reuses existing dense matrix-vector kernels without per-token dynamic overhead. We would now like to address your concerns point-by-point as follows:


[Weakness 1 + Question 1] Missing exact mathematical definition of "PPL retention"

Thank you for raising this and we agree that our definition should have been stated explicitly. We define it as follows:

Total PPL Retention. For each held-out position t∈1,…,Tt \in \\{1, \ldots, T\\} of HtestH_{\text{test}} with ground-truth next token yty_t, define the pruned next-token probability

p~S(ht)yt={pS(ht)ytyt∈S,ϵyt∉S,\tilde{p}^S(h_t)_{y_t} = \begin{cases} p^S(h_t)_{y_t} & y_t \in S, \\\\ \epsilon & y_t \notin S, \end{cases}

where pSp^S is the head renormalized over SS (Eq. 2) and ϵ=1e−10\epsilon = 1e-10. Then

PPLfull=exp⁡(−1T∑t=1Tlog⁡p⋆(ht)yt),PPLS=exp⁡(−1T∑t=1Tlog⁡p~S(ht)yt),\text{PPL}_{\text{full}} = \exp\Bigl(-\frac{1}{T}\sum_{t=1}^{T} \log p^{\star}(h_t)_{y_t}\Bigr), \qquad \text{PPL}_S = \exp\Bigl(-\frac{1}{T}\sum_{t=1}^{T} \log \tilde{p}^S(h_t)_{y_t}\Bigr),

Total PPL Retention(S)=100×PPLfullPPLS (\text{Total PPL Retention}(S) = 100 \times \frac{\text{PPL}_{\text{full}}}{\text{PPL}_S} \ (\\%).

"Total" means that no position is excluded in our PPL Retention based evaluation. When the target is pruned (yt∉Sy_t \notin S), pS(ht)yt=0p^S(h_t)_{y_t} = 0, so the position is scored with the fixed floor ϵ=1e−10\epsilon = 1e-10 which is identical for every method and keep fraction. In addition, for simplicity in PPL retention evaluation, no re-tokenization is involved and HtestH_{\text{test}} is tokenized once with the original tokenizer, and both heads are scored on the same TT positions. We will add this definition to Section 5 and to the captions of Figures 6–7.


[Weakness 2(a)] Equation 8 couples probability mass with logit gaps to nearest neighbors, but lacks direct baseline comparisons against simpler alternatives.

Equation 8 couples two per-token quantities which is mass and logit gap. To address this concern, we provide our new results which compares PASS with only expected decode-time Mass (mi=1N∑jp⋆(hj)im_i = \frac{1}{N}\sum_j p^{\star}(h_j)_i) v/s PASS with Mass * Gap (cost(i∣R)\text{cost}(i \mid R)) for Qwen2.5-1.5B:

K_fGSM8KHumanEvalHeld-out PPL Retention %
0.3PASS with Mass56.7914.496.153
PASS with (Mass * Gap)60.520.7396.924
0.5PASS with Mass60.8230.5298.426
PASS with (Mass * Gap)60.8832.9398.952

From the results, we make two observations: (a) contribution of Mass * Gap in performance is notably higher when we prune aggressive for both PPL retention as well as downstream tasks; (b) downstream tasks suffers significantly more in comparison to total PPL retention. Note that Mass only (∑jp⋆(hj)i\sum_j p^{\star}(h_j)_i) is intuitively very similar to our participation signal (π\pi) but Gap provide a complementary angle increasing the diversity of tokens within the target budget helping downstream tasks.


[Weakness 2(b) + Question 2] How are circular nearest-neighbor dependencies handled?

We would like to clarify that the nearest neighbor is considered within PASS only as a scoring proxy inside Eq. 8 and no substitution map is executed at inference. Because cost(i∣R)\text{cost}(i \mid R) uses R=Akeep∪AmidR = A_{\text{keep}} \cup A_{\text{mid}}, we fully understand reviewers concern related to mutual-nearest neighbour pair. During our method development phase, we did considered this case but our empirical analysis on Qwen2.5 tokenizer with 150K+ vocabulary, such pairs are extremely rare (<50 such pairs). Instead of complicating our method with such edge case (eg. breaking cycle with AkeepA_{\text{keep}} only, or a new rescoring pass over SS), we choose to trade them for simplicity of our algorithm.

AuthorsOpenReview ↗

Rebuttal By Authors (Part 2)

[Weakness 3] MMLU evaluations are artifacts of fixed multiple-choice option scoring, failing to reflect general vocabulary degradation.

We agree that MMLU scored by option-letter log-likelihood is insensitive to head pruning. Our evaluation in Table 1 found no performance degradation by any of the head pruning method at any keep-fraction ratio. We have explicitly flagged this in the footnote to Table 1 (Page 8 in draft).


[Question 3] Once special tokens are uniformly protected, what is the exact performance gap between the frequency baseline and PASS?

Thank you for suggesting this ablation (also raised by Reviewer BQXq, Q3). For this rebuttal, we define a whitelist set W\mathcal{W} = { special/control tokens ∪ byte-fallback pieces ∪ digits } which is default preserved before we prune using frequency. We report the performance with keep-fraction (k_f = 0.3) as follows for Qwen2.5-1.5B:

GSM8KHumanEval
Frequency02.44
Frequency + W\mathcal{W}27.9211.23
PASS60.520.73
PASS + W\mathcal{W}60.520.92

[Question 4] How are total memory footprint, peak memory, and end-to-end latency affected when accounting for input-output weight sharing in checkpoints like Qwen2.5?

Thank you for raising this question. For total memory footprint, peak memory and end-to-end latency, PASS safely allows pruning the tokenizer itself where the tokens outside SS are removed from the vocabulary, so the corresponding rows are deleted from both the input embedding and the LM head. We have very carefully examined the impact of pruning the tokenizer while preserving the byte-fallback pieces so every input string remains encodable with the pruned tokenizer.

In our deployment settings with a practical keep-fraction 0.5, we do not observe any performance degradation (under < 1.0 points with pruning tied head for both input and output vs tokenization with full-head at input) on downstream tasks. Moreover, since the tokenizer and the head are pruned together, the removed rows are deleted from every vocabulary indexed matrix. So the effect of PASS on total memory footprint, peak memory and end-to-end latency is the same whether or not a model ties its input embedding and LM head (input-output weight sharing).

AuthorsOpenReview ↗

Authors Message to Reviewer BQXq

Dear Reviewer BQXq,

We thank you for your time to review our work and your constructive comments to improve it, and we really hope to have a further discussion with you to see if our response solves your concerns.

We have replied to the important points raised by you such as comparison with mass-only CHES, frequency+special tokens baseline, definition of ppl retention, in our rebuttal response. Since the author-reviewer discussion period has now reached towards its end, we will deeply appreciate if you could check our response to your review comments soon. This way, if you have further questions and comments, we can still reply before the author-reviewer discussion period ends.

We again thank you for your time and efforts.

Best, Authors

Reviewer_js2jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

For small models, a large-vocabulary output head (LM-Head) incurs excessive overhead during on-device decoding. The authors propose PASS (Participation And Substitution Selection), a method to select a fixed subset of the vocabulary offline, effectively and permanently reducing the size of the output head without requiring any fine-tuning. Experiments cover Qwen2.5-0.5B, Qwen2.5-1.5B, and Gemma-2-2B with Nemotron-CC-v2 calibration.

Critical strengths
  1. Well-motivated, practically relevant problem.
  2. Sensible core insight.
  3. Simple, training-free, interpretable method.
Critical weaknesses
  1. Only three models at or below 2B parameters are evaluated. The introduction extensively motivates multimodal codebooks, but no multimodal experiments are included.
  2. Some key baselines are missing. i.e. ranking by 1N∑jp⋆(hj)i\frac{1}{N}\sum_j p^\star(h_j)_i. This is the natural ablation of Eq. 8 without the logit gap, and it would isolate whether substitutability actually matters. The difference between the "Frequency" and "VT (Ushio et al., 2023b)" baselines is not specified. Ushio et al. 2023a and 2023b also appear to be the same paper listed twice.
Questions for authors
  1. Can you add a "frequency + whitelist" baseline that always keeps special/control tokens, byte fallbacks, and digits? How much of the GSM8K and HumanEval gap at kf=0.3k_f=0.3 remains?
  2. How does PASS with α=1\alpha=1, K=1K=1 compare to VocabTrim? Please isolate the contributions of larger KK and of the cost refinement over VocabTrim.
  3. How is "Total PPL Retention" computed when the held-out target token is pruned?
  4. How was α≈0.65\alpha\approx0.65 selected? Was it tuned on the same held-out set used for reporting?
3 comments
AuthorsOpenReview ↗

Rebuttal By Authors (Part 1)

We would first like to thank you for your time and for identifying our work as addressing a well-motivated and practically relevant problem, built on a sensible core insight, with a simple, training-free and interpretable method. We would now like to address your concerns point-by-point as follows:


[Weakness 1] Only three models ≤2B are evaluated and no multimodal experiments

Thank you for raising this concern. We would respectfully like to clarify that the focus on small models is deliberate. Head Slack is a property of large-vocabulary small models, where the output projection occupies a large share of per-token cost (6–28% of parameters). As models scale, the head grows only linearly with the hidden dimension while the backbone grows roughly with its square (and also with depth), so the head's share of per-token cost falls. For example, the Qwen2.5-7B head which is 7% of the model unlike Qwen2.5-0.5B with 28%, in that case the absolute benefit of any head-pruning method will shrink.

During the rebuttal, we also conducted an additional experiment on Qwen2.5-3B and we report our new results as follows:

ModelK_fMethodGSM8KHumanEvalHeld-out PPL Retention %
Qwen2.5-3B1Full Head74.1739.32100
0.5Frequency72.5829.8696.02
PASS72.0237.9697.44
0.3Frequency22.615.3893.5
PASS71.5825.7495.87

Regarding multimodal heads, we would like to clarify that our multimodal discussion in our introduction is meant to highlight the trend of growing output vocabularies, among which unified visual codebooks can also be one component. Extending PASS to unified multimodal vocabularies is non-trivial and requires additional method innovation. Text tokens and visual-codebook tokens can operate on very different scales of both participation and substitution cost, which is out of the scope of this submission. Bridging the two signals across modalities, for example through modality-aware normalization or allocation, is an interesting direction that we leave to future work. We commit to rephrasing the introduction accordingly and to stating this explicitly in the Limitations and Future Work section.


[Weakness 2] Additional ablation on mass-only baseline 1N∑jp⋅(hj)i\frac{1}{N} \sum_j p \cdot (h_j)i in ablation + Clarification regarding Frequency and VT

Thank you for suggesting this ablation. We provide our new results which compares PASS with only expected decode-time Mass (mim_i) = 1N∑jp⋅(hj)i\frac{1}{N} \sum_j p \cdot (h_j)i v/s PASS with Mass * Gap (cost(i∣R)\text{cost}(i \mid R)) for Qwen2.5-1.5B:

K_fGSM8KHumanEvalHeld-out PPL Retention %
0.3PASS with Mass56.7914.496.153
PASS with (Mass * Gap)60.520.7396.924
0.5PASS with Mass60.8230.5298.426
PASS with (Mass * Gap)60.8832.9398.952

From the results, we make two observations:

(a) contribution of Mass * Gap in performance is notably higher when we prune aggressive for both PPL retention as well as downstream tasks;

(b) downstream tasks suffers significantly more in comparison to total PPL retention. Note that Mass only ( 1N∑jp⋅(hj)i\frac{1}{N} \sum_j p \cdot (h_j)i ) is intuitively very similar to our participation signal (π\pi) but Gap provide a complementary angle increasing the diversity of tokens within the target budget helping downstream tasks.

For clarification regarding "Frequency" and "VT" which both look at frequency of tokens, in our baseline implementation of VT, we first remove all non-en tokens using the mC4 dataset (https://huggingface.co/datasets/vocabtrimmer/mc4_validation), then we use default frequency count as the strategy to meet the target budget. We will make sure to incorporate our baseline design details within Appendix and also fix citation issues.

AuthorsOpenReview ↗

Rebuttal By Authors (Part 2)

[Question 1] "Frequency + whitelist" baseline: how much of the GSM8K/HumanEval gap at kfk_f=0.3 remains?

Thank you for suggesting this ablation (also raised by Reviewer BQXq, Q3). For this rebuttal, we define a whitelist set W = { special/control tokens ∪ byte-fallback pieces ∪ digits } which is default preserved before we prune using frequency. We report the performance with keep-fraction (kfk_f) as follows for Qwen2.5-1.5B:

GSM8KHumanEval
Frequency02.44
Frequency + W27.9211.23
PASS60.520.73
PASS + W60.520.92

[Question 2] PASS with α\alpha = 1 and K = 1, compare to VocabTrim?

PASS with α\alpha = 1 and K=1 reduces to using Participation Only signal which only looks at greedy Top-1 token decoded with a given hidden state. This setting is quite aggressive and requires a large calibration corpus to reduce auto-prune set of tokens (Stage 1 in Section 4.3). However, we run PASS with α\alpha = 1 and K=1 for Qwen2.5-1.5B and we report our results as follows:

K_fGSM8KHumanEval
0.3VocabTrim10.0415.24
PASS with α\alpha = 1 and K=151.318.46
PASS60.520.73
0.5VocabTrim60.2931.71
PASS with α\alpha = 1 and K=160.0231.78
PASS60.8832.93

[Question 3] How is "Total PPL Retention" computed when the held-out target is pruned?

Thank you for raising this (also raised by Reviewer BQXq) and we agree that our definition should have been stated explicitly. We define it as follows:

Total PPL Retention. For each held-out position t∈1,…,Tt \in \\{1, \ldots, T\\} of HtestH_{\text{test}} with ground-truth next token yty_t, define the pruned next-token probability

p~S(ht)yt={pS(ht)ytyt∈S,ϵyt∉S,\tilde{p}^S(h_t)_{y_t} = \begin{cases} p^S(h_t)_{y_t} & y_t \in S, \\\\ \epsilon & y_t \notin S, \end{cases}

where pSp^S is the head renormalized over SS (Eq. 2) and ϵ=1e−10\epsilon = 1e-10. Then

PPLfull=exp⁡(−1T∑t=1Tlog⁡p⋆(ht)yt),PPLS=exp⁡(−1T∑t=1Tlog⁡p~S(ht)yt),\text{PPL}_{\text{full}} = \exp\Bigl(-\frac{1}{T}\sum_{t=1}^{T} \log p^{\star}(h_t)_{y_t}\Bigr), \qquad \text{PPL}_S = \exp\Bigl(-\frac{1}{T}\sum_{t=1}^{T} \log \tilde{p}^S(h_t)_{y_t}\Bigr),

Total PPL Retention(S)=100×PPLfullPPLS (\text{Total PPL Retention}(S) = 100 \times \frac{\text{PPL}_{\text{full}}}{\text{PPL}_S} \ (\\%).

"Total" means that no position is excluded in our PPL Retention based evaluation. When the target is pruned (yt∉Sy_t \notin S), pS(ht)yt=0p^S(h_t)_{y_t} = 0, so the position is scored with the fixed floor ϵ=1e−10\epsilon = 1e-10 which is identical for every method and keep fraction. In addition, for simplicity in PPL retention evaluation, no re-tokenization is involved and HtestH_{\text{test}} is tokenized once with the original tokenizer, and both heads are scored on the same TT positions. We will add this definition to Section 5 and to the captions of Figures 6–7.


[Question 4] How was α=0.65 selected?

In our work, we have selected α directly using the calibration corpus (~1M tokens) used for estimation of participation and substitution signals. Our % PPL Retention based evaluation results (Figure 3, Figure 4a, and Figure 6) use a disjoint 20M held-out set (HtestH_{\text{test}}) and we have explicitly mention that either in our results figure/figure caption. We promise to incorporate this detail in Section - Ablation Studies on PASS Hyperparameters.

AuthorsOpenReview ↗

Author Message to Reviewer js2j

Dear Reviewer js2j,

We thank you for your time to review our work and your constructive comments to improve it, and we really hope to have a further discussion with you to see if our response solves your concerns.

We have replied to the important points raised by you such as comparison with Qwen2.5-3B model, ablations on "frequency+whitelist", "PASS with α\alpha= 1 and K=1", "mass-only", and definition of ppl retention, in our rebuttal response. Since the author-reviewer discussion period has now reached towards its end, we will deeply appreciate if you could check our response to your review comments soon. This way, if you have further questions and comments, we can still reply before the author-reviewer discussion period ends.

We again thank you for your time and efforts.

Best, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.