acceptodds
Under review as a conference paper at ICLR 2027

PAATRA: Parameter-Allocation-Aware Training for Small Language Models

Abstract

Small language models (SLMs) often inherit tokenizers designed for much larger model families. In our 64M-parameter GPT-2-style student using Qwen2.5's 152K-token vocabulary, 81.8% of the parameters are token embeddings. We study this vocabulary–transformer allocation with PAATRA, a pipeline that retains a frequency-ranked subset of the teacher vocabulary, distills from the teacher over that subset, and reinvests the freed parameters into transformer width, subject to an approximately fixed total budget. Distilling from Qwen2.5-0.5B on WikiText-103 with a shared training recipe, students with 47K and 20K tokens reach mean bits per byte (BPB) of and over three seeds, 4.9% below the full-vocabulary baseline (). A 10K vocabulary is slightly worse than the baseline (), and a 20K vocabulary without reinvestment is worse still (1.4322). Two results qualify this gain. First, in a single-seed learning-rate sweep, the full-vocabulary baseline tuned to reaches 1.3455, matching the best 20K student. Second, small, genuine BPE/Unigram tokenizers trained with plain cross-entropy under a different protocol achieve lower BPB (1.19–1.27) than all distilled students. We therefore present PAATRA as a controlled measure of the allocation effect under a fixed recipe, for a single teacher and a single corpus. We do not claim that reduced vocabularies win under per-configuration tuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Small language models, tokenizer vocabulary size, parameter allocation, embedding parameters, parameter-efficient training, knowledge distillation, vocabulary pruning, model capacity, transformer scaling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_TduSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

PAATRA targets a specific pathology of SLMs distilled from large teachers: inheriting the teacher's tokenizer wastes most of the parameter budget on embeddings (a 64M GPT-2-style student with Qwen2.5's 152K vocab spends ~81.8% on embeddings). The method keeps the top-K most frequent teacher tokens (K in {20K, 47K, 64K}), distills over this subset, and reinvests freed parameters into a wider transformer at a fixed budget. Main result: 4.9% BPB improvement over the full-vocab baseline on WikiText-103 across 3 seeds with Qwen2.5-0.5B as teacher.

Critical strengths

Unusually honest - both nullifying caveats are surfaced in the abstract, code and per-seed numbers are released. The parameter-matched controlled setup with 3 seeds is clean, and the embedding-share audit (Table 2, e.g., Gemma 3 270M spending ~63% on embeddings) is a useful community reference.

Critical weaknesses
  1. The paper's own results refute its central claim. Table 6: a full-vocab baseline tuned to LR 1e-3 reaches BPB 1.3455, matching the best PAATRA-20K student (1.3448). Table 7: genuine BPE/Unigram tokenizers at 20K-64K with plain CE achieve BPB 1.186-1.270, better than every PAATRA variant. The crucial matched-protocol comparison is explicitly "remains to be done." Vocabulary pruning with a real subword tokenizer dominates the proposed method.

  2. Toy scale and protocol issues that cannot support the claims: 64M params / 30.7M tokens / WikiText-103 only / single teacher; no validation split (selection effectively on test set); the surrogate-ID protocol does not yield a genuine tokenizer likelihood; the reinvestment ablation confounds parameter count with allocation (18.5M vs 64M).

  3. No efficiency gain and limited novelty. Table 10 shows PAATRA requires more compute per token than the full-vocab baseline. The core idea (prune teacher vocabulary to a frequency-ranked subset, distill over it) is well-covered by TextPruner, VocabTailor, VocabTrim, Phan et al. 2026, Tao et al. 2024.

Questions for authors
  1. Can you provide the matched-protocol comparison - PAATRA vs. a genuine 20K/47K BPE/Unigram tokenizer with plain CE at the same budget and LR?

  2. If every configuration receives its own multi-seed LR sweep, does the reallocation gain survive? Table 6 suggests a single LR change already erases most of it.

  3. Was any selection performed on the test set, given no validation split is described?

3 comments
AuthorsOpenReview ↗

Response to reveiwer TduS

We thank the reviewer for the careful evaluation, constructive criticism, and insightful questions. The concerns identify substantive limits that our revised claims and validation must reflect.

CW-01: Interpretation of existing results

Reviewer Concern

The paper's own results refute its central claim. Table 6: a full-vocab baseline tuned to LR 1e-3 reaches BPB 1.3455, matching the best PAATRA-20K student (1.3448). Table 7: genuine BPE/Unigram tokenizers at 20K-64K with plain CE achieve BPB 1.186-1.270, better than every PAATRA variant. The crucial matched-protocol comparison is explicitly "remains to be done." Vocabulary pruning with a real subword tokenizer dominates the proposed method.

Response

We agree that the evidence does not establish a tuned-performance advantage. Table 6's Full score (1.3455) nearly matches K20K (1.3448); the 4.9% result is specific to the shared recipe. Table 7 is concerning, but its different data, exposure, optimizer schedule, parameter counts and scored bytes prevent a controlled dominance claim (Appendix C). We will run matched controls and retain unfavorable outcomes. If the gain disappears, we will retract the advantage claim.

CW-02: Scope and protocol

Reviewer Concern

Toy scale and protocol issues that cannot support the claims: 64M params / 30.7M tokens / WikiText-103 only / single teacher; no validation split (selection effectively on test set); the surrogate-ID protocol does not yield a genuine tokenizer likelihood; the reinvestment ablation confounds parameter count with allocation (18.5M vs 64M).

Response

We accept these limitations. We will use validation-only tuning, audit OOV targets and contexts, and distinguish surrogate scores from full-text coding scores. The 18.5M ablation cannot isolate allocation; we will add a budget-matched depth alternative. Larger-budget, additional-corpus and second-teacher checks are planned, subject to compute availability; absent evidence, conclusions will remain restricted to the original setting. Historical test reuse cannot be undone.

CW-03: Efficiency and novelty

Reviewer Concern

No efficiency gain and limited novelty. Table 10 shows PAATRA requires more compute per token than the full-vocab baseline. The core idea (prune teacher vocabulary to a frequency-ranked subset, distill over it) is well-covered by TextPruner, VocabTailor, VocabTrim, Phan et al. 2026, Tao et al. 2024.

Response

We do not claim inference savings: Table 10 reports higher FLOPs/token and batch-1 CPU latency for the reallocated models. We will report measured latency, throughput, memory and total training cost separately. We will revise the related-work comparison, including Tao et al.'s vocabulary/non-vocabulary allocation and Phan et al.'s exact cross-tokenizer scoring. Our gathered-logit surrogate protocol is not that exact algorithm. Combining known components alone does not establish novelty.

QA-01: Matched tokenizer comparison

Reviewer Concern

Can you provide the matched-protocol comparison - PAATRA vs. a genuine 20K/47K BPE/Unigram tokenizer with plain CE at the same budget and LR?

Response

We plan BPE and Unigram controls at 20K and approximately 47K with corrected tied-parameter counting, identical raw training examples, update counts, optimizer schedules, and shared-LR results alongside validation-tuned results. We will score identical held-out bytes without dropped targets. Original PAATRA's lossy surrogate score cannot be ranked as raw-text likelihood; any full-support extension will be labelled a new variant, not a validation of the original method.

QA-02: Multi-seed LR tuning

Reviewer Concern

If every configuration receives its own multi-seed LR sweep, does the reallocation gain survive? Table 6 suggests a single LR change already erases most of it.

Response

This is unresolved. We will sweep every configuration over {1e-4,3e-4,6e-4,1e-3,3e-3} using three tuning seeds, select by mean validation score, and rerun selected settings with three fresh seeds. A predeclared boundary extension will check truncated optima. We will report paired differences and uncertainty, without equating a small observed gap with statistical equivalence.

QA-03: Test-set selection

Reviewer Concern

Was any selection performed on the test set, given no validation split is described?

Response

Yes. The manuscript's Discussion and the repository README explicitly state that all selection and evaluation used the test split. The LR sweep compared test scores; these results must be described as exploratory. We will audit which decisions used test feedback, move all new selection to official validation, and freeze settings before test reporting. Reusing the old test set does not make it pristine; an independently held-out corpus will provide a separate check. No new results are claimed here.

Reviewer_TduSOpenReview ↗

Response to Authors

Thank you for the candid response. However, the matched tokenizer comparison, independently tuned multi-seed results, and budget-matched allocation ablation remain planned rather than completed. The acknowledged test-set selection and limitations of surrogate scoring also undermine the evidence supporting the reported gains. While the proposed experiments are relevant, the current response does not establish a reliable performance advantage or sufficiently distinguish the contribution from prior work.

We thank the reviewer. The initial rebuttal is already posted; this update reports completed evidence (revised-paper changes: purple = your points only, orange = shared). Your central point is correct as the paper stood: the claim does not survive a tuned baseline. We now state that.

W1 / Q2 (own results refute the claim; per-configuration LR sweeps). We tuned each configuration's LR on a validation split (grid 1e-4..3e-3, tuning seeds 11/22, selection frozen by hash before test) and ran 3 fresh test seeds. Revised Table 5 (§6.2), grid in Table 10 (App. C).

ConfigLRBPB (legacy)Common-event NLL
Full1e-31.3430±.00404.019±.010
K50K3e-41.3438±.00043.956±.005
K20K3e-41.3453±.00274.000±.006
Full−K20K−0.0024 [−0.0114, 0.0067]0.019 [0.006, 0.031]

The 4.9% BPB gain disappears (intervals span zero): it was a baseline-tuning effect. A small consistent-target advantage remains (0.02-0.06 nats, n=3), and we make no claim of beating a tuned baseline. Abstract and conclusion are rewritten.

Q1 (matched BPE/Unigram with plain CE). Done at ~64M, same optimizer, steps and data; LR validation-selected (1 tuning seed); final seeds 101/202 (Unigram-47K seed 101 only). Revised Table 8 (§6.4), lossless test BPB on raw text:

ModelKBPB
Full (teacher IDs)151,6651.667±.005
BPE 20K / 47K20K / 47K1.706±.006 / 1.645±.001
Unigram 20K / 47K20K / 47K1.633±.001 / 1.614 (n=1)

Your point is partly confirmed: Unigram 20K/47K and BPE 47K beat the full vocabulary by 0.02-0.05 BPB, BPE 20K is 0.04 worse. A retrained lossless tokenizer is competitive; PAATRA's subset-of-teacher-vocabulary pipeline is not shown to beat it. The original submission's Table 7 numbers (BPB 1.19-1.27) used different data, optimizer and parameter counts and are kept only as unmatched context (App. E).

Q3 (test-set selection). In the original submission there was no validation split, and the LR sweep was effectively read on test. All new selections use a validation split only; selection files are hashed before test evaluation. The original sweep is retained and labelled as a shared-recipe measurement (§6.5).

W2 (scale, surrogate protocol, confounded ablation). The surrogate-ID score is not a raw-text likelihood: test OOV rates are 1.06% (K50K), 4.25% (K20K), 9.95% (K10K), and substituted targets carry 3.0/8.9/22.6% of NLL (Revised Table 4, §6.1); we now report a consistent-target metric and the lossless controls above. The 18.5M ablation is retained as a pruning-only reference; a budget-matched seen-only control (27.8M, B1 4.028 vs Full 4.019) and a depth-matched K20K (64.5M, 4.019) are in Revised Table 7 (§6.3). Scale, second teacher and additional corpora (256M students, FineWeb-Edu): NOT RUN; claims are limited to one teacher, 64M, WikiText-103.

W3 (efficiency, novelty). We ran no new efficiency measurement and make no efficiency claim; Table 12 (App. D) is analytical and shows no compute saving. Prior art (TextPruner, VocabTailor, VocabTrim, Phan et al. 2026, Tao et al. 2024) is now cited and tabulated (Table 1, §2); K50K equals seen-only pruning plus width.

Reviewer_5ELvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the allocation of parameters between vocabulary embeddings and Transformer blocks in small language models. PAATRA retains frequent tokens from the teacher’s vocabulary, trains students through distillation, and reallocates the saved embedding parameters to increase model width. Under an approximately 64M parameter budget, the 20K and 47K vocabulary configurations achieve about 4.9% lower reported BPB than the full-vocabulary baseline across three seeds. However, tuning the baseline’s learning rate largely closes the gap with K20K. The main evaluation also replaces OOV tokens with a surrogate token, leaving comparability across vocabularies unresolved.

Critical strengths
  1. Large vocabularies can consume most of a small model’s parameter budget. The paper provides a concrete analysis of this issue and compares models with approximately matched total parameter counts.

  2. The main configurations are evaluated across three seeds, and the learning-rate sweep shows that different parameter allocations respond differently to the training recipe.

Critical weaknesses
  1. Different vocabularies replace different sets of OOV tokens with a single existing token, changing both the prediction targets and input contexts. Using the same raw-byte denominator does not remove this difference. Without OOV rates and an analysis of their effect on scoring, the reported 4.9% improvement cannot be confidently attributed to parameter reallocation.

  2. Tuning the full-vocabulary baseline’s learning rate largely closes the gap with K20K, showing substantial dependence on the training recipe. The genuine-tokenizer baselines also differ in training data exposure, optimization settings, and parameter counts, so they do not isolate the effect of parameter allocation. The authors acknowledge these limitations, but the current evidence does not yet establish robust design guidance.

Questions for authors
  1. What are the test-set OOV rates for each vocabulary? How much loss comes from retained versus surrogate targets? Is there a comparison using consistent prediction targets that supports the reported performance ranking?

  2. How does the teacher perplexity calculation in Table 4 handle OOV targets and contexts? Renormalization alone should increase the probabilities of fixed retained targets. Which additional processing steps account for the reported perplexity increases?

  3. After separately tuning Full, K50K, and K20K on an independent validation set, do multi-seed results still support a benefit from parameter reallocation?

3 comments
AuthorsOpenReview ↗

Response to reviewer 5ELv

Thank you for the careful evaluation, constructive criticism, and insightful questions. We address each concern below; all additional evaluations described here are planned, not completed.

CW-01: OOV replacement and comparability

Reviewer Concern

Different vocabularies replace different sets of OOV tokens with a single existing token, changing both the prediction targets and input contexts. Using the same raw-byte denominator does not remove this difference. Without OOV rates and an analysis of their effect on scoring, the reported 4.9% improvement cannot be confidently attributed to parameter reallocation.

Response

We agree that equal byte denominators do not make different targets and contexts equivalent. Section 5.2 reports a 4.9% reduction under surrogate scoring, not an isolated reallocation gain. Appendix A already distinguishes this score from raw-text likelihood; Section 4's comparability statement is too strong. We will qualify it and the allocation claims, measure OOV coverage and loss contributions, and evaluate common mapped targets/contexts and unchanged-context retained targets. The common mapping remains lossy; matched lossless-tokenizer controls will test performance on the original text.

CW-02: Optimization and baseline controls

Reviewer Concern

Tuning the full-vocabulary baseline’s learning rate largely closes the gap with K20K, showing substantial dependence on the training recipe. The genuine-tokenizer baselines also differ in training data exposure, optimization settings, and parameter counts, so they do not isolate the effect of parameter allocation. The authors acknowledge these limitations, but the current evidence does not yet establish robust design guidance.

Response

Section 5.4 reports Full at 1.3455 versus K20K at 1.3448 after the single-seed LR sweep, so robustness to tuning is unestablished. We will tune Full, K50K, and K20K separately on validation data with equal search budgets, extend the LR range beyond 1e-3, freeze selection, and evaluate multiple seeds. We will also run genuine-tokenizer CE controls matched in parameter budget, raw training exposure, optimization protocol, and scored bytes. These controls test allocation within a lossless CE setting; they do not isolate distillation's contribution.

QA-01: Coverage, loss decomposition, and common targets

Reviewer Concern

What are the test-set OOV rates for each vocabulary? How much loss comes from retained versus surrogate targets? Is there a comparison using consistent prediction targets that supports the reported performance ranking?

Response

The manuscript does not provide these OOV rates or loss components, and no common-target ranking is established. We will report token- and byte-weighted OOV rates for every main vocabulary, retained/surrogate NLL sums and conditional means, and preceding-context OOV contamination. We will use the same target positions and common mapped histories for all primary models, with probability aggregation onto one alphabet, plus an unchanged-context retained-target diagnostic. Neither diagnostic alone establishes full raw-text performance.

QA-02: Teacher-screen perplexity

Reviewer Concern

How does the teacher perplexity calculation in Table 4 handle OOV targets and contexts? Renormalization alone should increase the probabilities of fixed retained targets. Which additional processing steps account for the reported perplexity increases?

Response

For fixed retained y and context c, pK(y∣c)=pT(y∣c)/ZK(c)p_K(y|c)=p_T(y|c)/Z_K(c), hence NLLK−NLLT=logZK(c)<=0NLL_K-NLL_T=log Z_K(c)<=0 (Appendix A). Thus renormalization alone cannot explain Table 4. The attached package omits the screen script and stores Table 4 as transcribed values, so its additional processing cannot be established from these files. We will reconstruct the implementation and separately audit target substitution, context substitution, masking, and averaging. We will correct or withdraw the screen's interpretation if its provenance cannot be established.

QA-03: Independently tuned multi-seed comparison

Reviewer Concern

After separately tuning Full, K50K, and K20K on an independent validation set, do multi-seed results still support a benefit from parameter reallocation?

Response

This remains unknown: Table 6 is single-seed, omits K50K, and was evaluated without a separate validation split (Section 6). We will select each configuration's LR on the official validation split and run three fresh evaluation seeds with matched training exposure. Selection will use a common-target diagnostic, with legacy scores reported separately. We will report paired differences and uncertainty, including ties or reversals, and retain the qualification that prior test use prevents treating the benchmark as wholly untouched.

Reviewer_5ELvOpenReview ↗

Thank you for the clarifications

Thank you for your response. You have clarified the limitations of surrogate scoring and proposed relevant additional experiments. However, OOV statistics, evaluations using common prediction targets, and multi-seed results after independently tuning each configuration have not yet been provided. The actual computation procedure for Table 4 also remains unclear. Therefore, the current response does not yet resolve the main concerns about the comparability of the reported performance and whether the gains can be attributed to parameter reallocation.

We thank the reviewer. The initial rebuttal is already posted; this update reports completed experiments (revised-paper changes: blue = your points only, orange = shared). Test split, 3 fresh seeds (101/202/303) unless noted.

CW-01 / Q1 (OOV, consistent targets). Test OOV rate (tokens/bytes): K50K 1.06/0.99%, K20K 4.25/5.55%, K10K 9.95/14.13%. Substituted targets carry 3.0/8.9/22.6% of legacy NLL (11.3/8.4/9.7 nats vs 3.9/3.8/3.7 on retained targets). On a common event (B1, out-of-set tokens collapsed in targets and contexts) the original-checkpoint ordering is Full 4.168 > K20K 3.997 > K50K 3.937 nats; on the clean slice K20K is lowest (3.832). The 4.9% is now described as a surrogate-score difference. Revised Table 4 (§6.1).

CW-02 / Q3 (tuning). We tuned LR per configuration on validation (grid 1e-4..3e-3, tuning seeds 11/22, frozen selection hash) before touching test seeds.

ConfigLRA: BPBB: common NLLC: clean NLL
Full1e-31.3430±.00404.019±.0103.909±.008
K50K3e-41.3438±.00043.956±.0053.872±.004
K20K3e-41.3453±.00274.000±.0063.835±.005
Full−K20K−0.0024 [−0.0114, 0.0067]0.019 [0.006, 0.031]0.074 [0.042, 0.107]
Full−K50K−0.0009 [−0.0112, 0.0095]0.064 [0.026, 0.101]0.037 [0.005, 0.069]

Revised Table 5 (§6.2); grid in Table 10 (App. C). The legacy BPB gap vanishes (intervals span zero), confirming your reading that the 4.9% was largely a baseline-tuning effect. A small consistent-target advantage remains (0.02-0.06 nats, n=3, descriptive intervals), but we do not claim it as established benefit and we claim no advantage over a tuned baseline. Gap at the shared LR was 0.169 nats.

Baselines that isolate allocation. We trained lossless controls (plain CE, same ~64M budget, optimizer and steps; BPE/Unigram trained on the same data; LR chosen on validation, 1 tuning seed; 2 final seeds, Unigram-47K 1 seed). Revised Table 8 (§6.4):

ModelKLossless BPB
Full (teacher IDs)151,6651.667±.005
BPE 20K / 47K20K / 47K1.706±.006 / 1.645±.001
Unigram 20K / 47K20K / 47K1.633±.001 / 1.614 (n=1)

Reduced tokenizers can beat the full vocabulary, but ordering is not monotone in size and these runs are not distilled, so they do not support the PAATRA construction itself. Budget-matched PAATRA controls (seen-only at d=344: 4.028 vs Full 4.019; depth-matched 4.019) are in Revised Table 7 (§6.3).

Q2 (Table 4 teacher perplexity). You are right: restriction on fixed retained targets cannot raise perplexity (NLL_K − NLL_T = log Z_K ≤ 0). A new teacher-only factorial confirms ratios 0.976 (47K), 0.945 (20K), 0.899 (10K), 0.855 (5K) on identical positions; increases arise only from target/context substitution. We could not recover the original script and no variant reproduces the old Table 4, so its interpretation is withdrawn. §6.6, Appendix F.

Limitations. One teacher, one corpus, small scale; controls use reduced grids and 2 seeds; the historical test split was consulted before validation selection was added (the new selection used validation only); Table 4's original processing is unrecovered.

Reviewer_d828OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

PAATRA shrinks the teacher vocabulary and reallocates embedding parameters to Transformer width under a fixed student budget. On WikiText-103, intermediate vocabularies achieve 4.9% lower reported BPB under a shared training recipe.

Critical strengths

S1. The main configurations approximately match total parameter budgets.

Tables 3 and 5 compare models within ±1% of a 64M-parameter target and report three random seeds. This controls an important variable and documents variability, providing a useful experimental foundation for investigating vocabulary–Transformer allocation.

Critical weaknesses

Compared with the datasets used to pretrain large language models, the WikiText-103 dataset used in this paper is very small. A smaller vocabulary is therefore an obvious choice, and it is unsurprising that reallocating embedding parameters to other Transformer components improves performance. Unfortunately, I do not see any novelty in this work.

Questions for authors

The small training corpus covers only a fraction of the teacher’s vocabulary. A smaller vocabulary can cover the observed training tokens. Pruning unused vocabulary entries frees embedding parameters for Transformer blocks. Performance gains from increased Transformer capacity are unsurprising. What is novel about PAATRA’s vocabulary pruning and parameter reallocation?

3 comments
AuthorsOpenReview ↗

Response to reveiwer d828

Thank you for the careful evaluation, constructive criticism, and direct question about novelty. We appreciate the recognition of the parameter matching and multi-seed evaluation, and address the limitations of our contribution below.

CW-01: Small-corpus explanation and scope of the contribution

Reviewer Concern

Compared with the datasets used to pretrain large language models, the WikiText-103 dataset used in this paper is very small. A smaller vocabulary is therefore an obvious choice, and it is unsurprising that reallocating embedding parameters to other Transformer components improves performance. Unfortunately, I do not see any novelty in this work.

Response

We agree that the limited training corpus makes vocabulary reduction a natural baseline. Our experiment uses only the first 80,000 qualifying WikiText-103 training rows, not the complete training split. K50K retains 47,181 tokens, essentially all types observed in that subset (§4, Table 3). Its result therefore does not establish an advantage beyond discarding unobserved entries and increasing width.

The intended contribution is a reproducible study of the vocabulary-width tradeoff in a teacher-aligned, approximately 64M student family, including where the apparent benefit fails, rather than the invention of vocabulary pruning or the observation that wider Transformers can help. The reported non-monotonic sweep (Table 5: Full 1.4147, K50K 1.3452, K20K 1.3460, K10K 1.4210 mean BPB) is descriptive evidence under one recipe. It does not establish novelty by itself: separately tuning Full nearly closes the gap (Table 6), and surrogate scoring changes targets and contexts.

We will audit observed token types and held-out coverage as training data grow, then compare Full, seen-token-only pruning, and frequency pruning on nested samples of a larger independent web-text corpus. We will hold training exposure fixed within that scaling comparison, tune on validation data, and report three final seeds with common-event scoring and a lossless-tokenizer control. These are planned tests of the small-corpus explanation, not completed evidence of generality.

QA-01: Novelty beyond pruning unseen tokens and reallocating parameters

Reviewer Concern

The small training corpus covers only a fraction of the teacher’s vocabulary. A smaller vocabulary can cover the observed training tokens. Pruning unused vocabulary entries frees embedding parameters for Transformer blocks. Performance gains from increased Transformer capacity are unsurprising. What is novel about PAATRA’s vocabulary pruning and parameter reallocation?

Response

The individual operations are established techniques; our claim should concern the specific experimental integration and resulting evidence, not first discovery of vocabulary allocation. TextPruner (Yang et al., 2022, §3.1) already removes corpus-unobserved tokens. Tao et al. (2024, §§4.1-4.3) explicitly study vocabulary/non-vocabulary allocation with training data and compute. We will correct Table 1's “No internal split” characterization of vocabulary-aware scaling and make this overlap explicit in the contribution statement.

PAATRA instantiates frequency-selected teacher subsets, gathered-logit distillation and width reinvestment at fixed depth and approximately fixed total parameters (§3). Those implementation choices define the setting; combining them does not alone establish a substantive new method. We will compare a literal “keep every observed token” baseline with and without reinvestment against Full and K20K, using matched optimization and consistent scoring. If its retained set matches K50K exactly, we will identify the configurations as the same treatment, not present them as competing methods.

The existing 18.5M pruning-only ablation (Table 5) is informative but not parameter-matched; it cannot establish novelty or isolate the cause of the gain. Also, “unobserved as input” does not mean an embedding row is completely unused: tied output rows can receive softmax/distillation gradients. We will verify the parameter and gradient accounting and distinguish this fact from any demonstrated performance benefit.

We will revise the related-work comparison using primary sources and restrict the final claims to verified differences. If the new controls explain the results through simple unused-token removal or do not support a benefit after tuning, we will report that and narrow the contribution accordingly. No additional experimental result is claimed in this response.

Reviewer_d828OpenReview ↗

Thank you for your reply

Thank you to the authors for their response. The authors appear to be planning additional controlled experiments, but these have not yet been provided. Please add any further discussion and experimental results as soon as possible. The current response is insufficient to change my assessment of the work’s novelty.

We thank the reviewer. The initial rebuttal is already posted; this update reports completed evidence (revised-paper changes: teal = your points only, orange = shared). Your concern, that pruning unused vocabulary and giving the parameters to the Transformer is unsurprising and not novel, is largely correct for the K50K configuration, and we now say so.

1. Is K50K just seen-only pruning? Yes. Regenerating the set of token types seen in the 80K-row training prefix gives exactly the K50K map (47,181 ids, zero differences either direction). K50K is therefore unused-token removal plus width, an established idea (TextPruner §3.1 removes tokens unseen in a corpus; Tao et al. 2024 split vocabulary vs non-vocabulary parameters under a fixed budget; VocabTailor, VocabTrim, PruneSLU, Phan et al. 2026 are related). We cite them in Related Work and a new taxonomy, Revised Table 1 (§2), and withdraw any priority claim for the pruning step. Revised §6.3, Table 6.

2. Coverage depends on corpus size (nested WikiText-103 prefixes).

Train rowsSeen types% of 151,665Test OOV, seen-onlyTest OOV, top-20K
20K38,23325.20.59%3.66%
80K (used)47,18031.10.09%3.27%
all (1.04M)62,72541.40.01%3.01%

Revised Table 6 (§6.3). On a larger corpus the seen set grows, so seen-only pruning stops being small; this supports your point that small-corpus coverage drives the result. We did not run larger independent corpora (FineWeb-Edu nested pools): NOT RUN; no claim is made beyond WikiText-103.

3. Does width, not pruning, carry the gain? Budget-matched controls (same distillation recipe; LR chosen on validation with tuning seed 11 and a reduced grid; 2 final seeds; common-event NLL B1 on the 20K-id set, lower is better):

ModelParamsB1 NLL
Full (tuned, 3 seeds)64M4.019±.010
Seen-only, d=344 (no reinvestment)27.8M4.028±.023
K50K = seen-only + width (tuned)64M3.956±.005
Top-20K, d=34418.5M4.109-4.130
K20K (width reinvested)64M4.000±.006
K20K, L=16 depth-matched64.5M4.019±.009

Revised Table 7 (§6.3). Removing unused tokens alone does not help (4.028 vs 4.019); the K50K advantage (0.064 nats vs Full, 95% interval [0.026, 0.101], n=3) comes with width reinvestment. Depth-matched K20K is not better than width (4.019 vs 4.000), but two seeds do not rank them.

4. Tuned comparison. With per-configuration validation-tuned LRs the legacy BPB gap vs Full vanishes (Full−K20K −0.0024 [−0.0114, 0.0067]); only a small common-event gap remains (0.019 nats for K20K). Revised Table 5 (§6.2).

5. Revised contribution. PAATRA is a controlled measurement of vocabulary-versus-width allocation for one teacher and corpus, not a new pruning method and not a method that beats a tuned baseline. Abstract, §1 and §7 are rewritten accordingly. Remaining limits: single corpus and teacher, 64M scale, controls use 2 seeds and reduced LR grids, no larger-pool experiment. Due to lack of time some of the experiments could not be completed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers (post-rebuttal revision)

We thank all three reviewers. The initial rebuttals were already posted. This note summarises the revised manuscript, which now reports the experiments we had committed to, including results that weaken our original claim.

Colour legend. Changes addressing 5ELv only are blue; changes addressing d828 only are teal; changes addressing TduS only are purple; revisions addressing two or more reviewers are orange.

Major shared changes

  1. Tuned comparison (orange, §6.2, Table 5). Each configuration's learning rate was selected on validation data, then tested on three fresh seeds. The 4.9% legacy BPB gain disappears (Full minus K20K = -0.0024 BPB, interval spans zero). A small consistent-target (common-event) advantage remains: 0.019 nats (K20K) and 0.064 nats (K50K), n=3. We claim no advantage over a tuned full-vocabulary baseline.
  2. Scoring audit (orange, §6.1, Table 4). OOV rates, loss shares of substituted targets, and consistent-target metrics. The legacy BPB is described as a surrogate score, not a raw-text likelihood.
  3. Coverage and controls (teal / orange, §6.3, Tables 6-7). K50K equals the set of token types seen in the training subset (seen-only pruning plus width). Seen-only at the baseline width gives no gain; width reinvestment does. Related work and a taxonomy table (Table 1) were added.
  4. Lossless-tokenizer controls (orange, §6.4, Table 8). Matched BPE/Unigram models with plain cross-entropy at about 64M parameters.
  5. Teacher-screen correction (blue, §6.6, Appendix F). The original perplexity table is withdrawn; renormalization alone lowers perplexity.
  6. Not run (Appendix I). Larger independent corpora, 256M students, a second teacher, PAATRA-Tail, a full 5-LR x 3-seed grid for the new controls, and new efficiency measurements.

Limits. One teacher, one corpus (WikiText-103), small scale. New controls use one tuning seed, reduced learning-rate grids, and two final seeds (Unigram 47K one seed).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.