- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
PAATRA targets a specific pathology of SLMs distilled from large teachers: inheriting the teacher's tokenizer wastes most of the parameter budget on embeddings (a 64M GPT-2-style student with Qwen2.5's 152K vocab spends ~81.8% on embeddings). The method keeps the top-K most frequent teacher tokens (K in {20K, 47K, 64K}), distills over this subset, and reinvests freed parameters into a wider transformer at a fixed budget. Main result: 4.9% BPB improvement over the full-vocab baseline on WikiText-103 across 3 seeds with Qwen2.5-0.5B as teacher.
Critical strengths
Unusually honest - both nullifying caveats are surfaced in the abstract, code and per-seed numbers are released. The parameter-matched controlled setup with 3 seeds is clean, and the embedding-share audit (Table 2, e.g., Gemma 3 270M spending ~63% on embeddings) is a useful community reference.
Critical weaknesses
-
The paper's own results refute its central claim. Table 6: a full-vocab baseline tuned to LR 1e-3 reaches BPB 1.3455, matching the best PAATRA-20K student (1.3448). Table 7: genuine BPE/Unigram tokenizers at 20K-64K with plain CE achieve BPB 1.186-1.270, better than every PAATRA variant. The crucial matched-protocol comparison is explicitly "remains to be done." Vocabulary pruning with a real subword tokenizer dominates the proposed method.
-
Toy scale and protocol issues that cannot support the claims: 64M params / 30.7M tokens / WikiText-103 only / single teacher; no validation split (selection effectively on test set); the surrogate-ID protocol does not yield a genuine tokenizer likelihood; the reinvestment ablation confounds parameter count with allocation (18.5M vs 64M).
-
No efficiency gain and limited novelty. Table 10 shows PAATRA requires more compute per token than the full-vocab baseline. The core idea (prune teacher vocabulary to a frequency-ranked subset, distill over it) is well-covered by TextPruner, VocabTailor, VocabTrim, Phan et al. 2026, Tao et al. 2024.
Questions for authors
-
Can you provide the matched-protocol comparison - PAATRA vs. a genuine 20K/47K BPE/Unigram tokenizer with plain CE at the same budget and LR?
-
If every configuration receives its own multi-seed LR sweep, does the reallocation gain survive? Table 6 suggests a single LR change already erases most of it.
-
Was any selection performed on the test set, given no validation split is described?
3 comments
Response to Authors
Thank you for the candid response. However, the matched tokenizer comparison, independently tuned multi-seed results, and budget-matched allocation ablation remain planned rather than completed. The acknowledged test-set selection and limitations of surrogate scoring also undermine the evidence supporting the reported gains. While the proposed experiments are relevant, the current response does not establish a reliable performance advantage or sufficiently distinguish the contribution from prior work.
We thank the reviewer. The initial rebuttal is already posted; this update reports completed evidence (revised-paper changes: purple = your points only, orange = shared). Your central point is correct as the paper stood: the claim does not survive a tuned baseline. We now state that.
W1 / Q2 (own results refute the claim; per-configuration LR sweeps). We tuned each configuration's LR on a validation split (grid 1e-4..3e-3, tuning seeds 11/22, selection frozen by hash before test) and ran 3 fresh test seeds. Revised Table 5 (§6.2), grid in Table 10 (App. C).
| Config | LR | BPB (legacy) | Common-event NLL |
|---|---|---|---|
| Full | 1e-3 | 1.3430±.0040 | 4.019±.010 |
| K50K | 3e-4 | 1.3438±.0004 | 3.956±.005 |
| K20K | 3e-4 | 1.3453±.0027 | 4.000±.006 |
| Full−K20K | −0.0024 [−0.0114, 0.0067] | 0.019 [0.006, 0.031] |
The 4.9% BPB gain disappears (intervals span zero): it was a baseline-tuning effect. A small consistent-target advantage remains (0.02-0.06 nats, n=3), and we make no claim of beating a tuned baseline. Abstract and conclusion are rewritten.
Q1 (matched BPE/Unigram with plain CE). Done at ~64M, same optimizer, steps and data; LR validation-selected (1 tuning seed); final seeds 101/202 (Unigram-47K seed 101 only). Revised Table 8 (§6.4), lossless test BPB on raw text:
| Model | K | BPB |
|---|---|---|
| Full (teacher IDs) | 151,665 | 1.667±.005 |
| BPE 20K / 47K | 20K / 47K | 1.706±.006 / 1.645±.001 |
| Unigram 20K / 47K | 20K / 47K | 1.633±.001 / 1.614 (n=1) |
Your point is partly confirmed: Unigram 20K/47K and BPE 47K beat the full vocabulary by 0.02-0.05 BPB, BPE 20K is 0.04 worse. A retrained lossless tokenizer is competitive; PAATRA's subset-of-teacher-vocabulary pipeline is not shown to beat it. The original submission's Table 7 numbers (BPB 1.19-1.27) used different data, optimizer and parameter counts and are kept only as unmatched context (App. E).
Q3 (test-set selection). In the original submission there was no validation split, and the LR sweep was effectively read on test. All new selections use a validation split only; selection files are hashed before test evaluation. The original sweep is retained and labelled as a shared-recipe measurement (§6.5).
W2 (scale, surrogate protocol, confounded ablation). The surrogate-ID score is not a raw-text likelihood: test OOV rates are 1.06% (K50K), 4.25% (K20K), 9.95% (K10K), and substituted targets carry 3.0/8.9/22.6% of NLL (Revised Table 4, §6.1); we now report a consistent-target metric and the lossless controls above. The 18.5M ablation is retained as a pruning-only reference; a budget-matched seen-only control (27.8M, B1 4.028 vs Full 4.019) and a depth-matched K20K (64.5M, 4.019) are in Revised Table 7 (§6.3). Scale, second teacher and additional corpora (256M students, FineWeb-Edu): NOT RUN; claims are limited to one teacher, 64M, WikiText-103.
W3 (efficiency, novelty). We ran no new efficiency measurement and make no efficiency claim; Table 12 (App. D) is analytical and shows no compute saving. Prior art (TextPruner, VocabTailor, VocabTrim, Phan et al. 2026, Tao et al. 2024) is now cited and tabulated (Table 1, §2); K50K equals seen-only pruning plus width.
Response to reveiwer TduS
We thank the reviewer for the careful evaluation, constructive criticism, and insightful questions. The concerns identify substantive limits that our revised claims and validation must reflect.
CW-01: Interpretation of existing results
Reviewer Concern
Response
We agree that the evidence does not establish a tuned-performance advantage. Table 6's Full score (1.3455) nearly matches K20K (1.3448); the 4.9% result is specific to the shared recipe. Table 7 is concerning, but its different data, exposure, optimizer schedule, parameter counts and scored bytes prevent a controlled dominance claim (Appendix C). We will run matched controls and retain unfavorable outcomes. If the gain disappears, we will retract the advantage claim.
CW-02: Scope and protocol
Reviewer Concern
Response
We accept these limitations. We will use validation-only tuning, audit OOV targets and contexts, and distinguish surrogate scores from full-text coding scores. The 18.5M ablation cannot isolate allocation; we will add a budget-matched depth alternative. Larger-budget, additional-corpus and second-teacher checks are planned, subject to compute availability; absent evidence, conclusions will remain restricted to the original setting. Historical test reuse cannot be undone.
CW-03: Efficiency and novelty
Reviewer Concern
Response
We do not claim inference savings: Table 10 reports higher FLOPs/token and batch-1 CPU latency for the reallocated models. We will report measured latency, throughput, memory and total training cost separately. We will revise the related-work comparison, including Tao et al.'s vocabulary/non-vocabulary allocation and Phan et al.'s exact cross-tokenizer scoring. Our gathered-logit surrogate protocol is not that exact algorithm. Combining known components alone does not establish novelty.
QA-01: Matched tokenizer comparison
Reviewer Concern
Response
We plan BPE and Unigram controls at 20K and approximately 47K with corrected tied-parameter counting, identical raw training examples, update counts, optimizer schedules, and shared-LR results alongside validation-tuned results. We will score identical held-out bytes without dropped targets. Original PAATRA's lossy surrogate score cannot be ranked as raw-text likelihood; any full-support extension will be labelled a new variant, not a validation of the original method.
QA-02: Multi-seed LR tuning
Reviewer Concern
Response
This is unresolved. We will sweep every configuration over {1e-4,3e-4,6e-4,1e-3,3e-3} using three tuning seeds, select by mean validation score, and rerun selected settings with three fresh seeds. A predeclared boundary extension will check truncated optima. We will report paired differences and uncertainty, without equating a small observed gap with statistical equivalence.
QA-03: Test-set selection
Reviewer Concern
Response
Yes. The manuscript's Discussion and the repository README explicitly state that all selection and evaluation used the test split. The LR sweep compared test scores; these results must be described as exploratory. We will audit which decisions used test feedback, move all new selection to official validation, and freeze settings before test reporting. Reusing the old test set does not make it pristine; an independently held-out corpus will provide a separate check. No new results are claimed here.