- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes Turnip, a neural tokenizer that treats tokenization as constrained predictive compression. Instead of discrete vocabulary entries, the input is split into segments of up to 256 bytes, each encoded into one continuous token without exact reconstruction, with boundaries set when the predicted entropy accumulated by a small causal model reaches a budget. The tokenizer is trained on next byte loss without any language model, then attached to a pretrained language model through adapters and finetuned with it, which the paper presents as a model agnostic retrofit. The paper derives a decomposition of next byte loss into source entropy, tokenization cost, and model loss, together with a cost model in which the asymptotic speedup equals the square of the compression ratio.
The contributions are the predictive compression formulation of tokenization with its decomposition and bound, entropy budget segmentation with content adaptive boundaries, and the model agnostic retrofit onto pretrained language models. Validation measures generation quality on IFEval and GPQA Diamond with Qwen2.5 at 0.5B and 1.5B against a freshly trained BPE baseline, and measures latency at batch size one on a single GPU, with BPE running in vLLM and Turnip in a custom pipeline.
Critical strengths
-
The formulation of tokenization as constrained predictive compression is clean and original. Lemma 1 decomposes next byte loss into source entropy, a tokenization term, and a model term, and turns tokenizer design into a problem with an explicit compression bound.
-
Entropy budget segmentation gives content adaptive boundaries with a single interpretable knob. The paper shows the budget trading compression against accuracy in a measurable way, and the ablation against fixed length segmentation supports the value of adaptive boundaries.
-
The system is complete. The tokenizer is trained once without any language model, applied to Qwen2.5 models from 0.5B to 7B with the same encoder and decoder serving multiple model sizes, and the paper reports measured speedups, up to 3.49x prefill at 7B with 128K prompts, together with a cost model that explains when and why the gains appear.
Critical weaknesses
Weakness 1: The design makes a weak frozen model decide what information survives, and later finetuning does not revisit the decision.
-
Turnip performs lossy compression. Up to 256 bytes per segment are encoded into one continuous token. The segmentation, and hence the rate allocation across the input, is determined by an entropy model with one layer, trained without any language model and then kept fixed. The encoder and decoder are likewise trained without any language model. The consuming model, whose task performance the paper claims to preserve, does not influence this allocation. Finetuning cannot alter it, since the entropy model is excluded from finetuning and every segment still passes through a bottleneck of one vector.
-
The guarantee concerns expected loss on the next byte (Eqs. 2 and 3, Lemma 1), whereas task failure is per instance. A single corrupted number, identifier, or instruction constraint suffices to fail a prompt. Compression that preserves the expectation may still concentrate the loss on the least predictable bytes, and neither the theory nor the evaluation examines whether this occurs. The reported curves of accuracy against compression ratio remain benchmark averages. What evidence is there that the damage avoids the content on which the benchmarks are scored?
-
The design is agnostic to the consuming model, so no property of that model enters the compression policy. The claim, however, concerns downstream task performance of pretrained language models. Where is optimality for any particular consuming model bounded or measured?
Weakness 2: The evaluation is not matched to the claims in baselines, tasks, or serving regime.
-
The BPE baseline is a freshly trained 160K vocabulary with a randomly initialized embedding table. How about the model's native tokenizer?
-
Generation quality is measured on IFEval and GPQA-Diamond, which test instruction following and four-choice QA. Turnip is lossy and optimizes next-byte predictability rather than task fidelity, yet the paper does not evaluate benchmarks like long-document QA, multi-hop retrieval, or summarization and code generation, the tasks most likely to expose the accuracy loss.
-
Efficiency is measured at batch size 1 on a single RTX A6000, with decode timed over 96 output bytes on synthetic prompts, all far from a real serving workload. The paper does not discuss how Turnip composes with the mechanisms serving relies on in vLLM and SGLang, such as prefix caching and continuous batching. How much of the reported speedup survives a realistic serving regime?
-
zip2zip (NeurIPS' 25) performs adaptive, context-dependent tokenization while keeping discrete tokens. This closely related work is neither discussed nor compared.
Questions for authors
Question 1: The entropy model decides what information is discarded, yet it is frozen and excluded from finetuning. Why freeze the component that makes this decision?
Question 2: The native tokenizer is never used as a baseline. How does Turnip compare against it once the 10K steps of per model finetuning are accounted for?
Question 3: No task is tested where fidelity of input content is decisive. What evidence is there that Turnip preserves performance on long document question answering, retrieval, summarization, or code generation?
Question 4: All latency numbers are at batch size one, on synthetic prompts, and across different stacks. At serving batch sizes and in the same stack, at what model sizes and compression ratios is Turnip actually faster?
Question 5: Continuous tokens are incompatible with prefix caching, continuous batching, constrained decoding, and speculative decoding, which all key on token IDs. How does Turnip compose with them?
Question 6: zip2zip achieves adaptive tokenization with discrete tokens and is not cited. What do continuous tokens buy that this discrete approach cannot, and is that worth the loss of serving compatibility?
3 comments
Efficiency is measured at batch size 1 on a single RTX A6000...
All latency numbers are at batch size one, on synthetic prompts...
We would like to mention that real-world performance is not homogeneous across hardware and software updates. We therefore mostly rely on FLOPs analysis for long-standing claims about efficiency.
Nevertheless, we have fully implemented Turnip on the same vLLM stack as BPE and measured performance on a single RTX A6000 across context lengths and batch sizes (B=1 and B=4), based on our evaluated Qwen2.5-1.5B checkpoints (CR8) with controlled contexts from real data:
| Batch | Prompt | Prefill (B/s, BPE / Turnip)↑ | Decode ~1KB (B/s, BPE / Turnip)↑ | Peak GPU Memory (GiB, BPE / Turnip)↓ |
|---|---|---|---|---|
| B=1 | 8K | 89.1K / 108.6K | 244.1 / 439.5 | 3.42 / 3.36 |
| B=1 | 16K | 87.4K / 116.6K | 313.6 / 456.1 | 3.72 / 3.46 |
| B=1 | 32K | 78.1K / 133.5K | 321.6 / 408.2 | 4.03 / 3.80 |
| B=4 | 8K | 101.5K / 109.5K | 1,271.7 / 874.1 | 4.10 / 3.86 |
| B=4 | 16K | 95.6K / 128.7K | 1,198.1 / 923.7 | 4.54 / 4.11 |
| B=4 | 32K | 83.3K / 79.4K | 1,009.4 / 1,080.0 | 5.38 / 4.73 |
As shown, BPE leads on short-prompt batched decoding on the 1.5B model, whereas scaling the context length (or scaling the backbone size to 3B and 7B, as analyzed in Table 3) shifts the bottleneck toward the LLM backbone and KV memory, making Turnip faster while using less GPU memory. We have added these shared-stack vLLM serving results in Section 3.3 and Appendix E.
Continuous tokens are incompatible with prefix caching, continuous batching, constrained decoding...
We implemented and verified all four serving primitives in our vLLM engine, as none of them requires discrete token IDs inside the LLM: (1) continuous batching and PagedAttention index KV blocks by sequence position rather than vocabulary ID; (2) prefix caching reuses shared KV blocks and tokenizer states directly because Turnip is strictly causal over UTF-8 byte prefixes; (3) constrained decoding applies finite-state masks directly over the byte decoder's logits, avoiding subword boundary issues; and (4) speculative decoding uses the already-resident byte-entropy model to draft bytes across segments for parallel verification.
zip2zip (NeurIPS' 25) performs adaptive, context-dependent tokenization while keeping discrete tokens. This closely related work is neither discussed nor compared.
zip2zip achieves adaptive tokenization with discrete tokens and is not cited...
We thank the reviewer for the suggestion. Unlike zip2zip (Geng et al., 2025), which merges discrete subwords losslessly at runtime and is therefore bounded by lossless discrete coding, Turnip uses continuous, context-dependent lossy representations to reach higher compression ratios without sacrificing serving compatibility as shown above. Moreover, we deem zip2zip a complementary and orthogonal approach, as Turnip can be applied over zip2zip's hypertoken stream in principle. We have cited zip2zip and added this discussion in Section 4.
Summary
We have clarified and added experiments correspondingly. Notably, the reviewer makes several incorrect assertions and uses them as premises. We have refuted such assertions point by point. We kindly ask the reviewer to reassess the paper thoroughly.
I thank the authors for the detailed response and the additional experiments. I apologize for two inaccurate statements in my review of the frozen entropy model and vLLM serving. These clarifications improve my understanding of the contribution.
The supplementary experiments partially address my earlier concerns. But still some conerns remain. The accuracy shows a loss for multi-hop tasks. And the gain drops clearly as the batch size grows. So I am worried about the scalability and practicality of the method. But overall, thanks to the new results provided by the authors, I will maintain my score.
Response (1/2)
We thank the reviewer for the comments.
As mentioned in Section 2.3, the frozen 1-layer entropy model does not decide what information is discarded. The retention is implicitly learned by the 9-layer encoder and 8-layer decoder, which are jointly fine-tuned end-to-end with the consuming language model.
We freeze the entropy model because there is no gradient anyway. Its sole purpose is to provide an entropy estimate. In fact, a weak entropy model predicts higher entropy per step, triggering boundaries sooner and emitting shorter segments (i.e., low compression). We have clarified this division of labor in Section 2.3.
Rather than concentrating error on unpredictable bytes, cumulative-entropy segmentation (Eq. 5) reaches the budget β faster on high-entropy spans, automatically emitting shorter segments (higher token density) on numbers, code identifiers, and prompt constraints. To directly test tasks where corrupting a single digit or identifier causes per-instance failure, alongside broader long-context benchmarks, we continue both BPE and Turnip (CR8) on 32K-byte contexts for 1,000 steps on
Qwen2.5-1.5Band evaluateLongBenchdocument QA (Qasper), multi-hop retrieval QA (2WikiMQA,HotpotQA), summarization (GovReport), single-file and repository-level code completion (LCC,RepoBench-P), and exact 7-digitPasskeyretrieval:Even at 2× compression, Turnip stays on par with or better than BPE on nearly all tasks with only 1,000-step training, confirming that predictive compression preserves exact numbers, identifiers, and constraints.
Turnip does not claim an optimality bound for a specific finite consuming model. Instead, alignment to a particular consuming LLM occurs during end-to-end joint fine-tuning after the tokenizer pre-training. In all of our experiments, the adapted model's downstream performance scales with its own size. Turnip resembles a typical tokenizer in this process.
Keeping
Qwen2.5-1.5B's native vocabulary while fine-tuning the sameQ, Oprojections on the identical data stream gives:Turnip outperforms both
Native Qwencheckpoints (at 2.5K and 10K steps) as well asRetrained BPEon both benchmarks at CR4 and CR8.