acceptodds
Under review as a conference paper at ICLR 2027

Turnip: Tokenizers Are Secretly Context Compressors for Transformers

19 views

Abstract

Transformer language models process text as sequences of tokens, and their self-attention cost grows quadratically with the number of tokens. As a result, tokenization largely determines the computational cost as contexts grow. However, the de facto standard tokenizer byte-pair encoding (BPE) produces token sequences longer than the information content of text requires. In this paper, we formulate tokenization as a constrained predictive-compression problem. Under this formulation, we show that the compression of BPE is limited by two constraints: its tokens are lossless and context-independent. Guided by this theory, we introduce Turnip (**t**he ne**ur**al toke**ni**zer **p**ipeline), a neural tokenizer that removes both constraints and provides direct control over the compression ratio. Turnip is model-agnostic and can be used with existing pretrained language models. On different models and benchmarks, Turnip nearly doubles the compression ratio of BPE while matching its generation quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM efficiency, Tokenization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_knHyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Turnip, a neural tokenizer that treats tokenization as constrained predictive compression. Instead of discrete vocabulary entries, the input is split into segments of up to 256 bytes, each encoded into one continuous token without exact reconstruction, with boundaries set when the predicted entropy accumulated by a small causal model reaches a budget. The tokenizer is trained on next byte loss without any language model, then attached to a pretrained language model through adapters and finetuned with it, which the paper presents as a model agnostic retrofit. The paper derives a decomposition of next byte loss into source entropy, tokenization cost, and model loss, together with a cost model in which the asymptotic speedup equals the square of the compression ratio.

The contributions are the predictive compression formulation of tokenization with its decomposition and bound, entropy budget segmentation with content adaptive boundaries, and the model agnostic retrofit onto pretrained language models. Validation measures generation quality on IFEval and GPQA Diamond with Qwen2.5 at 0.5B and 1.5B against a freshly trained BPE baseline, and measures latency at batch size one on a single GPU, with BPE running in vLLM and Turnip in a custom pipeline.

Critical strengths
  • The formulation of tokenization as constrained predictive compression is clean and original. Lemma 1 decomposes next byte loss into source entropy, a tokenization term, and a model term, and turns tokenizer design into a problem with an explicit compression bound.

  • Entropy budget segmentation gives content adaptive boundaries with a single interpretable knob. The paper shows the budget trading compression against accuracy in a measurable way, and the ablation against fixed length segmentation supports the value of adaptive boundaries.

  • The system is complete. The tokenizer is trained once without any language model, applied to Qwen2.5 models from 0.5B to 7B with the same encoder and decoder serving multiple model sizes, and the paper reports measured speedups, up to 3.49x prefill at 7B with 128K prompts, together with a cost model that explains when and why the gains appear.

Critical weaknesses

Weakness 1: The design makes a weak frozen model decide what information survives, and later finetuning does not revisit the decision.

  • Turnip performs lossy compression. Up to 256 bytes per segment are encoded into one continuous token. The segmentation, and hence the rate allocation across the input, is determined by an entropy model with one layer, trained without any language model and then kept fixed. The encoder and decoder are likewise trained without any language model. The consuming model, whose task performance the paper claims to preserve, does not influence this allocation. Finetuning cannot alter it, since the entropy model is excluded from finetuning and every segment still passes through a bottleneck of one vector.

  • The guarantee concerns expected loss on the next byte (Eqs. 2 and 3, Lemma 1), whereas task failure is per instance. A single corrupted number, identifier, or instruction constraint suffices to fail a prompt. Compression that preserves the expectation may still concentrate the loss on the least predictable bytes, and neither the theory nor the evaluation examines whether this occurs. The reported curves of accuracy against compression ratio remain benchmark averages. What evidence is there that the damage avoids the content on which the benchmarks are scored?

  • The design is agnostic to the consuming model, so no property of that model enters the compression policy. The claim, however, concerns downstream task performance of pretrained language models. Where is optimality for any particular consuming model bounded or measured?

Weakness 2: The evaluation is not matched to the claims in baselines, tasks, or serving regime.

  • The BPE baseline is a freshly trained 160K vocabulary with a randomly initialized embedding table. How about the model's native tokenizer?

  • Generation quality is measured on IFEval and GPQA-Diamond, which test instruction following and four-choice QA. Turnip is lossy and optimizes next-byte predictability rather than task fidelity, yet the paper does not evaluate benchmarks like long-document QA, multi-hop retrieval, or summarization and code generation, the tasks most likely to expose the accuracy loss.

  • Efficiency is measured at batch size 1 on a single RTX A6000, with decode timed over 96 output bytes on synthetic prompts, all far from a real serving workload. The paper does not discuss how Turnip composes with the mechanisms serving relies on in vLLM and SGLang, such as prefix caching and continuous batching. How much of the reported speedup survives a realistic serving regime?

  • zip2zip (NeurIPS' 25) performs adaptive, context-dependent tokenization while keeping discrete tokens. This closely related work is neither discussed nor compared.

Questions for authors

Question 1: The entropy model decides what information is discarded, yet it is frozen and excluded from finetuning. Why freeze the component that makes this decision?

Question 2: The native tokenizer is never used as a baseline. How does Turnip compare against it once the 10K steps of per model finetuning are accounted for?

Question 3: No task is tested where fidelity of input content is decisive. What evidence is there that Turnip preserves performance on long document question answering, retrieval, summarization, or code generation?

Question 4: All latency numbers are at batch size one, on synthetic prompts, and across different stacks. At serving batch sizes and in the same stack, at what model sizes and compression ratios is Turnip actually faster?

Question 5: Continuous tokens are incompatible with prefix caching, continuous batching, constrained decoding, and speculative decoding, which all key on token IDs. How does Turnip compose with them?

Question 6: zip2zip achieves adaptive tokenization with discrete tokens and is not cited. What do continuous tokens buy that this discrete approach cannot, and is that worth the loss of serving compatibility?

3 comments
AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for the comments.

The design makes a weak frozen model decide what information survives, and later finetuning does not revisit the decision.

Turnip performs lossy compression...

Q1. The entropy model decides what information is discarded...

As mentioned in Section 2.3, the frozen 1-layer entropy model does not decide what information is discarded. The retention is implicitly learned by the 9-layer encoder and 8-layer decoder, which are jointly fine-tuned end-to-end with the consuming language model.

We freeze the entropy model because there is no gradient anyway. Its sole purpose is to provide an entropy estimate. In fact, a weak entropy model predicts higher entropy per step, triggering boundaries sooner and emitting shorter segments (i.e., low compression). We have clarified this division of labor in Section 2.3.

The guarantee concerns expected loss on the next byte (Eqs. 2 and 3, Lemma 1)...

Generation quality is measured on IFEval and GPQA-Diamond,...

Question 3: No task is tested where fidelity of input content is decisive...

Rather than concentrating error on unpredictable bytes, cumulative-entropy segmentation (Eq. 5) reaches the budget β\beta faster on high-entropy spans, automatically emitting shorter segments (higher token density) on numbers, code identifiers, and prompt constraints. To directly test tasks where corrupting a single digit or identifier causes per-instance failure, alongside broader long-context benchmarks, we continue both BPE and Turnip (CR8) on 32K-byte contexts for 1,000 steps on Qwen2.5-1.5B and evaluate LongBench document QA (Qasper), multi-hop retrieval QA (2WikiMQA, HotpotQA), summarization (GovReport), single-file and repository-level code completion (LCC, RepoBench-P), and exact 7-digit Passkey retrieval:

Method (Qwen2.5-1.5B)Qasper↑2WikiMQA / HotpotQA↑GovReport↑LCC / RepoBench-P↑Passkey (4K / 16K / 32K)↑
BPE (8K)13.212.2 / 8.013.611.3 / 16.983.3 / 66.7 / 60.0%
BPE (32K)17.717.6 / 24.515.311.8 / 13.780.0 / 80.0 / 83.3%
Turnip (CR8, 32K)13.820.7 / 8.016.416.8 / 19.896.7 / 90.0 / 66.7%

Even at 2×2\times compression, Turnip stays on par with or better than BPE on nearly all tasks with only 1,000-step training, confirming that predictive compression preserves exact numbers, identifiers, and constraints.

The design is agnostic to the consuming model...

Turnip does not claim an optimality bound for a specific finite consuming model. Instead, alignment to a particular consuming LLM occurs during end-to-end joint fine-tuning after the tokenizer pre-training. In all of our experiments, the adapted model's downstream performance scales with its own size. Turnip resembles a typical tokenizer in this process.

The evaluation is not matched to the claims in baselines...

The BPE baseline is a freshly trained 160K vocabulary with a randomly initialized embedding table...

The native tokenizer is never used as a baseline...

Keeping Qwen2.5-1.5B's native vocabulary while fine-tuning the same Q, O projections on the identical data stream gives:

Tokenizer (Qwen2.5-1.5B)IFEval Prompt↑IFEval Instruction↑GPQA Diamond↑
Native Qwen (2.5K steps)37.5 ± 0.751.5 ± 0.721.0 ± 1.8
Native Qwen (10K steps)42.8 ± 1.956.4 ± 1.317.8 ± 2.1
Retrained BPE (10K steps)46.4 ± 0.558.4 ± 0.919.5 ± 3.2
Turnip (CR4, 10K steps)51.6 ± 0.862.3 ± 0.723.6 ± 1.3
Turnip (CR8, 10K steps)47.6 ± 0.358.6 ± 0.723.2 ± 2.8

Turnip outperforms both Native Qwen checkpoints (at 2.5K and 10K steps) as well as Retrained BPE on both benchmarks at CR4 and CR8.

Efficiency is measured at batch size 1 on a single RTX A6000...

All latency numbers are at batch size one, on synthetic prompts...

We would like to mention that real-world performance is not homogeneous across hardware and software updates. We therefore mostly rely on FLOPs analysis for long-standing claims about efficiency.

Nevertheless, we have fully implemented Turnip on the same vLLM stack as BPE and measured performance on a single RTX A6000 across context lengths and batch sizes (B=1 and B=4), based on our evaluated Qwen2.5-1.5B checkpoints (CR8) with controlled contexts from real data:

BatchPromptPrefill (B/s, BPE / Turnip)↑Decode ~1KB (B/s, BPE / Turnip)↑Peak GPU Memory (GiB, BPE / Turnip)↓
B=18K89.1K / 108.6K244.1 / 439.53.42 / 3.36
B=116K87.4K / 116.6K313.6 / 456.13.72 / 3.46
B=132K78.1K / 133.5K321.6 / 408.24.03 / 3.80
B=48K101.5K / 109.5K1,271.7 / 874.14.10 / 3.86
B=416K95.6K / 128.7K1,198.1 / 923.74.54 / 4.11
B=432K83.3K / 79.4K1,009.4 / 1,080.05.38 / 4.73

As shown, BPE leads on short-prompt batched decoding on the 1.5B model, whereas scaling the context length (or scaling the backbone size to 3B and 7B, as analyzed in Table 3) shifts the bottleneck toward the LLM backbone and KV memory, making Turnip faster while using less GPU memory. We have added these shared-stack vLLM serving results in Section 3.3 and Appendix E.

Continuous tokens are incompatible with prefix caching, continuous batching, constrained decoding...

We implemented and verified all four serving primitives in our vLLM engine, as none of them requires discrete token IDs inside the LLM: (1) continuous batching and PagedAttention index KV blocks by sequence position rather than vocabulary ID; (2) prefix caching reuses shared KV blocks and tokenizer states directly because Turnip is strictly causal over UTF-8 byte prefixes; (3) constrained decoding applies finite-state masks directly over the byte decoder's logits, avoiding subword boundary issues; and (4) speculative decoding uses the already-resident byte-entropy model to draft bytes across segments for parallel verification.

zip2zip (NeurIPS' 25) performs adaptive, context-dependent tokenization while keeping discrete tokens. This closely related work is neither discussed nor compared.

zip2zip achieves adaptive tokenization with discrete tokens and is not cited...

We thank the reviewer for the suggestion. Unlike zip2zip (Geng et al., 2025), which merges discrete subwords losslessly at runtime and is therefore bounded by lossless discrete coding, Turnip uses continuous, context-dependent lossy representations to reach higher compression ratios without sacrificing serving compatibility as shown above. Moreover, we deem zip2zip a complementary and orthogonal approach, as Turnip can be applied over zip2zip's hypertoken stream in principle. We have cited zip2zip and added this discussion in Section 4.


Summary

We have clarified and added experiments correspondingly. Notably, the reviewer makes several incorrect assertions and uses them as premises. We have refuted such assertions point by point. We kindly ask the reviewer to reassess the paper thoroughly.

I thank the authors for the detailed response and the additional experiments. I apologize for two inaccurate statements in my review of the frozen entropy model and vLLM serving. These clarifications improve my understanding of the contribution.

The supplementary experiments partially address my earlier concerns. But still some conerns remain. The accuracy shows a loss for multi-hop tasks. And the gain drops clearly as the batch size grows. So I am worried about the scalability and practicality of the method. But overall, thanks to the new results provided by the authors, I will maintain my score.

Reviewer_iPL6OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper formulates text tokenization as a constrained predictive-compression problem and proves that losslessness and context-independence each limit the achievable compression. Based on this analysis, the authors propose Turnip, a neural tokenizer that segments the byte stream using an accumulated entropy budget, encodes each segment into a continuous token representation, and predicts bytes with a context-dependent decoder. After adaptation to pretrained Qwen2.5 models, Turnip nearly doubles the compression ratio of BPE while maintaining comparable benchmark accuracy.

Critical strengths
  1. This paper provides a very interesting perspective on accelerating the overall LLM system: it formulates the text tokenizer as a lossy, context-dependent, trainable compressor.

  2. Turnip can be trained independently of a language model and subsequently attached to pretrained models with only minor modifications. It achieves even better benchmark scores at almost the same compression ratio as BPE, and nearly doubles BPE's compression ratio while maintaining comparable benchmark accuracy.

  3. The paper is clearly written.

Critical weaknesses
  1. My main concern is the lossiness of Turnip, though this is also what makes it interesting. Many high-frequency use cases of current LLMs involve long-context retrieval, such as needle-in-a-haystack and long-document QA, as well as tasks that are highly sensitive to context fidelity, such as code editing. If information is already lost at the tokenizer stage, the downstream LLM cannot perform well on these tasks. The paper does not report how accurately the trained tokenizer reconstructs its input as a function of the compression ratio. In addition, validating Turnip on long-context or fidelity-sensitive tasks would make the evaluation much more convincing.

  2. The comparison protocol is fair but not fully convincing. Table 1 trains a new BPE vocabulary rather than retaining the model's original tokenizer and embeddings. This is a useful control, where both tokenizers are equally new to the language model, but it may weaken the reference model. Indeed, the GPQA scores reported are relatively low, so the experiments cannot tell whether the two tokenizers differ in how they constrain the performance ceiling of a sufficiently trained LLM. If compute permits, longer fine-tuning, or training LLMs from scratch with different tokenizers, would make the comparison more informative.

Questions for authors

As Table 3 show, Turnip's prefill can be slower than BPE for short sequences and small models. This raises a further practical issue: in agentic usage (e.g., Codex, Claude Code), the context is re-organized across LLM calls, which means an LLM using Turnip may need to re-prefill the same passage repeatedly because its surrounding context has changed. I would like to hear the authors' view on how Turnip affects the serving infrastructure for agentic workloads, and whether this frequent re-prefilling of the same text limits the practical speedup of Turnip in agentic settings.

2 comments

We are grateful that the reviewer found our perspective on formulating tokenization as a trainable compressor "very interesting" and noted that the paper is "clearly written".

My main concern is the lossiness of Turnip...

We thank the reviewer for raising this important point. To validate Turnip on long-context retrieval and fidelity-sensitive code tasks, we continue both BPE and Turnip (CR8) on 32K-byte contexts for 1,000 steps on Qwen2.5-1.5B and evaluate LongBench document QA (Qasper), multi-hop QA (2WikiMQA, HotpotQA), summarization (GovReport), single-file and repository-level code completion (LCC, RepoBench-P), and exact 7-digit needle-in-a-haystack Passkey retrieval:

Method (Qwen2.5-1.5B)Qasper↑2WikiMQA / HotpotQA↑GovReport↑LCC / RepoBench-P↑Passkey (4K / 16K / 32K)↑
BPE (8K)13.212.2 / 8.013.611.3 / 16.983.3 / 66.7 / 60.0%
BPE (32K)17.717.6 / 24.515.311.8 / 13.780.0 / 80.0 / 83.3%
Turnip (CR8, 32K)13.820.7 / 8.016.416.8 / 19.896.7 / 90.0 / 66.7%

Even at 2×2\times compression, Turnip outperforms BPE on 2WikiMQA, GovReport, both fidelity-sensitive code benchmarks (LCC and RepoBench-P), and exact Passkey retrieval at 4K and 16K bytes, demonstrating that predictive lossy compression preserves code identifiers and exact numeric needles across long contexts.

Regarding input reconstruction across compression ratios, as mentioned in Sections 2.1 and 2.3, Turnip is trained for contextual next-byte prediction rather than autoencoding, so it does not reconstruct input segments directly. Instead, context fidelity across compression ratios (CR4, CR8, CR12) is directly tested by strict downstream constraints (IFEval in Table 1) and exact retrieval (Passkey, where 32K Turnip is continued for 1,000 steps only).

The comparison protocol is fair but not fully convincing...

We thank the reviewer for these constructive suggestions. As presented above, the experiments show that Turnip, at a 2x higher compression ratio than BPE, still performs on par with BPE on long-context evaluation.

As Table 3 show, Turnip's prefill can be slower than BPE for short sequences and small models...

Turnip is designed with agentic applications in mind. As mentioned in Section 2.3, Turnip is strictly causal and fully cached: nothing in an existing prefix ever changes as new input is appended. In standard multi-turn agentic workflows, both Turnip's tokenizer states and the LLM's KV cache are reused directly across calls without re-prefilling.


Summary

We thank the reviewer again for the insightful feedback and constructive questions. In this response, we added larger experiments and clarified several questions. If our new experiments and clarifications have resolved your concerns, we would be very grateful if you would consider updating your evaluation.

Reviewer_iPL6OpenReview ↗

Response to the Authors' Rebuttal

I would like to thank the authors for their response and the additional experiments, which fully resolve the "questions for authors" I raised and partially address my W1. Let me briefly explain why parts of W1 and W2 remain concerns for me.

W1. I am glad to see that Turnip outperforms BPE (32K) on most of the newly added benchmarks. Two observations still concern me: (1) on HotpotQA, the multi-hop QA benchmark, Turnip is significantly worse than BPE (32K) (8.0 vs. 24.5); (2) Turnip's Passkey accuracy decays rapidly as the context grows (96.7% / 90.0% / 66.7% at 4K / 16K / 32K bytes), whereas BPE (32K) remains essentially flat (80.0% / 80.0% / 83.3%). Even allowing for Turnip's higher compression ratio, these results only partially address my concern about lossiness. In addition, a reconstruction experiment would be inexpensive to run: shift the tokens fed to the decoder by one position so that each segment is decoded from its own token, and train for reconstruction instead of next-segment prediction. This would directly measure how much information the tokens preserve. The rebuttal does not provide an experiment of this kind.

W2. I understand that longer fine-tuning or from-scratch training may exceed the available compute. My concern therefore remains, though I appreciate the constraint.

Overall, since W2 and part of W1 remain unaddressed, I will keep my current rating.

Reviewer_gbwMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Turnip tries to make text generation cheaper by running the large model less often. A small encoder compresses each chunk of text into a vector for the large model to process. During generation, the large model runs once, and a small decoder uses its output to autoregressively generate a chunk of bytes. That chunk is then encoded and fed back into the large model for the next round. The authors first train the encoder and decoder on next-byte prediction, then connect them to an existing pretrained model and fine-tune them together. On Qwen2.5-0.5B and 1.5B, Turnip increases the amount of text per model position from about 4.4 to 8 bytes, reducing the number of positions by roughly 45%, while largely maintaining performance on instruction following and scientific question answering. It also delivers actual speedups: with the 1.5B model and a 128K-byte input, prefill is about 2.06× faster and decoding is 1.21× faster. The gains depend on the setup, though—decoding can be slower on the smaller model. The paper also measures speedups on 3B and 7B models, but does not test whether those models maintain answer quality.

Critical strengths

(1) The architecture is interesting and offers a practical way to reduce computation. Turnip lets a small decoder generate a chunk autoregressively while the large model runs only once per chunk. I think it's a good idea.

(2) The experiments show a promising balance between quality and efficiency. On the tested 0.5B and 1.5B models, Turnip increases bytes per model position from about 4.4 to 8 while largely maintaining performance on IFEval and GPQA Diamond. It also demonstrates actual latency improvements: for the 1.5B model with a 128K-byte input, prefill and decoding are accelerated by 2.06× and 1.21×, respectively. These results make the idea more convincing by showing that the shorter sequence can deliver useful speedups without a clear loss of quality in the tested setting.

Critical weaknesses

(1) The experiments are too limited to support the quality claims. For a language-modeling method, I would expect a basic perplexity evaluation—or BPB, which is more comparable across different tokenizers. The paper mentions using held-out BPB to select checkpoints but never reports those results. Downstream evaluation is also limited to IFEval and GPQA Diamond. I find GPQA questionable here because the scores are around 20–25%, close to or below random guessing on a four-option question. Keeping a similarly low score does not tell us much about whether reasoning ability is preserved. Broader capability tests are missing, and so are long-context quality tests. Since long contexts are where this method should offer the biggest benefits, I would want to see long-document summarization, retrieval, or document QA, with both quality and latency measured.

(2) The quality evaluation is limited to backbones that are too small to make the results convincing. The paper only evaluates quality on Qwen2.5-0.5B and 1.5B, whose performance is very weak. When the baseline barely solves the task, maintaining a similar score tells us little about whether compression preserves useful capabilities. I would expect quality evaluation on at least one stronger backbone around 7B, using tasks where it performs meaningfully above chance. The paper already profiles speed on 3B and 7B architectures, but those measurements do not establish whether the adapted models can maintain quality. Testing quality and latency together on a stronger model is necessary to show that the method delivers a useful trade-off beyond the small-model setting.

Questions for authors

(1) Why did you choose Qwen2.5-0.5B and 1.5B for the quality evaluation? The base model is too weak. For example, the GPQA scores are already close to or below random guessing, so keeping similar scores does not tell us much about whether useful capabilities are preserved. Do you have quality results on a stronger model(qwen3.5).

(2) I am particularly interested in long-context task : the paper shows speedups on long inputs, but does the model still perform well on tasks such as long-document summarization or document QA? And because you are doing inference efficiency. Therefore, it is well worth testing long context tasks.

2 comments

We thank the reviewer for the careful and constructive assessment. We are glad that you viewed our architecture as "a practical way to reduce computation" and noted its "promising balance between quality and efficiency".

The experiments are too limited to support the quality claims...

We thank the reviewer for the suggestion. We report held-out bits per byte (BPB) on the validation split of Dolci-Think-SFT-32B across our evaluated checkpoints:

BackboneBPETurnip (CR4)Turnip (CR8)Turnip (CR12)
Qwen2.5-0.5B0.4590.4750.5190.549
Qwen2.5-1.5B0.4120.4190.4760.518

Note that downstream generation quality shows no regression relative to BPE even at CR8, consistent with the well-known phenomenon in LLM training that small differences in validation perplexity/BPB do not necessarily reflect generation quality [1, 2].

I find GPQA questionable here because the scores are around 20–25%...

As mentioned in Section 3.1, GPQA is evaluated via open-ended chain-of-thought generation rather than four-way option ranking, where outputs without a parsed option letter are counted as wrong. Therefore, 20-25% should not be viewed as "below random guessing".

Broader capability tests are missing, and so are long-context quality tests...

I am particularly interested in long-context task...

We thank the reviewer for the suggestions. We added more benchmarks with longer contexts and broader domains. Specifically, we continue both BPE and Turnip (CR8) on 32K-byte contexts for 1,000 steps and evaluate LongBench, which includes document QA (Qasper), multi-hop QA (2WikiMQA, HotpotQA), summarization (GovReport), code completion (LCC, RepoBench-P), and exact 7-digit Passkey retrieval. The results are as follows:

Method (Qwen2.5-1.5B)Qasper↑2WikiMQA / HotpotQA↑GovReport↑LCC / RepoBench-P↑Passkey (4K / 16K / 32K)↑
BPE (8K)13.212.2 / 8.013.611.3 / 16.983.3 / 66.7 / 60.0%
BPE (32K)17.717.6 / 24.515.311.8 / 13.780.0 / 80.0 / 83.3%
Turnip (CR8, 32K)13.820.7 / 8.016.416.8 / 19.896.7 / 90.0 / 66.7%

As shown, Turnip tends to have higher generation quality than BPE with lower compute. We hypothesize that this is because Turnip utilizes fewer Transformer positions in the original Qwen model, within the regime where the pre-trained model is more capable, highlighting a promising direction for long-context extension.

We also implemented Turnip on the same vLLM stack as BPE and measured latency and memory on a single RTX A6000 using the evaluated Qwen2.5-1.5B checkpoints above with controlled contexts from real data:

Prompt (Qwen2.5-1.5B)Prefill (B/s, BPE / Turnip)↑Decode ~1KB (B/s, BPE / Turnip)↑Peak GPU Memory (GiB, BPE / Turnip)↓
4K94.6K / 72.9K419.3 / 419.93.27 / 3.26
8K89.1K / 108.6K244.1 / 439.53.42 / 3.36
16K87.4K / 116.6K313.6 / 456.13.72 / 3.46
32K78.1K / 133.5K321.6 / 408.24.03 / 3.80

As shown, Turnip matches BPE's performance at 4K context, and scaling the context length widens Turnip's advantage on the shared stack as the bottleneck shifts toward the LLM backbone rather than the small tokenizer. We have added these long-context and vLLM serving results in Section 3.3 and Appendices D and E.

The quality evaluation is limited to backbones that are too small to make the results convincing...

Why did you choose Qwen2.5-0.5B and 1.5B for the quality evaluation?...

We chose the Qwen2.5 family because these base models do not have built-in thinking behavior, whereas our adaptation data and downstream evaluations require reasoning; this allows us to isolate the effect of adaptation rather than pre-training quality. Within Qwen2.5, we focused on 0.5B and 1.5B due to our limited compute budget. Because adapting larger backbones requires substantially more compute and longer training, we cannot finish 3B and 7B adaptation at the moment; however, we are dedicated to adding 3B and 7B evaluations in the revision.


Summary

We thank the reviewer once again for the valuable suggestions to strengthen our empirical evaluation. In this response, we have conducted additional experiments and clarified the questions. We hope these additional experiments resolve your concerns and kindly invite you to reassess our submission.

[1] Liu, H., Xie, S. M., Li, Z., & Ma, T. (2023). Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models. ICML.

[2] Jaiswal, A. K., Gan, Z., Du, X., Zhang, B., Wang, Z., & Yang, Y. (2024). Compressing LLMs: The Truth is Rarely Pure and Never Simple. ICLR.

Thanks for the reply and the extra experiments during rebuttal. These address some of my concerns, and I appreciate the effort. I understand the compute constraints, so I’m not asking you to finish larger-model experiments now; that was a suggestion for stronger validation. I recognize the actual speed gains, especially the 1.71× prefill throughput at 32K. My concern is the quality loss on some long-context tasks: compared with BPE trained at 32K, HotpotQA drops from 24.5 to 8.0 and 32K Passkey from 83.3% to 66.7%. For document QA and retrieval, I’m not convinced the time savings are worth drops of this size. On GPQA, I understand that parsing failures can lower accuracy, but CoT generation alone does not make the 25% random-guessing reference irrelevant—the original GPQA paper also reports CoT results against that reference. If available, the parsing failure rates would help clarify how much of the low score comes from formatting rather than answering incorrectly. For now, I will keep my current rating.

Reviewer_RDtHOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper first formulates tokenization as a constrained predictive-compression problem. They provide some theoretical analysis under this formulation and suggests that the exiting BPE tokenizer is suboptimal and they propose replacing the lossless BPE tokenizer by a lossy neural tokenizer, which should perform better based on the theoretial analysis. They show that their method out-perform existing BPE methods under similar compression ratio and outperform existing work like BPE significantly.

Critical strengths

I don't see a critical strength of the paper, but it could be because I'm not able to understand the theoretical contribution of this paper (I have alerted the AC and set a low confidence level)

Critical weaknesses

The critical weakness of this paper is how it differs from existing works like BLT ? The overall architecture is very similar, we can see both this method as BLT as a two-level architecture, with the first level being merging bytes into latent token representation, and the second-level being the main model, powered by a transformer.

Your current argument, as appeared in the related work section, being

These boundary criteria are designed heuristically and decide each boundary from a local signal, so a patch closes regardless of how much information it already holds. Turnip instead accumulates the entropy within each segment up to a budget so that each token carries a similar amount of information, The entropy-based truncation rule has also been explored in the BLT work, so this argument is not valid to me.

Questions for authors

My first question is how does it differ from BLT and does it bring improvement on top of BLT ? My second question is, as you have mentioned in the section 3.2, the comparaion between BLT and Qwen2.5 models is not fair. While this is surely not a problem from you, but this does adds diffculty to assess how does this method stand against BLT. I wonder if we could perform other experiment that can provide more conclusive results ?

Another work that I think might be relevant is: https://neurips.cc/virtual/2025/loc/san-diego/poster/118871

1 comment

We thank the reviewer for the constructive comments and for noting that our presentation is "clear and easy to follow".

The critical weakness of this paper is how it differs from existing works like BLT ? The overall architecture is very similar, we can see both this method as BLT as a two-level architecture, with the first level being merging bytes into latent token representation, and the second-level being the main model, powered by a transformer.

My first question is how does it differ from BLT and does it bring improvement on top of BLT ?

As mentioned in Sections 2.3 and 4, Turnip differs from BLT in significant ways: (1) standalone compressing tokenizer vs. tokenizer-free model. BLT co-trains its local byte modules, cross-attention pooling, and multi-billion-parameter byte nn-gram hash embeddings jointly with a specific language model from scratch. By contrast, Turnip is pretrained once as a standalone tokenizer without any language model or byte-hash tables, and attaches directly to existing pretrained models via lightweight linear adapters. (2) theory-guided cumulative entropy budget vs. single-byte threshold. While BLT also uses a byte-entropy model (cited in Sections 2.3 and 4), BLT places boundaries from local single-byte criteria, so a long run of moderate-entropy bytes never closes a patch even as total uncertainty grows. Turnip instead accumulates entropy across the segment up to a budget β\beta (Eq. 5) so that every token carries a bounded amount of information, directly controlling ℓtok\ell^{\mathrm{tok}} in Lemma 1.

My second question is, as you have mentioned in the section 3.2, the comparaion between BLT and Qwen2.5 models is not fair. While this is surely not a problem from you, but this does adds diffculty to assess how does this method stand against BLT. I wonder if we could perform other experiment that can provide more conclusive results ?

Under the exact same Qwen2.5-0.5B backbone, training data, and compression ratio, we conduct two controlled experiments: (1) adapting pretrained modules (BLT-1B's encoder, decoder, and hash embeddings) to pretrained Qwen2.5-0.5B for 500 steps, and (2) pretraining both architectures from scratch on a Qwen2.5-0.5B backbone for 10K steps.

Setting (Qwen2.5-0.5B, CR8)MethodHash Params↓Total Params↓Held-Out BPB↓
Pretrained-module adaptation (500 steps)BLT (from BLT-1B)3.07B3.85B0.768
Pretrained-module adaptation (500 steps)Turnip (CR8)00.62B0.619
From-scratch pretraining (10K steps)BLT (CR8)3.07B3.85B0.707
From-scratch pretraining (10K steps)Turnip (CR8)00.62B0.541

In both post-hoc adaptation and from-scratch pretraining, Turnip achieves significantly lower held-out BPB than BLT while using 6.2×6.2\times fewer total parameters (zero byte-hash tables) and running roughly 6×6\times faster per training step. The results strongly confirm both the effectiveness and efficiency of Turnip against BLT.

Another work that I think might be relevant is: https://neurips.cc/virtual/2025/loc/san-diego/poster/118871

We thank the reviewer for the suggestion. We deem zip2zip (Geng et al., 2025) a complementary and orthogonal approach, as the two mechanisms could be combined in principle. For instance, Turnip can be applied over zip2zip's hypertoken stream. We have cited zip2zip and added this discussion in Section 4.


Summary

We again thank the reviewer for the helpful feedback and constructive comments. In this response, we clarified the important differences between Turnip and BLT, conducted matched comparisons against BLT, and discussed zip2zip as a complementary approach. Should the concerns be addressed, we would appreciate it if you could consider raising the score.

Reviewer_MTaaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Turnip, a neural tokenizer for Transformer models. The authors formulate tokenization as a constrained predictive-compression problem and analyze limitations associated with lossless tokenization and context-independent detokenization. Turnip uses an entropy model to create dynamic segments, encodes them into continuous representations, and uses a context-dependent decoder for next-byte prediction. After adaptation of Qwen2.5-0.5B and 1.5B, Turnip achieves accuracy comparable to the paper's retrained BPE baseline on IFEval and GPQA Diamond at approximately 8 bytes per token. Synthetic speed profiling, including larger models, shows benefits that generally increase with context length and model size.

Critical strengths
  1. The problem formulation is useful. The decomposition of next-byte loss into source, tokenization, and model terms provides a clear framework for discussing which information a tokenizer should retain for prediction.

  2. The system design is coherent. Entropy-accumulating segmentation, continuous representations, context-dependent decoding, and an adjustable segment budget form a consistent approach. The fixed-length segmentation ablation supports the proposed segmentation rule within the evaluated settings.

Critical weaknesses
  1. The theory does not yet establish the claimed explanation for the implemented system's advantage. The loss decomposition is useful, but the compression bound depends on a finite vocabulary through log⁡∣C∣\log |C|, and Proposition 2 additionally concerns discrete next-token prediction at the compression bound. Turnip instead emits continuous states and is trained through next-byte prediction. Finite-precision states could in principle define a finite alphabet, but the paper does not explain how its capacity should be compared with the 160K-entry BPE vocabulary or whether the resulting bound is informative. Moreover, Proposition 2 concerns the limiting case of attaining the bound, while Proposition 3 establishes an advantage for a constructed source. These results motivate the design but do not directly establish that the implemented tokenizer exceeds a comparable lossless limit on natural language. The experiments also do not isolate whether the gain comes from relaxing the two stated constraints or from greater representational capacity per model position.

  2. The quality and efficiency evidence supports a narrower conclusion than the overall claims suggest. Quality is evaluated on two benchmarks with 0.5B and 1.5B models. The 3B and 7B experiments profile speed with untrained models and therefore do not establish quality preservation at those sizes. The paper does account for tokenizer cost in its analytical model and reports both prefill and decoding speedups. Nevertheless, the timing evidence comes from synthetic inputs at fixed compression ratios, batch size one, and different execution stacks for BPE and Turnip. Component-level latency, memory usage, and fuller tokenizer architecture details would make these results easier to interpret and reproduce. Table 3 also shows decoding slowdowns for the 0.5B model, so the practical benefit is conditional on the workload and model size. A measurement on actual held-out text using an existing quality-evaluated checkpoint would more directly connect the reported quality and speed results.

Questions for authors
  1. Which theoretical conclusions are intended to apply directly to Turnip, and which serve as design motivation? How should the capacity of its continuous or finite-precision representations be related to log⁡∣C∣\log |C| and compared with BPE? Please clarify whether the claim of exceeding a lossless compression limit is established under comparable capacity constraints, or should be qualified.

  2. Please report tokenizer architecture sizes, trainable parameter counts, training-data volumes, and training compute for the two adaptation setups, including Turnip's tokenizer pretraining. How does the quality comparison change when Qwen retains its original tokenizer under a comparable adaptation schedule?

2 comments
AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for the thoughtful feedback and constructive assessment. We are encouraged that you found our problem formulation "useful" and our system design "coherent".

The theory does not yet establish the claimed explanation for the implemented system's advantage...

Which theoretical conclusions are intended to apply directly to Turnip, and which serve as design motivation?...

We thank the reviewer for raising this insightful point. The discrete-vocabulary bounds in Section 2.2 serve as design motivation by examining the bottlenecks of existing tokenizers (i.e., context-independence and losslessness). They only demonstrate the existence of an optimal tokenizer when removing these constraints. They do not argue that Turnip is that optimal tokenizer. We have clarified this in Section 2.2.

The theoretical capacity of each Turnip token spans all BF16 vectors. By contrast, BPE is limited to its fixed vocabulary. This is an additional advantage of Turnip that enables utilization of higher bits per token. To examine "whether the gain comes from relaxing the two stated constraints or from greater representational capacity per model position", we conduct the following experiments on Qwen2.5-1.5B (CR8).

Variant (Qwen2.5-1.5B, CR8)Token RepresentationCapacity / TokenDecoder ContextHeld-Out BPB↓
TurnipBF1616,384 bitsMulti-token window0.476
Single-token decoderSingle token+0.103
Lossy noisy channel (not retrained)+ 6dB Gaussian≤839\le 839 bits+0.011
64-bit codebookDiscrete grid64 bits+0.373

These results establish three conclusions. (1) Context-dependent decoding is essential as claimed, as restricting the byte decoder to a single token substantially degrades BPB even after full retraining, confirming that per-position capacity alone is insufficient. (2) Lossless reconstruction is unnecessary as claimed, as injecting Gaussian noise makes ckc_k strictly non-injective while barely affecting BPB even without retraining. (3) Additionally, our design allows Turnip to effectively operate at much higher bits per token than what BPE can achieve, as an 839-bit discrete budget would require an intractable 3.7×102523.7 \times 10^{252} BPE vocabulary.

These findings confirm our paper's claims and the consistency between our theoretical analysis and empirical results. We have added these ablations in Section 3.3 and Appendix C.

The quality and efficiency evidence supports a narrower conclusion than the overall claims suggest...

We thank the reviewer for the suggestions. We added more benchmarks with longer contexts and broader domains. Specifically, we continue both BPE and Turnip (CR8) on 32K-byte contexts for 1,000 steps and evaluate LongBench, which includes document QA (Qasper), multi-hop QA (2WikiMQA, HotpotQA), summarization (GovReport), code completion (LCC, RepoBench-P), and exact 7-digit Passkey retrieval. The results are as follows:

Method (Qwen2.5-1.5B)Qasper↑2WikiMQA / HotpotQA↑GovReport↑LCC / RepoBench-P↑Passkey (4K / 16K / 32K)↑
BPE (8K)13.212.2 / 8.013.611.3 / 16.983.3 / 66.7 / 60.0%
BPE (32K)17.717.6 / 24.515.311.8 / 13.780.0 / 80.0 / 83.3%
Turnip (CR8, 32K)13.820.7 / 8.016.416.8 / 19.896.7 / 90.0 / 66.7%

As shown, Turnip tends to have higher generation quality than BPE with lower compute. We hypothesize that this is because Turnip utilizes fewer Transformer positions in the original Qwen model, within the regime where the pre-trained model is more capable, highlighting a possible way for future long-context extension.

Given that adapting larger backbones requires substantially more compute and longer training, we cannot report 3B and 7B quality evaluations at the moment. However, we are dedicated to adding the 3B and 7B results in the revision for a fuller comparison.

AuthorsOpenReview ↗

Response (2/2)

The paper does account for tokenizer cost in its analytical model and reports both prefill and decoding speedups...

We would like to mention that real-world performance is not homogeneous across hardware and software updates. We therefore mostly rely on FLOPs analysis for long-standing claims about efficiency.

Nevertheless, we have fully implemented Turnip on the same vLLM stack as BPE and measured the performance on a single RTX A6000. All results below are based on the evaluated Qwen2.5-1.5B checkpoints above with controlled contexts from real data.

Prompt (Qwen2.5-1.5B)Prefill (B/s, BPE / Turnip)↑Decode ~1KB (B/s, BPE / Turnip)↑Peak GPU Memory (GiB, BPE / Turnip)↓
4K94.6K / 72.9K419.3 / 419.93.27 / 3.26
8K89.1K / 108.6K244.1 / 439.53.42 / 3.36
16K87.4K / 116.6K313.6 / 456.13.72 / 3.46
32K78.1K / 133.5K321.6 / 408.24.03 / 3.80

As shown, Turnip matches BPE's performance at 4K context, and scaling the context length widens Turnip's advantage on the shared stack (just as scaling the model size does in Table 3), as the bottleneck shifts more towards the LLM backbone rather than the small tokenizer.

Please report tokenizer architecture sizes, trainable parameter counts, training-data volumes, and training compute for the two adaptation setups, including Turnip's tokenizer pretraining.

We report them as follows.

Stage / SetupTrainable ParamsMax StepsData VolumeEst. Compute (A6000 h)
Turnip pretraining: Entropy model7.5M10K2.9 GB~1.5 (one-time)
Turnip pretraining: Encoder + Decoder126.1M50K13.2 GB~30 (one-time)
Qwen2.5-0.5B adaptation (BPE / Turnip)185.4M / 166.5M10K2.9 GB~10 / ~14
Qwen2.5-1.5B adaptation (BPE / Turnip)383.8M / 261.5M10K2.9 GB~16 / ~19

Turnip updates fewer parameters than BPE across both adaptation setups because BPE's tied 160K embedding table grows with the backbone hidden dimension whereas Turnip's tokenizer size stays fixed.

How does the quality comparison change when Qwen retains its original tokenizer under a comparable adaptation schedule?

Keeping Qwen2.5-1.5B's native vocabulary while fine-tuning the same Q, O projections on the identical data stream gives:

Tokenizer (Qwen2.5-1.5B)IFEval Prompt↑IFEval Instruction↑GPQA Diamond↑
Native Qwen (2.5K steps)37.5 ± 0.751.5 ± 0.721.0 ± 1.8
Native Qwen (10K steps)42.8 ± 1.956.4 ± 1.317.8 ± 2.1
Retrained BPE (10K steps)46.4 ± 0.558.4 ± 0.919.5 ± 3.2
Turnip (CR4, 10K steps)51.6 ± 0.862.3 ± 0.723.6 ± 1.3
Turnip (CR8, 10K steps)47.6 ± 0.358.6 ± 0.723.2 ± 2.8

Turnip outperforms both Native Qwen checkpoints (at 2.5K and 10K steps) as well as Retrained BPE on both benchmarks at CR4 and CR8.


Summary

We again thank the reviewer for the constructive comments and careful evaluation of our submission. In this response, we clarified the scope of our theoretical bounds, empirically isolated context-dependent decoding and lossy compression from per-position capacity, added 32K LongBench quality evaluations, benchmarked our evaluated 1.5B checkpoints on a shared vLLM stack, and reported full parameter, data, and compute budgets alongside the Native Qwen baseline. We believe the response addresses all your concerns, and would be grateful if you could raise the score in light of our revisions. Thank you for your time and support!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.