- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper builds on zip2zip [1], which lets a language model read and generate recurring token sequences as single hypertokens. This reduces the number of tokens the model uses, but also hurts quality. The authors find that ingesting and predicting a hypertoken benefit from different representations. Output embeddings depend strongly on the beginning of the sequence, while input embeddings respond more evenly to its contents. This motivates untying the input and output hypertoken embeddings. Merging also shifts the positions of subsequent tokens, so the authors propose a change to RoPE which keeps positions tied to the original token sequence, avoiding shifts. They combine these changes in zip2zip++, while avoiding merging digits, and adding a residual hyper-encoder with identity initialization.
For evaluation, the authors propose multi-view perplexity, sums the probabilities of alternative segmentations of the same text and more accurately captures the probability that the zip2zip/zip2zip++ model would have generated the sequence. This removes roughly a third of the perplexity gap to the uncompressed model. A substantial gap still remains.
The authors compare zip2zip++ with zip2zip and an uncompressed model, and ablate components. The results show lower perplexity and improved mathematical reasoning and long-context retrieval. Much of the reasoning recovery comes from maintaining individual digit tokens. Substantial gaps to the uncompressed models remain, particularly in code generation and instruction following.
[1] Geng et al. zip2zip: Inference-Time Adaptive Tokenization via Online Compression. NeurIPS, 2025.
Critical strengths
-
(S1) I found the embedding geometry analysis insightful and intuitive. It gives a clear motivation for untying the input and output hyper-encoders.
-
(S2) Accounting for equivalent tokenizations is the right way to evaluate text likelihood. The multi-view evaluation is a contribution in its own right which is also relevant to other dynamic tokenization methods.
Critical weaknesses
- (W1) The practical efficiency benefit is not demonstrated. The paper reports fewer tokens, but does not measure decoding speed after accounting for the hyper-encoders and codec. The improved quality also comes with less compression than zip2zip, while substantial quality losses remain on some tasks. For a method motivated by cheaper inference, I would like to see a convincing trade-off between actual inference cost and model quality.
Questions for authors
- (Q1) What decoding speed and memory savings does zip2zip++ achieve relative to zip2zip and the uncompressed model, including the hyper-encoder and codec overhead?
3 comments
Memory saving
At the request lengths in our GSM8K evaluation (about 410–490 base tokens held per request, prompt + output), zip2zip++ does not save memory: it costs slightly more than the uncompressed model, while zip2zip saves a little. zip2zip++ starts saving memory once a request holds about 830 base tokens, and its savings grow with context length. All numbers below are measured in vLLM.
Overhead breakdown
| Uncompressed | zip2zip | zip2zip++ | |
|---|---|---|---|
| Hyper-encoder weights (fixed) | — | 432 MiB | 864 MiB |
| Activation + CUDA-graph memory (fixed) | — | ~0.3 GiB | ~0.3 GiB |
| Hypertoken table (per request, 4,096 entries) | — | 24 MiB | 48 MiB |
| Net per-request memory vs. uncompressed (table − KV saving) | 0 | −23 MiB (saves) | +24 MiB (costs) |
| Break-even request length | — | below our lengths | ~830 base tokens |
Effect on serving capacity. Largest concurrency served without preemption at a fixed GPU budget:
| GPU budget | 12 GiB | 16 GiB | 24 GiB |
|---|---|---|---|
| Uncompressed | 16 | 36 | 72 |
| zip2zip | 18 | 36 | 80 |
| zip2zip++ (1,280-entry tables) | 12 | 36 | 80 |
| zip2zip++ (4,096-entry tables) | 12 | 32 | 64 |
zip2zip already saves memory on short requests. zip2zip++ pays a higher fixed cost (864 MiB of weights) plus a higher per-request cost (2x hypertoken table) that only pays off on longer contexts (≳830 base tokens).
Dear Reviewer,
Thank you for your time and effort in reviewing our paper. As the discussion period ends on November 18 (AoE), we would like to kindly check whether our responses have addressed your concerns.
Best regards,
Reply to Q1
We have now measured this in vLLM; full details are in our comment "Inference speed in vLLM, including concurrent requests." on top. All timings include the LZW codec, the hyper-encoders and the hypertoken output head. Decoding speed. Here are speedups over the uncompressed model on Phi-3.5-mini (one H100):
zip2zip++ is faster than the uncompressed model in every setting we measured. Its per-step overhead is 0.35–0.50 ms on a 3.9 ms base step, about the same as zip2zip (0.34–0.42 ms). It is slower than zip2zip mainly because digit protection lowers compression. Without digit protection, it nearly matches zip2zip's speed (328.6 vs 333.0 tok/s on GSM8K) yet yields a higher accuracy across all the tasks. Prompts are compressed too, so time to first token drops from 26.0 to 20.6 ms on GSM8K.