acceptodds
Under review as a conference paper at ICLR 2027

zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Loss

Abstract

Dynamic vocabularies built on-the-fly via Lempel–Ziv–Welch (LZW) compression let a language model represent the same text in fewer tokens, but prior work shows that this efficiency comes at a cost to reasoning and perplexity. We show that this quality gap is not a fundamental price of compression but can be substantially closed through targeted improvements to the compression layer and LZW codec, without modifying the underlying pretrained language model. Our analysis reveals four key issues: (1) LZW's compression algorithm creates an implicit conflict between the input embedding and the output embedding — a tension that a single shared embedding cannot resolve; (2) naively applying RoPE to LZW-compressed sequences ignores that merged tokens can span multiple tokens; (3) merging certain tokens (notably digits) disproportionately hurts downstream accuracy; and (4) dynamic vocabularies introduce an inherent segmentation ambiguity that confounds standard evaluation. We address these through decoupled input/output dynamic embedding layer, size-aware RoPE positioning, and a modified LZW algorithm supporting selective merges and multi-view segmentations. Across four models, our recipe improves perplexity on every corpus we evaluated as well as mathematical reasoning accuracy, closing roughly half the quality gap between zip2zip and the uncompressed baseline while achieving the same compression ratio. This represents a significant step toward lossless adaptive-vocabulary compression for language models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tokenization, adaptive vocabulary, LZW compression, LLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eMfqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper builds on zip2zip [1], which lets a language model read and generate recurring token sequences as single hypertokens. This reduces the number of tokens the model uses, but also hurts quality. The authors find that ingesting and predicting a hypertoken benefit from different representations. Output embeddings depend strongly on the beginning of the sequence, while input embeddings respond more evenly to its contents. This motivates untying the input and output hypertoken embeddings. Merging also shifts the positions of subsequent tokens, so the authors propose a change to RoPE which keeps positions tied to the original token sequence, avoiding shifts. They combine these changes in zip2zip++, while avoiding merging digits, and adding a residual hyper-encoder with identity initialization.

For evaluation, the authors propose multi-view perplexity, sums the probabilities of alternative segmentations of the same text and more accurately captures the probability that the zip2zip/zip2zip++ model would have generated the sequence. This removes roughly a third of the perplexity gap to the uncompressed model. A substantial gap still remains.

The authors compare zip2zip++ with zip2zip and an uncompressed model, and ablate components. The results show lower perplexity and improved mathematical reasoning and long-context retrieval. Much of the reasoning recovery comes from maintaining individual digit tokens. Substantial gaps to the uncompressed models remain, particularly in code generation and instruction following.


[1] Geng et al. zip2zip: Inference-Time Adaptive Tokenization via Online Compression. NeurIPS, 2025.

Critical strengths
  • (S1) I found the embedding geometry analysis insightful and intuitive. It gives a clear motivation for untying the input and output hyper-encoders.

  • (S2) Accounting for equivalent tokenizations is the right way to evaluate text likelihood. The multi-view evaluation is a contribution in its own right which is also relevant to other dynamic tokenization methods.

Critical weaknesses
  • (W1) The practical efficiency benefit is not demonstrated. The paper reports fewer tokens, but does not measure decoding speed after accounting for the hyper-encoders and codec. The improved quality also comes with less compression than zip2zip, while substantial quality losses remain on some tasks. For a method motivated by cheaper inference, I would like to see a convincing trade-off between actual inference cost and model quality.
Questions for authors
  • (Q1) What decoding speed and memory savings does zip2zip++ achieve relative to zip2zip and the uncompressed model, including the hyper-encoder and codec overhead?
3 comments

Reply to Q1

We have now measured this in vLLM; full details are in our comment "Inference speed in vLLM, including concurrent requests." on top. All timings include the LZW codec, the hyper-encoders and the hypertoken output head. Decoding speed. Here are speedups over the uncompressed model on Phi-3.5-mini (one H100):

GSM8K (1 req.)IFEval (1 req.)64 concurrent req.
zip2zip1.32×1.18×1.42×
zip2zip++1.15×1.06×1.18×
zip2zip++ w/o digit protection1.31×1.07×1.28×

zip2zip++ is faster than the uncompressed model in every setting we measured. Its per-step overhead is 0.35–0.50 ms on a 3.9 ms base step, about the same as zip2zip (0.34–0.42 ms). It is slower than zip2zip mainly because digit protection lowers compression. Without digit protection, it nearly matches zip2zip's speed (328.6 vs 333.0 tok/s on GSM8K) yet yields a higher accuracy across all the tasks. Prompts are compressed too, so time to first token drops from 26.0 to 20.6 ms on GSM8K.

AuthorsOpenReview ↗

Memory saving

At the request lengths in our GSM8K evaluation (about 410–490 base tokens held per request, prompt + output), zip2zip++ does not save memory: it costs slightly more than the uncompressed model, while zip2zip saves a little. zip2zip++ starts saving memory once a request holds about 830 base tokens, and its savings grow with context length. All numbers below are measured in vLLM.

Overhead breakdown

Uncompressedzip2zipzip2zip++
Hyper-encoder weights (fixed)—432 MiB864 MiB
Activation + CUDA-graph memory (fixed)—~0.3 GiB~0.3 GiB
Hypertoken table (per request, 4,096 entries)—24 MiB48 MiB
Net per-request memory vs. uncompressed (table − KV saving)0−23 MiB (saves)+24 MiB (costs)
Break-even request length—below our lengths~830 base tokens

Effect on serving capacity. Largest concurrency served without preemption at a fixed GPU budget:

GPU budget12 GiB16 GiB24 GiB
Uncompressed163672
zip2zip183680
zip2zip++ (1,280-entry tables)123680
zip2zip++ (4,096-entry tables)123264

zip2zip already saves memory on short requests. zip2zip++ pays a higher fixed cost (864 MiB of weights) plus a higher per-request cost (2x hypertoken table) that only pays off on longer contexts (≳830 base tokens).

Dear Reviewer,

Thank you for your time and effort in reviewing our paper. As the discussion period ends on November 18 (AoE), we would like to kindly check whether our responses have addressed your concerns.

Best regards,

Reviewer_UpELOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper builds on zip2zip, which lets a language model dynamically merge sequences of tokens into larger “hypertokens.” The idea is to represent the same text with fewer tokens, reducing the number of decoding steps and the space used in the context window. The catch is that the original approach can noticeably hurt model quality, especially on mathematical reasoning. zip2zip++ keeps the same basic framework and focuses on figuring out where that quality loss comes from. It separates the input and output hyper-encoders, arguing that reading a hypertoken and predicting one require different representations. It also improves embedding initialization, adjusts positional encoding to account for how many original tokens a hypertoken covers, and makes the LZW codec more robust when the model misses a merge. One particularly effective change is surprisingly simple: keeping digits out of the merging process. The paper also revisits evaluation. The same text can be generated through different combinations of base tokens and hypertokens, so scoring only one segmentation can unfairly penalize the model. Its multi-view perplexity accounts for equivalent segmentations of each target span. This corrects part of the measured quality gap, although it does not improve generation itself. The experiments show clear gains over zip2zip, particularly in perplexity and math. On Phi-3.5-mini, GSM8K accuracy rises from about 36% to 68%, compared with 75% for the uncompressed control. Digit protection alone brings it to around 61%. However, code generation and instruction following do not show reliable improvements, and generation compression becomes weaker. The paper also does not establish an end-to-end speedup.

Critical strengths

The paper identifies concrete weaknesses in zip2zip, including conflicting input/output representation needs, positional distortion, and sensitivity to digit merging. These analyses provide clear motivation for the proposed changes.

The method improves perplexity across four models and substantially recovers mathematical reasoning performance. Ablations help identify which changes contribute most to these gains.

Critical weaknesses

Although zip2zip++ substantially recovers performance on several benchmarks, code generation and instruction following remain well below the uncompressed control. On the 14B model, the reported HumanEval and IFEval scores also decrease relative to zip2zip, from 36% to 31% and from 43% to 40%, respectively. These results raise concerns about the task-dependent tradeoffs introduced by the proposed changes.

The added hyper-encoder computation introduces overhead, while digit protection reduces the achieved compression ratio. Without latency measurements against both zip2zip and the uncompressed model, it is difficult to assess whether the quality gains justify the efficiency cost. This is particularly important for tasks where quality does not improve: any substantial slowdown would further weaken the practical case for adopting the method.

Questions for authors

Question: RULER encompasses a wide range of tasks—are all of these tasks being evaluated? The paper should explicitly report the results for each RULER subtask and sub-category, and provide a control condition in which only "digit protection" (preservation of numerical data) is enabled. The current aggregate scores do not clarify whether the gains observed in long-context scenarios stem from a broader restoration of model capabilities, or if they are primarily driven by synthetic retrieval tasks that happen to be particularly well-suited to techniques involving data compression and the preservation of numerical digits.

3 comments

Reply to W1:

On the 14B model, the reported HumanEval and IFEval scores also decrease relative to zip2zip, from 36% to 31% and from 43% to 40%, respectively. These results raise concerns about the task-dependent tradeoffs introduced by the proposed changes.

We thank the reviewer for raising this point; we had the same question and addressed it in Appendix D.8 of the submission. Using paired two-sided McNemar tests on the per-problem outputs, we find that the differences between zip2zip++ and zip2zip on HumanEval and IFEval are not statistically significant, either at 14B (p = 0.32 and p = 0.16) or at 3.8B (p = 0.16 and p = 1.0). We therefore do not read these differences as a degradation (or an improvement) caused by our changes. In contrast, the gains on mathematical reasoning are large and highly significant. We re-ran the McNemar tests from the original per-problem result files and applied a Holm correction across the four comparisons:

Model / TaskNzip2zip → zip2zip++Gain (pp)Discordant (z2z only / ++ only)Holm-adj. p
Phi-3.5 (3.8B) / GSM8K1,31935.6% → 67.8%+32.260 / 4855.0e-83
Phi-3.5 (3.8B) / MATH-50050015.2% → 23.8%+8.624 / 671.5e-5
Phi-3 (14B) / GSM8K1,31951.4% → 69.5%+18.0127 / 3653.1e-27
Phi-3 (14B) / MATH-50050022.4% → 29.6%+7.230 / 663.1e-4

All four improvements stay significant after correction. These tests measure differences between the trained runs reported in the paper; they do not capture variation across training seeds.

For perplexity, we did not run a paired significance test. However, the improvement is consistent across all four corpora, both model scales, and both strict and multi-view scoring, so it is unlikely to be due to chance.

We will add the Holm-corrected table to the paper.


Reply to W2

The added hyper-encoder computation introduces overhead, while digit protection reduces the achieved compression ratio. Without latency measurements against both zip2zip and the uncompressed model, it is difficult to assess whether the quality gains justify the efficiency cost. This is particularly important for tasks where quality does not improve: any substantial slowdown would further weaken the practical case for adopting the method.

We have now measured inference speed in vLLM against both zip2zip and the uncompressed model; see our comment “Inference speed in vLLM, including concurrent requests” for details.

  • Faster than uncompressed everywhere: zip2zip++ decodes 1.15× faster on GSM8K and 1.06× faster on IFEval for single requests, and 1.10–1.18× faster under concurrent load (1–64 requests). Its time to first token is also lower.
  • Small overhead: The hyper-encoder adds 0.35–0.50 ms per decoding step (vs 0.34–0.42 ms for zip2zip, on a 3.9 ms base step). Without digit protection, zip2zip++ nearly matches zip2zip’s speed (328.6 vs 333.0 tok/s on GSM8K) while scoring 10 points higher. The remaining gap to zip2zip comes from digit protection, which is what buys the large GSM8K gain (66.3 vs 32.3).
  • Tasks without quality gains: On IFEval, zip2zip++ is 11% slower than zip2zip but still faster than the uncompressed model, and the accuracy differences there are not statistically significant (see above). Digit protection is optional and can be turned off where numerical accuracy matters less.

Reply to Q1

RULER encompasses a wide range of tasks—are all of these tasks being evaluated? The paper should explicitly report the results for each RULER subtask and sub-category, and provide a control condition in which only "digit protection" (preservation of numerical data) is enabled. The current aggregate scores do not clarify whether the gains observed in long-context scenarios stem from a broader restoration of model capabilities, or if they are primarily driven by synthetic retrieval tasks that happen to be particularly well-suited to techniques involving data compression and the preservation of numerical digits.

Yes, we evaluate all 13 tasks of the default RULER suite. Following the reviewer’s request, we have also run a control in which only digit protection is added to zip2zip, and we have broken down the results by task category. The table below summarizes the results for Phi-3.5-mini (4B) at 32K context:

Task categoryUncompressedzip2zipzip2zip + digit protectionzip2zip++
Retrieval (8)78.89.555.662.1
Multi-hop tracing (1)95.824.946.565.1
Aggregation (2)67.321.528.030.3
Question answering (2)48.227.227.331.9
All 13 tasks73.615.246.352.8

The results confirm the reviewer’s intuition that digit protection explains most of the aggregate recovery: it accounts for 31.1 of the 37.6-point gain (about 83%), with the largest effect on retrieval. However, the remaining improvement is not confined to synthetic retrieval: at 32K, zip2zip++ further improves over zip2zip + digit protection on every category, including multi-hop tracing (46.5 → 65.1) and question answering (27.3 → 31.9). This additional gain also grows with context length (6.3%, 11.3%, and 14.0% at 8K, 16K, and 32K), which is consistent with correcting the positional mismatch between compressed and original token streams.

We also have the complete results for all context lengths (4K–32K) and all individual subtasks, for both the 4B and 14B models. We will add these to the appendix of the revised paper.

Task categoryMethod4K8K16K32K
Retrieval (8)Uncompressed94.286.383.178.8
zip2zip44.422.618.69.5
zip2zip + digit protection87.566.459.055.6
zip2zip++85.070.564.262.1
Multi-hop tracing (1)Uncompressed77.593.895.295.8
zip2zip42.851.228.124.9
zip2zip + digit protection51.359.345.446.5
zip2zip++38.358.041.165.1
Aggregation (2)Uncompressed85.780.976.867.3
zip2zip59.637.623.521.5
zip2zip + digit protection62.341.528.128.0
zip2zip++50.346.538.430.3
Question answering (2)Uncompressed60.356.650.948.2
zip2zip44.128.724.727.2
zip2zip + digit protection47.533.323.727.3
zip2zip++50.036.230.531.9
All 13 tasksUncompressed86.481.478.173.6
zip2zip46.628.121.015.2
zip2zip + digit protection74.756.947.846.3
zip2zip++70.760.553.252.8

Dear Reviewer,

Thank you for your time and effort in reviewing our paper. As the discussion period ends on November 18 (AoE), we would like to kindly check whether our responses have addressed your concerns.

Best regards,

Reviewer_7wvvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the performance degradation caused by dynamic vocabulary compression based on LZW and proposes zip2zip++. The authors mitigate the quality loss due to compression by separating the input and output supertoken encoders, improving embedding initialization and positional encoding, and protecting digits from merging. Simultaneously, they introduce multi-view perplexity, incorporating different legitimate segments of the same target into the score. Experiments on four models in the Phi and Llama series show that zip2zip++ has lower perplexity than zip2zip, and also improves mathematical inference and long-context retrieval performance. Ablation experiments demonstrate that digit protection is the main source of improved mathematical performance, but overall performance still does not reach the level of the uncompressed model.

Critical strengths
  1. This paper identifies specific factors contributing to quality loss in dynamic vocabulary compression: First and last token substitution experiments showed that the output encoder relies more on the prefix, while the input encoder is more balanced in its responses to changes at both ends, supporting the separation of the two types of encoders. Furthermore, in a control with fixed model weights, preserving digits only during evaluation improves GSM8K performance.

  2. zip2zip++ substantially improves mathematical reasoning and lowers perplexity across multiple models. The perplexity gains remain when the methods are evaluated using the same scoring criteria.

Critical weaknesses
  1. The control experiments did not exclude the influence of training sample differences. Compressed training discards approximately 6.5% of windows with compression ratios exceeding 2×, while the uncompressed baseline retains all windows. This choice significantly impacts the results: removing the selection drops the full method's GSM8K accuracy from 67.7% to 59.5%. Digit protection also alters which windows are retained, thus potentially altering the training data for some component comparisons. This makes it difficult to separate the effects of the method changes from those of data selection. Key controls need to be added on fixed raw text windows, especially the uncompressed baseline using the same set of retained data.

  2. Mathematical performance improves, but the advantage at comparable compression ratios remains unclear. In Table 4, digit protection alone raises GSM8K accuracy from approximately 36% to 61%, accounting for most of the gain achieved by the full method (68%). Meanwhile, for GSM8K generation with Phi-3.5-mini, zip2zip and full zip2zip++ achieve compression ratios of 1.53× and 1.25×, respectively. The mathematical gains therefore come with less compression. Digit protection is a useful design choice, but a comparison at matched compression ratios, or an accuracy–compression curve, would help clarify whether it improves the trade-off between accuracy and compression.

Questions for authors
  1. Please fixate on a batch of original text windows and compare Uncompressed, zip2zip, and zip2zip++ on this batch of text. After controlling for differences in the training samples, how do the gains of the main results and key components change?
  2. Please provide the accuracy-compression ratio curves for both zip2zip and zip2zip++ methods, or offer a comparison with similar compression ratios. Does zip2zip++ still have a significant advantage at the same level of compression?
  3. Please explain why training windows with compression ratios greater than 2 are selected for dropping, what types of content these windows contain, and how digit protection alters the proportion and composition of the dropped windows.
3 comments

Reply to Q1

Please fixate on a batch of original text windows and compare Uncompressed, zip2zip, and zip2zip++ on this batch of text. After controlling for differences in the training samples, how do the gains of the main results and key components change?

We thank Reviewer 7wvv for pointing out a hidden confounding factor in our original comparison. In the original submission, the methods were not trained on exactly the same text windows, so part of the observed differences could be attributed to variations in training data rather than to the methods themselves.

Following the reviewer’s suggestion, we constructed a shared subset of training windows that are retained under all settings, and retrained Uncompressed, zip2zip, zip2zip++, and the ablations on exactly these windows, in the same order and with no skipped windows.

CorpusRecipeInspected chunksRuntime skips
oldUncompressed12801313
oldzip2zip14626018260
oldzip2zip++1371169116
new (sharedS)Uncompressed1280000
new (sharedS)zip2zip1280000
new (sharedS)zip2zip++1280000

The results are shown in rows No. 5–9 of the table in our general comment above. After controlling for training data, the main conclusions remain unchanged: zip2zip++ (No. 8) continues to outperform zip2zip (No. 6) on all tasks. In particular, the mathematical gains persist: digit protection improves GSM8K from 32.6% to 63.4%, and the full recipe reaches 69.5%. The gain of zip2zip++ over zip2zip is 36.9 points, compared with 32.1 points in the original setting. Notably, zip2zip++ adds another 6.1 points over digit protection alone, even though the two are trained on identical encoded inputs; this difference is statistically significant in a deterministic re-evaluation (paired exact McNemar test, Holm-adjusted p = 0.002).


Reply to Q2

Please provide the accuracy-compression ratio curves for both zip2zip and zip2zip++ methods, or offer a comparison with similar compression ratios. Does zip2zip++ still have a significant advantage at the same level of compression?

We thank the reviewer for this suggestion. Since compression is governed by the LZW algorithm, we cannot set the compression ratio arbitrarily; we only have coarse control via, e.g., the maximum merge size M, the codebook size K, or disabling certain merges. A fine-grained accuracy–compression curve is therefore difficult to produce by nature.

Matched compression. No. 6 (zip2zip) and No. 9 (zip2zip++ w/o digit protection) in our general comment have closely matched generation compression ratios (1.38x vs. 1.35x) and the same training data. At this level, zip2zip++ outperforms zip2zip on every task and metric (e.g., 6-MC Avg 66.6 → 68.4, GSM8K 32.6 → 38.4, TriviaQA 39.5 → 43.9, Byte-PPL 1.677 → 1.637), showing that its advantage does not come from compressing less.

Accuracy–compression sweep. We additionally swept M and K at inference time on the fixed checkpoints and evaluated GSM8K (table below). Across all measured operating points, zip2zip++ (66.5–68.7%) stays far above zip2zip (26.6–53.3%). At comparable compression, zip2zip++ reaches 67.6% at C = 1.20 versus 26.6% for zip2zip at C = 1.17.

MKzip2zip: accuracy / Czip2zip++: accuracy / C
1409653.30 / 1.000066.49 / 1.0000
2409636.92 / 1.379068.69 / 1.1983
3409634.04 / 1.464768.61 / 1.2400
425626.61 / 1.169267.55 / 1.1978
451232.98 / 1.463968.23 / 1.2524
4102433.97 / 1.479168.23 / 1.2524
4204833.97 / 1.479168.23 / 1.2524
4409633.97 / 1.479168.23 / 1.2524

Reply to Q3

Please explain why training windows with compression ratios greater than 2 are selected for dropping, what types of content these windows contain, and how digit protection alters the proportion and composition of the dropped windows.

The filter was an engineering choice rather than a deliberate data-selection step. Our loader produces fixed-length training sequences of 2,048 positions from 4,096-token raw windows. Windows that compress by more than about 2× become too short to fill a sequence, and padding them would waste compute, so we simply skipped them. We did not intend it as a quality filter.

To understand its effect, we analyzed the skipped windows (12.4% under zip2zip, 6.3% with digit protection). They are mostly highly repetitive text (71% repeated 4-grams vs. 17% in retained windows). However, the windows rescued by digit protection are digit-dense (31% digit tokens vs. 6%), so in the original setup, digit-protected models could have benefited from additional math-like data. Yet, the shared-window experiments in Q1 rule this out: with all methods trained on the same windows, zip2zip++ still improves GSM8K from 32.6% to 69.5%.

Thank you for the detailed rebuttal.

The additional experiments and explanations resolve most of my concerns.

  • For weakness 1, the experiments using the same training windows address my concern about differences in training data.
  • For weakness 2, the comparisons at similar compression ratios and the accuracy–compression sweep provide clearer evidence of an improved trade-off between accuracy and compression.
  • For question 3, the explanation of the window-filtering rule and the analysis of the dropped windows are reasonable. For these reasons, I raise my score from 2 (weak rejection) to 3 (weak acceptance).

Dear Reviewer,

Thank you very much for your thoughtful follow-up and for raising your score. We truly appreciate the time you spent reviewing our rebuttal.

The authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

New Experiments with Matched Training Data and Digit-Protection Ablations

#Variant (Phi-3.5-mini-instruct, 8k steps)DatasetRuntime Skip %ARC-C (norm)ARC-E (norm)HellaSwag (norm)OBQA (norm)PIQA (norm)WG6-MC AvgGSM8K (flex)TriviaQAWikiText Byte-PPL (Multi-View)Gen / In Comp. Ratio
1Uncompressed ControlOriginal0.01%58.183.573.647.880.674.769.974.244.11.6001.00x / 1.00x
2zip2zipOriginal12.48%53.881.570.346.678.974.767.635.635.81.6961.30x / 1.13x
3zip2zip + Digit protectionOriginal6.65%54.081.470.246.678.771.367.061.037.01.7111.25x / 1.11x
4zip2zip++Original6.65%57.083.071.846.279.473.268.467.739.41.6851.27x / 1.11x
5Uncompressed ControlsharedS0.00%59.484.073.447.681.574.570.171.247.11.577 (w=11.4)1.00x / 1.00x
6zip2zipsharedS0.00%52.080.169.945.478.973.566.632.639.51.677 (w=15.9)1.38x / 1.13x
7zip2zip + Digit protectionsharedS0.00%53.480.670.645.078.672.066.763.441.11.663 (w=15.2)1.23x / 1.10x
8zip2zip++sharedS0.00%55.682.272.446.680.074.368.569.543.21.628 (w=13.6)1.19x / 1.08x
9zip2zip++ - Digit protectionsharedS0.00%55.282.672.246.879.973.868.438.443.91.637 (w=13.9)1.35x / 1.08x

To address the reviewer 7wvv and UpEL’s concerns, we ran additional experiments, summarized in the table below. Rows No. 1–4 are the experiments reported in the original submission, while rows No. 5–9 show the new results. The only difference between the two groups is the training data: in the new experiments, all methods (Uncompressed, zip2zip, zip2zip++, and the ablations) are trained on exactly the same training windows. This removes training data selection as a confounding factor, as Reviewer 7wvv correctly pointed out.

The new results lead to the same conclusions as our submission: zip2zip++ improves over zip2zip on all benchmarks, with GSM8K rising from 32.6% to 69.5%. We have also run ablations to isolate the effect of digit protection, as requested by Reviewer UpEL. Digit protection provides the largest improvement (GSM8K 32.6% → 63.4%), the other design choices contribute a further noticeable gain (GSM8K 63.4% → 69.5%, WikiText byte-PPL 1.663 → 1.628), and the full recipe performs best.

We will refer to this table in our responses to each reviewer.

AuthorsOpenReview ↗

Inference speed in vLLM, including concurrent requests (UpEL, eMfq)

To provide the most realistic inference results, we integrated zip2zip and zip2zip++ into vLLM, so every number reported below comes from a real production serving engine.

Setup: one H100 80GB, BF16, greedy decoding, LoRA merged, and the same vLLM configuration for every model (CUDA graphs, asynchronous scheduling). We use the shared-data checkpoints of our first comment and the same 300 GSM8K and 200 IFEval prompts (budget 1,024 base tokens). Timings include everything zip2zip adds: the LZW codec, the hyper-encoders and the hypertoken output head. Throughput counts base tokens (original tokenizer) per second.

One request at a time (decoding throughput, speedup vs. uncompressed):

GSM8K acc.GSM8K tok/sIFEval acc.*IFEval tok/s
Uncompressed70.7251.5 (1.00×)40.0256.8 (1.00×)
zip2zip32.3333.0 (1.32×)30.5303.8 (1.18×)
zip2zip++66.3289.5 (1.15×)35.5271.3 (1.06×)
zip2zip++ w/o digit protection42.3328.6 (1.31×)35.0275.0 (1.07×)

95% CIs for zip2zip++: [1.13, 1.17] on GSM8K, [1.02, 1.10] on IFEval. GSM8K accuracy is scored on the vLLM outputs and is within 1.3 points of our reference implementation (BF16 near-ties change some greedy outputs). *IFEval accuracy: same checkpoints and prompts, reference implementation.

Concurrent requests (closed loop over all 500 prompts, aggregate base tok/s; the last row comes from a separate run against its own uncompressed baseline):

Concurrent requests183264
Uncompressed (tok/s)2491,5263,7844,960
zip2zip1.26×1.27×1.32×1.42×
zip2zip++1.10×1.09×1.14×1.18×
zip2zip++ w/o digit protection1.17×1.18×1.22×1.28×
  1. Overhead. A zip2zip++ decoding step takes 0.35–0.50 ms longer than an uncompressed one (3.9 ms), and a zip2zip step 0.34–0.42 ms longer. Speed therefore follows compression: zip2zip++ is faster than the uncompressed model in every setting we measured, and slower than zip2zip because it compresses less, mostly due to digit protection.
  2. Serving under load. With more concurrent requests, the per-step overhead is shared across the batch, and the decoder itself gets cheaper because compressed sequences are shorter (5.5 vs 6.1 ms per decoder forward at 32 requests). The speedup of zip2zip++ rises to 1.18× at 64 requests, and its 95th-percentile request time drops from 8.3 s to 5.2 s, partly because its answers are also shorter.
  3. Whole requests. Prompts are compressed too, so time to first token is lower (20.6 vs 26.0 ms on GSM8K), and a GSM8K request takes 0.61 s on average vs 0.84 s for the uncompressed model.
  4. Same compression, same speed. Without digit protection, zip2zip++ decodes GSM8K at nearly zip2zip's speed (328.6 vs 333.0 tok/s, compression 1.50× vs 1.47×) and scores 10 points higher (42.3 vs 32.3); see our reply to Reviewer 7wvv (Q2). On IFEval it is 9% slower than zip2zip.
  5. Memory. We did not profile memory inside vLLM; by accounting, the two hyper-encoders add 0.91 GB of weights (zip2zip: 0.45 GB), and each request slot holds hypertoken embedding tables of 48 MiB (zip2zip: 24 MiB). The KV cache stores compressed tokens, saving 0.375 MiB per token avoided. At zip2zip++'s compression this repays the tables once a request exceeds about 700 base tokens (zip2zip: about 230), so for the 500–1,000-token requests here zip2zip++ is roughly memory-neutral and zip2zip saves memory; the saving grows with context length.
AuthorsOpenReview ↗

Memory usage (measured in vLLM).

Fixed overhead. The hyper-encoders add 432 MiB of weights for zip2zip and 864 MiB for zip2zip++, plus about 0.3 GiB of activation and CUDA-graph memory.

Per-request trade-off. Each request slot holds hypertoken embedding tables (24 MiB for zip2zip, 48 MiB for zip2zip++), while the KV cache stores compressed tokens and therefore shrinks. On these workloads (300 GSM8K samples + 200 IFEval samples, 410–490 base tokens held per request):

  • zip2zip: the KV saving exceeds the table cost (net +23 MiB per request).
  • zip2zip++: the KV saving does not cover the table cost (net −24 MiB per request); break-even is at about 830 base tokens held.

Concurrency at a fixed GPU budget. Largest concurrency served without preemption:

Model12 GiB16 GiB24 GiBThroughput vs. uncompressed
Uncompressed1636721.00×
zip2zip1836801.39–1.54×
zip2zip++ (4,096-entry tables)1232640.96–1.16×
zip2zip++ (1,280-entry tables)123680—

The hypertoken embedding tables are sized for a 4,096-entry dictionary by default. Sizing them for 1,280 entries (the largest dictionary in any run had 1,236) leaves zip2zip++'s outputs unchanged on 100/100 prompts, and its concurrency then matches or exceeds the uncompressed model at 16 and 24 GiB.

Memory can be a cost of zip2zip++ for short requests, but this cost shrinks as contexts get longer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.