- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Q3.6, a lossless compressor for already-quantized LLM weights. It uses native quantization metadata, primarily scale codes, to partition weights into buckets and applies a separate entropy model to each bucket. It also introduces GPU decoding and fused decode-and-compute kernels for storage-offloaded inference. Experiments report consistent storage reductions over zstd and serving improvements under bandwidth and memory constraints.
Critical strengths
• The method preserves the existing quantized representation and explicitly accounts for native metadata, routing information, and coding tables. The shuffled-routing and random-bucket controls provide evidence that metadata conditioning contributes to the compression gains. • The evaluation covers several models, quantization formats, and bit widths. The GPU implementation demonstrates practical applicability beyond offline compression.
Critical weaknesses
• The methodological contribution is limited. The central approach combines metadata-based partitioning with established entropy coding. Although using native scales is useful, the paper does not sufficiently establish a substantial advance over existing context-conditioned coding principles. The additional storage reduction over zstd is modest—approximately 2–3% on average in the reported target-tensor evaluations. • The advantage over the closest related approach remains insufficiently established. Tan et al. also study lossless compression of quantized LLM weights with integrated GPU decompression, but no matched comparison is provided. The stated differences in checkpoints and deployment pipelines explain the difficulty of comparison but leave the relative contribution unclear. Furthermore, the largest serving gains combine compression with weight-buffer removal and increased batch capacity. The FlexGen control helps, but these gains should not be interpreted as evidence of a similarly large benefit from metadata conditioning itself.
Questions for authors
• Under matched quantized inputs and storage accounting, how much improvement does metadata conditioning provide over the per-layer shared-codebook approach of Tan et al.? • With the same streaming implementation, memory policy, and batch size, what end-to-end benefit does metadata-conditioned coding provide over a single unconditioned entropy model?
2 comments
Q2
"With the same streaming implementation, memory policy, and batch size, what end-to-end benefit does metadata-conditioned coding provide over a single unconditioned entropy model?"
Following your suggestions, we conducted additional experiments at a fixed batch size with the same streaming implementation and buffer management policies for all methods, including the uncompressed baseline (Raw). The results show that metadata-conditioned coding provides both higher throughput and smaller total deployment file sizes than single unconditioned entropy models.
Specifically, we evaluate Qwen3-8B-AWQ in vLLM [g] with native Marlin [h] and FP16 activations on an RTX PRO 6000 Blackwell, using batch 16, a 12 GiB GPU memory budget and 1.5 GB/s SSD offloading. Each request has 512 input and 32 output tokens, and throughput includes prefill. All methods release the full-model weight buffer and reuse two layer buffers with identical scheduling, prefetching and KV-cache allocation. The compressed variants use the same GPU decompressor. The throughput results are shown below:
| Method | Throughput gain over Raw |
|---|---|
| ANS, no scale-based buckets, shared table | +5.32% |
| ANS, same buckets, shared table | +6.92% |
| Q3.6, metadata-conditioned ANS | +8.56% |
Q3.6 raises the throughput gain over Raw by 3.24 percentage points relative to the unbucketed shared model and by 1.64 percentage points relative to the same-bucket shared model. In the last two rows, the buckets are identical; metadata conditioning replaces a single shared probability table with a separate table for each scale bucket. This comparison directly shows the end-to-end benefit of metadata-conditioned coding over a single unconditioned entropy model under the same streaming implementation, memory policy and batch size.
Besides improving throughput, metadata conditioning reduces total deployment file size. For Qwen3-8B, this size is 5,510.552 MiB with the unbucketed shared model and 5,512.513 MiB with the same-bucket shared model. Q3.6 reduces it to 5,489.548 MiB, saving 21.00 MiB and 22.97 MiB, respectively.
These results and the analysis have been added to revised Section 4.5 (Table 6) and Appendix C.5.
References
[a] Moshik Hershcovitch, Andrew Wood, Leshem Choshen, et al. ZipNN: Lossless Compression for AI Models. arXiv:2411.05239, 2024.
[b] Donghyeon Joo, Ramyad Hadidi, Soheil Feizi, and Bahar Asgari. Endor: Hardware-Friendly Sparse Format for Offloaded LLM Inference. arXiv:2406.11674, 2024.
[c] Yann Collet. FiniteStateEntropy. Software repository.
[d] NVIDIA. nvCOMP: GPU Data Compression. Official documentation.
[e] Hongshi Tan, Yao Chen, Gustavo Alonso, et al. Approaching Shannon Bound with Lossless LLM Weight Compression. ISCA, 2026.
[f] Ying Sheng, Lianmin Zheng, Binhang Yuan, et al. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. ICML, 2023.
[g] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP, 2023.
[h] Elias Frantar, Roberto L. Castro, Jiale Chen, et al. MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models. arXiv:2408.11743, 2024.
We thank you for your careful reading and thoughtful comments on the methodological contribution and the additional storage reduction over Zstd. We have added comparisons addressing prior work on the same inputs and the contribution of conditioning.
W1
Thank you for raising these concerns about the methodological contribution and the modest additional storage reduction over Zstd. Zstd is a strong, established baseline for this task; ZipNN [a] explicitly describes it as a state-of-the-art compressor. Our comparison on Qwen3-8B/AWQ/W4 illustrates this challenge: ZipNN, DeepCABAC and PNG-4 require 4.124, 3.913 and 4.020 bits/weight, respectively, all higher than Zstd's 3.802 (revised Table 14, Appendix B.7). Against this strong baseline, Q3.6 consistently achieves lower total storage rates in all 21 tested configurations, for both MLP and attention target tensors. With the FSE [c] backend, the average additional reductions are 2.17% and 3.10%, respectively. Q3.6 thus provides an additional lossless compression layer for already-quantized weights, with no storage regression across the tested models, quantization methods and bit widths.
Beyond storage reduction, Q3.6 also delivers higher GPU decompression throughput at lower storage rates than nvCOMP-Zstd [d]. Revised Table 4 reports that, for W3/W4/W6, our GPU decompressor achieves 769.5/650.6/522.9 GB/s, compared with 116.2/86.4/46.7 GB/s for nvCOMP-Zstd, 6.6×/7.5×/11.2× faster decompression. The corresponding total storage rates are also lower: 2.793/3.744/5.786 versus 3.026/3.821/5.849 bits/weight.
We have made the aforementioned storage and decompression benefits explicit in revised Section 4.4 and Appendix B.12.
W2 / Q1
Thank you for the helpful suggestions. To quantitatively compare with Tan et al. [e], we evaluated our implementation of their "per-layer shared-codebook approach" on the same public Qwen3-8B-AWQ checkpoint. The storage comparison covers all 252 projections. For serving, both methods use vLLM [g] with native Marlin [h], batch 16, a 12 GiB GPU memory budget and 1.5 GB/s SSD offloading. Revised Tables 3 and 6 report the storage rates and throughput gains over the uncompressed streaming baseline (Raw):
Q3.6 achieves both lower storage and higher throughput than Tan et al.'s "per-layer shared-codebook approach", reducing storage by 0.0122 bits/weight and increasing the throughput gain over Raw by 1.99 percentage points.
To distinguish the benefit of metadata conditioning from buffer removal and increased batch capacity, we further compared Q3.6 with unconditioned ANS controls using the same GPU decompressor. All methods, including Raw, release the full-model weight buffer and reuse two layer buffers, with identical KV-cache allocation, scheduling and a fixed batch size of 16. Throughput gains over Raw are 5.32% for an unbucketed shared model, 6.92% for a shared model using Q3.6's buckets, and 8.56% for Q3.6. With the buckets held fixed, conditioning increases the throughput gain by 1.64 percentage points. This provides additional support beyond the FlexGen [f] controls for the benefit of metadata conditioning itself (revised Table 6 and Appendix C.5).
Together, the matched comparison with Tan et al. shows lower storage and higher serving throughput, while the shared-table controls confirm an additional throughput benefit from conditioning at the same batch size and under the same memory policy.