acceptodds
Under review as a conference paper at ICLR 2027

Q3.6: Metadata‑Conditioned Lossless Compression for Quantized LLMs

Abstract

Low-bit quantization has become a popular way to deploy large language models (LLMs) on resource-constrained devices. However, after a model has already been quantized, residual redundancy in the quantized weights remains largely unexploited, while further quantizing to even lower bit widths can cause unacceptable generation-quality loss. This motivates dedicated lossless compressors for quantized LLMs: they reduce storage after quantization while preserving the exact quantized weights. Existing lossless LLM compressors target floating-point formats such as bfloat16 (BF16), re-quantize before coding, or treat weights as generic byte streams; none conditions on the native metadata of a fixed low-bit object. We propose Q3.6, a lossless compressor for already-quantized LLMs with bit-exact decoding. Q3.6 builds on the observation that stored metadata, such as the scales, is predictive of the symbol distribution of the weight blocks it describes. It derives deterministic entropy buckets from the native metadata and encodes each bucket with a matched table-based entropy model, so blocks with similar metadata share a bucket and a bitstream. Q3.6 stores less than a strong Zstandard (zstd) baseline in every evaluated setting, covering three models, five quantized formats, and three bit widths, and saves about 10% on average relative to the uncompressed target tensors. In storage-offloaded serving at 1.5 GB/s, Q3.6 gives the smallest stream and achieves the fastest decoding among the evaluated GPU codecs; fed to a fused decode-plus-GEMM kernel, the entropy-coded stream lowers the time per output token by 7 to 10%, and the weight buffers the kernel frees yield 16% to 90% more KV cache and up to 2.1× the throughput in vLLM on 8 to 24 GB GPU budgets, with tokens identical to the stock engines. Code is available at https://anonymous.4open.science/r/iclr2027-code-4B00.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
large language models, lossless compression, entropy coding, quantization metadata, memory-efficient inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6suzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Q3.6, a lossless compressor for already-quantized LLM weights. It uses native quantization metadata, primarily scale codes, to partition weights into buckets and applies a separate entropy model to each bucket. It also introduces GPU decoding and fused decode-and-compute kernels for storage-offloaded inference. Experiments report consistent storage reductions over zstd and serving improvements under bandwidth and memory constraints.

Critical strengths

• The method preserves the existing quantized representation and explicitly accounts for native metadata, routing information, and coding tables. The shuffled-routing and random-bucket controls provide evidence that metadata conditioning contributes to the compression gains. • The evaluation covers several models, quantization formats, and bit widths. The GPU implementation demonstrates practical applicability beyond offline compression.

Critical weaknesses

• The methodological contribution is limited. The central approach combines metadata-based partitioning with established entropy coding. Although using native scales is useful, the paper does not sufficiently establish a substantial advance over existing context-conditioned coding principles. The additional storage reduction over zstd is modest—approximately 2–3% on average in the reported target-tensor evaluations. • The advantage over the closest related approach remains insufficiently established. Tan et al. also study lossless compression of quantized LLM weights with integrated GPU decompression, but no matched comparison is provided. The stated differences in checkpoints and deployment pipelines explain the difficulty of comparison but leave the relative contribution unclear. Furthermore, the largest serving gains combine compression with weight-buffer removal and increased batch capacity. The FlexGen control helps, but these gains should not be interpreted as evidence of a similarly large benefit from metadata conditioning itself.

Questions for authors

• Under matched quantized inputs and storage accounting, how much improvement does metadata conditioning provide over the per-layer shared-codebook approach of Tan et al.? • With the same streaming implementation, memory policy, and batch size, what end-to-end benefit does metadata-conditioned coding provide over a single unconditioned entropy model?

2 comments

We thank you for your careful reading and thoughtful comments on the methodological contribution and the additional storage reduction over Zstd. We have added comparisons addressing prior work on the same inputs and the contribution of conditioning.

W1

"The methodological contribution is limited. The central approach combines metadata-based partitioning with established entropy coding. Although using native scales is useful, the paper does not sufficiently establish a substantial advance over existing context-conditioned coding principles. The additional storage reduction over zstd is modest, approximately 2–3% on average in the reported target-tensor evaluations."

Thank you for raising these concerns about the methodological contribution and the modest additional storage reduction over Zstd. Zstd is a strong, established baseline for this task; ZipNN [a] explicitly describes it as a state-of-the-art compressor. Our comparison on Qwen3-8B/AWQ/W4 illustrates this challenge: ZipNN, DeepCABAC and PNG-4 require 4.124, 3.913 and 4.020 bits/weight, respectively, all higher than Zstd's 3.802 (revised Table 14, Appendix B.7). Against this strong baseline, Q3.6 consistently achieves lower total storage rates in all 21 tested configurations, for both MLP and attention target tensors. With the FSE [c] backend, the average additional reductions are 2.17% and 3.10%, respectively. Q3.6 thus provides an additional lossless compression layer for already-quantized weights, with no storage regression across the tested models, quantization methods and bit widths.

Beyond storage reduction, Q3.6 also delivers higher GPU decompression throughput at lower storage rates than nvCOMP-Zstd [d]. Revised Table 4 reports that, for W3/W4/W6, our GPU decompressor achieves 769.5/650.6/522.9 GB/s, compared with 116.2/86.4/46.7 GB/s for nvCOMP-Zstd, 6.6×/7.5×/11.2× faster decompression. The corresponding total storage rates are also lower: 2.793/3.744/5.786 versus 3.026/3.821/5.849 bits/weight.

We have made the aforementioned storage and decompression benefits explicit in revised Section 4.4 and Appendix B.12.

W2 / Q1

"The advantage over the closest related approach remains insufficiently established. Tan et al. also study lossless compression of quantized LLM weights with integrated GPU decompression, but no matched comparison is provided. The stated differences in checkpoints and deployment pipelines explain the difficulty of comparison but leave the relative contribution unclear. Furthermore, the largest serving gains combine compression with weight-buffer removal and increased batch capacity. The FlexGen control helps, but these gains should not be interpreted as evidence of a similarly large benefit from metadata conditioning itself."

"Under matched quantized inputs and storage accounting, how much improvement does metadata conditioning provide over the per-layer shared-codebook approach of Tan et al.?"

Thank you for the helpful suggestions. To quantitatively compare with Tan et al. [e], we evaluated our implementation of their "per-layer shared-codebook approach" on the same public Qwen3-8B-AWQ checkpoint. The storage comparison covers all 252 projections. For serving, both methods use vLLM [g] with native Marlin [h], batch 16, a 12 GiB GPU memory budget and 1.5 GB/s SSD offloading. Revised Tables 3 and 6 report the storage rates and throughput gains over the uncompressed streaming baseline (Raw):

MethodOffline storage rate (bits/weight)Serving throughput gain over Raw
ANS following Tan et al. [e]3.7348+6.57%
Q3.63.7226+8.56%

Q3.6 achieves both lower storage and higher throughput than Tan et al.'s "per-layer shared-codebook approach", reducing storage by 0.0122 bits/weight and increasing the throughput gain over Raw by 1.99 percentage points.

To distinguish the benefit of metadata conditioning from buffer removal and increased batch capacity, we further compared Q3.6 with unconditioned ANS controls using the same GPU decompressor. All methods, including Raw, release the full-model weight buffer and reuse two layer buffers, with identical KV-cache allocation, scheduling and a fixed batch size of 16. Throughput gains over Raw are 5.32% for an unbucketed shared model, 6.92% for a shared model using Q3.6's buckets, and 8.56% for Q3.6. With the buckets held fixed, conditioning increases the throughput gain by 1.64 percentage points. This provides additional support beyond the FlexGen [f] controls for the benefit of metadata conditioning itself (revised Table 6 and Appendix C.5).

Together, the matched comparison with Tan et al. shows lower storage and higher serving throughput, while the shared-table controls confirm an additional throughput benefit from conditioning at the same batch size and under the same memory policy.

Q2

"With the same streaming implementation, memory policy, and batch size, what end-to-end benefit does metadata-conditioned coding provide over a single unconditioned entropy model?"

Following your suggestions, we conducted additional experiments at a fixed batch size with the same streaming implementation and buffer management policies for all methods, including the uncompressed baseline (Raw). The results show that metadata-conditioned coding provides both higher throughput and smaller total deployment file sizes than single unconditioned entropy models.

Specifically, we evaluate Qwen3-8B-AWQ in vLLM [g] with native Marlin [h] and FP16 activations on an RTX PRO 6000 Blackwell, using batch 16, a 12 GiB GPU memory budget and 1.5 GB/s SSD offloading. Each request has 512 input and 32 output tokens, and throughput includes prefill. All methods release the full-model weight buffer and reuse two layer buffers with identical scheduling, prefetching and KV-cache allocation. The compressed variants use the same GPU decompressor. The throughput results are shown below:

MethodThroughput gain over Raw
ANS, no scale-based buckets, shared table+5.32%
ANS, same buckets, shared table+6.92%
Q3.6, metadata-conditioned ANS+8.56%

Q3.6 raises the throughput gain over Raw by 3.24 percentage points relative to the unbucketed shared model and by 1.64 percentage points relative to the same-bucket shared model. In the last two rows, the buckets are identical; metadata conditioning replaces a single shared probability table with a separate table for each scale bucket. This comparison directly shows the end-to-end benefit of metadata-conditioned coding over a single unconditioned entropy model under the same streaming implementation, memory policy and batch size.

Besides improving throughput, metadata conditioning reduces total deployment file size. For Qwen3-8B, this size is 5,510.552 MiB with the unbucketed shared model and 5,512.513 MiB with the same-bucket shared model. Q3.6 reduces it to 5,489.548 MiB, saving 21.00 MiB and 22.97 MiB, respectively.

These results and the analysis have been added to revised Section 4.5 (Table 6) and Appendix C.5.

References

[a] Moshik Hershcovitch, Andrew Wood, Leshem Choshen, et al. ZipNN: Lossless Compression for AI Models. arXiv:2411.05239, 2024.

[b] Donghyeon Joo, Ramyad Hadidi, Soheil Feizi, and Bahar Asgari. Endor: Hardware-Friendly Sparse Format for Offloaded LLM Inference. arXiv:2406.11674, 2024.

[c] Yann Collet. FiniteStateEntropy. Software repository.

[d] NVIDIA. nvCOMP: GPU Data Compression. Official documentation.

[e] Hongshi Tan, Yao Chen, Gustavo Alonso, et al. Approaching Shannon Bound with Lossless LLM Weight Compression. ISCA, 2026.

[f] Ying Sheng, Lianmin Zheng, Binhang Yuan, et al. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. ICML, 2023.

[g] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP, 2023.

[h] Elias Frantar, Roberto L. Castro, Jiale Chen, et al. MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models. arXiv:2408.11743, 2024.

Reviewer_sv6nOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Q3.6, a lossless compressor for already-quantized LLM weights that preserves the quantized object bit for bit. Its key observation is that native quantization metadata, in particular the stored per-group scale codes, is predictive of the distribution of the low-bit payload symbols: groups with similar scales tend to have similar symbol statistics, and groups at the two ends of the scale range concentrate probability on a few codes. Q3.6 therefore derives deterministic, family-specific buckets from the stored scale codes using contiguous code intervals with stored split points, and encodes each bucket with its own table-based ANS model. The decoder recomputes the same bucket assignment from the stored metadata and scatters the decoded symbols back to their original positions. Native metadata, route metadata and coding tables are all charged in the reported rates. Across Qwen3-8B, Qwen3-14B and Llama-2-7B with AWQ, OmniQuant and MXFP4 at several bit widths, Q3.6 stores less than a strong zstd baseline in every setting, reducing the target tensors by about 10% on average relative to their uncompressed size. Negative controls with shuffled routes and random buckets confirm that the benefit of conditioning comes from the correspondence between metadata and payload. On the systems side, the paper contributes a GPU ANS decoder that is 2.1–2.3 times as fast as the public dietGPU decoder, and a fused decode-and-multiply kernel integrated into vLLM, SGLang and FlexGen. In a storage-offloaded serving setting, this reduces time per output token by 7–10%, and in vLLM the freed weight buffers allow 16–90% more KV cache and up to 2.1 times higher throughput, with tokens identical to the stock engines in the tested runs.

Critical strengths

The first strength is a practical and well-motivated idea that is carefully validated. Compressing an already-quantized model losslessly avoids the quality risk of moving to lower bit widths, and using the native quantization metadata, which the decoder already has, as an entropy-coding context is a natural and effective way to exploit the structure that generic byte-stream compressors ignore. The storage accounting is complete, charging native metadata, route descriptors and coding tables, and the shuffled-route and random-bucket controls convincingly show that the gain from conditioning stems from the matched pairing of scales and payload symbols rather than from simply using more streams. Q3.6 improves over zstd consistently across models, quantization methods and bit widths. The second strength is the substantial and complete systems contribution. The optimized GPU decoder, the fused decode-and-multiply kernel that avoids full-model weight materialization, and the integration into three serving engines turn the storage gain into measurable end-to-end benefits in latency, KV-cache capacity and throughput. The released artifact and the reproduction of the public AWQ checkpoints further strengthen the practical value of the work.

Critical weaknesses

My first concern is that the end-to-end gains are not decomposed at the system level. The reported serving results combine three components: the metadata-conditioned coding, the faster GPU decoder, and the fused kernel that removes the need to keep full weights resident. The paper provides a good decomposition for storage (Tables 12 and 13), but the serving experiments only compare the full Q3.6 system with uncompressed weights. A system-level ablation, for example the same decoder, fused kernel and memory policy with an unconditioned or family-only ANS stream, would make clear how each component contributes and help readers adopt the parts relevant to their setting. Related to this, in the vLLM capacity experiments the raw baseline streams weights while still keeping the full qweight resident in GPU memory, whereas Q3.6 does not. A streaming baseline that also releases the resident weight buffers would make the comparison more symmetric; the FlexGen experiments move in this direction, but the vLLM headline results do not include such a control. My second concern is the scope and positioning of the serving results. All serving experiments assume that the weights are read from storage on every forward pass under a 1.5 GB/s cap, and the GPU memory budgets are emulated as fractions of a 96 GB card rather than measured on smaller devices. The paper acknowledges that when the weights fit in GPU memory the engines’ native kernels remain preferable, so the serving benefits apply mainly to storage-bound deployments, and this scope could be stated more prominently. In addition, the concurrent work of Tan et al. (2026) also studies lossless compression of quantized LLM weights with ANS decoding fused into GEMM. The paper discusses it only qualitatively; a feature comparison or a controlled comparison on a common checkpoint would clarify the specific contribution of Q3.6 relative to this closest related work.

Questions for authors

Could the authors report the storage and serving results of the same system with an unconditioned or family-only ANS stream, keeping the decoder, the fused kernel and the memory policy fixed? Similarly, how does Q3.6 compare in the vLLM capacity experiments with an uncompressed streaming baseline that also releases the resident weight buffers? The answers to these two questions would most directly affect my assessment. Could the authors provide a clearer comparison with Tan et al. (2026), at least in terms of features and, where feasible, on a shared checkpoint? I would also appreciate a clearer distinction between what is guaranteed and what is observed: the quantized weights and metadata are recovered bit for bit, while the fused kernel changes the floating-point accumulation order, so identical tokens are an empirical observation on the tested runs. Finally, results on larger models, more publicly released quantized checkpoints, and whole-checkpoint compression rates would further support the positioning of Q3.6 as a general post-quantization storage layer; I see these as suggestions rather than requirements.

3 comments

We appreciate your thoughtful feedback and recognition of the value of post-quantization lossless compression. Following your suggestions, we have added system ablations and evaluations on larger public AWQ checkpoints, which we present below.

W1 / Q1 / Q2

"My first concern is that the end-to-end gains are not decomposed at the system level. The reported serving results combine three components: the metadata-conditioned coding, the faster GPU decoder, and the fused kernel that removes the need to keep full weights resident. The paper provides a good decomposition for storage (Tables 12 and 13), but the serving experiments only compare the full Q3.6 system with uncompressed weights. A system-level ablation, for example the same decoder, fused kernel and memory policy with an unconditioned or family-only ANS stream, would make clear how each component contributes and help readers adopt the parts relevant to their setting. Related to this, in the vLLM capacity experiments the raw baseline streams weights while still keeping the full qweight resident in GPU memory, whereas Q3.6 does not. A streaming baseline that also releases the resident weight buffers would make the comparison more symmetric; the FlexGen experiments move in this direction, but the vLLM headline results do not include such a control."

"Could the authors report the storage and serving results of the same system with an unconditioned or family-only ANS stream, keeping the decoder, the fused kernel and the memory policy fixed?"

"Similarly, how does Q3.6 compare in the vLLM capacity experiments with an uncompressed streaming baseline that also releases the resident weight buffers? The answers to these two questions would most directly affect my assessment."

Thank you for suggesting these complementary controls. We conducted additional experiments with an uncompressed vLLM baseline (Raw) using the same buffer management policy as Q3.6. In these experiments, neither method retains the full-model qweight buffer. The throughput experiment uses two reusable layer buffers and the same KV-cache allocation for all methods. All methods use native Marlin [a] with matched scheduling and prefetching; the ANS variants share the entropy decoder.

We use Qwen3-8B-AWQ with vLLM [b], FP16 and 512 input / 32 output tokens, following the 1.5 GB/s SSD offloading setting in our original submission. Throughput is measured at batch 16 with a 12 GiB GPU memory budget and includes prefill. We also compare the methods with the same weight-buffer allocation policy, using reusable module buffers at batch 32 within 8 GiB.

MethodTotal deployment file size (MiB)Throughput gain over Raw (B16, 12 GiB)
Unbucketed ANS, shared model5,510.552+5.32%
ANS with Q3.6's buckets, shared model5,512.513+6.92%
Q3.6, conditional models5,489.548+8.56%

Q3.6 reduces total deployment file size by 21.00 MiB relative to the unbucketed shared model and 22.97 MiB relative to the shared model with the same buckets. The serving results above use the common Marlin path.

With both methods releasing the full-model weight buffer and reusing small module buffers, the raw baseline and Q3.6 each support 32 concurrent requests within 8 GiB. These controls distinguish the two benefits: buffer management makes room for higher concurrency, while compression reduces total deployment file size by 5.87% relative to the raw baseline, lowers the weight data transferred, and improves throughput by 8.56% at a fixed batch size of 16. This comparison of memory policies is included in revised Appendix C.5; revised Table 6 reports the latest throughput results.

Q3

"Could the authors provide a clearer comparison with Tan et al. (2026), at least in terms of features and, where feasible, on a shared checkpoint?"

Thank you for suggesting both forms of comparison. We have added a feature comparison with Tan et al. [c] and a comparison on the same checkpoint, focusing on how each method chooses its probability model.

AspectTan et al.Q3.6
Probability modelOne shared model per weight tensorBucket-specific models conditioned on native scale metadata
Compression organizationIndependently coded spatial tiles aligned with GEMMWeight groups routed into buckets using native scale metadata

Q3.6 selects each group’s probability table at decoding time from its native scale, without storing context IDs for individual weights. This gives the decoder a more specific distribution for each group while preserving the original quantized object.

Our comparison on the same checkpoint covers all 252 projections across the 36 transformer layers of Qwen3-8B-AWQ. Our ANS implementation following Tan et al.'s description requires 3.7348 bpw, while Q3.6 requires 3.7226 bpw. These storage results appear in revised Table 3 and Appendix B.10.

To isolate the change in probability model, we also hold FSE [d] and the bucket streams fixed. Conditional tables save 0.0262 bpw over a shared table built from the same projection type across layers. Relative to unbucketed FSE with a shared table and the same pooling scope, the net reduction is 0.0259 bpw.

We also compare serving throughput on Qwen3-8B-AWQ using our GPU ANS implementation following Tan et al. Both methods use native Marlin at batch 16 with a 12 GiB GPU memory budget, the same buffer management policy and identical KV-cache allocation. SSD offloading follows the 1.5 GB/s setting in our original submission. The corresponding results are summarized in the table below.

MethodThroughput gain over Raw
ANS following Tan et al. [c]+6.57%
Q3.6+8.56%

Q4

"I would also appreciate a clearer distinction between what is guaranteed and what is observed: the quantized weights and metadata are recovered bit for bit, while the fused kernel changes the floating-point accumulation order, so identical tokens are an empirical observation on the tested runs."

Yes, your understanding is correct. Bit-exact recovery is guaranteed for the quantized weights and metadata. The fused kernel changes the floating-point accumulation order, so bit-exact logits are not guaranteed; identical generated tokens are an empirical observation on the tested workloads.

We evaluated Qwen3-8B-AWQ using 16 fixed sequences, a 128-token input and a 64-token output on RTX PRO 6000 Blackwell, RTX 5060 Ti and RTX 5080. Each comparison uses the native reference on the same GPU at the same batch size. We verified bit-exact recovery of weights, zero points and scales for all 144 target modules, confirming the recovery guarantee. The fused kernel's changed accumulation order leads to differences in logits, but the generated tokens remain identical to the native reference for all tested sequences on all three GPUs.

We have clarified this distinction in revised Section 4.5 and Appendix C.3. We limit the bit-exact guarantee to recovery of quantized weights and metadata, and report identical generated tokens as an empirical observation under the evaluated models and workloads. We do not claim bit-exact floating-point computation for the fused kernel.

Q5

"Finally, results on larger models, more publicly released quantized checkpoints, and whole-checkpoint compression rates would further support the positioning of Q3.6 as a general post-quantization storage layer; I see these as suggestions rather than requirements."

We appreciate these suggestions and have extended the storage evaluation to three larger public AWQ checkpoints: Qwen3-32B, DeepSeek-R1-Distill-Llama-70B and the Mixtral-8×7B MoE model. All use 4-bit weights and group size 128.

The table compares total checkpoint sizes after compression with Q3.6 and Zstd19, counting all weights, metadata and auxiliary files (revised Table 19, Appendix B.11).

Public AWQ checkpointRaw size (GB)Q3.6 size (GB)Q3.6 size reduction from RawZstd19 size (GB)Zstd19 size reduction from Raw
Qwen3-32B [e]19.34216.74613.42%17.00412.08%
DeepSeek-R1-Distill-Llama-70B [f]39.78535.12211.72%35.60710.50%
Mixtral-8×7B [g]24.65721.74811.80%22.17510.06%

These results further demonstrate Q3.6's storage benefits for larger dense models and MoE models.

W2

"My second concern is the scope and positioning of the serving results. All serving experiments assume that the weights are read from storage on every forward pass under a 1.5 GB/s cap, and the GPU memory budgets are emulated as fractions of a 96 GB card rather than measured on smaller devices. The paper acknowledges that when the weights fit in GPU memory the engines’ native kernels remain preferable, so the serving benefits apply mainly to storage-bound deployments, and this scope could be stated more prominently. In addition, the concurrent work of Tan et al. (2026) also studies lossless compression of quantized LLM weights with ANS decoding fused into GEMM. The paper discusses it only qualitatively; a feature comparison or a controlled comparison on a common checkpoint would clarify the specific contribution of Q3.6 relative to this closest related work."

Following your suggestion, we have revised Section 4.5 to state explicitly that our serving evaluation targets storage-bound deployments, while native kernels remain preferable when weights fit in GPU memory. Revised Tables 6 and 8 identify the GPU hardware, available memory and 1.5 GB/s SSD offloading condition. Appendix C.1 details how the bandwidth and server-GPU memory limits are applied.

Additionally, to evaluate actual consumer-GPU memory limits, we conducted experiments on RTX 5060 Ti (8 GiB) and RTX 5080 (16 GiB). These use Llama-2-7B-AWQ, vLLM, FP16 and 512 input / 32 output tokens, retaining the 1.5 GB/s SSD offloading condition.

At fixed batch sizes of 8/16 with identical KV-cache allocation, Q3.6 improves throughput by 8.23%/8.72%, respectively. The table below reports gains at each method's highest-throughput tested batch:

GPUConcurrent batch, Raw → Q3.6Throughput gain at the respective batches
RTX 5060 Ti11 → 18+76.91%
RTX 508034 → 42+33.32%

The original PRO 6000 sweep models per-model memory limits when multiple models share a single GPU, as in Cloudflare's Unweight deployment, which reduces memory per model to fit more models on each H100 GPU [h].

We also added feature and controlled comparisons with Tan et al. [c]. The table below highlights how the methods organize weights and choose probability models:

AspectTan et al.Q3.6
Probability modelOne shared model per weight tensorBucket-specific models conditioned on native scale metadata
Compression organizationIndependently coded spatial tiles aligned with GEMMWeight groups routed into buckets using native scale metadata

The storage comparison covers all 252 projections of the same Qwen3-8B-AWQ checkpoint. The serving comparison uses native Marlin at batch 16, a 12 GiB memory budget and 1.5 GB/s SSD offloading. The results are summarized below:

MethodOffline storage rate (bpw)Serving throughput gain over Raw
Our ANS implementation following Tan et al. [c]3.7348+6.57%
Q3.63.7226+8.56%

The detailed comparisons appear in the revised Tables 3 and 6 and Appendix B.10.

References

[a] Elias Frantar, Roberto L. Castro, Jiale Chen, et al. MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models. arXiv:2408.11743, 2024.

[b] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP, 2023.

[c] Hongshi Tan, Yao Chen, Gustavo Alonso, et al. Approaching Shannon Bound with Lossless LLM Weight Compression. ISCA, 2026.

[d] Yann Collet. FiniteStateEntropy. Software repository.

[e] Qwen. Qwen3-32B-AWQ. Public AWQ checkpoint.

[f] Valdemardi. DeepSeek-R1-Distill-Llama-70B-AWQ. Public AWQ checkpoint.

[g] TheBloke. Mixtral-8x7B-Instruct-v0.1-AWQ. Public AWQ checkpoint.

[h] Mari Galicer, Ivan Nikulin, and Chris Branch. Unweight: how we compressed an LLM 22% without sacrificing quality. Cloudflare Blog, April 17, 2026.

Reviewer_bLifOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents Q3.6, a lossless compressor for already-quantized LLM weights. The key observation is that the scales stored for quantization groups are statistically related to the distributions of the weight symbols within those groups. Q3.6 uses this metadata to assign groups to buckets, encodes each bucket with a separate ANS probability model, and reconstructs the exact quantized object using the stored metadata and bucket boundaries. The experiments report average storage savings of approximately 10% over the uncompressed target tensors, with additional reductions over zstd of roughly 2% for MLP tensors and 3% for attention tensors. The authors also optimize GPU decoding and integrate decompression with computation for storage-offloaded inference, reporting a 7-10% reduction in time per output token and up to 2.1x higher throughput under constrained GPU memory budgets.

Critical strengths

1.Simple design with a sound recovery mechanism. Bucket assignments are determined by the stored scales and routing rules, and coding within each bucket is reversible, making the argument for exact recovery straightforward. The reported storage costs include native metadata, routing descriptors, and coding tables rather than relying solely on ideal conditional entropy. Reusing existing quantization metadata also avoids the need to train an additional predictor. 2.Ablations support the proposed mechanism. Table 12 separates the gains from the coding backend and metadata conditioning, while Table 13 compares a single shared table, separate tables by tensor family, and scale-based bucketization. Additional controls include random buckets, shuffled routes, and bucketed zstd. Together, these results indicate that the gains cannot be explained solely by switching to ANS or increasing the number of streams. 3.The work addresses inference efficiency as well as storage. GPU decoder optimizations, fused execution, and integration with multiple serving engines extend the contribution beyond offline compression. The appendix also acknowledges slower performance with GPU-resident weights, numerical differences, and the use of simulated memory budgets, which improves the transparency of the evaluation.

Critical weaknesses
  1. Limited novelty and insufficient comparison with the closest work. The paper cites Approaching Shannon Bound with Lossless LLM Weight Compression by Tan et al., which also covers low-bit quantized weights, ANS coding, tile-level decoding, reconstruction in shared memory, and fusion with GEMM. These system components therefore do not, by themselves, establish the main novelty of Q3.6. Its clearest distinction is the use of native quantization metadata to select probability models. The authors do not provide a quantitative comparison, citing differences in checkpoints, quantization conventions, and deployment pipelines. These differences explain why the published numbers cannot be compared directly, but they do not remove the need for a controlled comparison on the same inputs. At a minimum, the evaluation should compare the following on the same quantized object: • Per-layer or per-tensor ANS probability models versus the metadata-conditioned models used by Q3.6. • Bitrate, indexing overhead, and computational cost at the same decoding granularity. Such a comparison is needed to establish whether the contribution is a practically meaningful modeling advance or a modest extension of an existing compressed-weight execution framework.
  2. The largest throughput gains conflate compression with weight-buffer management. The up to 2.1x throughput improvement in Table 5 cannot be attributed solely to transferring approximately 10% fewer weight bytes. Appendix C.5 shows that the raw streaming configuration retains the full Marlin qweight buffer, whereas Q3.6 releases approximately 3.24 GB of that buffer, making more memory available for the KV cache. This demonstrates the value of the overall implementation, but leaves an important question unanswered: how much capacity and throughput advantage would remain if the uncompressed baseline also loaded weights one module at a time into a small reusable buffer? The FlexGen comparison partially addresses this issue, but its raw path uses fp16 buffers and does not isolate the difference under the same low-bit execution path. Uncompressed and unconditional ANS baselines with identical scheduling, prefetching, and buffering strategies are needed.
  3. The incremental gains from metadata conditioning are small in absolute terms. Table 12 shows that conditioning contributes an average of approximately 0.034 bits per weight over unconditional tANS, accounting for about 45% of the total improvement over zstd. This is a real gain, but the headline storage savings include benefits already obtainable from conventional entropy coding, while the maximum throughput improvement also includes benefits from eliminating full-model weight buffers. The end-to-end contribution of metadata conditioning itself has not been directly measured. The paper should compare conditional and unconditional ANS within the same fused execution path. If the latency difference is small, the contribution should be framed more explicitly around storage compression.
  4. The system results have a narrow scope of applicability. The experiments primarily target storage-bound inference under an imposed bandwidth limit. This is a valid setting, and the appendix includes results at 3 and 6 GB/s. However, two limitations remain: • The 8-24 GB budgets are imposed on a 96 GB Blackwell GPU, rather than measured on GPUs with those actual memory capacities. • When weights are GPU-resident, the fused path is substantially slower than native Marlin. In Table 16, the example layer takes approximately 0.84 ms versus 0.17 ms for 16 tokens. The current evidence therefore supports benefits in storage-offloaded deployment. Broader claims about resource-constrained devices or inference acceleration require measurements on the target hardware and a more complete characterization of when the bottleneck shifts from transfer to computation.
  5. The relationship between storage-optimal and deployment configurations is unclear. The main storage results select the bucket count through a sweep, whereas the decoding benchmarks use a fixed count of 16 buckets per MLP family. The appendix also discusses configurations with many more buckets and conditioning on exact scales. In addition, Table 1 and Figure 4 use different accounting conventions for prebuilt decoding tables. Although these distinctions are disclosed, it remains difficult to determine how many tables, how much memory, and how much decoding work are required to deploy the representation that achieves the best storage result. A joint evaluation using the same compressed object and bucket configuration should report file size, runtime table memory, decoding throughput, and end-to-end performance.
Questions for authors

Please check the weaknesses.

4 comments

We thank you for your careful review and recognition of our recovery mechanism and ablations. Your suggestions guided the additional comparisons with prior work and deployment experiments below.

W1

''Limited novelty and insufficient comparison with the closest work. The paper cites Approaching Shannon Bound with Lossless LLM Weight Compression by Tan et al., which also covers low-bit quantized weights, ANS coding, tile-level decoding, reconstruction in shared memory, and fusion with GEMM. These system components therefore do not, by themselves, establish the main novelty of Q3.6. Its clearest distinction is the use of native quantization metadata to select probability models. The authors do not provide a quantitative comparison, citing differences in checkpoints, quantization conventions, and deployment pipelines. These differences explain why the published numbers cannot be compared directly, but they do not remove the need for a controlled comparison on the same inputs. At a minimum, the evaluation should compare the following on the same quantized object: • Per-layer or per-tensor ANS probability models versus the metadata-conditioned models used by Q3.6. • Bitrate, indexing overhead, and computational cost at the same decoding granularity. Such a comparison is needed to establish whether the contribution is a practically meaningful modeling advance or a modest extension of an existing compressed-weight execution framework.''

Thank you for suggesting this comparison.

We implemented shared-table ANS following the description in Tan et al. [a] and evaluated all 252 projections across the 36 transformer layers of Qwen3-8B-AWQ. This implementation retains one probability table per tensor and 64×256 spatial tiles. Our implementation uses FSE [b] and follows the original Q3.6 storage format: it pools corresponding projections across layers, partitions each projection type into scale-based buckets, and encodes each bucket as one complete stream. The FSE controls with shared tables use the same pooling scope. Rates include native metadata and all files needed to describe the representation.

MethodBits per weight
Shared-table ANS following Tan et al. [a]3.7348
FSE, no scale-based buckets, shared table3.7485
FSE, Q3.6 buckets, shared table3.7488
Q3.6, same buckets, separate tables3.7226

The compressed-block index in the ANS implementation following Tan et al. occupies 1.62 MiB, included in the rates above. Revised Table 3 and Appendix B.10 report this storage comparison.

With the bucket streams fixed, separate probability tables save 0.0262 bits per weight (bpw) over the FSE control with a shared table. Q3.6 saves 0.0122 bpw over Tan et al.

Q3.6 recovers bucket assignments and original positions from metadata without storing a separate stream of positions for individual weights. Its probability tables and other coding information total 326.86 KiB, included in the rates above.

For serving, we implemented GPU ANS following Tan et al. and integrated it alongside Q3.6 in vLLM [d], using native Marlin [c] for both. We compared the two methods on Qwen3-8B-AWQ with FP16 activations, a batch size of 16 and 512 input / 32 output tokens per request, under a 12 GiB GPU memory budget and the 1.5 GB/s SSD offloading setting from our original submission. For this additional comparison, we also included an uncompressed vLLM baseline (Raw) with the same buffer management policy. All methods release the full-model weight buffer, reuse two layer buffers and have the same KV-cache allocation.

MethodThroughput gain over Raw
ANS following Tan et al. [a]+6.57%
Q3.6+8.56%

Q3.6 improves serving throughput over the raw baseline by 8.56%, compared with 6.57% for ANS following Tan et al.

W2

"The largest throughput gains conflate compression with weight-buffer management. The up to 2.1x throughput improvement in Table 5 cannot be attributed solely to transferring approximately 10% fewer weight bytes. Appendix C.5 shows that the raw streaming configuration retains the full Marlin qweight buffer, whereas Q3.6 releases approximately 3.24 GB of that buffer, making more memory available for the KV cache. This demonstrates the value of the overall implementation, but leaves an important question unanswered: how much capacity and throughput advantage would remain if the uncompressed baseline also loaded weights one module at a time into a small reusable buffer? The FlexGen comparison partially addresses this issue, but its raw path uses fp16 buffers and does not isolate the difference under the same low-bit execution path. Uncompressed and unconditional ANS baselines with identical scheduling, prefetching, and buffering strategies are needed."

This is a helpful suggestion. We conducted additional experiments with an uncompressed vLLM baseline using the same buffer management policy as Q3.6. Both the raw baseline and Q3.6 release the full-model weight buffer and reuse two layer buffers in the throughput experiment, with the same scheduling, prefetching, KV-cache allocation and native Marlin [c] computation.

We ran Qwen3-8B-AWQ in vLLM [d] with FP16 activations, a batch size of 16 and 512 input / 32 output tokens per request, under a 12 GiB GPU memory budget. SSD offloading follows the 1.5 GB/s setting in our original submission. We also included unconditional ANS controls under the same buffer management policy and execution path. Across three paired runs, the mean throughput gains are:

MethodThroughput gain over the raw baseline
ANS, no scale-based buckets, shared table+5.32%
ANS, same buckets, shared table+6.92%
Q3.6, conditional ANS+8.56%

In a separate capacity test using small reusable module buffers, both the raw baseline and Q3.6 support 32 concurrent requests within 8 GiB on Qwen3-8B-AWQ with the same 512/32-token workload. The throughput results appear in revised Table 6, and Appendix C.5 describes the capacity comparison.

W3

"The incremental gains from metadata conditioning are small in absolute terms. Table 12 shows that conditioning contributes an average of approximately 0.034 bits per weight over unconditional tANS, accounting for about 45% of the total improvement over zstd. This is a real gain, but the headline storage savings include benefits already obtainable from conventional entropy coding, while the maximum throughput improvement also includes benefits from eliminating full-model weight buffers. The end-to-end contribution of metadata conditioning itself has not been directly measured. The paper should compare conditional and unconditional ANS within the same fused execution path. If the latency difference is small, the contribution should be framed more explicitly around storage compression."

Thank you for encouraging us to clarify the contribution of metadata conditioning. Q3.6 consistently improves upon unconditional ANS across all 21 evaluated combinations of models, quantization methods and bit widths (original Table 12, revised Table 15). In three settings, unconditional ANS produces larger files than Zstd; with conditioning, Q3.6 produces smaller files than Zstd in all three.

Following your suggestion, we added an end-to-end comparison on Qwen3-8B-AWQ at batch 16 to quantify the serving benefits of metadata conditioning beyond storage savings. The ANS variants use the same GPU decoder, and all methods, including the raw baseline, use native Marlin with the same buffer management policy and KV-cache allocation. With the bucket streams held fixed, conditioning raises the throughput gain over the raw baseline from 6.92% to 8.56%, an increment of 1.64 percentage points. The results are summarized below:

MethodThroughput gain over Raw
ANS, no scale-based buckets, shared table+5.32%
ANS, same buckets, shared table+6.92%
Q3.6, conditional ANS+8.56%

The offline FSE controls cover all 252 projections of the same checkpoint. Holding FSE and bucket streams fixed, conditioning saves 0.0262 bpw over the shared-table control; the reduction relative to unbucketed shared FSE is 0.0259 bpw. In the GPU deployment representation, Q3.6 reduces total deployment file size by 21.00 MiB relative to the unbucketed shared model and 22.97 MiB relative to the shared model using the same buckets.

The throughput results appear in revised Table 6, the offline storage comparison in Table 3 and Appendix B.10, and deployment file sizes in Appendix C.5. In the revised paper, we have framed Q3.6 as a post-quantization storage layer in Section 4 and scoped the serving results to storage-bound deployments in Section 4.5.

W4

" The system results have a narrow scope of applicability. The experiments primarily target storage-bound inference under an imposed bandwidth limit. This is a valid setting, and the appendix includes results at 3 and 6 GB/s. However, two limitations remain: • The 8-24 GB budgets are imposed on a 96 GB Blackwell GPU, rather than measured on GPUs with those actual memory capacities. • When weights are GPU-resident, the fused path is substantially slower than native Marlin. In Table 16, the example layer takes approximately 0.84 ms versus 0.17 ms for 16 tokens. The current evidence therefore supports benefits in storage-offloaded deployment. Broader claims about resource-constrained devices or inference acceleration require measurements on the target hardware and a more complete characterization of when the bottleneck shifts from transfer to computation."

Following your suggestion, revised Section 4.5 explicitly limits the serving evaluation to storage-bound deployments; native kernels remain preferable when weights fit in GPU memory. We also added measurements on RTX 5060 Ti and RTX 5080 using Llama-2-7B-AWQ, vLLM, FP16 and 512 input / 32 output tokens, retaining the 1.5 GB/s SSD offloading condition.

The table below reports fixed-batch results with optimized GPU staging and cached group maps in Q3.6. The raw baseline and Q3.6 have identical batch sizes and KV-cache allocation within each comparison.

GPU and memory settingBatchTTFT reductionTPOT reductionOutput tokens/s, Raw → Q3.6Throughput gain
RTX 5060 Ti (8 GiB)81.96%7.63%3.475 → 3.760+8.23%
RTX 5080 (16 GiB)169.51%7.97%7.145 → 7.768+8.72%

We repeated each comparison three times, with one warmup and five timed measurements per method in each round.

To more fully characterize when the bottleneck shifts from transfer to computation, as you suggested, we use a theoretical model to estimate the crossover bandwidth. Let SS be the compressed bytes per layer and TexecT_{\mathrm{exec}} the time to consume them and execute entropy decoding and GEMM. Assuming transfer and execution fully overlap, the time to process the layer's weights is Tweight≈max⁡(S/β,Texec)T_{\mathrm{weight}}\approx\max(S/\beta,T_{\mathrm{exec}}), with a crossover at β∗=S/Texec\beta^*=S/T_{\mathrm{exec}}. For the PRO 6000 example in original Table 16 (revised Table 21) at 16 tokens, S=83.1S=83.1 MB takes (83.1/1500)×1000=55.4(83.1/1500)\times1000=55.4 ms to read at 1.5 GB/s. The fused operator with inputs in GPU memory takes 0.84 ms, giving 83.1/0.84≈98.983.1/0.84\approx98.9 GB/s. In deployment, the GPU reads from a weight buffer in CPU memory; the measured time of 5.1 to 7.3 ms gives 83.1/7.3≈11.483.1/7.3\approx11.4 to 83.1/5.1≈16.383.1/5.1\approx16.3 GB/s. Thus, in this example, 1.5 GB/s lies below the estimated crossover; as storage bandwidth approaches this range, GPU access to host memory and fused execution increasingly determine performance.

The fixed-batch consumer-GPU results appear in revised Table 22 and Appendix C.5, and the crossover analysis appears in Appendix C.4.

W5

"The relationship between storage-optimal and deployment configurations is unclear. The main storage results select the bucket count through a sweep, whereas the decoding benchmarks use a fixed count of 16 buckets per MLP family. The appendix also discusses configurations with many more buckets and conditioning on exact scales. In addition, Table 1 and Figure 4 use different accounting conventions for prebuilt decoding tables. Although these distinctions are disclosed, it remains difficult to determine how many tables, how much memory, and how much decoding work are required to deploy the representation that achieves the best storage result. A joint evaluation using the same compressed object and bucket configuration should report file size, runtime table memory, decoding throughput, and end-to-end performance."

Thank you for this suggestion. The two configurations serve different objectives: the storage-optimal configuration supports fair comparisons with baseline methods in terms of final compressed size, while the deployment configuration balances storage savings with execution efficiency on current hardware. Table 1 counts serialized frequency tables; Figure 4 additionally considers their expanded decoding form.

With 16 buckets per MLP family, the deployment configuration achieves lower bitrates and higher decoding throughput than the nvCOMP [e] baselines at all three bit widths (original Table 3, revised Table 4).

Codec / configurationW3: bpwW3: GB/sW4: bpwW4: GB/sW6: bpwW6: GB/s
nvCOMP-LZ43.127766.54.168563.36.177382.8
nvCOMP-GDeflate2.974182.43.834308.35.874139.8
nvCOMP-Deflate2.95122.53.82239.35.85119.3
nvCOMP-Zstd3.026116.23.82186.45.84946.7
Q3.6, deployment configuration2.793769.53.744650.65.786522.9
Q3.6, storage-optimal configuration2.776635.13.737548.85.780401.6

The deployment row uses the original paper's inputs and CUDA-event timing; the storage-optimal row uses newly quantized inputs and full-pass wall timing (revised Table 20, Appendix B.12).

For the 108 MLP projections, the total storage-optimal file sizes are 1,799.0 / 2,421.8 / 3,745.5 MiB at W3/W4/W6, respectively. Expanded decoding tables require at most 512 KiB per call at each width. End-to-end serving results for the deployment configuration appear in our responses to W2 and W3.

References

[a] Hongshi Tan, Yao Chen, Gustavo Alonso, et al. Approaching Shannon Bound with Lossless LLM Weight Compression. ISCA, 2026.

[b] Yann Collet. FiniteStateEntropy. Software repository.

[c] Elias Frantar, Roberto L. Castro, Jiale Chen, et al. MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models. arXiv:2408.11743, 2024.

[d] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP, 2023.

[e] NVIDIA. nvCOMP: GPU Data Compression. Official documentation.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their rebuttals and updated their manuscript/responses. Please carefully read the authors' responses and engage in the discussion to clarify any remaining concerns.

In light of the rebuttal, please let the authors know whether your concerns have been adequately addressed, and update your review comments and scores if appropriate. Active participation is essential for us to reach a well-calibrated decision.

Please note that the author-reviewer discussion phase ends on November 18 at 11:59 PM AoE, after which no further responses can be posted. Thank you for your continued time and effort.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.