Q3.6: Metadata‑Conditioned Lossless Compression for Quantized LLMs

Low-bit quantization has become a popular way to deploy large language models (LLMs) on resource-constrained devices. However, after a model has already been quantized, residual redundancy in the quantized weights remains largely unexploited, while further quantizing to even lower bit widths can cause unacceptable generation-quality loss. This motivates dedicated lossless compressors for quantized LLMs: they reduce storage after quantization while preserving the exact quantized weights. Existing lossless LLM compressors target floating-point formats such as bfloat16 (BF16), re-quantize before coding, or treat weights as generic byte streams; none conditions on the native metadata of a fixed low-bit object. We propose Q3.6, a lossless compressor for already-quantized LLMs with bit-exact decoding. Q3.6 builds on the observation that stored metadata, such as the scales, is predictive of the symbol distribution of the weight blocks it describes. It derives deterministic entropy buckets from the native metadata and encodes each bucket with a matched table-based entropy model, so blocks with similar metadata share a bucket and a bitstream. Q3.6 stores less than a strong Zstandard (zstd) baseline in every evaluated setting, covering three models, five quantized formats, and three bit widths, and saves about 10% on average relative to the uncompressed target tensors. In storage-offloaded serving at 1.5 GB/s, Q3.6 gives the smallest stream and achieves the fastest decoding among the evaluated GPU codecs; fed to a fused decode-plus-GEMM kernel, the entropy-coded stream lowers the time per output token by 7 to 10%, and the weight buffers the kernel frees yield 16% to 90% more KV cache and up to 2.1× the throughput in vLLM on 8 to 24 GB GPU budgets, with tokens identical to the stock engines. Code is available at https://anonymous.4open.science/r/iclr2027-code-4B00.