Can LLMZip Encode Without an LLM?

Predictive compression methods such as LLMZip combine large language models (LLMs) with arithmetic coding to achieve strong text compression, but require costly LLM inference at both the encoder and decoder. We introduce text compression schemes that eliminate LLM inference at the encoder while retaining compression rates close to those of LLMZip. Our approach interleaves text into a matrix and compresses each column using syndrome source coding with error-correcting codes. Using the Slepian–Wolf (SW) theorem for general correlated sources, we establish that the scheme asymptotically matches LLMZip’s compression rate as the block length grows. For finite lengths, we develop a construction using low-density parity-check (LDPC) codes over $\mathrm{GF}(263)$, with a decoder that combines belief propagation with sequential LLM predictions across parallel text streams. On text8, using the byte-level BLT-7B model, our scheme achieves 1.2 and 1.73 bits per character (bpc) at block lengths of 4096 and 256 respectively compared to the 0.972 bpc of LLMZip. We further improve finite-length performance by introducing a weak language model at the encoder. Its predictions enable adaptive rate selection and optimized assignment of tokens to variable nodes in the LDPC Tanner graph, and the assignment requires no signaling. The two methods are complementary and together offer reductions of 0.09 and 0.264 bpc at block lengths of 4096 and 256 respectively, with feedback to account for rare predicted errors in rate to be truly lossless. Using a LoRA fine-tuned BLT-7B model for images as a pixel model at the decoder, we show that the scheme can also compress images losslessly. These results demonstrate that strong LLM-based compression can be achieved with substantially reduced encoder computation.