CHARM: Compressible Hilbert Order Autoregressive Model for Image Generation

In this paper, we present CHARM, a Compressible Hilbert Order AutoRegressive Model that jointly considers spatial locality and inference efficiency in generation-order design. CHARM adopts the Hilbert order, a recursively constructed traversal that keeps contiguous intervals in the generation sequence spatially concentrated. By introducing the notion of \(\beta\)-H\"older locality, we prove that the Hilbert order achieves the optimal exponent \(\beta=1/2\), which in turn guarantees high local-context coverage. In contrast, raster and diagonal zigzag orders admit only the trivial exponent \(\beta=0\), leaving their worst-case local-context coverage without any non-trivial guarantee. We further show that the Hilbert order naturally induces meaningful spatial blocks for compression. To show this, we introduce a power-of-four partition for any sequence whose intervals always correspond to square regions at multiple scales under Hilbert order. Using learned summary tokens to represent each such region with a single KV cache entry, CHARM reduces the attention computation to \(O(n^2\log n)\) and KV cache space to \(O(\log n)\). On class-conditional ImageNet generation, our recursively compressed 1.5B model achieves 1.64 FID at 256 resolution and 1.87 FID at 512 resolution. Across model scales and resolutions, CHARM improves throughput by up to \(19.75\times\) over matched uncompressed baselines, achieving a favorable balance between generation quality and efficiency. Additional experiments show that the same design extends to continuous-latent autoregression.