acceptodds
Under review as a conference paper at ICLR 2027

CHARM: Compressible Hilbert Order Autoregressive Model for Image Generation

Abstract

In this paper, we present CHARM, a Compressible Hilbert Order AutoRegressive Model that jointly considers spatial locality and inference efficiency in generation-order design. CHARM adopts the Hilbert order, a recursively constructed traversal that keeps contiguous intervals in the generation sequence spatially concentrated. By introducing the notion of \(\beta\)-H\"older locality, we prove that the Hilbert order achieves the optimal exponent \(\beta=1/2\), which in turn guarantees high local-context coverage. In contrast, raster and diagonal zigzag orders admit only the trivial exponent \(\beta=0\), leaving their worst-case local-context coverage without any non-trivial guarantee. We further show that the Hilbert order naturally induces meaningful spatial blocks for compression. To show this, we introduce a power-of-four partition for any sequence whose intervals always correspond to square regions at multiple scales under Hilbert order. Using learned summary tokens to represent each such region with a single KV cache entry, CHARM reduces the attention computation to \(O(n^2\log n)\) and KV cache space to \(O(\log n)\). On class-conditional ImageNet generation, our recursively compressed 1.5B model achieves 1.64 FID at 256 resolution and 1.87 FID at 512 resolution. Across model scales and resolutions, CHARM improves throughput by up to \(19.75\times\) over matched uncompressed baselines, achieving a favorable balance between generation quality and efficiency. Additional experiments show that the same design extends to continuous-latent autoregression.

Primary area:
generative models
Keywords:
Generative Model, Autoregressive image generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YCHaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CHARM combines Hilbert-order autoregressive image generation with recursive compression of the KV cache. Completed spatial blocks are represented by learned summary tokens, maintained through an online power-of-four partition. The paper establishes a uniform Hölder-locality property of Hilbert traversal and derives (O(n^2\log n)) attention-score computation and (O(\log n)) cached entries for an (n\times n) latent grid. Experiments evaluate generation quality and saturated throughput on ImageNet at 256 and 512 resolutions, with additional continuous-token experiments.

Critical strengths

The method provides a substantial quality–throughput improvement. In Table 1, recursive compression increases CHARM-H throughput by 5.37× at 256 resolution and 12.74× at 512 resolution, while changing FID from 1.56 to 1.64 and from 1.78 to 1.87, respectively. The benchmark explicitly measures saturated generator-only throughput, and the appendix describes common hardware, precision, guidance usage, and an optimized uncompressed baseline.

The spatial partition and online cache mechanism form a coherent design. The complexity analysis includes the additional summary tokens, and Appendix A.4 explains predictor alignment and the training-time visibility pattern. Results across two model scales and resolutions support the practical usefulness of this approach.

Critical weaknesses

The experiments do not fully isolate the contribution of Hilbert ordering or its interaction with compression. The ImageNet-256 comparison rows in Table 2 match RAR’s scan-order ablation, whose raster result uses randomness annealing. Appendix A.5 does not establish whether the Hilbert experiment matches the relevant annealing, architecture, augmentation, and guidance choices. Separately, Table 2 varies order without compression, whereas Table 3 varies compression only under Hilbert order. Consequently, the specific synergy claimed in lines 471–475 remains insufficiently tested. Applying the same recursive compression to Morton/Z-order would be particularly informative because its aligned power-of-four blocks also correspond to square regions.

The interpretation of the locality analysis is stronger than the established coverage result. Proposition 1 provides a sufficient lower bound; failure to satisfy positive Hölder locality does not imply the absence of all nontrivial coverage guarantees. Moreover, Eq. 10 is zero for integer radii (r\leq3), so it does not establish high coverage at these local scales. Using the traversal in Eq. 9, for example, the coverage at (n=16,r=1,t=34) is (1/9). This does not contradict the theorem, but it illustrates why optimality of the Hölder exponent should be distinguished from superior actual coverage. This distinction matters because coverage is a central explanation offered for the generation improvements.

Questions for authors

Which Table 2 results were newly trained, and which were taken from RAR? Please specify whether randomness annealing, backbone conditioning, data augmentation, and guidance schedules were matched between the Hilbert and comparison models.

How does the same recursive summary-cache mechanism perform under Morton/Z-order, using a matched backbone and training budget? This would help distinguish the benefit of Hilbert continuity from that of hierarchical spatial compression.

Can you provide actual coverage profiles across neighborhood radii and generation progress, and clarify which coverage comparisons follow from the theory? Please also qualify the abstract’s statement about raster and zigzag lacking nontrivial guarantees.

I look forward to the rebuttal and would be open to raising my score if it resolves the concerns about matched controls and the interpretation of the coverage guarantee.

Related-work suggestion for the revision: Section 2 could more clearly distinguish complementary routes to efficient generation. Please briefly discuss AiM (Scalable Autoregressive Image Generation with Mamba) and H2O, and cite them where relevant. AiM uses a Mamba backbone for AR image generation, while H2O studies heavy-hitter and recent-token KV retention in language models. Relating these to CHARM would clarify the differences between backbone efficiency, cache selection, and hierarchical summary compression.

Reviewer_xS4pOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a Hilbert ordering for rasterizing 2D continuous and discrete image tokens for autoregressive visual generation, and demonstrates that the induced ordering can be used to reduce the KV cache memory footprint for efficient sampling.

Critical strengths
  1. The proposed KV cache recursive merging strategy, based on Hilbert rasterization, is novel and results in minimal degradation in sample quality and a significant improvement in sampling throughput.
  2. Hilbert ordering-based generation improves generation fidelity as shown in Table 2.
Critical weaknesses
  1. The novelty of the proposed method is limited to demonstrating that Hilbert order generation can be used to recursively compress KV caches for efficient autoregressive sampling. Hilbert order for flattening 2D image tokens into 1D sequences has been explored in recent works, including STAR[1] and REOrder[2], and KV cache compression has been studied in the LLM literature[3,4]. This manuscript is also missing discussion of the proposed approach with the above works.

  2. The paper lacks comparisons with recent training-time adaptive tokenization strategies [5,6,7] and it is unclear if the proposed sampling-time approach offers better quality-efficiency tradeoff.

[1] Lee, S., Lee, C., Han, J., Song, H., You, T., Lim, H., ... & Yu, Y. (2026, September). Spanning Tree Autoregressive Visual Generation. In European Conference on Computer Vision (pp. 252-270). Cham: Springer Nature Switzerland.

[2] Kutscher, D., Chan, D., Bai, Y., Darrell, T., & Gupta, R. (2026). Reordering patches improves vision models. Advances in Neural Information Processing Systems, 38, 115268-115294.

[3] Zhang, P., Liu, Z., Xiao, S., Shao, N., Ye, Q., & Dou, Z. (2025, May). Long context compression with activation beacon. In International Conference on Learning Representations (Vol. 2025, pp. 101932-101948).

[4] Zhang, Y., Du, Y., Luo, G., Zhong, Y., Zhang, Z., Liu, S., & Ji, R. (2024, July). Cam: Cache merging for memory-efficient llms inference. In Forty-first international conference on machine learning.

[5] Srivastava, D., Mehra, A., Maneriker, P., Sanyal, D., Raj, V., Kamarshi, V., ... & Kimball, J. (2026). Dpar: Dynamic patchification for efficient autoregressive visual generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 23215-23226).

[6] Chen, Zeyuan, et al. "Soft tail-dropping for adaptive visual tokenization." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

[7] Miwa, K., Sasaki, K., Arai, H., Takahashi, T., & Yamaguchi, Y. (2025). One-d-piece: Image tokenizer meets quality-controllable compression. arXiv preprint arXiv:2501.10064.

Questions for authors

My main concern is limited novelty of the proposed approach. The authors should discuss the similarity of the proposed approach with works in visual modeling and LLM domain, and also compare the proposed strategy to existing training-time adaptive methods.

Reviewer_CrUPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CHARM, an AR image generation framework that combines Hilbert-order traversal with learned hierarchical KV cache compression. The Hilbert structure organizes generated tokens into spatially coherent regions, which are recursively summarized to reduce attention computation and cache memory. The paper provides a locality analysis and evaluates the method on class-conditional ImageNet generation across model sizes, resolutions, and token representations.

Critical strengths
  1. The paper addresses two relevant challenges in autoregressive image generation: choosing an appropriate generation order and managing the growing cost of attending to previously generated tokens. The connection between these challenges is well motivated, and the illustrations and theoretical analysis help explain the design.
  2. A coherent method that naturally exploits image structure. Combining hierarchical KV cache compression with the spatial organization induced by Hilbert traversal makes sense.
  3. Reasonably broad experimental coverage within the chosen setting.
Critical weaknesses
  1. The proposed method is fundamentally a learned KV cache compression approach adapted to image generation. Learned memory and context compression have been explored extensively in prior work on LLMs. While the Hilbert-based spatial hierarchy offers a sensible adaptation to image structure, the methodological contribution appears incremental. Clearer positioning relative to prior compression methods would help establish the novelty and significance of this adaptation.
  2. Unclear efficiency reporting. Table 1 reports saturated throughput without specifying the batch size for each result or the batch-size search range. This makes it difficult to separate improvements at a fixed batch size from gains enabled by larger batches.
  3. Table 2 studies scan order without compression, while Table 3 studies compression only under Hilbert order. These experiments do not directly establish that Hilbert traversal is particularly beneficial for compression.
  4. Evaluation is restricted to class-conditional ImageNet generation. Text-to-image experiments would help assess whether compression preserves text alignment and compositional consistency.
Questions for authors
  1. Could you report the batch size for each Table 1 result and provide matched-batch throughput, latency, and peak memory measurements, including batch size = 1?
  2. Could you compare Hilbert, raster, and Z-order using the same compression mechanism and cache budget, and include a simpler compression baseline?
1 comment

Thanks for the rebuttal. I maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.