acceptodds
Under review as a conference paper at ICLR 2027

Learning Spatial-Pyramid Visual Representations for Autoregressive Image Generation

Abstract

Autoregressive image generation relies on two coupled components: visual token representation and token distribution modeling. Existing tokenizers either preserve spatial correspondence through single-scale grids or aggregate global information with flat sequences, without explicitly organizing tokens across spatial granularities. Although end-to-end training aligns representation learning with generation objectives, it does not by itself determine how hierarchical visual information should be organized for conditional prediction. We introduce a spatial-pyramid 1D visual representation that organizes global tokens and regional tokens at multiple spatial granularities within a single 1D sequence. The corresponding context model generates global tokens causally and predicts each regional level in parallel, conditioned on all coarser groups, yielding a global-to-local factorization. This factorization retains all latent tokens while reducing sequential decoding depth compared with token-wise prediction. We jointly optimize the tokenizer and context model so that the learned representation adapts to the proposed factorization. On ImageNet at 256256 resolution, our method achieves a gFID of 1.21 with classifier-free guidance using 19 sequential decoding steps.

Primary area:
generative models
Keywords:
Autoregressive image generation, 1D visual tokenization, hierarchical representations, coarse-to-fine generation, end-to-end learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ToynOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper organizes global tokens and regional tokens at three spatial scales into a hierarchical sequence and designs a matching autoregressive factorization: global tokens are generated individually, followed by parallel generation of each regional level conditioned on all coarser levels. The default configuration retains all 240 tokens while reducing sequential sampling to 16+3=19 rounds. The tokenizer and generator are jointly trained through a straight-through estimator, with additional decoder-side DINOv2 representation alignment and distribution-alignment post-training of the generator. On ImageNet 256×256, the authors report gFID of 1.21 for the Large model with CFG. They also provide an extension to 512×512, baselines with matched training conditions, measured speed and memory use, token-group interventions, and ablations.

Critical strengths
  1. Joint design of representation structure and generation dependencies produces clear, measurable benefits.

    Under matched end-to-end training conditions, Table 1 in Section 5.2 reduces the flat baseline's rFID from 4.82 to 2.63 and gFID from 38.06 to 27.10, while increasing throughput from 4.43 to 49.57 images/s. This result holds before auxiliary alignment, showing that the structural contribution does not arise solely from the additional training used in the final system.

  2. Efficiency evidence extends beyond sampling-round counts, with competitive quality.

    Table 5 in Appendix B measures performance on the same RTX 4090 in FP32 at batch size 16. The Large model achieves 17.68 images/s, uses 6.44 GB of memory, and obtains gFID of 1.21. For comparison, VAR-d30 achieves 18.10 images/s, uses 15.46 GB, and obtains gFID of 1.92.

    Together with the 512×512 extension in Table 4, this establishes a meaningful trade-off among quality, speed, and memory, beyond a formal reduction in decoding rounds.

Critical weaknesses
  1. The structural benefit under the complete training recipe has not been directly isolated.

    Table 1 on page 8 reduces gFID from 38.06 to 27.10 under matched end-to-end training, supporting the joint spatial-pyramid and prediction design. However, Table 3 on page 9 shows that decoder alignment and distribution post-training further reduce gFID from 27.10 to 11.44 and 4.60. The main results additionally use a 150-epoch training configuration, and the 1.21 result in Table 2 comes from the larger Large model.

    A flat Vanilla baseline with matched model scale, training budget, both alignment components, and sampling settings is missing. It is therefore unclear how much structural benefit remains under the full recipe, or whether the representations benefit similarly from the auxiliary objectives. Table 1 remains positive evidence, but cannot independently explain the competitiveness of the final main result.

    This control affects the size of the explicit spatial hierarchy's contribution within the complete system, rather than the legitimacy of auxiliary training itself.

  2. The quality cost of parallel regional prediction has not been sufficiently separated from representation benefits.

    Equation (9) generates tokens within a regional level conditionally independently given all coarser levels, and Equation (13) thereby reduces sampling of 240 tokens to 19 sequential rounds. The paper explicitly describes this independence as a modeling choice, and the masks and Algorithm 1 support its implementation.

    However, Table 1 changes both representation organization and prediction factorization. The Parallel/Masked/Causal comparisons in Table 3 primarily change how global tokens are generated; they do not directly compare token-wise and parallel regional generation on the same hierarchical representation.

    The results therefore support the quality and speed benefits of the joint design, but do not quantify the quality cost of omitting within-level dependencies or how much a better hierarchical representation compensates for that cost. This matters for applicability at different quality requirements and for assessing the central 19-round sampling design.

  3. The main generation metrics are closely coupled to the post-training objective, with insufficient independent quality validation.

    Appendix F.2 explicitly updates the generator through frozen Inception-v3 features and a Fréchet-distance objective, while the main generation metrics in Appendix I, gFID and IS, also depend on Inception. Directly optimizing a related feature distribution is not itself an error or evidence of data leakage. Nevertheless, these metrics alone make it difficult to assess whether the advantage of 1.21 over nearby methods also appears in semantic correctness, detail realism, and diversity.

    Appendix I states that final gFID uses reference statistics provided by JiT, but the relationship between the real-image statistics used for post-training and the final reference, including data splits and preprocessing, could be clearer. Image examples and reconstruction metrics also do not replace independent validation of the generated distribution.

    It is therefore necessary to check whether gains before and after post-training persist under evaluation criteria not used in the training objectives, to clarify the practical meaning of the main metric improvement.

Questions for authors
  1. Attribution under the complete training recipe:

    What is the gap between Vanilla and the spatial-pyramid model after matching model scale, training budget, both alignment components, and sampling settings?

    At one representative scale, could you establish whether the structural advantage in Table 1 persists after adding decoder alignment and distribution post-training, or whether the result primarily depends on one representation benefiting more from the auxiliary objectives?

    If corresponding checkpoints or ablations already exist, please report them; retraining at every scale is unnecessary.

  2. Quality–speed trade-off of the prediction factorization:

    On the same hierarchical representation, could you compare token-wise prediction and the current parallel regional prediction in gFID, throughput, and sampling rounds?

    You could fix the tokenizer to isolate the generator factorization, or use matched joint-training controls while specifying any differences in settings.

    Table 3 already compares generation strategies for global tokens. The question here is the quality cost of parallelizing the regional levels themselves and the benefits and limitations of the 19-round design relative to more complete dependency modeling.

  3. Post-training objective and independent evaluation:

    Which data split and preprocessing are used for the real-image statistics in post-training, and how do they relate to the final gFID reference statistics from JiT described in Appendix I?

    Could you check whether gains from before to after post-training persist in a feature space not used in the training objectives, or in a blinded human-preference evaluation, considering whether realism and diversity improve together?

    The priority is validating the incremental effect of post-training, rather than extending the evaluation to a new task.

  4. Comparison with PAR, FAR, and BAR:

    Beyond the existing results for VAR, xAR, RAR, and other methods, could you add experimental comparisons and methodological discussion of PAR (Wang et al., 2025), FAR (H. Yu et al., 2025), and BAR (Tang et al., 2026)? Would like to see additional ImageNet 256×256256\times256 comparisons with all four methods in gFID, parameter count, token count, sampling rounds, and measured generation-plus-decoding latency or throughput. Even if time constraints prevent additional experiments, I would still appreciate a detailed comparative discussion of the methods above.

Reviewer_At7BOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a spatial-pyramid representation for autoregressive image generation. Global tokens and progressively finer regional tokens are organized in one sequence and learned jointly with the generator. At 256 x 256 resolution, the model generates 16 global tokens sequentially and each of three regional levels in parallel, producing 240 tokens in 19 rounds. Decoder feature alignment and distribution-alignment post-training further improve quality. Experiments on class-conditional ImageNet include reconstruction, generation, representation analysis, and a 512 x 512 extension. The Large model achieves guided gFID 1.21 at 256 x 256.

Critical strengths
  1. The representation and prediction schedule are well aligned. Table 1 demonstrates improved reconstruction and generation before auxiliary alignment, with sampling-and-decoding throughput increasing from 4.43 to 49.57 samples/s.
  2. The generation results are strong, and the paper provides measured throughput, a higher-resolution extension, and token-group interventions that help explain the roles of global and regional information.
Critical weaknesses
  1. Interpretation of the final quality gains: Table 1 supports the joint design, while Table 3 shows substantial additional gains from decoder alignment and distribution post-training: gFID decreases from 27.10 to 11.44 and then 4.60 in that ablation setting. The flat baseline is not evaluated with these objectives. Which conclusions about the joint representation and factorization design are supported by the existing comparisons, and which concern the complete training recipe?
  2. Limits of regional conditional independence: Section 4.2 clearly specifies independent regional sampling given the coarser prefix, with interaction between prediction queries. If the prefix still permits mutually exclusive configurations spanning several regions, what limitations does this factorization impose on sampling consistent details? Please use a short conceptual example to distinguish coordination of query features from dependencies between sampled same-level codes.
  3. Interpretation of distribution alignment: Appendix F.2 optimizes a Fréchet distance in Inception-v3 features, and gFID uses the same feature family. What evidence beyond this metric supports broader perceptual improvements from the generator-only post-training stage? Please discuss any existing independent evaluation or saved before/after samples; a small visual comparison cannot establish diversity gains.
  4. Comparison with two-stage sampling: The structure-first motivation is related to GtR [1], which applies a training-free checkerboard-based schedule to a pretrained MAR and completes the remaining tokens with greater parallelism. This paper instead jointly learns global and regional codes with a matching factorization. Could the authors compare how the two approaches establish the context that makes later prediction easier, and explain what the learned hierarchy contributes beyond changing the sampling schedule? Please relate the comparison to the existing representation analyses and token-group interventions.

[1] Generation then Reconstruction: Accelerating Masked Autoregressive Models via Two-Stage Sampling. ICLR 2026.

Questions for authors

Please refer to weaknesses.

Reviewer_nsbeOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a novel tokenization scheme for visual generation that tokenizes images in a coarse-to-fine manner for autoregressive visual generation. The proposed method obtains an impressive FID of 1.21 on the ImageNet-256 benchmark.

Critical strengths
  1. The proposed coarse-to-fine tokenization strategy is novel, and the authors demonstrate competitive reconstruction and generation performance on the ImageNet-256 benchmark.
  2. The paper is overall well written and easy to follow, with clear motivation and visualizations that illustrate the proposed tokenization learning behavior.
Critical weaknesses
  1. The paper is missing several important ablation studies:
  • End-to-end training: Existing seminal works in 1D and 2D tokenization[1,2,3] follow a two-stage strategy of training, i.e VQ-VAE followed by AR backbone, whereas the proposed approach trains both jointly in an e2e manner. The authors need to evaluate both strategies to determine the contribution of e2e training to the improvements in gFID and rFID.
  • Global token generation: While Table 3 compares parallel and causal generation strategies for global tokens, it is unclear whether the proposed bidirectional strategy is optimal for the regional tokens.
  • Scaling behavior: The work is missing generation performance ablation for tokenizer and AR model capacity, which is important to determine the scaling behavior for the proposed architecture. While scaling AR generative backbone might be computationally intensive, scaling the encoder and decoder could give a useful signal on the scaling behavior of the e2e model training.
  1. (Minor) The paper is missing comparisons with recent works in AR Image generation [4,5,6]

[1] Sun, P., Jiang, Y., Chen, S., Zhang, S., Peng, B., Luo, P., & Yuan, Z. (2024). Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525.

[2] Tian, K., Jiang, Y., Yuan, Z., Peng, B., & Wang, L. (2024). Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37, 84839-84865.

[3] Yu, Q., Weber, M., Deng, X., Shen, X., Cremers, D., & Chen, L. C. (2024). An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 37, 128940-128966.

[4] Srivastava, D., Mehra, A., Maneriker, P., Sanyal, D., Raj, V., Kamarshi, V., ... & Kimball, J. (2026). Dpar: Dynamic patchification for efficient autoregressive visual generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 23215-23226).

[5] Li, T., Tian, Y., Li, H., Deng, M., & He, K. (2024). Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37, 56424-56445.

[6] Liu, W., Zhuo, L., Xin, Y., Xia, S., Gao, P., & Yue, X. (2024). Customize your visual autoregressive recipe with set autoregressive modeling. arXiv preprint arXiv:2410.10511.

Questions for authors

Overall, I find the method sound and interesting and my main concerns are missing ablation studies as discussed in weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.