- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the limitation of supervising visual autoregressive generation with only Ground-Truth (GT) tokens. It proposes a training framework called Next Token Imagination (NTI). It constructs context-dependent non-GT soft supervision distributions through Context-Regulated Imagination (CRI), and uses Snapshot-based Context Approximation (SCA) to efficiently introduce model-generated context. Without changing the inference architecture, the method significantly improves image generation quality on ImageNet.
Critical strengths
-
The paper is clearly written. The logic is well organized and easy to follow.
-
The research angle is interesting and has practical application value.
Critical weaknesses
-
All experiments are limited to class-conditional generation on ImageNet-1K at 256x256. There is no text-to-image, no other resolutions, and no second dataset. This is a noticeable problem.
-
The paper should include an analysis of the additional computational overhead introduced by the method.
Questions for authors
-
In Section 3.2, the description of SCA uses the term "periodically refreshed frozen snapshot" at line 250. The meaning is somewhat vague. I can understand what the authors intend to do overall, but this expression confuses me a bit.
-
The font in Table 1 is a bit too small. It could be adjusted appropriately while still meeting the space requirements of the paper.
2 comments
Thank you for the author's response. The author's response basically solved my problem, and I will maintain my positive score. Good luck !
Response to Reviewer Xx5j
Thank you for your careful review. We are glad you found the NTI idea interesting. Below, we address the weaknesses and questions raised in your review. We have also uploaded a revised PDF of the manuscript.
Weakness 1
Great comment! Yes, this study focuses on class-conditional image generation, which is itself a fundamental research problem that can yield insights for visual generation more broadly, as exemplified by MaskGIT [1], RQ-Transformer [2], VAR [3], and ADM [4]. Our main experiments use ImageNet-1K at 256×256 and follow the ADM evaluation protocol [4].
Higher-resolution evaluation. Given our compute budget, we limited the 384×384 runs on ImageNet-1K to 50K training steps per model.
We believe the idea behind Next Token Imagination can generalize to other settings. Evaluating it on additional datasets and extending it to text-to-image generation would require substantial additional compute, so we leave these directions for future work.
[1] MaskGIT: Masked Generative Image Transformer, CVPR 2022.
[2] Autoregressive Image Generation Using Residual Quantization, CVPR 2022.
[3] Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction, NeurIPS 2024.
[4] Diffusion Models Beat GANs on Image Synthesis, NeurIPS 2021.
Weakness 2
Actually, we did provide the computational overhead discussion in Section 4.2. To make this more visible, we have revised as below:
We evaluate wall-clock times for 10K training steps (approximately 16 epochs), using a global batch size of 2,048 on 8 NVIDIA A100 GPUs. For scheduled sampling, the training time includes online context construction at each update. For NTI-S, the generator requires only one forward pass per update, while SCA construction runs once per 10K steps.
As shown above, compared with AR-S, NTI-S adds only 0.31 hours (about 19 minutes) per 10K steps, corresponding to approximately 10% additional overhead. Compared with scheduled sampling + CRI, NTI-S saves 2.69 hours (about 44%) while achieving a comparable FID (2.04 vs. 2.03). Overall, NTI-S achieves comparable generation performance with much lower computational overhead.
Question 1
Thank you for pointing this out. We have revised the sentence in Section 3.2 to avoid similar confusion for other readers.
Original wording. “We therefore introduce SCA, which uses a periodically refreshed frozen snapshot pθˉ of the current generator pθ to approximate model-generated contexts for NTI while preserving a single forward pass during training.”
Revised wording. “We therefore introduce SCA, which periodically copies the current generator pθ to obtain a snapshot pθˉ. The snapshot's weights remain fixed between updates and are used to construct approximate model-generated contexts for NTI. The trainable generator then uses these contexts with a single forward pass per update.”
Question 2
We have increased the font size in Table 1 and adjusted its layout so that the table is easier to read.
Thank you again for your valuable suggestions. We have revised the manuscript accordingly. We value the opportunity to discuss our work with the research community and welcome any further questions or comments.