acceptodds
Under review as a conference paper at ICLR 2027

Visual Autoregressive Generation with Next Token Imagination

Abstract

The space of possible visual content is enormous, and arguably unbounded. Any training set, however large, therefore covers only a tiny fraction of it. Yet generative models are typically trained to fit only the images they are given. In visual autoregressive (AR) generation, for example, each token is predicted from the preceding ground-truth (GT) tokens and supervised solely by the GT next token. Confining learning to observed data in this way limits how much of the visual distribution a model can capture. In this work, we hypothesize that letting a model imagine beyond its training data, by exploring plausible visual realizations it has never observed, can ease this limitation. We use visual AR generation as a proof of concept, since its fixed, discrete token space makes imagination easier to steer and constrain. We propose Next Token Imagination (NTI), a method that allows the model to explore beyond the GT next token, subject to a carefully designed criterion that keeps this imagination bounded by the underlying visual distribution. Extensive experiments on ImageNet show that NTI substantially and consistently improves the fidelity of generated images, approaching state-of-the-art diffusion and AR models while using a smaller model and lower training cost.

Primary area:
generative models
Keywords:
Visual autoregressive generation, Image generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Xx5jOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the limitation of supervising visual autoregressive generation with only Ground-Truth (GT) tokens. It proposes a training framework called Next Token Imagination (NTI). It constructs context-dependent non-GT soft supervision distributions through Context-Regulated Imagination (CRI), and uses Snapshot-based Context Approximation (SCA) to efficiently introduce model-generated context. Without changing the inference architecture, the method significantly improves image generation quality on ImageNet.

Critical strengths
  1. The paper is clearly written. The logic is well organized and easy to follow.

  2. The research angle is interesting and has practical application value.

Critical weaknesses
  1. All experiments are limited to class-conditional generation on ImageNet-1K at 256x256. There is no text-to-image, no other resolutions, and no second dataset. This is a noticeable problem.

  2. The paper should include an analysis of the additional computational overhead introduced by the method.

Questions for authors
  1. In Section 3.2, the description of SCA uses the term "periodically refreshed frozen snapshot" at line 250. The meaning is somewhat vague. I can understand what the authors intend to do overall, but this expression confuses me a bit.

  2. The font in Table 1 is a bit too small. It could be adjusted appropriately while still meeting the space requirements of the paper.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Xx5j

Thank you for your careful review. We are glad you found the NTI idea interesting. Below, we address the weaknesses and questions raised in your review. We have also uploaded a revised PDF of the manuscript.

Weakness 1

All experiments are limited to class-conditional generation on ImageNet-1K at 256x256. There is no text-to-image, no other resolutions, and no second dataset. This is a noticeable problem.

Great comment! Yes, this study focuses on class-conditional image generation, which is itself a fundamental research problem that can yield insights for visual generation more broadly, as exemplified by MaskGIT [1], RQ-Transformer [2], VAR [3], and ADM [4]. Our main experiments use ImageNet-1K at 256×256256\times256 and follow the ADM evaluation protocol [4].

Higher-resolution evaluation. Given our compute budget, we limited the 384×384384\times384 runs on ImageNet-1K to 50K training steps per model.

ModelTraining StepsFID ↓
AR baseline50K10.00
NTI-S50K4.66

We believe the idea behind Next Token Imagination can generalize to other settings. Evaluating it on additional datasets and extending it to text-to-image generation would require substantial additional compute, so we leave these directions for future work.

[1] MaskGIT: Masked Generative Image Transformer, CVPR 2022.

[2] Autoregressive Image Generation Using Residual Quantization, CVPR 2022.

[3] Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction, NeurIPS 2024.

[4] Diffusion Models Beat GANs on Image Synthesis, NeurIPS 2021.

Weakness 2

The paper should include an analysis of the additional computational overhead introduced by the method.

Actually, we did provide the computational overhead discussion in Section 4.2. To make this more visible, we have revised as below:

We evaluate wall-clock times for 10K training steps (approximately 16 epochs), using a global batch size of 2,048 on 8 NVIDIA A100 GPUs. For scheduled sampling, the training time includes online context construction at each update. For NTI-S, the generator requires only one forward pass per update, while SCA construction runs once per 10K steps.

MethodTraining (h)SCA construction (h)Total (h)Time per epoch (min)
AR-S3.10--3.1011.62
Scheduled sampling + CRI6.10--6.1022.90
NTI-S3.110.303.4112.80

As shown above, compared with AR-S, NTI-S adds only 0.31 hours (about 19 minutes) per 10K steps, corresponding to approximately 10% additional overhead. Compared with scheduled sampling + CRI, NTI-S saves 2.69 hours (about 44%) while achieving a comparable FID (2.04 vs. 2.03). Overall, NTI-S achieves comparable generation performance with much lower computational overhead.

Question 1

In Section 3.2, the description of SCA uses the term "periodically refreshed frozen snapshot" at line 250. The meaning is somewhat vague. I can understand what the authors intend to do overall, but this expression confuses me a bit.

Thank you for pointing this out. We have revised the sentence in Section 3.2 to avoid similar confusion for other readers.

Original wording. “We therefore introduce SCA, which uses a periodically refreshed frozen snapshot pθˉp_{\bar{\theta}} of the current generator pθp_\theta to approximate model-generated contexts for NTI while preserving a single forward pass during training.”

Revised wording. “We therefore introduce SCA, which periodically copies the current generator pθp_\theta to obtain a snapshot pθˉp_{\bar{\theta}}. The snapshot's weights remain fixed between updates and are used to construct approximate model-generated contexts for NTI. The trainable generator then uses these contexts with a single forward pass per update.”

Question 2

The font in Table 1 is a bit too small. It could be adjusted appropriately while still meeting the space requirements of the paper.

We have increased the font size in Table 1 and adjusted its layout so that the table is easier to read.

Thank you again for your valuable suggestions. We have revised the manuscript accordingly. We value the opportunity to discuss our work with the research community and welcome any further questions or comments.

Thank you for the author's response. The author's response basically solved my problem, and I will maintain my positive score. Good luck !

Reviewer_AK68OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Next Token Imagination (NTI), which trains visual AR models on modified prefixes with context-dependent soft targets. SCA caches prefixes that mix GT and sampled tokens, while CRI builds the targets from codebook similarity and model predictions. NTI improves ImageNet generation without changing the model or inference procedure.

Critical strengths
  • Adding NTI to the same AR-L backbone reduces FID from 3.02 to 1.67 and increases IS from 256.20 to 312.72, with both models trained for 400 epochs. FID also improves with GigaTok and AliTok, suggesting that the gains are not limited to one tokenizer.
  • With CRI, SCA gets an FID of 2.04 on NTI-S, almost the same as online scheduled sampling at 2.03. The estimated training overhead is about 10%, while online prefix construction nearly doubles the training cost. I find this a useful trade-off, especially since inference stays unchanged.
Critical weaknesses
  • I would like to see SCA tested with fixed soft targets. SCA with GT supervision already gets 2.11 FID, versus 2.04 with CRI, but Table 3 tests fixed soft targets only under GT contexts. Would fixed soft targets work just as well with SCA? This comparison would clarify what CRI adds beyond standard soft supervision.
  • I find the quality gains more convincing than the claim that NTI explores a broader visual space. Appendix C shows varied examples, but FID/IS and the similarity analyses do not directly establish broader coverage. Could the authors measure coverage or within-class diversity under matched guidance settings?
Questions for authors

Please see "Critical Weaknesses" part for reference.

2 comments
AuthorsOpenReview ↗

Response to Reviewer AK68

Thank you for your careful review. We appreciate your recognition of NTI's generality across visual tokenizers, as reflected in the gains with GigaTok and AliTok. Below, we address the weaknesses and questions raised in your review. We have also uploaded a revised PDF of the manuscript.

Weakness 1

I would like to see SCA tested with fixed soft targets. … This comparison would clarify what CRI adds beyond standard soft supervision.

Thank you for raising this question about CRI. We evaluate SCA with context-independent soft targets, and add the result and the plain AR-S baseline to the ablation table in the revised manuscript.

Ablation settingFID ↓
AR-S3.75
GT context + context-independent supervision2.10
SCA + GT supervision2.11
SCA + context-independent supervision2.26
NTI-S (SCA + CRI)2.04
NTI-S (Scheduled sampling + CRI)2.03

Compared with AR-S (FID 3.75), context-independent supervision with GT contexts and SCA with GT supervision improve FID to 2.10 and 2.11, respectively. However, combining SCA with context-independent supervision increases FID to 2.26.

This is consistent with Figure 4(b), where context-adaptive token selection achieves higher DINO scores than fixed selection as more GT tokens in the prefix are replaced. This suggests that context-independent soft targets are insufficient for SCA contexts, highlighting the importance of CRI. Under the same SCA setting, CRI reduces FID from 2.26 to 2.04.

Weakness 2

… Could the authors measure coverage or within-class diversity under matched guidance settings?

Thank you for this suggestion. To further evaluate the diversity and visual space explored by NTI, we report Precision, Recall, Coverage (measuring how well generated samples cover the training data), and Vendi Score [1] (measuring the sample diversity based on pairwise similarities). Further, we introduce the Novelty Rate to measure the feature-space novelty of generated images relative to the training set ImageNet. Specifically, we measure the DINOv2 feature distance (1 - cosine similarity) between each generated image and its nearest neighbor from the same class in the training set. For each class, P90 is the 90th percentile of the distances from real validation images to their nearest same-class training images. We define Novelty Rate as the proportion of generated images whose distances exceed their class-specific P90 thresholds.

AR is our own implementation here. We evaluated AR and NTI under matched guidance settings within each model size. The results are summarized below.

ModelFID ↓Precision ↑Recall ↑Coverage ↑Vendi ↑Novelty ↑
AR-S (Ours)3.750.830.460.902.0324.58%
NTI-S2.040.810.570.932.2930.30%
AR-L (Ours)3.140.840.490.921.9916.56%
NTI-L1.670.810.600.922.2921.98%

Precision and Recall: Across model size, NTI improves substantially the recall metric at a small cost of precision, while driving down the FID score significantly. This verifies the efficacy of NTI in improving both distribution coverage and generation quality.

Coverage. NTI maintains or improves Coverage. This metric measures how well generated samples cover the ImageNet training set, so it offers limited insight into the novelty of those samples. We therefore additionally report Vendi Score and Novelty Rate.

Vendi Score and Novelty: NTI improves both metrics clearly, indicating the capability of generating more diverse imagery beyond the training data.

[1] The Vendi Score: A Diversity Evaluation Metric for Machine Learning, TMLR 2023.

Thank you again for your valuable suggestions. We have revised the manuscript accordingly. We value the opportunity to discuss our work with the research community and welcome any further questions or comments.

Thanks for the authors' response. After carefully reviewing the response, I think the SCA ablation with fixed soft targets and the diversity evaluation under matched guidance settings address my main concerns. Thus, I will keep my score of 3.

Reviewer_SffPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Next Token Imagination (NTI), a training method for visual autoregressive (AR) models. The idea is that the ground-truth (GT) next token is not the only plausible target, especially once the context deviates from the GT sequence.

NTI has two parts:

  • Snapshot-based Context Approximation (SCA) replaces some GT tokens with tokens sampled from a periodically refreshed snapshot of the model.
  • Context-Regulated Imagination (CRI) trains on these perturbed contexts with soft targets. These targets move part of the probability mass from the GT token to non-GT tokens, weighted by codebook similarity and by the snapshot's predictions.

Inference is unchanged. With a MaskGIT-VQGAN tokenizer, NTI-L (461M) reaches an FID of 1.67, compared with 3.02 for the AR-L baseline. NTI also improves the GigaTok (3.94 → 3.66) and AliTok (1.61 → 1.51) generators.

Critical strengths
  1. Simple and effective. NTI only changes the training targets and inputs, leaves inference unchanged, and adds an estimated ~10% training cost. A standard raster-order AR-L model with the 1024-entry VQGAN tokenizer reaches FID 1.67, on par with RAR-L (1.70) at the same size, and the gains carry over to two other tokenizers. This, together with how easy the method is to adopt, is the main reason for my positive rating.

  2. An interesting perspective and a practical design. Treating off-GT contexts as possibly valid states, rather than only as errors to be corrected, is a nice angle on exposure bias. SCA makes this cheap: it nearly matches online scheduled sampling with CRI (2.04 vs. 2.03 FID) while keeping a single forward pass per training step. Code is also provided.

Critical weaknesses
  1. It is unclear how much of the gain comes from NTI, and from CRI in particular.

    The guidance scale and schedule parameters of the AR-L† baseline are not reported, while NTI uses power-cosine guidance with model-specific parameters (App. A.3). The FID of AR models is very sensitive to the guidance schedule. For reference, RAR's raster-order AR-L ablation reports 3.08 FID, which is close to the 3.02 here; that ablation uses a simple linear guidance schedule and a simpler setup overall.

    In addition, the ablation in Table 3 (NTI-S) shows only a small benefit of CRI:

    • SCA contexts with GT targets give 2.11;
    • codebook-based soft targets without SCA give 2.10;
    • full NTI gives 2.04.

    The plain AR-S baseline is missing. No ablation combines SCA contexts with context-independent soft targets, which would directly test the "context-regulated" part.

    Since CRI is the main technical contribution, this is my main concern.

  2. The claim that NTI expands the visual space lacks direct quantitative support.

    The generation results report only FID and IS, all under strong guidance; there is no precision/recall and no unguided FID. The large IS increase (256.20 → 312.72) could also mean sharper but less diverse samples, and Fig. 7 is only qualitative.

    Moreover, by Eq. 9, CRI moves the most probability mass away from the GT token when the snapshot is confident, disagrees with the GT, and is consistent across the two contexts. This is closer to self-distillation and could reinforce the model's own errors.

    This matters because the paper explains its gains through this "imagination" view. Overall, I still lean towards acceptance given the strong results, but I would like these points to be clarified.

Questions for authors

Questions that could change my assessment

  1. Table 1's † note says that AR-L† was trained with your implementation. Was it trained for this paper or taken from prior work, and which guidance schedule and scale were used? Could you also report AR-L† under the same guidance protocol as NTI-L, and the unguided FID of both models?
  2. Could you add the plain AR-S baseline, and an ablation with SCA contexts and context-independent soft targets?
  3. Could you report precision/recall (or another diversity metric) for AR-L† and NTI-L?
  4. NTI is turned off for the last 20% of training (after epoch 320). What happens if it is kept until the end? This would help separate a regularization effect from an actual change in the learned distribution.

Minor clarifications and suggestions

  • The ± in Table 1 is not defined. Is it the standard deviation over the three sampling seeds mentioned in App. A.3, i.e., from a single training run?
  • In Table 3, λ = 0.5 gives 3.63 and τ = 0.05 gives 3.55 at 80 epochs, although both are stated as defaults. Which settings were used in each row?
  • Were the GigaTok and AliTok baselines retrained? AliTok itself reports 1.61 at 400 epochs, the same number as here.
  • The 3.07 FID listed for LlamaGen-L corresponds to its 384×384 model; the 256×256 model reports 3.80.
4 comments
AuthorsOpenReview ↗

Response to Reviewer SffP (1/3)

Thank you for your careful review. We are happy that you appreciated our perspective on exposure bias and the idea that plausible off-GT contexts, often treated as errors, can give the model an opportunity to learn better continuations. Below, we address the weaknesses and questions raised in your review. We have also uploaded a revised PDF of the manuscript.

Weakness 1 & Question 2

W1: It is unclear how much of the gain comes from NTI, and from CRI in particular. …

Q2: Could you add the plain AR-S baseline, and an ablation with SCA contexts and context-independent soft targets?

Thank you for raising this concern about CRI. We address the guidance settings for AR-L† and NTI-L in our response to Question 1. Here we focus on the benefits of CRI.

Ablation settingFID ↓
AR-S3.75
GT context + context-independent supervision2.10
SCA + GT supervision2.11
SCA + context-independent supervision2.26
NTI-S (SCA + CRI)2.04
NTI-S (Scheduled sampling + CRI)2.03

We added the plain AR-S baseline and an additional ablation combining SCA with context-independent supervision. Compared with AR-S (FID 3.75), context-independent supervision with GT contexts and SCA with GT supervision improve FID to 2.10 and 2.11, respectively. However, combining SCA with context-independent supervision increases FID to 2.26.

This is consistent with Figure 4(b), where context-adaptive token selection achieves higher DINO scores than fixed selection as more GT tokens in the prefix are replaced. This suggests that context-independent soft targets are insufficient for SCA contexts, highlighting the importance of CRI. Under the same SCA setting, CRI reduces FID from 2.26 to 2.04.

Weakness 2

… This is closer to self-distillation and could reinforce the model's own errors. …

We report the evaluation results in our responses to Question 1 and Question 3, respectively. Here, we address the concern that CRI may reinforce its own errors.

Conceptually, our method can be viewed as a form of self-distillation, which is an insightful connection. Indeed, we acknowledge that the risk of error reinforcement exists in self-distillation. To mitigate this risk, we adopt two complementary mechanisms. First, the standard AR objective on GT prefixes remains active throughout training, providing direct supervision from GT next tokens rather than relying solely on the model's own predictions. Second, Eq. (9) controls the total probability assigned to non-GT tokens, while Eq. (10) retains probability mass on the GT token. When predictions under GT and model-generated contexts become inconsistent, the imagination weight decreases, while the supervision weight of the GT token increases. Thus, GT supervision acts as an adaptive anchor: it allows controlled exploration while avoiding excessive deviations.

Question 1

Table 1's †\dagger note says that AR-L$\dagger$ was trained with your implementation. Was it trained for this paper or taken from prior work, and which guidance schedule and scale were used? Could you also report AR-L$\dagger$ under the same guidance protocol as NTI-L, and the unguided FID of both models?

Thank you for carefully checking this detail, actually this is a typo. After double check, we found that this AR-L$\dagger$ result (FID 3.02) was taken from reAR [1]. We have corrected this. Also we reported the result of AR-L in our implementation using the same setting (power-cosine CFG with guidance scale 1414 and power 1.751.75) as NTI-L:

ModelFID ↓ (w/ CFG)FID ↓ (w/o CFG)
AR-L (reAR [1])3.02--
AR-L (our implementation)3.1416.61
NTI-L1.676.51

[1] reAR: Rethinking Visual Autoregressive Models via Token-wise Consistency Regularization, ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer SffP (2/3)

Question 3

Could you report precision/recall (or another diversity metric) for AR-L$\dagger$ and NTI-L?

Great suggestion. To further evaluate the diversity and visual space explored by NTI, we report Precision, Recall, Coverage (measuring how well generated samples cover the training data), and Vendi Score [1] (measuring the sample diversity based on pairwise similarities). Further, we introduce the Novelty Rate to measure the feature-space novelty of generated images relative to the training set ImageNet. Specifically, we measure the DINOv2 feature distance (1 - cosine similarity) between each generated image and its nearest neighbor from the same class in the training set. For each class, P90 is the 90th percentile of the distances from real validation images to their nearest same-class training images. We define Novelty Rate as the proportion of generated images whose distances exceed their class-specific P90 thresholds. AR is our own implementation here. The results are summarized below.

ModelFID ↓Precision ↑Recall ↑Coverage ↑Vendi ↑Novelty ↑
AR-S3.750.830.460.902.0324.58%
NTI-S2.040.810.570.932.2930.30%
AR-L3.140.840.490.921.9916.56%
NTI-L1.670.810.600.922.2921.98%

Precision and Recall: Across model size, NTI improves substantially the recall metric at a small cost of precision, while driving down the FID score significantly. This verifies the efficacy of NTI in improving both distribution coverage and generation quality.

Coverage. NTI maintains or improves Coverage. This metric measures how well generated samples cover the ImageNet training set, so it offers limited insight into the novelty of those samples. We therefore additionally report Vendi Score and Novelty Rate.

Vendi Score and Novelty: NTI improves both metrics clearly, indicating the capability of generating more diverse imagery beyond the training data.

[1] The Vendi Score: A Diversity Evaluation Metric for Machine Learning, TMLR 2023.

Question 4

NTI is turned off for the last 20% of training (after epoch 320). What happens if it is kept until the end? This would help separate a regularization effect from an actual change in the learned distribution.

As suggested, we additionally trained NTI-S with NTI enabled during the final 20% of training. The resulting FID is 2.03, compared with 2.04 when NTI is turned off during this stage. Both settings achieve comparable generation quality.

We do not consider regularization and the learned distribution to be decoupled. Regularization is part of the training process, while changes in the learned distribution are an outcome of this process. In this sense, NTI can be viewed as a distinct form of regularization through expanded next-token supervision. As reported in our response to Question 3, the higher Recall, Vendi Score, and Novelty Rate under matched guidance provide evidence of its impact on the learned distribution.

AuthorsOpenReview ↗

Response to Reviewer SffP (3/3)

Minor clarification 1

The ±\pm in Table 1 is not defined. Is it the standard deviation over the three sampling seeds mentioned in App. A.3, i.e., from a single training run?

Yes. For each NTI model, we evaluated the final checkpoint from a single training run using three sampling seeds (42, 142, and 242). Table 1 reports the mean ±\pm sample standard deviation of FID and IS across these evaluations. We have clarified this in the revised Table 1 caption.

Minor clarification 2

In Table 3, λ\lambda = 0.5 gives 3.63 and τ\tau = 0.05 gives 3.55 at 80 epochs, although both are stated as defaults. Which settings were used in each row?

Thank you for pointing this out. We rechecked the original training configurations and evaluation logs, and found a typo in the FID values reported for the temperature ablation in Table 5. We have corrected it in the revised manuscript.

The default setting is (λ,τ)=(0.5,0.05)(\lambda,\tau)=(0.5,0.05), which gives an FID of 3.63. In the loss-weight ablation, we fix τ=0.05\tau=0.05 and set λ=1.0\lambda=1.0, obtaining an FID of 3.61. In the temperature ablation, we fix λ=0.5\lambda=0.5 and set τ=0.10\tau=0.10, obtaining an FID of 3.55. These results also suggest that further tuning of λ\lambda and τ\tau may lead to additional improvements.

All evaluations use 50K-step checkpoints (approximately 80 epochs), a CFG scale of 22, a cosine guidance schedule with power 2.75, and sampling seed 42. We generate 50,000 images for each evaluation. We have also clarified the experimental configurations in the Appendix of the revised manuscript.

Minor clarification 3

Were the GigaTok and AliTok baselines retrained? AliTok itself reports 1.61 at 400 epochs, the same number as here.

We retrained GigaTok but not AliTok. For a fair comparison, we retrained the GigaTok baseline from scratch using the same tokenizer, batch size, and training budget as GigaTok + NTI. However, it is too expensive to retrain AliTok, so we used its original result as reported, and its training setting to train AliTok + NTI.

Minor clarification 4

The 3.07 FID listed for LlamaGen-L corresponds to its 384$\times$384 model; the 256$\times$256 model reports 3.80.

Thank you for catching this oversight! We have corrected the LlamaGen-L entry in Table 1 to report the FID of 3.80 for the 256×256256\times256 model.

Thank you again for your valuable suggestions. We have revised the manuscript accordingly. We value the opportunity to discuss our work with the research community and welcome any further questions or comments.

Reviewer_SffPOpenReview ↗

Post-rebuttal comment

Thank you for the new results. With NTI-L's guidance settings, your own AR-L reaches 3.14 FID versus 1.67 for NTI-L (16.61 vs. 6.51 unguided), and recall rises from 0.49 to 0.60 at a small precision cost, which resolves my guidance and diversity concerns. The SCA + context-independent run (2.26) also shows that CRI is needed once SCA is used. However, with AR-S at 3.75, codebook soft targets alone (2.10) or SCA with GT targets alone (2.11) come within 0.06-0.07 FID of full NTI (2.04), about twice the ±0.03 sampling spread of a single training run, so the combination adds little over either ingredient. Please report the controlled 3.14 baseline instead of the quoted 3.02 in the final version. I keep my rating at 3 (weak acceptance).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

Dear Reviewers and Area Chair,

We thank all reviewers for their time and helpful comments. We have revised the manuscript, with changes highlighted in blue, and provided additional experiments and clarifications in our responses. The main updates are summarized below, with corresponding reviewer comments indicated for reference.

  • Additional ablation studies [SffP W1, Q2; AK68 W1]. Added the plain AR-S baseline and SCA with context-independent supervision to Table 5. We also expanded the discussion of CRI in Section 4.3 and explained the connection to Figure 4(b) in our responses.

  • Diversity and novelty evaluation [SffP Q3; AK68 W2]. Added Precision and Recall to Table 1, Vendi Score and Novelty Rate to Table 3, with detailed Vendi and Novelty evaluation protocols in Appendix A.4. Our responses additionally report Coverage and AR–NTI comparisons under matched guidance settings.

  • Baseline and guidance clarifications [SffP Q1, Minor 3]. Corrected the attribution of the AR-L† result in Table 1 and added matched-guidance and unguided comparisons between our AR-L implementation and NTI-L in Appendix A.5. We also clarified the GigaTok/AliTok baseline provenance in our responses.

  • Self-distillation and learned distribution [SffP W2, Q4]. In our responses, we clarified the role of GT supervision in mitigating error reinforcement. We also reported an additional experiment keeping NTI enabled during the final 20% of training and discussed its implications for the learned distribution.

  • Higher-resolution evaluation [Xx5j W1]. Added the 384×384 evaluation under a matched 50K-step training budget to Section 4.2. Additional datasets and text-to-image generation are discussed as future directions.

  • Computational overhead [Xx5j W2]. Added a training-cost breakdown to Section 4.2 and Table 4. It distinguishes generator training from SCA construction and compares the AR baseline with two implementations of NTI: online context construction and SCA.

  • Experimental reporting and corrections [SffP Minor 1, 2, 4]. Clarified sampling statistics, corrected the temperature-ablation and LlamaGen-L entries, and documented the loss-weight and temperature ablation settings in Appendix A.3.

  • Manuscript clarity [Xx5j Q1, Q2]. Revised the SCA snapshot description in Section 3.2 and improved the readability of Table 1.

Please see our point-by-point responses below for further details. And thank you again for your time and consideration.

Sincerely, The Authors

AuthorsOpenReview ↗

Summary of Our Work and the Discussion

Dear Area Chair and Reviewers,

We sincerely thank all three reviewers for their feedback and follow up during discussion. At the close of the discussion, the reviewer ratings remained unchanged at 3/3/3. We briefly summarize our work and the main outcomes below.

Summary of Our Work

Standard visual autoregressive (AR) training uses ground-truth (GT) prefixes and supervises only the GT next token. Departures from GT sequences are often treated as exposure-bias errors that the model should learn to recover from. We instead recognize that plausible model-generated prefixes offer opportunities to explore alternative visual continuations. Based on this insight, we introduce Next Token Imagination (NTI), using Snapshot-based Context Approximation (SCA) to construct model-generated contexts and Context-Regulated Imagination (CRI) to guide next-token supervision.

Reviewers' Positive Feedback

  • Novelty and motivation. The reviewers appreciated the new perspective on exposure bias, particularly the idea of exploring plausible off-GT contexts rather than treating all deviations as errors.
  • Practicality and generality. The reviewers recognized NTI's compatibility with standard AR inference and its effectiveness across different tokenizers.

Main Outcomes of the Discussion

  • Generation quality and diversity. NTI-L improves FID from 3.14 to 1.67 over our AR-L baseline under matched guidance. Improvements in Recall, Vendi Score, and Novelty Rate further support greater sample diversity and an expanded visual space.

  • Contribution of CRI and GT supervision. Additional ablations show that CRI reduces FID from 2.26 to 2.04 under the same SCA setting. We also clarified how GT supervision acts as an adaptive anchor to mitigate error reinforcement.

  • Generality. NTI improves generation across GigaTok and AliTok. At 384×384384\times384, NTI-S achieves an FID of 4.66, compared with 10.00 for the AR baseline.

  • Computational efficiency. SCA achieves comparable FID to online context construction (2.04 vs. 2.03), while reducing training time by approximately 44%.

We sincerely appreciate the reviewers' feedback and the discussion. We hope NTI encourages the community to revisit the conventional view of exposure bias, considering plausible off-GT contexts not merely as errors to be corrected, but also as opportunities for learning.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.