acceptodds
Under review as a conference paper at ICLR 2027

Autoregressive Visual Generation Needs a Prologue

Abstract

In this work, we propose Prologue, an approach to bridging the reconstruction–generation gap in autoregressive (AR) image generation. Instead of modifying visual tokens to satisfy both reconstruction and generation, Prologue learns a small set of image-dependent prologue tokens prepended to the visual token sequence. These prologue tokens are trained exclusively with the AR cross-entropy (CE) loss, while visual tokens remain dedicated to reconstruction. This decoupled design lets us optimize generation through the AR model's true distribution while largely preserving reconstruction quality, which we further formalize from an ELBO perspective. On ImageNet 256×256, Prologue-Base reduces gFID from 21.01 to 10.75 without classifier-free guidance while keeping reconstruction almost unchanged; Prologue-Large reaches a competitive rFID of 0.99 and gFID of 1.46 using a standard AR model without auxiliary semantic supervision. Interestingly, driven only by AR gradients, prologue tokens exhibit emergent semantic structure: linear probing on 16 prologue tokens reaches 35.88% Top-1, far above the 23.71% of the first 16 tokens from a standard tokenizer; resampling with fixed prologue tokens preserves a similar high-level semantic layout. Our results suggest a new direction: generation quality can be improved by introducing a separate learned generative representation while keeping visual tokens dedicated to reconstruction.

Primary area:
generative models
Keywords:
Autoregressive Image Generation, Visual Tokenization, Discrete Representation Learning, Vector Quantization, Generative Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_J56nOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Prologue, an image-dependent discrete prefix learned through an auxiliary autoregressive model. Visual tokens support reconstruction, while prologue tokens receive direct AR supervision. A second-stage AR model generates the prefix followed by the visual tokens. The paper evaluates joint-training and frozen-tokenizer variants, together with architectural ablations and representation analyses.

Critical strengths
  1. The central empirical claim has convincing controlled support. In Table 1, Prologue improves guided gFID from 5.02 to 4.11 and no-CFG gFID from 21.01 to 10.75, while rFID changes only from 2.15 to 2.24. More importantly, Prologue-Post achieves 3.88 gFID with the visual tokenizer frozen and reconstruction unchanged. Table A3 also establishes a matched improvement at the larger tokenizer scale.

  2. The dedicated generation-oriented prefix is a useful design contribution, supported by substantial ablations. The paper examines direct AR regularization, simpler prefixes, prefix length, codebook size, visual dropout, and the frozen-tokenizer variant. Static and pooled prefixes explain some, but not all, of the improvement. These controls make the contribution informative beyond its best generation score.

Critical weaknesses
  1. The theoretical interpretation needs a clearer connection to the implemented encoder updates. Appendix C decomposes the full joint cross-entropy, whereas Appendix E states that Prob-STE applies only to prologue input embeddings and CE targets remain unchanged. The encoder therefore receives no direct target-side gradient from the cost of predicting a prefix code. The default visual-input dropout also introduces masked-input prediction, whereas the analysis concerns the unmasked objective. Please distinguish the formal likelihood objective from the surrogate gradient actually used to learn the prefix.

  2. Some mechanistic claims are stronger than the evidence establishes. Attention heatmaps show changes associated with direct AR regularization, but do not by themselves establish a “fundamental conflict” between spatial structure and raster ordering. Similarly, comparing globally informed prologue tokens with only the first 16 spatial tokens establishes useful semantic information, but does not isolate the effect of AR supervision from differences in spatial coverage. These issues affect the explanation more than the demonstrated generation gains.

Questions for authors
  1. What encoder-gradient objective is induced by input-only Prob-STE, and how does it relate to Equation (7), prefix prediction cost, and visual-input dropout?

  2. For Prologue-Post and its identical frozen visual tokenizer, can you report held-out total AR loss per image, including prefix cost, and no-CFG gFID at temperature 1? The separately tuned temperatures in Table A6 are valid sampler choices, but these additional measurements would more directly test the proposed improvement in prior modeling.

  3. How does the semantic probe compare with mean-pooling all baseline visual tokens or using 16 tokens distributed across the image?

I look forward to the rebuttal and would be open to raising my score if it resolves the concerns about the surrogate-gradient interpretation and the semantic-probe controls.

Related-work suggestion for the revision: The reconstruction–generation gap also connects to generation-friendly visual representations. Please briefly discuss HYDRA and REPA-E, and cite them where relevant. HYDRA progressively connects detail and semantic representations, while REPA-E uses representation alignment for joint VAE–diffusion training. This would help position the AR-supervised prologue, particularly its training objective and gradient path, relative to these alternative representation-learning designs.

2 comments

Thank you for your thoughtful review and for highlighting the frozen-tokenizer comparison, where Prologue-Post improves generation while keeping reconstruction fixed. Your questions helped us clarify how the prefix learns and strengthen the controls in our evaluation. We address each point below.

Prefix Gradients and Visual Input Dropout

Review (Q1 + W1): What encoder-gradient objective is induced by input-only Prob-STE, and how does it relate to Equation (7), prefix prediction cost, and visual-input dropout?

Equation (7) gives the unmasked joint CE objective for the prologue encoder, up to terms treated as constant in Appendix C. It includes both the cost of predicting the prefix and the cost of predicting the visual tokens given the prefix. In our default implementation, all targets are hard token IDs. When the AR model predicts a prefix token, the loss does not backpropagate through that token’s label. Once the same token is used as input, losses from predicting later prefix and visual tokens can update its representation through Prob-STE. The prediction targets are unchanged; the distinction is in how the gradient reaches the encoder. On our internal earily experiments, we tested a variant that allows gradients to flow through the prefix labels. Enabling or disabling this gradient path made little difference in these experiments.

With probability 0.5, we replace all visual input tokens with [EOS] while keeping the prediction targets unchanged. Without dropout, each visual token is predicted from the prefix and the preceding visual tokens. With dropout, each visual token is predicted from the prefix and its position, without access to the preceding visual token values. The prefix prediction loss is the same in both cases. The training objective therefore averages the losses under these two input conditions, and Equation (7) describes the unmasked case. We clarify both distinctions in Section 2.2 and Appendix C.

Held-Out AR Loss and Generation at Unit Temperature

Review (Q2): For Prologue-Post and its identical frozen visual tokenizer, can you report held-out total AR loss per image, including prefix cost, and no-CFG gFID at temperature 1? The separately tuned temperatures in Table A6 are valid sampler choices, but these additional measurements would more directly test the proposed improvement in prior modeling.

Following your suggestion, we measured the held-out AR loss averaged over all predicted tokens per image and gFID at T=1 without CFG for Prologue-Post and the 2D baseline. The full-sequence mean CEs for the 2D baseline and Prologue-Post are 7.73 and 7.31 nats/token, respectively, including the prefix and EOS for Prologue-Post.

ModelScopeTokensCE (nats/token)
2D baselineVisual / full sequence2567.73
Prologue-PostVisual2567.54
Prologue-PostPrefix164.10
Prologue-PostFull sequence, including EOS2737.31
Model (T=1, without CFG)gFID ↓IS ↑Precision ↑Recall ↑
2D baseline27.6846.60.600.61
Prologue-Post18.5162.90.660.59
ModelScopeTokensCE (nats/token)
Prologue B–B (λ=3)Prefix164.92
Prologue B–B (λ=3)Visual2567.28
Prologue B–B (λ=3)Full sequence, including EOS2737.12

CE is evaluated on 50,000 ImageNet validation images using teacher forcing.

Linear Probing with Spatially Distributed Tokens

Review (Q3 + W2): How does the semantic probe compare with mean-pooling all baseline visual tokens or using 16 tokens distributed across the image?

For the 2D baseline, sampling every 16th raster position selects one token per row across all 16 rows. Mean-pooling these tokens gives 27.19% Top-1 and 49.13% Top-5, both below the prologue tokens (35.88% and 60.28%).

ModelToken selectionTokensTop-1 accuracy (%) ↑Top-5 accuracy (%) ↑
2D baselineFirst 16 visual tokens1623.7144.17
2D baselineEvery 16th visual token, mean pooled1627.1949.13
ProloguePrefix tokens1635.8860.28

Results are from the checkpoint with the best validation Top-1 accuracy over 90 epochs.

Local Attention under Direct AR Regularization

Review (W2): Attention heatmaps show changes associated with direct AR regularization, but do not by themselves establish a “fundamental conflict” between spatial structure and raster ordering.

We have softened the wording in Sections 4.1–4.2 and Figure 6 to reflect what the attention maps show. With direct AR regularization on 2D visual tokens, attention to the left and upper neighbors becomes less pronounced, and both rFID and gFID worsen. One possible explanation is that gradients from AR regularization interfere with the local structure learned by the 2D tokenizer.

Relation to HYDRA and REPA-E

Review (Related-work suggestion): Please briefly discuss HYDRA and REPA-E, and cite them where relevant.

We now cite and discuss both works in Appendix B. HYDRA connects detail-preserving and semantic features, while REPA-E aligns representations during joint VAE–diffusion training. Prologue learns its generation-oriented prefix through next-token prediction.

Summary of revisions

Following your suggestions, we clarified the gradient path through prefix inputs and how visual dropout changes the objective in Equation (7) (Section 2.2, Appendix C). We now report gFID at temperature 1 without CFG (Table A5), held-out mean CE including the prefix cost (Table A6), and linear probing results using every 16th baseline visual token (Table 5). We also softened the interpretation of the attention maps (Sections 4.1 and 4.2, Figure 6) and added citations to HYDRA and REPA-E (Appendix B). The revisions are marked in blue.

Reviewer_6nmYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Prologue targets the reconstruction-generation gap in discrete AR generation. The authors place 16 image dependent prologue tokens from a separate codebook ahead of 256 standard visual tokens. Both types of tokens are produced by one shared ViT encoder. Visual tokens are trained for reconstruction, while prologue tokens are trained only by the next token prediction loss of a small AR model, through a straight through estimation trick. The tokenizer is then frozen and a full AR model is trained on the [prologue, visual] sequence. The paper motivates the design with an ELBO decomposition of the prior term. Experimental results cover class-conditional ImageNet generation in controlled and system-level settings, ablations of the loss weight and of simpler prefix baselines, a post-hoc variant on a frozen tokenizer, linear probing and attention visualizations.

Critical strengths
  1. The prologue tokens are a novel idea: a short prefix of image-dependent tokens trained only for AR likelihood gives the tokenizer a dedicated place to encode what helps generation, without touching the tokens used for reconstruction.
  2. The method is simple and scalable. Prologue uses a standard next-token AR, with similar sequence length and inference througput, while the gain holds at two tokenizer scales under controlled settings. The L-XL system, with a 685M model, reaches gFID 1.46, competitive with much larger AR systems.
  3. In Fig. 5, sending AR gradients into the visual tokens makes reconstruction collapse even at moderate loss weights, while Prologue's reconstruction stays nearly unchanged across the sweep. The post-hoc variant keeps reconstruction fixed and still improves generation. Training on the AR likelihood without damaging the visual code is a clear advance over earlier direct-AR-loss approaches, which need very small loss weights.
Critical weaknesses
  1. The gain seems not cleanly attributed to the proposed mechanism. A static learnable prefix, identical for every image, already gives more than half of the unguided gFID gain (Table A4). Some of the improvement may come from extra prefix positions rather than image-dependent information.
  2. The large tokenizer swaps the VGG16 perceptual loss for ConvNext, and that change alone improves L-B rFID and gFID more than Prologue itself does. If this backbone is an ImageNet-supervised classifier, it injects class-level features that Inception-based FID is known to reward [1], so the headline system-level numbers should be framed with that in mind.

[1] Kynkaanniemi et al. The role of ImageNet classes in Fréchet inception distance. ICLR, 2023.

Questions for authors
  1. In Table A4, why does CFG hurt class labels prefix generation?
  2. The compact AR that trains the prologue tokens is fixed at 40M, and its size is not ablated. Does the prologue help better or worse when this model is smaller, or closer in size to the Stage 2 AR model?
1 comment

Thank you for your thoughtful review and for recognizing that a short, image-dependent prefix improves generation at both tokenizer scales. We especially appreciate your questions about how much of this gain comes from the extra prefix positions and the choice of perceptual backbone. We address each point below.

Contribution of Static Prefix Positions

Review (W1): The gain seems not cleanly attributed to the proposed mechanism. A static learnable prefix, identical for every image, already gives more than half of the unguided gFID gain (Table A4). Some of the improvement may come from extra prefix positions rather than image-dependent information.

Yes, additional prefix positions can improve generation, especially when they are trained directly with the AR loss. Prologue learns an image-specific prefix shaped by the AR objective. Without CFG, the shared prefix accounts for 53% of the gFID improvement over the 2D baseline (21.01 to 15.59). With the same 16 prefix positions, Prologue reduces gFID by a further 4.84 to 10.75.

With CFG, the shared prefix improves gFID from 5.02 to 4.86, while Prologue improves it from 5.02 to 4.11. A shared prefix cannot provide an image-specific guidance signal. Prologue tokens can capture image-specific signals through the AR loss and use them to guide generation more effectively.

Perceptual Backbone and FID

Review (W2): The large tokenizer swaps the VGG16 perceptual loss for ConvNext, and that change alone improves L-B rFID and gFID more than Prologue itself does. If this backbone is an ImageNet-supervised classifier, it injects class-level features that Inception-based FID is known to reward [1], so the headline system-level numbers should be framed with that in mind.

[1] Kynkaanniemi et al. The role of ImageNet classes in Fréchet inception distance. ICLR, 2023.

You are right that ConvNeXt substantially improves FID. This is precisely why we included the backbone ablation. We hope these transparent ablations help the community understand how much each component contributes. The relative improvement from Prologue holds with both backbones. With VGG16 as well as with ConvNeXt-S, Prologue consistently improves over its corresponding baseline (Table A3). We now discuss in Appendix D how the FID gain from ConvNeXt-S may partly reflect the sensitivity to ImageNet class statistics reported by Kynkäänniemi et al. (2023).

CFG with Class-Conditioned Prefixes

Review (Q1): In Table A4, why does CFG hurt class labels prefix generation?

The class-conditioned prefix already encodes the class label explicitly. Applying CFG may therefore put too much weight on class information. This could explain why gFID worsens from 14.65 without CFG to 17.46 with CFG. We've searched different CFG scales and report the best.

Size of the Stage 1 AR Model

Review (Q2): The compact AR that trains the prologue tokens is fixed at 40M, and its size is not ablated. Does the prologue help better or worse when this model is smaller, or closer in size to the Stage 2 AR model?

You are right. We did not vary the size of the compact AR model because repeating the two-stage training procedure with a larger model would be too expensive, and we were concerned that a smaller model, such as a 10M-parameter model, might lack sufficient capacity.

However, we explored a related setting with Prologue-OneStage, where we train a full-size AR model and the tokenizer jointly in a single stage. This gave worse generation results overall. An additional second-stage fine-tuning run starting from Prologue-OneStage would allow us to examine the larger-AR setting you suggest. Given our current compute resources, we are unable to run this additional training experiment.

Summary of revisions

Following your suggestions, we now quantify how much of the unguided gFID gain the static prefix recovers and discuss why CFG may hurt the class-conditioned prefix (Appendix D.1, Table A4). Tables A2 and A3 give the backbone ablation and the matched baseline comparisons, respectively. We also clarified that all two-stage experiments use a 40M Stage 1 AR model (Appendix D). The revisions are marked in blue.

Reviewer_jSa5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Prologue, which addresses the reconstruction–generation gap in discrete AR image generation. Instead of changing the image tokens to suit the AR model, it adds a small set of image-dependent prologue tokens in front of them. These tokens are trained only with the AR loss, while the image tokens are trained only for reconstruction. At generation time, the AR model first generates the prologue tokens, then the image tokens conditioned on them. On ImageNet 256×256, Prologue clearly improves generation over a standard 2D tokenizer baseline while keeping reconstruction almost unchanged, and its best model reaches gFID 1.46. The paper also analyzes how the AR loss affects the tokenizer, and shows that the prologue tokens pick up semantic structure from the AR objective alone.

Critical strengths
  1. The direction matters. How the AR model should shape the tokenizer is an important open question for discrete AR generation. The paper proposes a simple paradigm: keep the image tokens for reconstruction and give the AR model its own small set of tokens to shape. The idea is clean and gives a useful new way to think about the reconstruction–generation gap.

  2. The analysis is insightful. The paper takes a fairly careful look at how the AR loss affects the tokenizer: the λ sweep, the attention patterns of 2D vs. 1D tokens, and what the prologue tokens end up encoding. These analyses give some useful observations and ideas. For example, applying the AR loss directly can break the local structure of 2D tokens, and semantic structure can emerge in the prefix from the AR objective alone.

Critical weaknesses
  1. Relation to AliTok’s prefix tokens. AliTok already puts 16 image-dependent prefix tokens before the image tokens and has the AR model generate them first, which is basically the same structure as the prologue. The main difference is that AliTok’s prefix is learned through reconstruction, while Prologue’s is learned through the AR loss. The paper doesn’t discuss this, and the ablations don’t compare against an AliTok-style prefix. Without that comparison, I can’t tell whether the gain comes from the AR-shaped prologue itself, or just from having a few global tokens generated first. Since AliTok still performs better at a similar scale, it would also help to show whether the two can be combined, e.g., by applying Prologue-Post to AliTok.

  2. Directly applying the AR loss to tokens. The paper argues that putting the AR loss on visual tokens hurts the codebook, but this is only shown through worse reconstruction. Codebook usage or entropy is never measured, and its own results show that direct AR regularization still helps 1D tokens at small λ. Prior work also suggests the direct route can work well with the right design: LARP trains a tokenizer jointly with an AR prior, and EOSTok trains the tokenizer and AR model end-to-end, with an extra loss that keeps the codebook from collapsing, and reports strong results. EOSTok is not cited or discussed. The paper should explain more clearly why routing the AR loss only to a separate prefix is preferable to these approaches, ideally with a direct comparison.

  3. Evaluation of the controlled comparison. The comparison with the 2D baseline only reports gFID. Without IS or Precision/Recall, it’s hard to tell whether the images actually look better or the distribution is just closer. The sampling setup also feels unusual: the big “w/o CFG” gain uses lower, separately tuned temperatures for the prologue and visual tokens, which already acts a bit like guidance. Prologue also has more sampling knobs to tune than the baseline. Results under a common, standard sampling setting would make the comparison much more convincing.

Questions for authors
  1. How does the prologue differ from AliTok's prefix tokens, which are also 16 image-dependent tokens generated first? Can you compare against an AliTok-style prefix learned through reconstruction, so we can see whether the gain comes from shaping it with the AR loss? Can Prologue-Post be applied to AliTok to show the two are complementary?

  2. How much do codebook usage and token entropy actually drop when the AR loss is applied directly to the image tokens? How does Prologue compare with direct approaches that work well in practice, such as LARP and EOSTok?

  3. Can you report IS and Precision/Recall for the controlled comparison with the 2D baseline? Can you also show results with both models sampled under the same standard setting (e.g., the same temperature for all tokens), to separate the method's gain from sampling tuning?

4 comments

Thank you for your thoughtful review and for recognizing the value of giving the AR model its own prefix while preserving the visual tokens used for reconstruction. We appreciate your comments on the analysis of direct AR regularization and address each point below.

Discussion about AliTok Prefixes

Review (Q1 + W1): How does the prologue differ from AliTok's prefix tokens, which are also 16 image-dependent tokens generated first? Can you compare against an AliTok-style prefix learned through reconstruction, so we can see whether the gain comes from shaping it with the AR loss? Can Prologue-Post be applied to AliTok to show the two are complementary?

We understand this question in three aspects.

Can a better reconstruction design, including the use of prefixes during reconstruction, improve performance? Yes, we believe it can. AliTok uses prefix tokens to predict the first row and imposes an autoregressive constraint during reconstruction. With this design, AliTok achieves strong performance. Our experiments in Appendix D also show that a prefix obtained by pooling existing visual tokens can improve generation without CFG (Table A4). When AR supervision is applied, can other methods also prevent collapse and improve performance? We agree that they can. EOSTok introduces APR to prevent collapse and uses DINOv2 alignment, which contributes substantially to its generation performance. DINOv2 alignment has also been shown to prevent collapse in a range of jointly trained VAEs. Then, what does Prologue do, and what advantages does it offer? Prologue provides a mechanism to improve generation while preserving the reconstruction objective. This allows us to train a prefix for an existing tokenizer, and also allows us to use the standard AR objective to address the gap between the tokenizer and the AR model without relying other semantic alignment or anti-collapse component. More broadly, this offers a different way to think about the problem. Asking one set of tokens to serve both reconstruction and generation may be harder than giving each task its own tokens.

Direct AR Regularization and Codebook Collapse

Review (Q2 + W2): How much do codebook usage and token entropy actually drop when the AR loss is applied directly to the image tokens? How does Prologue compare with direct approaches that work well in practice, such as LARP and EOSTok?

We further compare Prologue with direct AR regularization (LARP-Style Regularization) across visual codebook usage, rFID, and gFID at T=1 without CFG.

ModelλVisual Codebook Usage ↑rFID ↓gFID (T=1, no CFG) ↓
2D baseline—100.00%2.1527.68
2D + AR Reg0.03100.00%2.8045.86
2D + AR Reg0.3100.00%5.2634.40
2D + AR Reg31.39%54.5667.72
1D baseline—98.50%2.1127.83
1D + AR Reg0.0365.44%2.3421.17
1D + AR Reg0.316.55%4.1741.96
1D + AR Reg37.42%18.7641.83
Prologue B–B1100.00%2.1719.41
Prologue B–B3100.00%2.2418.00
Prologue B–B6100.00%2.3317.22
  1. For 1D tokens, even λ=0.03 reduces visual codebook usage from 98.50% to 65.44% and empirical token entropy from 13.90 to 13.36 bits, while rFID rises from 2.11 to 2.34. At λ=0.3 and 3, usage drops further to 16.55% and 7.42%, with entropies of 11.31 and 9.55 bits, respectively. For 2D tokens, usage remains at 100% at λ=0.03 and 0.3. Entropy changes little (13.92 bits at baseline, then 13.89 and 13.86), even as rFID rises from 2.15 to 2.80 and 5.26. At λ=3, usage drops to 1.39%, entropy falls to 7.35 bits, and rFID reaches 54.56. Prologue maintains 100% visual codebook usage at λ=1,3,6, with rFID between 2.17 and 2.33.
  2. At T=1 without CFG, the only direct AR regularization variant that improves gFID over its corresponding baseline is the 1D tokenizer with λ=0.03 (27.83 to 21.17). Its visual CE also falls from 7.89 to 7.16 nats/token (Table 3). Using fewer codes may make next-token prediction easier and help explain this gain. Prologue reaches gFID values of 19.41, 18.00, and 17.22 at λ=1, 3, and 6, respectively, with 100% visual codebook usage at all three weights.

These additional analyses clearly illustrate the central dilemma identified in our paper: directly applying an AR objective to reconstruction tokens introduces a trade-off between generation predictability and reconstruction fidelity. Decoupling generation and reconstruction nearly eliminates this trade-off for jointly trained Prologue. With Prologue-Post, the visual tokenizer is frozen and reconstruction stays fixed. EOSTok addresses codebook collapse through APR and also uses semantic alignment, offering a potentially complementary approach. A controlled comparison with EOSTok would require additional training and ablations, which we cannot complete within the rebuttal period given our available compute resources.

Generation Metrics and Matched Sampling

Review (Q3 + W3): Can you report IS and Precision/Recall for the controlled comparison with the 2D baseline? Can you also show results with both models sampled under the same standard setting (e.g., the same temperature for all tokens), to separate the method's gain from sampling tuning?

We added these metrics to the Baseline Table. With tuned sampling, Prologue improves IS, precision, and recall over the 2D baseline. At T=1 for all tokens without CFG, gFID improves from 27.68 for the 2D baseline to 18.00 for Prologue (Table 3).

Model (tuned sampling settings)gFID ↓IS ↑Precision ↑Recall ↑
2D baseline5.02206.60.820.46
1D baseline6.10201.10.830.42
Prologue4.11210.30.830.48
Prologue-Post3.88209.50.830.49

These experiments support the central idea of Prologue: separating generation-oriented prefixes from reconstruction tokens improves generation while maintaining reconstruction quality and visual codebook usage. Across λ=1, 3, and 6, Prologue consistently improves gFID over the baselines and retains 100% visual codebook usage, avoiding the severe reconstruction degradation and codebook collapse observed under strong direct AR regularization.

Summary of revisions

Following your suggestions, we added IS, precision, and recall to the controlled comparison (Table 1) and report gFID at a common temperature of 1 without CFG (Section 3.2, Tables 3 and A5). We also expanded the analysis of codebook usage and reconstruction under direct AR regularization (Section 4.1, Table 3) and clarified how Prologue relates to AliTok, LARP, and EOSTok (Section 5, Appendix B). The revisions are marked in blue.

Reviewer_jSa5OpenReview ↗

Maintain my weak reject recommendation

Thank you for the detailed response and the new experiments. The codebook usage analysis, the results at a common temperature (gFID 27.68 → 18.00 at T=1 without CFG), and the held-out CE including the prefix cost address my second and third questions well, and they show that the gain is real rather than an artifact of sampling tuning.

However, my main concern, the relation to AliTok’s prefix tokens, is not resolved. AliTok already places 16 image-dependent prefix tokens before the grid tokens and generates them first. The rebuttal discusses the difference in motivation but adds no experiment comparing an AR-shaped prefix with a learned prefix trained through reconstruction. The pooled prefix in Table A4 is not that comparison. Prologue-Post is also not applied to AliTok, even though it is designed for a frozen tokenizer. Without these, it remains unclear what the AR-shaped prologue adds beyond existing prefix designs, especially since AliTok still performs better at a similar scale.

Overall, I find the contribution too incremental relative to prior prefix-based tokenizers, and I maintain my weak reject recommendation.

Thank you for your follow-up and for recognizing that the additional experiments addressed your concerns about direct AR regularization and the sampling setup.

On the comparison with Other Methods

We understand your request for a direct comparison with AliTok. AliTok is an interesting and closely related work. Its first 16 tokens are trained with an auxiliary reconstruction loss, with each token corresponding to a patch in the first row. Together, they encode a compact representation for reconstructing that row. This is the conceptual connection to the pooled prefix in Table A4, which also gives the AR model a compact summary of information learned for reconstruction.

We agree that pooling existing visual tokens differs from learning a prefix through a reconstruction loss, and that Table A4 does not directly compare these two approaches. It does, however, address part of your original question. Can generating a short, image-dependent prefix first improve generation even when the prefix is derived from reconstruction-oriented tokens? In Table A4, the pooled prefix improves generation without CFG, but the gain is smaller than with Prologue. Could a better reconstruction-derived prefix works better? Yes. But that possibility does not change the fact that Prologue works well.

On the broader contributions of Prologue

Viewing Prologue solely as a prefix-token design captures only one part of the paper. Our starting point is the reconstruction–generation dilemma in discrete AR image generation. Through an ELBO decomposition, we identify how this tension arises when the same visual tokens must preserve information for reconstruction while remaining predictable under an AR prior. This motivates our proposal to augment the latent space so that the reconstruction and generation objectives can shape separate representations.

We instantiate this idea by prepending a small set of image-dependent prologue tokens to the visual token sequence. The AR loss shapes the prologue tokens, while the reconstruction objective shapes the visual tokens. Our analysis explains how this separation can improve AR prediction and protect the visual codebook from the collapse observed under strong direct AR regularization. It also gives us Prologue-Post, which can improve generation with an existing, frozen tokenizer while preserving its reconstruction exactly.

Our experiments examine when and how the reconstruction–generation dilemma arises. We study the respective strengths of 1D and 2D tokenizers for reconstruction and generation and show that they respond differently to direct AR regularization. For example, weak regularization improves generation for the 1D tokenizer, while strong regularization reduces visual codebook usage to 1.39% for the 2D tokenizer. Prologue retains 100% visual codebook usage across all three tested weights. At λ = 3, it improves gFID from 27.68 to 18.00 over the 2D baseline at a common temperature of 1 without CFG. Our visualizations and attention analyses further examine the structure of the visual tokens and the semantic information that emerges in the prologue from AR supervision alone. We also report detailed training and inference hyperparameters for reproducibility.

Even within the narrower question of prefix design, the paper already examines what a reconstruction-derived prefix contributes. In Table A4, we construct a prefix by pooling existing visual tokens trained for reconstruction. Generating this pooled prefix before the visual tokens improves generation without CFG. An exhaustive comparison across prefix-based methods is beyond the scope of this paper, especially given that our controlled experiments provide a self-contained test of both the paper’s broader claim and the specific question raised by this narrower perspective.

We therefore regret that the absence of one additional comparison with AliTok appears to carry more weight in the assessment of Prologue as “too incremental” than the paper’s theoretical contribution and the experiments that directly test its central claim.

Comments on the paper

2 comments

We sincerely thank the PCs, ACs, and all reviewers for their time and thoughtful, constructive feedback. We are glad that the reviewers found the problem important and the analysis insightful (jSa5), the method simple and scalable (6nmY), and the controlled evidence for our main claim convincing (J56n).

Following the reviews, we have revised the paper. The revisions are marked in blue. The main additions are:

  • Generation metrics and matched sampling. We added IS, precision, and recall to the controlled comparison. At a common temperature of 1 without CFG, gFID improves from 27.68 for the 2D baseline to 18.00 for Prologue (Sec. 3.2, Tables 1 and 3).
  • Codebook usage under direct AR regularization. We report how visual codebook usage and token entropy change with the AR loss weight, alongside reconstruction quality. Prologue retains 100% visual codebook usage at λ=1, 3, and 6 (Sec. 4.1, Table 3).
  • Contribution of the shared prefix. We now quantify how much of the unguided gFID gain the shared prefix accounts for (53%). With the same 16 prefix positions, Prologue further reduces gFID from 15.59 to 10.75 (Appendix D.1, Table A4).
  • Prefix training and prediction cost. We clarify how gradients reach the prefix and how visual-input dropout changes the objective. Including the prefix and EOS, Prologue-Post achieves a mean CE of 7.31 nats/token, compared with 7.73 for the 2D baseline (Sec. 2.2, Appendices C and D.2, Table A6).
  • Linear probing with distributed tokens. We added a comparison using every 16th visual token in raster order. Mean-pooling these 16 tokens gives 27.19% Top-1 accuracy, compared with 35.88% for the 16 prologue tokens (Sec. 4.4, Table 5).

Thank you again for your commitment to the review process, and we would be happy to hear your feedback.

AuthorsOpenReview ↗

Following Up as the Rebuttal Discussion Period Comes to a Close

Dear Reviewer,

Thank you again for your thoughtful review. As the rebuttal discussion period comes to a close, we wanted to follow up on our response. If you feel these updates address your concerns, we’d be grateful if you would consider updating your score. We’d also be happy to clarify any remaining points.

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.