- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Prologue, an image-dependent discrete prefix learned through an auxiliary autoregressive model. Visual tokens support reconstruction, while prologue tokens receive direct AR supervision. A second-stage AR model generates the prefix followed by the visual tokens. The paper evaluates joint-training and frozen-tokenizer variants, together with architectural ablations and representation analyses.
Critical strengths
-
The central empirical claim has convincing controlled support. In Table 1, Prologue improves guided gFID from 5.02 to 4.11 and no-CFG gFID from 21.01 to 10.75, while rFID changes only from 2.15 to 2.24. More importantly, Prologue-Post achieves 3.88 gFID with the visual tokenizer frozen and reconstruction unchanged. Table A3 also establishes a matched improvement at the larger tokenizer scale.
-
The dedicated generation-oriented prefix is a useful design contribution, supported by substantial ablations. The paper examines direct AR regularization, simpler prefixes, prefix length, codebook size, visual dropout, and the frozen-tokenizer variant. Static and pooled prefixes explain some, but not all, of the improvement. These controls make the contribution informative beyond its best generation score.
Critical weaknesses
-
The theoretical interpretation needs a clearer connection to the implemented encoder updates. Appendix C decomposes the full joint cross-entropy, whereas Appendix E states that Prob-STE applies only to prologue input embeddings and CE targets remain unchanged. The encoder therefore receives no direct target-side gradient from the cost of predicting a prefix code. The default visual-input dropout also introduces masked-input prediction, whereas the analysis concerns the unmasked objective. Please distinguish the formal likelihood objective from the surrogate gradient actually used to learn the prefix.
-
Some mechanistic claims are stronger than the evidence establishes. Attention heatmaps show changes associated with direct AR regularization, but do not by themselves establish a “fundamental conflict” between spatial structure and raster ordering. Similarly, comparing globally informed prologue tokens with only the first 16 spatial tokens establishes useful semantic information, but does not isolate the effect of AR supervision from differences in spatial coverage. These issues affect the explanation more than the demonstrated generation gains.
Questions for authors
-
What encoder-gradient objective is induced by input-only Prob-STE, and how does it relate to Equation (7), prefix prediction cost, and visual-input dropout?
-
For Prologue-Post and its identical frozen visual tokenizer, can you report held-out total AR loss per image, including prefix cost, and no-CFG gFID at temperature 1? The separately tuned temperatures in Table A6 are valid sampler choices, but these additional measurements would more directly test the proposed improvement in prior modeling.
-
How does the semantic probe compare with mean-pooling all baseline visual tokens or using 16 tokens distributed across the image?
I look forward to the rebuttal and would be open to raising my score if it resolves the concerns about the surrogate-gradient interpretation and the semantic-probe controls.
Related-work suggestion for the revision: The reconstruction–generation gap also connects to generation-friendly visual representations. Please briefly discuss HYDRA and REPA-E, and cite them where relevant. HYDRA progressively connects detail and semantic representations, while REPA-E uses representation alignment for joint VAE–diffusion training. This would help position the AR-supervised prologue, particularly its training objective and gradient path, relative to these alternative representation-learning designs.
2 comments
Linear Probing with Spatially Distributed Tokens
Review (Q3 + W2): How does the semantic probe compare with mean-pooling all baseline visual tokens or using 16 tokens distributed across the image?
For the 2D baseline, sampling every 16th raster position selects one token per row across all 16 rows. Mean-pooling these tokens gives 27.19% Top-1 and 49.13% Top-5, both below the prologue tokens (35.88% and 60.28%).
| Model | Token selection | Tokens | Top-1 accuracy (%) ↑ | Top-5 accuracy (%) ↑ |
|---|---|---|---|---|
| 2D baseline | First 16 visual tokens | 16 | 23.71 | 44.17 |
| 2D baseline | Every 16th visual token, mean pooled | 16 | 27.19 | 49.13 |
| Prologue | Prefix tokens | 16 | 35.88 | 60.28 |
Results are from the checkpoint with the best validation Top-1 accuracy over 90 epochs.
Local Attention under Direct AR Regularization
Review (W2): Attention heatmaps show changes associated with direct AR regularization, but do not by themselves establish a “fundamental conflict” between spatial structure and raster ordering.
We have softened the wording in Sections 4.1–4.2 and Figure 6 to reflect what the attention maps show. With direct AR regularization on 2D visual tokens, attention to the left and upper neighbors becomes less pronounced, and both rFID and gFID worsen. One possible explanation is that gradients from AR regularization interfere with the local structure learned by the 2D tokenizer.
Relation to HYDRA and REPA-E
Review (Related-work suggestion): Please briefly discuss HYDRA and REPA-E, and cite them where relevant.
We now cite and discuss both works in Appendix B. HYDRA connects detail-preserving and semantic features, while REPA-E aligns representations during joint VAE–diffusion training. Prologue learns its generation-oriented prefix through next-token prediction.
Summary of revisions
Following your suggestions, we clarified the gradient path through prefix inputs and how visual dropout changes the objective in Equation (7) (Section 2.2, Appendix C). We now report gFID at temperature 1 without CFG (Table A5), held-out mean CE including the prefix cost (Table A6), and linear probing results using every 16th baseline visual token (Table 5). We also softened the interpretation of the attention maps (Sections 4.1 and 4.2, Figure 6) and added citations to HYDRA and REPA-E (Appendix B). The revisions are marked in blue.
Thank you for your thoughtful review and for highlighting the frozen-tokenizer comparison, where Prologue-Post improves generation while keeping reconstruction fixed. Your questions helped us clarify how the prefix learns and strengthen the controls in our evaluation. We address each point below.
Prefix Gradients and Visual Input Dropout
Equation (7) gives the unmasked joint CE objective for the prologue encoder, up to terms treated as constant in Appendix C. It includes both the cost of predicting the prefix and the cost of predicting the visual tokens given the prefix. In our default implementation, all targets are hard token IDs. When the AR model predicts a prefix token, the loss does not backpropagate through that token’s label. Once the same token is used as input, losses from predicting later prefix and visual tokens can update its representation through Prob-STE. The prediction targets are unchanged; the distinction is in how the gradient reaches the encoder. On our internal earily experiments, we tested a variant that allows gradients to flow through the prefix labels. Enabling or disabling this gradient path made little difference in these experiments.
With probability 0.5, we replace all visual input tokens with [EOS] while keeping the prediction targets unchanged. Without dropout, each visual token is predicted from the prefix and the preceding visual tokens. With dropout, each visual token is predicted from the prefix and its position, without access to the preceding visual token values. The prefix prediction loss is the same in both cases. The training objective therefore averages the losses under these two input conditions, and Equation (7) describes the unmasked case. We clarify both distinctions in Section 2.2 and Appendix C.
Held-Out AR Loss and Generation at Unit Temperature
Following your suggestion, we measured the held-out AR loss averaged over all predicted tokens per image and gFID at T=1 without CFG for Prologue-Post and the 2D baseline. The full-sequence mean CEs for the 2D baseline and Prologue-Post are 7.73 and 7.31 nats/token, respectively, including the prefix and EOS for Prologue-Post.
CE is evaluated on 50,000 ImageNet validation images using teacher forcing.