- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GTA builds on top of the REPA-E and REG methods that unlocked end-to-end training of VAE + latent diffusion models through an alignment loss that fine-tunes the VAE encoder for better representation during latent model training. GTA adds to this framework a global representation token in the latent space. The first component is to add, in the VAE encoder, a projection of the global average pooling of internal features (just before output projection) to get a single token, and a loss aligning it with the global token of the REPA-E encoder. The second is to use this global token in the latent SiT (as input and in the flow matching objective). To ensure coherence between patch and global tokens, the method adds a similarity loss between the internal global token representation in SiT internal feature, and the pooled representation of the patches.
Critical strengths
The method directly addresses two issues of the end-to-end training method REG: learning the global token (instead of merely using the one from a representation auto-encoder) and ensuring coherence between the global token and the patch latents. This allows GTA to outperform previous end-to-end training methods.
Critical weaknesses
- The text-to-image experiment setting is too weak, while the generalization of the method to such benchmarks would clearly state if the global token only helps for object-centric generation, as in ImageNet, or if the method generalizes to more complex scenes. Specifically: 1) the choice of SD-VAE for this experiment seems odd, when modern alternatives such a Flux-VAE or even SD3 exists and 2) the experiment doesn’t show that GTA helps on text-to-image tasks, but only that fine-tuning of an underperforming VAE helps: a comparison with REPA-E and REG would be necessary.
- The paper clarity could be greatly improved: while the REPA-E / REG components are mentioned in the introduction, the method section would benefit from introducing those methods as they are parts of the training method, and mentioning them more explicitly in the method section instead of “original end-to-end framework”. Similarly, the method could be clearer by expliciting the full final training objective, as the current one uses “L_base” without clear and explicit definition in the paper.
Questions for authors
could the authors provide clear results on the T2I task or any results showing that GTA generalizes beyond object-centric datasets?
4 comments
Authors' Response to Reviewer JF2c - Part 2/2
W1-3 | GTA-VAE's contribution persists under matched REPA or REG objectives
To further separate the tokenizer's contribution from that of the downstream training objective, we compare E2E-VAE and GTA-VAE with both SD-VAE and VA-VAE initializations. For each paired comparison, we freeze the tokenizers and train SiT-XL with the same model configuration and the same downstream objective, either REPA or REG, for 400k steps (80 epochs) and evaluate without CFG.
Table R2. Frozen-tokenizer comparisons under matched downstream objectives and training budgets (SiT-XL, 400k steps, without CFG).
| Frozen VAE | gFID ↓ | sFID ↓ | IS ↑ | P ↑ | R ↑ |
|---|---|---|---|---|---|
| REPA | |||||
| E2E-SDVAE | 5.10 | 4.60 | 145.45 | 0.74 | 0.64 |
| GTA-SDVAE | 4.61 | 4.47 | 148.36 | 0.76 | 0.62 |
| E2E-VAVAE | 3.46 | 4.17 | 159.80 | 0.77 | 0.63 |
| GTA-VAVAE | 3.13 | 4.42 | 165.15 | 0.77 | 0.62 |
| REG | |||||
| E2E-SDVAE | 2.91 | 4.55 | 188.89 | 0.77 | 0.62 |
| GTA-SDVAE | 2.54 | 4.43 | 196.12 | 0.79 | 0.60 |
| E2E-VAVAE | 2.52 | 4.12 | 186.51 | 0.78 | 0.63 |
| GTA-VAVAE | 2.36 | 4.49 | 190.74 | 0.79 | 0.61 |
GTA-VAE achieves lower gFID and higher IS in all four matched comparisons. In particular, when both tokenizers are evaluated using REG, GTA-VAE retains its advantage with both initializations. These results demonstrate the independent downstream utility of the tokenizer learned by GTA: its benefits persist after controlling for the downstream training objective.
W2 | Clarifying the origins of existing components and the complete training objectives
We thank the reviewer for identifying aspects of the method description that require clarification. Although Appendix F of the original manuscript already provides the complete training objectives and loss weights, we agree that these definitions should also be explicit in the main method section so that readers can directly understand the origins of the components and GTA's additional contributions.
We have revised the manuscript accordingly. Sections 3.3 and 3.4 now explicitly present the complete training objectives and identify the components inherited from prior work.
I would like to thank the authors for the clear answers to the points raised in the comment.
- Soundness: the additional experiments provided clarify that the results extend beyond the initial limited setting, as they are not dependent of a class-centric dataset or a specific VAE
- Clarity: I believe the changes brought to the paper makes it much easier to follow, in particular the inclusion the loss formulation in the main paper. This greatly helps picturing the method.
I changed the rating for 3 to 4, as my concerns are resolved.
Authors' Response to Reviewer JF2c
Thank you for your positive feedback and constructive comments. We sincerely appreciate the time and care you have devoted to reviewing our work.
Authors' Response to Reviewer JF2c - Part 1/2
We thank the reviewer for the careful assessment and constructive comments. In response to your concerns regarding tokenizer choice, text-to-image transfer, contribution attribution, and the method description, we have conducted additional controlled comparisons and revised the manuscript. The main findings and revisions are summarized below:
W1-1 | Choice of SD-VAE and experimental rationale: We use SD-VAE to compare training methods with the same tokenizer architecture and initialization, thereby isolating GTA's contribution. Experiments with VA-VAE initialization also demonstrate gains from GTA, supporting its effectiveness with another tokenizer initialization.
Q1/W1-2 | Text-to-image transfer with end-to-end fine-tuned controls: We added two end-to-end fine-tuned controls, E2E-SDVAE and E2E(w/ GM)-SDVAE. GTA-SDVAE still achieves the highest GenEval (0.33) and DPG-Bench (70.10) scores, demonstrating additional transfer benefits beyond standard end-to-end fine-tuning.
W1-3 | The tokenizer's independent contribution under matched downstream objectives: With either REPA or REG held fixed and the same downstream training budget of 400k steps, GTA-VAE improves both gFID and IS for both tokenizer initializations. These results show that its benefits persist after controlling for the downstream training objective.
W2 | Origins of existing components and complete training objectives: We have revised Sections 3.3 and 3.4 to clarify how the REPA-E framework and REG-related components relate to GTA's additional alignment terms, and to explicitly present the complete training objectives.
W1-1 | Choice of SD-VAE and experimental rationale
We appreciate the reviewer's suggestion regarding tokenizer choice. We use SD-VAE primarily to maintain consistency with the baseline settings of REPA-E and related work, allowing us to compare training methods with the same tokenizer architecture and initialization and thereby isolate GTA's contribution more clearly.
For the text-to-image experiments, we additionally compare tokenizers obtained from the same SD-VAE initialization using different end-to-end training methods, as shown in Table R1. This design focuses the comparison on transfer benefits arising from the training method while controlling for differences in tokenizer architecture and initialization.
Our experiments are also not limited to SD-VAE. With VA-VAE initialization, GTA-VAE likewise improves both gFID and IS when the downstream objective is fixed to either REPA or REG, as reported in Table R2. These results provide evidence of effectiveness with another tokenizer initialization. As part of our future open-source work, we will further explore GTA with other VAEs to assess and broaden its applicability.
Q1/W1-2 | Text-to-image transfer with end-to-end fine-tuned controls
We thank the reviewer for pointing out that comparison with the original SD-VAE alone does not separate GTA's benefits from those of standard end-to-end fine-tuning. To directly address this concern, we added two controls: E2E-SDVAE, obtained from REPA-E, and E2E(w/ GM)-SDVAE, obtained from REPA-E (w/ Global Modeling). Both controls, as well as GTA-SDVAE, have undergone end-to-end fine-tuning.
We freeze each tokenizer and train a text-to-image model using DiT-2.5B and a Gemma-2B text encoder on CC-12M for 80k steps. Evaluation uses 25 sampling steps with a CFG scale of 6. All comparisons use identical downstream training and evaluation settings.
Table R1. Text-to-image transfer performance with different frozen tokenizers at 80k downstream training steps.
GTA-SDVAE achieves the highest scores on both metrics. Relative to the end-to-end fine-tuned E2E-SDVAE, GenEval increases from 0.32 to 0.33, and DPG-Bench increases from 68.67 to 70.10.
Thus, GTA provides additional text-to-image transfer benefits even when compared with end-to-end fine-tuned tokenizers. These experiments extend the evaluation from class-conditional generation on ImageNet to text-conditional generation trained on CC-12M, providing direct evidence that GTA's benefits transfer to this text-to-image setting. We have updated Section 4.3 and the corresponding figure in the revised manuscript to include the full comparison among the four tokenizers.