acceptodds
Under review as a conference paper at ICLR 2027

GTA: Global Token Alignment as a Bridge from Autoencoding to Denoising

Abstract

Latent generative models are jointly limited by two factors: the quality of the latent space and the modeling capacity of the denoiser. Although recent representation-enhanced methods improve tokenizers or denoisers, the two stages still lack a shared semantic bridge. Global semantics formed during autoencoding may not be effectively inherited by the denoiser. During denoising, the evolving global semantics may also drift from the spatial latent representations. We introduce GTA, a Global Token Alignment framework that bridges autoencoding and denoising through a shared global representation. GTA encourages the autoencoder to internalize image-level semantics during latent formation and maintains consistency with spatial latents throughout denoising. Experiments on ImageNet 256×256 show that GTA accelerates training and improves generation quality. With SD-VAE initialization, GTA achieves a gFID of 6.42 after 20 epochs. This outperforms REPA-E, which reaches 7.17 after 40 epochs. With VA-VAE initialization, GTA further achieves a gFID of 1.87 after 200 epochs. GTA also produces a more generation-friendly tokenizer. When GTA-VAE is frozen and new generative models are trained from scratch, it consistently outperforms the corresponding various types of VAEs under SiT, REPA, and REG.

Primary area:
generative models
Keywords:
Diffusion, Image Generation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_JF2cOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GTA builds on top of the REPA-E and REG methods that unlocked end-to-end training of VAE + latent diffusion models through an alignment loss that fine-tunes the VAE encoder for better representation during latent model training. GTA adds to this framework a global representation token in the latent space. The first component is to add, in the VAE encoder, a projection of the global average pooling of internal features (just before output projection) to get a single token, and a loss aligning it with the global token of the REPA-E encoder. The second is to use this global token in the latent SiT (as input and in the flow matching objective). To ensure coherence between patch and global tokens, the method adds a similarity loss between the internal global token representation in SiT internal feature, and the pooled representation of the patches.

Critical strengths

The method directly addresses two issues of the end-to-end training method REG: learning the global token (instead of merely using the one from a representation auto-encoder) and ensuring coherence between the global token and the patch latents. This allows GTA to outperform previous end-to-end training methods.

Critical weaknesses
  • The text-to-image experiment setting is too weak, while the generalization of the method to such benchmarks would clearly state if the global token only helps for object-centric generation, as in ImageNet, or if the method generalizes to more complex scenes. Specifically: 1) the choice of SD-VAE for this experiment seems odd, when modern alternatives such a Flux-VAE or even SD3 exists and 2) the experiment doesn’t show that GTA helps on text-to-image tasks, but only that fine-tuning of an underperforming VAE helps: a comparison with REPA-E and REG would be necessary.
  • The paper clarity could be greatly improved: while the REPA-E / REG components are mentioned in the introduction, the method section would benefit from introducing those methods as they are parts of the training method, and mentioning them more explicitly in the method section instead of “original end-to-end framework”. Similarly, the method could be clearer by expliciting the full final training objective, as the current one uses “L_base” without clear and explicit definition in the paper.
Questions for authors

could the authors provide clear results on the T2I task or any results showing that GTA generalizes beyond object-centric datasets?

4 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer JF2c - Part 1/2

We thank the reviewer for the careful assessment and constructive comments. In response to your concerns regarding tokenizer choice, text-to-image transfer, contribution attribution, and the method description, we have conducted additional controlled comparisons and revised the manuscript. The main findings and revisions are summarized below:

  • W1-1 | Choice of SD-VAE and experimental rationale: We use SD-VAE to compare training methods with the same tokenizer architecture and initialization, thereby isolating GTA's contribution. Experiments with VA-VAE initialization also demonstrate gains from GTA, supporting its effectiveness with another tokenizer initialization.

  • Q1/W1-2 | Text-to-image transfer with end-to-end fine-tuned controls: We added two end-to-end fine-tuned controls, E2E-SDVAE and E2E(w/ GM)-SDVAE. GTA-SDVAE still achieves the highest GenEval (0.33) and DPG-Bench (70.10) scores, demonstrating additional transfer benefits beyond standard end-to-end fine-tuning.

  • W1-3 | The tokenizer's independent contribution under matched downstream objectives: With either REPA or REG held fixed and the same downstream training budget of 400k steps, GTA-VAE improves both gFID and IS for both tokenizer initializations. These results show that its benefits persist after controlling for the downstream training objective.

  • W2 | Origins of existing components and complete training objectives: We have revised Sections 3.3 and 3.4 to clarify how the REPA-E framework and REG-related components relate to GTA's additional alignment terms, and to explicitly present the complete training objectives.


W1-1 | Choice of SD-VAE and experimental rationale


We appreciate the reviewer's suggestion regarding tokenizer choice. We use SD-VAE primarily to maintain consistency with the baseline settings of REPA-E and related work, allowing us to compare training methods with the same tokenizer architecture and initialization and thereby isolate GTA's contribution more clearly.

For the text-to-image experiments, we additionally compare tokenizers obtained from the same SD-VAE initialization using different end-to-end training methods, as shown in Table R1. This design focuses the comparison on transfer benefits arising from the training method while controlling for differences in tokenizer architecture and initialization.

Our experiments are also not limited to SD-VAE. With VA-VAE initialization, GTA-VAE likewise improves both gFID and IS when the downstream objective is fixed to either REPA or REG, as reported in Table R2. These results provide evidence of effectiveness with another tokenizer initialization. As part of our future open-source work, we will further explore GTA with other VAEs to assess and broaden its applicability.


Q1/W1-2 | Text-to-image transfer with end-to-end fine-tuned controls


We thank the reviewer for pointing out that comparison with the original SD-VAE alone does not separate GTA's benefits from those of standard end-to-end fine-tuning. To directly address this concern, we added two controls: E2E-SDVAE, obtained from REPA-E, and E2E(w/ GM)-SDVAE, obtained from REPA-E (w/ Global Modeling). Both controls, as well as GTA-SDVAE, have undergone end-to-end fine-tuning.

We freeze each tokenizer and train a text-to-image model using DiT-2.5B and a Gemma-2B text encoder on CC-12M for 80k steps. Evaluation uses 25 sampling steps with a CFG scale of 6. All comparisons use identical downstream training and evaluation settings.

Table R1. Text-to-image transfer performance with different frozen tokenizers at 80k downstream training steps.

Frozen tokenizerGenEval ↑DPG-Bench ↑
SD-VAE0.2164.36
E2E-SDVAE0.3268.67
E2E(w/ GM)-SDVAE0.2766.98
GTA-SDVAE0.3370.10

GTA-SDVAE achieves the highest scores on both metrics. Relative to the end-to-end fine-tuned E2E-SDVAE, GenEval increases from 0.32 to 0.33, and DPG-Bench increases from 68.67 to 70.10.

Thus, GTA provides additional text-to-image transfer benefits even when compared with end-to-end fine-tuned tokenizers. These experiments extend the evaluation from class-conditional generation on ImageNet to text-conditional generation trained on CC-12M, providing direct evidence that GTA's benefits transfer to this text-to-image setting. We have updated Section 4.3 and the corresponding figure in the revised manuscript to include the full comparison among the four tokenizers.

AuthorsOpenReview ↗

Authors' Response to Reviewer JF2c - Part 2/2

W1-3 | GTA-VAE's contribution persists under matched REPA or REG objectives


To further separate the tokenizer's contribution from that of the downstream training objective, we compare E2E-VAE and GTA-VAE with both SD-VAE and VA-VAE initializations. For each paired comparison, we freeze the tokenizers and train SiT-XL with the same model configuration and the same downstream objective, either REPA or REG, for 400k steps (80 epochs) and evaluate without CFG.

Table R2. Frozen-tokenizer comparisons under matched downstream objectives and training budgets (SiT-XL, 400k steps, without CFG).

Frozen VAEgFID ↓sFID ↓IS ↑P ↑R ↑
REPA
E2E-SDVAE5.104.60145.450.740.64
GTA-SDVAE4.614.47148.360.760.62
E2E-VAVAE3.464.17159.800.770.63
GTA-VAVAE3.134.42165.150.770.62
REG
E2E-SDVAE2.914.55188.890.770.62
GTA-SDVAE2.544.43196.120.790.60
E2E-VAVAE2.524.12186.510.780.63
GTA-VAVAE2.364.49190.740.790.61

GTA-VAE achieves lower gFID and higher IS in all four matched comparisons. In particular, when both tokenizers are evaluated using REG, GTA-VAE retains its advantage with both initializations. These results demonstrate the independent downstream utility of the tokenizer learned by GTA: its benefits persist after controlling for the downstream training objective.


W2 | Clarifying the origins of existing components and the complete training objectives


We thank the reviewer for identifying aspects of the method description that require clarification. Although Appendix F of the original manuscript already provides the complete training objectives and loss weights, we agree that these definitions should also be explicit in the main method section so that readers can directly understand the origins of the components and GTA's additional contributions.

We have revised the manuscript accordingly. Sections 3.3 and 3.4 now explicitly present the complete training objectives and identify the components inherited from prior work.

I would like to thank the authors for the clear answers to the points raised in the comment.

  • Soundness: the additional experiments provided clarify that the results extend beyond the initial limited setting, as they are not dependent of a class-centric dataset or a specific VAE
  • Clarity: I believe the changes brought to the paper makes it much easier to follow, in particular the inclusion the loss formulation in the main paper. This greatly helps picturing the method.

I changed the rating for 3 to 4, as my concerns are resolved.

AuthorsOpenReview ↗

Authors' Response to Reviewer JF2c

Thank you for your positive feedback and constructive comments. We sincerely appreciate the time and care you have devoted to reviewing our work.

Reviewer_YFmrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

GTA extends end-to-end tokenizer/denoiser training with a shared global semantic representation. It aligns pooled VAE features to a frozen image teacher and aligns a noised global denoising token with pooled spatial features during generation. Built on REPA-E and a REG-style global modeling component, the approach improves generation on ImageNet-256 and produces tokenizers that can be frozen and reused with several downstream denoising objectives.

Critical strengths
  1. The frozen-tokenizer experiments provide useful evidence that the gains are not confined to the original jointly trained system. In Table 3, replacing E2E-SDVAE with GTA-SDVAE improves gFID under no alignment, REPA, and REG from 25.18/14.76/9.02 to 22.78/13.21/7.56. Gains also appear with the VA tokenizer family. Holding the downstream denoising objective fixed makes these comparisons a substantive strength.
  2. The complete system achieves strong empirical improvements and includes relevant component ablations. For SD-based end-to-end training, Table 1 reports 6.42 versus 12.83 FID at 20 epochs and 2.51 versus 4.07 at 80 epochs. Tables 4 and 6 examine global modeling and the two alignment losses, and the supplementary material reports hyperparameters and reconstruction tradeoffs. This is a plausible and practically useful combination of tokenizer and denoiser supervision.
Critical weaknesses
  1. Much of the headline improvement comes from the pre-existing global modeling component, while the new alignment mechanism is less decisively validated. Table 4 improves 16.3 to 9.2 by adding global modeling, then to 8.1 with GTA. Table 6 gives approximately 9.18 without VAE alignment, 8.43 without SiT alignment, and 8.05 with both. These results support some incremental benefit, but the denoiser-side difference is small, and Table 7 is sensitive to the teacher: DINOv2-B reaches 8.05 gFID, while DINOv2-L, I-JEPA-H, and CLIP-L reach 11.43, 15.88, and 17.53. The paper does not directly measure whether semantics drift over noise levels or whether GTA reduces such drift. Thus, the claimed semantic bridge currently functions more as an interpretation of an effective combination than as a demonstrated mechanism.
  2. Several broader comparisons mix factors or budgets. Table 5 compares a GTA tokenizer with REG against an end-to-end-trained tokenizer with REPA, so its strongest result does not isolate the tokenizer contribution; the better-controlled Table 3 partly addresses this, but at a different setting. The qualitative text-to-image comparison uses the original SD tokenizer rather than a tokenizer fine-tuned without GTA. Moreover, convergence measured in epochs excludes differences in global-model, teacher, and tokenizer-training cost. These issues limit claims about unique transfer benefits and speedup, even if the complete pipeline improves FID.
Questions for authors
  1. Sensitivity and mechanism: Can the VAE-alignment and SiT-alignment ablation in Table 6 also be performed with DINOv2-L, whose full-system result is worse than DINOv2-B in Table 7? This would identify which alignment component is sensitive to teacher choice. Measuring global/spatial semantic consistency across noise levels would further test the proposed semantic bridge.
  2. Attribution test: At the headline downstream training budget, how do the GTA and REPA-E tokenizers compare when both use REG? For text-to-image transfer, does GTA still help relative to a tokenizer receiving the same end-to-end fine-tuning without the new losses? These comparisons would materially affect my assessment of the distinct contribution.
  3. Efficiency clarification: What are the total training FLOPs or GPU-hours, including the tokenizer preparation used by the frozen-tokenizer experiments, at matched generation quality? Please distinguish an epoch-to-quality reduction from an actual compute or wall-clock speedup.

If my concerns can be addressed, I will consider increasing my score.

9 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer YFmr - Part 1/4

We thank the reviewer for the careful assessment and constructive suggestions. We have conducted additional experiments in response to your comments and summarize the findings below, following the order of your questions:

  • Q1/W1-1 | Teacher sensitivity and stability of the gains from GTA components: Ablations with DINOv2-L help identify the source of teacher sensitivity. The VAE-alignment-only configuration remains relatively stable across the two tested DINOv2 teachers, whereas the observed sensitivity is primarily associated with introducing denoiser-side alignment. With DINOv2-B and global modeling (GM) held fixed, experiments across multiple random seeds show that GTA reduces mean gFID from 9.53 ± 0.25 to 8.05 ± 0.18, supporting the stability of its gains.
  • Q1/W1-2 | Global–spatial semantic consistency across noise levels: GTA improves consistency between global and spatial hidden representations at every evaluated noise level and improves the semantic consistency of predicted outputs at intermediate noise levels.
  • Q2/W2-1 | The independent contribution of GTA-VAE under matched downstream objectives: With either REPA or REG held fixed and the same downstream training budget of 400k steps, GTA-VAE improves both gFID and IS for both tokenizer initializations. Its benefits therefore persist after controlling for the downstream objective.
  • Q2/W2-2 | Text-to-image transfer with end-to-end fine-tuned controls: After adding E2E-SDVAE and E2E(w/ GM)-SDVAE as end-to-end fine-tuned controls, the results continue to support GTA's transfer benefits on GenEval and DPG-Bench.
  • Q3/W2 | Training cost analysis: GTA and REPA-E have comparable training time per step. GTA increases measured step time by only 1.8% and FLOPs by approximately 0.1%. Based on measured step times, GTA achieves a gFID of 3.52 at an estimated 478.5 A100 GPU-hours, outperforming REPA-E's 4.07 at 940.5 A100 GPU-hours.
AuthorsOpenReview ↗

Authors' Response to Reviewer YFmr - Part 2/4

Q1/W1-1 | Teacher sensitivity and stability of the gains from GTA components


Thank you for raising this question. We conducted two complementary sets of experiments. First, following your suggestion, we repeated the component ablation with DINOv2-L to examine how the two alignment components respond to teacher choice. Second, to assess the stability of the gains from GTA and its components, we repeated the experiments in the original Table 6 with multiple random seeds and report the means and standard deviations.


1. Ablation with DINOv2-L: separating the responses of the two alignment components to teacher choice

We repeated the ablation in the original Table 6 using DINOv2-L, with the results reported below.

Table R1. GTA component ablation with DINOv2-L. We follow the training setup of the original Table 6. All three GTA configurations include GM and differ only in whether VAE Align and SiT Align are enabled. REPA-E is the original reference without GM.

MethodgFID ↓sFID ↓IS ↑
REPA-E16.005.5977.7
GTA (w/o VAE Align)11.224.87109.5
GTA (w/o SiT Align)8.635.16125.2
GTA11.434.91106.7

These results indicate that the VAE-GTA-only configuration remains relatively stable across the tested DINOv2 teachers, whereas SiT-GTA is more sensitive to the global-token representation provided by DINOv2-L and can reduce the benefits obtained with VAE-GTA alone. A similar dependence on teacher choice when introducing a global token into the denoiser is reported in Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think. Table 3 of that paper reports an FID of 17.36 with DINOv2-L, compared with 15.22 with DINOv2-B. The same trend is also observed when using class-token entanglement without representation alignment (Table 6 of that paper). These findings provide additional evidence that global-token modeling can be sensitive to teacher choice. We therefore hypothesize that the denoiser's greater sensitivity to the quality of global-token representations contributes to the observed behavior.


2. Both alignment components provide reproducible improvements in mean performance with global modeling held fixed

We added REPA-E (w/ GM) as the reference for assessing the two alignment losses and repeated the experiments with different random seeds, as shown below.

Table R2. Component ablation across multiple random seeds with DINOv2-B. Main values denote means, and subscripts denote standard deviations. † denotes the original REPA-E, included only as a contextual reference. All other configurations include global modeling. We use SiT-L, SD-VAE, and DINOv2-B, train for 100k steps, and evaluate without CFG.

MethodgFID ↓sFID ↓IS ↑
REPA-E†16.316.35.695.6975.075.0
REPA-E (w/ GM)9.530.259.53_{0.25}4.710.094.71_{0.09}118.201.48118.20_{1.48}
GTA (w/o VAE Align)8.670.458.67_{0.45}4.940.234.94_{0.23}123.263.26123.26_{3.26}
GTA (w/o SiT Align)8.300.128.30_{0.12}5.230.345.23_{0.34}126.560.83126.56_{0.83}
GTA8.050.188.05_{0.18}5.070.385.07_{0.38}128.180.92128.18_{0.92}

These repeated-run results support stable gains in mean gFID and IS from GTA when using DINOv2-B as the teacher.


3. GTA also improves the tokenizer's independent downstream utility

We further emphasize that GTA's advantages over REPA-E (w/ Global Modeling) extend beyond end-to-end generation performance. In the evaluated downstream setting, the tokenizer obtained from REPA-E (w/ Global Modeling) yields worse gFID than E2E-VAE obtained from the original REPA-E. We compare the VAEs obtained from REPA-E, REPA-E (w/ Global Modeling), and GTA by freezing each tokenizer and training SiT-XL with REPA for 400k steps, as shown in Table R3.

Table R3. Downstream gFID with frozen tokenizers. Tokenizers are obtained from each method's 400k-step end-to-end checkpoint. The downstream SiT-XL is trained from scratch with REPA and evaluated without CFG.

Frozen VAE50K100K200K400K
E2E-SDVAE32.3811.977.065.10
E2E(w/ GM)-SDVAE37.6112.687.305.16
GTA-SDVAE34.0911.176.484.61

E2E(w/ GM)-SDVAE yields higher gFID than E2E-SDVAE at every reported checkpoint, whereas GTA-SDVAE achieves the best gFID at all reported checkpoints from 100k to 400k steps.

AuthorsOpenReview ↗

Authors' Response to Reviewer YFmr - Part 3/4

Q1/W1-2 | Global–spatial semantic consistency across noise levels


We thank the reviewer for suggesting a direct measurement of global–spatial semantic consistency across noise levels. Following your suggestion, we have added two complementary diagnostics to Section 4.5, “Global-Spatial Consistency across Noise Levels,” and the corresponding figure in the revised manuscript, examining GTA at both the hidden-representation and predicted-output levels.

We compare SD-VAE + SiT-XL/2 checkpoints obtained after 400k steps of end-to-end training with REPA-E (w/ Global Modeling) and GTA. Evaluation uses all 50,000 ImageNet validation images, with identical images and noise samples for both methods.


1. Hidden-representation consistency

After the eighth SiT block, we directly compute the cosine similarity between the global hidden token and the mean of the 256 spatial hidden tokens. As shown in Figure 6(a) of the revised manuscript, GTA achieves higher mean similarity at all 11 evaluated noise levels, with absolute gains ranging from 0.1465 to 0.2372. This result indicates that GTA strengthens consistency between global and spatial hidden representations at the alignment layer.


2. Semantic consistency of clean predictions

We further evaluate semantic consistency between the clean predictions of the global and spatial streams. At each nonzero noise level, we estimate the clean global token and spatial latents from a single velocity prediction, decode the estimated latents using the corresponding VAE, and compute the cosine similarity between the predicted global token and the DINOv2-B CLS feature of the decoded image.

As shown in the figure, GTA and REPA-E (w/ Global Modeling) exhibit similar semantic consistency at high and low noise levels. This may be related to the input states near the two endpoints: high-noise inputs are dominated by Gaussian noise, whereas low-noise inputs are already close to clean samples. GTA's main advantage is concentrated at intermediate noise levels, where its predicted global tokens exhibit stronger semantic agreement with the decoded images. This observation suggests that coordination between global and spatial representations at intermediate noise levels may be an important contributor to GTA's effectiveness. We appreciate this suggestion, which also motivates us to explore noise-dependent alignment strategies in future extensions.


Q2/W2-1 | The independent contribution of GTA-VAE under matched downstream objectives


We thank the reviewer for highlighting the importance of separating the tokenizer's contribution from that of the downstream training objective. To assess GTA-VAE's independent contribution, we conducted a fully crossed comparison involving two tokenizer initializations (SD-VAE and VA-VAE) and two downstream training objectives (REPA and REG).

Specifically, we freeze the E2E-VAE and GTA-VAE obtained from end-to-end training and train new SiT-XL models using either REPA or REG. For each paired comparison within the same tokenizer initialization, we keep the downstream objective, model configuration, and training budget fixed, changing only the frozen tokenizer. All downstream models are trained for 400k steps (80 epochs) and evaluated without CFG.

Table R4. Frozen-tokenizer comparisons under matched downstream objectives and training budgets.

Frozen VAEgFID ↓sFID ↓IS ↑P ↑R ↑
REPA
E2E-SDVAE5.104.60145.450.740.64
GTA-SDVAE4.614.47148.360.760.62
E2E-VAVAE3.464.17159.800.770.63
GTA-VAVAE3.134.42165.150.770.62
REG
E2E-SDVAE2.914.55188.890.770.62
GTA-SDVAE2.544.43196.120.790.60
E2E-VAVAE2.524.12186.510.780.63
GTA-VAVAE2.364.49190.740.790.61
  • GTA-VAE provides additional gains with REG held fixed. With SD-VAE initialization, GTA reduces gFID from 2.91 to 2.54 and increases IS from 188.89 to 196.12. With VA-VAE initialization, gFID decreases from 2.52 to 2.36, and IS increases from 186.51 to 190.74. These comparisons directly demonstrate that GTA-VAE's gains persist when both tokenizers are evaluated with REG.
  • The gains also hold under REPA. With SD-VAE and VA-VAE initialization, GTA reduces gFID from 5.10 to 4.61 and from 3.46 to 3.13, respectively, while improving IS in both cases. The observed tokenizer benefits are therefore not restricted to REG.
Reviewer_HiatOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GTA adds two alignment losses to end-to-end latent diffusion training. One aligns VAE features with a frozen image encoder's global token. The other aligns SiT patch features with the jointly denoised global token. The goal is to connect tokenizer learning and denoising through shared global semantics. The paper also tests the resulting tokenizers with separately trained generators.

Critical strengths
  • The frozen-tokenizer experiments are useful. Table 3 keeps the downstream objective matched and shows gains under no alignment, REPA, and REG. This supports a tokenizer benefit beyond improving the jointly trained system.

  • The convergence gains are substantial under both initializations. At 80 epochs, GTA improves gFID from 4.07 to 2.51 with SD-VAE and from 3.94 to 2.47 with VA-VAE. The appendix also reports reconstruction trade-offs.

Critical weaknesses
  • My main concern is how much improvement comes from the two new losses. Table 4 shows that global modeling alone improves gFID from 16.3 to 9.2. GTA then reaches 8.1. Section 4.4 compares the individual losses against 16.3, which also credits them with the global-modeling gain. The SiT alignment increment is smaller, from 8.43 to 8.05 in Table 6, without run variation.

  • Training time is not reported. Fewer epochs do not necessarily mean less compute once alignment operations are added. Appendix F gives hardware and batch size, but no measured time-to-quality comparison. I wonder how much faster GTA is than global modeling alone or REPA-E at matched quality.

  • The long-run comparison changes both tokenizer and downstream objective. Table 5 uses REG for GTA-VAVAE and REPA for E2E-VAVAE. Table 3 already gives a matched comparison at the shorter setting. However, Table 5 does not isolate the tokenizer's contribution to the 800-epoch result.

Questions for authors
  • With global modeling fixed, how stable are the gains from each alignment loss and their combination?

  • What GPU time do REPA-E, global modeling alone, and GTA need to reach the same gFID?

  • How do E2E-VAVAE and GTA-VAVAE compare at 800 epochs with the same REG objective and guidance protocol?

2 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer Hiat - Part 1/2

We thank the reviewer for recognizing the merits of GTA and for the careful review. Your constructive comments have helped us strengthen the evaluation and clarify GTA's contributions. We have conducted additional experiments addressing each concern, with the main findings summarized below:

  • Q1/W1 — Attribution and stability of the gains. We repeated the ablation experiments with different random seeds. With global modeling (GM) held fixed, GTA reduces mean gFID from 9.53 ± 0.25 to 8.05 ± 0.18. Each of its two alignment components also provides an improvement when used individually.
  • Q2/W2 — Training efficiency. GTA and REPA-E have comparable per-step training costs. GTA increases measured step time by only 1.8% and FLOPs by approximately 0.1% relative to REPA-E. GTA achieves 3.52 gFID with an estimated 478.5 A100 GPU-hours, outperforming REPA-E's 4.07 gFID with 940.5 A100 GPU-hours.
  • Q3/W3 — Comparisons under matched downstream objectives. We added fully crossed comparisons at 80 epochs (400k steps). Under both REPA and REG, GTA-VAE achieves lower gFID and higher IS than the corresponding E2E-VAE.

Q1/W1: Stability of the individual and combined alignment gains with global modeling held fixed


1. Controlled ablations across random seeds

Thank you for raising this point. The repeated-seed results support consistent gains from GTA. We added REPA-E (w/ GM) as an explicit reference in the ablation and repeated the experiments with different random seeds, using SiT-L, SD-VAE, and DINOv2-B for 100k steps, without CFG. The means and standard deviations are reported below.

Table R1. Component ablation with global modeling held fixed. Main values denote means, and subscripts denote standard deviations across random seeds. † denotes the original REPA-E without GM, included only as a reference; the remaining configurations all include GM.

MethodgFID ↓sFID ↓IS ↑
REPA-E†16.316.35.695.6975.075.0
REPA-E (w/ GM)9.530.259.53_{0.25}4.710.094.71_{0.09}118.201.48118.20_{1.48}
GTA (w/o VAE Align)8.670.458.67_{0.45}4.940.234.94_{0.23}123.263.26123.26_{3.26}
GTA (w/o SiT Align)8.300.128.30_{0.12}5.230.345.23_{0.34}126.560.83126.56_{0.83}
GTA8.050.188.05_{0.18}5.070.385.07_{0.38}128.180.92128.18_{0.92}

Relative to REPA-E (w/ GM):

  • Combining both losses reduces mean gFID by 1.48 and increases mean IS by approximately 10.0.
  • Using either loss individually also improves mean gFID: VAE Align reduces it by 1.23, while SiT Align reduces it by 0.86.

These repeated-seed results preserve the same ordering in mean gFID and IS, supporting the consistency of the gains from GTA and its individual components.


2. Tokenizer benefits beyond end-to-end generation

We also emphasize that GTA's advantages extend beyond outperforming REPA-E and REPA-E (w/ GM) in end-to-end training. Although GM accelerates end-to-end training, the resulting tokenizer yields higher downstream gFID than the original E2E-VAE obtained with REPA-E in our controlled evaluation. By comparison, the tokenizer learned with GTA outperforms both E2E-VAE and E2E(w/ GM)-VAE at the 400k-step downstream budget.

We evaluate the frozen tokenizers by training SiT-XL with REPA for 400k steps. As shown in Table R2, GTA-SDVAE achieves 4.61 gFID, compared with 5.10 for E2E-SDVAE and 5.16 for E2E(w/ GM)-SDVAE.

Table R2. Downstream gFID with frozen tokenizers. A new SiT-XL is trained with REPA for each frozen tokenizer. Columns indicate downstream training steps; lower is better.

Frozen VAE50k100k200k400k
E2E-SDVAE32.3811.977.065.10
E2E(w/ GM)-SDVAE37.6112.687.305.16
GTA-SDVAE34.0911.176.484.61

AuthorsOpenReview ↗

Authors' Response to Reviewer Hiat - Part 2/2

Q2/W2: Measurement and analysis of GPU training time


1. Measured per-step training cost

Thank you for this question. We clarify that REPA-E, REPA-E (w/ GM), and GTA have comparable per-step training times. Relative to the original REPA-E, the additional components include two average-pooling operations, a lightweight projection head, two cosine-similarity losses, and one additional token. These components introduce only a small computational overhead, which we quantify below.

We measured step latency and profiled FLOPs for all three methods on 8 × A100 GPUs, using SiT-XL, SD-VAE, and DINOv2-B. After 500 warm-up steps, we averaged the latency over the following 300 steps.

Table R3. Per-step training cost. Step time is averaged over 300 steps after 500 warm-up steps. Computational cost is reported in GFLOPs per image, with the relative increase calculated against REPA-E.

MethodMean step time (s/step)GFLOPs/imageFLOPs increase over REPA-E
REPA-E1.058033776.932+0.0000%
REPA-E (w/ GM)1.075623780.507+0.0946%
GTA1.076663780.568+0.0963%

These measurements show that GTA has comparable step latency and FLOPs to both baselines.


2. Generation quality versus estimated GPU-hours

Table R4 reports generation quality from 50k to 400k steps using SiT-XL, SD-VAE, and DINOv2-B, without CFG. Main values denote gFID, and subscripts denote estimated A100 GPU-hours, extrapolated from the measured mean step time on the same 8 × A100 setup.

At the evaluated checkpoints, REPA-E achieves 4.07 gFID with an estimated 940.5 A100 GPU-hours, whereas GTA achieves 3.52 gFID with only 478.5 A100 GPU-hours. GTA therefore obtains better generation quality with approximately half the estimated GPU-hour budget.

Table R4. End-to-end gFID and estimated training cost across training steps. The configuration is SiT-XL with SD-VAE and DINOv2-B, evaluated without CFG. Each entry reports gFID with estimated A100 GPU-hours in the subscript.

Method (gFIDGPU-hours\mathrm{gFID}_{\text{GPU-hours}})50k100k200k400k
REPA-E45.84117.645.84_{117.6}12.83235.112.83_{235.1}7.17470.27.17_{470.2}4.07940.54.07_{940.5}
REPA-E (w/ GM)27.38119.527.38_{119.5}7.28239.07.28_{239.0}3.92478.13.92_{478.1}2.71956.12.71_{956.1}
GTA25.97119.6\mathbf{25.97}_{119.6}6.42239.3\mathbf{6.42}_{239.3}3.52478.5\mathbf{3.52}_{478.5}2.51957.0\mathbf{2.51}_{957.0}

3. Accompanying improvements in downstream tokenizer performance

These efficiency gains are accompanied by improved downstream tokenizer performance. As shown in Table R2, with the tokenizer frozen and the same SiT-XL + REPA setup trained for 400k steps, GTA-SDVAE achieves 4.61 gFID. In comparison, the tokenizer obtained from REPA-E (w/ GM) yields 5.16 gFID, which is worse than the 5.10 achieved by E2E-SDVAE.


Q3/W3: Comparing GTA-VAE and E2E-VAE under matched downstream objectives and settings


Thank you for this valuable suggestion. Isolating the contribution of GTA-VAE is indeed important. However, an 800-epoch (4M-step) run requires approximately 300 wall-clock hours on 8 × A100 GPUs. Given the time and computational resources available during the rebuttal period, we are currently unable to provide the requested results at this budget. We have nevertheless conducted a comprehensive set of controlled experiments at 400k steps, all using SiT-XL, to isolate the contribution of GTA-VAE.

Table R5. Frozen-tokenizer comparisons under matched downstream objectives. All downstream models use SiT-XL, are trained for 80 epochs (400k steps), and are evaluated without CFG. P and R denote precision and recall, respectively.

Frozen VAEgFID ↓sFID ↓IS ↑P ↑R ↑
REPA
E2E-SDVAE5.104.60145.450.740.64
GTA-SDVAE4.614.47148.360.760.62
E2E-VAVAE3.464.17159.800.770.63
GTA-VAVAE3.134.42165.150.770.62
REG
E2E-SDVAE2.914.55188.890.770.62
GTA-SDVAE2.544.43196.120.790.60
E2E-VAVAE2.524.12186.510.780.63
GTA-VAVAE2.364.49190.740.790.61

Across both evaluated downstream objectives, GTA-VAE consistently achieves lower gFID and higher IS than the corresponding E2E-VAE. We plan to supplement these comparisons with results at 800 epochs (4M steps) and release the corresponding model weights and code on GitHub.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.