acceptodds
Under review as a conference paper at ICLR 2027

GuaVAE: Guidance-Aligned End-to-End VAE Training for Latent Diffusion

Abstract

Recent studies have demonstrated that pretrained visual encoders such as DINO can help to improve latent spaces for diffusion modeling. However, these approaches depend on an external representation model. We investigate whether diffusion transformers (DiTs) themselves can provide supervision for learning better latent spaces through VAE tuning. Prior work has shown that classifier-free guidance (CFG) can enhance internal DiT features, making them effective targets for representation alignment during training. We introduce GuaVAE, an end-to-end VAE tuning method that extends this objective to the VAE encoder. To encourage the VAE to learn a more generation-friendly latent space, we align a trainable DiT's features with CFG-enhanced features obtained at low noise levels from a frozen pretrained DiT. We backpropagate the alignment loss through the trainable DiT to the VAE encoder. Experiments show that the tuned VAE improves generation quality across DiT backbone sizes, achieving performance competitive with DINO-supervised VAEs. We also observe better generation quality when using VAEs tuned with larger DiTs, supporting diffusion representations as a scalable source of supervision. A DiT trained with the tuned VAE and CFG-enhanced feature alignment achieves a gFID of on ImageNet without CFG, outperforming self-contained representation alignment baselines as well as DINO-guided REPA. With CFG, it achieves a competitive gFID of .

Primary area:
generative models
Keywords:
Latent Diffusion, Variational Autoencoders, Representation Alignment, End-to-End Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fNX4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose GuaVAE, an extension of the existing REPA-E pipeline. Instead of using an external vision encoder such as DINOv2, they use CFG-enhanced features from a frozen pretrained DiT teacher as the alignment target. Additionally, they sample the teacher timestep independently from a cleaner time interval. The tuned VAE improves diffusability for generation over the original VAEs, reaching 4.97 gFID without CFG and 1.62 with CFG on ImageNet-256.

Critical strengths
  • The paper writing is clear and easy to follow.
  • The tuned VAE consistently improves generation over the original VAEs, e.g., generation FID drops from 37.02 to 22.53 on SD-VAE, and from 39.31 to 19.30 on IN-VAE (Table 1).
  • The paper provides detailed ablations on each component of the method, including the teacher timestep, feature guidance, teacher preparation, and alignment weight (Table 3).
Critical weaknesses
  • The motivation for avoiding external vision encoders is not clearly stated. GuaVAE does not outperform REPA-E (Table 1, Table 2) and still requires additional teacher training.
  • The novelty is limited. The method reads like REPA-E (alignment) + Self-Transcendence (teacher feature) + SRA (cleaner teacher timesteps).
  • The training cost is high. Training throughput is about 3.6× slower than REPA-E (Table 10), even when excluding 200K iterations of teacher preparation.
  • The abstract and introduction highlight the unguided generation results. However, real use cases for image generation models rely more on guided results. The guided result is worse than Self-Transcendence with SD-VAE (1.62 vs. 1.44, Table 7).
Questions for authors
  • Please refer to the weakness section above.
3 comments

Thank you for your careful reading and comments on the motivation and practical relevance of our work. For the corrected training-cost measurements and the discussion of CFG performance, please refer to our response to all reviewers.

We have expanded the motivation for avoiding external semantic encoders in the Introduction. Please see the revised version for this discussion.

To address your concern about guided generation in practical applications, we added text-to-image experiments with and without CFG (Section 4.5, Table 5). Generators trained on our tuned VAE improve over SD-VAE and perform comparably to REPA-E in both settings. Together with the lower VAE-training cost reported in Appendix A, these results demonstrate a practical use of our method beyond the unguided performance of the complete generative system.

Regarding novelty, our method builds on prior alignment approaches, but transferring their benefits from diffusion training to VAE tuning is not straightforward. Using CFG-enhanced teacher features with shared timesteps provides little improvement to the VAE. Table 3(b) separates the effects of independent timestep sampling and restricting the teacher to a low-noise interval, showing that both contribute to the improvement. The resulting design does not require the teacher input to be cleaner than the student's at every update. These findings establish the conditions under which internal diffusion representations become useful for improving the VAE itself.

Thank you for updating the timing measurements and adding the text-to-image results. It address my concern about VAE training cost. My concerns about novelty and the guided performance still remain. Therefore, I keep my score at 2.

Thank you for your response. We would like to clarify the remaining concern about guided performance. Our primary contribution is a method for improving VAE latent spaces, so we evaluated whether the tuned VAE benefits new generators when using CFG.

Please see the ImageNet and text-to-image results in our comment to all reviewers (Tables 16 and 5). In both settings, generators trained on our frozen VAE outperform those using SD-VAE and achieve quality comparable to REPA-E with CFG. These results demonstrate that the benefits of our VAE tuning extend to guided generation.

Reviewer_CmADOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GuaVAE tunes a VAE end-to-end for latent diffusion in the style of REPA-E, but replaces the pretrained vision foundation model (e.g., DINOv2) alignment target with features from a frozen SiT teacher. The teacher model is prepared as in Self-Transcendence and its features are enhanced with feature-level CFG. A trainable student aligns its features with these targets, and the alignment loss flows back through the student into the VAE encoder, while the diffusion loss uses detached latents. SiT generators are then trained from scratch on the frozen tuned VAE. Experimental results cover class-conditional ImageNet generation with two VAEs and several SiT sizes, a tuning-scale study, component ablations, representation analyses and full-system results.

Critical strengths
  1. Training new generators from scratch on the tuned VAE (Table 1) shows that the gain derived from aligned latent space. Large improvements on two latent formats without any external encoder are a meaningful result in a field that relies heavily on DINOv2.
  2. The teacher timestep ablation gives a useful, transferable insight. Shared time-steps give gFID 35.02, barely better than the untuned 37.02. Independent full-range sampling gives 27.68, and an independent low-noise interval gives 23.89. Together with the PCA observation that latent structure fades under shared timesteps, this explains why naive self-alignment fails to reach the VAE, which should help others doing end-to-end tuning.
Critical weaknesses
  1. The novelty is somewhat limited, and the final system does not beat its base method in the standard setting. The pipeline combines REPA-E [1] and Self-Transcendence [2]. The main new ingredient is independent low-noise teacher timesteps. With sampling CFG, GuaVAE reaches a gFID of 1.62, worse than Self-Transcendence [2] (1.44) and LayerSync [3] (1.49) in the authors' own Table 7. A recent end-to-end VAE training paper SpeeDiff [4] also reports 1.46 without REPA alignment. DINO-supervised VAE methods remain well ahead. The practical case for choosing GuaVAE over previous methods is not yet clear.
  2. Additional compute is not accounted for. Generator training is preceded by 200K teacher-preparation steps and 100K tuning steps, and each tuning step needs two teacher forward passes for CFG. Table 6's 80 epochs for the continued student leaves out the teacher stage.
  3. The method seems to need substantial hyperparameter tuning. On top of REPA-E's settings, it adds a teacher-preparation stage with its own auxiliary head, a feature-guidance scale, and a separate low-noise interval for teacher timesteps. The gain depends heavily on hyperparameter choices, and critical feature guidance weight ω\omega is not ablated.

[1] Leng et al. REPA-E: Unlocking VAE for end-to-end tuning of latent diffusion transformers. ICCV, 2025.

[2] Sun et al. Self-transcendence: Is external feature guidance indispensable for accelerating diffusion transformer training? arXiv:2601.07773, 2026.

[3] Haghighi et al. LayerSync: Self-aligning intermediate layers. ICLR, 2026.

[4] Zhang et al. SpeeDiff: Scalable pixel-anchored end-to-end latent diffusion model. CVPR, 2026.

Questions for authors
  1. What is the total training compute (GPU-hours or FLOPs) for each row of Tables 6 and 7, including teacher preparation and tuning, for GuaVAE, Self-Transcendence and REPA-E?
  2. How does the method perform with different choice of ω\omega?
2 comments

We appreciate the reviewer’s constructive feedback on practical applicability, training cost, and hyperparameter sensitivity. Our response to all reviewers presents the updated compute measurements and CFG evaluations. We discuss the practical benefits and hyperparameter choices below.

We acknowledge that our complete generative system does not outperform Self-Transcendence with CFG in the comparable setting. However, our tuned VAE achieves generation quality close to REPA-E both with and without CFG on ImageNet (Appendix B, Table 16) and text-to-image generation (Section 4.5, Table 5), while requiring an estimated 3.05×3.05\times less compute for VAE training, including teacher preparation. This provides a practical benefit: a VAE with comparable downstream generation quality at a substantially lower training cost.

Thank you also for pointing us to SpeeDiff. We will add it to Related Work. Our new result in Table 8 reaches a gFID of 1.43 with CFG, compared with 1.52 for SpeeDiff with CFG; its reported 1.46 uses AutoGuidance.

Regarding hyperparameters, we ablate the VAE alignment weight and teacher timestep interval in Tables 3(a) and 3(b). For teacher preparation and feature guidance, we follow the settings established in Self-Transcendence.

The improvements remain substantial across the tested settings. Varying the alignment weight from 0.1 to 2.0 gives gFID 24.27-26.14, while the tested independent low-noise intervals give 23.89-25.52, compared with 37.02 for the original SD-VAE. The shared-timestep and full-range settings test the importance of our timestep design. The remaining rows show that the gain is consistent across different low-noise intervals. Thus, the improvement does not depend on a narrowly tuned configuration.

We are evaluating additional feature-guidance weights and will provide the results soon.

We extended the feature-guidance ablation to ω∈{1,5,15,30,45,60}\omega \in \lbrace 1,5,15,30,45,60 \rbrace. The table below reports generation quality with and without CFG using the frozen VAEs, extending Table 3(c).

VAEω\omegagFID w/o CFGgFID w/ CFG
SD-VAE-37.026.30
Ours137.276.47
Ours533.575.59
Ours1527.774.54
Ours3023.894.35
Ours4523.974.63
Ours6023.834.39

Increasing ω\omega from 1 to 30 improves gFID both without CFG (37.27 to 23.89) and with CFG (6.47 to 4.35). Further increasing ω\omega provides no consistent improvement: across ω=30,45,60\omega=30,45,60, gFID ranges from 23.83 to 23.97 without CFG and from 4.35 to 4.63 with CFG. These results indicate that sufficiently strong feature guidance benefits both sampling settings, while performance is relatively insensitive to its precise weight within the tested range of 30–60. This trend is consistent with Self-Transcendence (Table 5), where ω=15\omega=15 also underperforms ω=30\omega=30, while FID varies by only 0.66 across ω=30,45,60\omega=30,45,60.

Reviewer_XoFuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GuaVAE tunes a VAE using representation targets from a frozen diffusion teacher, avoiding reliance on an external semantic encoder. The method uses classifier-free-guided teacher features at independently sampled, relatively low-noise timesteps. Alternating updates adapt the VAE and a diffusion student, after which the tokenizer can be frozen for downstream generator training.

Critical strengths
  1. Using the generative model's own features to improve its tokenizer is a coherent and useful direction. The independent low-noise teacher timestep is supported by the component experiments: Table 3 improves gFID from 35.02 with shared timesteps to 27.68 with independent full-range timesteps and 23.89 with the selected low-noise range.
  2. Training fresh generators with the frozen tuned VAE is a strong evaluation choice. It helps distinguish reusable tokenizer improvements from benefits restricted to the jointly trained student. The experiments cover two latent formats and multiple model sizes.
  3. The supplementary material reports reconstruction metrics, implementation details, and measured VAE-tuning time. It also makes the reconstruction/generation tradeoff visible rather than assuming that better generative performance implies uniformly better reconstruction.
Critical weaknesses
  1. GuaVAE shows significant improvement over Self-Transcendence in unguided FID, which could potentially come from incorporating guidance during VAE training. However, it shows no improvement in guided FID (1.62 versus 1.44). This leaves its contribution unclear: is GuaVAE a general improvement to end-to-end LDM training, or is it primarily distilling guidance into the VAE? I would like to see an ablation of teacher guidance during VAE training, reporting its effect on both unguided and guided FID.

  2. This is a question rather than a weakness: Have the authors considered using a slowly moving target instead of a frozen one? Since the VAE can improve itself, it seems natural to use an EMA version of the VAE as the target to enable bootstrapping. DINO and BYOL demonstrate the value of bootstrapping in representation learning, and I wonder whether similar benefits apply here. This also connects to the weakness above: does the improvement come from distilling guidance or from a more general improvement in the latent representation?

Questions for authors
  1. Can you ablate the use of teacher guidance during VAE training and report its effect on both unguided and guided FID? Does the improvement come primarily from distilling guidance into the VAE, or from a more general improvement to end-to-end LDM training?

  2. Have you considered using an EMA version of the VAE as a slowly moving target instead of a frozen one to enable bootstrapping? Could this bring benefits similar to those seen in DINO and BYOL, and help clarify whether the gains reflect guidance distillation or broader latent representation improvement?

1 comment

We thank the reviewer for the thoughtful questions regarding teacher guidance and EMA targets. Please see our response to all reviewers for the additional CFG and text-to-image results. Below, we address the teacher-guidance ablation and the proposed EMA alternatives.

To address your request, we report additional guided generation results for the VAE checkpoints from Table 3(c).

Frozen VAEVAE-tuning backbonegFID w/o CFGgFID w/ CFG
SD-VAE-37.026.30
REPA-ESiT-XL20.503.69
OursSiT-XL22.534.01

Teacher-guidance ablation, Table 3(c)

Frozen VAEVAE-tuning backbonegFID w/o CFGgFID w/ CFG
Ours, ω=1\omega=1SiT-B37.276.47
Ours, ω=30\omega=30SiT-B23.894.35

All results evaluate SiT-B/2 generators trained on frozen VAEs using only the diffusion loss.

Without teacher feature guidance (ω=1\omega=1), generation quality remains close to the original SD-VAE both with and without CFG. Using CFG-enhanced teacher features (ω=30\omega=30) improves both results. This shows that teacher guidance helps learn a VAE latent space that benefits downstream generation in both sampling regimes.

Regarding your suggestion to use an EMA target, we distinguish three possible variants:

  1. Updating only the teacher VAE through EMA. This would change the latent distribution received by the frozen diffusion teacher. We have not tested this variant because of the potential mismatch between the updated VAE and the diffusion teacher.

  2. Updating the diffusion teacher through EMA, with a fixed VAE. We tested this in the Self-Transcendence setting, where the teacher and student share one frozen VAE. We updated the diffusion teacher as an EMA of the student and used its CFG-enhanced features as alignment targets. Several variants, including delaying the start of EMA updates, were unsuccessful. These results motivated retaining a prepared, frozen diffusion teacher for VAE tuning.

  3. Updating both the teacher VAE and diffusion teacher through EMA. We have not tested this variant. The results described above raise concerns that CFG-enhanced features from an EMA teacher may also be ineffective when the teacher VAE is updated. Using EMA targets without feature CFG could behave differently, but would require a separate investigation. In our current setup, conditional teacher features without CFG amplification do not improve downstream generation, even with independent, low-noise teacher timesteps (Table 3(c)). We therefore retained the target construction supported by our experiments.

If your suggestion refers to an approach not covered above, we would appreciate further clarification.

Comments on the paper

2 comments

We thank the reviewers for their constructive feedback. Please see the revised paper, where additions are highlighted in green and deletions in red. Below, we address two concerns raised across the reviews: training cost and performance with CFG. All section and table references refer to the revised paper.

We identified a bug in our code that led to incorrect training-time measurements for REPA-E, and we apologize for this error. We have since fixed the bug and repeated the measurements using a revised, more thoroughly documented protocol. The updated Appendix A reports the time spent on each training stage and each component of an E2E step, together with the total training cost (Tables 11-14). The corrected measurements give an estimated 3.05×3.05\times speedup over REPA-E in VAE training, including teacher preparation.

Regarding CFG, we acknowledge that our complete generative system remains behind Self-Transcendence in the matched SD-VAE setting. However, as stated in the Introduction, our method primarily aims to improve VAE latent spaces for downstream generation. We therefore investigated whether this limitation also applies to the learned VAE itself.

We added comparisons of our tuned VAE, SD-VAE, and REPA-E for ImageNet generation (Appendix B, Table 16) and text-to-image generation (Section 4.5, Table 5), evaluating both tasks with and without CFG. In both settings, our tuned VAE improves over SD-VAE and performs comparably to REPA-E with and without CFG. These results show that the CFG limitation of the complete generative system does not extend to the learned VAE.

We address the remaining questions in our individual responses.

Text-to-image generation on COCO (Table 5). gFID of MMDiT trained on frozen VAEs.

Frozen VAEgFID w/o CFGgFID w/ CFG
SD-VAE12.825.92
REPA-E6.624.96
Ours6.764.95

ImageNet generation with frozen VAEs (Table 16). Results for the SiT-B/2 models from Table 1(a)

Frozen VAEgFID w/o CFGgFID w/ CFG
SD-VAE37.026.30
REPA-E20.503.69
Ours22.534.01

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.