- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose GuaVAE, an extension of the existing REPA-E pipeline. Instead of using an external vision encoder such as DINOv2, they use CFG-enhanced features from a frozen pretrained DiT teacher as the alignment target. Additionally, they sample the teacher timestep independently from a cleaner time interval. The tuned VAE improves diffusability for generation over the original VAEs, reaching 4.97 gFID without CFG and 1.62 with CFG on ImageNet-256.
Critical strengths
- The paper writing is clear and easy to follow.
- The tuned VAE consistently improves generation over the original VAEs, e.g., generation FID drops from 37.02 to 22.53 on SD-VAE, and from 39.31 to 19.30 on IN-VAE (Table 1).
- The paper provides detailed ablations on each component of the method, including the teacher timestep, feature guidance, teacher preparation, and alignment weight (Table 3).
Critical weaknesses
- The motivation for avoiding external vision encoders is not clearly stated. GuaVAE does not outperform REPA-E (Table 1, Table 2) and still requires additional teacher training.
- The novelty is limited. The method reads like REPA-E (alignment) + Self-Transcendence (teacher feature) + SRA (cleaner teacher timesteps).
- The training cost is high. Training throughput is about 3.6× slower than REPA-E (Table 10), even when excluding 200K iterations of teacher preparation.
- The abstract and introduction highlight the unguided generation results. However, real use cases for image generation models rely more on guided results. The guided result is worse than Self-Transcendence with SD-VAE (1.62 vs. 1.44, Table 7).
Questions for authors
- Please refer to the weakness section above.
3 comments
Thank you for updating the timing measurements and adding the text-to-image results. It address my concern about VAE training cost. My concerns about novelty and the guided performance still remain. Therefore, I keep my score at 2.
Thank you for your response. We would like to clarify the remaining concern about guided performance. Our primary contribution is a method for improving VAE latent spaces, so we evaluated whether the tuned VAE benefits new generators when using CFG.
Please see the ImageNet and text-to-image results in our comment to all reviewers (Tables 16 and 5). In both settings, generators trained on our frozen VAE outperform those using SD-VAE and achieve quality comparable to REPA-E with CFG. These results demonstrate that the benefits of our VAE tuning extend to guided generation.
Thank you for your careful reading and comments on the motivation and practical relevance of our work. For the corrected training-cost measurements and the discussion of CFG performance, please refer to our response to all reviewers.
We have expanded the motivation for avoiding external semantic encoders in the Introduction. Please see the revised version for this discussion.
To address your concern about guided generation in practical applications, we added text-to-image experiments with and without CFG (Section 4.5, Table 5). Generators trained on our tuned VAE improve over SD-VAE and perform comparably to REPA-E in both settings. Together with the lower VAE-training cost reported in Appendix A, these results demonstrate a practical use of our method beyond the unguided performance of the complete generative system.
Regarding novelty, our method builds on prior alignment approaches, but transferring their benefits from diffusion training to VAE tuning is not straightforward. Using CFG-enhanced teacher features with shared timesteps provides little improvement to the VAE. Table 3(b) separates the effects of independent timestep sampling and restricting the teacher to a low-noise interval, showing that both contribute to the improvement. The resulting design does not require the teacher input to be cleaner than the student's at every update. These findings establish the conditions under which internal diffusion representations become useful for improving the VAE itself.