acceptodds
Withdrawn from ICLR 2027

Exploring the Design Space of VAEs for Diffusion-Based Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Latent diffusion models often retain VAE encoder architectures introduced by early LDM systems, despite significant evolution in both decoders and diffu- sion backbones. We revisit the encoder from an efficiency perspective, and study how the computation should be allocated with regard to operations and resolution stages. We introduce ABCRP and LatentMaid: encoders built from pixel unshuffle, RMSNorm, and a depthwise + SwiGLU residual block. The design keeps spatial effects in the decoder and uses only local, lightweight components in the encoder, explicitly restricting cross-spatial dependencies in the encoder without requiring an auxiliary locality loss. At CFG=1, ABCRP with standard residual blocks achieves the best generation quality in our study (17.68 gFID), while LatentMaid achieves 18.32 gFID with 89% fewer encoder FLOPs than the SD-VAE baseline and increasing measured encoder throughput by approximately 3×, providing a substantially more efficient operating point. All code has been uploaded as supplementary material - available for testing and reproduction.

Primary area:
generative models
Keywords:
Diffusion, VAE, AE, LDM, latent diffusion models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5NUkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors analyse the receptive field (set of pixels) influencing each token, and replace components of the encoder by more spatially-constrained components, both reducing the receptive fields and improving encoder efficiency. They show that a local-biased encoder improves both rFID and gFID.

Critical strengths

The authors provide extensive experiments to study the impact of their design choices on rFID and gFID, providing clear justification for their design choices.

Critical weaknesses
  • The main concern about encoders for latent diffusion is the quality of the latent space for latent generative models, and as such, the gFID obtained with such models. The experiments in this direction are very weak: the paper only uses under-trained DiT models. Comparisons using fully-trained DiT, and possibly a more modern Flow-Matching baseline (SiT) would greatly strengthen the results. Additionally, evaluation without CFG doesn’t correlate with the actual quality of the model, and results should be reported with an optimal-CFG evaluation sweep. Results from Table 7 seem to indicate that, under per-setting optimal CFG, the SD-VAE baseline beats the proposed method.
  • The concern about the encoder efficiency seems mostly irrelevant in the studied setting (latent diffusion), as the decoder is used once a dataset for training, and never at sampling time. With this consideration, the second part of the method (LatentMaid) is unjustified and superfluous as it is consistently outperformed by the base ABCRP.
Questions for authors

Could the author provide proper evaluation of the resulting encoder(s) with complete training, modern latent model (SiT), and more importantly, with per-encoder optimal CFG?

Reviewer_94k2OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper explores the design space of VAE encoders for latent diffusion models, focusing on reducing computational cost while maintaining downstream generation quality. The authors introduce ABCRP and LatentMaid, which simplify the encoder through locality-preserving operations, lightweight residual blocks, and optimized stage allocation.

Critical strengths

The paper provides a relatively comprehensive study of VAE encoder architectures, examining the effects of self-attention, normalization, downsampling, residual blocks, and stage allocation on downstream diffusion generation.

Critical weaknesses
  1. Limited practical significance. In modern latent diffusion systems, the DiT backbone (often billions of parameters) is substantially larger than the VAE encoder. Moreover, VAE latents are commonly precomputed for DiT training, making encoder efficiency less critical. Improving the latent compression ratio would have a more direct impact on diffusion training and inference costs. However, this paper primarily focuses on reducing encoder FLOPs while retaining the relatively low f8c4 compression setting.

  2. Insufficient comparisons and uncompetitive generation quality. The paper mainly compares against the conventional SD-VAE and its own architectural variants, without sufficient comparisons to stronger modern VAEs. For example, VA-VAE with LightningDiT achieves approximately 5.14 gFID without CFG after only 64 epochs, while using a higher spatial compression ratio (f16 versus f8), compared to 17.68 gFID after approximately 80 epochs in this paper. Although the training configurations differ, this substantial gap raises concerns about competitiveness, especially given more recent approaches such as RAE, RAEv2, and GenFirst.

Questions for authors
  1. Practical efficiency. Given that VAE latents are commonly precomputed and cached during DiT training, could the authors clarify the practical benefits of reducing encoder FLOPs compared to improving the latent compression ratio?

  2. Comparison with stronger baselines. Could the authors provide comparisons with modern VAE architectures such as VA-VAE, RAE, and RAEv2 under comparable DiT backbones and training budgets? How do the authors explain the substantial generation quality gap despite using a lower spatial compression ratio?

  3. Effectiveness under CFG. According to Table 7, the improvement over the SD-VAE baseline becomes marginal at CFG=2 (4.74 vs. 4.80 gFID). Could the authors clarify the marginal improvement?

Reviewer_3SFMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper empirically refines VAE encoder architectures to reduce computational cost, following the principle of limiting unnecessary spatial dependencies. The experiments show reduced parameter counts and computation while maintaining competitive generation performance for selected configurations.

Critical strengths

The experiments show reduced parameter counts and computation while maintaining competitive generation performance for selected configurations.

Critical weaknesses
  1. The paper is not clear enough in its title, abstract and introduction about the basic motivation, the essential mechanisms of the proposed ABCRP and LatentMaid (particularly in abstract) , and their distinction from related work.

  2. The paper reads more like an empirical exploration that leads to a final framework, with some refinements not sufficiently motivated or justified. Although empirical exploration is valuable, some connections between the design choices and their claimed benefits remain unclear. 2.1 — Removing self-attention. The author motivates this in 4.2 by saying that it leads to performance upgrade in experiments. However, the authors’ own results do not consistently support this, i.e., see Table 2. This makes the reviewer question how broadly this design choice applies across different model scales and configurations. 2.2 — Replacing spatial normalization. The authors use per-token RMSNorm to avoid the spatial coupling introduced by computing mean and variance across spatial positions in GroupNorm. This structural argument is reasonable, but how does removing this coupling contribute to better generation performance, and what computational savings does this particular change provide?

Similarly for the other module refinement.

  1. The paper's core contribution should be proposing a lightweight VAE encoder, but without sufficient discussion to related papers. It also lacks simulation comparisons with prior methods and baselines at similar computational budgets.

  2. The paper needs substantial rewriting to improve its organization and readability. There are several abrupt transitions and justifications whose connection to the motivation or proposed methods is unclear.

  3. Fig. 2, especially (a) and (b), is not clear enough about what the authors want to convey. The two panels look largely the same except for the exchanged colors, yet the authors do not explicitly explain what each color represents.

Questions for authors
  1. The paper is not clear enough in its abstract and introduction about the basic motivation, the essential mechanisms of the proposed ABCRP and LatentMaid (particularly in abstract) , and their distinction from related work.

  2. The paper reads more like an empirical exploration that leads to a final framework, with some refinements not sufficiently motivated or justified. Although empirical exploration is valuable, some connections between the design choices and their claimed benefits remain unclear. 2.1 — Removing self-attention. The author motivates this in 4.2 by saying that it leads to performance upgrade in experiments. However, the authors’ own results do not consistently support this, i.e., see Table 2. This makes the reviewer question how broadly this design choice applies across different model scales and configurations. 2.2 — Replacing spatial normalization. The authors use per-token RMSNorm to avoid the spatial coupling introduced by computing mean and variance across spatial positions in GroupNorm. This structural argument is reasonable, but how does removing this coupling contribute to better generation performance, and what computational savings does this particular change provide?

Similarly for the other module refinement.

  1. The paper's core contribution should be proposing a lightweight VAE encoder, but without sufficient discussion to related papers. It also lacks simulation comparisons with prior methods and baselines at similar computational budgets.

  2. The paper needs substantial rewriting to improve its organization and readability. There are several abrupt transitions and justifications whose connection to the motivation or proposed methods is unclear.

  3. Fig. 2, especially (a) and (b), is not clear enough about what the authors want to convey. The two panels look largely the same except for the exchanged colors, yet the authors do not explicitly explain what each color represents.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.