acceptodds
Under review as a conference paper at ICLR 2027

Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling

Abstract

Latent generative models typically train a VAE for reconstruction and then fit a generative model on the frozen latent space. Since reconstruction-optimized latents are not necessarily generation-friendly, jointly training both models is an appealing alternative. However, direct end-to-end training remains challenging, as it is prone to latent collapse and faces a generation–reconstruction conflict. We revisit this problem and identify two key insights. First, the entropy term in the Kullback-Leibler divergence (KL) objective is essential for preventing collapse. Second, reconstruction and generation exhibit asymmetric learning dynamics: reconstruction is fast and strongly supervised, whereas generation is slower and harder to optimize. Based on these observations, we introduce an entropy-preserving end-to-end objective for stable joint training and propose GenFirst, a simple generation-before-reconstruction strategy that first shapes a generation-friendly latent space and then refines reconstruction. GenFirst generalizes across exact-likelihood continuous autoregressive priors and SiT-based flow-matching priors. With our end-to-end objective and GenFirst, SiT achieves a gFID of with CFG and without CFG on ImageNet-, while MMDiT reaches a GenEval score of on text-to-image generation. We further extend the framework to shared visual latents for generation, representation learning, and reconstruction, as well as continuous text latents for unified text–image generation, demonstrating its applicability across generative priors, objectives and modalities.

Primary area:
generative models
Keywords:
end-to-end, variational autoencoder, image generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hDXcOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper targets the task of latent generative modeling, where image generation is achieved in two steps:

  1. learn a VAE to compress image data into a compact latent space
  2. Fit a generative model in the latent space, to learn the latent distribution and allow meaningful ancestral sampling

While foundation papers in the field approached the two stages separately (learning the VAE first, then freeze it and learn the prior), the paper proposes an end-to-end solution. The need for an end-to-end solution is motivated by recent observations that a latent spaced learned for reconstruction (stage-1) is often sub-optimal for generation, complicating the task for stage-2 and preventing overall better results.

The authors start from a "naive" end-to-end objective, which simply sum the two stage terms into a unique loss, and show why that solution leads to collapse (equation 3 of the paper and short discussion). They further propose to remove the KL term, and in particular the prior p0p_0 fitting term, while keeping an entropy term whose weight can be readjusted. This solution results in the loss of equation 4, which is capable of preventing latent collapse, as experiments show.

The paper continues by observing that the new loss now has a balancing problem, and in particular in the weight λprior\lambda_\text{prior}. As visible in Figure 5(a), they recover an empirical trade-off between reconstruction and generation capability (high λprior\lambda_\text{prior} achieves better generation but worse distortion and vice-versa).
After exploring various empirical solutions (Sections 3.2 and 3.3) they find a viable way to tame such coefficient: start with a high value, to favor generation in first iterations, then lower it to enhance reconstruction quality. They motivate this solution as "treating the conflict as an optimization-order rather than a loss-balancing problem. GenFirst first shapes a generation-friendly latent space and then strengthens reconstruction to recover details."

As for priors, the authors propose two main alternatives in Section 4. An exact likelihood solution, which forms the main model, and a velocity field (Equation 6) as an alternative.

Experimentally, the method is tested on various baselines, consisting in both class conditioned and text-to-image generation baselines, showing overall credible to good results with respect to the main baselines.

Critical strengths

The greater strength in the paper is the proposed objective in Equation (4), and only partially the discussion that motivates it. The objective itself is a good contribution, and can be a starting point for a much stronger paper. The entropy term has a precise role in the loss function and the authors summarize it correctly with the bold sentence: "In this view, the entropy term is not a minor regularizer, but rather provides a counteracting force to learned prior fitting and prevents latent collapse"

Critical weaknesses

The proposed Equation 4 and the GenFirst method are likely a correct direction to follow, but they are both presented without enough justifications. The discussion that surrounds the motivations for the entropy term is preliminary and not deep enough, while the balancing of the λprior\lambda_\text{prior} term is presented empirically rather than mathematically.

This lack of sound justifications is particularly evidenced by the following points:

  1. The choice of setting λprior=λent\lambda_\text{prior} = \lambda_\text{ent} is not justified in the main paper.
  2. A deeper discussion on why λprior\lambda_\text{prior} sets a tradeoff between reconstruction and generation in Equation (4) is missing, as the method is presented only empirically.
  3. It remains unclear why the authors choose not to vary the reconstructions weights, which have an important role in the loss picture.

These points are related together by a mathematical interpretation of the objective itself, which the author miss and can likely be derived using basic mathematical tricks and known results of VAE literature only (references at the end). In the following, I have tried to sketch it.

The derivation is based on a decomposition of Lprior\mathcal{L}_\text{prior}, which in principle can be applied also to the naive end-to-end objective:

= \mathbb{E}_{x}\\,\mathbb{E}_{z \sim q_{\phi}(z|x)}[-\log p_{\theta}(z)]$$ Step 1: add and subtract $\log q_{\phi}(z|x)$: $$\mathcal{L}_{\text{prior}} = \mathbb{E}_{x}\\,\mathbb{E}_{z \sim q_{\phi}(z|x)} [\log q_{\phi}(z|x) - \log p_{\theta}(z) - \log q_{\phi}(z|x)]$$ Step 2: divide the expectations into two terms: $$\mathcal{L}_{\text{prior}} = \mathbb{E}_{x}\\,\mathbb{E}_{z \sim q_{\phi}(z|x)} [\log q_{\phi}(z|x) - \log p_{\theta}(z)] + \mathbb{E}_{x}\\,\mathbb{E}_{z \sim q_{\phi}(z|x)} [-\log q_{\phi}(z|x)]$$ Step 3: the two quantities are now well defined as: $$\mathcal{L}_{\text{prior}} = \mathbb{E}_{x} D_{\text{KL}}(q_{\phi}(z|x) \\| p_{\theta}(z)) + \mathbb{E}_{x}\\,\mathcal{H}(q_{\phi}(z|x))$$ Final step: apply the result of [1] to the KL term, which recovers $$\mathcal{L}_{\text{prior}} = I_{q_{\phi}}(\mathbf{X}; \mathbf{Z}) + D_{\text{KL}}(q_{\phi}(\mathbf{z}) \\| p_{\theta}(\mathbf{z})) + \mathbb{E}_{x}\\,\mathcal{H}(q_{\phi}(z|x))$$ With $\mathcal{L}_{\text{prior}}$ in this form, the full Equation 4 loss writes: $$\mathcal{L}_{\text{E2E}} = \lambda_{\text{rec}}\mathcal{L}_{\text{rec}} + \lambda_{\text{LPIPS}}\mathcal{L}_{\text{LPIPS}} + \lambda_{\text{GAN}}\mathcal{L}_{\text{GAN}} + \lambda_{\text{prior}} I_{q_{\phi}}(\mathbf{X}; \mathbf{Z}) + \lambda_{\text{prior}} D_{\text{KL}}(q_{\phi}(\mathbf{z}) \\| p_{\theta}(\mathbf{z})) + (\lambda_{\text{prior}} - \lambda_{\text{ent}}) \mathbb{E}_{x}\\!\left[\mathcal{H}(q_{\phi}(\mathbf{z}|\mathbf{x}))\right].$$ Such derivation sets the ground for a stronger explanation of the paper. First, note that in the two stage setup, $q_\phi$ is fixed, so $\mathcal{L}_\text{prior}$ is optimizing the second term only (under $\theta$), which is the one relevant for generation. Second, the naive E2E objective unfreezes $q_\phi$, making the training unstable, since term 1 and 3 come into play from the beginning. The KL term of Equation 3 is insufficient to balancing these forces, as its weight is too small for the entropy term and the mutual information remains untamed. Therefore, the authors identify the correct direction of removing that term, while adding an entropy term alone (Equation 4 of the paper). Its role is to counterbalance the entropy force introduced by the prior loss, and the $\lambda_\text{prior} = \lambda_\text{ent}$ acquires a sound justification (which the authors don't discuss). A sound justification is also given to the role of $\lambda_\text{prior}$: high value adds pressure to the KL term between the aggregate posterior and learnable prior, which is the goal of the generative model (generation improves), but **at the same time** adds pressure to minimize the Information Rate term [2], which constraints rate and makes reconstruction harder. The opposite happens if $\lambda_\text{prior}$ is too low. Therefore, the interpretation of GenFirst itself becomes: at early phases of training, it keeps high pressure on MI term, learning a generative model in low rate region of the Rate--Distortion plane [2]. Then, low rate pressure is released, moving in RD regions more favorable for distortion, while making generation harder. In light of this, the main point in favor of GenFirst is not merely to: "explicitly separates latent-space formation from reconstruction refinement", as authors propose. The real point is that the $\lambda_\text{prior}$ coefficient has the impossible role of balancing two terms that want opposite things. GenFirst proposes one possible way to counter-balance this fact. However, this does not explain why e.g. a cosine decay is not working as well, and further experiments are needed to shed light on this fact. On the other hand, the choice of keeping the reconstruction terms coefficients unbalanced is a completely unexplored direction, with no apparent motivation. A possible solution could be to keep $\lambda_\text{prior}$ fixed (e.g. at 1.0, favoring good prior fitting) while increasing the Distortion term to counter-balance the rate coefficient (i.e. $\lambda_\text{prior}$ itself) on the MI term. That could be done by tuning $\lambda_\text{rec}$, $\lambda_\text{LPIPS}$ and $\lambda_\text{GAN}$, which likely need separate re-balancing. With this direction unexplored, it is difficult to judge the goodness of the proposed GenFirst solution. Overall, these observations make me conclude that the paper is going into the right direction with Equation (4), but a substantial re-thinking with additional justifications, discussion and experiments is necessary before reaching the acceptance threshold. Core references: - [1] Hoffman, Matthew D., and Matthew J. Johnson. "Elbo surgery: yet another way to carve up the variational evidence lower bound." Workshop in advances in approximate Bayesian inference, NIPS. Vol. 1. No. 2. 2016. - [2] Alemi, Alexander, et al. "Fixing a broken ELBO." International conference on machine learning. PMLR, 2018.
Questions for authors

I have no specific questions for the authors. Note that the derivation in the weaknesses section is only a sketch, and needs further sharpening and reasoning. In any case, I remain open to engage in further discussions with the authors on these points, hoping to be helpful in improving their work.

Reviewer_FDkSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes stable joint training of VAE and latent generative models. The proposed idea combines an explicit posterior-entropy term with a two-stage schedule that induces learning generation early and reconstruction later. To be specific, the first stage makes the prior relatively strong influence over the encoder and the second stage reduces the prior and entropy weights. This framework is demonstrated with both exact-likelihood continuous autoregressive prior and flow matching prior.

Critical strengths
  1. Comprehensive study on practical joint-training failure
  • This study shows an imbalance between prior fitting and posterior entropy preservation as a root of collapse in naive end-to-end training. It gives a useful explanation of the observed failure and motivates reformulation of total loss objective. Also, the analysis in this paper helps connect the training collapse to specific loss components rather than treating it as a general limitation of end-to-end training.
  1. Strong empirical evidences that jointly learned latent benefits the performance of generation from both design ablations and benchmark comparisons.
  • This paper identifies design choices and evaluates both autoregressive and flow-matching models. The experimental results show impressive generation performance, including both ImageNet-256 and ImageNet-512. Together, the ablations and broader comparisons demonstrate the effectiveness of the proposed approach
Critical weaknesses
  1. Clarification between optimization ordering and loss balancing is required
  • Section 3.3 describes GenFirst as addressing an optimization-order problem rather than a loss-balancing problem. However, in Section 4.2, it implements this ordering by changing the relative loss weights in two stages. From my understanding, GenFirst is therefore a particular loss-weight schedule motivated by the relative learning speeds of reconstruction and generation. Therefore, It would be helpful to clarify whether "optimization ordering" refers to the motivation behind this schedule, or whether the authors intend a distinction beyond loss scheduling.
Questions for authors
  1. Can the generation quality gains be maintained at similar reconstruction quality?
  • Can this approach improve gFID while retaining similar rFID under the same total training budget? A reconstruction-generation trade-off curve would clarify whether both generation quality and reconstruction quality can be achieved at the same time or not.
Reviewer_se8wOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how to stably end-to-end train both the VAE latent tokenizer and the generative model for image generation. They provide two interesting insights and respective design choices for how to make the end-to-end training stable:

  • The KL regularizer in the VAE can be decomposed to a prior-fitting term (that encourages the sampled latents to match the Gaussian prior) and a negative posterior-entropy term, that preserves posterior uncertainty and prevents latent collapse. Because the KL term in VAE has typically a very low loss weight, the entropy term remains essentially suppressed, while the generative objective brings another strong prior-fitting force, leading to latent collapse. Based on this, the authors propose to keep only the entropy-term but with a much higher loss weight.
  • During end-to-end training there is a generation-reconstruction trade-off and because the reconstruction objective is learnt much faster than the generation objective, the latent space is shaped to be more reconstruction-oriented and less "friendly" to generation. Based on that, they proposed a Generation-first strategy, where during the beginning of the training the generation objective has much higher loss weight.

The authors evaluate their approach with continuous autoregressive and SiT models on class-conditional image generation (on ImageNet) and with MMDiT on text-to-image generation, presenting strong results.

Critical strengths
  • The work offers valuable insights into the impact of the loss terms and learning dynamics in the end-to-end training of latent generative models. Based on these insights, they propose simple techniques that enable stable end-to-end training without requiring additional REPA loss terms or complex gradient-detachment schemes. I believe this work would be valuable to the community.
  • The proposed approach is extensively evaluated and shown to lead to strong empirical results, surpassing many prior approaches, such as REPA-E.
Critical weaknesses
  • From the results in Table 1 it seems that end-to-end approach is not adequate for achieving good gFID results for SiT. The end-to-end training with SiT achieves 92.38 gFID, which is very poor. How is this poor gFID result explained? In contrast, in REPA-E, end-to-end training from scratch for 80 epochs achieves 4.34 (see Table 7 of REPA-E); so REPA-E does not depends from an additional training stage with the VAE frozen as the current work does. This seems to break the "story" of end-to-end training in the case of SiT. Also, if I understood correctly, the analyses presented in Figure 2-5, based on which the insights are extracted, are specific to the continuous AR model only. So, it is unclear if the same observations / insights also hold for SiT and flow-matching models in general. Overall, my impression is that the study and the proposed end-to-end training strategy is more specific to the continuous AR model. For flow-matching models, which are the standard models for image generation, it is unclear whether the same insights hold and to achieve strong / good results it requires several "hacks" like the additional training stage with the VAE frozen (which based in Table 18 is around 10x longer than the end-to-end training) plus, for Table 5, an additional stage of training only the VAE decoder.
  • Minor weakness: there is no clear "recipe" for how to allocate training epochs / iterations between the the "Generation-First" and "Reconstruction-refinements" stages.
Questions for authors
  • How do you explain the poor gFID results when only using end-to-end training for the SiT model? Why is it necessary to have an additional frozen VAE stage in the SiT (which is not the case for REPA-E)?
  • The insights extracted from Figures 2-5 (based on the continious AR model), do they also hold in the case of SiT?
  • What is the proposed "recipe" for allocating training epochs / iterations between the the "Generation-First" and "Reconstruction-refinements" stages.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.