- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper targets the task of latent generative modeling, where image generation is achieved in two steps:
- learn a VAE to compress image data into a compact latent space
- Fit a generative model in the latent space, to learn the latent distribution and allow meaningful ancestral sampling
While foundation papers in the field approached the two stages separately (learning the VAE first, then freeze it and learn the prior), the paper proposes an end-to-end solution. The need for an end-to-end solution is motivated by recent observations that a latent spaced learned for reconstruction (stage-1) is often sub-optimal for generation, complicating the task for stage-2 and preventing overall better results.
The authors start from a "naive" end-to-end objective, which simply sum the two stage terms into a unique loss, and show why that solution leads to collapse (equation 3 of the paper and short discussion). They further propose to remove the KL term, and in particular the prior fitting term, while keeping an entropy term whose weight can be readjusted. This solution results in the loss of equation 4, which is capable of preventing latent collapse, as experiments show.
The paper continues by observing that the new loss now has a balancing problem, and in particular in the weight . As visible in Figure 5(a), they recover an empirical trade-off between reconstruction and generation capability (high achieves better generation but worse distortion and vice-versa).
After exploring various empirical solutions (Sections 3.2 and 3.3) they find a viable way to tame such coefficient: start with a high value, to favor generation in first iterations, then lower it to enhance reconstruction quality. They motivate this solution as "treating the conflict as an optimization-order rather than a loss-balancing problem. GenFirst
first shapes a generation-friendly latent space and then strengthens reconstruction to recover details."
As for priors, the authors propose two main alternatives in Section 4. An exact likelihood solution, which forms the main model, and a velocity field (Equation 6) as an alternative.
Experimentally, the method is tested on various baselines, consisting in both class conditioned and text-to-image generation baselines, showing overall credible to good results with respect to the main baselines.
Critical strengths
The greater strength in the paper is the proposed objective in Equation (4), and only partially the discussion that motivates it. The objective itself is a good contribution, and can be a starting point for a much stronger paper. The entropy term has a precise role in the loss function and the authors summarize it correctly with the bold sentence: "In this view, the entropy term is not a minor regularizer, but rather provides a counteracting force to learned prior fitting and prevents latent collapse"
Critical weaknesses
The proposed Equation 4 and the GenFirst method are likely a correct direction to follow, but they are both presented without enough justifications. The discussion that surrounds the motivations for the entropy term is preliminary and not deep enough, while the balancing of the term is presented empirically rather than mathematically.
This lack of sound justifications is particularly evidenced by the following points:
- The choice of setting is not justified in the main paper.
- A deeper discussion on why sets a tradeoff between reconstruction and generation in Equation (4) is missing, as the method is presented only empirically.
- It remains unclear why the authors choose not to vary the reconstructions weights, which have an important role in the loss picture.
These points are related together by a mathematical interpretation of the objective itself, which the author miss and can likely be derived using basic mathematical tricks and known results of VAE literature only (references at the end). In the following, I have tried to sketch it.
The derivation is based on a decomposition of , which in principle can be applied also to the naive end-to-end objective:
= \mathbb{E}_{x}\\,\mathbb{E}_{z \sim q_{\phi}(z|x)}[-\log p_{\theta}(z)]$$ Step 1: add and subtract $\log q_{\phi}(z|x)$: $$\mathcal{L}_{\text{prior}} = \mathbb{E}_{x}\\,\mathbb{E}_{z \sim q_{\phi}(z|x)} [\log q_{\phi}(z|x) - \log p_{\theta}(z) - \log q_{\phi}(z|x)]$$ Step 2: divide the expectations into two terms: $$\mathcal{L}_{\text{prior}} = \mathbb{E}_{x}\\,\mathbb{E}_{z \sim q_{\phi}(z|x)} [\log q_{\phi}(z|x) - \log p_{\theta}(z)] + \mathbb{E}_{x}\\,\mathbb{E}_{z \sim q_{\phi}(z|x)} [-\log q_{\phi}(z|x)]$$ Step 3: the two quantities are now well defined as: $$\mathcal{L}_{\text{prior}} = \mathbb{E}_{x} D_{\text{KL}}(q_{\phi}(z|x) \\| p_{\theta}(z)) + \mathbb{E}_{x}\\,\mathcal{H}(q_{\phi}(z|x))$$ Final step: apply the result of [1] to the KL term, which recovers $$\mathcal{L}_{\text{prior}} = I_{q_{\phi}}(\mathbf{X}; \mathbf{Z}) + D_{\text{KL}}(q_{\phi}(\mathbf{z}) \\| p_{\theta}(\mathbf{z})) + \mathbb{E}_{x}\\,\mathcal{H}(q_{\phi}(z|x))$$ With $\mathcal{L}_{\text{prior}}$ in this form, the full Equation 4 loss writes: $$\mathcal{L}_{\text{E2E}} = \lambda_{\text{rec}}\mathcal{L}_{\text{rec}} + \lambda_{\text{LPIPS}}\mathcal{L}_{\text{LPIPS}} + \lambda_{\text{GAN}}\mathcal{L}_{\text{GAN}} + \lambda_{\text{prior}} I_{q_{\phi}}(\mathbf{X}; \mathbf{Z}) + \lambda_{\text{prior}} D_{\text{KL}}(q_{\phi}(\mathbf{z}) \\| p_{\theta}(\mathbf{z})) + (\lambda_{\text{prior}} - \lambda_{\text{ent}}) \mathbb{E}_{x}\\!\left[\mathcal{H}(q_{\phi}(\mathbf{z}|\mathbf{x}))\right].$$ Such derivation sets the ground for a stronger explanation of the paper. First, note that in the two stage setup, $q_\phi$ is fixed, so $\mathcal{L}_\text{prior}$ is optimizing the second term only (under $\theta$), which is the one relevant for generation. Second, the naive E2E objective unfreezes $q_\phi$, making the training unstable, since term 1 and 3 come into play from the beginning. The KL term of Equation 3 is insufficient to balancing these forces, as its weight is too small for the entropy term and the mutual information remains untamed. Therefore, the authors identify the correct direction of removing that term, while adding an entropy term alone (Equation 4 of the paper). Its role is to counterbalance the entropy force introduced by the prior loss, and the $\lambda_\text{prior} = \lambda_\text{ent}$ acquires a sound justification (which the authors don't discuss). A sound justification is also given to the role of $\lambda_\text{prior}$: high value adds pressure to the KL term between the aggregate posterior and learnable prior, which is the goal of the generative model (generation improves), but **at the same time** adds pressure to minimize the Information Rate term [2], which constraints rate and makes reconstruction harder. The opposite happens if $\lambda_\text{prior}$ is too low. Therefore, the interpretation of GenFirst itself becomes: at early phases of training, it keeps high pressure on MI term, learning a generative model in low rate region of the Rate--Distortion plane [2]. Then, low rate pressure is released, moving in RD regions more favorable for distortion, while making generation harder. In light of this, the main point in favor of GenFirst is not merely to: "explicitly separates latent-space formation from reconstruction refinement", as authors propose. The real point is that the $\lambda_\text{prior}$ coefficient has the impossible role of balancing two terms that want opposite things. GenFirst proposes one possible way to counter-balance this fact. However, this does not explain why e.g. a cosine decay is not working as well, and further experiments are needed to shed light on this fact. On the other hand, the choice of keeping the reconstruction terms coefficients unbalanced is a completely unexplored direction, with no apparent motivation. A possible solution could be to keep $\lambda_\text{prior}$ fixed (e.g. at 1.0, favoring good prior fitting) while increasing the Distortion term to counter-balance the rate coefficient (i.e. $\lambda_\text{prior}$ itself) on the MI term. That could be done by tuning $\lambda_\text{rec}$, $\lambda_\text{LPIPS}$ and $\lambda_\text{GAN}$, which likely need separate re-balancing. With this direction unexplored, it is difficult to judge the goodness of the proposed GenFirst solution. Overall, these observations make me conclude that the paper is going into the right direction with Equation (4), but a substantial re-thinking with additional justifications, discussion and experiments is necessary before reaching the acceptance threshold. Core references: - [1] Hoffman, Matthew D., and Matthew J. Johnson. "Elbo surgery: yet another way to carve up the variational evidence lower bound." Workshop in advances in approximate Bayesian inference, NIPS. Vol. 1. No. 2. 2016. - [2] Alemi, Alexander, et al. "Fixing a broken ELBO." International conference on machine learning. PMLR, 2018.Questions for authors
I have no specific questions for the authors. Note that the derivation in the weaknesses section is only a sketch, and needs further sharpening and reasoning. In any case, I remain open to engage in further discussions with the authors on these points, hoping to be helpful in improving their work.