- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper adress the problem of decoders being trained on denoising tasks and being drop-in solutions for generative tasks. The authors motivate their solution by stating that there is mostly non correlation between reconstruction FID and generation FID. For this they present a solution to adapt the decoder for the generation task by fine-tuning it in a Generative-Aware Reconstruction (GAR) task, where the image is encoded into a latent space and then its perturbed by some noise and then decoded. The paper presents analysis on the latent distribution mismatch and decoder behaviour finishing with experiments that show how GAR improves generation. The main contributions of the paper are: identifying the problem of using reconstruction decoder in a generative task, introduce GAR as a method that constructs a trajectory from reconstruction to generation.
Critical strengths
- The motivation of the problem and solution are easy to follow and well explained.
- The paper shows that using decoders from reconstruction can hurt the performance of generative models and how it can be easily prevented improving FID of the images generated.
- The in-depth analysis (Section 3 and 4) present results that corroborate with the motivation showing that the problem is impacting generative tasks.
Critical weaknesses
- Despite being informative and explaining the problem, the initial analysis occupy most of the main pages making the final experiments rely mostly on the Annex.
- The decoder adaption could be presented with more depth on ablations such as training with different noise levels and presenting failure cases and possible justifications for them.
Questions for authors
- The work focuses on adapting the decoder with a fixed range of noise levels (0.25-0.45). What would happen if the adaption is done with lower ranges of noise like 0.10-0.25 or only with high levels of noise?
- What do you think enables the decoder to perform better on examples such as Fig.14 but then struggling on samples such as Fig.17? It would be interesting to have a failure case analysis to understand if the problem still related with the capability of decoding latents produced during the generation.
Suggestions:
- In Figure 3 change one of the legend for latents from diagonal left to diagonal right, by doing that the overlap is much easier to understand.
- Figure 3 and Figure 5 captions should have more vertical space so it is easier to read and distinguish from the main text
- The arrows inside table 2 should follow the header by being green and pointing down, otherwise it is too difficult to understand.
5 comments
Response to Reviewer FikZ (2/3)
(2) Failure cases
We observe three cases where DA does not help or comes at a cost. In each, the outcome depends on which latent distribution the decoder is trained on.
To examine the first two, we added a new experiment in the revision. On iMF-B/2, we follow one-epoch CFG-aware DA and keep the objectives, discriminator, optimizer, and number of decoder updates fixed, changing only the latents used to train the decoder: clean encoder latents, encoder latents with added noise but no generator denoising (), or GAR latents denoised by the generator. We report gFID under the OpenAI protocol:
| Training latents (1 epoch) | gFID w/ CFG ↓ | gFID w/o CFG ↓ |
|---|---|---|
| None (w/o DA) | 3.47 | 16.41 |
| Clean () | 3.74 | 16.21 |
| Directly perturbed, | 9.91 | 21.68 |
| GAR (DA), | 3.01 | 13.21 |
- Directly perturbed latents. With the same noise range as DA, direct noise worsens gFID, whereas GAR latents improve it. Direct noise pushes the latents toward an isotropic Gaussian, not toward the generation-time distribution, so the decoder is trained on perturbations it does not see during generation.
- Clean latents. Fine-tuning on clean encoder latents worsens gFID with CFG and gives only a small gain without CFG. Without generation-time latents, the fine-tuning objectives alone do not improve generation.
- Reconstruction on encoder latents. DA lowers standard reconstruction quality: on iMF-B/2, FID on encoder latents rises from 0.73 to 0.85, while FID on latents closer to the generation-time distribution improves, for example from 4.38 to 3.10 at (Table 17). The adapted decoder works better on generation-time latents and slightly worse on encoder latents.
These results are summarized in Section 8 (Table 5), and the full results, including fixed noise levels from 0.1 to 1.0 for directly perturbed latents, are given in Appendix O (Table 21).
R 3.3 The work focuses on adapting the decoder with a fixed range of noise levels (0.25-0.45). What would happen if the adaption is done with lower ranges of noise like 0.10-0.25 or only with high levels of noise?
Thank you for this question. In addition to the fixed-noise ablation without CFG in R 3.2, we ran CFG-aware DA with different noise ranges. All settings follow our main CFG-aware DA protocol (one epoch, GAR latents generated with CFG) and change only the noise range. We report gFID with CFG under the OpenAI protocol:
| Noise range | iMF-B/2 | iMF-M/2 | iMF-L/2 | iMF-XL/2 |
|---|---|---|---|---|
| w/o DA | 3.47 | 2.40 | 1.90 | 1.80 |
| 3.50 | 2.38 | 1.84 | 1.85 | |
| 3.41 | 2.32 | 1.85 | 1.81 | |
| (default) | 3.01 | 2.19 | 1.70 | 1.61 |
| 3.04 | 3.21 | 2.98 | 1.98 |
- Low noise. With , the GAR latents stay close to the encoder distribution and give the decoder little exposure to generation-time latents. The effect is small and mixed: gFID improves slightly on iMF-M/2 and iMF-L/2 but is slightly worse on iMF-B/2 and iMF-XL/2. Extending the range to gives small gains at most scales (e.g., 3.47 → 3.41 on iMF-B/2), still far from the default range.
- High noise only. With , gFID on iMF-B/2 stays close to the default range (3.04), but becomes worse than the original decoder on iMF-M/2, iMF-L/2, and iMF-XL/2. This is another failure case of DA: at high noise levels, the GAR latents keep too little correspondence with their source images, so the paired supervision no longer matches the content in the latents.
Compared with the setting without CFG in R 3.2, where a fixed worked best, the useful range is lower with CFG. This is why our default range , which gives the best results here at all four scales, was chosen for CFG-aware DA. These results have been added to Section 7 (Table 4).
Response to Reviewer FikZ (3/3)
R 3.4 What do you think enables the decoder to perform better on examples such as Fig.14 but then struggling on samples such as Fig.17? It would be interesting to have a failure case analysis to understand if the problem still related with the capability of decoding latents produced during the generation.
Thank you for this question. In Figures 13-24, the two samples in each column are decoded from the same generated latent ; only the decoder parameters differ, before and after DA. DA therefore changes how a given is rendered, but not what encodes. We see two sources of error:
- Decoder-side errors (e.g., Fig. 14). In samples such as the agama in Fig. 14, the generated latent already encodes a plausible image, and the remaining artifacts come mainly from decoding, such as smoothed textures and inconsistent fine details. These are the errors DA targets, so the adapted decoder produces sharper and more coherent details.
- Generator-side errors (e.g., Fig. 17). In the weaker samples of Fig. 17 (golden retriever), structural distortions, such as irregular faces or body shapes, appear in both the original and the adapted decoding of the same . These errors are therefore more likely to come from the generated latent itself.
The training setup of DA explains why it can fix the first kind of error but not the second. DA is trained on GAR latents with , where the global structure is still largely determined by the source image (Figure 8), so the paired supervision mainly teaches the decoder local appearance. Global structure is formed at higher noise levels, where the latents keep too little correspondence with their source images for paired supervision to work, as shown in R 3.2 and R 3.3. DA can thus refine local appearance but is unlikely to correct global structure.
For these failure cases, we therefore think the remaining problem lies mainly in the generated latents rather than in the decoder's ability to decode them. Such errors are better addressed by improving the generative model, which is complementary to DA (Section 5). We have added this discussion to Appendix L.
R 3.5 In Figure 3 change one of the legend for latents from diagonal left to diagonal right, by doing that the overlap is much easier to understand. Figure 3 and Figure 5 captions should have more vertical space so it is easier to read and distinguish from the main text. The arrows inside table 2 should follow the header by being green and pointing down, otherwise it is too difficult to understand.
Thank you for pointing these out. We have revised the figures and table according to your suggestions.
Please let us know if you have any remaining questions.
Response to Reviewer FikZ (1/3)
We appreciate that the reviewer found our motivation and solution clear and easy to follow, recognized that reconstruction-trained decoders can hurt generative performance and that decoder adaptation readily improves generation FID, and found that our in-depth analyses in Sections 3 and 4 corroborate this motivation. Thank you for taking the time to review our paper and for your thoughtful feedback. We address your questions and concerns below.
Thank you for this suggestion. Sections 3 and 4 establish the latent distribution mismatch, show that the decoder amplifies it, verify that GAR moves decoder inputs from Pe toward Pg, and relate this shift to generation quality across 13 tokenizers. Decoder adaptation (DA) builds directly on these findings, so we gave them most of the main pages, and several DA results ended up in the Appendix.
We agree that the experiments should be more visible in the main text. Within the page limit, the revision adds the following DA experiments to the main text:
Section 7 also points to the remaining analyses in the Appendix, including the system-level comparison (Table 18) and the failure-case analysis (Appendix L).
Thank you for this suggestion.
(1) Noise-level ablation
This ablation is already included in the original submission, in Table 13 of Appendix L (Table 16 in the revision), which we may not have pointed to clearly enough from the main text. It varies only the noise level used for decoder adaptation (DA), fixed at ηt∈{0.0,…,0.6}, under the DA w/o CFG setting (GAR latents without CFG, 10 epochs). We report gFID w/o CFG (↓) under the OpenAI protocol:
The trend is the same at all four scales. As ηt increases, the training latents move from the encoder distribution toward the generation-time distribution, and gFID improves up to ηt=0.5. At ηt=0.6, gFID is worse than at 0.5 at every scale, because the latents keep too little correspondence with their source images for the paired supervision to stay reliable. rFID shows this directly: on iMF-B/2, it rises from 1.20 at ηt=0.5 to 1.81 at ηt=0.6.