- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work introduces a method for post-training of a latent decoder model to improve generation quality for latent generative models of image-type data, called FDDA. The key idea is to adapt recent ideas from Frechet Distribution losses as a training objective for generator model post-training into a method for latent decoder post-training. During FDDA, the latent generator is frozen and the decoder is unfrozen. The EMA version of FD post-training is used, and the FD loss only flows through the EMA FD statistics contributed by the current training batch. A suite of 6 different encoder models are used for FD loss in a balanced way to avoid overfitting to the FD distribution of a single encoder. Experiments show that the proposed method can consistently improve generation quality of a variety of pre-trained image generation models both in the case where the original generator is used and only the decoder is post-trained, and in the case where the generator first undergoes FD post-training followed by FD post-training of the decoder with FDDA.
Critical strengths
- This method provides a well-scoped and effective method of improving the quality of latent generator models. The FD-loss methodology is naturally suitable for decoder as well as generator post-training, and the FDDA approach is simple and makes intuitive sense.
- The experimental settings provide convincing evidence that FDDA provides generation benefits across a variety of pre-trained generative models.
Critical weaknesses
- There is no theoretical guarantee that FD loss will lead to better perceptual generative quality or to a closer match of the data distribution. Overfitting to the FD distribution of the different encoders at the expense of visual quality remains a theoretical possibility, although in practice it looks like FD loss does lead to better perceptual quality. This weakness is shared by other works using FD loss for the generator model, etc.
- The gains diminish in the more realistic setting where the generator is first FD post-trained, and the method itself is not especially surprising given the recent trend of FD loss for the generator model.
Questions for authors
My main request is for the visualization in Appendix D. In the cases where generator-side FD and FDDA are both used, could you display three rows: pretrained, generator FD only, and generator FD + FDDA? Without the middle row of generator FD only, it is difficult to assess the improvement of the FDDA component after generator FD.
3 comments
Thanks for the response. I will stick with my score. The proposed FDDA method follows naturally (although not especially surprisingly) from generator-side FD learning and appears to be a useful part of the practitioner tool-kit of FD related techniques.
Thank you!
We're glad you find FDDA a useful addition to the toolkit of FD-related techniques. Thank you for your feedback and for supporting the paper being presented at ICLR!
Response to Reviewer gYLR
We appreciate that the reviewer recognized FDDA as a well-scoped, simple, and intuitive method that naturally extends FD-loss post-training from the generator to the decoder. We also appreciated the recognition that our experiments provide convincing evidence of consistent generation benefits across a variety of pre-trained generative models. Thank you for taking the time to review our paper and for your thoughtful feedback. We address your questions and concerns below.
We agree that FD loss has no such theoretical guarantee, and we note this in Appendix B. To check for overfitting, FDDA trains with only three representations (Inception, MAE, SigLIP2) and holds out DINOv2, CLIP and ConvNeXt for evaluation. The held-out DINOv2 and CLIP FDs decrease in all 22 configurations of Tables 1 and 2.
We agree that the gains are smaller after generator-side FD post-training, but they remain consistent across all 11 configurations (Table 2). Starting from the same generator-side FD-post-trained checkpoint, we continue either generator-side or decoder-side FD post-training for the same additional wall-clock time on a single H100. The decoder-side time includes building the latent pool, which takes 2.68 h for GigaTok-S-S, so its first result is at 3 h.
Table R1. gFDr6 (↓) under the same additional wall-clock time, starting from generator-side FD-post-trained models. Percentages are relative to 0 h.
After 5 h, continued generator-side post-training reduces gFDr6 by only 1.8-6.7%, while FDDA reduces it by 10.2-24.9%. This comparison is added to Section 5.4 (Table 6) of the revised manuscript.
Thank you for the suggestion. In Appendix I of the revised manuscript, we add three-row comparisons on TiTok-L-32 (Figures 18-26): the pretrained model, generator-side FD only, and generator-side FD + FDDA. The pretrained model uses a different generator, so its samples cannot share the same generated latents with the other two rows; all three rows use the same class labels and random seeds. The middle and bottom rows share the same generated latents and differ only in the decoder, so their difference shows the effect of FDDA after generator-side FD post-training.
Please let us know if you have any remaining questions.