acceptodds
Withdrawn from ICLR 2027

Latent Prior GANs: Efficient Decoder Refinement via Inherited Generative Structure

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern latent generative models learn strong generative structure through stable, non-adversarial training objectives in their latent generators. Their success changes a key premise of adversarial generation: this structure need no longer be learned from scratch through adversarial optimization, but can instead be inherited from a pretrained model. We refer to this inherited generative structure, comprising the learned generation-time latent distribution and pretrained latent-to-image decoding knowledge, as the latent prior. We introduce Latent Prior GANs (LP-GANs), which repurpose adversarial learning from discovering generative structure from scratch to targeted generation-time decoder refinement. LP-GANs freezes the pretrained latent generator and adversarially optimizes its decoder on generation-time latents, directly correcting the decoder train-generation mismatch: conventional decoders are optimized on reconstruction latents but deployed on generated latents. The reconstruction and perceptual losses require paired images, which generated latents do not provide, so we construct intermediate latents that still correspond to a real image. Across four latent generators and their model scales, five epochs of adversarial adaptation are enough to improve generation quality over the pretrained model. We hope this makes adversarial training cheaper when strong pretrained models are already available.

Primary area:
generative models
Keywords:
Generative Adversarial Networks, Latent Generative Models, Adversarial Generation, Latent Priors, Decoder Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ALqxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LP-GAN (Latent Prior GAN), which enhacnes the vae decoder by post-training of it using adversarial and paired reconstruction loss. The main motivation is that two-stage latent generative models does not consider the gap between the distribution of generated latents and ground-truth latents (encoded by vae). To reduce this gap, they suggest to finetune the vae decoder with extra adversarial training for reducing the divergence between the rgb images rendered from generated latents and groud-truth one. Also, for maintaining the reconstruction quality (correspondence to its input), they additionally introduce the reconstruction objective for real latent. Experimental results show that LP-GAN successfully reduces FID compared to iMF-XL/2 with lower computational cost compared to generative models (e.g., GAN) training from scratch.

Critical strengths
  • The motivation and approach of LP-GAN make sense for me. The problem they suggest is well-motivated and important research direction, as I believe. Similarly, the solution they propose is intuitive and expect to work well.

  • Quantitative results show that the proposed method successfully enhances the metrics such as FID compared to the pure latent generative models which does not use any post-training or finetuning methods.

Critical weaknesses
  • The most critical weakness is that their motivation is not that new. In other words, the importance of correction of generated latent and ground-truth latent is already considered in prior works. For example, every teacher forcing relevant techniques (especially for VAR [2, 3]) suggests to reduce the distributional gap between the generated and actual tokens. Thus, as I understand, the motivation and approach of LP-GAN is not that new and its contribution would be a proper application of finetuning of decoder in one-step generative model.

  • The paper shows the quantitaive comparison mainly focused on FID. However, as the generative backbone (such as diffusion transformers) are frozen, the metrics beyond distributional matching should be reported, as this part would be already modeled in backbone. For example, what property of samples from the enhanced decoder actually enhances? It is not intuitive for me and qualitative examples in appendix also does not imply actual difference.

  • Training of LP-GAN uses pretrained vision encoder-based discriminator (e.g., dino). As these pretrained encoder have features space favorable to the FID, as they are both trained on ImageNet dataset and downstream task. I suspect that this may largely affect to FID enhancement rather it actually enhances the visual quality of distributional gap between the generated and actual data. I believe this discriminator design should be analyzed more specifically for proving the effectiveness of LP-GAN.

[1] Zheng, Boyang, et al. "Diffusion transformers with representation autoencoders." ICLR, 2026.

[2] Rajagopalan, Sudarshan, Kartik Narayan, and Vishal M. Patel. "Restorevar: Visual autoregressive generation for all-in-one image restoration." ICLR, 2026.

[3] Han, Jian, et al. “Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis.” CVPR, 2025.

Questions for authors

First of all, please see the critical weaknesses.

  • Can the post-trained decoder be applied to the other types of generative models? For example, if decoder trained with improved meanflow is used for original meanflow, does it enhances FID scores?
Reviewer_enZhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets a mismatch in latent generation: the decoder is trained on reconstruction latents but used on generated latents, which have no paired images. LP-GANs freezes the generator and refines only the decoder. An adversarial loss aligns the decoder on fully generated latents, and a paired loss on intermediate latents along a noise-and-resample trajectory (taken from GAR) anchors it against drift. Five epochs of decoder-only refinement cut iMF-XL/2's 1-NFE gFID from 1.73 to 1.32 on ImageNet-256, and the same recipe improves MeanFlow, DMF, and the autoregressive TiTok.

Critical strengths
  1. The strongest evidence in the paper is the update-scope comparison in Table 3. On iMF-B/2, refining the decoder alone (2.14) beats joint training (2.40), and updating the generator alone (3.91) is worse than doing nothing at all (3.47). That surprised me a little, and it is the right experiment to run for the paper's central claim.

  2. The ablations are also informative about where the method stops working. Without CFG the adversarial term on its own makes things worse (16.58→19.43), and only the paired anchor brings it back to 13.39. The authors could have left that out. They didn't.

  3. The gains hold across four generator families, including a distilled model and an autoregressive one (Table 6, Fig. 4). However, all experiments use ImageNet-256 and the same shared decoder, as the authors acknowledge in App. B. The results show breadth across generators, but not across settings.

Critical weaknesses

Overall, the results are strong, but the contribution is incremental and the evaluation narrow: one new term added to GAR's setup, tested on one benchmark with one shared decoder.

  1. My main concern is attribution. The adversarial-only arm shows a benefit (2.55, Table 2), but there is also no control for simply training longer than 5 epochs, e.g., the joint training arm. so it shows decoder-only wins at that budget, not necessarily beyond it. Without these comparisons, the contribution of the new term remains unclear.

  2. Novelty is limited as well. GAR, published just days earlier, already introduced the diagnosis, trajectory, and paired adaptation on intermediate latents. The main addition here is the adversarial loss on fully generated latents. This is a meaningful step, but a relatively practical finding to an otherwise inherited method.

  3. The efficiency claim is also hard to justify. The 88 H100 hours cover refinement only; adding the prior's hours, it could be well above StyleGAN-XL's and other baselines. GAR's cost and the η sweep required for each new model are also unreported. The current accounting does not establish the claimed efficiency.

  4. Finally, the paper never measures the effect on reconstruction quality. The paired loss is meant to prevent the decoder from drifting, yet the paper reports gFID throughout and no rFID. A before-and-after comparison would show whether reconstruction quality is preserved. This matters to coverage.

Questions for authors

What would change my mind: a longer-training control, before-and-after rFID or recall/coverage measurement, and address the above concerns. With these, I would move to weak acceptance; otherwise, I lean toward weak rejection.

Reviewer_85tMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Latent Prior GANs (LP GANs), which improve a pre-trained image generator by finetuning its decoder while keeping the encoder and latent generator frozen. It addresses the mismatch between latents obtained by encoding real images and generated latents. The method applies an adversarial loss to fully generated samples and uses partially regenerated latents for reconstruction and perceptual supervision. Experiments on ImageNet using iMF, MeanFlow report improved FID when finetuned for five epochs.

Critical strengths
  1. Improving existing generators without increasing inference cost is a relevant problem, and the paper provides a simple approach that only updates the decoder in the generation pipeline.
  2. The paper reports improvements across multiple generators and model sizes. The ablations on paired supervision, and which components to update help analyze the proposed method.
Critical weaknesses

It would be helpful for readers if the following points were clarified or improved:


  1. GAR baseline approach fine-tunes the decoder using adversarial losses on generated latents starting with noisy real latents. The paper proposes to apply the adversarial loss to generated latents from pure random noise. It would be helpful to give more details about the comparison setup for these approaches (i.e. for fair comparison under the same training budget and settings, like same number of epochs fine-tuning for the decoder). This would help readers understand the efficacy of the proposed change.


  2. Experiments are evaluated on ImageNet at 256 × 256 resolution. Maybe experiments on another dataset or a public text-to-image model could help readers understand whether the benefits extend to more diverse images and conditioning.


  3. The quantitative evaluation mainly reports FID. It would be helpful to check whether generated images continue to match their requested classes and whether sample diversity is preserved after refinement.

Questions for authors

Please refer to the Weakness section for questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.