Latent Prior GANs: Efficient Decoder Refinement via Inherited Generative Structure

Modern latent generative models learn strong generative structure through stable, non-adversarial training objectives in their latent generators. Their success changes a key premise of adversarial generation: this structure need no longer be learned from scratch through adversarial optimization, but can instead be inherited from a pretrained model. We refer to this inherited generative structure, comprising the learned generation-time latent distribution and pretrained latent-to-image decoding knowledge, as the latent prior. We introduce Latent Prior GANs (LP-GANs), which repurpose adversarial learning from discovering generative structure from scratch to targeted generation-time decoder refinement. LP-GANs freezes the pretrained latent generator and adversarially optimizes its decoder on generation-time latents, directly correcting the decoder train-generation mismatch: conventional decoders are optimized on reconstruction latents but deployed on generated latents. The reconstruction and perceptual losses require paired images, which generated latents do not provide, so we construct intermediate latents that still correspond to a real image. Across four latent generators and their model scales, five epochs of adversarial adaptation are enough to improve generation quality over the pretrained model. We hope this makes adversarial training cheaper when strong pretrained models are already available.