acceptodds
Withdrawn from ICLR 2027

Adversarial Training for Pixel Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Pixel diffusion models generate RGB images directly, avoiding the bottleneck of an autoencoder, yet their outputs still systematically underrepresent fine-scale natural-image statistics. We show that adversarial learning provides an effective post-training correction for this deficiency. Starting from a pretrained model, we retain its original diffusion or flow-matching objective and add an adversarial loss to the predicted output at non-high-noise timesteps, leaving the model architecture and sampling procedure unchanged. To our knowledge, this is the first systematic study of adversarial-based post-training for pixel diffusion. Across two pixel backbones, the method jointly improves distribution fidelity, coverage, prompt alignment, and perceptual quality. We further investigate why it works. Frequency-band and power-law analyses show that the original models systematically underproduce natural-image high-frequency content, while adversarial post-training restores this missing spectral power. In contrast, perceptual loss also increases high-frequency content but sacrifices distribution fidelity and prompt alignment. And nearest-neighbor, recall, and matched no-GAN SFT controls further rule out memorization, mode dropping, and additional optimization as simple explanations. Finally, we examine the boundary of this effect. Under the tested latent diffusion configurations, the same procedure does not produce comparable joint gains and adds almost no decoded high-frequency power. These results identify direct output access to the image statistics being corrected as a key factor governing when adversarial post-training succeeds.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Pixel Diffusion, Adversarial loss

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_L3GPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies adversarial post-training for pretrained pixel-space text-to-image diffusion models. The method retains the original diffusion or flow-matching objective and adds a hinge adversarial loss to the predicted clean image at non-high-noise timesteps, without changing the model architecture or sampling procedure. Experiments on DeCo and PixelGen show improvements in distributional fidelity, recall, prompt alignment, and no-reference image quality. The authors analyze changes in high-frequency image statistics and compare the method with perceptual-loss training, sharpening controls, and latent diffusion models.

Critical strengths
  1. The paper addresses a relevant question and presents a simple post-training method. Results across two pixel diffusion backbones show improvements across several complementary metrics.

  2. The analysis goes beyond performance comparisons. The spectral study, perceptual-loss and sharpening controls, discriminator and timestep ablations, and pixel-latent comparisons provide useful insight into the method and its limitations.

Critical weaknesses
  1. The novelty seems insufficient. The core mechanism of this paper is simply adding adversarial loss to pixel diffusion. Both pixel diffusion and adversarial loss are established techniques in the literature. Moreover, the paper does not appear to introduce a particularly novel mechanism for applying adversarial loss. The paper "PixelGen: Improving Pixel Diffusion with Perceptual Supervision" explicitly states that adding adversarial loss affects training stability, which is why PixelGen does not use adversarial loss. The authors also provide insufficient discussion of whether adding adversarial loss causes training instability.

  2. The writing quality is unsatisfactory. First, the paper lacks a flowchart to help readers understand the method, which is necessary. Figures 1, 2, and 5 all present qualitative experimental results; retaining just one would be sufficient. The writing is verbose, particularly in Sections 4 and 5, which contain many unnecessary details.

Questions for authors
  1. Could the authors discuss the specific innovations of their proposed method relative to existing techniques in the literature?

  2. Does adding adversarial loss affect the stability of model training?

  3. Do the authors plan to release the code, training configurations, and evaluation scripts?

  4. Could the authors briefly clarify the exact objectives used for the PixelGen no-GAN and +GAN results, including which perceptual losses are retained or replaced? The appendix notes that PixelGen +GAN replaces native perceptual supervision. A clearer explanation of this setup would help readers compare the PixelGen results with the DeCo experiments.

Reviewer_eeiNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an additional layer of adversarial (GAN) to provide post-training correction to generated frequency enhanced images from the pretrained models (they selected two pixel diffusion models). The training retains the existing algorithm's original objective and add an adversarial GAN based loss for post-training. It appears to offer limited methodological novelty as the proposed approach is simply adding the GAN to the existing methods. The authors did not even provide a diagram to describe the implementation.

Critical strengths

This paper proposed an adversarial post-training to utilize GAN for post-training correction to generated images from the pretrained models (they selected two pixel diffusion models).

Critical weaknesses
  • The described algorithm seems like traditional image enhancement (or simply GAN image enhancement) to me. While traditional image enhancement could out perform, the paper did not provide a clear justification. The experimental results seem to show that the texture is enhanced. However, what is causing this enhancement? or does the image even needs the texture enhancement?
  • The paper stated that the existing methods indicates measurable high-frequency deficit. However, the original experiment results from the two papers they selected shows that they generated images of details texture quite well, but somehow, in this paper, the same algorithms created results that lost high frequency content. I wonder whether this inconsistency is due to the current paper's use of non-optimal parameters.
Questions for authors
  1. Could you explain the notations in this formula in page 15? D(x, y) = Duncond(x) + ⟨V (y), ϕ(x)⟩, and why this enhances the high frequency content in the post training?
  2. Why some of the image reconstructed w/ Gan has larger field of view (FOV) than the original images generated from the existing algorithms, some images has smaller FOV, and some seems approximately equal?
Reviewer_sbpvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Based on the of Ma. et. al, authors say that trained pixel diffusion models underrepresent fine scale image statistics (high frequency components). They also confirm this by providing a quantitative values of radial spectrum of generated images (Table 2).

To resolve this, they propose a adversarial training method by adding GAN loss and continue the training. First, they empricially show that adding this training head improves the metrics like FID, Patch FID etc. Author also observe that for newly trained pixel difusion model restores High frequencies when compared with NO GAN version (Figure 3). In Section 4.3, they provide empirical evidence that this gain doesn't effect the diversity (traditional GAN models have this problem).

In Section 5, authors argue that the same methodology cannot be effectively applied to Latent Diffusion Models and their main observation is because of Frozen VAE which hinders the performance gains. (Section 5.2).

Critical strengths

Paper is written clearly, easy to follow. Authors also tried to answer why there was a performance gain in the proposed method which I personally liked.

Critical weaknesses

W1. In Section 5.2, authors argue that same gains on latent diffusion models is not observed because of limited output access to frozen VAE decoder (they used the wording "strong evidence"). To actually conclude this, authors should replace or finetune the decoder. The natural experiment, jointly fine tuning the decoder with the adversarial loss or using a higher fidelity autoencoder, is missing.

W2. Continuing on this, in Section 2, authors say that VAE were trained on LPIPS + Patch GAN objective. But Section 5.2 says that VAE reconstruction of real images steepens the spectral slope to α=2.48\alpha = 2.48 and removes 0.2 dex of High Frequency components. This means, it doesn't matter what diffusion model produces, it is limitation of VAE decoder (even though it is trained with GAN type loss). So this weakens "adversarial post-training" value as evidence for the proposed mechanism.

Combining above points, in this particular case, I feel that authors did strong claims, without performing complete controlled experiments.

W3. (From my "AI Documentation For Reviewing") : prior works$[1, 2, 3, 4]$ are not discussed by the paper. Authors should cite them in their Section - 2 as I think they are closely related to the proposed methodology (I skimmed through these papers very briefly and felt that they are related to proposed method.). Also, discuss how the proposed method differs from them methodologically : A distinction based only on the model family or setting (as these works target consistency or score-based models while the current paper targets pixel diffusion) would not by itself be sufficient, as I want to understand whether the underlying technique differs.

[1][1]: ACT-Diffusion: Efficient Adversarial Consistency Training for One-step Diffusion Models

[2][2]: Phased Consistency Models

[3][3]: Adversarial score matching and improved sampling for image generation

[4][4]: Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion

Minor Weakness

W4. According to Section 2 of the paper (around lines 129129 - 133133), training with specific loss and adding a adversarial objetive objective has been explored before in cases of VAE/distiallation. I also agree that authors didn't claim that they are first ones to propose this idea. What I suggest is that they should may be make this explicit in Contirbution 1.

Questions for authors
  1. According to Appendix B, PixelGen has perceptual loss included. So when authors say +GAN variant, they remove perceptual loss and add GAN loss (Table 10). In this sense, for PixelGen, the Table 1 is misleading because +GAN is understood as adding GAN loss without changing anything in the baseline. Could authors report a PixelGen fine tuned with the flow matching objective only (no perceptual loss, no GAN), for the same number of steps, so that the GAN's contribution can be isolated.

  2. Pointing out to W1, W2, I think "when it works" contribution (Contribution 3) should either be supported by a direct test (e.g., an unfrozen decoder variant) or toned down to a hypothesis consistent with the observations.

Based on response from authors, I'm open to increasing the Rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.