acceptodds
Under review as a conference paper at ICLR 2027

Rethinking the Design Space of Vanilla Pixel Diffusion Transformers

11 views

Abstract

Pixel-space diffusion models have recently re-emerged as a compelling alternative to latent diffusion, avoiding the lossy latent bottleneck and separately trained autoencoder. JiT shows that a vanilla diffusion transformer can generate directly in pixel space by replacing the commonly used velocity prediction () with clean image prediction (). In this work, we revisit this choice and introduce **Mꜰʟᴏᴡ**, an adaptive parameterization that learns to interpolate between and prediction across principal directions of the data, which can be learned end-to-end with diffusion model training. We find that the preferred parameterization varies systematically with the data representation and model capacity. Notably, increasing model capacity shifts preference toward prediction from $x_0$ prediction. We evaluate Mꜰʟᴏᴡ across ImageNet, UCF-101 video, and large-scale text-to-image generation, demonstrating competitive or improved generative performance across diverse settings. In addition, we find that vanilla pixel diffusion transformers, irrespective of parameterization, retain a persistent high-frequency deficit relative to real images. To address this, we introduce **_ConvRefiner_**, a lightweight convolutional module trained post-hoc with adversarial loss while keeping the diffusion backbone frozen. With only a small amount of additional training, _ConvRefiner_ restores the high-frequency statistics of generated images while preserving FID.

Primary area:
generative models
Keywords:
Diffusion Models, Flow Models, Generative models, Pixel Diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_THynOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MFlow, which extends AsymFlow's binary PCA-based prediction targets to continuous learnable weights, allowing the model to adaptively combine x-prediction and v-prediction. The authors also introduce ConvRefiner, a lightweight convolutional module for recovering missing high-frequency details.

Critical strengths
  1. MFlow is simple and does not require changes to the transformer architecture. By learning prediction weights for different PCA directions, it avoids manually selecting the cutoff rank in AsymFlow. The authors also evaluate the method across image, video, and text-to-image generation tasks.

  2. The authors identify a clear high-frequency deficit in JiT-style models and introduce ConvRefiner to address it. The module can be trained without updating the diffusion backbone and substantially improves high-frequency statistics while largely preserving FID.

Critical weaknesses
  1. The novelty over AsymFlow is limited. AsymFlow already uses different prediction targets across PCA directions, while MFlow mainly replaces its binary selection with continuous learnable weights. Although this provides more flexibility, the paper does not clearly establish why continuous weights are necessary or what they offer beyond the existing binary formulation.

  2. The performance gains are relatively small. Compared with AsymFlow, MFlow shows only marginal improvements across image, video, and text-to-image generation, and even performs worse in some settings. Its ImageNet results also fall behind recent methods such as PixelDiT and DeCo, although these use different architectures.

Questions for authors
  1. Could you provide more evidence that continuous weights are necessary? For example, how would MFlow perform if its learned weights were binarized and the model retrained under the same setup?

  2. Given the small performance differences, could you report results from multiple training runs to demonstrate that the improvements are consistent? It would also be useful to compare the total training and hyperparameter tuning costs of MFlow and AsymFlow to reach similar FID scores.

4 comments

Dear Reviewer,

Thank you again for your thoughtful review. We have posted a detailed rebuttal addressing your concerns, along with additional experimental results.

We would be grateful if you could please review our responses and let us know whether they address your concerns, or whether any further clarification would be helpful.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Additional Experiment: Binarizing Learned Mꜰʟᴏᴡ Gains

We thank the reviewer for suggesting this experiment. We conducted two additional ImageNet-256 B/16 training runs, where the learned Mꜰʟᴏᴡ gains were binarized using thresholds (mi>0.5m_i > 0.5) and (mi>0.1m_i > 0.1), and subsequently held fixed while retraining the generator under the same training recipe.

  • ImageNet-256, B/16, with LPIPS. Results are from a single training seed, with FID evaluated using the default JiT-B sampling hyperparameters. | Parameterization | Gain configuration | FID | |------------------|--------------------------|-----| | JiT | Rank 0 (pure x0x_0) |3.424| | AsymFlow | Rank 8 |3.400| | Binarized Mꜰʟᴏᴡ |mi>0.5m_i>0.5 (rank 2, frozen)|3.370| | Binarized Mꜰʟᴏᴡ |mi>0.1m_i>0.1 (rank 5, frozen)|3.449| | Mꜰʟᴏᴡ (Ours) | Continuous, learned gains|3.327|

In this controlled comparison, Mꜰʟᴏᴡ achieves FID 3.327, compared with 3.370 and 3.449 for the two binarized variants. Although these differences are modest and we cannot establish their statistical significance from a single training seed, the results suggest that continuously learned gains remain competitive with fixed binary parameterizations and may offer additional benefits beyond identifying a binary subspace.

Furthermore, the two binarization thresholds yield different results, pointing towards the slight sensitivity to the choice of binary mask. Together, these observations provide additional empirical support for Mꜰʟᴏᴡ's flexible parameterization without requiring manual cutoff or threshold selection.

Dear Reviewer,

In our rebuttal, we addressed the binarization experiment suggested by you, supporting Mꜰʟᴏᴡ's flexible parameterization empirically, while saving thousands of hours of parameterization search. We furthermore provided five-seed inference evaluations and additional clarification of Mꜰʟᴏᴡ's mathematical and analytical contributions. We provide more results including the effectiveness of ConvRefiner in our Anonymous Website.

Based on these additional results and clarifications, would the reviewer kindly consider upgrading their rating?

Reviewer_THynOpenReview ↗

response to authors

Thank you for the detailed rebuttal. However, my main concern remains unresolved. Despite the results across five random seeds, the performance gains over JiT and AsymFlow are marginal, with even larger gaps relative to PixelDiT and DeCo. Moreover, the improvements from ConvRefiner are barely noticeable in the provided visualizations, and the methodological contribution remains incremental. Therefore, I will maintain my original score.

Reviewer_4VzHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes two methods to improve vanilla pixel diffusion transformers. The starting place is the JiT (just image transformers) approach, which runs on clean image pixels instead of velocity. The first method, MFlow, generalizes another method, AsymFlow, which splits up the PCA directions and does velocity prediction in the top-r directions and image prediction on the others. MFlow instead proposes a continuous, learnable gain per direction. The second method is a small refiner trained post hoc to improve the high spatial frequencies. These are tested on ImageNet, UCF-101 video, RAEv2 latents, and text-to-image on an internal dataset. Claimed results include competitive or improved FID/FVD vs. JiT and AsymFlow without a parameterization search, best scores among pixel-space models, and restoration of the high-frequency spectrum with unchanged FID.

Critical strengths

(1). The formulation is clean and clear. MFlow other prediction methods and AsymFlow as special cases. The velocity-recovery map is exact, and the pseudocode in the appendix makes the method easy to reproduce.

(2). The paper provides good comparisons to relevant methods. JiT, AsymFlow, and MFlow are retrained in the same codebase with the same backbone, FLOPs, loss, and training schedule. Hyperparameters are tuned per model for ImageNet-H and UCF-101, and shared for T2I. Training and sampling configurations are specified per model.

(3). A wide variety of settings are tested, with the same parameterization code throughout.

(4), The limitations section is honest, saying that gains over fixed parameterizations are sometimes modest, that lower diffusion loss does not translate into better FID, and that FID is insensitive to the refiner's effect.

Critical weaknesses

(1). Overall the novelty of the paper is somewhat minor. The first method is a softer version of AsymFlow and the second is a Conv/GAN head. These do not amount to a big change in what vanilla pixel transformers can do.

(2). The experiments lack statistical validity. It seems that every value is based on a single training run evaluated once; no seeds, standard deviations, or confidence intervals appear in the paper. The paper's figures show the noise is comparable to the claimed effects.

(3). Based on the results in the tables, it appears that MFlow does not improve on AsymFlow, and in several settings not on JiT. And the claimed wins in the text-to-image setting are not really supported by the numbers.

Questions for authors

(1). How many training seeds were run, that underlie each result? It would be good to report mean and std over several seeds for the main results and ablations. And the default MFlow configuration appears to have different results across the tables. Do these reflect variance across runs or is the configuration somehow different?

(2). Have you tried any other metrics to evaluate ConvRefiner? It would be good to explore a metric that is sensitive to local texture, such as precision/recall, or to try a small human preference study. These could help validate that the high frequency modifications are worthwhile.

(3). What is the contribution of adding LPIPS? Some of the best results in the paper involve adding LPIPS to JiT, at B level, but it seems to not add anything at H scale. Can the authors make sense of this result?

5 comments

Dear Reviewer,

Thank you again for your thoughtful review. We have posted a detailed rebuttal addressing your concerns, along with additional experimental results.

We would be grateful if you could please review our responses and let us know whether they address your concerns, or whether any further clarification would be helpful.

Thank you for your time and consideration.

Dear Reviewer,

In our rebuttal, we provided evidence of the effectiveness of ConvRefiner using a user study (81.2% preference) you suggested, as well as visual results on our Anonymous Website. We furthermore provided five-seed inference evaluations and additional clarification of Mꜰʟᴏᴡ's mathematical and analytical contributions. We also clarified the question regarding LPIPS's contribution.

We would further like to emphasize Mꜰʟᴏᴡ's ability to save 1000s of hours in parameterization search and our systematic high-frequency deficit analysis in JiT-style backbones.

Based on these additional results and clarifications, would the reviewer kindly consider upgrading their rating?

Reviewer_4VzHOpenReview ↗

Response to Author Comments

Thank you for the human preference study and the multi-seed evaluation. The preference for ConvRefiner is encouraging, and the study protocol should be added to the paper. However, the multi-seed numbers support my comments, as the spread from sampling seeds is as large as the gaps between JiT, AsymFlow and MFlow. So claims that MFlow improves on them, such as “best on every benchmark” for T2I, should be restated. Wrt the inconsistency, I meant two places: the MFlow ablation paragraph, where the u-loss setting gives FID 3.66 but the final default is 3.74; and Figure 7, where the frozen MFlow-B base is ~3.39 versus 3.33 in Table 2, even though both seem to use fixed evaluation settings and JiT-B matches across them. Because these discrepancies are as large as the ablation steps used to choose the default, I’m keeping my score. The paper would be stronger if it did not focus on FID gains but instead on the methods -- removing the rank search the M-curve analysis.

Comments on the paper

3 comments

Dear Reviewer,

Thank you again for your thoughtful review. We have posted a detailed rebuttal addressing your concerns, along with additional experimental results.

We would be grateful if you could please review our responses and let us know whether they address your concerns, or whether any further clarification would be helpful.

Thank you for your time and consideration.

Dear Reviewer,

In our rebuttal, we provided additional experiments and analyses addressing several of your concerns, like controlled B/H capacity comparisons, full M-curves, quantification of parameterization search costs, and five-seed inference evaluations. We also provided additional text-to-image ConvRefiner results and a human preference study (81.2% preference). We further clarified the mathematical and analytical contributions, implementation details, and interpretation of the RAEv2 restuls.

We hope these additional results address your main concerns. Based on these, would the reviewer kindly consider upgrading their rating?

I maintain my original rating. While the work is technically sound and provides useful analysis, the core methodological contribution remains incremental, and the rebuttal does not sufficiently change my assessment of its novelty.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.