acceptodds
Under review as a conference paper at ICLR 2027

Ordered Representations for Pixel Diffusion: Denoising with Learned Velocity Correction

Abstract

Recent pixel diffusion models improve generation through clean-image prediction (-pred) or encoder–decoder architectures that reduce the noise-modeling burden on the backbone. Combining these designs, we find that -pred learns stronger representations and achieves lower FID than velocity prediction (-pred) with sufficient sampling steps, yet underperforms in few-step sampling. Examining its induced sampling velocity reveals that the -to- conversion amplifies prediction errors, which are most pronounced along the leading, high-energy directions of RGB images. Motivated by this structure, we propose **OiT**, an Ordered-representation image Transformer that uses nested channel masking to make dominant image information progressively accessible from the leading feature channels. A compact prefix of this representation guides a full-dimensional velocity correction, while the full representation remains responsible for clean-image prediction. This allows OiT to improve the representation quality and long-step generation of -pred while achieving the strong few-step performance of -pred. We further introduce bottlenecked Representation Alignment (bREPA) and a Hann Decoder as lightweight designs for representation alignment and patch decoding. On ImageNet , OiT-B and OiT-XL achieve FID-50K scores of 2.52 and 1.55, respectively, reaching state-of-the-art performance among pixel diffusion models at comparable compute.

Primary area:
generative models
Keywords:
image generation, pixel diffusion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nKHiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OiT to combine the strengths of x-pred and v-pred, aiming to improve few-step sampling while retaining the advantages of x-pred. It uses nested channel masking to learn ordered representations and a small feature prefix for velocity correction. The paper also introduces bREPA and a Hann Decoder to improve feature alignment and image reconstruction.

Critical strengths
  1. The motivation is interesting. The paper highlights the trade-off between x-pred and v-pred in representation learning and sampling quality. Using velocity correction to combine their strengths is a reasonable direction.

  2. The overall results are strong. OiT-XL achieves 1.55 FID on ImageNet-256, with promising results in 10-step sampling. The paper also includes useful ablations on nested masking, prefix width, bREPA, and the Hann Decoder.

Critical weaknesses
  1. The gains from the core design are modest. In Table 3, adding nested masking and velocity correction only improves FID from 5.84 to 5.68. Since JiT already trains x-pred with a velocity loss, the need for a separate Correction Decoder is not well justified. Without a compute-matched comparison against a larger Clean Decoder, it is unclear whether the improvement comes from the proposed correction design or simply the extra model capacity.

  2. The benefit of Ordered Representation is unclear. The authors observe larger velocity errors along the leading PCA directions, but the connection to the leading feature channels learned through nested masking is not well established. Tables 4 and 6 provide some useful ablations, but do not fully separate the effects of nested masking and prefix conditioning. It remains unclear whether ordered features offer any advantage over a simple low-dimensional representation.

Questions for authors
  1. Why is a separate Correction Decoder preferable to simply increasing the capacity of the Clean Decoder, given that JiT already uses a velocity loss? Could the authors compare the two designs with matched parameters and FLOPs, including results at different sampling steps?

  2. Is Ordered Representation necessary for velocity correction? What happens if the correction branch uses the first 8 channels or a learned low-dimensional projection without nested masking? Conversely, how does nested masking perform with full-feature correction? These comparisons would help clarify whether the learned ordering itself matters.

4 comments
AuthorsOpenReview ↗

Response to Reviewer nKHi

We thank the reviewer for the thoughtful assessment and for recognizing OiT's motivation and strong results. These concrete suggestions helped us design more direct comparisons of decoder capacity and feature ordering. We address both concerns below.


W1 / Q1: Velocity supervision and decoder capacity

We agree that JiT already uses velocity-space supervision. The distinction is what the network predicts: JiT predicts the clean image, with a v-loss equivalent to a timestep-weighted x-loss (Eq. (7)). Sampling still relies on a fixed x-to-v conversion that amplifies reconstruction errors. OiT retains clean-image prediction while learning a separately supervised velocity residual to refine the converted field. As shown in Table 1, x-pred remains weaker than direct v-pred in few-step sampling.

Following the suggestion, we removed OiT-B's two-layer Correction Decoder and expanded the Clean Decoder from two to four layers to closely match the parameter and FLOP budgets. We also include a control with two two-layer Clean Decoders whose outputs are averaged:

Decoder architectureProbing Acc. (%)↑FID@10↓FID@50↓
2-layer Clean Decoder44.1215.138.36
2-layer Clean Decoder + 2-layer Clean Decoder44.0115.018.29
4-layer Clean Decoder44.2315.298.12
2-layer Clean Decoder + 2-layer Correction Decoder (OiT)45.3012.067.67

Relative to the four-layer Clean Decoder, OiT improves probing accuracy by 1.07 percentage points and reduces FID by 3.23/0.45 at 10/50 steps. Its advantage over both capacity controls supports the contribution of explicit velocity correction.

Regarding the modest gain in Table 3, the baseline is already much stronger after the bREPA stage, leaving less room for further improvement. We therefore also evaluated correction before adding bREPA, keeping all other settings unchanged.

ConfigurationFID↓IS↑
Vanilla x-pred9.81147.8
+ Decoder8.36162.3
+ Hann Decoder7.79165.5
Hann Decoder + correction (new)6.87183.9
Hann Decoder + bREPA5.84206.7
Hann Decoder + bREPA + correction5.68211.4

Before bREPA, correction reduces FID by 0.92 and improves IS by 18.4.


W2 / Q2: Feature ordering and compact correction inputs

(1) Why nested masking encourages ordering

The ordering mechanism has a simple explanation in a linear orthogonal model. Let xx be centered data and uiu_i orthonormal directions. Reconstructing from a prefix of length KK gives x^K=∑i=1K(ui⊤x)ui\hat{x}_K=\sum_{i=1}^{K}(u_i^{\top}x)u_i. When KK is sampled independently of xx,

=\mathbb E\|x-\hat{x}_K\|_2^2 =\mathbb E\|x\|_2^2 -\sum_i\Pr(K\ge i)\,\mathbb E[(u_i^\top x)^2].$$ The first term is fixed total variance; the sum rewards captured variance weighted by how often each direction is retained. Since earlier directions are retained at least as often, placing the leading principal components first minimizes this objective, consistent with the linear theory of Nested Dropout (Rippel et al., 2014). Related ordering is also observed empirically in nonlinear models. FlexTok (Bachmann et al., 2025, Sec. 5.1) shows that short prefixes capture high-level semantic and geometric structure, while additional tokens refine details. Semanticist (Wen et al., 2025, Sec. 5.3, Fig. 4) reports a PCA-like hierarchy in tokens' contributions to reducing diffusion loss. These observations support the motivation for learning informative leading channel groups in OiT. Here, "PCA-like" refers to importance ordering. **(2) Controlled ablations** We appreciate the suggestion to separate the effects of nested masking and correction input width. We added the two requested controls to Table 4(a). All variants use the same training and evaluation settings. | Nested masking | Correction input | FID↓ | IS↑ | | --- | --- | ---: | ---: | | No | None | 5.84 | 206.7 | | No | Full features | 5.81 | 208.8 | | No | First 8 channels | 5.84 | 207.0 | | Yes | None | 5.77 | 209.9 | | Yes | Full features | 5.75 | 210.6 | | Yes | First 8 channels | **5.68** | **211.4** | With the correction input fixed to eight channels, nested masking improves FID from 5.84 to 5.68. With nested masking enabled, the eight-channel prefix also outperforms full-feature correction (5.68 vs. 5.75 FID). Restricting the input width without nested masking gives no FID improvement over the baseline without correction. These results support learning informative prefixes for correction. We especially appreciate the reviewer's precise suggestions, which led to direct tests of both design choices. Thank you for the time and care devoted to examining the method and helping us strengthen its justification.

Dear Reviewer nKHi,

Thank you very much for the time and care you have given our paper and for your thoughtful feedback. We have posted a detailed response to your comments and hope it addresses the concerns you raised. If anything remains unclear or insufficiently addressed, we would be grateful for the chance to clarify it before the discussion period closes. If you feel that the main concerns have been resolved, we would also greatly appreciate your consideration of a higher rating.

Best regards,

Authors of Submission 30812

Reviewer_nKHiOpenReview ↗

response to author

Thank you for the detailed response. The rebuttal has addressed most of my concerns, so I have decided to raise my rating.

Thank you very much for carefully considering our response and for your positive reassessment! Your thoughtful feedback has helped us strengthen the experiments and clarify the method. We sincerely appreciate your time and constructive engagement, and we will incorporate the additional results and clarifications into the revised manuscript.

Reviewer_yJ1pOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OiT, an ordered representation method for pixel-space diffusion, aiming to combine the strong representational ability of x-pred with the better few-step sampling performance of v-pred. OiT learns information-ordered feature representations through nested channel masks and uses compact feature prefixes to predict full-dimensional velocity corrections. Additionally, the paper proposes bREPA and the Hann Decoder, which are used for representation alignment and reducing patch reconstruction boundary artifacts, respectively. On ImageNet 256×256, OiT-B and OiT-XL achieve FIDs of 2.52 and 1.55, respectively; with 10-step sampling, OiT-B reaches an FID of 5.42, a clear improvement over JiT-B/16’s 7.16.

Critical strengths

The ordered feature prefix used for speed adjustment is simple in design and logically clear, effectively combining the representational advantages of x-pred with better performance in few-step sampling.

It achieves strong generative results on ImageNet, and scales well from B to XL, while maintaining competitive computational costs.

Critical weaknesses
  1. The independent gain of the OiT core speed correction mechanism is quite small. After adding the nested-v residual in Table 3, the FID only drops from 5.84 to 5.68, and the main performance improvement actually comes from the decoder and bREPA.
  2. Some experiments lack proper control and explanation. Especially for Tables 1 and 3, the training/evaluation setups are different, but the paper doesn't clearly explain this; also, loss weights and the variance across multiple runs are not reported.
Questions for authors

Please clarify why the training and assessment settings used in Table 1 and Table 3 differ.

Can x-pred, v-pred, x-pred + full residual, and OiT be fairly compared under the same architecture and training budget?

1 comment
AuthorsOpenReview ↗

Response to Reviewer yJ1p

We appreciate the reviewer's encouraging assessment of ordered-prefix correction and its scalability. The questions about the correction mechanism and experimental controls are especially helpful, and we have added targeted experiments and clarifications below.


W1: Contribution of velocity correction

The bREPA stage substantially strengthens the baseline, which makes the additional gain from correction appear smaller. To better assess correction's contribution before this stage, we added it to the Hann Decoder baseline while keeping all other settings unchanged.

ConfigurationFID↓IS↑
Vanilla x-pred9.81147.8
+ Decoder8.36162.3
+ Hann Decoder7.79165.5
Hann Decoder + correction (new)6.87183.9
Hann Decoder + bREPA5.84206.7
Hann Decoder + bREPA + correction5.68211.4

Before the bREPA stage, correction reduces FID by 0.92 and improves IS by 18.4.


W2 / Q1: Experimental settings, loss weights, and variability

We sincerely thank the reviewer for pointing out that some experimental details were not clearly described. We will clarify the differences between Tables 1 and 3 and explicitly report the detailed configurations in the revised manuscript.

Table 1 searches for the best CFG scale for each method to present its best generation performance, whereas Table 3 fixes CFG at 2.75 across all rows for a controlled component ablation. Table 1 also disables REPA/bREPA for all methods to evaluate classification probing without additional representation-alignment supervision.

SettingTable 1Table 3
CFG scalev-pred: 4.25; x-pred: 3.75; OiT: 3.02.75 for all rows
REPA/bREPADisabled for all methodsbREPA enabled as specified by each row
Clean-loss weight (OiT)11
Correction-loss weight (OiT)11
Alignment-loss weight (OiT)00.1

To assess training variability, we also resumed OiT-XL from the same 220-epoch checkpoint using three different training seeds and trained all runs to the same endpoint.

RunFID↓IS↑
Run 11.5532304.21
Run 21.5488305.76
Run 31.5492304.97
Mean ± SD1.5504 ± 0.0024304.98 ± 0.78

Q2: Comparison under the same architecture and training budget

Following the suggestion, we retrained the baselines under the same architecture and training budget. The x-pred and v-pred variants each use two decoders and average the outputs. All four methods are trained for 80 epochs without REPA/bREPA. The generation results use the best CFG scale found for each method.

MethodProbing Acc. (%)↑FID@10↓FID@50↓
v-pred (2 v-decoders)42.5612.1310.47
x-pred (2 x-decoders)44.0115.018.29
x-pred + full residual44.2215.098.33
OiT45.3012.067.67

OiT achieves the highest probing accuracy and the lowest FID at both sampling budgets, showing that its advantages persist under the same architecture and training budget. These results support its ability to combine strong representations with improved few-step generation. We will update Table 1 with this controlled comparison in the revised manuscript.

These perceptive questions have helped us strengthen the experimental controls and better substantiate the contribution of velocity correction. We warmly appreciate your engagement with this work and hope the new evidence and clarifications address your concerns.

Reviewer_Gz2gOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the trade-off between clean-image prediction named as x-pred and velocity prediction named as v-pred in pixel-space diffusion. It proposes OiT, which uses nested channel masking to learn ordered representations and lets a compact feature prefix condition a full-dimensional velocity-correction branch. The paper also introduces bottlenecked Representation Alignment and a Hann Decoder for auxiliary representation supervision and overlapping patch reconstruction. Experiments on class-conditional ImageNet report FID scores, as well as improved 10-step sampling results. Ablations provide evidence for the contributions of the proposed components.

Critical strengths

1.The method is motivated by a clear optimization issue that converting the errors of x-prediction into velocity errors can amplify them, particularly at later timesteps. Using an ordered feature prefix to guide velocity correction is a coherent and potentially userful response to the issue.

2.The results over ImageNet are competitive among diffusion models in pixel space, and the paper includes ablations and a few-step sampling evaluation which provide initial evidence for OiT’s effectiveness

Critical weaknesses
  1. The robustness of the headline results is difficult to assess because the paper does not report results across multiple training seeds or uncertainty estimates. In particular, the reported OiT-XL’s FID of 1.55 is close to the cited AsymFlow-H/16 result of 1.57.

  2. The compute comparison would be more informative if it accounted consistently for the full sampling cost, including both prediction branched and classifier-free guidance, and matched sampling budgets across methods. The few-step evaluation is also limited in its set of comparisons.

Questions for authors
  1. How much do FID and Inception Score vary across independent training runs? Can the authors provide multi-seed results or uncertainty estimates, especially for OiT-XL and its closest competitors?

  2. Can the authors provide a compute-matched comparison that includes the full sampling cost of both OiT branched and classifier-free guidance, with the number of sampling steps stated for each baseline?

3.Does the benefit of nested-prefix velocity correction persist under a matched training configuration and compute budget when compared with a standard full-width residual head?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Gz2g

We thank the reviewer for the constructive feedback and for recognizing the motivation and competitive results of OiT. We address the three concerns below with additional experiments and clearer sampling-cost reporting.


W1 / Q1: Robustness across training seeds

To assess variability across training seeds within the available rebuttal time window, we resumed OiT-XL from the 220-epoch checkpoint using three different training seeds and trained all three runs to the same endpoint.

RunFID↓IS↑
Original1.55304.2
Run 11.5532304.21
Run 21.5488305.76
Run 31.5492304.97
Mean ± SD (runs 1–3)1.5504 ± 0.0024304.98 ± 0.78

The three continuations closely reproduce the reported FID, supporting stable performance across continuation seeds. OiT-XL achieves this competitive result with 624M parameters and 300 GFLOPs per forward, compared with 953M parameters, 364 GFLOPs, and FID 1.57 for AsymFlow-H/16.

To further evaluate OiT beyond ImageNet, we followed PixelGen's T2I setting and evaluated a 1.0B OiT model on GenEval at 512×512:

ModelParamsSingleTwoCountingColorsPositionColor attr.Overall↑
PixNerd1.2B0.970.860.440.830.710.530.73
PixelGen1.1B0.990.880.590.900.700.700.79
PixelDiT-T2I1.3B1.000.940.700.900.530.650.78
OiT (ours)1.0B1.000.900.620.900.750.730.82

OiT achieves the highest overall score with the fewest parameters in this comparison, improving on PixelGen in five categories and matching its Colors score. These results complement the consistent ImageNet continuations by demonstrating OiT's effectiveness in text-to-image generation.


W2 / Q2: Complete sampling cost and few-step comparisons

The reported OiT GFLOPs include the shared encoder and both decoders, and we estimate sampling costs by multiplying the per-forward cost by the sampler's evaluation count and doubling for CFG.

ModelGFLOPs/forwardSamplerStepsGuidanceEst. TFLOPs/imageFID↓
PixelREPA-B/1650Heun50Yes9.903.17
JiT-L/16176Heun50Yes34.852.79
OiT-B74Heun50Yes14.572.52
PixelDiT-XL/16311FlowDPMSolver100Yes62.201.61
JiT-H/16363Heun50Yes71.871.86
AsymFlow-H/16364Heun50Yes72.071.57
OiT-XL300Heun50Yes59.401.55

At comparable sampling cost, OiT-XL achieves FID 1.55 versus 1.61 for PixelDiT-XL/16, using 59.40 versus 62.20 estimated TFLOPs/image. We will update the complete main comparison table in the revised manuscript with sampling configurations and cumulative sampling costs for all baselines.

We also expanded Table 6(b) to 5, 10, and 20 steps for OiT-B, JiT-B/16, JiT-L/16, and PixelREPA-B/16:

Model5 steps (FID↓ / IS↑)10 steps (FID↓ / IS↑)20 steps (FID↓ / IS↑)
JiT-B/1610.32 / 211.67.17 / 243.15.80 / 262.2
JiT-L/1610.01 / 208.36.56 / 246.74.83 / 271.0
PixelREPA-B/1610.31 / 213.36.45 / 262.84.89 / 288.2
OiT-B8.85 / 242.75.42 / 289.14.12 / 282.4

OiT-B achieves the lowest FID at all three step counts and the highest IS at 5 and 10 steps. At 10 steps, OiT-B also improves on JiT-B/16 at 20 steps (FID 5.42 vs. 5.80).


Q3: Controlled comparison with full-width correction

Following the reviewer's suggestion, we completed the two additional controls and expanded Table 4(a):

Nested maskingCorrection inputFID↓IS↑
NoNone5.84206.7
NoFull width5.81208.8
NoFirst 8 channels (new)5.84207.0
YesNone5.77209.9
YesFull width (new)5.75210.6
YesFirst 8 channels (ours)5.68211.4

Without nested masking, restricting the input to 8 channels gives no FID improvement. With nested masking, the compact prefix achieves the best FID and IS, outperforming full-width correction both with and without masking. Compared with the standard full-width baseline, FID improves from 5.81 to 5.68 and IS from 208.8 to 211.4. These results support the combination of ordered representation learning and compact-prefix velocity correction.

We sincerely thank the reviewer again for the thoughtful and constructive feedback. Your suggestions have directly helped us strengthen the evidence for training stability, make sampling-cost comparisons more transparent, and isolate the contribution of nested-prefix correction. We hope these additions address your concerns, and we are grateful for the time and care you have devoted to helping us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.