acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Estimate: Wiener-Informed Decoding for Easier Training of Pixel-Space Flow Models

Abstract

Learning velocity fields directly in pixel space is difficult and slow. We show that coarse second-order image statistics can ease this task: frozen Wiener estimates improve velocity learning, with gains extending beyond the linear predictions they explicitly provide. Moreover, we observe similar but complementary gains from frozen bases and pixel decoding. These observations motivate Wiener-informed Decoding (WiD), which uses statistical knowledge both for direct prediction and to organize decoder computation without modifying native blocks. WiD combines a trainable Wiener prediction path with coefficient interfaces that inject features into the native decoder and return its predicted coefficient corrections, with these components initialized with data statistics. All introduced parameters, including the spectrum and coordinate transforms, are trained jointly under the original pixel-space velocity objective. Across model scales and decoder architectures, WiD accelerates training and improves generation quality with modest parameter overhead. It also yields smoother gradient trajectories and remains effective with representation alignment. On ImageNet 256×256, PixelDiT-XL with WiD reaches FID 1.61 at 260 epochs, matching the baseline's reported quality at 320 epochs with less than 1% additional parameters. We also validate the early gains on text-to-image tasks. These results show that statistical denoisers can contribute beyond the estimate: the computational structure also helps pixel generation learning.

Primary area:
generative models
Keywords:
Pixel-space flow matching, diffusion models, generative modeling, deep learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nLnuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Wiener-informed Decoding (WiD), a method for improving the training efficiency and generation quality of pixel-space flow models by incorporating second-order image statistics into the pixel decoder. Motivated by the observation that frozen Wiener estimators substantially improve velocity prediction, the authors propose a trainable statistical prediction pathway with PCA-initialized coordinate transforms, noise-dependent spectral gains, and coefficient interfaces that interact with the native decoder. The method retains the original decoder blocks and flow-matching objective while introducing a relatively small number of additional parameters. Experiments on ImageNet-1K 256×256 demonstrate that PixelDiT-XL with WiD achieves an FID of 1.61 at 260 epochs, matching the reported baseline performance at 320 epochs with approximately 0.94% additional parameters. The approach also improves DeCo-XXL on text-to-image generation, increasing GenEval from 0.43 to 0.49 and DPG-Bench from 73.1 to 78.8. Additional ablations and diagnostic analyses suggest that statistical initialization and coefficient-space interactions provide complementary benefits.

Critical strengths
  1. The paper identifies a meaningful difficulty in pixel-space velocity learning and connects it to classical Wiener estimation. The progression from frozen analytic predictors to trainable decoder components is intuitive and supported by preliminary experiments. This provides an interesting connection between traditional image estimation and modern flow-based generative modeling.

  2. WiD introduces statistical prediction and coefficient interfaces without modifying the internal blocks of the original decoder. The initialization strategy is particularly well designed: it preserves the native decoder's initial input representation while initializing the overall velocity prediction with a Wiener estimate. This makes the method relatively straightforward to integrate into existing architectures.

  3. The improvements on PixelDiT and DeCo suggest that WiD is not restricted to a particular decoder implementation. The ImageNet results demonstrate faster progress toward competitive FID values, while the text-to-image experiments provide additional evidence of practical relevance. The gains on GenEval and DPG-Bench are particularly encouraging.

  4. The ablations indicate that statistical initialization, direct Wiener prediction, and coefficient interfaces each contribute to the overall performance. The parameter and FLOP increases are also relatively small, making the approach attractive for integration into existing pixel-space generation systems.

Critical weaknesses
  1. The paper argues that Wiener priors facilitate learning beyond their explicit linear predictions. However, the error decomposition measures agreement with a pretrained PixelDiT-XL reference rather than directly evaluating the learned velocity against its training target. Furthermore, the remaining component can still contain linear dependencies outside the selected affine fitting family, as acknowledged in Appendix B.1. Consequently, the reported improvements in the remaining error do not necessarily establish improved nonlinear representation learning. The diagnostic analysis also focuses primarily on frozen Wiener bases rather than the final WiD architecture.

  2. The experimental design does not sufficiently isolate Wiener-specific benefits from more general preconditioning and additional model capacity. WiD combines statistical initialization, trainable analysis and synthesis transforms, time-dependent coefficient scaling, feature injection, and an additional prediction head. Although individual ablations are provided, it remains unclear how much of the improvement specifically requires Wiener estimation. Comparisons with parameter-matched decoder extensions, generic trainable residual pathways, and simpler preconditioning schemes would help establish whether the proposed statistical structure is essential.

  3. The highlighted result of FID 1.61 at 260 epochs uses different CFG weights and guidance intervals from the reported PixelDiT-XL baseline at 320 epochs. Appendix A.3 shows that the 260-epoch WiD checkpoint obtains FID 1.65 under CFG 2.75, while the reported FID 1.61 uses CFG 3.25 with a different guidance interval. Although selecting favorable inference configurations is reasonable, a matched evaluation protocol would better isolate the contribution of the training method. Moreover, the paper does not report whether the improvements persist when WiD is trained to the same 320-epoch budget or beyond.

  4. The paper primarily measures efficiency through training epochs rather than end-to-end training time or total computational expenditure. Appendix A.3 reports approximately 1.33% additional forward FLOPs and reduced inference throughput, with observed slowdowns of roughly 5–6%. These overheads are relatively small, but they highlight the importance of reporting training time and quality-versus-compute curves. The current results support improved convergence in epochs more directly than actual wall-clock training efficiency.

Questions for authors
  1. Can the authors provide direct comparisons between PixelDiT-XL and PixelDiT-XL+WiD under identical CFG weights, guidance intervals, and sampling settings? It would also be useful to compare both methods at the same 320-epoch budget and report training time versus generation quality.

  2. How does WiD compare with simpler alternatives, such as a frozen Wiener residual connection, a trainable Wiener branch without coefficient interfaces, a parameter-matched additional decoder head, or generic preconditioning? These comparisons would help identify the essential source of the improvement.

  3. Does WiD remain effective at higher resolutions, such as 512×512, or under different patch sizes and image distributions? Additionally, the text mentions an ablation removing Wiener gain, but the corresponding numerical result is not included in Table 3. Could the authors provide this result?

5 comments
AuthorsOpenReview ↗

Response to Reviewer nLnu (1/3)

Thank you for the comments. We have added reference-independent diagnostics, controls, and extended experiments to address these concerns. We have also revised the manuscript accordingly.

W1: The error decomposition and new diagnostics.

The original analysis. We agree that the remaining prediction is not necessarily purely nonlinear: it excludes the chosen affine family but may retain other linear dependencies. Our claim is that gains can extend beyond the explicit additive prediction, not that the decomposition proves improved nonlinear representations. At each fixed t, our affine fits can represent the corresponding frozen base exactly. Therefore, subtracting the fitted component removes the base’s direct additive contribution. The smaller remaining error reflects closer agreement of the learned remainder with the reference.

Direct validation against flow targets. We now compare complete predictions against v=x−ϵv=x-\epsilon. Although the sampled target differs from the marginal velocity, the expected MSE against this target equals the marginal-velocity MSE plus a predictor-independent term. This term cancels when comparing models (Appendix B.3, Eq.20).

We use 3,072 held-out images, two fixed noises per image, and five fixed time steps, with EMA and no CFG. We report mean MSE reductions in units of 10−310^{-3} in the table below.

Comparison10 epochs40 epochs
DiT + PCA base vs. DiT18.8413.50
DiT + DCT base vs. DiT19.5513.85
Native PixelDiT vs. DiT10.1915.94
PixelDiT + PCA base vs. PixelDiT10.611.47
PixelDiT + WiD vs. PixelDiT23.024.40

All five comparisons favor the added base, decoder, or WiD at both checkpoints (Table 11). This validates prediction improvement independently of the chosen reference.

Analysis of trained WiD. We examine the direct statistical prediction, coefficient correction, and native pixel output separately. At 40 epochs, the direct path's MSE is 0.24440. Adding the coefficient correction reduces it to 0.11797, and the full output reaches 0.09864. Removing the direct statistical, coefficient-correction, or native pixel term from the trained output, with all other computation held fixed, increases MSE by 1.14861, 0.10750, and 0.01933, respectively. These measurements show that all three output terms contribute to the combined prediction (Appendix B.4, Table 12).

AuthorsOpenReview ↗

Response to Reviewer nLnu (2/3)

W2 & Q2: Comparisons with simpler alternatives.

We added two controls on native PixelDiT-B/16. All use 40 epochs, global batch 256, REPA 0.5, and EMA/no-CFG FID-50K with FlowDPMSolver-100:

MethodFID
Native PixelDiT-B/1660.01
+ time-conditioned linear skip48.46
+ scalar flow preconditioning56.18
+ WiD41.71

Time-conditioned linear skip. For each noisy patch zt,i∈R768z_{t,i}\in\mathbb R^{768}, we add

ri(zt,i,t)=Bdiag⁡(1+g(t))Azt,ir_i(z_{t,i},t)=B\operatorname{diag}(1+g(t))Az_{t,i}

to the corresponding native velocity output. The trainable matrices A∈R4096×768A\in\mathbb R^{4096\times768} and B∈R768×4096B\in\mathbb R^{768\times4096} are shared across patches. The time modulation g(t)g(t) is produced by a 256→256→4096256\to256\to4096 MLP with SiLU. We randomly initialize AA and zero-initialize BB and the MLP's last layer. This branch is linear in the image input at fixed tt, and adds 7.41M parameters versus WiD's approximately 7.48M.

Scalar flow preconditioning. We adapt EDM's scalar input, skip, and output preconditioning [1] to our linear-flow velocity parameterization. From the same patch second-moment matrix used by WiD, we compute σx2=tr⁡(M^patch)/768\sigma_x^2=\operatorname{tr}(\widehat M_{\rm patch})/768 and set qt=t2σx2+(1−t)2q_t=t^2\sigma_x^2+(1-t)^2. The prediction is

v^(zt,t,y)=tσx2−(1−t)qtzt+σxqtfθ(ztqt,t,y),\hat v(z_t,t,y)=\frac{t\sigma_x^2-(1-t)}{q_t}z_t+\frac{\sigma_x}{\sqrt{q_t}}f_\theta\left(\frac{z_t}{\sqrt{q_t}},t,y\right),

where fθf_\theta is native PixelDiT and yy is the class label. This combines an isotropic Wiener skip with noise-dependent input and output scaling, without adding trainable parameters.

The frozen additive-base comparison is already shown separately without REPA in Fig.1(c): its early gains narrow with training. In Table 5, we have tested the trainable statistical path without coefficient interfaces, which also underperforms full WiD. Random initialization of UencU_{enc} and UdecU_{dec} gives FID 43.99, while replacing only the empirical spectrum with ones gives FID 48.92. Both are worse than full WiD's 41.71. Together, these controls support the value of statistical initialization and coefficient interaction.

[1] Karras et al. Elucidating the Design Space of Diffusion-Based Generative Models.

AuthorsOpenReview ↗

Response to Reviewer nLnu (3/3)

W3 & Q1: Extended training and comparable settings.

Our original comparison reported tuned generation quality. CFG scale and guidance interval are inference hyperparameters. As discussed in JiT [1], the best CFG scale can vary across model configurations. We agree that shared-setting evaluation complements reporting tuned performance, and we are pleased to provide additional results in the following.

Shared-setting evaluation at 256 x 256. We now report both models under CFG 4.0 and interval [0.14, 0.55], with EMA, FlowDPMSolver-100, and 50K samples:

ModelEpochsFID
PixelDiT-XL/163201.58
PixelDiT-XL/16 + WiD2601.59
PixelDiT-XL/16 + WiD3201.58

Under identical guidance settings, WiD reaches comparable quality with 18.75% fewer training epochs and matches the baseline at 320 epochs. These results correspond with our claim of accelerated training with comparable quality retained. These comparisons are included in Tables 1 and 7.

Longer training. At 500 epochs, WiD reaches FID 1.53, comparable to PixelDiT's officially reported 1.54 at 800 epochs, with 37.5% fewer epochs. We use the published baseline result because its 800-epoch checkpoint is not available. This longer-training comparison further supports the benefit in training efficiency.

Higher-resolution support (also Q3). At 512 x 512, WiD reaches FID 1.68 using the baseline's reported CFG 3.5 and interval [0.10, 1.00], compared with its published 1.81. Keeping CFG at 3.5 and adjusting only the interval to [0.10, 0.90] further lowers WiD's FID to 1.64. Our method uses 260 epochs of 256-resolution pretraining plus 50 epochs of 512-resolution fine-tuning, while the baseline reports 320 + 530 epochs. This provides additional support for the training-efficiency benefit under the baseline's own reported guidance settings.

W4 & Q1: Training time estimations.

With 8 H200 GPUs, we obtain median update times of 0.174s for native PixelDiT-XL/16 and 0.183s with WiD on ImageNet 256×256. Extrapolating the rates, we estimate 66.0 hours for PixelDiT-XL/16+WiD at 260 epochs versus 77.1 hours for the native model at 320 epochs, a 14.4% saving. For the longer-training comparison, the estimates are 127.0 hours for PixelDiT-XL/16+WiD at 500 epochs versus 192.9 hours for the native model at 800 epochs, accounting for a 34.2% saving.

Q3: Patch size and image distributions.

Patch size. These experiments use B-size models: native PixelDiT-B and PixelDiT-B + WiD, each trained for 40 epochs with global batch 256 and REPA 0.5. WiD lowers FID at all three patch sizes:

Patch sizeNative PixelDiT-B+ WiD
846.5332.83
1660.0141.71
3282.9275.27

Text-to-image training. We initialize DeCo-XXL/16 + WiD using statistics from 51,200 images in the 256-resolution pretraining set. Following DeCo's three-stage recipe, we train for 200K steps at 256 resolution, continue for 80K at 512, and fine-tune for 40K on BLIP3-o-60K at 512. WiD remains effective after both resolution and training-data changes, with its parameters adapting jointly with the network.

At 512×512, our method achieves GenEval 0.86 and DPG-Bench 82.6, versus 0.82 and 82.3 from our evaluation of the official DeCo-XXL/16 checkpoint, strictly following its official sampling settings. We report detailed results in Table 4.

DeCo officially reports GenEval 0.86 and DPG-Bench 81.4. Our reevaluated GenEval 0.82 is also consistent with independent studies: RaPD [2] and DMA² [3]. These studies likewise obtain GenEval around 0.82–0.83. Our results extend WiD's validation to another decoder architecture and image-text distribution, including subsequent higher-resolution training and fine-tuning.

Q3: Numerical result for removing the Wiener gain.

Thank you for pointing out the ambiguous wording. This referred to removing the direct Wiener prediction, not setting the spectrum to one. The row labeled "w/o Wiener prediction" gives FID 49.54, sFID 10.88, and IS 29.9, versus full WiD's 41.71, 9.18, and 35.1. We have aligned the text with this row name in Sec.5.3, Table 5.

[1] Tianhong Li and Kaiming He. Back to Basics: Let Denoising Generative Models Denoise.

[2] Ge et al. RaPD: Resolution-Agnostic Pixel Diffusion via Semantics-Enriched Implicit Representations.

[3] Lin et al. DMA²: Pixel-space Distribution Matching with Adversarial and Anchor Losses.

Reviewer_kDeCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the difficulty of velocity prediction and inefficient training in pixel-space flow models. The authors observe that adding a frozen DCT/PCA-based Wiener velocity estimate substantially improves plain DiT training; after subtracting fitted affine responses, the remaining predictions also agree more closely with a well-trained reference model; and the direct pixel pathway is, on average, more aligned with Wiener responses than the contextual pathway. Motivated by these observations, they propose Wiener-informed Decoding (WiD), which integrates a PCA-initialized, trainable statistical prediction path into the pixel decoder through coefficient inputs and coefficient-correction outputs. On ImageNet, WiD reduces FID from 2.36 to 2.09 at 80 epochs and reaches FID 1.61 at 260 epochs, matching the baseline at 320 epochs. For text-to-image generation, it improves GenEval from 0.43 to 0.49 and DPG-Bench from 73.1 to 78.8.

Critical strengths
  1. A simple, well-motivated idea. Incorporating the classical Wiener estimation pipeline of transformation, coefficient scaling, and reconstruction into the pixel decoder offers a clear and practical way to improve generative-model training using image statistics.
  2. Low overhead and good compatibility. On PixelDiT-XL, WiD increases parameters and theoretical FLOPs by only approximately 0.94% and 1.33%, respectively. It works with both PixelDiT and DeCo decoders and is compatible with REPA. The initialization preserves the native decoder's input features while making the initial total output equal to the PCA-based Wiener prediction.
  3. Early-training gains across multiple settings. Improvements are observed at B and XL/XXL scales, across two decoder architectures, and in both class-conditional and text-conditional generation. The substantial improvement from adding a frozen statistical base to plain DiT in Figure 1(a) is also noteworthy.
Critical weaknesses
  1. The evidence for training efficiency is not yet sufficient. The main benefit is reaching an FID comparable to the baseline with fewer training epochs, but the sampling settings are not fully matched. For example, in Table 5, WiD reaches FID 1.61 using CFG 3.25 over [0.12, 0.70], whereas the baseline uses CFG 2.75 over [0.10, 0.90]. Actual wall-clock training times are also not compared. A fairer evaluation would help establish the extent of the speedup.
  2. The method's distinct contribution remains unclear. Controlled comparisons with simpler alternatives, such as statistical prediction or preconditioning, are missing. One example would be adding a simple trainable linear skip connection to the same backbone. It is therefore difficult to determine whether the gains depend on WiD's specific design.
  3. Ablation and robustness evidence is limited. Ablations focus on small models early in training, and results across multiple random seeds are not reported. The contributions of key components and the consistency of their benefits at larger scales and later training stages remain to be verified.
  4. The text-to-image validation is preliminary. Experiments cover only low-resolution generation and short training durations. They do not establish whether the gains persist through convergence or at higher resolutions, and overall image quality is not sufficiently evaluated.
Questions for authors

See "Weakness".

5 comments
AuthorsOpenReview ↗

Response to Reviewer kDeC (1/3)

Thank you for the comments. We have added shared-setting evaluations, simpler controls, and extended XL/XXL training to address the four concerns below, and revised the manuscript accordingly.

W1: Matched settings and training-time evidence.

CFG scale and guidance interval are inference hyperparameters. As discussed in JiT [1], the preferred CFG scale can vary across model configurations. Our original comparison reported tuned generation quality. We agree that a shared-setting comparison is also needed to assess training efficiency, and now provide it together with longer-training results.

Shared-setting evaluation at 256 x 256. We evaluate the public PixelDiT-XL/16 checkpoint and WiD using EMA weights, 50K samples, FlowDPMSolver-100, CFG 4.0, and interval [0.14, 0.55] (Tables 1 and 7):

ModelTraining epochsFID
PixelDiT-XL/16, reevaluated public checkpoint3201.58
PixelDiT-XL/16 + WiD2601.59
PixelDiT-XL/16 + WiD3201.58

WiD reaches comparable quality with 18.75% fewer training epochs and matches the baseline at 320 epochs. These results correspond with our claim of accelerated training with comparable quality retained.

Longer training. At 500 epochs, PixelDiT-XL/16+WiD reaches FID 1.53, comparable to PixelDiT's officially reported 1.54 at 800 epochs, with 37.5% fewer epochs. We use the published baseline result because its 800-epoch checkpoint is not available.

Higher-resolution support. At 512 x 512, PixelDiT-XL/16+WiD obtains FID 1.68 using the baseline's reported CFG 3.5 and interval [0.10, 1.00], compared with its published 1.81. WiD uses 260 epochs of 256-resolution pretraining plus 50 epochs of 512-resolution fine-tuning, versus the baseline's reported 320 + 530 epochs. Keeping CFG at 3.5 and adjusting only the interval to [0.10, 0.90] further lowers our FID to 1.64. We include these results in Tables 2 and 7.

Training-time estimate. With 8 H200 GPUs, we obtain median update times of 0.174s for native PixelDiT-XL/16 and 0.183s with WiD on ImageNet 256 x 256. Both use global batch 256, BF16, and REPA 0.5. We estimate training time by extrapolating the rates.

Training comparisonWiD hoursNative hoursEstimated saving
WiD 260 vs. native 320 epochs66.077.114.4%
WiD 500 vs. native 800 epochs127.0192.934.2%

[1] Tianhong Li and Kaiming He. Back to Basics: Let Denoising Generative Models Denoise.

AuthorsOpenReview ↗

Response to Reviewer kDeC (2/3)

W2: Comparisons with alternatives.

We added the suggested trainable linear skip and a scalar preconditioning control on native PixelDiT-B/16. All models use 40 epochs, global batch 256, and REPA 0.5. We evaluate EMA checkpoints without CFG using 50K samples and FlowDPMSolver-100:

MethodFID
Native PixelDiT-B/1660.01
+ time-conditioned linear skip48.46
+ scalar flow preconditioning56.18
+ WiD41.71

Time-conditioned linear skip. For each noisy patch zt,i∈R768z_{t,i}\in\mathbb R^{768}, we add ri(zt,i,t)=Bdiag⁡(1+g(t))Azt,ir_i(z_{t,i},t)=B\operatorname{diag}(1+g(t))Az_{t,i} to the native velocity output. The trainable matrices A∈R4096×768A\in\mathbb R^{4096\times768} and B∈R768×4096B\in\mathbb R^{768\times4096} are shared across patches; g(t)g(t) is a 256→256→4096256\to256\to4096 MLP with SiLU. We randomly initialize AA and zero-initialize BB and the MLP's last layer. The added branch is linear in the image input at fixed tt, and adds 7.41M parameters, close to WiD's 7.48M.

Scalar flow preconditioning. We adapt EDM's input, skip, and output preconditioning [1] to linear-flow velocity prediction. From the same patch statistics used by WiD, we compute σx2=tr⁡(M^patch)/768\sigma_x^2=\operatorname{tr}(\widehat M_{\rm patch})/768 and set qt=t2σx2+(1−t)2q_t=t^2\sigma_x^2+(1-t)^2. The prediction is v^(zt,t,y)=tσx2−(1−t)qtzt+σxqtfθ(ztqt,t,y),\hat v(z_t,t,y)=\frac{t\sigma_x^2-(1-t)}{q_t}z_t+\frac{\sigma_x}{\sqrt{q_t}}f_\theta\left(\frac{z_t}{\sqrt{q_t}},t,y\right), where fθf_\theta is native PixelDiT and yy is the class label.

Both alternatives improve on native PixelDiT, but WiD achieves lower FID than either. These results support its benefit beyond the tested generic linear capacity and scalar normalization (Appendix A.4, Table 8).

W3: Additional Ablations.

We have expanded the B-size component controls and tested different patch sizes. These experiments use the same 40-epoch, REPA-0.5 training and no-CFG FID-50K evaluation setting as W2.

Statistical initialization and trainable coordinates. We separately vary the initial basis and spectrum, and test freezing the statistically initialized transforms:

PixelDiT-B/16 + WiD variantFID
Full WiD41.71
Random orthogonal basis + empirical spectrum43.99
PCA basis + unit spectrum48.92
Statistical initialization, frozen transforms42.95

Replacing either initialization factor worsens FID in this setting (Table 5). Freezing only UencU_{\rm enc} and UdecU_{\rm dec} retains much of the benefit. Allowing the transforms to adapt further improves FID from 42.95 to 41.71 (Appendix A.4, Table 8). We keep them trainable to allow adaptation when data or resolution changes during multistage training.

Patch size. We also compare native PixelDiT-B and WiD at patch sizes 8, 16, and 32 under the same training and evaluation protocol:

Patch sizePixelDiT-B+ WiD
846.5332.83
1660.0141.71
3282.9275.27

WiD lowers FID at all three patch sizes, so the benefit is not specific to the default 16 x 16 patches (Table 8).

Larger models and later training. W1 reports XL results at 320 and 500 epochs and at 512 resolution, and W4 below extends XXL text-to-image training through the full three-stage recipe. These evaluate the complete method beyond the original early-training experiments.

[1] Karras et al. Elucidating the Design Space of Diffusion-Based Generative Models.

AuthorsOpenReview ↗

Response to Reviewer kDeC (3/3)

W4: Text-to-Image Experiments.

Complete multistage training. We have now completed DeCo-XXL/16 + WiD's full three-stage recipe on the BLIP3-o data (Table 4): 200K steps at 256 x 256, 80K further steps at 512 x 512, and 40K fine-tuning steps on BLIP3-o-60K at 512 x 512. WiD is initialized from statistics of 51,200 images in the 256-resolution pretraining set, and its parameters are trained jointly throughout the subsequent higher-resolution training and fine-tuning stages.

Final 512-resolution evaluation. We evaluate the official DeCo-XXL/16 checkpoint strictly following its official sampling settings.

ModelGenEvalDPG-Bench
DeCo-XXL/16 (reported)0.8681.4
DeCo-XXL/16 (reevaluated)0.8282.3
DeCo-XXL/16 + WiD0.8682.6

Our reevaluated GenEval 0.82 is close to independent evaluations: RaPD [1] explicitly reruns the released DeCo model and reports 0.83, while DMA² [2] reports 0.8216 for the official DeCo teacher. WiD improves both scores over the reevaluated baseline. Relative to the published results, WiD matches the reported GenEval score and improves DPG-Bench. We also provide visualizations in Appendix D.

[1] Ge et al. RaPD: Resolution-Agnostic Pixel Diffusion via Semantics-Enriched Implicit Representations.

[2] Lin et al. DMA²: Pixel-space Distribution Matching with Adversarial and Anchor Losses.

Reviewer_13NgOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Wiener-informed Decoding (WiD) to improve the training of pixel-space flow models. WiD initializes a prediction branch using second-order image statistics and connects its coefficient representation to the native pixel decoder through additional input and output projections. The statistical transforms and spectrum are jointly trained with the network under the original velocity objective. Experiments show improvements across PixelDiT and DeCo, including class-conditional and text-to-image generation.

Critical strengths

The paper presents a well-motivated way to incorporate inexpensive statistical knowledge into generative model training. The observation that a frozen Wiener estimate already improves velocity learning is compelling, and WiD extends this idea with consistent empirical gains. On PixelDiT-B, WiD improves FID from 60.01 to 41.71, while component ablations support the usefulness of statistical initialization, direct prediction, and decoder interaction. Improvements across two decoder architectures and compatibility with REPA further strengthen the practical contribution.

Critical weaknesses

The large-scale training comparison is incomplete. WiD reaches the reported 320-epoch PixelDiT FID after 260 epochs, but its own 320-epoch result is not provided. Reaching the same quality earlier is useful evidence of acceleration, yet an equal-budget comparison is needed to understand whether the advantage persists later in training. Furthermore, the exact FID match uses different CFG settings: with the baseline’s listed settings, WiD achieves 1.65 rather than 1.61. The headline comparison therefore combines differences in training duration and inference configuration.

The paper provides limited analysis of what the Wiener branch learns after initialization. Sections 3.2–3.3 analyze frozen Wiener bases and the input pathways of native PixelDiT, rather than the trained WiD branches. Since the analysis and synthesis transforms evolve independently without orthogonality constraints, the final branch need not retain its initial statistical interpretation. The ablations establish that the design helps, but do not explain whether the trained branch continues to provide a statistical velocity estimate or becomes a more generic learned prediction path. Examining its output, its contribution relative to the native pixel head, and changes in the learned spectrum would help connect the proposed explanation to the final model. The random-initialization ablation also changes both the basis and spectrum, leaving their individual contributions unresolved.

Questions for authors
  1. Why does the large-scale WiD experiment stop at 260 epochs? If a 320-epoch checkpoint is available, does its improvement over PixelDiT persist under comparable CFG settings?

  2. Have the authors examined the trained Wiener branch separately from the native pixel output? In particular, does it retain a role resembling the initial statistical estimator, and how do the learned transforms and spectrum change during training?

  3. Do existing ablations distinguish the benefit of the PCA basis from that of the empirical spectrum, or establish whether learning the transforms is necessary?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 13Ng (1/2)

Thank you for the comments. We have extended the training experiments, added diagnostics of the trained WiD model, and revised the manuscript accordingly.

W1 & Q1: Extended training and comparable settings.

We have extended the large-scale experiment beyond the originally reported 260 epochs and now provide results at both 320 and 500 epochs.

Why adjust CFG? CFG scale and guidance interval are inference hyperparameters. As discussed in JiT [1], the best CFG scale can vary across model configurations. Our original comparison reported tuned generation quality. We additionally provide a shared-setting comparison to assess the trained models under identical guidance settings.

Shared-setting evaluation at 256 x 256. We evaluate the public PixelDiT-XL/16 checkpoint and WiD with EMA weights, 50K samples, FlowDPMSolver-100, CFG 4.0, and interval [0.14, 0.55]:

ModelTraining epochsFID
PixelDiT-XL/163201.58
PixelDiT-XL/16 + WiD2601.59
PixelDiT-XL/16 + WiD3201.58

Under identical guidance settings, WiD reaches comparable quality with 18.75% fewer training epochs and matches the baseline at 320 epochs. These results support our claim of accelerated training, with comparable quality retained at the equal training budget. We have added these comparisons to Tables 1 and 7.

Longer training. At 500 epochs, WiD reaches FID 1.53, comparable to PixelDiT's officially reported 1.54 at 800 epochs, with 37.5% fewer epochs. We use the published baseline result because its 800-epoch checkpoint is not available. This longer-training comparison further supports the benefit in training efficiency.

Higher-resolution support. Higher-resolution results also support this training-efficiency benefit. At 512 x 512, WiD obtains FID 1.68 using the baseline's reported CFG 3.5 and interval [0.10, 1.00], compared with the its published 1.81. WiD uses 260 epochs of 256-resolution training and only 50 epochs of 512-resolution fine-tuning, while the baseline reports 320 + 530 epochs. Tables 2 and 7 report the corresponding results and guidance settings.

[1] Tianhong Li and Kaiming He. Back to Basics: Let Denoising Generative Models Denoise.

AuthorsOpenReview ↗

Response to Reviewer 13Ng (2/2)

W2 & Q2: The trained Wiener branch.

We now examine the final WiD architecture. The direct statistical path at fixed time is the patch-linear operator

Kt=Udecdiag⁡(a(t;λ))Uenc⊤.K_t=U_{\rm dec}\operatorname{diag}(a(t;\lambda))U_{\rm enc}^{\top}.

It adapts while remaining close to its statistical initialization. At 40 epochs, the relative Frobenius changes in the encoder and decoder transforms are 16.3% and 15.5%, and the RMS change in log-spectrum parameters is 0.757. Because the transforms evolve independently, these parameters need not remain covariance eigenvalues. We therefore also inspect the effective operator. Across five time steps, its relative change is 3.2%-19.9%, while its matrix cosine with the initial operator is 0.985-0.999. Subtracting each operator's scalar-identity component still gives cosines of 0.959-0.990. Thus, the learned path remains closely aligned with its statistical initialization while adapting to the jointly learned prediction.

Its role is joint prediction. We evaluate EMA checkpoints at 10 and 40 epochs on 3,072 held-out images, two fixed noises per image, and five time steps, without CFG. At 40 epochs, the mean per-coordinate flow-target MSE is:

Prediction readoutMSE
Initial fixed Wiener estimate0.21174
Learned direct statistical path0.24440
Statistical path + coefficient correction0.11797
Full output, including the native pixel head0.09864

The direct path is optimized jointly with the coefficient correction and native pixel output. Although its standalone MSE increases, the combined prediction becomes substantially more accurate. Removing the direct, coefficient-correction, or pixel output after training, while keeping all other computation unchanged, increases MSE by 1.14861, 0.10750, and 0.01933, respectively. Both checkpoints and every tested time show a positive removal cost. We report details in Appendix B.4 and Table 12.

Q3: Separate contributions of the PCA basis, empirical spectrum, and trainable transforms.

We added controls that change one initialization factor at a time. The following PixelDiT-B/16 models use 40 epochs, global batch 256, REPA weight 0.5, and we report no-CFG FID-50K from EMA checkpoints with FlowDPMSolver-100:

Initialization / constraintFID
PCA basis + empirical spectrum, full WiD41.71
Random orthogonal basis + empirical spectrum43.99
PCA basis + unit spectrum48.92
Random orthogonal basis + unit spectrum49.02
PCA basis + empirical spectrum, frozen encoder/decoder transforms42.95

The first four rows separate the initialization factors: replacing the statistical initialization of either the basis or the spectrum worsens FID in this setting. These results are reported in Table 5.

Freezing the transforms retains much of WiD's benefit, and learning the transforms yields a further improvement from 42.95 to 41.71. Our choice to keep the transforms trainable is motivated by practical multistage training of modern models, where data, resolution, or objectives may change. Our additional text-to-image experiments in Table 4 provide an example of such a training pipeline. This choice allows the coordinates to evolve with the network instead of fixing them to the initial data basis. This design rationale and the frozen-transform control are added in Appendix A.4, Table 8.

Reviewer_13NgOpenReview ↗

Thanks for the rebuttal

I appreciate authors for the rebuttal. Especially, the Wiener branch diagnosis clarifies the design choice, and I like the new ablations showing the effect of PCA basis. While I do believe the apples-to-apples comparison with JiT is not arguably better in terms of training efficiency as 260 to 320 epochs does not really change FID score and the score gap is too narrow, I value this paper's work on it's method and analysis. I don't have any further concerns, and will maintain my positive score.

Thank you for your thoughtful assessment and encouraging feedback. We appreciate your recognition of the Wiener branch diagnosis and the ablations, and we understand your perspective on the efficiency comparison. Your feedback helps us further strengthen the analysis and comparison. For example, we provide 500-epoch and 512-resolution results as additional evidence. Thank you again for your time and careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.