acceptodds
Under review as a conference paper at ICLR 2027

Exploring the Design Factors of Pixel-Space Diffusion

Abstract

In this paper, we introduce EiT by exploring the key design factors of pixel-space diffusion from three perspectives: model architecture, representation, and frequency domain. Inspired by Denoising Autoencoders (DAE) which performs denoising through a bottleneck, we construct a U-shaped pixel diffusion transformer that performs explicit compression along the channel dimension. In terms of representation, we find that both pixel and latent diffusion transformers exhibit a three-stage representation dynamics across layers and timesteps: denoising/local feature expansion, basis lift-up, and posterior projection. Moreover, earlier denoising and a longer basis lift-up stage are associated with better generation, which EiT naturally achieves through its channel bottleneck and increased depth. From the frequency perspective, we establish the equivalence between pixel-space and frequency-space diffusion, and show that the latter enables flexible frequency-band loss reweighting, yielding performance gains at no extra cost. On ImageNet 256256 and 512512, EiT achieves significantly better generation quality than JiT (Just image Transformers). Our code will be made publicly available.

Primary area:
generative models
Keywords:
diffusion models, pixel diffusion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_g8ZUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EiT, a U-shaped Transformer with channel compression. The authors analyze three-stage representation dynamics using SPR and Noise Fraction, and introduce a DWT-based frequency-weighted loss. Experiments show that EiT outperforms JiT and some PixelU configurations on ImageNet-256/512, and also improves performance in latent diffusion.

Critical strengths
  1. Channel compression is a simple and reasonable design. With comparable parameter counts and computational budgets, EiT achieves clear improvements over JiT, showing the practical value of this approach.

  2. The experiments cover ImageNet-256/512, different model sizes, and latent diffusion. The paper also includes ablations on different channel compression configurations and frequency loss weights, making the evaluation relatively comprehensive.

  3. The analysis using SPR and Noise Fraction offers a new perspective on how representations evolve inside diffusion Transformers. This is an interesting direction worth exploring further.

Critical weaknesses
  1. The contribution of channel compression is not clearly isolated. Compared with JiT and PixelU, EiT changes not only the compression dimension but also the network depth and skip connection design. For example, EiT-B uses 24 layers, while PixelU-B has only 12. Without comparisons at matched depth and compute, it is unclear whether the improvements mainly come from channel compression or from the deeper network and other architectural changes.

  2. The evidence for the three-stage representation dynamics is not fully convincing. A lower Noise Fraction does not necessarily mean better denoising, and a higher SPR does not always indicate more useful features. The lack of comparison with PixelU also makes it unclear whether these patterns are specific to channel compression or common to U-shaped architectures.

  3. The novelty and empirical gains of the frequency-domain component are relatively limited. The equivalence between pixel-space and frequency-space diffusion mainly follows from the fact that orthogonal transforms preserve the L2 norm, while the DWT loss essentially gives more weight to low-frequency errors. Similar frequency-weighted losses have already been used in DeCo, so the novelty over prior work is unclear. The FID improvements are also relatively small.

Questions for authors
  1. Could you directly compare channel compression, spatial compression, and a Transformer without compression under matched depth, similar FLOPs, and the same skip connection design? This would help clarify the core architectural contribution of EiT.

  2. Could you compare the layer-wise SPR and Noise Fraction of EiT and PixelU under comparable settings? This would help clarify whether earlier denoising and longer basis lift-up are specific to channel compression.

  3. What are the key differences between EiT's DWT loss and DeCo's Frequency-Aware FM loss? A comparison using the same model and training setup would help determine whether the DWT loss achieves better results.

Reviewer_fQtPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EiT, a U-shaped pixel diffusion Transformer that compresses the channel dimension while preserving the spatial token grid. The authors also investigate the representation dynamics of diffusion Transformers and introduce frequency-domain loss reweighting to improve generation quality. Experiments on ImageNet 256×256 and 512×512 show improvements over JiT and PixelU under comparable computational budgets.

Critical strengths
  1. The proposed architecture is simple and achieves promising generation results, particularly under comparable parameter and FLOP budgets.

  2. The representation analysis provides interesting observations about denoising and feature evolution in diffusion Transformers.

Critical weaknesses
  1. The main experiments primarily compare EiT with JiT and PixelU, while several recent methods, such as DeCo, DiP, PixelDiT, and Simpler Diffusion, are not included in the quantitative comparisons. Given the rapid progress in pixel-space generation, it is difficult to assess how competitive EiT is against more recent approaches.

  2. Although the paper reports parameter counts and theoretical GFLOPs, it does not provide actual training time, inference latency, or throughput comparisons. EiT also uses more Transformer blocks than JiT, so lower FLOPs do not necessarily translate into faster execution.

Questions for authors
  1. Can the authors provide comparisons with more recent pixel-space diffusion methods, or explain why these methods are excluded from the quantitative evaluation?

  2. How do EiT and JiT compare in actual training throughput, wall-clock convergence time, and sampling latency on identical hardware?

  3. The proposed three-stage representation dynamics are interesting, but the interpretation of increased effective rank as "basis lift-up" remains somewhat speculative. Is there more direct evidence connecting this behavior to improved posterior prediction and generation quality?

Reviewer_fEveOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EiT, a U-shaped pixel-space diffusion Transformer with channel compression, analyzes the three-stage representation dynamics of diffusion models, and introduces frequency-aware loss re-weighting to improve generation quality and computational efficiency.

Critical strengths
  1. Effective and efficient architecture design. The proposed channel-compression strategy is simple yet effective, consistently outperforming JiT and PixelU under comparable parameter and FLOP budgets on ImageNet 256×256 and 512×512. The improvements in both generation quality and training efficiency demonstrate its practical value.

  2. Insightful analysis of diffusion representations. The paper provides an interesting analysis of representation dynamics in diffusion transformers through Spatial Participation Ratio and Noise Fraction, revealing a three-stage pattern shared by pixel-space and latent-space diffusion models.

Critical weaknesses
  1. Insufficient isolation of the architectural contribution. The proposed channel-compression strategy simultaneously changes network depth and introduces U-shaped skip connections. Although EiT achieves better FID under comparable parameter and FLOP budgets, it remains unclear whether the improvements primarily come from channel compression, increased depth, or skip connections. More controlled ablations are needed to isolate the contribution of each component.

  2. Limited evidence supporting the representation dynamics interpretation. The proposed three-stage representation dynamics are primarily characterized by Spatial Participation Ratio (SPR) and Noise Fraction. However, a higher SPR does not necessarily indicate more useful representations for posterior prediction, and the observed correlations do not establish causality. Stronger evidence is needed to validate whether earlier denoising and longer basis lift-up stages directly contribute to improved generation quality.

Questions for authors
  1. Clarification on the architectural contribution. Since EiT simultaneously introduces channel compression, increased depth, and U-shaped skip connections, can the authors clarify which component contributes most to the performance improvement? Could controlled ablations with matched depth and computational budgets help isolate the benefits of channel compression itself?

  2. Validation of the representation dynamics. The paper interprets higher SPR as basis lift-up and associates longer lift-up stages with better generation quality. Can the authors provide more direct evidence, such as layer-wise linear probing or posterior prediction error, to demonstrate that these high-dimensional representations contain more useful information for generation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.