acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing Pixel-Space Diffusion Transformers: From Depth-Wise Connectivity to Layer-Wise Attention Patterns

22 views

Abstract

Recent studies have demonstrated the potential of Diffusion Transformers (DiTs) in pixel space. However, pixel-space models still suffer from slow convergence, high training costs, and the lack of native training at high resolutions. In previous work, fine-tuning a 1.0B text-to-image model at resolution in pixel space required industrial-scale cluster resources. In this work, we diagnose two potential factors underlying the inefficiency of pixel-space DiTs: depth-wise connectivity and layer-wise attention patterns, and propose the Adaptive Connection Image Transformer (AceiT). AceiT achieves reliable convergence with as little as 7 GFLOPs per sample, reducing the computational cost to one quarter of JiT-B/16 and demonstrating substantial gains in computational efficiency. Meanwhile, AceiT achieves faster convergence, alleviating the reliance on representation alignment (e.g., REPA) while remaining free from additional network components (e.g., pixel decoders), thereby offering better simplicity and scalability. More importantly, with only A100 GPUs, AceiT can be pretrained at resolution with up to 2.1B parameters, making native high-resolution training from scratch in pixel space feasible. We will release our code to facilitate further research and development in the community.

Primary area:
generative models
Keywords:
Image Generation, Diffusion Models, Generative Models

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_w8VmOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a new type of Image transformers called AceiT. The problem with the diffusion generation is that UNets are not scalable while providing high quality results, while Image transformers are computationally expensive. This paper uses adaptive attention residuals to make it computationally feasible to use pixel level image transformers. While increasing scalability with chunk downsampling.

Critical strengths

1- The paper is very well written, it motivates the problem at hand and considers the challenges that is associated with it.

2- The adaptive attention residuals to overcome pixel space inefficiencies and limiting the scalability of conventional skip connections.

3- Query sharing and hybrid residuals improves generation quality while reducing computation and memory.

4- Chunk downsampling and adaptive attentions allows for arbitrary compression ratios, hence ensuring the scalability of the model.

Critical weaknesses

(Minor)

1- How would the method works for different resolutions and it is required for a method that is scalable to show performance in different resolutions.

2- While FID improves, it is a very broad measure and GenEval is not broken down to its subtask, so it is not clear how the proposed solution will affect the downstream tasks.

Questions for authors

1- In table 5 DeCo-XXL/16 has a quite similar performance to the proposed model but with half the parameters. Is the throughput has considered the fact that due to less memory, two of that model can be used parallelly on the same resource as AceiT.

2- How would performance of AceiT differ for different resolutions?

3- How AceiT affect downstream tasks, and how would latent space models in Table 5 compare based on FID and IS to the method?

Suggestions:

1- SDXL and Flux 1 are not very recent and currently SD3.5 and Flux2 are the most recent versions. I believe since AceiT throughput is much better than those models while having much less parameters, it would be good to also include them, since I believe the GenEval performance for those models still is less than the AceiT

2- The figures and tables are usually very far from their explanation in the text, that makes it quite hard for reader to go 2 pages back to see the explanation, while at the same time reader becomes confused when they see a figure which its explanation starts 2 pages later, so it would be great if these can be fixed.

Reviewer_F37WOpenReview ↗
Rating
4(+1)
Confidence
3(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AceiT, a pixel-space diffusion Transformer combining adaptive depth connections, finer spatial representations at the network endpoints, and chunk downsampling. It adapts Attention Residuals through query sharing and hybrid residual connections, while chunk downsampling supplies multiple representations to depth attention. Experiments report improved quality–compute trade-offs on ImageNet and demonstrate native 1024×1024 text-to-image training without representation alignment or an auxiliary pixel decoder.

Critical strengths
  1. Chunk downsampling integrates naturally with depth attention. Multiple downsampled representations remain independently accessible while each subsequent block retains its configured width. This is a simple and useful architectural design.
  2. The experimental evaluation is substantial.
  3. The ImageNet results show clear benefits at smaller compute budgets and competitive performance at larger scales without REPA. The extension to native high-res generation demonstrates practical relevance beyond benchmarks.
  4. The paper is generally easy to follow.
Critical weaknesses
  1. The contribution relative to closely related architectures could be positioned more precisely. HDiT [1] and PixelU [2], both cited in the paper, already combine spatial downsampling with shallow-to-deep information transfer; their reported results alone do not isolate the advantage of AceiT’s particular combination.
  2. The endpoint-attention diagnosis is stronger than the supporting evidence. Temperature interventions identify sensitive layers, but do not establish that finer endpoint patches help specifically by correcting ineffective attention rather than by changing spatial resolution and compute allocation.
  3. The chunk ablations demonstrate effectiveness but do not fully separate its proposed information-preservation mechanism from the additional representation capacity and resource usage introduced by retaining multiple chunks.
  4. End-to-end efficiency comparisons remain incomplete. Matched training epochs and forward FLOPs provide useful evidence, but do not establish training-time savings at matched quality; the text-to-image throughput comparison also uses different recommended inference settings.
  5. Some figures are placed far from the text discussing them, requiring frequent back-and-forth reading. Their captions provide limited explanation of how to interpret the visual elements and connect them to the main argument.

References

[1] Katherine Crowson et al. Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers. arXiv:2401.11605, 2024.

[2] Zipeng Guo et al. PixelU: A U-Shaped Transformer for Efficient End-to-End Pixel Diffusion. arXiv:2606.27760, 2026.

Questions for authors
  1. What skip-connection placement and merging operation are used in the U-Net baseline in Table 2, and which aspects of HDiT [1] or PixelU [2] does this control capture?
  2. For the conventional and chunk downsampling variants in Table 2, what are the corresponding parameter counts and FLOPs?
Reviewer_EDemOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the computational inefficiency of pixel-space Diffusion Transformers (DiTs) and attributes it to two potential issues: (1) insufficient propagation of high-dimensional information across network depth, and (2) ineffective attention patterns arising when large patch sizes are used to reduce computation. Based on these observations, the authors propose AceiT (Adaptive Connection Image Transformer).

Critical strengths
  • The strongest aspect of the paper is the empirical efficiency improvement. AceiT-B/32 reaches FID 7.98 at roughly 7 GFLOPs, while the larger AceiT variants also show substantially faster convergence than comparable pixel-space DiTs. Importantly, these results are obtained without REPA. This is consequential because reducing reliance on pretrained representation networks makes the pixel-space training pipeline considerably simpler and makes the efficiency result more compelling.

  • The experiments demonstrate that the architecture can be trained directly at 1024² resolution and can scale to multi-billion-parameter models under relatively modest hardware configurations. Even with the qualifications discussed below regarding the dataset and evaluation protocol, demonstrating that a 2.1B-parameter pixel-space model can be trained at 1024² on 8 A100 GPUs is practically significant.

Critical weaknesses
  • The attention-temperature and entropy analyses show correlations with large patch size, but do not clearly demonstrate that these attention patterns cause poor convergence. Similarly, finer endpoint patches may help simply because early/late layers need higher spatial resolution. The paper therefore supports the architecture more strongly than the claimed mechanism. The adaptive connection is also closely related to existing Attention Residuals, so the novelty is better viewed as an effective adaptation to pixel-space DiTs than as a fundamentally new connection mechanism.

  • The high-resolution claims require more careful qualification. The 1024² GPIC data are created using 4× Real-ESRGAN upsampling, and the final tuning stage uses images generated by external text-to-image models. Thus, “training from scratch” applies to model initialization, but the high-resolution training pipeline still relies on substantial external priors. In addition, FLOPs alone do not fully establish training efficiency. Matched wall-clock throughput, memory usage, and total GPU-hours against the main baselines would make the efficiency claim substantially stronger.

Questions for authors
  • Can the authors provide a more direct intervention showing that the endpoint attention patterns identified in Sec. 3.2 are a cause of poor convergence rather than merely correlated with large patch size? For example, can attention behavior be modified without changing tokenization, or vice versa?

  • How much of the gain from adaptive connections comes from the existing Attention Residual mechanism itself versus the AceiT-specific modifications such as query sharing and hybrid residuals? Clarifying this would help assess the novelty of the proposed connection mechanism.

  • For the 1024² experiments, what are the original resolutions of the GPIC images before Real-ESRGAN upsampling? Can the authors also report the main results before the additional synthetic instruction/aesthetic tuning stage?

  • Can the authors report matched training throughput, peak memory, and total GPU-hours for AceiT versus JiT and PixelDiT on the same hardware? This would better support the training-efficiency claims than forward-pass FLOPs alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.