acceptodds
Withdrawn from ICLR 2027

PixelDDT: Depth-Routed Decoding for Pixel Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pixel-space diffusion generates images directly in pixel space without a learned image autoencoder. Recent decoupled architectures pair a global transformer with a lightweight pixel decoder, aiming to separate global semantic modeling from local detail rendering. However, pixel-decoder architectures remain insufficiently compared under a shared implementation. We first compare representative decoder families in a shared codebase and identify a compact multiscale U-Net as a competitive design. This architectural split, however, does not determine how the two modules share information. The pixel-decoder receives backbone information only through the final transformer state at its bottleneck. Probes show that this state retains coarse and fine image content, while class separation declines across later backbone blocks. Earlier states, on the other hand, already expose visual content with stronger class separation, motivating direct access to these features at different decoder scales. We therefore introduce PixelDDT, which routes distinct backbone depths to successive U-Net stages while retaining the final-state bottleneck. These connections allow local multiscale reconstruction to draw on earlier features without passing through the terminal backbone state. With matched backbone and decoder widths, depth routing improves ImageNet-256 FID from 2.74 to 2.31 at 400K updates. Gated window attention further exchanges features across neighboring patches, bringing FID to 2.27 at the same training budget. With longer training, PixelDDT achieves FID 1.63 after approximately 380 epochs with classifier-free guidance.

Primary area:
generative models
Keywords:
pixel-space diffusion, diffusion transformers, pixel decoders, multiscale conditioning, representation analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bSf1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PixelDDT, which makes two main improvements over DiP. First, it connects Transformer features from different depths to different stages of the U-Net decoder. Second, it introduces Window Attention to enable direct feature interaction between neighboring patches.

Critical strengths
  1. The motivation is reasonable. DiP separates global modeling from local decoding, but its decoder mainly relies on the final Transformer features. PixelDDT improves this information flow with only a small increase in parameters and computational cost.

  2. The paper includes useful ablation studies. Depth Routing reduces FID from 2.74 to 2.31. The authors also test broadcasting the final-layer features to all decoder stages, showing that the improvement is not simply due to adding more connections. In addition, feature probes and intervention experiments provide insights into how the internal representations change.

Critical weaknesses
  1. The novelty is somewhat limited. Connecting intermediate Transformer features to a multiscale decoder has already been explored in prior work such as DPT, and Shifted Window Attention is also a well-established technique. PixelDDT mainly combines these ideas within the DiP framework.

  2. The paper does not clearly explain why deeper Transformer features should be routed to coarser decoder stages and shallower features to finer stages. Without comparisons to reversed routing or other depth assignments, it is unclear whether the proposed routing strategy is necessary.

  3. The contribution of Cross-Patch Attention is unclear. Adding this module only reduces FID from 2.31 to 2.27. It is unclear how much of this improvement comes from cross-patch interaction itself, rather than the additional feature processing introduced by the attention module.

Questions for authors
  1. How important is the Depth Routing order? What happens if the routing order is reversed, or if all decoder stages use features from the REPA-aligned layer? These comparisons would help separate the benefits of using multiple feature depths from those of the specific routing strategy.

  2. How much of the improvement from Cross-Patch Attention comes from cross-patch interaction itself? Could the authors include a Patch-Only Attention baseline, using the same attention module but restricting interactions to within each patch? This would help distinguish the benefits of cross-patch communication from those of additional feature processing, and better justify the need for Cross-Patch Attention.

Reviewer_1djXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the conditioning interface between a Transformer backbone and a local decoder in pixel-space diffusion models. The authors argue that conditioning exclusively on the backbone’s final state may limit the decoder’s access to useful intermediate representations. PixelDDT retains the final-state bottleneck while connecting features from multiple backbone depths to different stages of a local U-Net decoder. It additionally introduces cross-patch attention using regular and shifted windows. Under a shared training configuration, depth routing improves FID from 2.74 to 2.31, with cross-patch attention further reducing it to 2.27. After approximately 380 training epochs, the model achieves an FID of 1.63 on ImageNet-256.

Critical strengths
  • The paper addresses a concrete architectural question with meaningful controlled experiments. Final-state bottleneck conditioning, final-state broadcasting to multiple stages, and depth routing achieve FIDs of 2.74, 2.46, and 2.31, respectively. The latter two configurations match receiving sites, projections, and fusion modules, providing evidence that source-depth selection contributes beyond adding conditioning locations.

  • The modifications are inexpensive, and the analysis extends beyond aggregate generation metrics. The final system achieves competitive generation quality. Representation probes, frequency-selective interventions, and cross-patch edge masking provide useful diagnostic evidence.

Critical weaknesses
  • The distinctive contribution and mechanism of the proposed routing remain insufficiently established. Connecting features from different Transformer depths to a multi-scale convolutional decoder has precedent in DPT, while jointly using regular and shifted windows has precedent in generative models such as StyleSwin. These methods are not equivalent to PixelDDT, but they increase the importance of establishing what new insight the specific design provides.

  • The system-level comparison does not sufficiently establish the emphasized efficiency–quality advantage. Table 3 omits the complete PixelDiT system, whose CVPR 2026 paper and supplementary material report an FID of 1.61 with 797M parameters, approximately 311 GFLOPs, and 320 training epochs. PixelDDT reports an FID of 1.63 with approximately 700M parameters, 275 GFLOPs, and 380 epochs. Under the paper’s own proxy of forward TFLOPs multiplied by epochs, the corresponding exposures are approximately 99.6 and 104.4.

  • The absence of a matched long-training baseline leaves unclear whether the short-run ablation gains persist at the final training budget.

Questions for authors
  • What happens when all four receiving stages use block-8 features while keeping the projections and fusion modules unchanged? Can this control distinguish multi-depth complementarity from direct access to the REPA-supervised representation?

  • Can the authors include the published complete PixelDiT in Table 3 and the compute-exposure comparison, and clarify which cost and quality dimensions support PixelDDT’s advantage?

  • Are matched long-training results available for the bottleneck-only or final-broadcast baseline to determine whether the improvements observed at 400K updates persist at the final training budget?

Reviewer_mr6POpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigate routing mechanisms from different depths of pixel diffusion transformer backbones to pixel decoder. This paper studies a typical pixel diffusion network, DiP, a U-Net-like input projection and head, and put hyper-connections between transformer backbones and output DiP head. The authors find that hyper-connections from early layers benefits pixel diffusion systems, and curate a fast-converging system, named PixelDDT, where the performance is competitive to SOTA academic settings on pixel diffusion.

Critical strengths
  1. This paper substantially studies the impact of different pixel decoders in pixel diffusion systems under the same setting and the same codebase in Table 1, which should be encouraged (a note: where is pixnerd and jit? they're also 16x16 diffusion transformers+mlp/shortcut decoding head, you can see jit prediction transform as a shortcut decoding head). The authors find DiP achieve superior performance on FID with and without CFG, which is informative.

  2. The authors provide many supplementary metrics with experiments, which add soundness. The authors also conduct many ablation experiments, to investigate many design options and space, and finally curate a fast-converging pixel diffusion system.

Critical weaknesses
  1. As a research effort on image generation, this paper seems lacking any visualizations on final outputs/inference results in the paper and the supplementary. This is a very concerning factor, reducing the soundness of this paper. First, one can hack FID but with poor inference results with diffusion models. Second, PIXEL diffusion models, which are without pretrained VAEs, often exhibit BAD low-level details but with competitive FID on ImageNet (however, my expectation for this paper is, this paper should improve low details due to the multiscale shortcuts). The authors are highly encouraged to include actual inference results, especially compared with DiP and other diffusion models

  2. The novelty of this paper seems very limited: the hyper connections, either in U-Net or transformers, are well investigated by previous work, where this paper falls in the scope, either for understanding or generation. Low-level feature shortcuts help seems too obvious, rather being a new finding. Also, the proposed methods make the resulting architecture over-complicated without finding supporting key factors. The authors are highly encouraged to find the key components that actually matters for pixel diffusion (especially convergence speed), and simplify the design.

Questions for authors
  1. Why do authors have a preference for over-complicated architectures for pixel diffusion systems in Figure 2? Conv-U Net, cross path attn in patch-wise decoder, concat Rs, Ls, Cs, 3x3 conv in final output, etc…… The authors just cannot justify the necessity of every component in the figure.

  2. Why do authors only study /16 patch size setting? It seems the motivation and design also help /32 setting?

  3. Can authors provide some principled view for what really matters on pixel diffusion from this paper design?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.