acceptodds
Withdrawn from ICLR 2027

BandDiT: Band-Aligned Diffusion Transformers for Pixel-Space Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pixel-space diffusion models denoise pixel values directly, without the pretrained autoencoder that latent diffusion models use to compress images. At each denoising step, the pixel-space transformer must extract meaningful features from raw pixels, reason over these representations, and map its predictions back into image space. In contrast, latent diffusion models operate in a learned representation space that abstracts away some low-level detail, leaving the pretrained decoder to reconstruct fine details and textures. This places the full burden of spectral reconstruction on the transformer, raising a central question: how are different image frequencies represented across network depth, and how can this organization guide learning? We find that pixel-space diffusion transformers build the spectrum progressively across their blocks: low-frequency bands become decodable from shallow blocks, whereas high-frequency bands emerge near the output. This hierarchy lies in internal representations within a denoising step, distinct from the known coarse-to-fine evolution of samples along the sampling trajectory. We introduce \our, a band-aligned diffusion transformer that turns this hierarchy into explicit supervision. Wavelet bands of the clean image, which need no pretrained teacher, serve as targets for intermediate blocks at matching depths, from low frequencies in shallow blocks to high frequencies in deep ones. On ImageNet , \our-B/16 matches the baseline's FID and 5-crop FID with fewer epochs and reduces the high-frequency Fr\'echet Wavelet Distance by , indicating improved global coherence, local structure, and texture fidelity.

Primary area:
generative models
Keywords:
Diffusion Models, Image Generation, Diffusion Transformers, Wavelet Decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_o8gvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BandDiT, which applies frequency-specific supervision at different Transformer depths based on the observation that low-frequency information is decoded earlier than high-frequency details. Experiments on ImageNet-256 show modest improvements in FID, 5-crop FID, and hFWD over JiT, with faster convergence in terms of training epochs.

Critical strengths
  1. Supervising different layers according to their frequency characteristics is an intuitive idea. The method does not require a pretrained teacher or changes to the original generation architecture or sampling procedure, making it relatively easy to adopt.

  2. The evaluation is fairly comprehensive. In addition to FID, the authors report 5-crop FID and hFWD to assess local structure and texture quality, and evaluate the method at multiple model scales.

Critical weaknesses
  1. The motivation is not fully supported by Figure 3. The authors use the final output head to decode intermediate features, but this head was trained on the final layer. Poor high-frequency decodability in shallow layers may simply come from this mismatch rather than missing information.

  2. It's still unclear whether band-depth alignment is the main reason for the improvements. In Table 3, G0 alone achieves an FID of 4.23, while adding all bands slightly worsens it to 4.25. A comparison with standard intermediate supervision is also missing. Some gains are quite small, such as the H model improving FID from 1.91 to just 1.90, though with fewer training epochs. The extra adapters also make the actual training efficiency unclear.

Questions for authors
  1. If a separate linear probe were trained for each intermediate layer, rather than reusing the final output head, would the same frequency hierarchy still be observed? Does this trend hold across different noise levels?

  2. Given the three additional adapters used during training, does BandDiT require fewer GPU-hours or training FLOPs than JiT to reach the same FID?

Reviewer_FSu3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies an implicit curriculum that occurs in diffusion transformers - shallow blocks can decode coarse information, which deeper blocks increasingly decode higher level/finer features. They turn this observation into a training-only method which potentially improves results, while preserving the JiT architecture and setup - valuable when other comparable methods require architecture changes. Validation occurs through gFID, gFID 5 clip, and frechet wavelet distance.

Contributions are: Empircal observations of depth based coarse to fine decoding, in both pixel and latent space models. Teacher free training - natural usage of the input images to provide extra training signal Practical JiT improvements - maintain architecture and improve results

Critical strengths

The mechanism is interesting, well-motivated, and relevant. There is a lot of interest in understanding the relationship between frequency and diffusion.

The method doesn't require external supervision, which is a huge plus towards the method. It doesn't modify the architecture (inference), and minimal changes to objectives.

Critical weaknesses

Intermediate decoding already exists in papers like: https://arxiv.org/abs/2512.24176 My concern is that the current method might not be because of the band portion, but because it's doing something similar. I would like the authors to test two additional runs. The first, decode full image at all depths, no frequency information. The second, all wavelet bands at all depths. This is important because we need to know if the improvement actually comes from the design choices - the inverse ablation was useful, not sufficient.

Secondary weakness: Marginal improvements over baseline. The difference between results is at the level of "random evaluation noise", and I am not convinced by the 500 epoch vs 600.

Questions for authors

I would like to see a graph of the gFID over steps (every 50 epochs, 100 worst case), or at least just the raw numbers. I'd like to know what the gFID at 600 epochs is - or understand why training appears.

See weaknesses for the other 2 runs I'd like to see.

For figure 9, I would like to know if the input noise (and random seeding) was the same for all images. If not, I would like to see a version containing that stipulation.

Reviewer_yp25OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper is a diffusion-transformer-on-imagenet paper in the vein of DiT, JiT, REPA, etc. The key contribution is a series of adapters and auxilliary losses which are sequential in depth and intended to induce the model to learn a coarse-to-fine bias, if I understand correctly. The evaluation is w.r.t. FID and comparable metrics.

Critical strengths

The proposed method is simple and coarse-to-fine bias is reasonable to try to induce in a diffusion model. I appreciate that the formulation is comparable to accepted techniques in the diffusion model community such as REPA, without requiring auxilliary data. Moreover, the benchmark comparison is thorough and contains many matched comparisons at small-scale while still comparing against SOTA at large scale.

Critical weaknesses

The main issue is that this technique feels too simplistic to work at scale. Since the frequency statistics of natural images are consistent and well-known, the fact that different coarse-to-fine schedules and biases can be induced by varying the noise schedule of diffusion models is in fact the entire point of "Diffusion is spectral autoregression". Varying the noise schedule is one way to achieve a coarse or fine bias, and is considerably more elegant than adding auxilliary modules and losses. For instance, in the Simple/Simpler diffusion line of work, the key idea is to bias the schedule towards higher levels of noise at higher resolutions, which produces a stronger coarse-to-fine bias than copying low-resolution noise schedules. An ablation sweeping the noise schedule of the baseline is needed to justify the proposed method.

I'm also not sure why metrics such as 5-crop FID and hFWD need to be included if the ultimate goal is training a good generative model. It seems a little like biasing the evaluation in favor of the method, when more standard metrics such as Precision/Recall could've been used.

Finally, the large-scale comparison results are not encouraging.

Questions for authors

What is the advantage of spectral biases when the ultimate evaluation is with still respect to proxies for human perception such as FID? Why not optimize for human perception more directly, such as with perceptual networks like LPIPS/DreamSim?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.