- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper argues that pixel-space diffusion transformers do not avoid a conditioning bottleneck, because conditioning is computed before the image or latent is touched. For class-conditional models, it measures how similar the combined conditioning vectors c = t_emb + y_emb are across classes. It links high similarity to a mismatch between the timestep embedding’s frequency schedule and the t range, and proposes the norm ratio ‖t_emb‖/‖y_emb‖ as a checkpoint-level diagnostic. Across 12 public checkpoints, this ratio correlates with how similar the conditioning vectors of different classes are. For text-conditioned MMDiTs, it argues that token homogenization only happens when blocks can write to the text stream, and tests this with embedding edits on SD3.5 and FLUX. Finally, it proposes CNB, a loss that keeps the two norms equal, and reports gains in top-1 accuracy and FID on SiT-B/2 and PixelDiT-B at each baseline’s best guidance scale.
Critical strengths
-The diagnostic is cheap, needing only two embedding norms from a checkpoint, and it covers 12 public models across 9 families.
-CNB is simple. It adds no parameters, and Fig. 5 shows that it clearly separates the conditioning vectors of different classes on both a latent and a pixel model.
Critical weaknesses
-The introduction attributes to prior pixel-space work a premise that these papers do not state, namely that removing the VAE removes representation bottlenecks in general. JiT [1] is motivated by clean-data prediction and a self-contained pipeline, and PixelFlow [2] by simplicity and end-to-end training. PixelDiT [3] does mention lossy VAE reconstruction, but only in terms of image fidelity, not conditioning. The authors should either support this framing with quotes from these papers or revise it, and remove the word “unexamined”.
-The causal test is too thin to support the claims. CNB is the paper’s only causal test of Mechanism A and its main practical contribution, yet each arm is a single training run of a small B-scale model at batch size 32. The reported p-values only capture sampling noise, not variation across training runs, so the gains may fall within run-to-run differences. CNB also hurts on SiT at low guidance (cfg ≤ 1.5), where well-trained models usually operate.
-The core observation is not new, and the added evidence largely follows from how c is constructed. Pham et al. [4] already report over 99% cross-class similarity in DiT conditioning embeddings, and show that pruning up to two-thirds of the embedding leaves generation quality largely unaffected. Beyond this, when a shared timestep embedding dominates c, the high similarity between classes and their sensitivity to small perturbations follow directly from the norm ratio. Their correlation with the ratio is therefore expected rather than a new finding.
[1] Li, T. and He, K. Back to Basics: Let Denoising Generative Models Denoise. CVPR 2026.
[2] Chen, S., Ge, C., Zhang, S., Sun, P., and Luo, P. PixelFlow: Pixel-Space Generative Models with Flow. arXiv:2504.07963, 2025.
[3] Yu, Y., Xiong, W., Nie, W., Sheng, Y., Liu, S., and Luo, J. PixelDiT: Pixel Diffusion Transformers for Image Generation. CVPR 2026.
[4] Pham, T. X., Zhang, K., Hong, J. W., and Yoo, C. D. A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformers. ICLR 2026.
Questions for authors
-Could you train a matched pair that differs only in max_period, and report the norm ratio and class accuracy, to test the mismatch as a cause?
-What max_period does your PixelDiT-B baseline use, and why does its ratio (4.3-4.6) exceed every official PixelDiT-XL checkpoint
Response by Reviewer srTJ
Thanks for the authors' response. I will retain my weak acceptance rating.