acceptodds
Under review as a conference paper at ICLR 2027

Pixel Space Doesn't Save You: Two Mechanisms of Conditioning Homogenization in Diffusion Transformers

Abstract

A key motivation for pixel-space Diffusion Transformers is that deleting the VAE removes a lossy representation bottleneck. We show this fails for the conditioning pathway: the bottleneck was never in the VAE. Across 12 checkpoints and 9 architecture families, pixel- and latent-space models interleave along the homogenization axis, with conditioning vectors degenerating toward near-identical representations across instances. We trace this to two mechanisms—a frequency-schedule mismatch in pooled conditioning, and text-stream writeability in token conditioning—and derive from the first a training-free, checkpoint-level diagnostic, the norm ratio (, ), validated out-of-sample on four held-out architectures under pre-registered predictions. The same ratio predicts class-code fragility across those checkpoints (), showing functional consequences in models we did not train. We validate the second mechanism by perturbing only the prompt-specific residual, which changes the named attribute in 92–100% of trials on SD3.5 and FLUX. We then propose Conditioning Norm Balancing (CNB), a training-time regularizer that restores balanced conditioning geometry, and evaluate it under a shared-noise paired protocol (10,000 pairs per cell, McNemar's exact test, FID and accuracy from the same samples). CNB holds the ratio at parity and improves both metrics at each model's own optimal guidance, on both sides of the pixel/latent divide: top-1 on SiT-B/2 () and on PixelDiT-B, which has no VAE (), with FID improving in both cases. At lower guidance, the two metrics give opposite verdicts on identical weights. A single FID number cannot certify a conditioning pathway.

Primary area:
generative models
Keywords:
Conditions Imbalance, Diffusion Transformers, Homogenization Mechanism, Multimodal Generative AI, Conditioning Bottleneck

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CEK8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that pixel-space diffusion transformers do not avoid a conditioning bottleneck, because conditioning is computed before the image or latent is touched. For class-conditional models, it measures how similar the combined conditioning vectors c = t_emb + y_emb are across classes. It links high similarity to a mismatch between the timestep embedding’s frequency schedule and the t range, and proposes the norm ratio ‖t_emb‖/‖y_emb‖ as a checkpoint-level diagnostic. Across 12 public checkpoints, this ratio correlates with how similar the conditioning vectors of different classes are. For text-conditioned MMDiTs, it argues that token homogenization only happens when blocks can write to the text stream, and tests this with embedding edits on SD3.5 and FLUX. Finally, it proposes CNB, a loss that keeps the two norms equal, and reports gains in top-1 accuracy and FID on SiT-B/2 and PixelDiT-B at each baseline’s best guidance scale.

Critical strengths

-The diagnostic is cheap, needing only two embedding norms from a checkpoint, and it covers 12 public models across 9 families.

-CNB is simple. It adds no parameters, and Fig. 5 shows that it clearly separates the conditioning vectors of different classes on both a latent and a pixel model.

Critical weaknesses

-The introduction attributes to prior pixel-space work a premise that these papers do not state, namely that removing the VAE removes representation bottlenecks in general. JiT [1] is motivated by clean-data prediction and a self-contained pipeline, and PixelFlow [2] by simplicity and end-to-end training. PixelDiT [3] does mention lossy VAE reconstruction, but only in terms of image fidelity, not conditioning. The authors should either support this framing with quotes from these papers or revise it, and remove the word “unexamined”.

-The causal test is too thin to support the claims. CNB is the paper’s only causal test of Mechanism A and its main practical contribution, yet each arm is a single training run of a small B-scale model at batch size 32. The reported p-values only capture sampling noise, not variation across training runs, so the gains may fall within run-to-run differences. CNB also hurts on SiT at low guidance (cfg ≤ 1.5), where well-trained models usually operate.

-The core observation is not new, and the added evidence largely follows from how c is constructed. Pham et al. [4] already report over 99% cross-class similarity in DiT conditioning embeddings, and show that pruning up to two-thirds of the embedding leaves generation quality largely unaffected. Beyond this, when a shared timestep embedding dominates c, the high similarity between classes and their sensitivity to small perturbations follow directly from the norm ratio. Their correlation with the ratio is therefore expected rather than a new finding.

[1] Li, T. and He, K. Back to Basics: Let Denoising Generative Models Denoise. CVPR 2026.

[2] Chen, S., Ge, C., Zhang, S., Sun, P., and Luo, P. PixelFlow: Pixel-Space Generative Models with Flow. arXiv:2504.07963, 2025.

[3] Yu, Y., Xiong, W., Nie, W., Sheng, Y., Liu, S., and Luo, J. PixelDiT: Pixel Diffusion Transformers for Image Generation. CVPR 2026.

[4] Pham, T. X., Zhang, K., Hong, J. W., and Yoo, C. D. A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformers. ICLR 2026.

Questions for authors

-Could you train a matched pair that differs only in max_period, and report the norm ratio and class accuracy, to test the mismatch as a cause?

-What max_period does your PixelDiT-B baseline use, and why does its ratio (4.3-4.6) exceed every official PixelDiT-XL checkpoint

Reviewer_srTJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies conditioning homogenization in pixel- and latent-space diffusion transformers. It analyzes timestep/class embedding imbalance and text-stream updates, and proposes Conditioning Norm Balancing (CNB) for pooled conditioning. CNB improves FID and target-class accuracy on two controlled models at the baselines’ optimal guidance settings.

Critical strengths
  • The paper highlights a useful distinction between distributional quality and conditioning fidelity.
  • Shared-noise, shared-label evaluation on 10,000 pairs makes the comparisons more informative. Guidance and sampling-budget sweeps help establish the limits of the results.
  • The authors report negative results, failed predictions, and scoring errors rather than presenting only favorable settings.
Critical weaknesses
  • The most relevant simple baselines are missing. Comparisons against separate normalization, fixed or learned scaling, and EDM2-inspired mixing are needed to establish the value of CNB’s specific objective.

  • Similarity and sensitivity to injected noise do not establish loss of usable semantic information. The frequency mismatch is not isolated through matched training with corrected encodings.

  • CNB can increase label-table homogenization, while the proposed centered correction is untested. For token conditioning, CLIP-based editing demonstrates semantic influence but does not show that homogenization causes poorer prompt adherence.

Questions for authors
  • How does CNB compare with normalization, learned scaling, and magnitude-preserving mixing under matched training budgets?
  • Does correcting the timestep encoding during training prevent the imbalance without CNB?
  • For token conditioning, can reducing homogenization improve directly measured prompt adherence, beyond demonstrating editability?
1 comment
Reviewer_srTJOpenReview ↗

Response by Reviewer srTJ

Thanks for the authors' response. I will retain my weak acceptance rating.

Reviewer_Kq5MOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This submission falsifies a common assumption behind the recent development of pixel-based generative models: VAE encoder-decoders are representation bottlenecks limiting generation quality. Instead, this work proposes a mechanism that creates an internal bottleneck in Diffusion Transformers termed conditioning homogenization. During training norm mismatch between text condition vectors and image tokens is large and this homogenize the semantic effects on generation, in other words making different-class conditions similar, harming the generation quality. Conditioning Norm Balancing (CNB) is proposed to mitigate this problem. In experiments, 12 models, including pixel and latent DiTs are tested and adding CND is shown to improve both pixel and latent DiTs in ImageNet class-conditioned generation.

Critical strengths
  • The mismatch between conditioning and image-token vectors is a real problem and, to my knowledge, overlooked.
  • The hypothesis of norm imbalance and homogenization is tested using massive numbers of pretrained models and validated.
Critical weaknesses
  • Overcriticism of the pixel DiTs: I think that the motivation behind pixel DiTs is multifaceted: computation saving, simplifying training and architecture, and removing the representation bottleneck. The proposed analyses are only relevant to the final one. Moreover, the major part of the manuscript's contribution is on the detailed analyses of the homogenization problem and its mitigation measure, so I did not find much critical.

  • The mechanistic cause of homogenization: how the norm imbalances cause homogenization is not clear to me despite the convincing diagnostic results. Norm imbalance would be universal in adding new modules to the training network, for example, fine-tuning with new task heads, adding new conditioning, and multimodal fusion. I am not fully convinced why this is especially problematic in this setting.

Questions for authors
  • Term "frequency-schedule mismatch": Is this the timestep-embedding frequency? It is guessable but better to be clear.
  • 5.3: norm balancing is done by adding a loss, but is it possible to achieve balancing via a some normalization method? I think it is simpler.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A common response to all reviewers

We thank all reviewers for their feedback. The three sets of comments converged on one question — whether our result is a property of conditioning geometry or of a single training run — so we ran the controls experiments that question requires rather than only arguing the point. All changes are marked in red in the revised PDF manuscript.

In summary, we have:

  1. Added five further SiT-B/2 arms and a matched SiT-XL/2 pair (675M), each identical to its own baseline except for the conditioning rule, under a shared-noise paired protocol (10,000 pairs per cell, McNemar's exact test, FID and accuracy from the same samples).
  2. Ruled out seed and the timestep-frequency constant as explanations: a second seed reproduces the effect cell for cell, and max_period 10^4 → 10 changes neither metric at any guidance scale.
  3. Trained the normalization alternative that was asked for (RMS-norm with learned gains). It beats CNB at 100k, which we report in full; it also re-creates the imbalance through its gains, and its advantage decays over 100k–400k while CNB's does not.
  4. Extended training 4×, to 400k steps, so the comparison no longer rests on a single checkpoint.
  5. Corrected our own account. The norm ratio is a diagnostic for additive conditioning, not a causal parameterisation; what orders all seven arms at a matched budget is the effective dimensionality of the class codes. The abstract, Sec. 3 and the conclusion are narrowed accordingly.
  6. Narrowed the framing of pixel-space DiTs to the conditioning pathway, their case for the image pathway being well supported, and state plainly what we did not test.

Which item answers which comment: (1) srTJ W1/Q1 and W3, CEK8 W2 - (2) CEK8 Q1/Q2, Kq5M Q1, srTJ W2/Q2 - (3) srTJ W1/Q1, Kq5M Q2 - (4) CEK8 W2, Kq5M Q2 - (5) Kq5M W2 - (6) Kq5M W1, CEK8 W1/W3, srTJ Q3.

The revised PDF is on OpenReview, with a new section in the main paper and an extended appendix. We invite reviewers to read the point-by-point responses together with the revision. We look forward to further discussion. Thank you!


Best regards,

Submission5792 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.