acceptodds
Withdrawn from ICLR 2027

Learning Representations to Guide Pixel Diffusion Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual representations are largely developed for understanding, yet what makes them useful for generation remains unclear. We study this question in pixel diffusion transformers and uncover two key insights. First, better representation quality does not necessarily translate into greater generative utility, as externally aligned semantics can be progressively lost during pixel synthesis. Second, architectural separation with specialized pixel decoders mainly redistributes semantic and spatial modeling, without necessarily yielding representations better suited to generation. These insights suggest that a useful generative representation must be both properly formed and actively exploited. Motivated by these insights, we introduce Representation Guidance (RG), which directly forms and exploits internal representations while retaining a plain Transformer architecture. A low-dimensional bottleneck forces generation-relevant information into a compact representation, which is then explicitly used to guide decoder generation by modifying the flow-matching target according to its induced prediction offset. RG learns clearer semantic and spatial structure while scaling efficiently in both parameters and computation. On ImageNet , RG-L/16 with only 459M parameters outperforms substantially larger pixel diffusion transformers, while RG-XL/16 achieves an FID of 1.55. RG further reaches an FID of 1.74 at and extends to text-to-image generation, achieving 0.89 on GenEval and 87.1 on DPG-Bench.

Primary area:
generative models
Keywords:
Diffusion Models, Image Generation, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BDafOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how internal representations are formed and used in pixel-space diffusion Transformers and proposes Representation Guidance (RG). Building on a decoupled encoder–decoder architecture, the method introduces a low-dimensional linear bottleneck between the two components and modifies the flow-matching target using the difference between EMA model predictions with and without the internal representation. The authors examine the components through layer-wise representation analysis and ablation studies, and report competitive results on ImageNet at 256×256 and 512×512 resolutions, as well as text-to-image generation.

Critical strengths
  1. The method is easy to understand, with reasonably extensive component ablations.

  2. RG retains standard Transformer blocks and improves generation by modifying the training objective. Compared with a model using the same architecture and cfg settings, it does not require additional computation of the RG prediction offset during sampling.

  3. The generation results are competitive. RG-XL achieves FID of 1.55 and 1.74 on ImageNet at 256×256 and 512×512, respectively. Experiments across multiple model sizes and text-to-image generation also demonstrate the potential scalability of the approach.

Critical weaknesses
  1. The abstract emphasizes that RG “directly forms and exploits internal representations,” but does not explicitly explain the role of REPA. Considered alongside the abstract’s discussion of the limitations of external alignment, it can easily lead readers to underestimate the method’s dependence on pretrained representation supervision and overestimate the contribution of this paper. In addition, the broad conclusion of the first insight—that better representation quality does not necessarily translate into better generation—has already been explored in prior work. PixelREPA [1] directly documents the degradation caused by REPA as JiT training progresses.

  2. DDT [2] already uses an encoder to extract internal representations, conditions a decoder on these representations, and applies REPA supervision to intermediate encoder features. Thus, the use of internal representations alone does not distinguish RG from DDT. Moreover, Eq. (2) closely resembles Model Guidance (MG) [3], which modifies the training target using differences between model predictions. Although these works are cited, the incremental contribution of RG remains insufficiently isolated. Authors may distinguish the inherited design choices from the proposed advances and add a matched MG baseline using the same encoder–decoder architecture, bottleneck, REPA supervision, and training budget.

  3. The model still uses cfg during sampling in addition to RG during training, and both affect generation quality. I suggest reporting results for the baseline and RG without cfg, to assess the benefit of RG itself and its dependence on sampling-time guidance.

  4. Table 8 shows that, under the same bottleneck setting, adding RG reduces FID from 3.79 to 3.28 with REPA, whereas adding guidance increases FID from 4.71 to 8.99 without REPA. This indicates that, under the current experimental settings, the effectiveness of guidance clearly depends on external representation supervision. How should this phenomenon be explained?

[1] Representation Alignment for Just Image Transformers is not Easier than You Think. arXiv: 2603.14366

[2] DDT: Decoupled Diffusion Transformer. arXiv:2504.05741, 2025.

[3] Diffusion Models without Classifier-free Guidance. arXiv:2502.12154, 2025.

Questions for authors
  1. Please report the training time of RG relative to the baseline, including the GPU-hours. The cost should include the additional EMA predictions.

  2. RG introduces a guidance scale ss, a guidance timestep interval, and a warm-up setting, while also requiring additional EMA predictions during training. The ablations indicate that performance is sensitive to the guidance scale and timestep interval. Although Table 8 shows a meaningful FID improvement for B/16, it remains unclear whether these hyperparameters transfer directly across model sizes and tasks. Hyperparameter tuning can be particularly expensive at larger training scales.

Reviewer_SUaZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what makes visual representations useful for generation in pixel diffusion transformers. Based on two observations about representation quality and architectural separation, the authors propose Representation Guidance (RG), which forms compact internal representations through a low-dimensional bottleneck and explicitly uses them to guide generation. RG retains a simple Transformer architecture while improving semantic and spatial modeling. Experiments on ImageNet and text-to-image generation demonstrate strong performance and favorable scaling efficiency.

Critical strengths
  1. The proposed method is simple and easy to understand.
  2. The proposed method achieves good results on both c2i and t2i benchmarks.
Critical weaknesses
  1. I would like the authors to clarify the main contribution of this work. Is the primary contribution the analysis of different representations? The proposed method itself seems to offer limited methodological novelty: the bottleneck design has already been explored in JiT, while the modified training target appears to mainly align the model with the CFG-enhanced prediction. From this perspective, the inference procedure also resembles an enhanced form of CFG.

  2. In Table 2, it seems inappropriate to categorize VA-VAE as a pre-trained model. VA-VAE is trained only on ImageNet and, to my understanding, is not pre-trained on a larger-scale external dataset.

  3. RG requires two additional gradient-free EMA forward passes during each training step, and also relies on a pre-trained visual encoder to compute the REPA loss. Therefore, parameter count, per-forward FLOPs, and the number of training epochs are insufficient to fully characterize its training cost. The paper should report actual training throughput, memory consumption, total GPU hours, and FID under a matched training compute budget to better substantiate the claimed efficiency.

  4. The paper lacks sufficient qualitative results for text-to-image generation. More generated samples, together with their corresponding text prompts, should be provided to better demonstrate the model's generation quality and prompt-following ability.

Questions for authors

Please refer to weakness.

Reviewer_LCcQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes to tackle pixel transformer image generation through what they term "representation guidance". The approach is a combination of existing techniques from across the recent image generation literature -- model guidance, JiT, REPA, DDT -- that achieves SoTA performance on image generation benchmarks.

Critical strengths
  1. The paper text is generally easy to follow and the visual presentation of figures is pleasing.
  2. The paper reports generally strong results across a large number of image generation experiments. They establish SotA generation quality with strong parameter efficiency.
Critical weaknesses
  1. The paper story is confusing. At a high level, it seems to be about both producing "good" representations and using them well during training. However, what makes a "good" representation is poorly defined. The three measures used, RMSC, LDS and linear probing accuracy are not directly related to generation quality and are also subjective descriptors of the model features. Yet the authors seem to use them as an objective measure of representation "quality". It also doesn't follow clearly how to measure how well-used a representation is. Surely it is used if it is the input to later stages of the model? In the end, this makes it difficult to understand the motivation for the final proposed approach.
  2. The approach is a composition of existing work with little additional insight. Given the previous weakness, I find it easier to view the method in this paper simply as the combination of a number of existing papers: model-guidance/auto-guidance, JiT, DDT, REPA. It is not that surprising that the experimental results are strong when you are putting the best ideas from the recent literature together, and the paper doesn't seem to offer any additional knowledge or insight into the strong performance beyond that supplied in the original papers for each component. Furthermore, the complexity from all these components are considerable hyperparameter burden to implementing the approach: REPA depth and weight, encoder-decoder depth, model guidance scale and interval, etc.
Questions for authors

What are the practical advantages of performing image generation in the pixel space vs a latent space?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.