acceptodds
Withdrawn from ICLR 2027

RiT: Vanilla Diffusion Transformers Are Enough in Representation Space

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Flow matching with -prediction—regressing the clean data point rather than the ambient velocity—is known to exploit low-dimensional manifold structure effectively in pixel space. We ask whether a pretrained representation space, while containing a low-dimensional data manifold of comparable intrinsic dimensionality, offers a distribution more favorable for flow-matching learning. Comparing pixel, SD-VAE, and DINOv2 features along four geometric axes, we find that pixel and DINOv2 share nearly identical intrinsic dimensionalities (both ) yet DINOv2 exhibits higher effective rank, better covariance conditioning, lower excess kurtosis, and lower on-manifold interpolation error; SD-VAE latents are consistently intermediate, indicating that the advantage stems from representation-learning objectives rather than mere compression. These statistical properties render the flow-matching regression well-conditioned and remove the need for the specialized prediction heads or Riemannian transport used by prior DINOv2 diffusion methods. We propose the Representation Image Transformer (RiT): a vanilla Diffusion Transformer trained by -prediction on frozen DINOv2 features, augmented only by a dimension-aware noise schedule and joint [CLS]-patch modeling. On ImageNet , RiT attains FID 1.45 without guidance and 1.14 with classifier-free guidance, outperforming DiT -XL with fewer parameters (676M vs. 839M). The resulting ODE is efficiently solvable at coarse discretizations: Heun steps already reach FID 2.0 and steps reach 1.25, without distillation or consistency training. Code and model weights will be released.

Primary area:
generative models
Keywords:
representation learning, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_wSg8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RiT studies a training recipe for a modernized standard DiT operating in frozen representation space. It combines element-wise standardization, x-prediction, a dimension-aware time distribution, and joint diffusion of global and patch features, without a representation-specific wide head or non-Euclidean transport. Experiments on class-conditional ImageNet 256×256 show improved FID relative to a matched-encoder/decoder RAE baseline and useful few-step sampling. The paper also analyzes representation spaces using intrinsic-dimension estimates, covariance spectra, and kurtosis, and reports transfer to SigLIP2 features.

Critical strengths
  1. A useful implementation reference with controlled component comparisons. The paper presents a relatively simple recipe and reports improvements over the RAE-XL DINOv2-S baseline at the same encoder, decoder, and denoiser parameter count: FID decreases from 1.87 to 1.45 without guidance and from 1.41 to 1.14 with guidance. The within-code ablations also show separate benefits from x-prediction and CLS diffusion. These results establish some practical value for the recipe.

  2. Clear presentation of the empirical scope. The manuscript documents sampling grids and several implementation choices, and acknowledges the descriptive nature of its geometry measurements and the distinction between epochs and compute. This helps readers interpret and potentially reproduce the experimental findings.

Critical weaknesses
  1. Limited new insight beyond combining established components. Standardization and dimension-dependent shifting are inherited from RAE/SD3, x-prediction follows JiT, and joint semantic-token diffusion follows REG. Moreover, the matched RAE-XL DINOv2-S baseline already omits a wide head, so removing such a head is not a new simplification relative to that baseline. The paper mainly establishes that this combination works, without deriving a sufficiently clear mechanistic explanation or transferable design principle. The geometry diagnostics remain correlational, the time-distribution ablation changes both the base distribution and its shift, and the CLS gain is not separated from additional-token or auxiliary-supervision effects. Given the limited component novelty, these gaps substantially limit the research contribution.

  2. The empirical advantage is insufficient to compensate for the limited conceptual contribution. RiT’s headline guided FID of 1.14 does not improve on the original RAE result of 1.13; its longer EDM sampling configuration in Appendix M also reaches 1.13. The original RAE comparison uses a larger DINOv2-B model with a wide head and AutoGuidance, so this is a comparison of best reported performance rather than a matched-setting test. RiT’s smaller model and improvement over RAE-S deserve credit, but a compelling quality–compute advantage over the strongest RAE configuration is not established under a common evaluation. Likewise, epoch savings are not measured compute savings, and the few-step baselines retain different samplers. With evaluation limited to ImageNet 256×256, the demonstrated practical advance is not strong enough to establish the significance of an otherwise largely empirical recipe.

Questions for authors
  1. What generalizable insight does the paper establish beyond showing that known components work well together? In particular, what evidence separates the CLS token’s semantic contribution from auxiliary supervision, and the time shift from the changed base distribution?
  2. Can the authors reconcile their reproduced RAE FID of 1.16 with the original report of 1.13, and clarify the remaining quality–compute advantage when guidance, sampling budget, and evaluation are made comparable? This clarification could establish whether the practical simplification provides a sufficiently consequential contribution despite similar best FID.
Reviewer_iYQCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RiT, a method for training modern, standard Diffusion Transformers (DiTs) directly within the feature space of a frozen representation encoder. The approach integrates several techniques: element-wise feature normalization, xx-prediction based on a velocity loss, dimension-aware temporal distribution, and joint diffusion of both [CLS] tokens and patch tokens from the encoder. On the ImageNet dataset (256×256256 \times 256 resolution), RiT-XL achieves an FID of 1.45 without a guidance setting, and 1.14 when using classifier-free guidance. Furthermore, the model excels in few-step sampling: it attains a guided FID of 2.0 with 5-step Heun sampling, and an FID of 1.25 with 10-step sampling. Additionally, this paper compares the geometric structures of pixel space, SD-VAE space, DINOv2 space, and SigLIP2 space using various metrics, including intrinsic dimensionality estimation, truncated-PCA effective rank, normalized conditioning diagnostics, and marginal excess kurtosis.

Critical strengths
  1. Empirical performance. Its core empirical finding—that a modern DiT (Diffusion Transformer) operating within the DINOv2 representation space can achieve a highly competitive ImageNet FID score, even without a DDT head or Riemannian paths, and performs well with few-step sampling.

  2. Several informative ablations. Without [CLS], the xx- and vv-prediction runs use the same clamped velocity objective and mainly differ in functional parameterization. The final FID difference of 1.63 versus 1.78, therefore, provides reasonable evidence for the benefit of xx-parameterization.

  3. The paper is clearly written and unusually transparent about its limitations. The appendices provide detailed hyperparameters, pseudocode, sampling schedules, and geometric estimators.

Critical weaknesses
  1. Limited methodological novelty. My primary concern is that most ingredients in the proposed recipe have already appeared in closely related work. Specifically, xx-prediction—together with the motivation that noisy interpolants leave the data manifold—is a central contribution of JiT [1]; element-wise latent standardization is inherited from RAE [2]; and the dimension-aware time shift follows the same effective-dimension scaling rule introduced in RAE and subsequently retained in Scale-RAE [2,3]. Moreover, jointly noising and predicting a vision-foundation-model class token alongside image latents was previously explored by REG [4], while Scale-RAE had already shown that a standard DiT can dispense with the wide DDT head when the backbone width sufficiently exceeds the representation dimension [3]. RiT does offer a cleaner integration than REG because its [CLS] and patch tokens originate from the same encoder, eliminating the need for a second encoder and an explicit alignment loss. Nevertheless, I view this primarily as a useful simplification and recombination of existing techniques rather than a substantively new mechanism. The remaining potentially novel contribution is the collection of geometric diagnostics, but these analyses are largely descriptive and correlational: they do not establish a causal connection between the reported geometric properties and the effectiveness of the proposed recipe.

[1] Back to Basics: Let Denoising Generative Models Denoise (JiT). arXiv:2511.13720 [2] Diffusion Transformers with Representation Autoencoders (RAE). arXiv:2510.11690 [3] Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders (Scale-RAE). arXiv:2601.16208 [4] Representation Entanglement for Generation (REG). arXiv:2507.01467

  1. Empirical advantages are less compelling when compared to stronger RAE variants. Recent improvements to RAE have rendered the numerical results reported for RiT less significant. RAEv2 [5] achieves a guided FID of 1.06 in just 80 epochs, with an unguided FID falling below 2 after only 35 epochs; in contrast, RiT requires 760 epochs of training to reach a guided FID of 1.14. When a methodological contribution consists primarily of a combination of existing components, one typically expects it to yield significantly better final results, offer a superior trade-off between computational cost and generation quality, or undergo more extensive validation. RiT has yet to demonstrate such advantages. Notably, its requirement for 760 epochs of training is difficult to reconcile with its central claim of "training efficiency."

[5] Improved Baselines with Representation Autoencoders. arXiv:2605.18324

Questions for authors
  1. The geometric metrics mentioned (anisotropy, kurtosis, and off-manifold statistics) are purely descriptive. It remains unclear whether they can predict differences in generative performance, or if they are, in fact, the underlying causes of such differences. Controlling for latent dimension and reconstruction quality, can the authors demonstrate that these metrics predict performance across different encoders or representation transformations? Alternatively, could the authors intentionally manipulate a specific geometric property (e.g., via whitening, rotation, or rescaling) to show whether generative quality shifts in the expected direction?

  2. Evaluation Beyond FID. When all compared methods achieve FID scores in the 1–2 range, marginal differences become difficult to interpret; they are often sensitive to factors such as estimator variance, reference statistics, preprocessing, sampling methods, and the choice of guidance strategy.

Reviewer_8yZfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

RiT shows that a standard DiT trained on standardized DINOv2 features needs no representation-specific machinery, outperforms RAE at matched parameter count, transfers to SigLIP2, and that covariance statistics order generation quality across tokenizers.

Critical strengths

The paper convincingly shows that the specialized machinery built for representation space can be dropped while still improving results. Combined with much faster convergence and few-step sampling without distillation, this gives the community a simpler and cheaper recipe that is easy to adopt.

Careful and honest experimentation.

Critical weaknesses
  1. The geometric diagnostics are self-described as correlational. They cannot explain why the recipe works, and they fail where it matters. , that is the less conditioned encoder variant trains to a better FID, and the tokenizer comparison has only three cases.

So this part may has neither explanatory nor predictive value.

  1. Everything is validated only on class-conditional ImageNet at one resolution, with a denoiser much wider than the token dimension. Prior work shows the wide head matters for narrow models, so the claim that a standard DiT is enough may not hold in capacity-constrained settings.

Text-to-image and higher resolutions are untested.

Questions for authors

See weaknesses.

Please answer the question carefully. And I may increase my score.

Reviewer_p5TWOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper present RiT, which shows how to training the DiT in the DINO V2 features domain. This method propose a training receipt: through adjust the training target function, training noise distribution, and jointly generate the global [CLS] token. Results show that this receipt is efficient, such as decrease the unguided FID from 1.87 to 1.45. My core concern is that these receipt are all from the existing works.

Another core concern is that this paper contains several reference errors, such as the author list of PixelDiT, the year of the ReDi, MaskDiT.

Critical strengths
  1. The method is simple, without adding machinery specific to representation spaces, such as a wide prediction head, or Riemannian transport on the hypersphere.
  2. The method also have a good performance in the few step generation, which do not need extra distillation training.
  3. The training recipt is effective, as shown in the experiment that this method converage quickly.
Critical weaknesses
  1. My core concern is that these components in the receipt are all from the existing works, such as the normalization and dimension-specific time shift are from RAE.
  2. Another core concern is that this paper contains several reference errors, such as the author list of PixelDiT, the year of the ReDi, MaskDiT.
  3. The ablation for v-prediction + [CLS] only shows results at 80 epochs, which cannot make the fair comparision with the full model.
Questions for authors

Please see Critical Weaknesses.

  1. Is there more result about the few-step generation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.