- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies why DiT width needs to be comparable to the token dimension in high-dimensional latent space. The paper shows that the bottleneck comes from the final linear layer instead of the transformer itself. The last linear layer of a width-d model can only output vectors in a d-dimensional subspace. If the target contains isotropic noise, the loss floor is proven to be 1 − d/n. The paper also shows that the sampling process suffers from the same rank bottleneck. The paper recommends predicting the clean data directly to avoid this loss floor, as the isotropic noise is not part of the learning target.
Critical strengths
- The paper writing is coherent and easy to follow.
- The paper studies a fundamental question of why diffusion transformers require a width comparable to the token dimension, through analyzing the rank of the final linear layer and the dimensionality of the data.
- The detailed theoretical proofs for velocity parameterization, EDM parameterization, and clean parameterization are sound.
- The theory makes accurate predictions on expected loss floor without training the model. It predicts the failures reported by RAE within 5% error (Table 5).
Critical weaknesses
- The discussion of related work is insufficient. The paper misses the discussion and comparison with two closely related works:
- RiT (https://arxiv.org/abs/2605.21981), which trains a vanilla DiT with x-prediction on DINOv2 features without a wide head.
- k-Diff (https://arxiv.org/abs/2601.21419), which relates the optimal prediction target to data dimensionality.
- The conclusion “A width-d feature is mapped to an n-dimensional token by W, whose rank is at most d” is closely related to the prior work Eckart–Young. There is limited novelty.
- Other than RAE and pixel-space generation, there is no exploration of high-dimensional VAE latent space, e.g., a VAE trained for reconstruction with 768+ latent channels.
- The paper lacks real-scale generation experiments. The generation results in Section 5.3 only use raw pixels and training is far from convergence (FID 94 to 355). So, the impact on generation quality in the RAE or RAEv2 setting with 80+ epochs is unclear.
Questions for authors
- Please refer to the weakness section above.
5 comments
Follow-up to Reviewer WLKt
1. Related work and novelty:
The criterion goes beyond Eckart-Young because it predicts what sampling does outside the head's span. On saved trained single-image fits (50 steps):
| Prediction | Sampler | Predicted energy ratio | Measured ratio |
|---|---|---|---|
| Noise | Euler | 608.9 | 608.9 |
| Noise | Heun | 716.2 | 716.2 |
| Half skip | Euler | 0.006334 | 0.006334 |
| Half skip | Heun | 0.004646 | 0.004646 |
For the specified SDE, the half-skip variance is against predicted.
The generation results are consistent with this mechanism. At width 256 (64×64, 150k steps, Euler FID; each arm trained with its own natural objective):
| Target | Complementary energy ratio(50-step Euler) | FID |
|---|---|---|
| Clean | 0 | 109.4 ± 3.6 (6) |
| Half skip | 0.0063 | 160.4 ± 11.8 (3) |
| Direct | 1 | 235.2 ± 4.0 (6) |
| EDM | 0.95 | 239.7 ± 3.1 (6) |
| Noise | 608.9 | 344.2 ± 0.2 (3) |
Clean and the half skip, which remove all or 99.4% of the out-of-span component, reach lower FID than direct and EDM, which keep nearly all of it; noise prediction amplifies it and has the highest FID. This supports the mechanism, although the complementary energy alone does not determine FID: EDM keeps slightly less than direct but scores slightly worse. The noise factor 608.9 is specific to the tested setting, with floored at 0.1 in the conversion and sampling started at .
2. Reconstruction-trained latents:
To test whether the mechanism extends beyond DINOv2 features, we used reconstruction-pretrained MAE-B/16 tokens with dimension 768. The measured MAE spectrum gives a direct-prediction excess interval of [0.1926, 0.7092] at , close to DINOv2-B’s [0.1930, 0.7173]. Together with the single-image fits below, this shows that reconstruction pretraining does not eliminate the bottleneck in this tested representation. MAE addresses the reconstruction-objective aspect of your concern; the VAE-specific experiment remains open.
| Width | Direct velocity error | Floor | Clean velocity error | Seeds |
|---|---|---|---|---|
| 192 | 0.7769 | 0.750 | 0.0002 | 1 |
| 384 | 0.5202 ± 0.0035 | 0.500 | 0.0004 ± 0.0003 | 3 |
| 768 | 0.0195 | 0 | 0.0003 | 1 |
A VAE with 768+ channels would have to be trained first; the same spectrum test applies to it unchanged.
3. Real-scale generation and training length:
The full head and the rank-256 head were each fine-tuned from the same trained width-768 backbone, for each of three seeds (64×64 Euler FID):
| Prediction | Full head | Rank-256 head | Difference of means |
|---|---|---|---|
| Direct | 171.7 ± 11.9 | 246.0 ± 12.5 | +74.3 |
| EDM | 161.3 ± 4.5 | 245.5 ± 1.0 | +84.2 |
| Clean | 89.4 ± 2.0 | 92.0 ± 2.0 | +2.6 |
On training length: between 25k and 150k steps, clean keeps improving (149.5 to 108.9), while direct and EDM stop improving by 25k steps and drift up (224.0 to 235.2, 226.7 to 239.7).
In a separate width-256 experiment, 10,000-sample Euler FID improves from 149.5 to 108.9 for clean prediction between 25k and 150k steps (three seeds). Direct and EDM change from 224.0 to 235.2 and from 226.7 to 239.7, respectively (six new seeds each). Thus extending training over this window does not close the quality gap. These trajectories do not establish optimization convergence; the sampling law itself holds for any weights under its stated architectural and sampler assumptions. The sampling law itself holds for any weights.
At real scale, RAE's DiT-DH uses a 2048-wide head, above its 768-dimensional tokens, so its success sits on the safe side of our boundary. We did not complete the probe of RAE's released checkpoint due to time constraint in the discussion period;
Follow-up to Reviewer WLKt
Dear Reviewer WLKt,
Thank you again for your thoughtful review. As the discussion period closes soon, we would like to confirm how the revised PDF answers your points.
- Novelty. The sampling law is tested on five parameterizations and three samplers, and the measured energies match the predictions (Tables 1 and 16). EDM lowers the training floor by 23% but pays the same rank penalty as direct (Table 3), which a floor-based account does not predict.
- RiT and k-Diff. Both are now discussed in Related Work (Section 7).
- Representation and scale. MAE-B/16 features follow the same floor: direct reaches 0.5202 against 0.500, and clean 0.0004 (Table 7). A test on a released RAE-scale model is not yet done, and we list it as a limitation (Appendix J).
We would be glad to know whether this addresses your concerns, and we are happy to clarify anything further.
Best regards,
The authors
Response to Reviewer WLKt
Thank you for the careful reading and the references. We clarify the connection to prior work and the purpose of the additional experiments.
1. Novelty beyond low-rank approximation
We build on the classical projection result and RAE's regression bound. The paper adds three concrete conclusions:
The new tests make the criterion concrete: 608.9× noise-energy amplification and 0.006334 half-skip energy retention, both matching 50-step Euler measurements with untrained weights. Noise conversion has a 0.1 floor and starts at t=0.001. The half skip still has the nonzero isotropic regression bound 0.25(1−r/n). The specified-SDE prediction is 0.6381 total complementary variance, versus 0.6384 untrained and 0.6390 trained. The direct/EDM/clean Euler checks remain evidence from the submitted Appendix G.
2. RiT and k-Diff
RiT's main setup has width 1152 and token dimension 384, so it has no architecturally forced output-rank deficit. This is consistent with our boundary; it does not test the rank-deficient regime.
For k-Diff's constant-k target u=kz−(1−k)ϵ, applying our criterion to a fixed restricted head gives Qx1=((1−k)/k)Qx0 for 0<k<1, under unclipped rectified-flow ODE or Euler sampling. The corresponding energy ratio is ((1−k)/k)2. At their Theorem 3.2 optimum k∗=D/(D+d), the amplitude ratio is d/D and the energy ratio is (d/D)2, under that theorem's assumptions. Here D is ambient and d intrinsic dimension. Clipping or time-varying k(t) requires the corresponding recurrence rather than this endpoint formula.