acceptodds
Under review as a conference paper at ICLR 2027

The Rank Bottleneck in Diffusion Transformers: Why and When Width Must Track Token Dimension

Abstract

Diffusion transformers on high-dimensional latent tokens have been reported to require a width comparable to the token dimension, even when fitting a single training example. We show that this requirement can arise from a rank bottleneck in the final per-token linear layer, rather than from a limitation of the transformer backbone itself. A width-d output layer can emit only a rank-d subspace of an n- dimensional token. When the regression target contains an isotropic component, that component cannot be represented through the layer, giving an error floor of 1 − d/n; more generally, the floor is determined by the target covariance beyond rank d. With nothing fitted, this floor predicts the reported failures with a mean error of 5%. The same bottleneck persists during sampling: a rank-d velocity field cannot alter the component of a sample outside its output subspace, leaving the corresponding part of the initial noise unchanged. This also explains why EDM preconditioning does not remove the floor. In contrast, predicting the clean token and reconstructing the isotropic component from the network input routes that component around the rank-limited output layer. For the clean target, the remain- ing floor can be bounded directly from the target spectrum, giving a criterion for when the ambient token dimension is or is not the relevant width scale. Across latent and pixel representations, our experiments support this view: the observed width requirement follows the effective rank of the emitted target rather than its ambient dimension. These results identify the output parameterization, rather than transformer capacity alone, as the source of the reported width constraint.

Primary area:
generative models
Keywords:
diffusion models, flow matching, diffusion transformers, latent diffusion, parameterization, representation encoders, rank constraints, preconditioning, generative models

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_WLKtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why DiT width needs to be comparable to the token dimension in high-dimensional latent space. The paper shows that the bottleneck comes from the final linear layer instead of the transformer itself. The last linear layer of a width-d model can only output vectors in a d-dimensional subspace. If the target contains isotropic noise, the loss floor is proven to be 1 − d/n. The paper also shows that the sampling process suffers from the same rank bottleneck. The paper recommends predicting the clean data directly to avoid this loss floor, as the isotropic noise is not part of the learning target.

Critical strengths
  • The paper writing is coherent and easy to follow.
  • The paper studies a fundamental question of why diffusion transformers require a width comparable to the token dimension, through analyzing the rank of the final linear layer and the dimensionality of the data.
  • The detailed theoretical proofs for velocity parameterization, EDM parameterization, and clean parameterization are sound.
  • The theory makes accurate predictions on expected loss floor without training the model. It predicts the failures reported by RAE within 5% error (Table 5).
Critical weaknesses
  • The discussion of related work is insufficient. The paper misses the discussion and comparison with two closely related works:
  • The conclusion “A width-d feature is mapped to an n-dimensional token by W, whose rank is at most d” is closely related to the prior work Eckart–Young. There is limited novelty.
  • Other than RAE and pixel-space generation, there is no exploration of high-dimensional VAE latent space, e.g., a VAE trained for reconstruction with 768+ latent channels.
  • The paper lacks real-scale generation experiments. The generation results in Section 5.3 only use raw pixels and training is far from convergence (FID 94 to 355). So, the impact on generation quality in the RAE or RAEv2 setting with 80+ epochs is unclear.
Questions for authors
  • Please refer to the weakness section above.
5 comments
AuthorsOpenReview ↗

Response to Reviewer WLKt

Thank you for the careful reading and the references. We clarify the connection to prior work and the purpose of the additional experiments.

1. Novelty beyond low-rank approximation

We build on the classical projection result and RAE's regression bound. The paper adds three concrete conclusions:

  • What sampling cannot change. Submitted Proposition 2 gives an exact invariant outside an arbitrary learned output span, for the stated deterministic updates and any network weights.
  • How parameterization changes that constraint. The scalar-skip criterion determines whether those directions are preserved, attenuated or amplified. It unifies the submitted EDM and clean-prediction analysis and extends it during rebuttal: a nonzero regression floor can coexist with near elimination of complementary noise.
  • What remains on full distributions. Submitted Proposition 3 gives clean-prediction bounds from the data spectrum; the rebuttal adds the direct-prediction counterpart. Both concern optimal conditional prediction rather than assuming that a trained narrow-versus-wide difference equals the total-risk floor.

The new tests make the criterion concrete: 608.9× noise-energy amplification and 0.006334 half-skip energy retention, both matching 50-step Euler measurements with untrained weights. Noise conversion has a 0.1 floor and starts at t=0.001t=0.001. The half skip still has the nonzero isotropic regression bound 0.25(1−r/n)0.25(1-r/n). The specified-SDE prediction is 0.6381 total complementary variance, versus 0.6384 untrained and 0.6390 trained. The direct/EDM/clean Euler checks remain evidence from the submitted Appendix G.

2. RiT and k-Diff

RiT's main setup has width 1152 and token dimension 384, so it has no architecturally forced output-rank deficit. This is consistent with our boundary; it does not test the rank-deficient regime.

For k-Diff's constant-kk target u=kz−(1−k)ϵu=kz-(1-k)\epsilon, applying our criterion to a fixed restricted head gives Qx1=((1−k)/k)Qx0Qx_1=((1-k)/k)Qx_0 for 0<k<10<k<1, under unclipped rectified-flow ODE or Euler sampling. The corresponding energy ratio is ((1−k)/k)2((1-k)/k)^2. At their Theorem 3.2 optimum k∗=D/(D+d)k^*=D/(D+d), the amplitude ratio is d/Dd/D and the energy ratio is (d/D)2(d/D)^2, under that theorem's assumptions. Here DD is ambient and dd intrinsic dimension. Clipping or time-varying k(t)k(t) requires the corresponding recurrence rather than this endpoint formula.

AuthorsOpenReview ↗

Follow-up to Reviewer WLKt

1. Related work and novelty:

The criterion goes beyond Eckart-Young because it predicts what sampling does outside the head's span. On saved trained single-image fits (50 steps):

PredictionSamplerPredicted energy ratioMeasured ratio
NoiseEuler608.9608.9
NoiseHeun716.2716.2
Half skipEuler0.0063340.006334
Half skipHeun0.0046460.004646

For the specified SDE, the half-skip variance is 4.75×10−44.75\times10^{-4} against 4.751×10−44.751\times10^{-4} predicted.

The generation results are consistent with this mechanism. At width 256 (64×64, 150k steps, Euler FID; each arm trained with its own natural objective):

TargetComplementary energy ratio(50-step Euler)FID
Clean0109.4 ± 3.6 (6)
Half skip0.0063160.4 ± 11.8 (3)
Direct1235.2 ± 4.0 (6)
EDM0.95239.7 ± 3.1 (6)
Noise608.9344.2 ± 0.2 (3)

Clean and the half skip, which remove all or 99.4% of the out-of-span component, reach lower FID than direct and EDM, which keep nearly all of it; noise prediction amplifies it and has the highest FID. This supports the mechanism, although the complementary energy alone does not determine FID: EDM keeps slightly less than direct but scores slightly worse. The noise factor 608.9 is specific to the tested setting, with tt floored at 0.1 in the conversion and sampling started at t=0.001t=0.001.

2. Reconstruction-trained latents:

To test whether the mechanism extends beyond DINOv2 features, we used reconstruction-pretrained MAE-B/16 tokens with dimension 768. The measured MAE spectrum gives a direct-prediction excess interval of [0.1926, 0.7092] at d=n/2d=n/2, close to DINOv2-B’s [0.1930, 0.7173]. Together with the single-image fits below, this shows that reconstruction pretraining does not eliminate the bottleneck in this tested representation. MAE addresses the reconstruction-objective aspect of your concern; the VAE-specific experiment remains open.

Width ddDirect velocity errorFloor 1−d/7681-d/768Clean velocity errorSeeds
1920.77690.7500.00021
3840.5202 ± 0.00350.5000.0004 ± 0.00033
7680.019500.00031

A VAE with 768+ channels would have to be trained first; the same spectrum test applies to it unchanged.

3. Real-scale generation and training length:

The full head and the rank-256 head were each fine-tuned from the same trained width-768 backbone, for each of three seeds (64×64 Euler FID):

PredictionFull headRank-256 headDifference of means
Direct171.7 ± 11.9246.0 ± 12.5+74.3
EDM161.3 ± 4.5245.5 ± 1.0+84.2
Clean89.4 ± 2.092.0 ± 2.0+2.6

On training length: between 25k and 150k steps, clean keeps improving (149.5 to 108.9), while direct and EDM stop improving by 25k steps and drift up (224.0 to 235.2, 226.7 to 239.7).

In a separate width-256 experiment, 10,000-sample Euler FID improves from 149.5 to 108.9 for clean prediction between 25k and 150k steps (three seeds). Direct and EDM change from 224.0 to 235.2 and from 226.7 to 239.7, respectively (six new seeds each). Thus extending training over this window does not close the quality gap. These trajectories do not establish optimization convergence; the sampling law itself holds for any weights under its stated architectural and sampler assumptions. The sampling law itself holds for any weights.

At real scale, RAE's DiT-DH uses a 2048-wide head, above its 768-dimensional tokens, so its success sits on the safe side of our boundary. We did not complete the probe of RAE's released checkpoint due to time constraint in the discussion period;

AuthorsOpenReview ↗

Follow-up to Reviewer WLKt

Dear Reviewer WLKt,

Thank you again for your thoughtful review. As the discussion period closes soon, we would like to confirm how the revised PDF answers your points.

  • Novelty. The sampling law is tested on five parameterizations and three samplers, and the measured energies match the predictions (Tables 1 and 16). EDM lowers the training floor by 23% but pays the same rank penalty as direct (Table 3), which a floor-based account does not predict.
  • RiT and k-Diff. Both are now discussed in Related Work (Section 7).
  • Representation and scale. MAE-B/16 features follow the same floor: direct reaches 0.5202 against 0.500, and clean 0.0004 (Table 7). A test on a released RAE-scale model is not yet done, and we list it as a limitation (Appendix J).

We would be glad to know whether this addresses your concerns, and we are happy to clarify anything further.

Best regards,

The authors

Reviewer_KTMqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper argues that the rank of the final linear output head in diffusion transformers acts as a bottleneck, necessitating an increase in the model's width. Specifically, it demonstrates that while modeling velocity leads to a theoretical error bound, this limitation can be bypassed by predicting clean tokens and utilizing the input via a skip connection. The authors explain that the effective rank of the model's output is the critical factor determining the required width.

Critical strengths

The paper provides a link between the representational constraints of the final output layer, learning error, and the exact conservation quantities during the generation process. In single-image experiments, the predicted lower bounds align well with empirical measurements. Furthermore, the authors acknowledge and analyze the persistent constraints that remain even when predicting clean tokens within the context of the full data distribution.

Critical weaknesses

[W1] The paper does not clearly distinguish its theoretical and methodological contributions from prior work. Theorem 1 and Appendix B.1 of the original RAE paper [1] already present a lower bound based on the covariance tail and use the rank constraint of the final output matrix. The current paper removes input-layer constraints and adds a bias term, but does not clarify the substantive advance. The authors should distinguish these existing results from their specific contributions, such as the exact sampling conservation equations and bounds for the full data distribution.

[W2] Changing both the prediction method and training loss makes the source of performance gains unclear. Direct velocity prediction and EDM use velocity error, while clean-token prediction uses token-reconstruction error with different time-dependent weights. Thus, the performance differences in Section 5.3 may reflect optimization differences as well as output-layer constraints. A comparison using identical time weights and loss spaces would help isolate these effects.

[W3] The paper does not establish that the full-distribution error gap is caused by a rank constraint. Section 5.2 and Table 6 interpret the narrow–wide performance gap as the cost of a rank constraint, comparing direct velocity prediction with (1−d/n)(1-d/n). But Proposition 1 bounds total regression error; it does not directly bound the error difference between narrow and wide models. That difference also depends on the covariance of the conditional mean and, in trained models, on width-dependent expressivity and optimization error. The additional error for direct velocity prediction therefore needs separate justification.

[W4] The evidence does not support quantitatively attributing generation-quality degradation to rank constraints. Section 5.3 and Table 1 attribute part of the FID increase from reducing model width to rank constraints, assuming model size has the same effect across prediction methods. However, the reduced-width direct-velocity model has only one successful run and had not fully converged. “Ideally constructed” samples illustrate residual-noise effects but do not justify decomposing the total FID gap by cause. Without controlled experiments or more replicated runs, this attribution should be framed as an observation.

[1] Zheng, Boyang, et al. "Diffusion transformers with representation autoencoders." ICLR 2026.

Questions for authors

[Q1] What does the theory add beyond prior work? Could you provide a direct comparison between Proposition 1 and Equation (5) and Theorem 1 (along with Appendix B.1) from the RAE paper? Specifically, please clarify which scenarios are now covered by removing input-layer constraints and allowing for bias terms.

[Q2] Do clean-token prediction gains persist with aligned loss weights? Could you present results using identical time-dependent velocity error weights, while varying only the model’s direct output (i.e., predicting velocity versus predicting clean tokens)? It would be particularly helpful to see this comparison both with and without rank constraints on the output layer.

[Q3] When does the additional error from direct velocity prediction approach (1−d/n)(1-d/n)? How do you distinguish the total-error lower bound from the error increase relative to a wider model, and justify Table 6 using the covariance of the conditional mean? Please provide learning curves and individual error plots for the wide and narrow models.

[Q4] Can output-layer rank be isolated from overall model width? Could you conduct a comparison where different rank constraints are applied exclusively to the output matrix, while keeping the backbone architecture constant? I would like to see how such controlled experiments support the attribution of the FID results discussed in Section 5.3.

7 comments
AuthorsOpenReview ↗

Response to Reviewer KTMq (Part 1)

We thank the reviewer for four precise questions.

1. What is new beyond the regression bound? Q1

Our contribution has three parts:

  • Exact sampling law: Proposition 2 identifies the component of the initial noise that a direct velocity head cannot change, independently of training.
  • Parameterization criterion: the input-skip coefficient determines both the isotropic regression bound and the evolution outside the head's span. It explains the submitted EDM result and yields the new noise-prediction, half-skip and SDE predictions.
  • Full-distribution bounds: Proposition 3 treats clean prediction and the new Proposition 4 (see Q3) treats direct prediction. Both use the conditional-mean spectrum, which separates optimal rank excess from total regression error.

The formulations also cover different scenarios:

AspectRAE Theorem 1 as statedOur formulation
PredictorBf(Axt,t)Bf(Ax_t,t)Wh+bWh+b
Input accessThrough a rank-limited input projectionFull-dimensional input and other available context permitted
OutputLinear projection, no explicit biasRank-rr affine head

Here hh may be any function of the input, time and available context, so our formulation covers a wide backbone with full-dimensional input and a narrow readout. RAE's proof already uses output rank, so this wider scope is not our central novelty claim.

The criterion predicts sampling, not only the floor. Each value was computed before the run; all checks use 50 steps:

CheckPredictedUntrained weightsTrained fits
Noise prediction, Euler608.9608.9608.9
Noise prediction, Heun716.2716.2716.2
Half skip, Euler0.0063340.0063340.006334
Half skip, Heun0.0046460.0046460.004646
Direct, SDE0.63810.63840.6381

The noise-prediction conversion floors tt at 0.1, and sampling starts at t=0.001t=0.001. SDE values are the total out-of-span variance, which includes both surviving initial noise and injected noise; a trained 64×64 generator gives 0.6390. These identities hold for any weights, so they test the criterion itself rather than training quality.

AuthorsOpenReview ↗

Response to Reviewer KTMq (Part 2)

2. Does the gain survive when both arms use the same loss? Q2

Yes. We trained each arm with each loss. FID at 64×64 (Euler, mean ± SD, seeds in parentheses):

WidthLossDirectClean
256 (rank-limited)Velocity235.2 ± 4.0112.9 ± 2.1 (3)
256 (rank-limited)Clean-space237.3 ± 5.9 (3)109.4 ± 3.6 (6)
768 (full rank)Velocity173.8 ± 11.3 (3)91.4 (1)
768 (full rank)Clean-space165.5 (1)93.8 ± 2.5 (3)

Clean is better by about 125 FID at width 256 and by 70 to 80 at width 768, whichever loss is used. Changing the loss moves an arm by only 2 to 8 FID. So the gain comes from what the network predicts, not from how the loss is weighted, and it is largest when the head is rank-limited. Clean prediction with velocity loss uses the previously stated 0.05 conversion cap.

These results show that the advantage persists across both tested objectives. The joint fixed-backbone rank-by-loss refit, which directly compares restricted and unrestricted heads under aligned loss weights, will be reported separately.

3. Can rank sensitivity be tested at fixed backbone width? Q4

At backbone width 768, both full-head and rank-256 models were each fine-tuned from the same trained backbone, for each of three seeds. The completed 64×64 Euler FIDs are mean ± SD over three seeds:

PredictionFull headRank-256 headDifference of means
Direct171.7 ± 11.9246.0 ± 12.5+74.3
EDM161.3 ± 4.5245.5 ± 1.0+84.2
Clean89.4 ± 2.092.0 ± 2.0+2.6

The large direct/EDM differences persist after fine-tuning, while the observed clean difference is much smaller. These extend the frozen-feature refits already reported and support rank sensitivity at fixed backbone width without subtracting width penalties across parameterizations.

In a separate width-256 experiment, 10,000-sample Euler FID from 25k to 150k steps changes from 149.5 to 108.9 for clean (three seeds), 224.0 to 235.2 for direct, and 226.7 to 239.7 for EDM (six new seeds each). Thus the quality gap persists over the evaluated training window.

AuthorsOpenReview ↗

Response to Reviewer KTMq (Part 3)

3. What does Table 6 establish? Q3

Proposition 1 bounds total regression error. Table 6 measures something else: how much more error a rank-limited model makes than a less restricted one. Irreducible conditional noise contributes to both models' errors and cancels from their difference, so the difference depends only on the predictable part of the target, its conditional mean. The revision states this as a new result.

Proposition 4 (new). Both models see (xt,t)(x_t, t), upstream computation is unrestricted, and the rank-rr affine head Wh+bWh+b is shared across tt. Errors are per dimension and averaged over t∼U(0,0.95)t \sim U(0, 0.95). Let m=E[v∣xt,t]m = E[v \mid x_t, t] and Σm=Et Cov(m∣t)\Sigma_m = E_t\,\mathrm{Cov}(m \mid t), with eigenvalues μ1≥⋯≥μn\mu_1 \ge \dots \ge \mu_n, and let λ1≥⋯≥λn\lambda_1 \ge \dots \ge \lambda_n be the eigenvalues of the data covariance (the noise has unit variance). Then:

  • (i) The excess of the best rank-rr head over an unrestricted model is exactly 1n∑i>rμi\frac{1}{n}\sum_{i>r}\mu_i.
  • (ii) This excess lies between 1n\frac{1}{n} times the sum of the n−rn-r smallest values of G(λi)G(\lambda_i), where G(λ)=Et[(tλ−(1−t))2/(t2λ+(1−t)2)]G(\lambda) = E_t\big[(t\lambda-(1-t))^2/(t^2\lambda+(1-t)^2)\big], and (1−r/n)+1n∑i>rλi(1-r/n) + \frac{1}{n}\sum_{i>r}\lambda_i.

Proof sketch. (i) Since E[m∣t]=E[v]E[m \mid t] = E[v] for every tt, the best head projects mm onto the top-rr principal subspace of Σm\Sigma_m (Eckart-Young). (ii) Σm\Sigma_m is at least the covariance explained by the best linear predictor, which is diagonal in the data eigenbasis with entries G(λi)G(\lambda_i) and is attained by Gaussian data. It is at most Cov(v)=Σ+I\mathrm{Cov}(v) = \Sigma + I. Sums of the smallest eigenvalues preserve this order.

When does the excess reach 1−r/n1-r/n? The upper end is reached when the Bayes error is zero, so that m=vm = v. It equals exactly 1−r/n1-r/n when, in addition, the clean data have rank at most rr, because outside rr directions v=z−ϵv = z - \epsilon then contains only the isotropic noise. A single image meets both conditions: given xtx_t, tt and the token position, vv is determined, and its 256 tokens span at most 255 dimensions. At n=768n=768, r=384r=384, the fixed-backbone fit gives 0.5016 against 0.500. Full distributions have Bayes error, so their excess lies inside the interval.

At r=n/2r=n/2, the intervals below come from (ii) with the measured spectra (clean uses Proposition 3), and the updated results are:

nnArmOptimal-class intervalWidth gapFixed-width fine-tuning gapRefit gap
384Direct[0.1936, 0.7117]0.52330.42680.4530
768Direct[0.1930, 0.7173]0.48400.39440.4208
1024Direct[0.1933, 0.7123]0.4602Not run0.3827
384Clean[0.0814, 0.2114]0.17660.15670.1662
768Clean[0.0835, 0.2164]0.17350.15440.1664
1024Clean[0.0811, 0.2111]0.1624Not run0.1488

Width gap changes backbone width. Fixed-width fine-tuning changes head rank while retaining backbone width; refit uses frozen features. Direct uses velocity MSE and clean uses clean-space MSE. Clean intervals retain the submitted Table 6 values.

What this answers: positive refit gaps occur without changing backbone features, providing evidence of rank sensitivity beyond a width sweep. The intervals concern optimal predictors; measured gaps also contain differences in learning error. Therefore we do not identify the width gap with an exact representational penalty.

Learning curves. Held-out velocity error for direct at n=1024n=1024, the one width pair whose curves were logged every 10k steps:

Step10k20k30k40k
Narrow (width 512)1.2881.2441.2251.214
Wide (width 1024)0.8660.8000.7720.755
Gap0.4220.4440.4540.460

Both models still improve, but their gap grows.

Error against noise level. Narrow minus wide held-out error of the final direct models, by bins of tt (t=0t=0 is pure noise):

nn[0,0.2)[0, 0.2)[0.2,0.4)[0.2, 0.4)[0.4,0.6)[0.4, 0.6)[0.6,0.8)[0.6, 0.8)[0.8,0.95)[0.8, 0.95)
3840.520.580.570.520.37
7680.520.560.530.450.30
10240.520.550.510.410.26

Both ends agree with Proposition 4 applied at a single noise level. As t→0t \to 0 the input is pure noise, the predictable part of vv is the noise itself, and the excess tends to 1−r/n=0.51-r/n = 0.5. As t→1t \to 1 only the tail of the data spectrum remains, about 0.21 here. The revision will include these curves and the error-versus-tt plots for every wide and narrow model.

Reviewer_H3ZmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies diffusion transformers whose final per-token output map has rank below the token dimension. It derives a regression-error floor, an invariant showing that deterministic sampling retains noise outside the output span, and clean-prediction excess-risk bounds governed by the data spectrum. Validation covers single-image fitting, full latent distributions, and image generation, with comparisons across prediction targets and model widths.

Critical strengths
  1. The sampling analysis gives a precise architectural limitation. Proposition 2 shows that velocity-based deterministic updates preserve the component outside span(W,b), regardless of training, leaving at least n−d−1 residual dimensions when bias is included. The EDM calculation predicts approximately 95% energy retention after 50 Euler steps, consistent with Appendix G. This connects the output restriction to sampling behavior more directly than a regression bound alone and is a useful theoretical contribution.

  2. Several predictions receive quantitative validation. Across twelve constrained settings, direct-prediction error is 1.03–1.07 times the proposed floor. Proposition 3 extends the clean-prediction analysis beyond single-image fitting to full distributions. Together, these results provide a useful account of how the target spectrum affects a narrow output head. The explicit disclosure of divergence, incomplete convergence, and scoring differences also helps delimit the empirical evidence.

Critical weaknesses
  1. The contribution of the regression floor substantially overlaps with prior work. RAE[1] (ICLR 2026), Theorem 1 and Appendix B.1, already derives the covariance-tail bound and the isotropic specialization 1−d/n using the final projection's image. The affine formulation and weaker upstream assumptions refine this result. Clean prediction with input-based velocity reconstruction is established in JiT[2]; analytic treatment of ambient components also appears in AsymFlow[3] and Chen et al.[4]. These works do not establish the same arbitrary-output-span deterministic invariant: Chen et al. assume subspace-supported data and stochastic dynamics, while AsymFlow uses a designed target-noise projector. The sampling invariant, EDM analysis, and clean-spectrum bounds therefore deserve credit, but the overlap reduces the novelty attributable to the paper's central regression-floor claim.

  2. The experiments do not isolate how much of the quality gap is caused by output rank. Width sweeps also change backbone capacity and potentially input information. The proposed rank-attributable FID difference assumes other effects cancel across objectives, without an established additive decomposition. A fixed, sufficiently wide backbone and full-rank input, with only readout rank varied, would isolate this more directly. Loss weighting also differs: clean MSE corresponds to (1−t)²-weighted velocity MSE. JiT[2], Table 2, already crosses prediction and loss spaces; the attribution needs a comparable control in this paper's settings.

    Although Equation (6) is explicitly presented as an empirical model, subtracting the unconstrained-width baseline in the full-distribution experiments does not by itself isolate the representational penalty: estimation and optimization errors may also change with width. Proposition 1 bounds total risk, not necessarily avoidable error. For independent standard Gaussian data and noise at t=1/2, Cov(v)=2I but E[v|x_t]=0, so rank-induced excess risk is zero. With only output rank constrained and otherwise unrestricted computation, averaging over t ∼ U(0, 0.95) gives optimal per-dimension excess velocity risk of approximately 0.402(1−d/n), versus the target-projection floor 2(1−d/n). This limits the interpretation of Table 6; it does not refute Proposition 1, Proposition 2, or the near-zero-Bayes-risk single-image results.

[1] https://proceedings.iclr.cc/paper_files/paper/2026/file/3c4141c12660ad3625eb4ae845e0a6f9-Paper-Conference.pdf | Zheng et al. Diffusion Transformers with Representation Autoencoders. ICLR 2026.

[2] https://openaccess.thecvf.com/content/CVPR2026/papers/Li_Back_to_Basics_Let_Denoising_Generative_Models_Denoise_CVPR_2026_paper.pdf | Li and He. Back to Basics: Let Denoising Generative Models Denoise. CVPR 2026.

[3] https://arxiv.org/pdf/2605.12964v2 | Chen et al. Asymmetric Flow Models. Accepted at NeurIPS 2026; cited sections from v2. Venue record: https://hanshengchen.com/asymflow/

[4] https://proceedings.mlr.press/v202/chen23o.html | Chen, Huang, Zhao, and Wang. Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data. ICML 2023.

Questions for authors
  1. What does Proposition 1 add beyond RAE[1], Theorem 1 and Appendix B.1? Please distinguish changes in assumptions from new conclusions, and identify which claims rest on Propositions 2–3 instead.
  2. What justifies interpreting the width-baseline difference in Table 6 as a representational penalty when estimation and optimization errors may vary with width? How does this interpretation account for the time-averaged Gaussian example above?
  3. Is there a comparison that varies only readout rank while holding the backbone, input information, and converted loss weights fixed? Such a control would directly affect my assessment of the output-rank attribution, especially the interpretation of the FID differences.

[1] https://proceedings.iclr.cc/paper_files/paper/2026/file/3c4141c12660ad3625eb4ae845e0a6f9-Paper-Conference.pdf | Zheng et al. Diffusion Transformers with Representation Autoencoders. ICLR 2026.

[2] https://openaccess.thecvf.com/content/CVPR2026/papers/Li_Back_to_Basics_Let_Denoising_Generative_Models_Denoise_CVPR_2026_paper.pdf | Li and He. Back to Basics: Let Denoising Generative Models Denoise. CVPR 2026.

[3] https://arxiv.org/pdf/2605.12964v2 | Chen et al. Asymmetric Flow Models. Accepted at NeurIPS 2026; cited sections from v2. Venue record: https://hanshengchen.com/asymflow/

[4] https://proceedings.mlr.press/v202/chen23o.html | Chen, Huang, Zhao, and Wang. Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data. ICML 2023.

5 comments
AuthorsOpenReview ↗

Response to Reviewer H3Zm

Thank you for the careful and detailed review. We distinguish the prior bound, our contribution and the new controls below.

1. Novelty: three concrete results

We build on RAE's covariance-tail bound and established clean prediction, including JiT. Our contribution is a theory of which directions a rank-limited diffusion model can represent and how sampling evolves outside its output span.

A. An exact sampling law (submitted Proposition 2)

Let QQ project orthogonally outside span(W,b)\mathrm{span}(W,b). For direct velocity prediction, the stated velocity-based deterministic updates preserve this component exactly: QxK=Qx0Qx_K=Qx_0. This holds for any trained weights and data distribution. It identifies the part of the initial noise that the architecture cannot modify.

B. A parameterization criterion

For v^=a(t)xt+c(t)(Wh+b)\widehat v=a(t)x_t+c(t)(Wh+b), the skip coefficient a(t)a(t) determines both the isotropic regression lower bound and the evolution outside the head's span. This unifies the submitted EDM and clean-prediction results and extends them during rebuttal. It quantifies regression limits and sampling residuals separately: EDM retains approximately 95% of complementary noise energy after 50 Euler steps, while the half skip retains only 0.6334% despite a nonzero bound of 0.25(1−r/n)0.25(1-r/n).

C. Bounds for full-distribution prediction

The submitted Proposition 3 bounds the clean-prediction rank excess using the data spectrum; the rebuttal extends this analysis to direct velocity prediction. These bounds concern the covariance of the conditional mean, separating optimal rank excess from the total-target projection floor. The projection identity is classical; the contribution is its diffusion-specific spectral bounds.

The rebuttal adds the following 50-step checks; the direct/EDM/clean Euler checks were already in Appendix G:

New check, 50 stepsPredictionMeasurement
Noise prediction, Euler energy ratio608.9608.9
Half skip, Euler energy ratio0.0063340.006334
Direct, specified SDE variance ratio0.63810.6384 untrained; 0.6390 trained

The first two use untrained weights; noise conversion has denominator floor 0.1 and starts at t=0.001t=0.001. SDE values are total complementary variance. These measurements test the criterion under the specified conversion and sampler settings.

2. Table 6

With unrestricted upstream computation and an optimally chosen shared rank-rr affine output space, excess risk is the tail of the time-averaged conditional-mean covariance divided by nn. This recovers your Gaussian value 0.401881(1−r/n)0.401881(1-r/n) for t∼U(0,0.95)t \sim U(0, 0.95).

At r=n/2r=n/2, the reported references and measurements are:

nnArmOptimal-class intervalWidth gapFixed-backbone gapRefit gap
384Direct[0.1936, 0.7117]0.52330.42680.4530
768Direct[0.1930, 0.7173]0.48400.39440.4208
1024Direct[0.1933, 0.7123]0.4602not yet reported0.3827
384Clean[0.0814, 0.2114]0.1766not yet reported0.1662
768Clean[0.0835, 0.2164]0.1735not yet reported0.1664
1024Clean[0.0811, 0.2111]0.1624not yet reported0.1488

* We will provide the remaining fixed-backbone results in our final update.

Direct uses velocity MSE; clean uses clean-space MSE. The intervals describe ideal classes, not guaranteed bounds on trained gaps. Different learning errors remain, so differences between these measurements are not a causal decomposition.

3. Rank-only and loss controls

With backbone width and token dimension fixed at 384, direct head ranks 96, 192 and 288 give errors 0.7584, 0.5030 and 0.2518, versus floors 0.750, 0.500 and 0.250, over two seeds. On frozen width-768 generators, rank-256 versus full-rank refits change mean FID by +82.1 direct, +82.4 EDM and −2.8 clean, over three seeds.

Separately, under clean-space loss at width 256, direct scores 237.3 ± 5.9 versus clean's 109.8 ± 2.8, over three seeds. These support the mechanism through complementary controls; the joint fixed-backbone rank-by-loss comparison is a planned additional control.

AuthorsOpenReview ↗

Follow-up to Reviewer H3Zm: Additional results

We now update the full-distribution table and report three-seed controls.

1. Full-distribution excess: answering Q2

Thank you for your thoughtful review as your Gaussian example helped us clarify an important distinction. Proposition 1 bounds the total regression error, while Table 6 measures the additional error relative to a less constrained model. For the optimal predictor, this additional error is determined by the covariance of the conditional mean, the part of the target that can be predicted from the input. Irreducible conditional noise contributes to both models' errors and therefore does not determine their difference.

For t∼U(0,0.95)t\sim U(0,0.95), this gives your value of 0.401881(1−r/n)0.401881(1-r/n). Accordingly, we use the conditional-mean analysis, rather than the total-target error floor, to characterize optimal rank excess over the full distribution.

The updated measurements at r=n/2r=n/2 are shown below. The width gap compares different backbone widths. Fixed-width fine-tuning changes head rank while keeping the backbone width unchanged. Refit also varies head rank but uses frozen backbone features.

nnArmOptimal-class intervalWidth gapFixed-width fine-tuning gapRefit gap
384Direct[0.1936, 0.7117]0.52330.42680.4530
768Direct[0.1930, 0.7173]0.48400.39440.4208
1024Direct[0.1933, 0.7123]0.46020.3827
384Clean[0.0814, 0.2114]0.17660.15670.1662
768Clean[0.0835, 0.2164]0.17350.15440.1664
1024Clean[0.0811, 0.2111]0.16240.1488

Direct uses velocity MSE; clean uses clean-space MSE. (We did not run the fine-tuning comparison at n = 1024; the refit column covers that case.)

What this establishes: positive refit gaps occur even with frozen backbone features, so the evidence for rank sensitivity extends beyond changing backbone width. The intervals describe optimal-class rank penalties; trained-model gaps can also contain unequal estimation and optimization errors. Thus these results support the mechanism without treating the width gap as an exact representational penalty.

2. Controlled generation: addressing Q3

At backbone width 768, the full head and the rank-256 head were each fine-tuned from the same trained backbone, for each of three seeds. The completed 64×64 Euler FIDs are mean ± SD over three seeds:

PredictionFull headRank-256 headDifference of means
Direct171.7 ± 11.9246.0 ± 12.5+74.3
EDM161.3 ± 4.5245.5 ± 1.0+84.2
Clean89.4 ± 2.092.0 ± 2.0+2.6

The large differences in Direct and EDM remain after fine-tuning both models with the same backbone width, while the difference in Clean remains much smaller. This extends the earlier frozen-feature refit results and removes the need to assume that width effects cancel out across different parameterizations.

FID trajectories during training further support this difference: In a separate width-256 experiment, Clean improves from 149.5 to 108.9 between 25k and 150k steps, while Direct and EDM stop improving after 25k steps and gradually worsen (224.0 to 235.2 and 226.7 to 239.7), respectively.

The single-image rank-only control also now has three seeds. At fixed backbone width and token dimension 384, head ranks 96, 192 and 288 give velocity errors 0.7589 ± 0.0007, 0.5031 ± 0.0002 and 0.2518 ± 0.0001, against floors 0.750, 0.500 and 0.250.

These address the fixed-width rank intervention in Q3. The joint fixed-backbone rank-by-loss refit will be reported separately.

3. Further evidence for the novelty in Q1

We attribute the covariance-tail regression floor to prior work. Our contributions are the exact sampling law for arbitrary output spans, diffusion-specific spectral bounds, and the parameterization criterion.

To further test our sampling predictions, we conducted 50-step checks on saved, trained single-image fits. The results show noise-energy amplification of 608.9 with Euler and 716.2 with Heun. The half-skip retains 0.006334 and 0.004646, respectively. For the specified SDE, the complementary variance, including both retained and injected noise, is 4.75×10−44.75\times10^{-4}, closely matching the predicted 4.751×10−44.751\times10^{-4}. These results extend our earlier untrained-weight experiments and provide further validation of the quantitative sampling predictions on trained models.

AuthorsOpenReview ↗

Additional Results to Reviewer H3Zm (Part 2)

Q3. A comparison that varies only the readout rank

Design. For each arm and each of three seeds, we froze the trained width-768 backbone (64×64, n=768n = 768) and refitted only its linear readout, at rank 768 and at rank 256. We did this under two loss weightings, and in each case the same weighting is applied to all three arms:

  • Velocity-space weighting: squared velocity error. For clean, the conversion denominator is clipped at 0.05, as stated before.
  • Clean-space weighting: squared clean-token error, which is the velocity error weighted by (1−t)2(1-t)^2.

Within each column of the table below, the backbone, its input, the fitting data, the noise draws, the sampler and the loss weighting are identical. Only the readout rank changes between the two refits being compared.

Check. Refitting direct seed 0 with its own loss through the new code reproduces our earlier refit exactly: FID 251.90 at rank 256 and 171.21 at rank 768.

Results. FID change from rank 768 to rank 256 (Euler, 10,000 samples), as mean ± SD of the three paired seed differences:

PredictionVelocity-space weightingClean-space weightingOut-of-span energy kept at rank 256
Direct+82.1 ± 30.1+81.1 ± 27.11
EDM+82.4 ± 5.6+77.0 ± 6.80.9499
Clean−2.1 ± 0.2−2.8 ± 0.23×10−143\times10^{-14}

The large SD for direct reflects seed-to-seed variation of its backbones, already visible in our earlier refit (SD 20.6 at rank 256).

What this establishes

  • The rank cost is set by the parameterization, not by the loss weighting. Under either weighting, reducing only the readout rank costs direct about 81 FID and EDM 77 to 82, while clean loses nothing.
  • The FID differences follow the mechanism. At rank 256 the sampler keeps exactly 1 (direct), 0.9499 (EDM) and essentially 0 (clean) of the noise outside the readout span, under both weightings. These are the values Proposition 2 and Corollaries 1 and 2 assign to each parameterization, so the FID cost tracks the noise each one keeps.
  • A side result on the refit protocol. Refitted with the velocity weighting at full rank, the clean head scores 94.3, close to the trained head's 93.8 (98.4 with its own loss). The small refit cost we noted earlier therefore comes from the refit loss, not from the refit procedure.

Together with the single-image head-rank fits (0.7589, 0.5031 and 0.2518 against floors 0.750, 0.500 and 0.250) and the fine-tuned rank-256 heads (+74.3, +84.2, +2.6), this gives three controls in which only the readout rank changes. All three point to the same attribution.

Reviewer_2sw6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that the reported width requirement in diffusion transformers is primarily an output-rank bottleneck, not an inherent limitation of the transformer backbone: a width-d linear head cannot emit all directions of an n-dimensional, noise-containing target when d<n.

Main contributions

  • Quantitative error floor: The irreducible representational error is the target covariance's tail beyond rank d. For velocity targets containing unit isotropic noise, this gives a per-coordinate lower bound of 1-d/n, regardless of backbone capacity.
  • Sampling consequence: Direct velocity prediction preserves initial noise outside the output span. The paper's rectified-flow EDM parameterization reduces the training floor but ultimately preserves that residual noise too.
  • A parameter-free escape and design rule: Predict the clean token and reconstruct velocity as (\hat z-x_t)/(1-t), bypassing the head for the noise component. Width then depends on the clean target's effective linear rank—not necessarily its ambient dimension. This offers large savings for pixel patches, but modest savings for high-rank DINOv2 latents.

The strongest result is the precise mechanism linking output rank, regression error, and sampling behavior. The generation evidence is supportive: models are undertrained, several runs diverge, and some comparisons have only one surviving seed. The guarantees are specific to fixed linear heads and the analyzed deterministic sampling setups.

Critical strengths
  • A precise, quantitatively validated explanation of the width bottleneck. The paper identifies the final linear output map as the source of an unavoidable rank constraint, derives the 1-d/n isotropic error floor, and matches previously reported fitting failures within approximately 5% without fitting parameters. Although the underlying linear algebra is classical, identifying it as the operative mechanism is consequential: it replaces a broad capacity explanation with a falsifiable prediction and directly informs architectural choices.

  • A connection between regression limitations, sampling behavior, and an actionable remedy. The paper shows that the bottleneck preserves out-of-span initial noise during direct deterministic sampling, explains why the analyzed EDM preconditioning does not eliminate it, and demonstrates how clean prediction bypasses the isotropic constraint without adding parameters. Importantly, it also establishes that clean prediction remains limited by the data's spectrum, explaining why the benefit is much greater for pixels than for DINOv2 latents. This combination of mechanism, remedy, and clearly delineated applicability makes the contribution useful beyond the motivating single-image experiment.

Critical weaknesses
  • The generation experiments do not robustly establish the practical magnitude of the bottleneck. Models are far from converged; the constrained direct result at 6464 has only one surviving seed, and the 256256 experiments have one seed per configuration and non-monotonic width results. Moreover, the "ideal" samples are diagnostic constructions, not formal FID lower bounds. These limitations make the claimed rank-attributable generation penalty substantially less secure than the algebraic mechanism, weakening the evidence for practical impact.

  • The theoretical floor does not by itself establish the width needed for successful learning. The covariance-tail calculation characterizes an output-subspace representational constraint, but achievable regression performance also depends on the input-conditioned target, optimization, and backbone capacity. The additive model in Equation (6) assumes approximately unchanged in-subspace error across widths, while the full-distribution direct-error comparison lacks the conditional-mean treatment developed for clean prediction. Consequently, the proposed width rule is better supported as a necessary representational constraint than as a sufficient design prescription; this distinction matters to the paper's broader claim of separating output rank from transformer capacity.

Questions for authors
  • What justifies predicting the full-distribution direct model's excess error by 1-d/n? Proposition 1 bounds total regression error, whereas Section 5.2 subtracts an unconstrained model's error. Can you explain the assumptions needed to connect these quantities, accounting for Bayes error and the covariance of the conditional mean \mathbb E[v\mid x_t,t]? This would clarify whether the comparison validates a theoretical prediction or an empirical approximation.
  • How well supported is the "rank-attributable" FID decomposition? Subtracting clean prediction's width penalty from direct/EDM's assumes comparable non-rank effects of reducing width, despite different objectives and optimization behavior. What existing training curves or convergence evidence support that assumption, particularly given the divergent runs and single surviving constrained-direct seed? The answer would materially affect how strongly I credit the quantitative generation claims.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 2sw6

Thank you for the thorough reading and constructive suggestions.

1. Full-distribution excess

For unrestricted upstream computation and an optimally chosen shared rank-rr affine output space, the optimal excess is the tail of the time-averaged conditional-mean covariance divided by nn. It recovers the Gaussian value 0.401881(1−r/n)0.401881(1-r/n) for t∼U(0,0.95)t \sim U(0, 0.95).

For direct prediction at r=n/2r=n/2, the intervals at dimensions 384, 768 and 1024 are [0.1936, 0.7117], [0.1930, 0.7173] and [0.1933, 0.7123]; measured width gaps are 0.5233, 0.4840 and 0.4602. These are descriptive comparisons to optimal-class references, not guarantees for trained-model gaps. We will make that distinction explicit and retain Equation (6) as an empirical approximation. The spectral analysis does not guarantee successful optimization or sufficient backbone capacity.

2. How well is the FID decomposition supported?

The original subtraction does not establish a causal rank contribution: it assumes other width effects cancel across parameterizations. Re-auditing also found that the sole finite width-256 direct run was degenerate, with velocity MSE 47.12; for these bounded pixels, the zero predictor has risk at most 2. We will therefore remove the additive attribution and replace it with direct interventions.

With each width-768 backbone frozen, rank-256 and full-rank refits give:

PredictionFull-rank FIDRank-256 FID
Direct175.0 ± 10.5257.1 ± 20.6
EDM163.2 ± 2.4245.6 ± 3.4
Clean98.4 ± 2.795.6 ± 2.7

These are three-seed means ± SD, with both heads refitted. Separate fine-tuning gives increases of 81.6, 85.0 and 7.0, respectively, with one seed so far. The intervention does not assume equal convergence across independently trained widths; its FID magnitude applies to the evaluated checkpoints.

We will provide individual curves, retain finite poor outcomes, and report the two nonfinite direct and one nonfinite EDM runs explicitly. Oracle samples remain diagnostics, not FID bounds. The new specified-SDE check gives complementary variance 0.6384 untrained and 0.6390 trained, versus 0.6381 predicted, including both surviving initial and injected noise.

AuthorsOpenReview ↗

Follow-up to Reviewer 2sw6: Remaining results

1. Is the 1−r/n1-r/n comparison a theoretical prediction or an empirical approximation? Q1

For full distributions it is an empirical approximation. Proposition 1 bounds total regression error; Section 5.2 measures an excess over a less restricted model. Connecting them needs the following, which the revision adds as Proposition 4:

  • Assumptions: both models see (xt,t)(x_t, t), upstream computation is unrestricted, the rank-rr affine head is shared across tt, and errors are averaged over t∼U(0,0.95)t \sim U(0, 0.95).
  • Result: the excess of the best rank-rr head over an unrestricted model equals 1n\frac{1}{n} times the sum of the n−rn-r smallest eigenvalues of Σm=Et Cov(E[v∣xt,t]∣t)\Sigma_m = E_t\,\mathrm{Cov}(E[v \mid x_t,t] \mid t). Irreducible conditional noise, the Bayes error, adds to both risks and cancels from the difference.
  • Why: since E[v∣t]E[v \mid t] does not depend on tt, the best shared head projects the conditional mean onto its top-rr principal subspace (Eckart-Young). Comparing the conditional mean with the best linear predictor, and with the target itself, bounds the excess below by its value for Gaussian data with the same spectrum and above by the total-error floor of Proposition 1.
  • When it equals 1−r/n1-r/n: only if the Bayes error is zero, so that E[v∣xt,t]=vE[v \mid x_t,t] = v, and the clean data have rank at most rr. Single-image fits meet both conditions, which is why they sit at 1−r/n1-r/n (0.5016 against 0.500 at n=768n=768, r=384r=384). Full distributions have Bayes error, so the excess lies strictly inside an interval computed from the data spectrum.

The measurements at r=n/2r=n/2:

nnArmOptimal-class intervalWidth gapFixed-width fine-tuning gapRefit gap
384Direct[0.1936, 0.7117]0.52330.42680.4530
768Direct[0.1930, 0.7173]0.48400.39440.4208
1024Direct[0.1933, 0.7123]0.4602Not run0.3827
384Clean[0.0814, 0.2114]0.17660.15670.1662
768Clean[0.0835, 0.2164]0.17350.15440.1664
1024Clean[0.0811, 0.2111]0.1624Not run0.1488

Width gap changes backbone width; fine-tuning changes only the head rank on one backbone; refit replaces the head on frozen features. Direct uses velocity MSE and clean uses clean-space MSE. The intervals describe the best head, and trained gaps also contain learning error, so these values are consistent with the theory rather than tests of it. Equation (6) stays an empirical approximation.

2. How well supported is the rank-attributable FID decomposition? Q2

Not well enough and we replace it with controls that need no subtraction across arms. Both heads are fine-tuned from the same trained width-768 backbone, for each of three seeds (Euler FID, mean ± SD):

PredictionFull headRank-256 headDifference of means
Direct171.7 ± 11.9246.0 ± 12.5+74.3
EDM161.3 ± 4.5245.5 ± 1.0+84.2
Clean89.4 ± 2.092.0 ± 2.0+2.6

These replace the one-seed values (+81.6, +85.0, +7.0) in our previous response, which compared heads started from different seeds' backbones. The frozen-backbone refits reported earlier agree (+82.1, +82.4, −2.8).

Seeds. The constrained direct cell now has six new runs, all finite and non-degenerate (velocity error 0.77 to 0.78), at 235.2 ± 4.0; EDM has six at 239.7 ± 3.1. Including the original runs: direct 252.4 ± 45.5 over 7 finite runs with 2 non-finite, and EDM 264.4 ± 47.9 over 8 with 1 non-finite. All failures are among the original seeds, and we do not claim a cause.

Training curves. At width 256, FID from 25k to 150k steps goes from 149.5 to 108.9 for clean, but from 224.0 to 235.2 for direct and 226.7 to 239.7 for EDM: they stop improving early. In the regression curves for direct at n=1024n=1024, both narrow and wide models still improve, but their gap grows (0.422, 0.444, 0.454, 0.460 at 10k to 40k steps). Longer training widens the gap rather than closing it.

AuthorsOpenReview ↗

Follow-up to Reviewer 2sw6

Dear Reviewer 2sw6,

Thank you again for your helpful review. As the discussion period closes soon, we would like to confirm that the revised PDF covers the two points you raised.

  • Full-distribution error. Proposition 4 now bounds the velocity excess on both sides. At d=n/2d=n/2 the interval is about [0.19,0.71][0.19, 0.71], and both the width gaps and the fixed-backbone gaps fall inside it (Table 11).
  • Generation attribution. The earlier additive estimate is replaced by interventions that change only the head on a fixed backbone (Table 3). The fine-tuning result now has three seeds: +74.3 for direct, +84.2 for EDM and +2.6 for clean. Every run is listed in Table 20.

We would be glad to know whether this resolves your questions, and we are happy to clarify anything further.

Best regards,

The authors

We sincerely thank the reviewer for their helpful feedback. As the discussion period is coming to an end, we would really appreciate it if the reviewer could reconsider their rating based on our responses. Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

Dear Reviewers and AC,

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. We have uploaded a revised PDF. Updates are temporarily highlighted in blue for your convenience; rewording alone is not marked.

Diffusion transformers on high-dimensional tokens seem to need a width close to the token dimension. We show that the requirement comes from the linear output layer, not the backbone. A head of rank d<nd<n cannot emit the target in n−dn-d directions. Under velocity prediction this includes Gaussian noise, which then reaches every sample unchanged; one skip coefficient in the parameterization decides whether that noise survives sampling, grows or is removed. Clean prediction removes it, so a clean model needs only the effective rank of the data, not the token dimension. On full distributions the remaining penalty is bounded on both sides by quantities measured from the data spectrum before training.

Our claims.

The paper adds what the output-rank floor alone does not say: an exact identity for the noise a rank-limited head passes to every sample (Proposition 2), a criterion that decides this for any linear input skip (Theorem 1), two-sided bounds on the rank-induced excess on full distributions (Propositions 3 and 4), and experiments that change only the head rank.

New evidence

  • Only the head rank changes. With the backbone fixed, single-image fits reach 0.7589, 0.5031 and 0.2518 against floors of 0.75, 0.50 and 0.25 (Table 5). On trained width-768 generators, a rank-256 head raises FID by +82.1 (direct), +82.4 (EDM) and −2.8-2.8 (clean) after a refit, and by +74.3, +84.2 and +2.6 after fine-tuning (Table 3).
  • The loss does not explain it. Refitting under the other arm's loss leaves the rank cost unchanged (+81.1, +77.0, −2.1-2.1). Swapping the loss in whole models moves FID by 2 to 8, while changing the target moves it by 72 to 128 (Table 24).
  • The sampling law holds. Predicted and measured out-of-span energy agree for five parameterizations under Euler, Heun and a stochastic sampler, e.g. 608.9-fold growth for noise prediction (Tables 1 and 16). EDM lowers the floor by 23% but pays the same rank penalty as direct.
  • Full distributions. Proposition 4 gives the interval [0.19,0.71][0.19, 0.71] at d=n/2d=n/2; the width gaps and the fixed-backbone gaps all fall inside it (Table 11, Figure 5).
  • Robustness. A second encoder family, MAE-B/16 (Table 7). Fifteen new seeds for the constrained arms all trained normally; Table 20 lists every run, including the five original failures.

The revision includes

  • Structure. The paper is built around three questions, stated and answered in Section 1 and each tested in one part of Section 5.
  • Related work. Section 7 and Appendix I are expanded, covering RAE's Theorem 2, AsymFlow, RiT and the kk-Diff targets.
  • Reporting details. The entropy-rank formula, the training and evaluation ranges of tt, the sampler's final step and the FID implementation are stated. Learning curves and the gap by noise level are new (Figure 5).

We thank you again for the reviews, which made the claims clearer and better tested.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.