acceptodds
Under review as a conference paper at ICLR 2027

Mean Mode Screaming: Mean–Variance Split Residuals for 1000-Layer Diffusion Transformers

Abstract

Scaling Diffusion Transformers (DiTs) to hundreds of layers introduces a structural vulnerability: networks can enter a silent, mean-dominated collapse state that homogenizes token representations and suppresses centered variation. Through mechanistic auditing, we isolate the trigger event of this collapse as Mean Mode Screaming (MMS). MMS can occur even when training appears stable, with a mean-coherent backward shock on residual writers that opens deep residual branches and drives the network into a mean-dominated state. We show this behavior is driven by an exact decomposition of these gradients into mean-coherent and centered components, compounded by the structural suppression of attention-logit gradients through the null space of the Softmax Jacobian once values homogenize. To address this, we propose Mean–Variance Split (MV-Split) Residuals, which combine a separately gained centered residual update with a leaky trunk-mean replacement. On a 400-layer single-stream DiT, MV-Split prevents the divergent collapse that crashes the un-stabilized baseline; it tracks close to the baseline's pre-crash trajectory while remaining substantially better than token-isotropic gating methods such as LayerScale across the full schedule. Finally, we present a 1000-layer DiT as a scale-validation run at boundary scales, establishing that the architecture remains stably trainable at extreme depth.

Primary area:
generative models
Keywords:
Diffusion Transformers, training stability, deep residual networks, representation collapse, residual scaling, extreme depth, gradient dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UinTOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates mean-dominated representation collapse in very deep Diffusion Transformers (DiTs), using gradient decomposition at residual output projections and training diagnostics to examine the relationship between mean-coherent gradient spikes (MMS) and Q/K gradient suppression. To address this failure mode, the authors propose MV-Split Residuals, which apply separate gains to the mean and centered residual components and introduce a leaky update to the trunk mean to limit its accumulation. In controlled 400-layer experiments on ImageNet, the method prevents the collapse observed in the unstabilized baseline and reaches comparable generation quality faster than LayerScale, while a separate 1000-layer experiment demonstrates its trainability at extreme depth.

Critical strengths
  1. The paper provides a thorough analysis of the MMS event, supported by clear and informative visualizations. These make the progression of the failure and its connection to the proposed mechanism easy to follow.
  2. The experimental design is well aligned with the paper’s claims and demonstrates clear benefits of MV-Split. In particular, the controlled 400-layer experiments show that the method avoids the collapse observed in the unstabilized baselines while achieving faster convergence than LayerScale.
Critical weaknesses
  1. The paper provides substantial evidence linking mean-coherent gradient spikes to representation collapse, but it remains unclear whether the spikes initiate the collapse or just emerge as part of a coupled process. The gradient decomposition and training trajectories are consistent with the proposed mechanism, yet a targeted intervention on the mean-gradient component is missing. The evidence is less conclusive about what initiates them.
  2. MV-Split combines separate gains for the mean and centered components with attenuation of the trunk mean, but the experiments do not establish their individual contributions. The attention-only control in the appendix helps identify which residual branches require protection, but does not determine whether both design choices are necessary. It remains unclear which modification primarily drives the gains and whether the full design is needed.
  3. In the main quality comparison, LayerScale uses standard initialization, whereas MV-Split uses zero-initialized residual output projections, so both residual design and initialization differ. The LayerScale + Zero-Init control in Figure 6 is valuable, but only gradient diagnostics are reported, without corresponding generation-quality and long-term stability results.
Questions for authors
  1. The observed association between gradient spikes and collapse does not establish which occurs first in the causal chain. It therefore remains unclear whether the spikes initiate collapse or develop alongside representation deterioration.
  2. MV-Split changes both gains and the propagation of the trunk mean, so the full model’s results do not isolate their individual effects. This leaves uncertainty about which component drives the gains and whether both are necessary.
  3. The main quality comparison differs in both residual design and initialization, while the matched-initialization control reports only gradient diagnostics. Consequently, the extent to which MV-Split’s convergence advantage comes from the residual design itself remains unclear.
1 comment
AuthorsOpenReview ↗

Response to Reviewer UinT: causal ordering, component contributions, initialization

Thank you for the detailed review.

W1/Q1 (do spikes initiate collapse?). We agree this is a coupled feedback process, and that our evidence does not show the spike is its first cause. What Figure 3 establishes is the order inside the fast event: the gradient-norm spike is concentrated in the mean-coherent component, and the drop in Q/K gradients and the rise of deep-layer token similarity follow it. What it does not establish is what precedes the spike. A slower build-up of token alignment before ignition is plausible: Eq. 5 makes the coherent term grow with alignment. We have not measured it in the collapsing runs. We will revise the text to describe MMS as the ignition event of a feedback loop, not as its root cause, and state the unmeasured precursor as a limitation.

Two controls run after submission bear on this. In a matched 400-layer Pre-LN run, the token-mean energy fraction drifts up to ~0.56 in one layer with no spike and no collapse over 50k steps, so a rising mean fraction does not by itself produce collapse. At 128 layers, adding explicit timestep conditioning delays onset ×2.8 (step 526 → 1,481) and leaves the signature and the collapsed plateau unchanged (loss 1.96 vs 1.94). A direct intervention on the mean-gradient component is missing; we agree it is the right experiment.

W2/Q2 (individual contributions). The two effects come from one parameter: the merged token mean is (1−α) ⊙ JX + α ⊙ JF, a convex interpolation, so the leak and the mean-write gain are not independent add-ons. Without the leak, the carried mean passes every layer with unit gain, and Prop. 1 shows attention preserves it exactly. We do not have the three-arm experiment that would isolate each effect empirically and will state this as a limitation.

W3/Q3 (initialization). The reviewer is right: in Table 1 the LayerScale and MV-Split rows differ in both the residual merge and the writer initialization, and for the matched-initialization control (Fig. 6) we report gradient diagnostics only. We therefore cannot attribute the full quality gap to the residual design, and we will reword the claim to "better than LayerScale in its standard configuration." We used standard initialization for LayerScale because that is its original setup. We also expect zero-initialized writers to slow it further: the branch output is the product of a small gate and a zero writer, so the gate receives no gradient at initialization and the writer's gradient is scaled down by the gate. This is an argument, not a measurement. Our stability claims do not depend on this comparison.

Reviewer_AKLCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to train an ultra-deep diffusion transformer (up to 1000 layers). The authors identify a training failure case, Mean Mode Screaming, where the representations of all tokens become the same. To solve the problem, the paper proposes MV-Split Residuals. It splits the residual update into a mean component and a centered component. The centered part is added to the trunk as a normal update, while the mean part uses a leaky update. MV-Split prevents the collapse and converges faster than LayerScale in 400-layer DiT experiments. The paper also shows that the method remains stable in 1000-layer DiT training.

Critical strengths
  • The paper writing is coherent and easy to read.
  • The paper provides detailed analysis of why scaling DiT training to up to 1000 layers would collapse. The Figure 3 clearly shows the failure process.
  • The proposed MV-Split is simple and directly motivated by the analysis. It makes 400-layer and 1000-layer DiT training stable.
Critical weaknesses
  • The paper focuses on post-norm DiT models. However, the first version of DiT (https://arxiv.org/abs/2212.09748) and most DiT research since then use pre-norm. Also, there has been research showing that pre-norm has better stability than post-norm (https://arxiv.org/abs/2002.04745). The authors should explore if the phenomenon still exists with pre-norm transformer architecture.
  • The model scaling is unusual. The transformer scales up to 1000 layers while the hidden size is still kept at 1024. The ideal width–depth ratio can be referred to language transformers (https://arxiv.org/pdf/2001.08361). There is no comparison with a wider and shallower model of the same size to check whether the instability issue still happens.
  • The authors do experiments on ImageNet 256 resolution with batch size 1024 and 100k training steps, which is equivalent to ~26B image tokens trained. However, for 5.45B (400-layer) and 13.64B (1000-layer) models, according to https://arxiv.org/pdf/2608.17286, ideally the models should be trained with ~1.09T and ~2.73T tokens to fully utilize the model capacity. Additionally, the paper does not use EMA for any of the experiments, which is used in ImageNet conventional setup.
Questions for authors
  • Please refer to the weakness section above.
1 comment
AuthorsOpenReview ↗

Response to Reviewer AKLC: Pre-LN control, width–depth ratio, training budget

Thank you for the careful review.

W1 (pre-norm). We ran this control: a matched 400-layer Pre-LN model with identical data, batch, optimizer and learning rate (7.8e-5); only the normalization placement changes. Pre-LN shows no MMS event over 50k steps, 23.7k steps beyond the point where the matched Post-LN baseline collapses (step 26,423). So we agree: Pre-LN is stable at this depth.

What we add is a measurement of how that stability is obtained. For the deepest quartile of layers, the layer-to-output response stays O(1) before the final norm (1.03–1.05 at every checkpoint) but is divided by the growing stream RMS at the output:

stepinit5k10k20k30k50k
output attenuation×1.0×4.5×6.7×9.3×10.8×12.7

Matched-step quality (FID / IS, same learning rate):

1k5k10k15k20k
Pre-LN71.9 / 16.026.2 / 38.112.7 / 66.37.00 / 95.65.33 / 112.5
Post-LN vanilla (collapses at 26,423)68.1 / 17.413.4 / 64.65.92 / 108.63.55 / 138.83.22 / 152.2

For reference, MV-Split reaches 3.64 / 139.9 at 20k at twice the learning rate, a rate at which the vanilla baseline collapses at step 3,400.

Two caveats: this is a single-variable control, so Pre-LN's learning rate was not tuned separately; and the link between attenuation and the quality gap is correlational. Our claim is limited to (i) the mechanism of the post-norm failure and (ii) that MV-Split makes this regime trainable without uniform attenuation. We will add these results to the appendix.

W2 (width–depth ratio). We agree that 1000×1024 is not a recommended allocation, and we do not claim deep-narrow is the best use of parameters. Depth is the stressor we study; the 1000-layer run is a trainability check only. Two points suggest the phenomenon is not an artifact of an extreme ratio. The full MMS signature reproduces at 128 layers, and the ignition learning rate falls with depth (6.25e-4 at 128L vs 1.56e-4 at 400L). Also, the coherent term in Eq. 5 scales with token count, not width. We have no matched-parameter wide-shallow comparison and will list this as a limitation.

W3 (training budget, EMA). We agree that our budget is far below the regime the reviewer cites, and we make no claim about capacity utilization. Within this budget the models reach good sample quality: 400-layer MV-Split reaches FID 2.60 at 50k steps, and the 1000-layer model stays between 2.64 and 2.77 from 30k onward (CFG 2.0, 25-step Euler, non-EMA). Our claims concern stability and matched-step comparison, for which the budget is sufficient: the collapse occurs at step 3,400 (full LR) or 26,423 (half LR), and all arms share the same budget. EMA is an averaged copy of the weights used for evaluation; it does not enter the gradient path and cannot prevent the event. All reported FIDs are non-EMA, which we will state clearly.

Reviewer_P4uBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates a mean-dominated failure mode in deep Diffusion Transformers, termed Mean Mode Screaming. The proposed explanation connects residual-writer gradients, token homogenization, and suppression of attention-logit gradients. To address this failure, the authors introduce MV-Split residuals, which control mean and centered branch updates while attenuating the carried trunk mean. Experiments show the effectiveness.

Critical strengths
  1. This paper studies a practical obstacle to extreme-depth training and provides diagnostics beyond aggregate loss curves in diffusion models.

  2. The method is simple. The gradient decomposition and local backward routing are interpretable and standard-initialization control and attention-only intervention provide useful additional evidence.

Critical weaknesses
  1. MV-Split simultaneously introduces separate gains for mean and centered branch components and attenuation of the carried trunk mean. The experiments do not isolate their individual contributions. The negative controls do not replace direct comparisons between branch-gain separation alone, trunk-mean attenuation alone, and the complete method.

  2. The mean attenuation is applied before RMSNorm, which can cancel uniform attenuation and recouple mean and centered components. Authors need to give more details.

Questions for authors
  1. How do branch-gain separation alone and trunk-mean attenuation alone perform? Is their combination necessary?

  2. From the same pre-collapse state, does selective mean-gradient suppression outperform update-magnitude-matched uniform shrinkage?

1 comment
AuthorsOpenReview ↗

Response to Reviewer P4uB: RMSNorm and mean attenuation, component contributions

Thank you for the review.

W2 (attenuation before RMSNorm). RMSNorm cancels only a rescaling that is uniform over the whole token vector: Norm(cz) = Norm(z). The MV-Split merge is not of that form. Writing token i as m + c_i (token mean plus centered part), the merge gives

z_i = [(1−α) ⊙ m + α ⊙ f̄] + [c_i + β ⊙ (PF)_i],

so only the mean part is contracted, and α is a per-channel vector, so the contraction is not even uniform across channels. RMSNorm then rescales each token by one scalar, which leaves the mean-to-centered ratio of that token unchanged. The attenuation therefore survives the norm as a change in ratio: with f̄ = 0 and scalar α, Norm((1−α)m + c_i) = Norm(m + c_i/(1−α)). The reviewer is right on one point, which we will state in the paper: the norm restores the overall magnitude, so MV-Split controls the mean-to-centered balance, not the absolute size of the mean; and because the per-token scale depends on both parts, the split is not exactly preserved through the norm.

W1/Q1 (individual contributions). The two effects are produced by a single parameter. The token mean after the merge is (1−α) ⊙ JX + α ⊙ JF: a convex interpolation between the carried mean and the branch mean for α in [0,1], which holds in all our runs. Removing the leak gives JX + α ⊙ JF, where the carried mean passes through every layer with unit gain and branch means add up over depth; row-stochastic attention preserves this subspace exactly (Prop. 1), so nothing else in the block decays it. Removing the separate gain and keeping only the leak leaves the centered and mean writes tied to one scale, which is the LayerScale situation. The extreme of mean removal, hard centering, degraded optimization in our early trials (reported qualitatively in the appendix). We agree these arguments do not replace a three-arm experiment (gain separation only, leak only, full). We do not have it, and we will state this as a limitation.

Q2 (selective vs magnitude-matched uniform shrinkage). We have not run this from a shared pre-collapse checkpoint. The closest existing evidence is indirect: LayerScale, a uniform shrinkage, bounds the mean-coherent gradient but shrinks the centered gradient as well, and trails MV-Split at every checkpoint (FID 6.50 vs 3.64 at 20k; 2.90 vs 2.60 at 50k); global gradient clipping, used in all runs, limits update magnitude and does not remove the event. These comparisons differ in more than one factor, and we will say so.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.