- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates mean-dominated representation collapse in very deep Diffusion Transformers (DiTs), using gradient decomposition at residual output projections and training diagnostics to examine the relationship between mean-coherent gradient spikes (MMS) and Q/K gradient suppression. To address this failure mode, the authors propose MV-Split Residuals, which apply separate gains to the mean and centered residual components and introduce a leaky update to the trunk mean to limit its accumulation. In controlled 400-layer experiments on ImageNet, the method prevents the collapse observed in the unstabilized baseline and reaches comparable generation quality faster than LayerScale, while a separate 1000-layer experiment demonstrates its trainability at extreme depth.
Critical strengths
- The paper provides a thorough analysis of the MMS event, supported by clear and informative visualizations. These make the progression of the failure and its connection to the proposed mechanism easy to follow.
- The experimental design is well aligned with the paper’s claims and demonstrates clear benefits of MV-Split. In particular, the controlled 400-layer experiments show that the method avoids the collapse observed in the unstabilized baselines while achieving faster convergence than LayerScale.
Critical weaknesses
- The paper provides substantial evidence linking mean-coherent gradient spikes to representation collapse, but it remains unclear whether the spikes initiate the collapse or just emerge as part of a coupled process. The gradient decomposition and training trajectories are consistent with the proposed mechanism, yet a targeted intervention on the mean-gradient component is missing. The evidence is less conclusive about what initiates them.
- MV-Split combines separate gains for the mean and centered components with attenuation of the trunk mean, but the experiments do not establish their individual contributions. The attention-only control in the appendix helps identify which residual branches require protection, but does not determine whether both design choices are necessary. It remains unclear which modification primarily drives the gains and whether the full design is needed.
- In the main quality comparison, LayerScale uses standard initialization, whereas MV-Split uses zero-initialized residual output projections, so both residual design and initialization differ. The LayerScale + Zero-Init control in Figure 6 is valuable, but only gradient diagnostics are reported, without corresponding generation-quality and long-term stability results.
Questions for authors
- The observed association between gradient spikes and collapse does not establish which occurs first in the causal chain. It therefore remains unclear whether the spikes initiate collapse or develop alongside representation deterioration.
- MV-Split changes both gains and the propagation of the trunk mean, so the full model’s results do not isolate their individual effects. This leaves uncertainty about which component drives the gains and whether both are necessary.
- The main quality comparison differs in both residual design and initialization, while the matched-initialization control reports only gradient diagnostics. Consequently, the extent to which MV-Split’s convergence advantage comes from the residual design itself remains unclear.
Response to Reviewer UinT: causal ordering, component contributions, initialization
Thank you for the detailed review.
W1/Q1 (do spikes initiate collapse?). We agree this is a coupled feedback process, and that our evidence does not show the spike is its first cause. What Figure 3 establishes is the order inside the fast event: the gradient-norm spike is concentrated in the mean-coherent component, and the drop in Q/K gradients and the rise of deep-layer token similarity follow it. What it does not establish is what precedes the spike. A slower build-up of token alignment before ignition is plausible: Eq. 5 makes the coherent term grow with alignment. We have not measured it in the collapsing runs. We will revise the text to describe MMS as the ignition event of a feedback loop, not as its root cause, and state the unmeasured precursor as a limitation.
Two controls run after submission bear on this. In a matched 400-layer Pre-LN run, the token-mean energy fraction drifts up to ~0.56 in one layer with no spike and no collapse over 50k steps, so a rising mean fraction does not by itself produce collapse. At 128 layers, adding explicit timestep conditioning delays onset ×2.8 (step 526 → 1,481) and leaves the signature and the collapsed plateau unchanged (loss 1.96 vs 1.94). A direct intervention on the mean-gradient component is missing; we agree it is the right experiment.
W2/Q2 (individual contributions). The two effects come from one parameter: the merged token mean is (1−α) ⊙ JX + α ⊙ JF, a convex interpolation, so the leak and the mean-write gain are not independent add-ons. Without the leak, the carried mean passes every layer with unit gain, and Prop. 1 shows attention preserves it exactly. We do not have the three-arm experiment that would isolate each effect empirically and will state this as a limitation.
W3/Q3 (initialization). The reviewer is right: in Table 1 the LayerScale and MV-Split rows differ in both the residual merge and the writer initialization, and for the matched-initialization control (Fig. 6) we report gradient diagnostics only. We therefore cannot attribute the full quality gap to the residual design, and we will reword the claim to "better than LayerScale in its standard configuration." We used standard initialization for LayerScale because that is its original setup. We also expect zero-initialized writers to slow it further: the branch output is the product of a small gate and a zero writer, so the gate receives no gradient at initialization and the writer's gradient is scaled down by the gate. This is an argument, not a measurement. Our stability claims do not depend on this comparison.