- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the relationships among high-norm tokens, massive activations, and attention sinks in Diffusion Transformers (DiTs). The authors identify a shared representation direction dominated by a massive activation channel. Through intervention experiments, they argue that direction determines attention sink identity, while large activation norms help maintain this direction across layers.
Critical strengths
The paper attempts to provide a unified perspective on several previously observed phenomena in DiTs. The controlled intervention experiments help distinguish the effects of representation direction and magnitude on attention behavior.
Critical weaknesses
-
Generalizability of the findings. The experiments focus primarily on FLUX.1, with additional results on PixArt-Σ. Both FLUX.1-Dev and FLUX.1-Schnell belong to the same model family, making the architectural diversity rather limited. The paper does not investigate other representative DiTs, such as SD3, SD3.5, or FLUX.2. As a result, several conclusions, particularly those concerning specific massive activation channels, shared directions, and register dynamics, appear highly model-specific. It remains unclear whether these observations reflect general properties of DiTs or particular characteristics of the selected architectures.
-
Distinction from existing studies on massive activations and attention sinks. several central findings closely overlap with prior work, while the paper does not clearly explain its additional contributions. The spatial correspondence between high-norm tokens and attention sinks is closely related to the previously observed concentration of attention on massive activations [1]. The shared representation direction is also closely related to previous findings that massive activations concentrate in a few fixed dominant dimensions, leading to high directional similarity among feature vectors [2, 3]. The observation that these activation structures affect local image details has also been investigated in [3]. Although the present paper provides additional analyses and interventions, the distinction between these findings and existing results is not sufficiently articulated. A clearer discussion of what is genuinely new would help establish the paper's contribution.
-
Broader significance. The paper presents several observations about shared directions, attention sinks, activation magnitudes, and layer-wise dynamics, but these findings remain somewhat fragmented. The work does not sufficiently establish a broader understanding of their functional roles or demonstrate a concrete application based on the identified properties. A practical application can make the overall contribution more compelling.
References
[1] Sun et al., Massive Activations in Large Language Models, COLM 2024.
[2] Gan et al., Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations, NeurIPS 2025.
[3] Gan et al., Massive Activations are the Key to Local Detail Synthesis in Diffusion Transformers, ICLR 2026.
Questions for authors
see weakness
Response to Reviewer 1Pe7
W1 (Generalizability). We thank the reviewer and agree that the scope of our claims should be much more prominent.
SD3 and SD3.5 fall outside this scope because their token-level sparse structures are reported almost exclusively in the text stream. Our analysis covers the image stream only. This is stated in the Introduction (lines 079-080) and as a limitation in Section 7 (lines 515-520), which says that the analysis “is limited to the image stream of the DiTs considered in this work” and that generalization “across other DiT architectures remains open”. Wu & Summa (2026) locate over 99.9% of the attention sinks in SD3’s joint attention on text tokens. Starodubcev et al. (2026, arXiv:2605.16147, App. C.2) measure token norms in SD3.5-Large and place the outliers in the text sequence, with image-token norms “comparatively uniform”. By these reports, the object we trace, a high-norm image token that acts as an attention sink, is not reported in these models, so they can neither confirm nor refute a mechanism defined on image-stream registers. We have not tested FLUX.2 and make no claim about it.
W2 (Distinction from prior work). First, the correspondence between massive activations and attention sinks is related to Sun et al. (2024), and Gan et al. (2025, 2026) show that massive activations occur in a few fixed dimensions across image tokens. We add that the dominant channel contains both a large activation shared by ordinary image tokens and sparse, much larger peaks at the high-norm register positions, which coincide with the attention sinks in FLUX.1 and PixArt-Σ (Sections 3.2 to 3.4). By contrast, channel 1446 in FLUX.1-Dev ranks second by mean activation but has no comparable sparse peaks (Section 3.4.2).
Second, a shared direction dominated by one massive channel is partly expected, but the channel magnitude does not explain sink identity. On 50 generations per model (Appendix D.1, Table 7), doubling the dominant channel alone reduces attention to 0.75 in FLUX.1-Dev and 0.07 in PixArt-Σ. Setting the channel to zero once leaves 0.90 in FLUX.1-Dev and makes the PixArt-Σ register a 31 times stronger sink. Writing the channel into ordinary tokens at the registers' magnitude creates no sink. Removing only the part of v⋆ outside the dominant channel, about 1% and 6% of its squared length, leaves 0.34 and 0.04 of the attention. In FLUX.1-Dev, removing the full v⋆ component at fixed norm leaves 0.05; in PixArt-Σ, the same edit also removes most of channel 293 and instead strengthens the sink. Suppressing the sink by editing its keys leaves the norm and dominant channel essentially unchanged (Appendix D.2). The part of v⋆ outside the dominant channel is also reproducible across the two FLUX.1 checkpoints (cosine 0.988 after channel 154 is excluded; Appendix B.2).
Finally, Gan et al. (2026) connect massive activations to local detail synthesis, and we do not present that connection as new. We intervene on the sparse register tokens rather than disrupting the massive dimensions themselves, and vary the layer at which the state is present. In FLUX.1-Dev, inducing the state outside its natural layer range changes the image by LPIPS 0.07 to 0.09 while CLIP-I remains above 0.98. In PixArt-Σ, introducing it late, maintaining it beyond its endpoint, or removing it gives LPIPS 0.70 to 0.94 (Section 6, Appendix C.5, Table 5).
W3 (Broader significance). We agree the original presentation was fragmented. The revision follows one register state through its lifecycle, in which it is written, read by attention, preserved by its large norm, and erased (Sections 3 to 5). In FLUX.1-Dev, the additional angular drift at quarter norm is 3.08 times that at half norm, against a predicted 3 (Section 4.2, Appendix D.3). A large norm therefore protects the state, and shrinking a register destabilizes it.
New experiments identify the units that write and erase this state (Section 5, Appendix D.7). 205 of the 12,288 neurons of the layer-18 MLP (FLUX.1-Dev) and 97 of the 4,608 of the layer-16 MLP (PixArt-Σ) account for 90% of each layer's write. Silencing the writer neurons leaves 0.13 and 0.063 of the register state. MLPs erase the state in FLUX.1-Dev, while in PixArt-Σ three self-attention heads of layer 21 account for 64%, 18% and 12% of that layer's removal. Silencing the eraser units leaves 5.1 and 14 times as much state after the erasing layers, and it persists to the final layer. The state can thus be removed or kept through a few units.
Practically, Table 7 shows that modifying the dominant channel does not necessarily suppress sinks; methods should target the part of the register direction outside it. Moving or removing the state changes the image, far more in PixArt-Σ (Section 6), a possible editing target. We do not yet demonstrate an application; Section 7 names activation-aware quantization and compression as future directions.