acceptodds
Under review as a conference paper at ICLR 2027

It’s All About One Direction

Abstract

Diffusion Transformers exhibit several sparse structures, including high-norm tokens, massive activation channels, and attention sinks, but their relationship remains unclear. Across FLUX.1-Dev, FLUX.1-Schnell, and PixArt-, we find that high-norm register tokens collapse onto a direction shared across prompts and seeds. This direction is dominated by a single massive activation channel that accounts for most of the register tokens' high norm. Controlled interventions show that direction, rather than magnitude, determines sink identity, while large norms stabilize the register direction across subsequent layers. Query–key geometry explains this directional preference. The -aligned state primarily affects image detail, and its effect depends on the Transformer layer at which it is present. Finally, we trace a coordinated lifecycle in which projection onto grows as high-norm registers and attention sinks emerge, persists through intermediate layers, and is later attenuated as these sparse structures disappear. Together, these results link massive activation channels, high-norm tokens, attention sinks, and image refinement in Diffusion Transformers through a shared internal direction.

Primary area:
interpretability and explainable AI
Keywords:
Diffusion Transformers, mechanistic interpretability, massive activations, attention sinks, high-norm tokens, attention routing, internal representations

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1Pe7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the relationships among high-norm tokens, massive activations, and attention sinks in Diffusion Transformers (DiTs). The authors identify a shared representation direction dominated by a massive activation channel. Through intervention experiments, they argue that direction determines attention sink identity, while large activation norms help maintain this direction across layers.

Critical strengths

The paper attempts to provide a unified perspective on several previously observed phenomena in DiTs. The controlled intervention experiments help distinguish the effects of representation direction and magnitude on attention behavior.

Critical weaknesses
  1. Generalizability of the findings. The experiments focus primarily on FLUX.1, with additional results on PixArt-Σ. Both FLUX.1-Dev and FLUX.1-Schnell belong to the same model family, making the architectural diversity rather limited. The paper does not investigate other representative DiTs, such as SD3, SD3.5, or FLUX.2. As a result, several conclusions, particularly those concerning specific massive activation channels, shared directions, and register dynamics, appear highly model-specific. It remains unclear whether these observations reflect general properties of DiTs or particular characteristics of the selected architectures.

  2. Distinction from existing studies on massive activations and attention sinks. several central findings closely overlap with prior work, while the paper does not clearly explain its additional contributions. The spatial correspondence between high-norm tokens and attention sinks is closely related to the previously observed concentration of attention on massive activations [1]. The shared representation direction is also closely related to previous findings that massive activations concentrate in a few fixed dominant dimensions, leading to high directional similarity among feature vectors [2, 3]. The observation that these activation structures affect local image details has also been investigated in [3]. Although the present paper provides additional analyses and interventions, the distinction between these findings and existing results is not sufficiently articulated. A clearer discussion of what is genuinely new would help establish the paper's contribution.

  3. Broader significance. The paper presents several observations about shared directions, attention sinks, activation magnitudes, and layer-wise dynamics, but these findings remain somewhat fragmented. The work does not sufficiently establish a broader understanding of their functional roles or demonstrate a concrete application based on the identified properties. A practical application can make the overall contribution more compelling.

References

[1] Sun et al., Massive Activations in Large Language Models, COLM 2024.

[2] Gan et al., Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations, NeurIPS 2025.

[3] Gan et al., Massive Activations are the Key to Local Detail Synthesis in Diffusion Transformers, ICLR 2026.

Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer 1Pe7

W1 (Generalizability). We thank the reviewer and agree that the scope of our claims should be much more prominent.

SD3 and SD3.5 fall outside this scope because their token-level sparse structures are reported almost exclusively in the text stream. Our analysis covers the image stream only. This is stated in the Introduction (lines 079-080) and as a limitation in Section 7 (lines 515-520), which says that the analysis “is limited to the image stream of the DiTs considered in this work” and that generalization “across other DiT architectures remains open”. Wu & Summa (2026) locate over 99.9% of the attention sinks in SD3’s joint attention on text tokens. Starodubcev et al. (2026, arXiv:2605.16147, App. C.2) measure token norms in SD3.5-Large and place the outliers in the text sequence, with image-token norms “comparatively uniform”. By these reports, the object we trace, a high-norm image token that acts as an attention sink, is not reported in these models, so they can neither confirm nor refute a mechanism defined on image-stream registers. We have not tested FLUX.2 and make no claim about it.

W2 (Distinction from prior work). First, the correspondence between massive activations and attention sinks is related to Sun et al. (2024), and Gan et al. (2025, 2026) show that massive activations occur in a few fixed dimensions across image tokens. We add that the dominant channel contains both a large activation shared by ordinary image tokens and sparse, much larger peaks at the high-norm register positions, which coincide with the attention sinks in FLUX.1 and PixArt-Σ (Sections 3.2 to 3.4). By contrast, channel 1446 in FLUX.1-Dev ranks second by mean activation but has no comparable sparse peaks (Section 3.4.2).

Second, a shared direction dominated by one massive channel is partly expected, but the channel magnitude does not explain sink identity. On 50 generations per model (Appendix D.1, Table 7), doubling the dominant channel alone reduces attention to 0.75 in FLUX.1-Dev and 0.07 in PixArt-Σ. Setting the channel to zero once leaves 0.90 in FLUX.1-Dev and makes the PixArt-Σ register a 31 times stronger sink. Writing the channel into ordinary tokens at the registers' magnitude creates no sink. Removing only the part of v⋆v^\star outside the dominant channel, about 1% and 6% of its squared length, leaves 0.34 and 0.04 of the attention. In FLUX.1-Dev, removing the full v⋆v^\star component at fixed norm leaves 0.05; in PixArt-Σ, the same edit also removes most of channel 293 and instead strengthens the sink. Suppressing the sink by editing its keys leaves the norm and dominant channel essentially unchanged (Appendix D.2). The part of v⋆v^\star outside the dominant channel is also reproducible across the two FLUX.1 checkpoints (cosine 0.988 after channel 154 is excluded; Appendix B.2).

Finally, Gan et al. (2026) connect massive activations to local detail synthesis, and we do not present that connection as new. We intervene on the sparse register tokens rather than disrupting the massive dimensions themselves, and vary the layer at which the state is present. In FLUX.1-Dev, inducing the state outside its natural layer range changes the image by LPIPS 0.07 to 0.09 while CLIP-I remains above 0.98. In PixArt-Σ, introducing it late, maintaining it beyond its endpoint, or removing it gives LPIPS 0.70 to 0.94 (Section 6, Appendix C.5, Table 5).

W3 (Broader significance). We agree the original presentation was fragmented. The revision follows one register state through its lifecycle, in which it is written, read by attention, preserved by its large norm, and erased (Sections 3 to 5). In FLUX.1-Dev, the additional angular drift at quarter norm is 3.08 times that at half norm, against a predicted 3 (Section 4.2, Appendix D.3). A large norm therefore protects the state, and shrinking a register destabilizes it.

New experiments identify the units that write and erase this state (Section 5, Appendix D.7). 205 of the 12,288 neurons of the layer-18 MLP (FLUX.1-Dev) and 97 of the 4,608 of the layer-16 MLP (PixArt-Σ) account for 90% of each layer's write. Silencing the writer neurons leaves 0.13 and 0.063 of the register state. MLPs erase the state in FLUX.1-Dev, while in PixArt-Σ three self-attention heads of layer 21 account for 64%, 18% and 12% of that layer's removal. Silencing the eraser units leaves 5.1 and 14 times as much state after the erasing layers, and it persists to the final layer. The state can thus be removed or kept through a few units.

Practically, Table 7 shows that modifying the dominant channel does not necessarily suppress sinks; methods should target the part of the register direction outside it. Moving or removing the state changes the image, far more in PixArt-Σ (Section 6), a possible editing target. We do not yet demonstrate an application; Section 7 names activation-aware quantization and compression as future directions.

Reviewer_noAxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes high-norm register tokens, massive activation channels, and attention sinks in three text-to-image Diffusion Transformers: FLUX.1-Dev, FLUX.1-Schnell, and PixArt-Σ. It finds that high-norm register tokens coincide with attention sinks. Their representations align with a direction v*. This direction is dominated by one massive activation channel, which accounts for most of the register tokens’ high norm. Through controlled interventions, the paper shows that the direction, not the norm, determines which token becomes the attention sink, while the large norm keeps that direction stable across subsequent layers. It shows that the state’s effect on the generated image depends on the layer at which it is present.

Critical strengths
  1. Coverage of two DiT architectures. The analysis spans two different families of Diffusion Transformers: MMDiT-style joint attention (FLUX.1-Dev and FLUX.1-Schnell) and cross-attention conditioning (PixArt-Σ). The same pattern appears in both, which suggests the findings are not tied to a single architecture.
  2. Disentanglement of direction and magnitude. The paper ablates the register representation to distinguish the role of direction and magnitude. The results suggest that direction determines sink identity, whereas massive magnitude helps preserve the register direction across subsequent layers.
Critical weaknesses
  1. Incremental novelty. High-norm tokens, massive activations, and attention sinks have each been studied in Transformers, including Diffusion Transformers, as mentioned in Sec. 1-2. Prior work has also connected massive activation to attention sinks in language models [1] and high-norm tokens are created by a few register neurons that write into a few specific channels, resulting in sinks in ViT [2]. While the paper's direction and magnitude disentanglement in attention sinks is insightful, the overall contribution appears incremental relative to prior work. The authors should clarify what new findings this connection provides beyond prior work and why they matter.
  2. Key definitions are written in prose rather than formulas. Several key measurements and procedures are described in text (e.g. Sec. 3.1 and Sec. 4), which makes it hard to understand exactly what is measured and how. Writing these as equations would improve the clarity and reproducibility of the analysis.
  3. Inconsistent evaluation settings. The analyses are performed at different denoising steps: the final step in Sec. 3, step 19 in Fig. 5, and step 21 of 28 in Tab. 3. Some analyses use very few samples. Fig. 4 uses only five samples for one model variant. In addition, Sec. 6 and Fig. 5 and 6 do not clearly report the number of samples used. I encourage the authors to clarify the rationale for these differing evaluation settings and report the sample counts consistently. Where appropriate, evaluating under more consistent settings would help establish the robustness of the findings.

[1] "Massive Activations in Large Language Models", Sun et al., COLM 2024.
[2] "Vision Transformers Don’t Need Trained Registers", Jiang et al., NeurIPS 2025.

Questions for authors
  1. Denoising step. Main analysis are done at the one denoising step. However, as [3] showed, diffusion features are known to change across timesteps. An analysis on how register state evolves across denoising steps (e.g. the number of register tokens, attention sinks, etc) would show whether the one-direction finding holds throughout the denoising process or only at the final step.

[3] "Massive Activations are the Key to Local Detail Synthesis in Diffusion Transformers", Gan et al, ICLR 2026

Minor comments

  1. Notation of v*. The direction v* is different in the main text and some figures (e.g. Fig. 1). Please unify the notation.
  2. Section organization. The paper uses 12 sections, several of which are short and closely related, which makes the overall argument hard to follow. I encourage the authors to use a few top-level sections for the main topics, with subsections for individual findings. For example, Sec. 3–5 could be merged into one section on the register state, and Sec. 6–8 into a section on the direction and magnitude.
  3. Typos.
    • Fig. 1 caption: "High-norm" → “high-norm”.
    • L80: "prompts, seeds." → "prompts and seeds.".
    • L454: "broke" → "break"
    • The register-zones appear inconsistent: PixArt-Σ ranges 13–21 (Sec. 3.2), 13–22 (Sec. 9), and 13–20 (Tab. 3 caption). and FLUX-Dev 19-35 (Sec. 9), 18-39 (Tab. 3 caption), and 18-41 (C.5)
3 comments
AuthorsOpenReview ↗

Response to Reviewer noAx ( Part 1/3): Novelty Beyond Prior Work

We thank the reviewer for their comments and valuable feedback. We have revised the manuscript where needed and address the concerns below.

Section numbers refer to the revised PDF, which merges the old Secs. 3–5 into Sec. 3 (The Register State) and the old Secs. 6–8 into Sec. 4 (Direction and Magnitude). New experiments (App. D) use FLUX.1-dev and PixArt-Σ, with 50 held-out generations per model (10 prompts × 5 seeds) and 95% confidence intervals calculated by bootstrap resampling of prompts.

W1. Incremental novelty relative to [1] and [2]

Our contribution: First, we establish that three previously studied phenomena, high-norm tokens, massive activation channels, and attention sinks, co-localize at the same image-token positions in DiTs and are linked by a shared, model-specific register direction v*. No previous DiT work has established this three-way co-localization or identified a common representation connecting all three. Second, we go beyond this observation through causal interventions that distinguish the roles of channel magnitude, representation direction, and attention, and identify the components that write and erase the register state.

What prior work established. Previous DiT research studied these phenomena separately: massive activation channels (Gan et al. 2025, 2026; Turri et al. 2026), high-norm outlier tokens (Wu et al. 2026), and attention sinks (Wu & Summa 2026). None established that all three occur at the same token positions in DiTs. In other architectures, [1] connects massive activations with sinks in LLMs, while [2] identifies neurons that produce high-norm tokens associated with sinks in ViTs. These studies provide important foundations, but neither establishes the three-way relationship or shared register representation especially in DiTs. We clarify these distinctions in Sec. 2.

Why v* matters. The three phenomena were previously studied without a common explanation for their occurrence at the same positions. We find that register representations are nearly collinear along v*, estimated independently from unit-normalized register states. Its dominant channel accounts for most of the register norm, but smaller components are important for sink attention. Thus, v* provides a common representational link between all three phenomena and enables mechanistic tests that channel-only or norm-only analyses would miss.

More broadly, identifying a shared register direction suggests a potential approach to controlling attention. Future work could investigate whether analogous directions exist in LLMs and ViTs, and whether injecting them into selected tokens can redirect attention.

We provide the causal evidence supporting these contributions in Response 2/3.

AuthorsOpenReview ↗

Response to Reviewer noAx (Part 2/3): Causal Evidence and Formal Definitions

W1. Incremental novelty relative to [1] and [2] (continued)

What our experiments establish (FLUX.1-dev / PixArt-Σ; App. D):

FindingCausal evidenceBeyond [1] and [2]
Norm does not determine immediate sink attentionScaling whole register tokens by ×0.25 or ×4 leaves incoming attention at 1.00× in both modelsSeparates magnitude from attention under normalization
The massive channel alone is insufficientDoubling it reduces attention to 0.75× / 0.07×; inserting it into ordinary tokens produces top keys for only 1% / 0% of headsDirectly tests whether the massive channel creates sinks
Smaller components of v* matterRemoving the component outside the dominant channel (1% / 6% of v* squared norm) reduces attention to 0.34× / 0.04×Reveals a causal contribution missed by channel-only analysis
Sink attention does not immediately sustain the high-norm stateSuppressing sink attention leaves register norm and dominant-channel activation approximately unchangedTests the causal relationship in the reverse direction
Large norms stabilize direction across depthIn FLUX, directional drift at ×0.25 norm is 3.08× that at ×0.5 (first-order prediction: 3); PixArt-Σ loses its register state at ×0.25Identifies a persistence role for magnitude, distinct from attention selection
Specific components write and erase the stateSilencing writers reduces v* projection to 0.13× / 0.063×; eraser silencing preserves 5.1× / 14× more stateExtends [2]'s writer analysis to register erasure
Channel interventions differ across architecturesZeroing the dominant channel leaves 90% of incoming attention in FLUX but increases it 31.1× in PixArt-ΣShows that channel magnitude does not have a universal effect on sink attention

Why these findings matter. Prior work established important relationships between pairs of phenomena, but did not explain their joint occurrence in DiTs. Our analysis connects them through a shared register representation and distinguishes their mechanisms: the dominant channel supplies most of the norm, the norm helps preserve the representation across layers, and sink attention depends on the normalized representation, including components outside the dominant channel. We further identify the components responsible for writing and erasing this state. These findings provide a more precise basis for understanding and intervening on register behavior than treating massive activations, high-norm tokens, and attention sinks independently (Secs. 3–5; App. D.1–D.3, D.7).

W2. Key definitions in prose

We have replaced the prose definitions with explicit equations in Sec. 3:

  • Eq. 1: High-norm tokens satisfy ‖xᵢ‖ ≥ 3 · medianⱼ ‖xⱼ‖; at most the eight largest are selected as registers.
  • Eq. 2: Sink share is defined per attention head and image key, renormalized over image keys and expressed relative to uniform attention. A head has a sink when its maximum share exceeds 10× uniform.
  • Eq. 3: v* is the top right-singular vector of the unit-normalized register matrix, with alignment cos(x, v*) = xᵀv* / ‖x‖.
  • App. D: The new experiments additionally require alignment above the 99.9th percentile, and App. D.6 reports the register key's rank among image keys.
AuthorsOpenReview ↗

Response to Reviewer noAx (Part 3/3): Evaluation Settings, Denoising Steps, and Minor Comments

W3. Inconsistent evaluation settings and sample counts

We agree that the original submission did not sufficiently explain the differing evaluation settings or consistently report sample counts. We have addressed this as follows:

  1. Standardized protocol. All new intervention experiments use the same relative denoising point (75% of the trajectory: step 21/28 for FLUX.1-dev and 15/20 for PixArt-Σ) and 50 held-out generations per model (10 prompts × 5 seeds; App. D).
  2. Evaluation across denoising steps. We additionally measure register properties at eight or nine sampled steps spanning the trajectory (Apps. D.4, D.6), rather than relying on a single denoising step.
  3. Explicit sample counts. We have clarified the settings and sample counts in the captions of Figs. 4–6.

Addressing the specific concerns:

  • Fig. 4 (5 generations, FLUX.1-schnell). We extended the central direction-removal intervention to 50 generations each of FLUX.1-dev and PixArt-Σ (App. D.1). Whole-token scaling leaves immediate attention unchanged (1.00×). Removing v* at fixed norm reduces FLUX attention to 5% [4%, 6%] of baseline but increases PixArt-Σ attention 26×. This difference arises because PixArt-Σ's dominant channel suppresses sink attention; removing the smaller component of v* outside that channel instead reduces attention to 4% of baseline.
  • Sec. 4 [old Sec. 6] (norm versus direction). The experiments are repeated on 50 generations per model (App. D.1).
  • Figs. 5–6. The analyses are repeated on 50 generations per model across nine sampled denoising steps (App. D.6).

Q1. Does the register state persist across denoising steps?

Yes. We analyzed unedited generations across the denoising trajectory, measuring register properties across blocks at nine sampled steps (App. D.6). The following ranges summarize register-zone measurements across steps:

QuantityFLUX.1-devPixArt-Σ
High-norm tokens per block (before top-eight selection)6–25; fewest at final stepExactly 1 in blocks 13–20; none elsewhere (450 generation–steps)
Heads with a sink (≥10× uniform)20–22 / 2414–15 / 16
Distinct sink tokens per block5–61.0–1.4
Sink heads whose top key is high-norm90–95%97–100%
All heads whose top key is high-norm64–88%89–92%
|cos(high-norm token, v*)|0.994–0.9980.984–0.993
Dominant channel's share of squared norm98–99%94–96%
Register key among top 10 by mean-query cosine98–99%93–95%
Depth of formationNorm ratio peaks at block 18Register state begins at block 13
  • The shared direction persists: high-norm tokens remain strongly aligned with v* throughout the sampled denoising trajectory, with consistently high overlap between high-norm positions and attention sinks.
  • The depth of formation is stable: FLUX's norm ratio peaks at block 18 and PixArt-Σ's register state begins at block 13 at every sampled step.
  • The number of tokens varies: FLUX has fewer high-norm tokens and lower head overlap at the final step. Thus, the original final-step descriptive figures do not represent the maximum observed overlap.
  • Relationship to [3]: Consistent with Gan et al., activation properties change across timesteps. However, the register direction and its association with attention sinks remain stable across the sampled steps.

These results support the one-direction finding beyond the single denoising step originally emphasized.

Minor comments

  1. Notation of v*. We have standardized the notation to v* throughout the revised manuscript, including the main text, equations, figures, and captions.
  2. Section organization. We adopted the suggested restructuring: Sec. 3, The Register State (old Secs. 3–5), and Sec. 4, Direction and Magnitude (old Secs. 6–8).
  3. Typos. Corrected the Fig. 1 caption, L80 (“prompts, seeds, and denoising steps”), and L454 (“break”).
  4. Register zones. We now use one definition (Sec. 3.2; App. C.5): the first through last layer where the register state reaches at least half its peak, giving layers 18–41 for FLUX.1-dev and 13–20 for PixArt-Σ. The earlier 19–35 range describes the strongest part of the FLUX register state, not its full zone. Table 3 explicitly identifies its evaluated layers (18–39). App. D.6 further confirms that PixArt-Σ's high-norm tokens occupy blocks 13–20 at every sampled step.
Reviewer_8m3COpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper does phenomenology on diffusion transformers for image generation. It investigates the links between three previously-observed phenomena: (1) individual tokens whose activations have huge norm, (2) "massive activations": individual channels of the activations (e.g., dimension 154) that have much larger magnitude than typical dimensions, and (3) attention sink tokens. The paper argues that these three phenomena are really one and the same.

Sound evidence is presented that mostly supports this claim. The evidence comes from examining the activations and attention patterns for some example generations. A few further claims are made about these phenomena in the later sections, which seem reasonable but have less clear evidence and implications.

Critical strengths

The paper addresses an interesting and well-scoped question using sound methods. It grounds most key claims in reasonably good evidence.

Critical weaknesses

I personally learned something interesting from reading this paper. However, when I read the related work, it seems that most of what I learned was already described there. In addition, some points seem simpler than the authors make them out to be.

The first part of the argument is that phenomena (1) and (2) are the same: activations with huge norm are caused by individual large channels, particularly channel 154 in Flux. The authors argue this point convincingly in sections 4 and 5. However, it seems to me that this point is simpler than they make it. Just pick a layer and show a heatmap of token vs. channel. There will be a few massive spikes at channel 154. Pick another layer and you'll see an almost same heatmap. I don't see a need to derive v* or do the experiments in section 5. Moreover, it seems like this fact is already known in the literature, as in the cited work of Jiang et al, Sun et al. (2024), and Gan et al. See, e.g., Figure 1 of Sun et al. (2024).

The second part of the argument is that phenomena (2) and (3) are the same; tokens with high activation in channel 154 are also the attention sink tokens (in the single-stream block). Figure 2b effectively shows that phenomena (1) and (3) occur for the same tokens. Section 6 clarifies that high activation norm is not per se what makes a token into an attention sink. In the paper's framing, it is the direction of the activation vector that "determines sink identity". This seems basically correct. However, I am not sure this finding is surprising. Contrary to line 300, we know how attention layers work, and they don't measure norms. They don't exactly measure "direction" either. If xix_i is the hidden embedding of token ii, x2⊤WkWq⊤x1=⟨x2,WkWq⊤x1⟩x_2^\top W_k W_q^\top x_1 = \langle x_2, W_k W_q^\top x_1\rangle determines how much token 1 attends to token 2. To me, it seems more correct to say that the dot product with v*---that is, the magnitude of channel 154---is what determines sink identity. I wish that this hypothesis were tested directly. Furthermore, I am still not sure what differentiates this finding from cited work like section 4 of Sun et al. (2024) or Jiang et al.; it seems to me they already associated attention sinks with large activations in specific channels.

Sections 7 and 8 speculate that making channel 154 very large helps cement the token's role as a sink token. This is perfectly sensible, but would not necessarily be surprising, and no supporting empirical or theoretical evidence is presented. Section 9 and 10 describe how the role of these special tokens is localized to the middle layers. I believe this was more or less understood previously. The finding in section 10 about local details holds only for Flux, not PixArt, and anyway it is a bit of a tangent.

Questions for authors

My main question for the authors in the rebuttal is, how does the present work advance our understanding beyond the previous work? For instance, lines 69 presents the relationship between massive activations and attention sinks as an active mystery by citing Sun et al. (2026): "yet the two phenomena can also be decopuled and serve distinct roles". But the present work doesn't really engage with the argument of that paper, which already showed that these two phenomena co-occur and tried to explain why on a mechanistic level (its answer: pre-norm). Am I missing something?

One potential source of novelty is the focus on diffusion transformers as opposed to LLMs or ViTs, though Gan et al, and Turri et al. also study DiTs. Do the three phenomena look different in DiTs than in the other architectures?

Additional questions:

  • For attention heads that attend to attention sinks, is it true that WkWq⊤x1≈v∗W_k W_q^\top x_1 \approx v^*, as I suspect (see above)? Am I right that the dot product some vector / the magnitude of channel 154 is what determines sink identity, rather than "direction"?
  • Line 130: "The strongest sink of approximately 90-100% of heads lies among the high-norm tokens". This is an absolutely key claim for the paper, but I don't see a figure or table supporting that. Is Figure 6 supposed to justify it?
  • Line 120: "All measurements are taken at the final denoising step". Why? What happens at other denoising steps? What does the related work (e.g. Gan et al.) say about the 3 phenomena at other denoising steps?
  • Line 119 and Figure 6: What is plotted on the y-axis of the middle row in Figure 6? "Attention sinks (relative to uniform)" sounds like the attention sinkiness score described in line 119 "Giving each image token a share of the image-to-image attention mass...relative to uniform." That gives a score to each token. But then, which token's score is being plotted in Figure 6? Same for the third row, "massive channels". Are we taking an average across tokens for each channel?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 8m3C (Part 1/3): Novelty, Causal Evidence, and Relation to Prior Work

We thank the reviewer for their careful assessment and constructive feedback. We have revised the manuscript and conducted additional experiments where needed to clarify our contributions and address the concerns raised. Section numbers refer to the revised manuscript; numbers in brackets refer to the original submission. We answer the main question first, then the weaknesses in the review's order, and then the additional questions.

1. Main question: what does the paper add beyond prior work

In DiTs, no previous paper has investigated that the three structures sit on the same tokens so that claim in DiTs is new, and our main contribution goes beyond it.

  • Prior DiT work studies each structure separately: massive channels (Gan et al. 2025, 2026; Turri et al. 2026), high-norm outlier tokens (Wu et al. 2026, without channels or attention), and sinks (Wu & Summa 2026). None show the three on the same tokens in DiTs. This is what motivated us for this paper, these structures were being studied separately and not through a common lens. We not only prove their co-occurence but also show that v⋆ is the common point in these three structures.
  • Even in other architectures like LLMs and ViTs, co-occurrence is known only pairwise: massive activations with sinks in LLMs (Sun et al., 2024), high-norm tokens with sinks in ViTs (Jiang et al., 2025).
  • Beyond co-localization, we give a causal analysis of the shared register state: which representation components govern attention, what the large norm does, and which model components write and erase the state.

A globally massive channel is not itself a register. The dominant channel is large across image tokens but is selectively amplified at sparse register positions. These tokens exhibit all three signatures and align with a shared, model-specific direction v⋆v^\star. We use targeted interventions to distinguish the roles of channel magnitude, representation direction, and attention (App. D; FLUX.1-Dev / PixArt-Σ):

InterventionMeasured effect (FLUX / PixArt-Σ)What it shows
Replace register tokensNorm falls to 11% / 6%, dominant channel to 8% / 4%, and incoming attention to 2% / 0% of baselineAll three signatures are concentrated at register positions
Suppress sink attention (replace register keys with mean key)Register norm and dominant channel remain approximately unchangedSink attention does not sustain either signature in this intervention
Double the dominant channelIncoming attention falls to 75% / 7% of baselineIncreasing channel magnitude does not strengthen sink attention
Remove the component of v outside the dominant channel*Incoming attention falls to 34% / 4% of baselineThe small, non-dominant component is important for sink attention
Reduce register norm while preserving directionFLUX: directional drift increases 3.08× (predicted 3×); PixArt-Σ: register state is lost at 0.25× normLarge norms help preserve register direction across layers
Silence identified writer / eraser unitsWriter silencing reduces v* projection to 0.13× / 0.063×; eraser silencing preserves 5.1× / 14× more stateSpecific model components causally contribute to writing and erasing the state

Resulting picture. The dominant channel accounts for most of the register norm, which helps stabilize its direction across layers. However, channel magnitude alone does not explain sink attention: the remaining components of v* also matter. Suppressing sink attention leaves the norm and dominant channel approximately unchanged. We further identify the components that write and erase the register state.

Sun et al. (2026) [line 69]. Their account is architecture-level and concerns two structures in LLMs: pre-normalization makes massive activations and sinks co-occur. Ours is consistent with it and adds the token-level mechanism in DiTs:

  • Normalization is why whole-token magnitude drops out (×0.25 and ×4 leave attention unchanged).
  • What matters instead is the normalized representation, including a low-energy part of v⋆; increasing the massive channel weakens incoming attention.
  • Their observation that the two can be decoupled has a token-level counterpart: zeroing the dominant channel removes most of the norm but keeps 90% of the FLUX sink and strengthens the PixArt-Σ sink.

Line 69 and Sec. 2 are rewritten accordingly.

Do these phenomena differ in DiTs?

  1. The massive channel is present on every image token (as Gan et al. 2025 note, unlike the token-specific outliers of ViTs). The registers are a sparse population that carries it far more strongly (about 6× in PixArt-Σ), only inside a middle-depth zone (App. D.5, D.6).
  2. A larger activation does not attract attention: increasing the dominant channel weakens incoming attention in both models, whereas in LLMs and ViTs large activations and high norms are associated with sinks.
AuthorsOpenReview ↗

Response to Reviewer 8m3C (Part 2/3): Why the Shared Direction Matters and What Determines Sink Attention

2. "Phenomena (1) and (2) are the same; a token × channel heatmap shows it; v⋆ is unnecessary"

We agree that the channel is massive and do not claim it as new: massive channels are known in LLMs (Sun et al. 2024, Fig. 1), and Gan et al. (2025) identified FLUX dimensions 154 and 1446; both are now credited (Secs. 2, 3). The heatmap is added (App. D.5, Fig. 25). It identifies the large coordinate, but not its relation to the registers and sinks:

  • The channel is large on every token. In PixArt-Σ, channel 293 holds 71% of an ordinary token's squared norm (median 50); in FLUX, channel 154 has mean |activation| 1,945 against 6.3 for the median channel (Table 3).
  • Only a sparse subset of tokens becomes high-norm registers and sinks. On them the channel is far larger (in PixArt-Σ about sixfold: 290, 94% of the squared norm), and the full token representations align with v⋆ (|cos| 0.98–0.998 at nine sampled denoising steps).
  • A second massive channel (1446, mean 880) marks no registers.

Why v* matters. Previous work identified high-norm tokens, massive activation channels, and attention sinks, but did not establish what connects all three at the same token positions in DiTs. We find that these register tokens share a stable, model-specific direction v*, providing a common representational link between the three phenomena.

  1. v* is identified independently of channel selection. It is fitted to unit-normalized register representations, its 99% squared mass on channel 154, rather than the also-massive channel 1446, is an empirical finding.

  2. The dominant channel explains the high norm but not sink behavior. Although the channel is large across image tokens, only a sparse subset becomes registers. Inserting the channel alone into ordinary tokens makes them the top key for just 1% / 0% of heads (FLUX / PixArt-Σ), whereas inserting the v* projection yields 43% in FLUX (and increases attention 4.7-fold in PixArt-Σ without creating top keys).

  3. The remaining components of v* are causally important. The 1% outside channel 154 is reproducible across independent FLUX.1-Dev and FLUX.1-Schnell fits (absolute cosine similarity 0.988; App. B.2). Removing the corresponding component reduces incoming attention to 34% / 4% of baseline (FLUX / PixArt-Σ).

Thus, v provides a common representational link between the three structures, while our interventions distinguish the roles of its dominant channel and smaller components. A channel-only analysis would miss this distinction.*

3. "(2) and (3): the dot product with v⋆, i.e. the magnitude of channel 154, determines sink identity. I wish this were tested directly"

We agree that Fig. 2b shows the high-norm tokens and the sinks on the same positions. The question the paper adds is which property of the token the attention responds to.

The formula. x₁ᵀ W_Qᵀ W_K x₂ omits the normalization before attention. In both models the key is computed from LayerNorm(x₂) with a timestep-dependent scale and shift, and FLUX also normalizes each head's query and key. Rescaling x₂ therefore leaves its key unchanged. We agree that attention does not measure norms; the submission's line 300 overstated the ambiguity, and and Sec. 4.1 and App. D.1 now state this.

Direct test (Sec. 4, App. D.1, Table 7). Attention received by the register tokens after one edit at one block, relative to unmodified (unmodified: 28.1× / 137.5× uniform):

EditFLUX.1-devPixArt-Σ
Whole token ×0.25 / ×41.00 / 1.001.00 / 1.00
Dominant channel ×2 / ×0.50.75 / 1.160.07 / 16.7
Dominant channel set to 00.9031.1
Part of v⋆ outside the channel removed0.340.04
v⋆ removed, norm kept0.0525.8
  • Not the raw dot product with v⋆: it scales with the token, yet ×4 changes nothing.
  • Not the magnitude of channel 154: increasing it weakens incoming attention in both models; with it zeroed, 62% of FLUX heads still pick a register token (79% unmodified).
  • What matters is the normalized representation, including the low-energy part of v⋆, which is causally important. Whether it acts through the key projection or the normalization statistics is not yet separated (App. D.1).
  • PixArt-Σ differs: removing v⋆ at fixed norm (94% of it is channel 293), or zeroing channel 293, strengthens the sink (26×, 31×), so there the channel limits the sink. A plausible reason, which we have not tested, is the absence of query–key normalization.

This separates the paper from Sun et al. (2024, Sec. 4) and Jiang et al.: they associate sinks with large activations; here, increasing the activation lowers attention, and a register reduced to an ordinary token's norm remains the sink of 89% of the affected heads (Sec. 4).

AuthorsOpenReview ↗

Response to Reviewer 8m3C (3/3): Secs. 4–6, lines 119–130, Fig. 6

4. Secs. 4.1–4.2 [7–8]: "no supporting empirical or theoretical evidence is presented"

The paper claims that the large norm keeps the register's direction stable across layers (Sec. 4.2), not that the norm makes the token a sink (§3 shows it does not). Evidence in the submission, now cited:

  • Holding the norm through depth keeps the direction (cos 0.99 through layer 50) and the sink (85 of 100 generations at layer 50), extending alignment with v⋆ by 14 / 3 layers (Table 4).
  • After the register is damaged, restoring its norm alone does not recover the sink; restoring the full register does (Fig. 18).
  • The register key is among the image keys most aligned with the mean image query (Fig. 5).

New:

  • Theory and test (App. D.3). To first order, an update u rotates a state s·x by ‖u⊥‖ / (s‖x‖), predicting drift ratios 3 and −0.5 for s = 0.25 and 2, relative to s = 0.5. FLUX measures 3.08 and −0.25: the prediction holds when the norm shrinks and overstates the gain from a larger norm. PixArt-Σ at s = 0.25 ends 49° off v⋆ and loses its sink.
  • Maintenance (App. D.2). Keeping the channel at zero over four blocks leaves 26% / 2% of the sink, against 90% after a single zeroing in FLUX.
  • Query–key geometry at scale (App. D.6). At nine sampled denoising steps, a register key is among the ten keys most aligned with the mean image query for 98–99% / 93–95% of heads; its cosine is 0.61–0.78, against −0.14 to 0.07 for the median image key.

5. Secs. 5–6 [9–10]: "localization was more or less understood"; "local detail holds only for Flux; a tangent"

  • The middle-layer localization is now credited to Wu et al. (2026) (Sec. 2). Sec. 5 (Fig. 7) and App. D.7 add which units write and erase the state, selected on 40 generations and tested on 60 held-out ones:
    • Writers: sparse MLP neurons, in both models. In FLUX.1-dev, 205 of the 12,288 neurons of the layer-18 MLP carry 90% of the write (three carry half); in PixArt-Σ, 97 of 4,608 in layer 16. Silencing the writers leaves 0.13× / 0.063× of the state; random and matched controls leave 1.0×.
    • Erasers: MLP neurons in FLUX.1-dev; in PixArt-Σ, attention heads 9, 6 and 12 of layer 21 (94% of that layer's removal). Silencing them leaves 5.1× / 14× as much state, which then persists to the final layer.
    • Rejected alternative: competition from the second massive channel leaves the end within ±1 layer (Table 4).
  • The state forms at the same depth at all nine sampled denoising steps (App. D.6).
  • Agreed that the local-detail result is FLUX-only; Sec. 6 now scopes it to FLUX and states that PixArt-Σ is substantially more sensitive (late introduction and removal break the image). We keep it because it shows that the state's effect depends on its depth.

6. Line 130: "a key claim, but no supporting figure or table"

Correct, and the statement was imprecise. Measured at eight sampled steps (App. D.4, Fig. 24), the most-attended image token is a high-norm token for 90–95% / 97–100% of the heads that have a sink (≥ 10× uniform), and for 64–88% / 89–92% of all heads. Sec. 3.2 now states these numbers and cites the figure.

7. Line 120: why the final step; other steps; related work

  • v⋆ is fitted on states pooled across denoising steps (Sec. 3.3); only the descriptive read-outs used the final step.
  • At the sampled steps (App. D.4, D.6), registers align with v⋆ (|cos| 0.994–0.998 / 0.984–0.993) and the channel holds 98–99% / 94–96% of their squared norm. In FLUX the norm ratio peaks at block 18 at every sampled step; in PixArt-Σ every generation has exactly one high-norm token in each block 13–20 and none elsewhere. The FLUX final step has the fewest high-norm tokens (6 per block, against 14–25) and the lowest head overlap, so the final-step figures understate the overlap.
  • Gan et al. (2026) report that massive activations change across timesteps; they do not study high-norm tokens or sinks.

8. Line 119 and Fig. 6: what is plotted

The sink share is now Eq. 2 (per head and image key, relative to uniform). Fig. 6 plots, per block, the mean over generations of: top, largest / median token norm; middle, the strongest sink, i.e. the largest share any head gives a single image token; bottom, not an average over tokens: each channel's peak |activation| over image tokens, largest channel's peak divided by the median channel's. The caption now defines all rows.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.