- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work studies attention sinks in Vision Transformer models. It begins with hypothesizing that attention sinks are associated with one of two possible mechanisms: First, allowing an otherwise difficult zero-update to the residual stream; second, a global rank-1 update to all attending tokens. These are referred to as NOP, and broadcasting, respectively. Toy examples that can potentially rely on such mechanisms demonstrate that they indeed arise in the toy setting. A subsequent analysis finds both mechanisms in pretrained DINOv2 models as well. Studying the register tokens of a DINOv2 model additionally reveals the same behavior there, which is further supported by a causal intervention method that demonstrates different behavior between NOP and Broadcast tokens. Finally, the manuscript presents two complementary architectural interventions that are geared to move computational mechanisms from both types of sinks into specialized additional components. This convincingly removes sinks from the toy models, and also shows (more mixed) effects in a specifically trained JEPA model.
Critical strengths
The overall flow and argumentation of the work is strong; the progression from hypothesis to toy examples to analysis of large-scale models yields interesting mechanistic insights into attention sinks in vision models. To the best of my knowledge, these complementary mechanisms have not been described before and constitute a valuable contribution to the field. The evidence found from the experiments presented in Sections 3 and 4 is convincing. The causal intervention on register tokens is particularly strong, and clearly demonstrates distinct patterns for the two types of sinks in DINOv2.
Critical weaknesses
While the sink-token transition is demonstrated in multiple models, the main claim regarding the two distinct types of sink tokens is shown only for a single model (DINOv2-G). The generality of the findings therefore remains unclear. Further, the experiments presented in Section 5 do not appear to offer a clear conclusion. While the proposed interventions work well on the toy examples, the effectiveness for the JEPA model is limited; the combination of both interventions results in more sink tokens than the baseline (reported in l.455, correct me if I am wrong). It is not really clear whether this is due to a failure of the interventions (though they make sense intuitively and work on the toy models) or because the pathology does not directly generalize from DINOv2 to the JEPA model. This could be discussed more explicitly as a limitation.
Questions for authors
- What is actually plotted in Fig.6? How do the bars and the individual points relate to each other?
- There are some low-quality figures in the Appendix which should ideally be fixed for the camera-ready version.
3 comments
Question 1: What is plotted in Figure 6?
We apologize for the incomplete caption. Figure 6 shows the relative change in final-layer patch representations caused by interventions on individual sink heads in pretrained DINOv2 models. Specifically, we compute the RMS difference between the original and intervened patch tokens, after the final LayerNorm, divided by the RMS magnitude of the original tokens, excluding the sink position. Thus, a value of 5% means that the final patch representations change by 5% of their original magnitude.
Each point represents one sink head, averaged over the images in which that head exhibits sink behavior. Points are colored by model (DINOv2-L and DINOv2-G, each with and without registers), while each bar reports the mean across heads. Blue bars indicate sink-value interventions, and gray bars indicate routing interventions or non-sink controls. We analyze layers 1 onward, excluding block 0.
The interventions distinguish the two mechanisms: NOP-like heads are largely insensitive to modifying their near-null sink values (approximately 0.4–1% change), but replacing the sink value with an ordinary value or removing attention to the sink causes substantially larger changes (8% and 3.5%, respectively). Conversely, broadcast-like heads are sensitive to zeroing the sink value (5.5%), while replacement with the head's mean sink value or a sink value from another image has substantially smaller effects (approximately 0.9–1.4%). We have updated the caption to clarify the metric, aggregation, and meaning of the individual points.
Question 2: Figure quality in the Appendix
We thank the reviewer for pointing this out. We will regenerate the affected appendix figures at higher resolution, increase font and label sizes where needed, and verify readability at the final publication size.
Thank you for the response, as well as running the additional experiments. My concerns have been addressed, even though the new experiments do not provide very strong additional support of the paper's main story. I think that the (lack of) prevalence of different sink types across model families should be discussed briefly in the main paper rather than being relegated to the appendix. Currently, it still reads overly general in my opinion.
I think that overall the paper is interesting and retain my score.
We thank the reviewer for their constructive feedback and for recognizing the mechanistic insights provided by our analysis, particularly the causal interventions. We have extended the pretrained-model analysis and substantially revised the architectural-intervention experiments to address the concerns raised.
Weakness 1: Generality of the two mechanisms
We agree that the original main text illustrated the two-regime analysis primarily through DINOv2-G. We have now extended the same analysis to ten pretrained ViTs (new Figure 18): DINOv2-L and DINOv2-G, each with and without registers; EVA-g/14; OpenAI CLIP ViT-L/14; MAE ViT-L/16; DeiT-III ViT-L/16; AugReg ViT-L/16; and SigLIP ViT-L/16. Using consistent sink-head diagnostics, we find that both regimes coexist in six of the ten models: all four DINOv2 variants, EVA-g, and AugReg. NOP-like sinks are widespread, occurring in nine of ten models across self-supervised, supervised, and language-supervised training. Broadcast-like sink heads are most prevalent in DINOv2 and substantially rarer elsewhere. Under the new Figure 18 analysis, CLIP, MAE, and DeiT-III exhibit only NOP-like sinks, while SigLIP exhibits essentially no detected sink heads. Importantly, the causal interventions in Figure 6 also cover four model variants (DINOv2-L and DINOv2-G, each with and without registers), rather than DINOv2-G alone. These additional results substantially strengthen the generality of our findings while also qualifying their scope: NOP-like suppression appears broadly across pretrained ViTs, whereas broadcast-like sinks are less universal and particularly prominent in DINOv2. We have updated the manuscript to reflect this distinction.
Weakness 2: Interpretation of the architectural interventions
We thank the reviewer for highlighting the ambiguity of the original LeJEPA results. To address this concern, we trained all six variants under a common experimental recipe (LeJEPA ViT-L, 100 epochs, the same training configuration and three seeds per variant) and updated Table 1 and the accompanying analysis. The results establish that the interventions reorganize sink computation in different ways and that reducing sink prevalence does not necessarily improve representation quality.
In particular, the penalized global pathway alone eliminates detected sinks of both types under our primary criterion (Definition 1, =0.5): 0.00 NOP-like and 0.00 broadcast-like sinks per image. Adding gating preserves the absence of NOP-like sinks but reintroduces 0.49 ± 0.54 broadcast-like sinks per image. These are not merely negligible attention artifacts: Figure 19 shows that their effective residual writes, measured after gating, output projection, and LayerScale, are approximately 5% of the residual-stream norm, with a median shared fraction of 0.91 across tokens. Thus, the combined model retains consequential shared updates through attention even though an explicit global pathway is available. Our sensitivity analysis in Appendix E.2 also shows that exact sink counts depend on the detection criterion.
The downstream results reveal an additional tradeoff. Gating + Global achieves the highest ImageNet linear-probe accuracy (0.697) but underperforms gating alone on dense probes (ADE20K: 0.159 versus 0.186; VOC12: 0.452 versus 0.480). In contrast, Gating + Registers achieves the strongest dense-probe performance (ADE20K: 0.200; VOC12: 0.522). The global pathway alone substantially suppresses sinks, but this does not translate into the strongest downstream representations.
We therefore agree that the interventions should not be interpreted as independent fixes that necessarily eliminate sinks when combined. Our updated interpretation is that providing an alternative mechanism for one computation can change how the network implements another. The controlled experiments demonstrate that broadcast computation can be relocated to a position-free pathway, but the LeJEPA experiments show that this relocation is not guaranteed when gating is introduced simultaneously. We now discuss this interaction and its dependence on architectural choices as a limitation, rather than presenting sink reduction as a universally successful mitigation strategy.