acceptodds
Under review as a conference paper at ICLR 2027

A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions

Abstract

When attention concentrates on a single token, a *sink*, what is the model actually computing? Attention sinks are ubiquitous in softmax transformers, yet this shared visual signature can hide fundamentally different algorithms. We show that visually similar sink patterns can reflect two distinct mechanisms: *(i)* adaptive *NOP*, where a head suppresses its update by routing to a null token, and *(ii)* broadcast, where a sink aggregates and redistributes global information. Each mechanism leaves distinct traces (*NOP* sinks exhibit negligible value norms; broadcast sinks induce low-rank outputs), which we formalize on synthetic tasks and use to derive practical diagnostics. Applied to pretrained vision transformers, these diagnostics reveal that both mechanisms exist at scale: sinks transition from `CLS` in early layers to patches in deeper layers and concentrate in specialized heads. Causal interventions further connect these signatures to near-null suppression and shared residual contributions. We then use architectural interventions to show how these computations can be reorganized: gating eliminates detected *NOP*-like sinks but increases broadcast-like sinks, registers relocate rather than remove sink computation, and our position-free global pathway provides an explicit route for shared communication that reduces the broadcast-like sinks induced by gating. On dense probes, combining gating with the global pathway gives the strongest results among the tested variants, despite retaining some broadcast-like sinks. Overall, we find that the same attention pattern can reflect two very different computations, and that effective intervention depends not only on identifying the computation, but also on providing architectural alternatives through which the model can reorganize it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision transformers, attention sinks, gated attention, register tokens, global pathway

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Sms5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work studies attention sinks in Vision Transformer models. It begins with hypothesizing that attention sinks are associated with one of two possible mechanisms: First, allowing an otherwise difficult zero-update to the residual stream; second, a global rank-1 update to all attending tokens. These are referred to as NOP, and broadcasting, respectively. Toy examples that can potentially rely on such mechanisms demonstrate that they indeed arise in the toy setting. A subsequent analysis finds both mechanisms in pretrained DINOv2 models as well. Studying the register tokens of a DINOv2 model additionally reveals the same behavior there, which is further supported by a causal intervention method that demonstrates different behavior between NOP and Broadcast tokens. Finally, the manuscript presents two complementary architectural interventions that are geared to move computational mechanisms from both types of sinks into specialized additional components. This convincingly removes sinks from the toy models, and also shows (more mixed) effects in a specifically trained JEPA model.

Critical strengths

The overall flow and argumentation of the work is strong; the progression from hypothesis to toy examples to analysis of large-scale models yields interesting mechanistic insights into attention sinks in vision models. To the best of my knowledge, these complementary mechanisms have not been described before and constitute a valuable contribution to the field. The evidence found from the experiments presented in Sections 3 and 4 is convincing. The causal intervention on register tokens is particularly strong, and clearly demonstrates distinct patterns for the two types of sinks in DINOv2.

Critical weaknesses

While the sink-token transition is demonstrated in multiple models, the main claim regarding the two distinct types of sink tokens is shown only for a single model (DINOv2-G). The generality of the findings therefore remains unclear. Further, the experiments presented in Section 5 do not appear to offer a clear conclusion. While the proposed interventions work well on the toy examples, the effectiveness for the JEPA model is limited; the combination of both interventions results in more sink tokens than the baseline (reported in l.455, correct me if I am wrong). It is not really clear whether this is due to a failure of the interventions (though they make sense intuitively and work on the toy models) or because the pathology does not directly generalize from DINOv2 to the JEPA model. This could be discussed more explicitly as a limitation.

Questions for authors
  1. What is actually plotted in Fig.6? How do the bars and the individual points relate to each other?
  2. There are some low-quality figures in the Appendix which should ideally be fixed for the camera-ready version.
3 comments

We thank the reviewer for their constructive feedback and for recognizing the mechanistic insights provided by our analysis, particularly the causal interventions. We have extended the pretrained-model analysis and substantially revised the architectural-intervention experiments to address the concerns raised.

Weakness 1: Generality of the two mechanisms

We agree that the original main text illustrated the two-regime analysis primarily through DINOv2-G. We have now extended the same analysis to ten pretrained ViTs (new Figure 18): DINOv2-L and DINOv2-G, each with and without registers; EVA-g/14; OpenAI CLIP ViT-L/14; MAE ViT-L/16; DeiT-III ViT-L/16; AugReg ViT-L/16; and SigLIP ViT-L/16. Using consistent sink-head diagnostics, we find that both regimes coexist in six of the ten models: all four DINOv2 variants, EVA-g, and AugReg. NOP-like sinks are widespread, occurring in nine of ten models across self-supervised, supervised, and language-supervised training. Broadcast-like sink heads are most prevalent in DINOv2 and substantially rarer elsewhere. Under the new Figure 18 analysis, CLIP, MAE, and DeiT-III exhibit only NOP-like sinks, while SigLIP exhibits essentially no detected sink heads. Importantly, the causal interventions in Figure 6 also cover four model variants (DINOv2-L and DINOv2-G, each with and without registers), rather than DINOv2-G alone. These additional results substantially strengthen the generality of our findings while also qualifying their scope: NOP-like suppression appears broadly across pretrained ViTs, whereas broadcast-like sinks are less universal and particularly prominent in DINOv2. We have updated the manuscript to reflect this distinction.

Weakness 2: Interpretation of the architectural interventions

We thank the reviewer for highlighting the ambiguity of the original LeJEPA results. To address this concern, we trained all six variants under a common experimental recipe (LeJEPA ViT-L, 100 epochs, the same training configuration and three seeds per variant) and updated Table 1 and the accompanying analysis. The results establish that the interventions reorganize sink computation in different ways and that reducing sink prevalence does not necessarily improve representation quality.

In particular, the penalized global pathway alone eliminates detected sinks of both types under our primary criterion (Definition 1, =0.5): 0.00 NOP-like and 0.00 broadcast-like sinks per image. Adding gating preserves the absence of NOP-like sinks but reintroduces 0.49 ± 0.54 broadcast-like sinks per image. These are not merely negligible attention artifacts: Figure 19 shows that their effective residual writes, measured after gating, output projection, and LayerScale, are approximately 5% of the residual-stream norm, with a median shared fraction of 0.91 across tokens. Thus, the combined model retains consequential shared updates through attention even though an explicit global pathway is available. Our sensitivity analysis in Appendix E.2 also shows that exact sink counts depend on the detection criterion.

The downstream results reveal an additional tradeoff. Gating + Global achieves the highest ImageNet linear-probe accuracy (0.697) but underperforms gating alone on dense probes (ADE20K: 0.159 versus 0.186; VOC12: 0.452 versus 0.480). In contrast, Gating + Registers achieves the strongest dense-probe performance (ADE20K: 0.200; VOC12: 0.522). The global pathway alone substantially suppresses sinks, but this does not translate into the strongest downstream representations.

We therefore agree that the interventions should not be interpreted as independent fixes that necessarily eliminate sinks when combined. Our updated interpretation is that providing an alternative mechanism for one computation can change how the network implements another. The controlled experiments demonstrate that broadcast computation can be relocated to a position-free pathway, but the LeJEPA experiments show that this relocation is not guaranteed when gating is introduced simultaneously. We now discuss this interaction and its dependence on architectural choices as a limitation, rather than presenting sink reduction as a universally successful mitigation strategy.

Question 1: What is plotted in Figure 6?

We apologize for the incomplete caption. Figure 6 shows the relative change in final-layer patch representations caused by interventions on individual sink heads in pretrained DINOv2 models. Specifically, we compute the RMS difference between the original and intervened patch tokens, after the final LayerNorm, divided by the RMS magnitude of the original tokens, excluding the sink position. Thus, a value of 5% means that the final patch representations change by 5% of their original magnitude.

Each point represents one sink head, averaged over the images in which that head exhibits sink behavior. Points are colored by model (DINOv2-L and DINOv2-G, each with and without registers), while each bar reports the mean across heads. Blue bars indicate sink-value interventions, and gray bars indicate routing interventions or non-sink controls. We analyze layers 1 onward, excluding block 0.

The interventions distinguish the two mechanisms: NOP-like heads are largely insensitive to modifying their near-null sink values (approximately 0.4–1% change), but replacing the sink value with an ordinary value or removing attention to the sink causes substantially larger changes (8% and 3.5%, respectively). Conversely, broadcast-like heads are sensitive to zeroing the sink value (5.5%), while replacement with the head's mean sink value or a sink value from another image has substantially smaller effects (approximately 0.9–1.4%). We have updated the caption to clarify the metric, aggregation, and meaning of the individual points.

Question 2: Figure quality in the Appendix

We thank the reviewer for pointing this out. We will regenerate the affected appendix figures at higher resolution, increase font and label sizes where needed, and verify readability at the final publication size.

Thank you for the response, as well as running the additional experiments. My concerns have been addressed, even though the new experiments do not provide very strong additional support of the paper's main story. I think that the (lack of) prevalence of different sink types across model families should be discussed briefly in the main paper rather than being relegated to the appendix. Currently, it still reads overly general in my opinion.

I think that overall the paper is interesting and retain my score.

Reviewer_gUn1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper distinguishes suppression through sink values that are near 0 from broadcasting a shared small but consequential update. This analysis is conducted on synthetic tasks and also on pre-trained transformers and the data suggests that both sink nodes trapping communications and sink nodes that carry broadcasting exist. Analysis considers a pathway that includes both pooling and broadcast explicitly, with computation moving onto this pathway and ViT-L showing a reduced sink count as a result. The paper therefore makes a general case of different roles for sinks, and a direct connection between proposed suppression and broadcasting mechanisms and the existence of said sinks.

Critical strengths
  1. The controlled interventions applied do well at distinguishing routing, necessity, and transferable content - for the NOP task, identity-target queries are assigned to the target sink at a rate above 99% which feels like a convincing statistic.
  2. The experiment that connects to a particular architectural variant in the explicit sink/broadcast mechanism relocates shared computation while maintaining similar task error. It's important to note that providing the pathway alone does not guarantee relocation, but some relocation nevertheless is observed suggesting an important role for what the particular sink nodes (and types) are doing.
  3. Effects of controlled interventions are strong in general, with the exception that gating needs to be carefully examined
Critical weaknesses
  1. The pathway comparison has issues in that LayerScale initialization varies without a control - this undermines the strength of the effects claimed to a degree. The authors might consider some careful analysis / ablation around this. I think this probably doesn't threaten the general arguments of the paper, but it would rule out one concern a critic might raise.
  2. The gated sink labels don't necessarily establish broadcasting since a large sink value can be suppressed by gating, and query dependent gates can destroy the shared update structure. This is something that needs to be addressed very directly (either through argument in the text, or additional ablations/experiments). The argument for broadcasting interacts with gating in a critical way.

EDIT FOR REBUTTAL: The authors have taken both of the stated weaknesses and questions, turned them into experiments, and shown that these either don't threaten the argument made or that a substantive part of the argument survives, so I have raised my score accordingly.

Questions for authors
  1. After applying the gate, output projection, and LayerScale, how large are the counted sinks actual residual contributions and to what degree are these shared?
  2. Do sink-count and probe-performance conclusions survive matched LayerScale initialization, and how variable are they across backbone seeds? (I understand running extra seeds may not be possible for a rebuttal period, but perhaps the authors can at least share expectations based on what they have seen in the experiments).
4 comments

We thank the reviewer for their careful assessment and constructive suggestions. We particularly appreciate their recognition of the causal interventions and the distinction between the computational roles of attention sinks. We have addressed both principal concerns through controlled retraining and additional measurements of effective residual contributions.

Weakness 1: Unmatched LayerScale initialization

We thank the reviewer for identifying this confound. We have now standardized the LayerScale initialization across all six LeJEPA variants in Table 1: the attention and MLP residual branches use the same initialization (10-4), and every model is trained from scratch for 100 epochs under the same base recipe, with three seeds (9000, 9001, 9002). The global-pathway variants were retrained under these conditions. The auxiliary global pathway itself has no LayerScale, as specified in Appendix C.2. All reported results are from the matched epoch-100 checkpoints.

The revised Table 1 reports the following mean ± standard deviation across three independently trained backbones:

VariantImageNetADE20KVOC12
Baseline0.694 ± 0.0010.166 ± 0.0060.438 ± 0.016
Registers0.693 ± 0.0020.187 ± 0.0020.499 ± 0.001
Gating0.695 ± 0.0000.186 ± 0.0040.480 ± 0.002
Global0.671 ± 0.0040.174 ± 0.0050.470 ± 0.008
Gating + Registers0.694 ± 0.0010.200 ± 0.0060.522 ± 0.017
Gating + Global0.697 ± 0.0010.159 ± 0.0270.452 ± 0.067

The sink-reduction result survives matched LayerScale initialization: the penalized global pathway eliminates detected sinks under our primary criterion (Definition 1, =0.5). However, the downstream conclusions are more nuanced. Gating + Global achieves the highest ImageNet classification accuracy, but its dense-probe performance is lower and more variable than gating alone. Gating + Registers instead achieves the strongest dense-probe results among the six variants. We have revised the manuscript to distinguish sink reduction from improvements in representation quality.

Weakness 2: Gated sink labels do not necessarily establish broadcasting

We agree with the reviewer that a large pre-gate sink value and concentrated attention do not suffice to establish broadcasting. Elementwise, query-dependent gates can both suppress the magnitude of a sink's effective contribution and destroy its shared structure across recipients. We therefore added Figure 19, which directly examines the effective residual writes associated with the broadcast-like sinks counted in Gating + Global, after applying the attention gate, output projection, and LayerScale.

The results show that the gate retains approximately 40% of the ungated write, leaving an effective contribution of approximately 5% of the incoming residual-stream RMS, comparable in magnitude to ordinary attention heads. Importantly, these writes remain strongly shared across recipient tokens: the median fraction of write energy contained in the common component is 0.91, compared with 0.54 for non-sink heads in the same layers. We compute this shared fraction using the common-component energy measure of Eq. (18), applied to the effective residual writes. These findings directly address the possibility that gating eliminates the broadcast contribution or makes it strongly recipient-dependent. Indeed, the shared-energy fraction also bounds effective-write stable rank by sr⁡(W)≤1/Eshared(W)\operatorname{sr}(W) \leq 1/E_{\mathrm{shared}}(W), implying a median stable rank no greater than approximately 1.10.

We therefore distinguish the original pre-gate diagnostic labels from effective broadcasting. For the counted sinks in Gating + Global, the post-gate measurements establish that their associated attention heads retain non-negligible, predominantly shared residual updates. This does not by itself establish image-specific payload transfer, but it confirms the broadcast-like computational structure relevant to our hypothesis. We have clarified this distinction in the revised manuscript.

Question 1: How large are the actual residual contributions, and how shared are they?

Figure 19 directly addresses this question for the counted broadcast-like sinks in Gating + Global. Measurements are taken over (image, head) pairs across three independently trained backbones. After gating, output projection, and LayerScale, the corresponding residual writes retain approximately 40% of their ungated magnitude and contribute approximately 5% of the residual-stream RMS. Their median shared-energy fraction is 91%, compared with 54% for non-sink heads. Thus, the counted sink heads are not merely attracting attention to values that are subsequently silenced: they retain substantial and predominantly shared residual contributions. In contrast, Global alone has no detected sinks under our primary criterion (Table 13), so there are no corresponding counted sink heads to evaluate.

Question 2: Robustness to matched LayerScale initialization and backbone seeds

All models in revised Table 1 use matched attention/MLP LayerScale initialization and three independently trained backbones per variant. The sink-count conclusions survive this control. Under Definition 1 with =0.5, Global produces 0.00 ± 0.00 NOP-like and 0.00 ± 0.00 broadcast-like sinks per image, while Gating + Global retains 0.00 ± 0.00 NOP-like and 0.49 ± 0.54 broadcast-like sinks per image (Table 13). Gating therefore reintroduces broadcast-like sinks even with the explicit global pathway. The latter counts vary substantially across seeds (0.01–1.24 per image), but remain nonzero in all three. Appendix E.2 further shows that the qualitative ordering persists under alternative sink criteria, although absolute counts depend on the definition.

The probe conclusions are more nuanced. Gating + Global achieves the highest ImageNet accuracy (0.697 ± 0.001), but underperforms gating alone on ADE20K (0.159 versus 0.186) and VOC12 (0.452 versus 0.480). Its dense-probe results also vary substantially across seeds (ADE20K: 0.139–0.190; VOC12: 0.384–0.518). By contrast, Gating + Registers achieves the best dense-probe performance among the Table 1 variants (ADE20K: 0.200 ± 0.006; VOC12: 0.522 ± 0.017). The other variants exhibit substantially smaller standard deviations, at most 0.006 on ADE20K and 0.017 on VOC12.

These findings reinforce our central interpretation: architectural interventions can reorganize sink computation, but removing sinks is not equivalent to improving representation quality. The effectiveness of an intervention also depends on how it interacts with the other computational pathways available to the model. We have updated the discussion to make this limitation explicit.

Dear Reviewer,

As the author–reviewer discussion deadline approaches, I would like to remind you to review the authors’ rebuttal and indicate whether your concerns have been adequately addressed.

If you have any remaining concerns or questions, please share them with the authors before the discussion period ends.

Thank you again for your time and valuable feedback.

Reviewer_gUn1OpenReview ↗

Response to reviewer comments

Rather than post an additional comment, I updated my original review, but it would be more obvious if I were to post this as an additional comment instead, so I am doing so:

The authors have taken both of the stated weaknesses and questions, turned them into experiments, and shown that these either don't threaten the argument made or that a substantive part of the argument survives, so I have raised my score accordingly.

An additional note: I feel that work that shows mechanistic results tends to get penalized for not showing something shiny and new that people can use to squeeze some extra percentage points out of whatever benchmark is the flavour of the week. The reality is that mechanistic results will tend to be more enduring.

Reviewer_CS4QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates two mechanisms underlying visually similar attention sinks: adaptive NOP, where attention to a near-zero value suppresses a head's update, and broadcast, where attention to a non-negligible value produces a shared, approximately rank-one update. The authors derive diagnostics based on value norms and update stable rank, validate them through synthetic tasks, and apply them to pretrained vision transformers using causal interventions on sink values and attention routing. They further propose a position-free global pathway with a common-mode penalty and evaluate its interaction with gated attention in LeJEPA ViT-L. The results suggest that gating suppresses NOP-like sinks but increases broadcast-like sinks, while combining gating with the global pathway reduces this increase and achieves the strongest dense segmentation-probe results among the tested variants.

Critical strengths
  • Clear and meaningful conceptual distinction: The paper distinguishes attention sinks used for conditional suppression from those producing shared residual updates. This provides a useful framework for interpreting attention patterns that appear visually similar but may serve different computational roles.

  • Well-designed causal experiments: The synthetic experiments distinguish attention routing from information transfer through targeted interventions, including donor-target comparisons that test whether information is actually communicated. Additional interventions on pretrained ViTs further support the proposed functional interpretations.

  • Simple and interpretable diagnostics: The proposed value-norm and stable-rank measurements are straightforward to compute and are evaluated across several pretrained vision transformers, providing evidence that both mechanisms occur beyond controlled synthetic settings.

Critical weaknesses
  • Limited theoretical and conceptual novelty: Lemmas 1–2 largely formalize direct consequences of concentrated softmax attention.

  • Unclear information-broadcasting role: In pretrained ViTs, replacing broadcast-like sink values with those from other images produces relatively small changes. This suggests that many shared updates may be largely image-invariant rather than communicating input-specific information. A constant-offset baseline would help distinguish genuine global communication from learned shared biases.

  • Limited functional validation and generality: The pretrained-model interventions measure changes in final-layer representations rather than downstream task performance, leaving the functional importance of individual sink mechanisms unclear. The architectural experiments are restricted to LeJEPA ViT-L, with no corresponding validation in language models or supervised ViTs, limiting the broader applicability of the conclusions.

  • Weak and potentially confounded architectural improvements: The mitigation experiments are limited to a single 50-epoch training setting, with small gains over gating alone (ADE20K: 0.217 vs. 0.216) and slightly worse ImageNet performance (0.606 vs. 0.610). The global pathway alone degrades all reported probes

Questions for authors
  • If broadcast sink values are largely image-invariant, how do they differ from a learned per-head bias? Would replacing them with a constant vector preserve performance?

  • Could the authors report the number and proportion of attention heads classified as NOP-like, broadcast-like, or neither for each model?

See Critical weaknesses as well.

7 comments

We thank the reviewer for their constructive feedback and for recognizing the conceptual distinction, causal experiments, and interpretable diagnostics as strengths of our work. We have expanded the pretrained-model analysis, added downstream intervention experiments, and substantially revised the architectural comparisons to address their concerns.

Weakness 1: Limited theoretical and conceptual novelty

We agree that Lemmas 1–2 formalize relatively direct consequences of concentrated softmax attention. Their purpose is to establish the precise assumptions and diagnostic implications of the two mechanisms, rather than to constitute our main theoretical contribution. Our central contribution is the joint mechanistic account: visually similar attention sinks can implement either conditional suppression through near-null values or non-negligible shared residual updates. Distinguishing these roles requires examining values and actual contributions rather than attention concentration alone.

Importantly, our additional pretrained-model interventions reveal a further computational distinction: a sink can deliver an approximately constant payload while still implementing a functionally important, attention-dependent update. On ImageNet, replacing the sink value with a fixed vector while preserving attention routing leaves accuracy nearly unchanged (86.12% to 86.08%), whereas replacing its contribution with an unconditional fixed offset reduces accuracy to 85.62%. Thus, payload content and conditional delivery are functionally distinct, even when the payload itself is approximately image-invariant. This distinction is not implied by attention concentration or Lemmas 1–2 alone and cannot be established from observational diagnostics without causal interventions.

Our contribution therefore lies in identifying and experimentally distinguishing these computational roles, deriving operational diagnostics, validating them through controlled and pretrained-model interventions, and connecting them to downstream performance and architectural design. We have clarified this positioning in the revised manuscript, distinguishing the elementary formal results from the empirical and mechanistic contributions.

Weakness 2: Unclear information-broadcasting role

We agree that a nonzero shared update need not communicate image-specific information. Crucially, our analysis distinguishes three properties that attention concentration alone cannot separate: (i) producing a non-negligible shared residual update, (ii) delivering that update conditionally through attention, and (iii) transferring an input-specific payload. Our broadcast hypothesis concerns the first property; the controlled broadcast task demonstrates the stronger case of input-specific payload transfer. Our new pretrained-model experiments directly test whether conditional delivery remains functionally important when the payload itself is approximately constant.

To address this distinction directly, we conducted additional interventions on pretrained DINOv2-L/14, evaluating ImageNet-1K classification and ADE20K segmentation on their complete validation sets (50,000 and 2,000 images, respectively). We compared four conditions: the unmodified model, replacement of the sink value by a fixed vector while preserving the original attention routing, replacement of the sink contribution by the same fixed offset for every recipient and image, and zero ablation. The backbone and downstream probes remain fixed, and the intervention is applied on every evaluated image.

The results are:

InterventionImageNet top-1 (%)ADE20K mIoU (%)
Unmodified86.1248.32
Fixed value (routing preserved)86.0848.32
Fixed offset (routing removed)85.6248.20
Zero ablation85.5646.48

First, the sink payload is largely replaceable by a constant: preserving attention routing while substituting a fixed value changes ImageNet accuracy by only −0.04 percentage points and leaves ADE20K mIoU unchanged. This confirms that input-specific variation in the sink value is not essential for the evaluated tasks.

Second, the importance of conditional routing is task-dependent. On ImageNet, replacing the sink contribution with an unconditional fixed offset reduces accuracy by 0.50 points, nearly the full 0.56-point loss from zero ablation. Thus, a fixed payload delivered through the original attention weights is functionally different from an unconditional bias. On ADE20K, however, the fixed offset reduces mIoU by only 0.12 points, compared with 1.84 points for zero ablation, recovering approximately 93% of the lost performance. For this task, the sink's contribution behaves much more like a learned shared bias. Restricting interventions to images where the head is actually detected as a sink yields the same qualitative pattern: on ImageNet, the fixed value has no effect, the fixed offset costs 0.07 points, and zero ablation costs 0.15 points.

These findings clarify the distinction between an approximately constant payload and its conditional delivery. Writing the sink contribution as

oi(s)(x)=Ais(x)WOvs(x),o_i^{(s)}(x)=A_{is}(x)W_Ov_s(x),

the fixed-value intervention replaces vs(x)v_s(x) with vs, preserving Ais(x)WOvsA_{is}(x)W_Ov_s. In contrast, an unconditional bias delivers the same vector to every recipient independently of Ais(x)A_{is}(x). Our ImageNet results demonstrate that these two operations are not functionally equivalent, even when the payload itself is approximately constant.

We therefore distinguish shared residual updates, conditional delivery through attention, and input-specific payload transfer. The pretrained results establish that a shared update can be functionally important without carrying substantial image-specific information in its value. They also show that conditional routing can independently contribute to downstream performance, although its importance varies by task. We have clarified this distinction and its limitations in the revised manuscript.

Weakness 3: Limited functional validation and generality

We thank the reviewer for this suggestion. We have extended our causal interventions to measure downstream task performance directly.

Pretrained ViTs. Using DINOv2-L/14 with its released linear classifier, we evaluated ImageNet-1K top-1 accuracy on all 50,000 validation images, intervening simultaneously on heads of a given sink type. Relative to the unmodified accuracy of approximately 86.1%:

  • Zeroing NOP-like sink values leaves accuracy effectively unchanged (−0.00 percentage points).
  • Zeroing broadcast-like sink values reduces accuracy by 0.55 points, compared with a 0.12-point reduction for the random non-sink control.
  • Replacing broadcast-like sink values with those of other images reduces accuracy by only 0.04 points.

The additional constant-value and constant-offset experiments described in our response to Weakness 2 provide a complementary functional test on ImageNet and ADE20K. Together, these interventions establish that broadcast-like sink contributions can matter for downstream performance even when their value content is approximately image-invariant, and that conditional routing is important for ImageNet classification in the tested setting.

Model generality. We have also extended the value-norm and stable-rank diagnostics to ten pretrained ViTs (Figure 18), covering self-supervised, supervised, and language-supervised pretraining. NOP-like sinks occur in nine of ten models, while both regimes coexist in six. Broadcast-like heads are most prevalent in DINOv2 and less common elsewhere. These results strengthen the generality of the observational findings, although causal intervention experiments remain concentrated on DINOv2.

Architectural validation. We additionally trained six LeJEPA ViT-L variants for 100 epochs under matched conditions, with three independently trained backbones each. We evaluate both ImageNet classification and frozen-backbone ADE20K and VOC12 segmentation. Gating + Registers achieves the strongest dense-probe results among the Table 1 variants, improving ADE20K by 3.4 points and VOC12 by 8.4 points relative to baseline, with essentially unchanged ImageNet accuracy.

We agree that architectural validation remains restricted to bidirectional vision transformers and that extending the interventions to language models is an important direction for future work. Autoregressive causal masking imposes additional constraints on shared communication, since earlier tokens cannot receive information from later positions, but it does not preclude broadcast-like updates from earlier positions to later recipients.

Comments on the paper

1 comment

General Response

We thank all reviewers for their careful and constructive feedback. We appreciate their recognition of the "clear and meaningful conceptual distinction" (CS4Q) between two mechanisms underlying attention sinks: adaptive NOP, which suppresses residual updates, and broadcast, which produces shared updates. Reviewers also highlighted our "simple and interpretable diagnostics" and "well-designed causal experiments" (CS4Q). In particular, gUn1 noted that the interventions "do well at distinguishing routing, necessity, and transferable content," while Sms5 described the causal intervention on register tokens as "particularly strong." We also appreciate the positive assessment of the paper's "overall flow and argumentation" and "interesting mechanistic insights" (Sms5). Building on this feedback, we have substantially strengthened the manuscript through additional experiments, causal interventions, and more detailed analyses.

Our revisions focus on three areas. First, we extend our analysis across ten pretrained ViTs and add downstream causal interventions. Zeroing broadcast-like sinks reduces DINOv2-L ImageNet accuracy by 0.55 percentage points, while zeroing NOP-like sinks has negligible effect. Second, we disentangle broadcast payloads from conditional routing. Replacing sink values with a fixed vector preserves performance almost exactly, whereas replacing the attention-dependent contribution with an unconditional fixed offset causes a 0.50-point ImageNet accuracy drop. These results establish that broadcast-like sinks can provide functionally important conditional shared updates without necessarily communicating image-specific content. Third, we address a shared reviewer concern about experimental scale and validity: Table 1 now comprises 18 ViT-L models (six variants × three seeds), all trained for 100 epochs with matched LayerScale and initialization. We also add controlled causal experiments and measurements of actual residual contributions. Broadcast-like sink updates remain substantial and highly shared after gating (median shared-energy fraction 0.91), while our expanded comparisons reveal that gating combined with the global pathway achieves the highest ImageNet accuracy, and gating combined with registers achieves the strongest dense-prediction performance.

We believe these additions directly address the central concerns raised in the reviews and substantially strengthen the empirical support for our conclusions. Taken together, the results reinforce our central finding: attention sinks implement distinct computations, and effective architectural interventions depend on preserving or reorganizing these functions rather than simply eliminating attention concentration. We thank the reviewers again for helping us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.