acceptodds
Under review as a conference paper at ICLR 2027

Attention Sinks in VLMs Are Borrowed, Not Seen

10 views

Abstract

We study how to detect and mitigate visual attention sinks in vision-language models (VLMs). Visual attention sinks are image tokens that absorb a large share of attention over the image regardless of the query or task, and they mislead methods that treat attention maps as a record of where the model “looked.” Prior work has detected sinks through massive activation dimensions and proposed redistributing their attention, but none have addressed what makes a token become a sink. Comparing six candidate detectors on ten VLMs from four families, we find that only the centered cosine similarity between an image token's key and the key of the language model's text-sink detects visual sinks. Thus, a visual sink is an image token whose key aligns with the text-sink's key. Moving a sink's key component along this direction to the median of the image keys sends its attention back to the text-sink (Are Borrowed) while leaving model answers unchanged(Not Seen). Building on this, we propose a training-free, entropy-triggered re-look that first cleans the attention map, then crops the most-attended region of the cleaned map and inserts it into the reasoning trace. Cleaning adds negligible computation to the forward pass, and the re-look improves accuracy with a mean gain of points on fine-detail VQA benchmarks across model families.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLMs, attention mechanism

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fiibOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies "visual attention sinks": image tokens that absorb a large share of in-image attention regardless of the question. Across ten VLMs from four families (Sec. 3.2, Table 1) it reports that (i) the language-model backbone largely decides which models have sinks; (ii) the centered cosine between an image token's key and the text-sink key (Eq. 2) separates sinks with AUC >= 99.0 in all seven sink-bearing models, whereas norm- and logit-based detectors are family-specific (Table 3); (iii) projecting that direction out of sink keys removes sink attention and writing it into random image tokens creates it (Table 7); (iv) a gated key edit (Eq. 3) returns the sinks' attention to the text-sink with little output change (Table 2), hence "borrowed, not seen". Sec. 4 then proposes a training-free "re-look": a crop of the window with most attention on the cleaned map is inserted into the reasoning trace at the first high-entropy step. Table 4 reports paired gains over 40 (model, benchmark) pairs, mean +4.3 points; cleaned-map selection beats raw-map selection (+4.1 vs +2.9 on 28 pairs).

Critical strengths
  1. A simple, falsifiable, single-direction account of sinks. Tying visual sinks to the text-sink key unifies Sec. 3.2 (InternVL3-8B with a Qwen2.5 LM has a 2% sink share versus 24% for InternVL3.5-8B with Qwen3 and the same encoder) and Sec. 3.3 (centered-cosine minimum AUC 99.0, against minimum AUCs of 8.0-49.0 for the prior-work norm features, Table 3). The breadth (ten models, four families, HR-Bench replication in Tables 11-12) is unusual, and the family-specific failure of prior detectors is useful to anyone reading attention maps.
  2. A paired downstream evaluation with a raw-map control and reported discordant counts. Shared seeds make traces identical up to the trigger (Sec. 5.1). App. Table 15 shows the cleaned map beats the raw map significantly in 7 of 28 pairs and never loses significantly, and the V* crop-hit rate is 0.90 vs 0.22 for Qwen3-VL-8B. Raw and cleaned crops are similar on sink-free models, consistent with cleaning mattering where sinks exist.
Critical weaknesses
  1. Most of the +4.3 is not attributable to sink cleaning, and the key controls are absent. The +4.3 is the unweighted mean of cleaned gains over four benchmarks (6.4/2.8/5.9/2.1, Table 4); the abstract calls it a gain on "fine-detail" benchmarks, yet the V*/HR-Bench mean is 5.0 and the 4.3 includes MME-RealWorld-Lite (Sec. 5.2). With a raw-map crop the re-look already gains +2.9, so roughly 70% of the gain needs no cleaning. App. C states "We do not report an always-trigger or random-step control", and Table 14 shows the trigger fires on more than 90% of questions for six of ten models, so it is close to an unconditional re-look there. I found no random- or center-window control, no re-insertion of the full image, no sensitivity analysis for H0 = 1 nat or c = 32, and Table 4 compares only against the base run and raw-map selection, not the zoom methods named in Sec. 2.2. Extra high-resolution pixels may thus explain much of the gain. Fixable.
  2. Selection and statistics are thin, and "not seen" is narrower than the title. Each pair is one sampled run (seed = question index, App. C); I found no multiplicity correction across the 40 + 28 McNemar tests and no confidence intervals, and V* has only 191 questions. Selector layers were chosen using 30 V* images (Qwen3-VL-8B) and 60 V* images (Gemma-4), so Gemma's V* column is "not fully held out" (App. B). The "not seen" evidence covers four full-attention models (Table 2: accuracy shift at most three points, KL about 1e-3; 50 HR images in Table 12) and a 60-question, two-model trace check (App. A.10). The all-layer edit costs Qwen3-VL-2B 7 points on V* (.68 to .61, Table 2) and 13-14 on HR-Bench (Table 12); this does not enter Table 4 since the edit is applied only in the selector prefill, and Gemma-4 is excluded because it reads its sinks (KL up to 0.53). The Table 7 interventions use 20 V* images and four models, with random-direction and random-key controls, though "sufficiency" largely follows from the attention formula. Partly fixable; the title should state the scope.
Questions for authors
  1. (New experiment) Can you report the re-look with a random window, a center window, and the full original image re-inserted with the same text, for the sink-bearing models? How much of the +4.3 survives?
  2. (New experiment) Can you add the always-trigger and random-step controls and an H0/c sweep, given the trigger already fires on over 90% of questions for six models (Table 14)?
  3. (Clarification) Can you give bootstrap CIs for the benchmark means in Table 4 and results under several sampling seeds, and state how many of the 7 raw-vs-cleaned significant pairs survive multiplicity correction?
  4. (Clarification) How does the re-look compare with ZoomEye or the attention-crop method of Zhang et al. 2025a on the same models?
  5. (Future work) Does the cleaned map help other attention readers (pruning, hallucination mitigation, RL grounding signals)?
Reviewer_nUTzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies visual attention sinks through the alignment between image-token keys and the text-sink key. It exploits this property to clean attention maps through targeted key edits and uses the cleaned maps to guide entropy-triggered crop-and-relook. Experiments across ten VLMs and four benchmarks report an average accuracy gain of 4.3 percentage points.

Critical strengths

1)The paper provides a useful explanation of why high attention need not indicate meaningful visual contribution, supported by key interventions and measurements of value contributions in the evaluated sink-bearing full-attention models.

2)The study covers ten models across four families and shows that sink contamination can impair crop selection, establishing practical relevance to methods that rely on attention maps.

3)The proposed method is clearly described and easy to follow, with a straightforward progression from attention-map cleaning to crop selection and entropy-triggered re-look.

Critical weaknesses

1)The advantage over model-directed crop-tool use is not empirically established. Although related visual-tool methods are discussed, the experiments do not directly compare the proposed heuristic pipeline with a model deciding when and where to invoke a crop tool.

2)Compatibility with FlashAttention is insufficiently addressed. The selector requires explicit attention weights, which are not ordinarily returned by FlashAttention. The paper does not explain or evaluate how these weights can be extracted while retaining efficient attention kernels, leaving the practical integration and associated overhead unclear.

3)A substantial portion of the improvement comes from re-look, while the additional benefit of sink cleaning is model-dependent. Across the 28 comparable pairs, raw-map re-look gains 2.9 points on average, versus 4.1 points with cleaning. Cleaning can also reduce gains: for Qwen3-VL-2B, raw versus cleaned gains are +3.1 versus +1.6 on V*, and +4.1 versus +3.1 on HR-8K. Although this model is classified as sink-free, these results limit the case for applying cleaning uniformly across models.

4)The computational cost of the complete pipeline is not adequately evaluated. Every question requires an additional selector prefill, and triggered questions incur another image encoding and additional context tokens. Without end-to-end latency, peak GPU memory, and throughput comparisons, the practical cost of the accuracy improvements is difficult to assess.

Questions for authors

see weakness

2 comments

I appreciate the authors’ response, which addresses some of my concerns. However, two key concerns remain: (1) a substantial portion of the improvement comes from re-look rather than the proposed “sink clean”, and (2) the method is highly model-dependent, limiting its generalizability. I therefore maintain my original score.

We thank the reviewer for carefully reading our response and for the follow-up comments. We are glad that some of the concerns are addressed, and we appreciate the chance to clarify the two that remain. We will answer them one by one below.

1

The goal of the paper is to propose how to remove the sinks. The re-look is a simple method that uses the cleaned attention to get a better result. As the paper shows, the visual sinks do not affect the model's output, so removing them cannot improve accuracy by itself. What it improves is a method that reads the attention, such as our re-look: the content picked from the cleaned map matches the question better than the content picked from the raw map. Therefore we compare the results between raw map and clean map.

2

We use one sink-removal method with one setting for every full-attention model: the detector finds the image keys aligned with the text-sink key, and the edit moves them to the median image key at the layers at or above L/4. On a model without sinks the detector finds almost nothing and the edit is practically a zero operation. We re-ran this single rule on seven models.

Share of image attention held by the largest image token at the peak layer, before → after the edit. Cells are V* / HR-4K / HR-8K, 100 questions each.

ModelSinksShare, before → afterAccuracy changeAnswer agreement
Qwen3-VL-8Byes.33 → .03 / .32 → .01 / .32 → .01.00 / −.01 / .00.97 / .98 / 1.00
Qwen3-VL-4Byes.30 → .03 / .26 → .02 / .28 → .02−.01 / −.01 / .00.99 / .98 / 1.00
InternVL3.5-8Byes.42 → .03 / .39 → .02 / .42 → .02.00 / −.01 / +.01.99 / .98 / .99
Pixtral-12Byes.70 → .02 / .70 → .01 / .73 → .02.00 / +.01 / −.01.99 / .96 / .99
Qwen3-VL-2Bnounchanged−.01 / −.01 / −.02.98 / .97 / .96
InternVL3.5-2Bnounchanged.00 / −.01 / −.01.98 / .95 / .99
InternVL3-8Bnounchanged.00 / .00 / .001.00 / 1.00 / 1.00

The same rule removes the sinks at the edited layers on all four models that have them, including Pixtral-12B, whose LLM is Mistral and not Qwen3, and it leaves the attention of the three sink-free models unchanged. Accuracy changes by at most .02 on all 21 sets. So the method does not need to know in advance whether a model has sinks. The only exception is Gemma-4, as the paper describes: its sliding-window layers cannot reach the text sink, so a few image tokens stand in for it and the model reads them.

We will add the split table in appendix and state the single rule in §4.1.

Reviewer_LeWAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the visual attention sink in VLMs and its relation with a text-sink of language model. Across ten VLMs from four family, authors show that a visual sink is an image token aligned to the text-sink, and they propose the centered key cosine as detector. Moving the aligned key component to image median return the sink attention to the text-sink and leave the model output almost not changed, while the attention map become more accurate. Based on this cleaned map, authors propose training-free, entropy-triggered re-look which crop the most attended region and insert it to the reasoning trace. The method improve accuracy in 39 of 40 (model, benchmark) pair.

Critical strengths
  • Clear and easy-to-follow writing: The paper builds its argument step by step, from which models have sinks to what a sink is and what happens when it is removed. Figures 1 and 2 make the key-direction signature easy to grasp.
  • Interesting finding: Visual sinks borrow the attention of the text-sink, and the model does not read them. This reverses the prior view that treats sink attention as misplaced image attention to be redistributed. It seems other readers can get some insights from this paper.
  • Extensive experiments that verify the claims: The claims are tested on ten VLMs from four families with causal interventions and controls, including masking, redistribution, and value-contribution analyses. The appendix also reports unfavorable cases such as Gemma-4 and the 2B models.
Critical weaknesses
  • No comparison with competing methods or simple controls: The paper cites ViCrop, ZoomEye, Look Twice, and uncertainty-guided lookback but compares against none of them. Random-window and always-trigger controls are also missing, which leaves the source of the +4.3-point gain unclear.
  • Inconsistent behavior across models: The two properties in the title hold in four full-attention models, three of which share the Qwen3 backbone. Gemma-4 skips the edit, Pixtral-12B needs a median target, and the 2B models need a depth restriction.
  • Understated and unmeasured cost: The abstract calls the added computation negligible, but the method runs one extra full-prompt prefill per question (section 4.2). The paper reports no latency, FLOPs, or memory.
  • Narrow evidence for the causal and origin claims: The necessary-and-sufficient claim rests on four models, one layer, and 20 images (Table 7). The language-model origin claim rests on a single model pair that also differs in training recipe.
Questions for authors
  • Are the keys used in the centered cosine and the key edit taken before or after RoPE?
  • Which layer does Figure 3 show, and could the crop hit rates in Table 15 be cited alongside it?
  • Could the authors share any insight or preliminary evidence on using the cleaned map as an annotation-free grounding signal in RL pipelines?
  • Would the authors consider separating Section 3.1 as Preliminaries or presenting Section 4 as an application of the findings in Section 3?
  • Please read the above weakness section and I'd like to hear from the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.