- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies "visual attention sinks": image tokens that absorb a large share of in-image attention regardless of the question. Across ten VLMs from four families (Sec. 3.2, Table 1) it reports that (i) the language-model backbone largely decides which models have sinks; (ii) the centered cosine between an image token's key and the text-sink key (Eq. 2) separates sinks with AUC >= 99.0 in all seven sink-bearing models, whereas norm- and logit-based detectors are family-specific (Table 3); (iii) projecting that direction out of sink keys removes sink attention and writing it into random image tokens creates it (Table 7); (iv) a gated key edit (Eq. 3) returns the sinks' attention to the text-sink with little output change (Table 2), hence "borrowed, not seen". Sec. 4 then proposes a training-free "re-look": a crop of the window with most attention on the cleaned map is inserted into the reasoning trace at the first high-entropy step. Table 4 reports paired gains over 40 (model, benchmark) pairs, mean +4.3 points; cleaned-map selection beats raw-map selection (+4.1 vs +2.9 on 28 pairs).
Critical strengths
- A simple, falsifiable, single-direction account of sinks. Tying visual sinks to the text-sink key unifies Sec. 3.2 (InternVL3-8B with a Qwen2.5 LM has a 2% sink share versus 24% for InternVL3.5-8B with Qwen3 and the same encoder) and Sec. 3.3 (centered-cosine minimum AUC 99.0, against minimum AUCs of 8.0-49.0 for the prior-work norm features, Table 3). The breadth (ten models, four families, HR-Bench replication in Tables 11-12) is unusual, and the family-specific failure of prior detectors is useful to anyone reading attention maps.
- A paired downstream evaluation with a raw-map control and reported discordant counts. Shared seeds make traces identical up to the trigger (Sec. 5.1). App. Table 15 shows the cleaned map beats the raw map significantly in 7 of 28 pairs and never loses significantly, and the V* crop-hit rate is 0.90 vs 0.22 for Qwen3-VL-8B. Raw and cleaned crops are similar on sink-free models, consistent with cleaning mattering where sinks exist.
Critical weaknesses
- Most of the +4.3 is not attributable to sink cleaning, and the key controls are absent. The +4.3 is the unweighted mean of cleaned gains over four benchmarks (6.4/2.8/5.9/2.1, Table 4); the abstract calls it a gain on "fine-detail" benchmarks, yet the V*/HR-Bench mean is 5.0 and the 4.3 includes MME-RealWorld-Lite (Sec. 5.2). With a raw-map crop the re-look already gains +2.9, so roughly 70% of the gain needs no cleaning. App. C states "We do not report an always-trigger or random-step control", and Table 14 shows the trigger fires on more than 90% of questions for six of ten models, so it is close to an unconditional re-look there. I found no random- or center-window control, no re-insertion of the full image, no sensitivity analysis for H0 = 1 nat or c = 32, and Table 4 compares only against the base run and raw-map selection, not the zoom methods named in Sec. 2.2. Extra high-resolution pixels may thus explain much of the gain. Fixable.
- Selection and statistics are thin, and "not seen" is narrower than the title. Each pair is one sampled run (seed = question index, App. C); I found no multiplicity correction across the 40 + 28 McNemar tests and no confidence intervals, and V* has only 191 questions. Selector layers were chosen using 30 V* images (Qwen3-VL-8B) and 60 V* images (Gemma-4), so Gemma's V* column is "not fully held out" (App. B). The "not seen" evidence covers four full-attention models (Table 2: accuracy shift at most three points, KL about 1e-3; 50 HR images in Table 12) and a 60-question, two-model trace check (App. A.10). The all-layer edit costs Qwen3-VL-2B 7 points on V* (.68 to .61, Table 2) and 13-14 on HR-Bench (Table 12); this does not enter Table 4 since the edit is applied only in the selector prefill, and Gemma-4 is excluded because it reads its sinks (KL up to 0.53). The Table 7 interventions use 20 V* images and four models, with random-direction and random-key controls, though "sufficiency" largely follows from the attention formula. Partly fixable; the title should state the scope.
Questions for authors
- (New experiment) Can you report the re-look with a random window, a center window, and the full original image re-inserted with the same text, for the sink-bearing models? How much of the +4.3 survives?
- (New experiment) Can you add the always-trigger and random-step controls and an H0/c sweep, given the trigger already fires on over 90% of questions for six models (Table 14)?
- (Clarification) Can you give bootstrap CIs for the benchmark means in Table 4 and results under several sampling seeds, and state how many of the 7 raw-vs-cleaned significant pairs survive multiplicity correction?
- (Clarification) How does the re-look compare with ZoomEye or the attention-crop method of Zhang et al. 2025a on the same models?
- (Future work) Does the cleaned map help other attention readers (pruning, hallucination mitigation, RL grounding signals)?
I appreciate the authors’ response, which addresses some of my concerns. However, two key concerns remain: (1) a substantial portion of the improvement comes from re-look rather than the proposed “sink clean”, and (2) the method is highly model-dependent, limiting its generalizability. I therefore maintain my original score.