- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates visual attention sinks during multimodal pre-training, where the vision encoder and language model are frozen and only the projector is trained. Through layer-wise attention analysis, activation-based identification, attention knockout interventions, and checkpoint tracking, the authors argue that sinks reduce the model’s reliance on fine-grained visual evidence. They propose ViSAlign, which adaptively masks text-to-visual attention toward top-attended visual tokens when their attention share exceeds a threshold. The intervention is applied only during pre-training, with standard attention restored for instruction tuning and inference. Validation includes captioning with four language-model backbones, attention-based localization on RefCOCO-family datasets, downstream multimodal benchmarks, and masking-strategy and architecture ablations.
Critical strengths
1. A valuable focus on the formation of multimodal alignment. The paper examines sinks during projector pre-training, extending the discussion beyond the inference behavior of instruction-tuned models. Combining attention analysis, activation-based identification, interventions, and training dynamics provides a useful empirical perspective on a potentially consequential alignment bottleneck.
2. Clear pre-training improvements with no additional inference module. Captioning metrics improve across all four evaluated language-model backbones, and attention-based localization improves across all eight RefCOCO-family splits. The method also produces improvements on several downstream tasks and is evaluated with alternative encoders and projectors. These results support its practical promise.
Critical weaknesses
1.High attention does not necessarily identify harmful sinks, and benefits may diminish with model strength. Equations (7)–(9) mask top- visual tokens based on attention concentration without checking their semantic relevance. Correct attention focused on the target object could therefore also trigger masking. Appendix B.4 provides useful overlap analysis with activation-defined sinks, but only on a completed baseline model; it does not establish that this correspondence holds during early training, after ViSAlign changes the attention distribution, or across backbones.
Moreover, Table 1 shows substantially smaller gains for Vicuna-v1.5-13B than for Vicuna-v1.5-7B: CIDEr improves by only 0.028 (0.803→0.831), compared with 0.243 (0.493→0.736). This suggests diminishing benefits as baseline capability increases and raises the question of whether concentration-based masking remains an effective strategy for stronger VLMs.
2.The intervention results do not distinguish improved visual evidence use from increased fragility. Masking sinks may remove useful information pathways, forcing the model to rely more heavily on the remaining target-region evidence. Perturbing that region could then cause a larger loss increase even if masking has worsened unperturbed prediction quality. Thus, a positive alone does not establish that sinks impair effective use of local evidence; prediction quality before and after masking should also be reported.
3.Table 2 may overstate the evidence for fine-grained alignment. Its metric counts a hit if any top-5 attention location falls within the ground-truth box. By suppressing attention concentration, ViSAlign may spread these locations more broadly and increase the chance of a hit without improving semantic alignment. Functional grounding evaluations are needed to distinguish these explanations.
Questions for authors
-
Sink identification and model scaling: How do you distinguish harmful sinks from correctly concentrated attention on relevant objects during training? Given the substantially smaller gains for Vicuna-13B than Vicuna-7B, could you provide a controlled size sweep within one model family—for example, Qwen2.5-3B, 7B, and 14B—to determine whether benefits diminish with model strength? A comparable sweep across Qwen3 sizes would also be informative.
-
Intervention interpretation: Does masking sinks improve or worsen unperturbed prediction quality? Could you report the losses before and after masking alongside , to distinguish improved use of local evidence from increased vulnerability after removing useful information pathways?
-
Fine-grained alignment: Can you show that Table 2’s gains reflect improved semantic grounding rather than broader top-5 spatial coverage? Top-1 localization and a functional local-object or attribute-discrimination evaluation would help distinguish these explanations.
9 comments
The smaller captioning improvement observed with Vicuna-13B compared to Vicuna-7B may be attributed to differences in the severity of visual attention sinks between the two models. As illustrated in Figure 13 (Appendix G), LLaVA-1.5-13B exhibits a less pronounced visual attention sink phenomenon than LLaVA-1.5-7B, with less T2I attention on visual sinks, while previous literature [1] has also reported fewer visual sinks in LLaVA-1.5-13B. Consequently, the potential benefits of sink masking may be reduced when visual attention sinks are less prevalent or less pronounced, which could explain the smaller improvement achieved by ViSAlign on Vicuna-13B. A similar trend could be found in Qwen families, as shown in Figure 15 (Appendix G): Qwen2.5-1.5B exhibits the least pronounced visual attention sinks, with the smallest number of visual sinks and less T2I attention. Coincidentally, Qwen2.5-1.5B exhibits the least performance gain on BLEU and CIDEr metrics.
Nevertheless, these results suggest that the effectiveness of ViSAlign is not inherently dependent on model scale; rather, as long as visual attention sinks are present, ViSAlign can effectively mitigate their adverse effects and improve vision-language alignment.
Weak2 & Question 2. Intervention Interpretation Needs Loss Report
Thank you for your valuable suggestion. To comprehensively investigate the mechanisms underlying visual attention sink pathway interventions, we compare the loss before and after pathway masking on a curated dataset derived from RefCOCO and 1,000 samples from the PACO subset of the Describe Anything dataset [2], which extends the original PACO dataset [3] with detailed region-level captions. Both datasets provide bounding box annotations and corresponding localized captions.
The results shown in Figure 12 (Appendix F) demonstrate consistent trends: masking the visual sinks pathway reduces the loss and increases the proposed metric. Specifically, random masking with a comparable number of visual sinks produces minor and ignorable loss change within 0.001; however, masking V-sinks and L-sinks reduces captioning losses. These results indicate that masking sinks improves unperturbed prediction quality; thus, the larger loss increase from region perturbation doesn’t come from increased fragility but from improved use of visual evidence. Thanks for this detailed observation and thinking, which indeed improves the logical cautiousness of our work.
Beyond intervention interpretation, an interesting problem is why masking visual sinks improves prediction quality instead of remaining close or becoming vulnerable. A potential explanation for this phenomenon might be that visual sinks help with global semantic modeling [4,5,6]; however, producing a local caption requires the model to suppress visual sinks and distribute attention to specific visual evidence.
Weak 3 & Question 3. Fine-grained Alignment
3.1 Top-1 PiB Acc Thank you for this question and valuable suggestion. As suggested, we test top-1 PiB accuracy, and the results are shown in the table below. The Top-1 PiB Acc of both the baseline and ViSAlign is poor, around 10%-20%, and ViSAlign could achieve improvements, but with performance fluctuations. The reason behind this is that ViSAlign doesn’t eliminate visual attention sinks. Instead, we focus on suppressing the harmful attention sinks of the model to encourage T2I attention localization for semantically rich items in the image. As shown in Figure 1(d), more attention is centralized on the text-related region of flowers, while visual sinks remain. Consequently, when , the results of PiB Acc will be affected by the position of L-sinks, resulting in limited improvements.
| Method | RefCOCO | RefCOCO+ | RefCOCOg | |||
|---|---|---|---|---|---|---|
| val | test | val | test | val | test | |
| Baseline | 13.91 | 14.17 | 12.70 | 14.10 | 14.58 | 14.12 |
| ViSAlign | 15.14 | 12.72 | 13.24 | 12.87 | 15.29 | 14.67 |
3.2 Functional Semantic Grounding Analysis To further verify the claimed fine-grained alignment, we conduct both visualization and quantitative analysis. In Figure 5, we select a 6 6 patch grid and apply Embed Lens [7] to obtain semantic grounding. ViSAlign achieves superior alignment where it correctly correspond "who is also a drummer" and "cheese" in probed images. Besides, we further conduct a controlled analysis based on the PACO subset of the Describe-Anything dataset with image, object-level bounding box, object attributions, and object name. Then, we apply Embed Lens within the bounding box patches and test whether it matches the object attributions and names. The results are shown in the table below. ViSAlign achieves superior performance on fine-grained object-level semantic probing, further supporting our claims that ViSAlign improves vision-language alignment.
| Method | Acc |
|---|---|
| Baseline | 30.29 |
| ViSAlign | 32.51 |
| +2.22 |
[1] "SMART: Sink-based Modality-Aware Redistribution of Transformer Attention", EMNLP 2026
[2] "Describe Anything: Detailed Localized Image and Video Captioning", arXiv 2504.16072
[3] "PACO: Parts and Attributes of Common Objects", CVPR 2023
[4] "See What You Are Told: Visual Attention Sink in Large Multimodal Models", ICLR 2025
[5] "To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models", ICLR 2026
[6] "When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models", ECCV 2026
[7] "What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models", CVPR 2026
We would greatly appreciate any further concerns or suggestions you may have regarding our work. We also welcome continued technical discussions and hope that our clarifications and additional evidence can help address your concerns and earn your support.
Thank you very much for your constructive comments.
Due to OpenReview's formatting limitations, we have incorporated all additional experimental figures into Appendices E, F, and G of the revised manuscript. We refer reviewers to the corresponding figures throughout our responses.
Weak 1 & Question 1. Sink Identification and Model Scaling
1.1 Sink Identification During Training Thank you for your insightful question. Following Appendix B.4 as suggested, we save intermediate checkpoints when ViSAlign is applied and monitor the overlap between L/V-sinks and Top-K attention selected by ViSAlign during training. As shown in Figure 10 (Appendix E), ViSAlign can successfully capture L-sinks and V-sinks during training while inevitably containing a few non-sink tokens. However, as we typically select a small value k=6 for the experiments, the influence of this mismatching is minimized. These results have shown that high attention does not necessarily identify harmful sinks, but ViSAlign can effectively capture them during training.
Surprisingly, beyond the coverage, we identify an interesting phenomenon brought by ViSAlign. Previously, as shown in Figure 3(a) in our main text, the V-sinks and L-sinks have distinct activation patterns: V-sinks have relatively uniform activation but together a higher feature norm, while L-sinks have extremely high activation on specific dimensions like 2533 in LLaVA-1.5-7B. However, our experiments have shown that L-sinks and V-sinks are merged when ViSAlign is applied, denoted as L&V-sinks in Figure 10 (Appendix E). We further visualized their activation pattern in Figure 11 (Appendix E), where the patches 471 and 469 are L&V-sink, which have extremely high activation on dimension 2533 meanwhile identified as V-sink in the vision encoder.
This unexpected merging behavior, induced by a simple masking strategy, may offer new insights into the mechanisms underlying visual attention sinks and inspire future approaches to their mitigation and utilization. We hypothesize that this phenomenon is closely related to the underlying factors driving the emergence of L-sinks, which warrants further investigation in future work.
1.2 Effectiveness Might Diminish as Model Scaling To further investigate the generalizability of ViSAlign across different model scales and LLM architectures, we extend our experiments to the Qwen2.5 family, including 1.5B, 7B, and 14B models, and evaluate their performance on the image captioning task.
As shown in the supplementary results, ViSAlign consistently improves CIDEr scores across all three model scales, achieving gains of +0.017, +0.064, and +0.050 for Qwen2.5-1.5B, 7B, and 14B, respectively. Similar improvements are observed in BLEU-1, BLEU-2, and BLEU-3 across all configurations. Although minor fluctuations occur in individual metrics, particularly BLEU-4 and ROUGE-L for Qwen2.5-14B, the overall results demonstrate that ViSAlign remains effective across different model capacities.