acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Multimodal Pre-training for Vision-Language Alignment in MLLMs

Abstract

Vision-language alignment is a key prerequisite for MLLMs to perceive and understand the visual world. Although existing MLLMs typically employ a multimodal pre-training stage for building alignment, we find that the fine-grained, patch-level cross-modal correspondence remains insufficiently established. As text-to-image (T2I) attention plays a critical role in building cross-modal correspondence, we examine it on a pre-trained model and find that T2I attention rapidly shifts from broadly distributing over the image to concentrating on a small subset of textually irrelevant visual tokens, termed visual attention sinks, rather than attending to text-relevant visual evidence. Through further analyses of visual sinks' attention flow and training dynamics, we observe that they emerge in the early loss-reduction phase and persistently hinder the projector from learning fine-grained cross-modal correspondence. To address this issue, we propose Visual Sink-aware Alignment (ViSAlign), a simple and effective method that temporarily blocks the visual sinks' attention flow during multimodal pre-training. By preventing visual sinks from dominating the attention, it encourages the model to capture text-relevant visual evidence, thus learning a finer-grained cross-modal correspondence. Extensive experiments across diverse multimodal settings demonstrate that ViSAlign improves projector learning, strengthens vision-language alignment, and yields broad improvements over a variety of downstream tasks.

Primary area:
interpretability and explainable AI
Keywords:
Vision-language alignment, MLLMs, Visual attention sinks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NbqxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates visual attention sinks during multimodal pre-training, where the vision encoder and language model are frozen and only the projector is trained. Through layer-wise attention analysis, activation-based identification, attention knockout interventions, and checkpoint tracking, the authors argue that sinks reduce the model’s reliance on fine-grained visual evidence. They propose ViSAlign, which adaptively masks text-to-visual attention toward top-attended visual tokens when their attention share exceeds a threshold. The intervention is applied only during pre-training, with standard attention restored for instruction tuning and inference. Validation includes captioning with four language-model backbones, attention-based localization on RefCOCO-family datasets, downstream multimodal benchmarks, and masking-strategy and architecture ablations.

Critical strengths

1. A valuable focus on the formation of multimodal alignment. The paper examines sinks during projector pre-training, extending the discussion beyond the inference behavior of instruction-tuned models. Combining attention analysis, activation-based identification, interventions, and training dynamics provides a useful empirical perspective on a potentially consequential alignment bottleneck.

2. Clear pre-training improvements with no additional inference module. Captioning metrics improve across all four evaluated language-model backbones, and attention-based localization improves across all eight RefCOCO-family splits. The method also produces improvements on several downstream tasks and is evaluated with alternative encoders and projectors. These results support its practical promise.

Critical weaknesses

1.High attention does not necessarily identify harmful sinks, and benefits may diminish with model strength. Equations (7)–(9) mask top-kk visual tokens based on attention concentration without checking their semantic relevance. Correct attention focused on the target object could therefore also trigger masking. Appendix B.4 provides useful overlap analysis with activation-defined sinks, but only on a completed baseline model; it does not establish that this correspondence holds during early training, after ViSAlign changes the attention distribution, or across backbones.

Moreover, Table 1 shows substantially smaller gains for Vicuna-v1.5-13B than for Vicuna-v1.5-7B: CIDEr improves by only 0.028 (0.803→0.831), compared with 0.243 (0.493→0.736). This suggests diminishing benefits as baseline capability increases and raises the question of whether concentration-based masking remains an effective strategy for stronger VLMs.

2.The intervention results do not distinguish improved visual evidence use from increased fragility. Masking sinks may remove useful information pathways, forcing the model to rely more heavily on the remaining target-region evidence. Perturbing that region could then cause a larger loss increase even if masking has worsened unperturbed prediction quality. Thus, a positive CFC_F alone does not establish that sinks impair effective use of local evidence; prediction quality before and after masking should also be reported.

3.Table 2 may overstate the evidence for fine-grained alignment. Its metric counts a hit if any top-5 attention location falls within the ground-truth box. By suppressing attention concentration, ViSAlign may spread these locations more broadly and increase the chance of a hit without improving semantic alignment. Functional grounding evaluations are needed to distinguish these explanations.

Questions for authors
  1. Sink identification and model scaling: How do you distinguish harmful sinks from correctly concentrated attention on relevant objects during training? Given the substantially smaller gains for Vicuna-13B than Vicuna-7B, could you provide a controlled size sweep within one model family—for example, Qwen2.5-3B, 7B, and 14B—to determine whether benefits diminish with model strength? A comparable sweep across Qwen3 sizes would also be informative.

  2. Intervention interpretation: Does masking sinks improve or worsen unperturbed prediction quality? Could you report the losses before and after masking alongside CFC_F, to distinguish improved use of local evidence from increased vulnerability after removing useful information pathways?

  3. Fine-grained alignment: Can you show that Table 2’s gains reflect improved semantic grounding rather than broader top-5 spatial coverage? Top-1 localization and a functional local-object or attribute-discrimination evaluation would help distinguish these explanations.

9 comments

Thank you very much for your constructive comments.

Due to OpenReview's formatting limitations, we have incorporated all additional experimental figures into Appendices E, F, and G of the revised manuscript. We refer reviewers to the corresponding figures throughout our responses.

Weak 1 & Question 1. Sink Identification and Model Scaling

1.1 Sink Identification During Training Thank you for your insightful question. Following Appendix B.4 as suggested, we save intermediate checkpoints when ViSAlign is applied and monitor the overlap between L/V-sinks and Top-K attention selected by ViSAlign during training. As shown in Figure 10 (Appendix E), ViSAlign can successfully capture L-sinks and V-sinks during training while inevitably containing a few non-sink tokens. However, as we typically select a small value k=6k=6 for the experiments, the influence of this mismatching is minimized. These results have shown that high attention does not necessarily identify harmful sinks, but ViSAlign can effectively capture them during training.

Surprisingly, beyond the coverage, we identify an interesting phenomenon brought by ViSAlign. Previously, as shown in Figure 3(a) in our main text, the V-sinks and L-sinks have distinct activation patterns: V-sinks have relatively uniform activation but together a higher feature norm, while L-sinks have extremely high activation on specific dimensions like 2533 in LLaVA-1.5-7B. However, our experiments have shown that L-sinks and V-sinks are merged when ViSAlign is applied, denoted as L&V-sinks in Figure 10 (Appendix E). We further visualized their activation pattern in Figure 11 (Appendix E), where the patches 471 and 469 are L&V-sink, which have extremely high activation on dimension 2533 meanwhile identified as V-sink in the vision encoder.

This unexpected merging behavior, induced by a simple masking strategy, may offer new insights into the mechanisms underlying visual attention sinks and inspire future approaches to their mitigation and utilization. We hypothesize that this phenomenon is closely related to the underlying factors driving the emergence of L-sinks, which warrants further investigation in future work.

1.2 Effectiveness Might Diminish as Model Scaling To further investigate the generalizability of ViSAlign across different model scales and LLM architectures, we extend our experiments to the Qwen2.5 family, including 1.5B, 7B, and 14B models, and evaluate their performance on the image captioning task.

As shown in the supplementary results, ViSAlign consistently improves CIDEr scores across all three model scales, achieving gains of +0.017, +0.064, and +0.050 for Qwen2.5-1.5B, 7B, and 14B, respectively. Similar improvements are observed in BLEU-1, BLEU-2, and BLEU-3 across all configurations. Although minor fluctuations occur in individual metrics, particularly BLEU-4 and ROUGE-L for Qwen2.5-14B, the overall results demonstrate that ViSAlign remains effective across different model capacities.

ModelViSAlignBLEU_1BLEU_2BLEU_3BLEU_4ROUGE_LCIDEr
Qwen2.5-1.5B-0.6130.3990.2520.1590.3940.549
✓\checkmark0.6270.4090.2580.1620.3920.566
Δ\Delta+0.014+0.010+0.006+0.003-0.002+0.017
Qwen2.5-7B-0.5460.3720.2480.1630.4430.464
✓\checkmark0.5770.3960.2630.1730.4600.528
Δ\Delta+0.031+0.024+0.015+0.010+0.017+0.064
Qwen2.5-14B-0.6230.4400.2990.2050.4870.679
✓\checkmark0.6450.4480.3030.1980.4630.729
Δ\Delta+0.022+0.008+0.004-0.007-0.024+0.050

The smaller captioning improvement observed with Vicuna-13B compared to Vicuna-7B may be attributed to differences in the severity of visual attention sinks between the two models. As illustrated in Figure 13 (Appendix G), LLaVA-1.5-13B exhibits a less pronounced visual attention sink phenomenon than LLaVA-1.5-7B, with less T2I attention on visual sinks, while previous literature [1] has also reported fewer visual sinks in LLaVA-1.5-13B. Consequently, the potential benefits of sink masking may be reduced when visual attention sinks are less prevalent or less pronounced, which could explain the smaller improvement achieved by ViSAlign on Vicuna-13B. A similar trend could be found in Qwen families, as shown in Figure 15 (Appendix G): Qwen2.5-1.5B exhibits the least pronounced visual attention sinks, with the smallest number of visual sinks and less T2I attention. Coincidentally, Qwen2.5-1.5B exhibits the least performance gain on BLEU and CIDEr metrics.

Nevertheless, these results suggest that the effectiveness of ViSAlign is not inherently dependent on model scale; rather, as long as visual attention sinks are present, ViSAlign can effectively mitigate their adverse effects and improve vision-language alignment.

Weak2 & Question 2. Intervention Interpretation Needs Loss Report

Thank you for your valuable suggestion. To comprehensively investigate the mechanisms underlying visual attention sink pathway interventions, we compare the loss before and after pathway masking on a curated dataset derived from RefCOCO and 1,000 samples from the PACO subset of the Describe Anything dataset [2], which extends the original PACO dataset [3] with detailed region-level captions. Both datasets provide bounding box annotations and corresponding localized captions.

The results shown in Figure 12 (Appendix F) demonstrate consistent trends: masking the visual sinks pathway reduces the loss and increases the proposed CFC_F metric. Specifically, random masking with a comparable number of visual sinks produces minor and ignorable loss change within 0.001; however, masking V-sinks and L-sinks reduces captioning losses. These results indicate that masking sinks improves unperturbed prediction quality; thus, the larger loss increase from region perturbation doesn’t come from increased fragility but from improved use of visual evidence. Thanks for this detailed observation and thinking, which indeed improves the logical cautiousness of our work.

Beyond intervention interpretation, an interesting problem is why masking visual sinks improves prediction quality instead of remaining close or becoming vulnerable. A potential explanation for this phenomenon might be that visual sinks help with global semantic modeling [4,5,6]; however, producing a local caption requires the model to suppress visual sinks and distribute attention to specific visual evidence.

Weak 3 & Question 3. Fine-grained Alignment

3.1 Top-1 PiB Acc Thank you for this question and valuable suggestion. As suggested, we test top-1 PiB accuracy, and the results are shown in the table below. The Top-1 PiB Acc of both the baseline and ViSAlign is poor, around 10%-20%, and ViSAlign could achieve improvements, but with performance fluctuations. The reason behind this is that ViSAlign doesn’t eliminate visual attention sinks. Instead, we focus on suppressing the harmful attention sinks of the model to encourage T2I attention localization for semantically rich items in the image. As shown in Figure 1(d), more attention is centralized on the text-related region of flowers, while visual sinks remain. Consequently, when K=1K=1, the results of PiB Acc will be affected by the position of L-sinks, resulting in limited improvements.

MethodRefCOCORefCOCO+RefCOCOg
valtestvaltestvaltest
Baseline13.9114.1712.7014.1014.5814.12
ViSAlign15.1412.7213.2412.8715.2914.67

3.2 Functional Semantic Grounding Analysis To further verify the claimed fine-grained alignment, we conduct both visualization and quantitative analysis. In Figure 5, we select a 6 ×\times 6 patch grid and apply Embed Lens [7] to obtain semantic grounding. ViSAlign achieves superior alignment where it correctly correspond "who is also a drummer" and "cheese" in probed images. Besides, we further conduct a controlled analysis based on the PACO subset of the Describe-Anything dataset with image, object-level bounding box, object attributions, and object name. Then, we apply Embed Lens within the bounding box patches and test whether it matches the object attributions and names. The results are shown in the table below. ViSAlign achieves superior performance on fine-grained object-level semantic probing, further supporting our claims that ViSAlign improves vision-language alignment.

MethodAcc
Baseline30.29
ViSAlign32.51
Δ\Delta+2.22

[1] "SMART: Sink-based Modality-Aware Redistribution of Transformer Attention", EMNLP 2026

[2] "Describe Anything: Detailed Localized Image and Video Captioning", arXiv 2504.16072

[3] "PACO: Parts and Attributes of Common Objects", CVPR 2023

[4] "See What You Are Told: Visual Attention Sink in Large Multimodal Models", ICLR 2025

[5] "To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models", ICLR 2026

[6] "When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models", ECCV 2026

[7] "What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models", CVPR 2026

We would greatly appreciate any further concerns or suggestions you may have regarding our work. We also welcome continued technical discussions and hope that our clarifications and additional evidence can help address your concerns and earn your support.

Reviewer_Z9KCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MLLMs can currently generate relatively accurate image descriptions Visual attention sinks, but have not sufficiently established fine-grained correspondences between local image regions and textual semantics. The authors believe the reason is the Visual Attention Sinks generated during multimodal pre-training: irrelevant tokens absorb too much attention, making it difficult for the model to fully utilize truly relevant visual evidence. The authors propose the VisAlign method, which dynamically identifies Attention Sinks during the pre-training stage and blocks text attention to these Tokens, encouraging the model to better learn fine-grained vision-language correspondences.

Critical strengths
  1. Through relatively detailed experiments, the internal model mechanism by which Attention Sinks arise during pre-training is investigated, and the visual and language model origins of the Sinks and identification methods are analyzed.
  2. Proposes an adaptive ViSAlign method that, during visual pre-training, dynamically identifies and excludes sinks that occupy too much attention based on attention proportion, improving model performance on fine-grained visual tasks.
  3. The experiments are detailed and complete, including both analysis of the pre-training mechanism for Visual Attention Sinks and validation of method performance based on downstream tasks, and ablation experiments further demonstrate the effectiveness of the method.
Critical weaknesses
  1. Regarding the authors' core assumption: Does high Attention necessarily equal harmful Sinks? The authors verify in experiments the correlation between high-Attention Tokens and Visual Sinks, but do not sufficiently verify that the Attention masked by ViSAlign is all truly harmful Sinks.
  2. Some hyperparameters are insufficiently reported; the specific values of parameters k and /alpha in the actual experiments are unknown; and how the parameters are specifically set for different Backbone models; these all need to be reported in more detail in the paper.
Questions for authors
  1. How effectively can Top-K Attention identify harmful Sinks? Have the authors further counted and analyzed what proportion of the attention filtered by TopK is actually harmful sinks?
  2. How were the hyperparameters used in the experiments selected (e.g., k and alpha), and have the authors considered adding experiments on hyperparameter sensitivity?
  3. During pre-training, how much computation and memory overhead do the dynamic Sink identification and Attention Masking methods add?
2 comments

Thanks very much for your valuable suggestions for our work.

Due to OpenReview's formatting limitations, we have incorporated all additional experimental figures into Appendices E, F, and G of the revised manuscript. We refer reviewers to the corresponding figures throughout our responses.

Question 1. Identify Visual Sinks via Top-K Attention during Training

Thank you for your insightful question. Following Appendix B.4 as suggested, we save intermediate checkpoints when ViSAlign is applied and monitor the overlap between L/V-sinks and Top-K attention selected by ViSAlign during training. We add the experiment results in Figure 10 (Appendix E). ViSAlign can successfully capture L-sinks and V-sinks during training while inevitably containing a few non-sink tokens. However, as we typically select a small value k=6k=6 for the experiments, the influence of this mismatching is minimized. These results have shown that high attention does not necessarily identify harmful sinks, but ViSAlign can effectively capture them during training.

Surprisingly, beyond the coverage, we identify an interesting phenomenon brought by ViSAlign. Previously, as shown in Figure 3(a) in our main text, the V-sinks and L-sinks have distinct activation patterns: V-sinks have relatively uniform activation but together a higher feature norm, while L-sinks have extremely high activation on specific dimensions like 2533 in LLaVA-1.5-7B. However, our experiments have shown that L-sinks and V-sinks are merged when ViSAlign is applied, denoted as L&V-sinks in Figure 10 (Appendix E). We further visualized their activation pattern in Figure 11 (Appendix E), where the patches 471 and 469 are L&V-sink, which have extremely high activation on dimension 2533 meanwhile identified as V-sink in the vision encoder.

This unexpected merging behavior, induced by a simple masking strategy, may offer new insights into the mechanisms underlying visual attention sinks and inspire future approaches to their mitigation and utilization. We hypothesize that this phenomenon is closely related to the underlying factors driving the emergence of L-sinks, which warrants further investigation in future work.

Question 2. Hyperparameter Selection and Sensitivity Analysis

Thank you for your suggestion. Our parameter selection is guided by empirical observations across different model architectures, particularly the proportion of text-to-image (T2I) attention concentrated on visual sinks and the coverage of visual sinks achieved by the top-kk selection in Figures 13-17 (Appendix G). These observations provide an empirical basis for balancing effective sink suppression against the preservation of useful visual information.

In our main experiments, we typically set k=6k=6 and α=0.3\alpha=0.3 to capture most harmful visual sinks that attract disproportionately high attention, while minimizing the risk of masking potentially informative visual tokens. A smaller kk may leave some visual sinks unmasked; however, these sinks generally receive less attention and therefore have a relatively limited impact on the pre-training stage. As shown in Figure 18 (Appendix G), typically the top 2-3 visual sinks receive most of the attention while others shrink significantly for Vicuna-7B, Vicuna-13B, and Phi3-3.8B. For Qwen2.5-7B-Instruct, we retain these settings without further modification, although it contains more visual sinks and increasing kk won't mask useful visual tokens. In conclusion, our hyperparameter selection is well-grounded in the analysis of visual attention sinks and T2I attention patterns across different backbones. Our observations further suggest that the reported configuration can generalize across different model architectures.

We plan to add a sensitivity analysis on the hyperparameters kk and α\alpha on the baseline model LLaVA-1.5-7B. The full training process would include pre-training on the 558K subset of the LAION-CC-SBU dataset and fine-tuning 7B parameters on 665K instruction-tuning data, leading to approximately 120 GPU-hours for a single experiment. However, due to limited computational resources, it’s difficult for us to complete all experiments during the rebuttal period.

Question 3. Computation and Memory Overhead of ViSAlign

Thank you for your question. We conducted additional profiling experiments to quantify the training overhead introduced by ViSAlign. Specifically, we measured the average optimizer-step latency over 100 steps following 10 warmup steps, as well as the peak GPU memory allocation. The results shown in the table below reveal that ViSAlign introduces a moderate training-time overhead of 12.58%, while increasing peak allocated GPU memory by only 0.16%. These results further demonstrate that ViSAlign is a simple yet effective method for improving vision-language alignment, with only modest computational overhead and negligible additional memory consumption.

MetricBaselineViSAlignΔ\Delta
Step latency(s)12.66214.255+12.58%
Peak Memory Allocated(GiB)26.72526.768+0.16%

We would greatly appreciate any further questions or suggestions you may have regarding our work.

Reviewer_ZVEeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Summary: In LLaVA-style MLLMs, pre-training trains only the projector (encoder and LLM frozen). This paper studies text-to-image attention in pre-trained, LLaVA-1.5. It finds T2I attention starts near-uniform for two layers then collapses onto textually irrelevant visual attention sinks. It runs only during pre-training: per layer, identify top-k visual tokens by T2I attention; when sinks absorb more than threshold α of total T2I attention, block caption tokens from attending to them.

Contributions: (1) Diagnostic of sinks during projector pre-training. LLM-emerged L-sinks (massive activations) dominate and cause the layer-2 entropy collapse; ViT-propagated V-sinks (high L₂ norms) prominent deep; knockout + perturbation shows both causally suppress reliance on in-box evidence; sinks form during the early rapid loss-reduction phase. (2) ViSAlign — sink ID directly from T2I attention, backbone-agnostic. (3) Better alignment (COCO captioning, RefCOCO pointing-in-box) and downstream gains after instruction tuning across four LLM backbones.

Validation. Pre-training-stage: COCO2014 captioning for Vicuna-7B/13B, Qwen2.5-7B, Phi-3-3.8B — every metric improves for every backbone (Vicuna-7B CIDEr 0.493→0.736; Phi-3 0.310→0.583); pointing-in-box +3.7 to +9.1 on all eight RefCOCO splits. After identical instruction tuning: 7 benchmarks × 4 backbones vs inference-time methods (VAR, SinkF) and trainable LSG.

Critical strengths

The diagnosis is the real contribution and it's causally supported, not just visualized. The sink knockout increases sensitivity to evidential-region perturbation while random knockout doesn't. The sinks emerge exactly as loss first collapses. It converts "sinks exist" into "sinks form during pre-training and actively prevent fine-grained correspondence".

Critical weaknesses

The key hyperparameters k (sink set size) and α (dominance threshold) unreported and unablated. It is hard to reproduce or judge fragility; the entire section of analysis comes from one pre-trained LLaVA-1.5-7B on 1,000 samples, and the top-k proxy only partially coincides with activation-based sink identity.

Questions for authors

N/A

3 comments

We greatly appreciate your endorsement and valuable suggestions.

Due to OpenReview's formatting limitations, we have incorporated all additional experimental figures into Appendices E, F, and G of the revised manuscript. We refer reviewers to the corresponding figures throughout our responses.

Question 1. Hyperparameter Selection and Sensitivity Analysis

Thank you for your suggestion. Our parameter selection is guided by empirical observations across different model architectures, particularly the proportion of text-to-image (T2I) attention concentrated on visual sinks and the coverage of visual sinks achieved by the top-kk selection in Figures 13-17 (Appendix G). These observations provide an empirical basis for balancing effective sink suppression against the preservation of useful visual information.

In our main experiments, we typically set k=6k=6 and α=0.3\alpha=0.3 to capture most harmful visual sinks that attract disproportionately high attention, while minimizing the risk of masking potentially informative visual tokens. A smaller kk may leave some visual sinks unmasked; however, these sinks generally receive less attention and therefore have a relatively limited impact on the pre-training stage. As shown in Figure 18 (Appendix G), typically the top 2-3 visual sinks receive most of the attention while others shrink significantly for Vicuna-7B, Vicuna-13B, and Phi3-3.8B. For Qwen2.5-7B-Instruct, we retain these settings without further modification, although it contains more visual sinks and increasing kk won't mask useful visual tokens. In conclusion, our hyperparameter selection is well-grounded in the analysis of visual attention sinks and T2I attention patterns across different backbones. Our observations further suggest that the reported configuration can generalize across different model architectures.

We plan to add a sensitivity analysis on the hyperparameters kk and α\alpha on the baseline model LLaVA-1.5-7B. However, the full training process would include multimodal pre-training on the 558K subset of the LAION-CC-SBU dataset and visual instruction tuning of 7B parameters on 665K instruction-tuning data, leading to approximately 120 GPU-hours for a single experiment. Due to limited computational resources, it’s difficult for us to complete all experiments during the rebuttal period.

Question 2. Analysis section only from LLaVA-1.5-7B and 1,000 samples.

We extend attention sinks analysis to pre-trained models using different LLM backbones, including LLaVA-1.5-13B, Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, and Phi3-mini-4k-instruct-3.8B, as shown in Figures 13- 17 (Appendix G). Our observations demonstrate that visual attention sinks, including L-sinks and V-sinks, are widely observed phenomena. In addition to 1,000 samples curated based on RefCOCO, we add analysis and experiments on 5,000 data points in a well-known dataset, the PACO[1] subset of the Describe-Anything dataset [2], which contains bounding box and local caption annotations. In Figure 12 (Appendix F), we add causal sensitivity analysis on this dataset and obtain cohesive results with our crucial claim: Visual sinks harm the model's attention acquisition on visual evidence, undermining vision-language alignment.

[1] "PACO: Parts and Attributes of Common Objects", CVPR 2023

[2] “Describe Anything: Detailed Localized Image and Video Captioning”, arXiv 2504.16072

We look forward to hearing whether you have additional questions or suggestions for our work.

Thank you for the response.

On hyperparameters: I appreciate the reported settings and the empirical rationale. But the sensitivity analysis is still missing (I understand it is due to compute constraint). I would suggest that the final version state the k and α values and their selection rationale in the main text, and that the sensitivity analysis be completed for the camera-ready if possible (or at least note it as a limitation if compute constraints continue to be an issue).

On the analysis breadth: the extension to more backbones and the 5,000-sample PACO analysis address my concern.

Overall, I would maintain my rating of 3.

We sincerely appreciate your positive assessment and encouraging comments on our work.

We're happy to know that your concerns about the analysis have been addressed, and we will follow your suggestion on hyperparameters to conduct experiments and revise our work. Thank you very much for your understanding of the compute constraints and your valuable suggestions in reviewing our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on the Rebuttal Discussion

Dear reviewers,

As the rebuttal discussion period nears its end, we would appreciate your feedback on whether our responses have successfully addressed your concerns. We would be happy to answer any remaining questions.

Thank you again for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.