- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper incorporates foreground-background discrimination with attention contrastive decoding for image captioning. In particular, it locates sink visual tokens, tokens likely belonging to backgrounds, and attention heads concentrating on foreground with heuristics and prior methods. These identified tokens and attention heads are then used to define the attention maps used for contrastive decoding. Experimental results on several public models show that the proposed method achieves favorable improvements across different settings.
Critical strengths
(1) It is a promising direction to enhance models’ reasoning capabilities by redirecting their attention to pertinent visual information.
(2) The method shows consistent improvements over three baselines, and appears to generate captions with more details.
(3) The paper proposes extensive ablation study, which facilitates understanding the contribution of individual components.
Critical weaknesses
(1) It is not a new idea to redirect models’ attention. While prior work may not specifically emphasize explicit discrimination between foreground and background, a direct comparison with quantitative evaluation is still important here. For instance, do existing attention reallocation methods fail to differentiate foreground and background? What’s the corresponding classification accuracy? Why is the proposed heuristic-based approach better than prior work?
(2) Related to the previous comment. Looking at Table 1, it appears that the improvement achieved by the proposed method is relatively incremental compared to prior methods. It is therefore unclear how advantageous is the explicit discrimination between foreground and background.
(3) Several key comparisons in the paper are rather incomplete in my opinion. Specifically, Table 2 compares the visual token classifications with the proposed method on different backbones, without involving any prior methods. Similarly, Table 4 focuses on hallucination score but no baseline/prior method is compared. Table 3 compares Qwen3 VL 4B with the proposed method against Qwen3 VL 8B, but fails short of analyzing the scaling capabilities of the method, e.g., what’s the performance of different methods on models with increasing scales?
(4) Table 8 shows that redirecting attention from sink tokens to visual tokens achieves over 80% of the improvement for the full methods. As discussed in Section 2, there is a body of research on utilizing/alleviating attention sink, and some recent models (e.g., DINOv3) have shown to be less prone to sink tokens. How would the proposed method be helpful for these more recent vision models?
(5) The paper only provides two qualitative examples on how the method helps image captioning. However, neither of the examples directly show how attention reallocation, with a focus on differentiating foreground and background, is involved in the inference process. For example, captions generated by the proposed method seem to contain more details, but without a visualization of the corresponding attention, it is uncertain where these additional details are attributed to and whether it is genuinely about background removal. I would also expect some quantitative scores here about how the method shifts attention from background to foreground, and how the foreground attention is linked to the generated captions.
Questions for authors
(1) Please justify the key advantages of the proposed method over related work on attention reallocation, and directly compare them with attention distribution.
(2) How does the proposed method improve over previous ones in terms of addressing incorrect attention allocation, hallucination, and scaling? Please refer to my comments on the weaknesses.
(3) How does the method work with vision models with fewer or no sink tokens?
(4) Please consider incorporating additional qualitative analysis and quantitative analysis on the relationship between shift in attention distribution and its impacts on generated captions.