acceptodds
Withdrawn from ICLR 2027

FoAReD: Foreground-guided Attention Reallocation Decoding for Comprehensive Image Captioning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite their strong image captioning capabilities, VLMs often fail to capture fine-grained details in complex scenes. We find that this issue is associated with insufficient utilization of visual information during autoregressive decoding, where attention is dominated by redundant background and sink tokens. To address this issue, we propose FoAReD, a training-free image captioning framework that enhances visual information utilization through attention contrastive decoding. Specifically, FoAReD employs foreground-guided attention reallocation to construct a contrastive attention condition. During autoregressive decoding, the contrastive and original attention conditions are contrasted to enhance the generation preference toward informative visual details. Experiments on CompreCap and CaptionQA across multiple VLMs show that FoAReD consistently improves both caption comprehensiveness and accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VLM, Image Captioning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xGneOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper incorporates foreground-background discrimination with attention contrastive decoding for image captioning. In particular, it locates sink visual tokens, tokens likely belonging to backgrounds, and attention heads concentrating on foreground with heuristics and prior methods. These identified tokens and attention heads are then used to define the attention maps used for contrastive decoding. Experimental results on several public models show that the proposed method achieves favorable improvements across different settings.

Critical strengths

(1) It is a promising direction to enhance models’ reasoning capabilities by redirecting their attention to pertinent visual information.

(2) The method shows consistent improvements over three baselines, and appears to generate captions with more details.

(3) The paper proposes extensive ablation study, which facilitates understanding the contribution of individual components.

Critical weaknesses

(1) It is not a new idea to redirect models’ attention. While prior work may not specifically emphasize explicit discrimination between foreground and background, a direct comparison with quantitative evaluation is still important here. For instance, do existing attention reallocation methods fail to differentiate foreground and background? What’s the corresponding classification accuracy? Why is the proposed heuristic-based approach better than prior work?

(2) Related to the previous comment. Looking at Table 1, it appears that the improvement achieved by the proposed method is relatively incremental compared to prior methods. It is therefore unclear how advantageous is the explicit discrimination between foreground and background.

(3) Several key comparisons in the paper are rather incomplete in my opinion. Specifically, Table 2 compares the visual token classifications with the proposed method on different backbones, without involving any prior methods. Similarly, Table 4 focuses on hallucination score but no baseline/prior method is compared. Table 3 compares Qwen3 VL 4B with the proposed method against Qwen3 VL 8B, but fails short of analyzing the scaling capabilities of the method, e.g., what’s the performance of different methods on models with increasing scales?

(4) Table 8 shows that redirecting attention from sink tokens to visual tokens achieves over 80% of the improvement for the full methods. As discussed in Section 2, there is a body of research on utilizing/alleviating attention sink, and some recent models (e.g., DINOv3) have shown to be less prone to sink tokens. How would the proposed method be helpful for these more recent vision models?

(5) The paper only provides two qualitative examples on how the method helps image captioning. However, neither of the examples directly show how attention reallocation, with a focus on differentiating foreground and background, is involved in the inference process. For example, captions generated by the proposed method seem to contain more details, but without a visualization of the corresponding attention, it is uncertain where these additional details are attributed to and whether it is genuinely about background removal. I would also expect some quantitative scores here about how the method shifts attention from background to foreground, and how the foreground attention is linked to the generated captions.

Questions for authors

(1) Please justify the key advantages of the proposed method over related work on attention reallocation, and directly compare them with attention distribution.

(2) How does the proposed method improve over previous ones in terms of addressing incorrect attention allocation, hallucination, and scaling? Please refer to my comments on the weaknesses.

(3) How does the method work with vision models with fewer or no sink tokens?

(4) Please consider incorporating additional qualitative analysis and quantitative analysis on the relationship between shift in attention distribution and its impacts on generated captions.

Reviewer_PTvgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors argue that VLMs fails to capture fine grained details in complex scenes because VLMs insufficiently utilize visual information. The authors attribute this to attention being concentrated on background or sink tokens in later layer. To address this, the authors propose FoAReD, a contrastive decoding method that uses logits generated from attention re-allocation toward foreground tokens.

Critical strengths

None

Critical weaknesses

Comparison with directly related prior works on attention steering/re-allocation is insufficient, and the findings claimed as contributions are already known.

The authors suggest that existing methods improve captioning by introducing external supervision or guidance. However, many works already utilize internal visual representations such as attention for better utilization of visual information.

PAI [1] also steers the attention to image tokens during decoding and combines it with contrastive decoding, which is close to the attention contrastive decoding proposed in this paper.

SPARC [2] demonstrates that VLMs’ attention to key visual tokens weakens and attention shows a noisy pattern (concentrating on irrelevant(background) or sink tokens) especially during long context text generation in detailed image captioning tasks, and it selectively amplifies the attention to key visual tokens.

ASCD [3] addresses weak visual dependency by identifying key heads based on the text-to-image attention ratio and reallocating attention to strengthen these heads. It then performs contrastive decoding between the resulting logits and the logits obtained from decoding with weakened visual dependency.

Therefore, FoAReD’s components, foreground-guided attention reallocation(FAR), visual token classification(VTC), and attention contrastive decoding(ACD) have all been proposed in prior works in a broad sense, so the novelty is limited. Morerover, none of these methods are compared in the experiments.

Also, it is already known that attention to the relevant visual tokens in VLMs is high in the mid layers and low in the later layers. Furthermore, the later layers are considered to be the textual decoding stage, so the low visual attention in these later layers is not necessarily a problem [4,5].

[1] Liu et al. Paying More Attention to Images: A Training-Free Method for Alleviating Hallucination in LVLMs. ECCV2024

[2] Jung et al. Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models. ICML 2025

[3] Wang et al. ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM. AAAI 2026

[4] Jiang et al. Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens. CVPR 2025

[5] Zhang et al. MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs. ICLR 2025

Questions for authors
  1. There are too many hyperparameters, which is concern. Currently, ablations are provided only for some of the parameters on Qwen3-VL-4B and CompreCap. Could the authors provide ablations for all parameters, and on other models and benchmarks?

  2. The choice of layers for attention reallocation seems to be the most important, however there is no description of it. Could the authors describe the used layer and add an ablation on the layer?

  3. The authors suggest that the background tokens have small L2 norms of visual token features. This contradicts prior work [6], which shows that tokens whose ViT feature norm is high often contain globally redundant visual information (background). Could the authors provide further explanation related with this?

  4. Pruning on the background tokens seems to be orthogonal to the baselines in the main experiments. Therefore, applying the token pruning method only to FoAReD seems unfair. Also, since latency is strongly affected by response length, could the authors measure latency in s/token?

[6] Gong et al. DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination. EMNLP 2024

Reviewer_K2esOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FoAReD is a training-free framework for comprehensive image captioning. It uses feature norms to identify background candidates, prunes redundant tokens, and selectively reallocates attention from background and sink tokens to foreground tokens, favoring less-attended ones. Logit-level contrastive decoding amplifies the resulting prediction changes. Experiments across three VLM backbones report improved CompreCap and CaptionQA performance and reduced AMBER hallucination scores, supported by token-classification evaluation, ablations, and a length-controlled comparison. The method requires internal attention access and additional inference computation.

Critical strengths
  1. Cross-model gains with supporting controls. CompreCap Unified improves by 2.00, 1.91, and 5.30 points on Qwen3-VL-4B, LLaVA-v1.6-7B, and InternVL3-2B, respectively, alongside lower AMBER hallucination scores. The Qwen3-VL length-controlled comparison suggests that longer captions alone do not explain the improvement. Together, these results support broader caption coverage without worse performance on the reported hallucination benchmark.

  2. A useful training-free approach with competitive reported latency. Selective foreground reallocation and contrastive decoding require neither parameter updates nor external visual experts. FoAReD outperforms VCD in Unified score with slightly lower reported latency on all three backbones, supporting its practical value beyond simply adding computation to the base model.

Critical weaknesses
  1. The core intervention is not fully specified. Sections 3.2.2 – 3.2.3 provide head-classification statistics and recipient-side attention increments, but do not explicitly specify the classification thresholds and complete decision rules, source-side deductions, or handling of foreground/sink overlap. Although renormalization is stated, the complete source-to-recipient update is not spelled out. These details affect reproducibility and verification of the proposed mechanism. This is a specification concern, not evidence of an implementation error.

  2. Ablation interpretation and evaluation protocols remain unclear. Table 6 does not fully define how FAR and ACD are disabled. If “ACD disabled” corresponds to α = 0 in Equation (3) under otherwise identical settings, its score of 65.70 appears inconsistent with the approximately 65.3 read from Figure 7e. Different configurations could explain this, but the relationship is not clarified. Actual evaluation sample sizes, decoding/scoring configurations, and hyperparameter selection and transfer procedures are also insufficiently specified. These gaps limit attribution of component gains and assessment of robustness despite the positive overall results.

Questions for authors
  1. Intervention specification: Please provide the complete rules underlying Weakness 1, distinguishing inherited settings from modifications and specifying the renormalization scope. Does background detection use vision-encoder or post-projector features?

  2. Ablation correspondence: What is retained or replaced when FAR and ACD are disabled, and do both branches share the same pruned tokens? How does α = 0 relate to “ACD disabled,” and what configuration differences, if any, explain the corresponding results in Table 6 and Figure 7e?

  3. Evaluation and cost protocols: Please specify the evaluation and tuning settings underlying Weakness 2 and any generation/scoring randomness. Paired uncertainty estimates, if available, would help assess robustness. For latency, please report hardware, batch size, output lengths, and timing scope, and reconcile the “no additional computational costs” claim with the acknowledged overhead.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.