acceptodds
Under review as a conference paper at ICLR 2027

Focus where it Matters: Training-Free Semantic Re-Focus for Vision Language Models

Abstract

Vision-language models (VLMs) often fall back on linguistic priors even when the visual input contains the evidence needed to answer a query. We systematically investigate this failure through controlled input perturbations and decoder attention analysis. While visual context is essential for accurate generation, visual tokens receive disproportionately low attention. Moreover, this attention is concentrated in a small subset of heads and layers and is often poorly aligned with query-relevant image regions. Motivated by these insights, we introduce Semantic Re-Focus (SRF), an inference-time method that intervenes at two complementary stages. A semantic relevance map is estimated and is used to apply foveation before visual encoding and a selective, query-conditioned enhancement of visual attention is performed during decoding. Across five benchmark suites and two VLM architectures, SRF improves performance in fine-grained perception, counterfactual settings, object hallucination, and open-ended generation. On Qwen2.5-VL-3B and LLaVA-1.5-7B, it improves MMVP pair accuracy from to , and lowers the MMHal-Bench hallucination rate from to respectively. Our results suggest that query-conditioned control of visual evidence is a promising direction for building more reliably grounded multi-modal models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision language models, visual grounding, language bias, training-free, inference-time intervention, attention re-weighting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_72f7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Semantic Re-Focus (SRF) uses an external, frozen CLIP model to identify image regions relevant to a query. This relevance map controls both a spatially varying blur before image encoding and attention-logit adjustments in selected decoder heads and layers. The head selection uses a small unlabeled calibration set. The paper motivates these interventions with input perturbations and attention measurements, and evaluates them on fine-grained discrimination, counterfactual reasoning, perception, and hallucination benchmarks using Qwen2.5-VL-3B and LLaVA-1.5-7B. An optional extension adds a no-image contrastive decoding pass.

Critical strengths
  1. The intervention is straightforward and the reported improvements span several useful settings. Qwen MMVP pair accuracy increases from 40.0% to 45.3%, while VLMBias average accuracy increases from 18.1% to 20.2% (Table 1). The LLaVA results also show improvements across all four reported MME subtasks and lower MMHal hallucination (Table 2). Keeping the external alignment model fixed across architectures is a sensible design choice. Reporting actual latency, including relevance estimation, is more informative than reporting only the number of VLM passes (§C, Table 6).

  2. The paper includes useful checks beyond the headline results. The random-map ablation largely removes the MMVP gain (Table 3), which supports the relevance map carrying useful information. The POPE appendix reports nearly unchanged results on two splits instead of presenting the method as uniformly better, and the VLind extension shows that plain SRF does not improve every counterfactual setting (Tables 4–5). These qualifications make the scope of the empirical claim easier to assess.

Critical weaknesses
  1. Essential parts of the evaluation pipeline are not specified well enough to reproduce or assess the results. Section 4.1 defines the grounding target as salient nouns or noun phrases, but does not explain how g(x) is implemented, how multiple entities or relational questions are handled, or whether any manual choices are involved. This matters because the same target drives both interventions. Appendix C supplies numerical hyperparameters, but not the source of the 20 calibration examples or the procedure/data used to select the layer ranges and logit strengths. It also describes the MMHal evaluator only as the “GPT model used in SOTA works,” without a concrete model version or grading setup. The LLaVA comparison uses published baseline numbers rather than rerunning them (§C), so matching the generation and judging protocols is particularly important.

  2. The evidence does not yet establish the claimed complementarity and reliability of the two-stage design. Table 3 adds components in one order, but lacks foveation alone and a uniform-relevance control, so it cannot show how much of the gain requires selective decoder reweighting rather than preprocessing or a global visual boost. A random spatial map is helpful but does not answer those comparisons. Moreover, the attention-mass imbalance in §3 is descriptive: a low fraction of raw attention does not by itself establish that visual information is insufficiently used. The main Qwen pair-accuracy gain corresponds to eight additional correct pairs out of 150, yet no paired uncertainty analysis is provided. The intermediate MMVP pair scores in Table 3—41.9, 42.2, and 43.0—also need explanation, since a single evaluation of 150 pairs has increments of approximately 0.67 percentage points (§C). Averaging across calibration runs could explain this, but such averaging is not described. Together, these gaps make the strength and source of the gain harder to judge.

Questions for authors
  1. Please specify g(x), the calibration-data source, the hyperparameter-selection procedure and split, and the exact MMHal judge/version/prompt. Were the LLaVA baseline and SRF generations judged using the same protocol?

  2. How do foveation alone and the full intervention with a uniform relevance map perform under the same tuning budget? These comparisons would establish whether both semantic stages are needed.

  3. Are the Table 3 ablation scores averages over multiple runs or different evaluation subsets? Please report the evaluation counts, calibration variability, and a paired confidence interval or test for the full SRF versus baseline MMVP result.

4 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for the careful and constructive review. The answers are below and also in the paper (all changes in Blue)

  • g(x) is a deterministic, rule-based parser with no learned component and no manual choice per sample. The benchmarks follow a few fixed question formats in which the queried object is named explicitly, so g(x) is an ordered list of regular-expression templates applied to the lower-cased question, and the first template that matches returns its captured phrase: possessive ("the X's wings" → X), existence and counting ("Is there a X …", "How many X …"), position and attribute ("Is the X above …", "… of the X") and a generic subject template ("Is the/this X …"). Captures are a single noun for attribute questions and a full noun phrase for existence and position questions. If the target matches no image region, the confidence gate (Eq.~4-5) makes the map uniform, so whole image has to be looked at thoroughly (similar to humans searching whole scene to check if object is absent or describe the scene) [1][2]. Check Figure 8 in appendix

For free-form prompts that name several objects, we also now extended g(x)g(x) to a benchmark-agnostic multi-target extractor. The prompt is part-of-speech tagged and chunked into noun phrases with NLTK [3]. We also checked on a new dataset GQA, whose free-form questions involve several objects and spatial relations and the merged maps cover the named objects and SRF with this extractor. Check **Figure 9 in appendix **

  • The purpose of calibration is to identify a structural property of the model: which attention heads route their attention predominantly to visual rather than to textual tokens. It uses 20 image-question pairs drawn at random, with a fixed seed, from the evaluated benchmark. One forward pass of the unmodified model per pair gives each head's share of attention on image tokens (Eq.8); in each layer of the band, the heads with the highest average share are kept (Eq.9). Labels and predictions are never used. We did a new experiment with nine calibrations with different images and sample sizes (3 draws × 20 / 40 / 60) select 72 (Qwen2.5-VL-3B) and 68 (LLaVA-1.5-7B) of 78 heads identically, with an average Jaccard overlap of 0.94 / 0.90 against 0.10 by chance (Table 8 in Appendix)
Qwen2.5-VL-3BLLaVA-1.5-7B
Selected (layer, head) pairs per calibration7878
Pairs selected by all 9 calibrations7268
Avg. overlap between two calibrations0.940.90
Overlap expected by chance (random selection)0.100.10
  • The layer band is fixed once per architecture and spans the central decoder layers, in which vision-responsive heads are concentrated. For the new Qwen2.5-VL-7B, the parameters translate to gains in the bigger VLM as well. We follow ILVAD, the source of the LLaVA baselines in Table 2: greedy generation with at most 512 new tokens, judged by GPT-4o at temperature 0 with the official MMHal-Bench grading prompt and 0–6 rubric [4] We now also evaluate every model on all four benchmarks and add the stronger Qwen2.5-VL-7B (Table7 in Appendix).
ModelMMVP (pair acc.)VLMBias (acc.)MME (score)MMHal score ↑MMHal Hal% ↓
Qwen2.5-VL-3B40.0 / 45.318.1 / 20.2726.7 / 736.73.10 / 3.3350.0 / 42.7
LLaVA-1.5-7B22.7 / 26.015.6 / 15.3631.6 / 693.32.01 / 2.4667.0 / 57.2
Qwen2.5-VL-7B52.7 / 54.722.1 / 22.5723.3 / 726.73.41 / 3.5038.5 / 36.5
  • Ablation - On its own, foveation gives a small part of the gain Its benefit also depends on where the blur is placed: the same amount of blur spread uniformly, or placed by a shuffled map, is clearly worse than the semantic placement on both models. Foveation helps more when it is semantically placed and combined with decoder re-weighting. With a uniform relevance map, the full intervention gives the same boost to every image token. It improves over the model without SRF. Query-conditioned, selective re-weighting is needed for the full gain: the two semantic stages are complementary, and neither preprocessing nor a uniform boost can replace them.
ConfigurationMMVP pair acc.
Model without SRF40.0
Foveation only41.3
Decoder re-weighting only42.7
Uniform relevance map (global boost)42.7
Random map40.7
Full SRF45.3

[1] J.~M.~Wolfe. What can 1 million trials tell us about visual search? \emph{Psychological Science} [2] S.~Liu, K.~Zheng and W.~Chen. Paying more attention to image: A training-free method for alleviating hallucination in LVLMs. In \emph{ECCV}, 2024. [3] Xie et al. ILVAD: Mitigating hallucination in LVLMs via inter-layer visual attention discrepancy. ICML 2026. [4] Z. Sun et al. Aligning large multimodal models with factually augmented RLHF (MMHal-Bench). Findings of ACL 2024.

AuthorsOpenReview ↗

Response 2/2

  • The attention-mass imbalance - We agree that a low share of raw attention does not by itself establish under-use, and we added interventional and quantitative evidence on LLaVA-1.5-7B (Tables 9-10 in Appendix). First, re-directing attention towards the image changes the answers, and does so most when the calibrated vision-responsive heads are steered rather than all heads or random heads. Second, measured against ground-truth object boxes, the decoder's attention singles out the queried object less well than the CLIP relevance map, and objects are missed precisely when attention is less aligned with them. Third, blurring the queried object makes accuracy collapse, whereas blurring an equal area elsewhere does not, and SRF raises alignment mainly on the questions it corrects. The imbalance has measurable consequences for the predictions, and SRF acts on them.
POPE adversarialMME
Decoder attention, model without SRF0.740.70
CLIP relevance map0.850.87
Heads modified by SRF, without -> with SRF0.822 -> 0.829 (p = 2e-8)0.770 -> 0.786 (p = 0.04)
Accuracy (%)POPE adversarialMME
Original image86.782.7
Queried object blurred37.556.8
Same-size region elsewhere blurred84.179.7
  • ablation rows of Table 3 are means over several calibration draws. We repeated the calibration with six different random draws of the 20 unlabeled samples. SRF is at or above the model without SRF for all eleven draws. The selected heads themselves barely change between draws (Table 8 In Appendix) For the run of Table 1, SRF gains 19 pairs and loses 11 (exact McNemar p = 0.20; paired bootstrap 95% CI of the gain -2.0 to +12.7 points); at image level it gains 23 and loses 16 of 300 (p = 0.34).

  • We provide more ablations on LLAva and robustness to hyperparameters in response to Reviewer Wpyi. Kindly have a look.

We hope these answers and the new results address your concerns, and we are happy to clarify any remaining point.

Reviewer_72f7OpenReview ↗

Thanks for replying

Thanks for illustrating the implementation and supplmenting the experiments, which address several of my original questions. My remaining concern is the reliability of the gains: the paired confidence interval on MMVP includes zero, and the full method’s advantage over the simpler controls is small. The results are encouraging, but I remain uncertain about how consistently the full two-stage design helps. I will keep my score at 2.

We thank the reviewer for the follow-up

  • The MMVP interval in context. The interval has to be read against the benchmark. MMVP consists of 150 image pairs chosen because CLIP cannot tell them apart, each differing in one fine visual detail, and a pair counts as correct only if both of its images are answered correctly; the MMVP paper reports GPT-4V at 38.7 and Gemini at 40.7 pair accuracy, against 95.7 for humans. Under this metric, none of the compared training-free methods improves Qwen2.5-VL-3B at all: VAF, VCD, ILVAD and VHR land between -2.7 and 0 points relative to the baseline (Tab. 1; table below). SRF is the only method above the baseline (+5.3), and the mean over eleven calibration draws (+2.7) still exceeds every competitor's best. On 150 pairs, however, no gain of this magnitude can reach p < 0.05 in a single run: given how many answers change on this benchmark, exact McNemar requires a net gain of about +8 points, half as much again as the largest gain in Tab. 1. The interval therefore measures the resolution of the benchmark, not the reliability of the method, and the appropriate evidence on MMVP is replication. That evidence is now in place: all eleven calibration draws are above the model without SRF (sign test p = 0.001) at pair level and significant at image level (p = 0.02); each of the one-at-a-time hyper-parameter settings on LLaVA MME and Qwen MMVP is at above the model without SRF ; and SRF improves both paper models on MMVP, MME and MMHal-Bench.

    MMVP on Qwen2.5-VL-3B-Instruct (Tab. 1). Every compared method is at or below the baseline in pair accuracy; SRF is the only one above it, with no extra decoding pass.

    Method+passMMVP PairMMVP ImgΔ\Delta Pair vs baseline
    Baseline (no intervention)--40.067.7--
    VAF (CVPR'25)040.069.00.0
    VCD (CVPR'24)+137.365.7-2.7
    ILVAD (ICML'26)+138.768.0-1.3
    VHR (ACL'25)+238.768.3-1.3
    SRF (ours)045.370.0+5.3
  • Most existing training-free methods focus on a single intervention, relying primarily on the VLM's internal signals. However, re-weighting existing attention may reinforce irrelevant regions when the model fails to prioritize the queried evidence. SRF introduces a complementary two-stage design: an external semantic signal identifies relevant regions, while encoder-side foveation preserves their details and decoder-side re-weighting prioritizes them. Our ablations demonstrate the benefit of combining both stages.

    We believe our contribution introduces a promising direction for improving visual grounding through lightweight external semantic guidance, without retraining or scaling the VLM. This is particularly relevant for smaller, resource-constrained VLMs, where better use of existing visual evidence may offer a practical alternative to increasing model capacity. We hope the reviewer will consider this broader methodological contribution alongside the additional evidence when reassessing the work.
    
Reviewer_WpyiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reliance of vision-language models on linguistic priors despite the availability of relevant visual evidence. Through controlled input perturbations and decoder attention analysis, it identifies several issues: disproportionately low attention to visual tokens, uneven attention allocation across layers and heads, and spatial misalignment between decoder attention and query-relevant image regions. To address these issues, this work propose Semantic Re-Focus (SRF), a training-free method combining two interventions: (1) encoder-side semantic foveation, which preserves query-relevant image details while blurring less relevant regions using a CLIP-based relevance map; and (2) decoder-side attention reweighting, which selectively enhances relevant visual tokens within calibrated vision-responsive layers and heads. Experiments on Qwen2.5-VL-3B and LLaVA-1.5-7B across five benchmarks demonstrate improvements in fine-grained perception, counterfactual reasoning, and hallucination mitigation.

Critical strengths
  1. Well-motivated and practical method design. The paper establishes a coherent connection between its empirical observations and the proposed method. Unlike approaches that uniformly amplify visual attention, SRF considers both query-relevant image regions and vision-responsive attention heads. Combining encoder-side semantic foveation with decoder-side attention reweighting is intuitive and potentially effective.
  2. Encouraging results across diverse benchmarks. The evaluation covers fine-grained perception, counterfactual reasoning, and hallucination mitigation rather than focusing on a single benchmark. The improvements on MMVP, MME, and MMHal-Bench are encouraging. The random-map ablation also provides evidence that query-conditioned semantic information contributes to the gains.
Critical weaknesses
  1. Insufficient evidence for the proposed explanation of visual grounding failures. The paper motivates SRF through attention imbalance and spatial misalignment, but the connection between these observations and actual prediction failures remains insufficiently established. First, the discrepancy between the proportion of visual tokens and their attention allocation does not necessarily indicate inadequate visual information utilization. Second, the spatial misalignment analysis relies primarily on qualitative examples in Figure 3, without quantitative evaluation of attention localization or its relationship to prediction correctness.
  2. Incomplete ablations and limited evidence of generalization. The ablation study does not include a semantic-foveation-only configuration, making it difficult to isolate the contributions of the encoder and decoder interventions. Comparisons with alternative relevance maps and head-selection strategies are also missing, leaving the necessity of several design choices insufficiently validated. And the evaluation is limited to two relatively small VLMs, with different main benchmarks evaluated on different architectures. Overall, more comprehensive ablations and consistent cross-model evaluations are needed to establish the robustness and generalizability of the method.
Questions for authors
  1. Can the authors provide quantitative evidence supporting the attention-misalignment hypothesis? Beyond the qualitative examples in Figure 3, can the authors quantitatively evaluate the alignment between decoder attention and query-relevant image regions? More importantly, is attention misalignment systematically associated with incorrect predictions, and does SRF improve this alignment when correcting errors?
  2. Can the authors provide additional ablations to isolate the contributions of the proposed components? In particular, comparisons involving semantic foveation alone, alternative relevance maps, and different head-selection strategies (e.g., random or all-head intervention) would help establish the necessity and complementarity of the proposed components.
  3. How robust are the results across architectures and hyperparameter choices? Can the authors evaluate representative benchmarks consistently on both architectures, ideally including a stronger VLM?
2 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for the detailed and constructive review, and for finding our method well motivated and the evaluations encouraging. We answer the questions below and all the changes are Blue in the revised Appendix.

  • evidence supporting the attention-misalignment - To answer this question, we added multiple new analyses on LLaVA-1.5-7B, an intervention test of under-use, a quantitative alignment measure, its relation to prediction correctness, and the change in alignment when SRF corrects an error.

    • [1] Under-use of visual information - Removing the image drops POPE accuracy from 87.2 to 42.0 (Fig. 1). Re-allocating attention to image tokens improves accuracy when applied in the calibrated vision-responsive heads, but less when applied in random or all heads (Table 12). The visual evidence is under-used, and it is recovered most effectively through the heads that SRF selects.
    • [2] Measuring alignment - To replace the qualitative Fig 3 by a measurement, we use questions whose queried object has a ground-truth COCO bounding box . For each question, we take the decoder's attention from the answer position to the 576 image patches. The {alignment score} is the probability that a randomly chosen patch inside the box receives more attention than a randomly chosen patch outside it. This is the area under the ROC curve (AUC) when attention is used to separate objects from the background: 0.5 means attention does not favour the object (chance), and 1.0 means every object patch receives more attention than every background patch.
    • [3] On both benchmarks, the decoder's attention is clearly less aligned with the queried object than the CLIP map ((a) and Table 9a in Appendix ). On POPE, where every analysed question concerns an object that is present, the model misses an object precisely when its attention is less aligned with it, for small, medium and large objects alike ((b) and Table 9b in Appendix ). Errors coincide with misaligned attention, and the CLIP map localises the object better than the model itself, which is the premise of SRF.

(a)

POPE adversarialMME
Decoder attention, model without SRF0.740.70
CLIP relevance map ↑0.850.87
Heads modified by SRF, without → with SRF ↑0.822 → 0.829 (p = 2×10⁻⁸)0.770 → 0.786 (p = 0.04)

(b)

Object sizeAccuracy (%)Alignment, correct (n)Alignment, incorrect (n)p
Small (< 3%)77.90.97 (278)0.93 (79)3×10⁻⁶
Medium (3–24%)89.80.85 (378)0.78 (43)3×10⁻⁴
Large (> 24%)96.90.68 (378)0.54 (12)0.01
  • [4] To test whether the model actually relies on the evidence in the queried region, we blurred, either the object's box, a region of identical shape and area moved to a non-overlapping location, or the background (Table 10). Blurring the queried object lowers accuracy , whereas blurring a region of identical shape and size elsewhere doesnt change much. Conversely, blurring everything except the object raises the alignment. \textbf{The answer depends on the query-relevant region, and attenuating the rest of the image moves attention onto it, which is the mechanism of semantic foveation (Eq.~6).}
Accuracy (%)POPE adversarialMME
Original image86.782.7
Queried object blurred37.556.8
Same-size region elsewhere blurred84.179.7
  • Hyper-parameters - We varied each decoder strength and the head fraction one at a time on the LLaVA-1.5-7B, by a factor of two to three in each direction, on MME. Every setting stays above the model without SRF
ParameterLowerDefaultHigher
lamda_sem (relevance boost)0.25: 686.70.5: 693.31.0: 693.3
lamds_bg (background attenuation)0.1: 683.30.2: 693.30.4: 693.3
lamda_sys (system-token suppression)0.1: 686.70.3: 693.30.6: 693.3
k (head fraction)0.1: 686.70.2: 693.30.3: 683.3

And the selected heads are stable across calibrations (Table 8 in Appendix)

Qwen2.5-VL-3BLLaVA-1.5-7B
Selected (layer, head) pairs per calibration7878
Pairs selected by all 9 calibrations7268
Avg. overlap between two calibrations0.940.90
Overlap expected by chance (random selection)0.100.10
  • We now evaluate every model on all four benchmarks and add the stronger Qwen2.5-VL-7B (Table7 in Appendix). | Model | MMVP (pair acc.) | VLMBias (acc.) | MME (score) | MMHal score ↑ | MMHal Hal% ↓ | |---|---|---|---|---|---| | Qwen2.5-VL-3B | 40.0 / 45.3 | 18.1 / 20.2 | 726.7 / 736.7 | 3.10 / 3.33 | 50.0 / 42.7 | | LLaVA-1.5-7B | 22.7 / 26.0 | 15.6 / 15.3 | 631.6 / 693.3 | 2.01 / 2.46 | 67.0 / 57.2 | | Qwen2.5-VL-7B | 52.7 / 54.7 | 22.1 / 22.5 | 723.3 / 726.7 | 3.41 / 3.50 | 38.5 / 36.5 |
AuthorsOpenReview ↗

Response 2/2

  • Ablations We ran the requested ablations on LLaVA-1.5-7B with MME: Table 12 changes one component of SRF at a time.

Neither stage (foveation or decoder attention) alone changes a lot but together they give the best performance. This follows from the design: foveation keeps the query-relevant region sharp, decoder re-weighting directs attention to it, and a sharper region helps only if it is attended to. \textbf{The two stages are complementary, and the full gain requires both.}

With everything else fixed, replacing the CLIP map by a shuffled map (its values permuted over the image), a uniform map or a random map lowers the accuracy. Other alignment models perform on par or better, and hence are not very dependent on a particular alignment model. The gain comes from query-relevant localisation rather than from a generic visual boost, and it does not depend on a particular alignment model. Intervening in all heads, in an equal number of random heads, or only in early or late layers yields clearly smaller gains than intervening in the calibrated heads

CategoryConfigurationRelevance mapFoveationDecoder re-weightingHeadsLayersMME
ReferenceFull SRFCLIP ViT-B/32ononcalibratedSRF band693.3
StagesFoveation onlyCLIP ViT-B/32onoffnonenone673.3
StagesDecoder re-weighting onlyCLIP ViT-B/32offoncalibratedSRF band683.3
Relevance mapShuffled map (mean of 6 draws)CLIP, permuted (a)ononcalibratedSRF band676.1
Relevance mapUniform mapconstant (b)no blur (b)oncalibratedSRF band680.0
Relevance mapRandom map (mean of 6 draws)random (c)ononcalibratedSRF band669.4
Alignment modelCLIP ViT-B/16CLIP ViT-B/16 (d)ononcalibratedSRF band696.7
Alignment modelCLIP ViT-L/14CLIP ViT-L/14 (d)ononcalibratedSRF band693.3
Heads and layersAll headsCLIP ViT-B/32ononallSRF band676.7
Heads and layersRandom heads (mean of 6 draws)CLIP ViT-B/32ononrandom (e)SRF band680.0
Heads and layersEarly layersCLIP ViT-B/32ononcalibratedearly673.3
Heads and layersLate layersCLIP ViT-B/32ononcalibratedlate680.0
  • We hope these answers and the new results address your concerns, and we are happy to clarify any remaining point.
Reviewer_88TAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This work proposes Semantic Re-Focus (SRF), a training-free inference-time pipeline to improve visual grounding in MLLM. Given an input image and query, the method extracts target noun phrases from the query and uses an external frozen CLIP model to estimate a spatial relevance map. This map is used first to blur non-relevant background regions in the raw image before visual encoding, and second to boost attention logits on relevant image tokens while attenuating background and system tokens within calibrated middle-layer visual heads during decoding. The authors evaluate SRF on Qwen2.5-VL-3B (on MMVP, VLMBias, and POPE) and LLaVA-1.5-7B (on four MME subtasks and MMHal-Bench), showing gains on MMVP, MME, and MMHal-Bench, small improvements on VLMBias, and neutral or slightly negative results on POPE.

Critical strengths
  • Two-stage intervention: Intervening both at the input level before the visual encoder and inside the decoder attention layers is an interesting take that separates this work from some previous literature,

  • Training-free inference pipeline: The method requires no fine-tuning of the VLM weights and avoids running a second full VLM decoding pass.

Critical weaknesses

A. Reliance on noun extraction and external CLIP: The pipeline assumes that a compact grounding target can be extracted from nouns in the query and localized by an external CLIP ViT-B/32. Details on how the target noun extraction is actually implemented are missing from the work, making a complete criticism hard. However the settings seem rather limiting for more general use cases outside hallucination benchmarks.The setting also poses the method at an unfair advantage over competitors that are not relying on any external model.

B. CLIP behavior in wrongly posed questions: when a query asks about an absent or hallucinated object (as in negative POPE queries), the resulting CLIP grounding mask will highlight spurious regions (or falls back to uniform boosting via Equation 5), this behavior cannot help the model in any way and indeed performance on POPE are neutral.

C. Repeatability of target head extraction: Section 4.3.1 selects vision-responsive heads using 20 unlabeled calibration samples without specifying which dataset they come from or analyzing how stable the selected head set is across different calibration samples.

D. Limited evaluation: Experiments are restricted to only two models: a small Qwen2.5-VL-3B and a dated LLaVA-1.5-7B. Benchmarks and baselines are split across the two without overlap. MMVP, VLMBias, and POPE are only reported on Qwen2.5-VL-3B against four baselines, whereas MME (using only 4 of the 14 subtasks) and MMHal-Bench are only reported on LLaVA-1.5-7B using copied baseline numbers.

E. [minor] Presentation: I don’t think Section 3 does a good job at motivating the rest of the paper. The experiments in Section 3 are not particularly novel and do not really support the need for foveation or the need for boosting attention to specific parts of the image.

Questions for authors
  1. How is the grounding target t = g(x) extracted in practice, and how does the CLIP relevance map behave when a prompt contains multiple objects, abstract relations, or generic open-ended instructions without a single target noun?

  2. How stable is the attention head selection when you sample different images for the calibration set?

5 comments
AuthorsOpenReview ↗

Response 1/3

We thank the reviewer for the careful review, and for recognising the two-stage intervention and the training-free, single-pass design. We address each weakness and question below.

  • Reliance on noun extraction
    • The grounding target is extracted by a deterministic, rule-based parser (no learned component, no labels). It returns the salient noun or noun phrase of the question. The benchmarks SRF targets ask questions in a few fixed formats in which the queried object is named explicitly, so g(x) is an ordered list of regular-expression templates applied to the question; the first template that matches returns its captured phrase. In order: possessive ("the X's wings" → X), existence and counting ("Is there a X …", "Are there any X …", "Are there a total of n X …", "How many X …"), position and attribute ("Is the X above …", "Is the X facing …", "… of the X") and a generic subject template ("Is the/this X …"). Captures are a single noun for attribute questions and a full noun phrase for existence and position questions. The target is encoded as one CLIP text query. Figure 8 in Appendix shows the resulting maps and the complete rules are added to Appendix C.

    • The approach is not restricted to these formats. To verify this, we now extended g(x) to a benchmark-agnostic multi-target extractor. Answer-format instructions are stripped, the prompt is part-of-speech tagged with NLTK's averaged-perceptron tagger [1] and noun phrases are chunked with a small grammar that keeps modifiers with their noun and coordinated adjectives together. Each target is localised by the unchanged CLIP map, and the maps of the targets that pass the confidence gate (Eq. 4) are merged by an element-wise maximum. To test this extension on free-form questions with several objects, we chose a new dataset GQA [2] : its questions are generated from scene graphs of real images, so most name several objects and a spatial relation between them, (Figure 9 in appendix shows the visualizations )

This also provides a direct basis for future work on stronger grounding models.

[1] Bird, Klein and Loper. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit, O'Reilly Media, 2009 [2] Hudson and Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering.{CVPR}, 2019.

  • External alignment model

    • The alignment model is a frozen CLIP ViT-B/32 (about 151M parameters, 2–5% of the VLMs used), evaluated once per image, shared by all VLMs and never adapted to a benchmark. SRF adds no VLM pass, unlike VCD, ILVAD and VHR (Tab. 1, "+pass"); its end-to-end latency, including relevance estimation and foveation, is reported below and also in Table 11 in Appendix and is lower than that of every method requiring an additional pass. | Method | Extra VLM passes | ms / sample | MMVP pair acc. | |---|---|---|---| | VAF | 0 | 367 | 40.0 | | ILVAD | +1 | 668 | 38.7 | | VCD | +1 | 680 | 37.3 | | VHR | +2 | 802 | 38.7 | | SRF | 0 | 613 | 45.3 |

    • Relying on extra computation or an auxiliary model is common among training-free methods, and Table below lists these requirements: the contrastive methods (VCD, CODE, AGLA, ILVAD, VHR) run the VLM one or two additional times per generated token, CODE first generates a full image description, and AGLA also relies on an external image–text matching model (BLIP-ITM) and computes GradCAM on it to mask the image. | Method | Venue | Extra VLM passes | External model | Other requirement | |---|---|---|---|---| | VCD | CVPR'24 | +1 per token | – | noised copy of the image | | CODE | NeurIPS'24 | +1 per token | – | self-generated image description (one extra generation) | | AGLA | CVPR'25 | +1 per token | BLIP-ITM | GradCAM on the matching model to mask the image | | VAF | CVPR'25 | 0 | – | – | | VAR | ICLR'25 | 0 | – | selection of image-centric heads | | SPARC | ICML'25 | 0 | – | attention tracking across decoding steps | | ONLY | ICCV'25 | +1 layer per token | – | per-head text-to-visual entropy ratio | | VHR | ACL'25 | +2 | – | – | | ILVAD | ICML'26 | +1 | – | – | | SRF | – | 0 | CLIP ViT-B/32 (frozen) | 20 unlabeled calibration samples, once per model |

    -SRF does not depend on a particular CLIP: ViT-B/16 and ViT-L/14, and performs on par on both architectures (Table 6 in appendix) . SRF's requirements are lighter than those of most compared methods, and its gains come from the query-conditioned map rather than from a specific alignment model.

Alignment modelQwen2.5-VL-3B, MMVPLLaVA-1.5-7B, MME
CLIP ViT-B/32 (as in the paper)45.3693.3
CLIP ViT-B/1645.3696.7
CLIP ViT-L/1444.7693.3
AuthorsOpenReview ↗

Response 2/3

  • CLIP behavior in wrongly posed questions
    • SRF cannot help by localising an object that is not present, and for this case it is designed not to harm. Eq. 5 in paper describes a soft confidence that blends the mask towards uniform. If the target is not found, its alignment s is below τ, so the confidence becomes small and mask M̄ = cM + (1−c)1 approaches the uniform map. Nothing is then blurred (Eq. 6), and Eq. 11 adds the same shift to every image token, so an absent target produces neither a spurious foveation region nor a spatially misplaced boost; only the weight of the image relative to the system prompt increases. Also see more examples on Figure 8 in Appendix Deciding that an object is absent requires looking everywhere, which is why target-absent visual search in humans is exhaustive and slower [1][2], and uniformly re-weighting attention towards the image is a known training-free way to reduce “yes'' hallucinations [3]. Stronger grounding models are a natural extension for future work.

[1] Chun and Wolfe. Just say no: How are visual searches terminated when there is no target present? {Cognitive Psychology}, [2] Wolfe. What can 1 million trials tell us about visual search? {Psychological Science} [3] Liu, Zheng and Chen. Paying more attention to image: A training-free method for alleviating hallucination in LVLMs. In {ECCV}, 2024.

  • Repeatability of target head extraction: The 20 calibration samples are drawn at random from the unlabeled images of the evaluation dataset. Calibration reads only the attention statistics of one forward pass (the vision attention share). It never uses answers, labels or model predictions.

    • To test stability, we performed a new experiment. We repeated the calibration nine times, with three random draws each of 20, 40 and 60 samplesQwen2.5-VL-3B was calibrated on MMVP images and LLaVA-1.5-7B on MME images. Table 7, Fig 10 - Of the 78 (layer, head) pairs selected per calibration, 72 (Qwen2.5-VL-3B) and 68 (LLaVA-1.5-7B) are selected by all nine calibrations, and the average Jaccard overlap between any two calibrations is 0.94 and 0.90, against 0.10 for a random selection of the same size. (Overlap compares the selections of two calibrations. It is the number of (layer, head) pairs selected by both, divided by the number selected by either (Jaccard index): 1 means identical selections, 0 means none in common.)
Qwen2.5-VL-3BLLaVA-1.5-7B
Selected (layer, head) pairs per calibration7878
Pairs selected by all 9 calibrations7268
Avg. overlap between two calibrations0.940.90
Overlap expected by chance (random selection)0.100.10
  • Limited evaluation: We now evaluate every model on all four benchmarks, and also add the larger Qwen2.5-VL-7B (Table 7 in Appendix). SRF improves both paper models on the benchmarks previously reported only for the other model. Qwen2.5-VL-7B starts from a much stronger baseline than the 3B model but still SRF outperformsl.

On MME we follow the established hallucination protocol: the four subsets existence, count, position and colour, which probe object-level and attribute-level hallucination (30 images and 60 questions each). This subset was introduced for training-free hallucination mitigation by the SOTA methods and is the protocol under which every method in Table 2 is reported.

ModelMMVP (pair acc.)VLMBias (acc.)MME (score)MMHal score ↑MMHal Hal% ↓
Qwen2.5-VL-3B40.0 / 45.318.1 / 20.2726.7 / 736.73.10 / 3.3350.0 / 42.7
LLaVA-1.5-7B22.7 / 26.015.6 / 15.3631.6 / 693.32.01 / 2.4667.0 / 57.2
Qwen2.5-VL-7B52.7 / 54.722.1 / 22.5723.3 / 726.73.41 / 3.5038.5 / 36.5
AuthorsOpenReview ↗

Response 3/3

  • Section 3 presentation - We agree that Sec. 3 should motivate the design more directly. Section 3 now points to three new controlled experiments, in the revised appendix. The perturbation study (Sec. 3.1) shows that global degradations are tolerated; we localised this effect: on LLaVA-1.5-7B without SRF, blurring the queried object (ground-truth COCO box) collapses accuracy, whereas blurring a region of identical shape and area elsewhere leaves it almost unchanged, on POPE and MME alike (Table 10 in appendix). The answer depends on a small, query-specific region while the rest can be attenuated, which is exactly what semantic foveation (Eq. 6) exploits. The attention analysis now has a quantitative counterpart: against ground-truth boxes, the decoder's own attention singles out the queried object markedly less well than the CLIP relevance map, wrong answers coincide with lower alignment, and SRF raises alignment mainly on the questions it corrects (Table 9 in appendix); steering all heads or random heads instead of the calibrated heads yields clearly smaller gains (Table 12). Each component is now motivated by a measured effect: local, query-specific evidence → semantic foveation; visual attention concentrated in few heads → re-weighting restricted to those heads; decoder attention off the queried region → an external, query-conditioned map rather than a uniform boost.

We hope these answers and the new results address the reviewer's concerns, and we are happy to clarify any remaining point.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank the reviewers for their careful and constructive reviews. They found the two-stage, training-free design interesting and well motivated, and the evaluation diverse. Their concerns were on four points: the specification of the pipeline, the evidence behind the attention-misalignment motivation, ablations that isolate the two stages, and consistent cross-model evaluation with a stronger VLM

Goal VLMs often answer from linguistic priors even when the image contradicts them. Our analysis locates the bottleneck not in whether visual information is available but in whether the evidence relevant to the current query is prioritised. SRF addresses this specific failure by re-focusing the model on the query-relevant region at two stages, before encoding and during decoding, with no training and no additional VLM pass, and improves fine-grained perception, counterfactual reasoning and hallucination.

New in the revision. All revisions to paper are highlighted with BLUE color

  • Target extraction g(x) fully specified (rule templates, safeguards), plus a new benchmark-agnostic multi-target extractor (NLTK noun-phrase chunking, one map per phrase, merged by maximum), tested on a newly added benchmark, GQA (Figure 8).
  • Inference requirements and cost of all compared methods, and SRF ablations with two further CLIP backbones or alignment models (Tables 6, 11).
  • Calibration data and procedure, with a stability study: nine calibrations with different images and sample sizes select nearly identical heads on both architectures (Table 8, Fig. 10).
  • New results - All three models on all four benchmarks, with an additional larger Qwen2.5-VL-7B (Table 7)
  • Quantitative attention-alignment analysis against ground-truth object boxes: decoder attention is markedly less aligned with the queried object than the CLIP map, wrong answers coincide with lower alignment, and SRF raises alignment precisely where it corrects errors; a region-blur intervention shows the answer depends on the queried region (Tables 9–10).
  • Component ablations: foveation only, decoder only, shuffled / uniform / random maps, all / random heads, early / late layers and the two stages crossed with blur placement on both models
  • On the MMVP paired interval. As detailed in our replies, MMVP is 150 CLIP-blind pairs on which every compared training-free method is at or below the baseline (-2.7 to 0 points); SRF is the only method above it (+5.3). the interval measures the resolution of the benchmark rather than the reliability of the method. The reliable evidence is replication, and it is consistent: all eleven calibration draws, both architectures, and all hyper-parameter settings we ran are above the model without SRF, and the controls that remove the map's content carry paired significance (Tables R5, R8-R11). The gain is small per benchmark, as training-free gains in this literature are, but it is the only positive one on MMVP and it holds in every replication.

Scientific contribution and impact. Beyond individual benchmark gains, SRF introduces a distinct approach to training-free visual grounding. While existing methods primarily rely on contrastive decoding or modifying the VLM's internal attention, SRF uses an independent, query-conditioned semantic signal to guide two complementary stages. Our new analyses and ablations provide evidence for both design choices and their complementarity. The methodological contribution lies not simply in introducing an external model, but in selectively integrating its semantic guidance at two stages of VLM inference. Importantly, grounding failures persist with scale: even the 7B model scores only 52.7% on MMVP and 22.1% on VLMBias. The 3B model with SRF even exceeds the unmodified 7B on MME (736.7 vs. 723.3)

  • We believe SRF opens a promising direction for inference-time visual grounding, moving beyond methods that solely manipulate a VLM's internal attention. Our two-stage design provides a foundation for future research on how complementary external signals can be selectively integrated into pretrained VLMs (and particularly relevant for smaller and resource-constrained models) improve visual grounding and multimodal reasoning

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.