- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Semantic Re-Focus (SRF) uses an external, frozen CLIP model to identify image regions relevant to a query. This relevance map controls both a spatially varying blur before image encoding and attention-logit adjustments in selected decoder heads and layers. The head selection uses a small unlabeled calibration set. The paper motivates these interventions with input perturbations and attention measurements, and evaluates them on fine-grained discrimination, counterfactual reasoning, perception, and hallucination benchmarks using Qwen2.5-VL-3B and LLaVA-1.5-7B. An optional extension adds a no-image contrastive decoding pass.
Critical strengths
-
The intervention is straightforward and the reported improvements span several useful settings. Qwen MMVP pair accuracy increases from 40.0% to 45.3%, while VLMBias average accuracy increases from 18.1% to 20.2% (Table 1). The LLaVA results also show improvements across all four reported MME subtasks and lower MMHal hallucination (Table 2). Keeping the external alignment model fixed across architectures is a sensible design choice. Reporting actual latency, including relevance estimation, is more informative than reporting only the number of VLM passes (§C, Table 6).
-
The paper includes useful checks beyond the headline results. The random-map ablation largely removes the MMVP gain (Table 3), which supports the relevance map carrying useful information. The POPE appendix reports nearly unchanged results on two splits instead of presenting the method as uniformly better, and the VLind extension shows that plain SRF does not improve every counterfactual setting (Tables 4–5). These qualifications make the scope of the empirical claim easier to assess.
Critical weaknesses
-
Essential parts of the evaluation pipeline are not specified well enough to reproduce or assess the results. Section 4.1 defines the grounding target as salient nouns or noun phrases, but does not explain how g(x) is implemented, how multiple entities or relational questions are handled, or whether any manual choices are involved. This matters because the same target drives both interventions. Appendix C supplies numerical hyperparameters, but not the source of the 20 calibration examples or the procedure/data used to select the layer ranges and logit strengths. It also describes the MMHal evaluator only as the “GPT model used in SOTA works,” without a concrete model version or grading setup. The LLaVA comparison uses published baseline numbers rather than rerunning them (§C), so matching the generation and judging protocols is particularly important.
-
The evidence does not yet establish the claimed complementarity and reliability of the two-stage design. Table 3 adds components in one order, but lacks foveation alone and a uniform-relevance control, so it cannot show how much of the gain requires selective decoder reweighting rather than preprocessing or a global visual boost. A random spatial map is helpful but does not answer those comparisons. Moreover, the attention-mass imbalance in §3 is descriptive: a low fraction of raw attention does not by itself establish that visual information is insufficiently used. The main Qwen pair-accuracy gain corresponds to eight additional correct pairs out of 150, yet no paired uncertainty analysis is provided. The intermediate MMVP pair scores in Table 3—41.9, 42.2, and 43.0—also need explanation, since a single evaluation of 150 pairs has increments of approximately 0.67 percentage points (§C). Averaging across calibration runs could explain this, but such averaging is not described. Together, these gaps make the strength and source of the gain harder to judge.
Questions for authors
-
Please specify g(x), the calibration-data source, the hyperparameter-selection procedure and split, and the exact MMHal judge/version/prompt. Were the LLaVA baseline and SRF generations judged using the same protocol?
-
How do foveation alone and the full intervention with a uniform relevance map perform under the same tuning budget? These comparisons would establish whether both semantic stages are needed.
-
Are the Table 3 ablation scores averages over multiple runs or different evaluation subsets? Please report the evaluation counts, calibration variability, and a paired confidence interval or test for the full SRF versus baseline MMVP result.
4 comments
Response 2/2
- The attention-mass imbalance - We agree that a low share of raw attention does not by itself establish under-use, and we added interventional and quantitative evidence on LLaVA-1.5-7B (Tables 9-10 in Appendix). First, re-directing attention towards the image changes the answers, and does so most when the calibrated vision-responsive heads are steered rather than all heads or random heads. Second, measured against ground-truth object boxes, the decoder's attention singles out the queried object less well than the CLIP relevance map, and objects are missed precisely when attention is less aligned with them. Third, blurring the queried object makes accuracy collapse, whereas blurring an equal area elsewhere does not, and SRF raises alignment mainly on the questions it corrects. The imbalance has measurable consequences for the predictions, and SRF acts on them.
| POPE adversarial | MME | |
|---|---|---|
| Decoder attention, model without SRF | 0.74 | 0.70 |
| CLIP relevance map | 0.85 | 0.87 |
| Heads modified by SRF, without -> with SRF | 0.822 -> 0.829 (p = 2e-8) | 0.770 -> 0.786 (p = 0.04) |
| Accuracy (%) | POPE adversarial | MME |
|---|---|---|
| Original image | 86.7 | 82.7 |
| Queried object blurred | 37.5 | 56.8 |
| Same-size region elsewhere blurred | 84.1 | 79.7 |
-
ablation rows of Table 3 are means over several calibration draws. We repeated the calibration with six different random draws of the 20 unlabeled samples. SRF is at or above the model without SRF for all eleven draws. The selected heads themselves barely change between draws (Table 8 In Appendix) For the run of Table 1, SRF gains 19 pairs and loses 11 (exact McNemar p = 0.20; paired bootstrap 95% CI of the gain -2.0 to +12.7 points); at image level it gains 23 and loses 16 of 300 (p = 0.34).
-
We provide more ablations on LLAva and robustness to hyperparameters in response to Reviewer Wpyi. Kindly have a look.
We hope these answers and the new results address your concerns, and we are happy to clarify any remaining point.
Thanks for replying
Thanks for illustrating the implementation and supplmenting the experiments, which address several of my original questions. My remaining concern is the reliability of the gains: the paired confidence interval on MMVP includes zero, and the full method’s advantage over the simpler controls is small. The results are encouraging, but I remain uncertain about how consistently the full two-stage design helps. I will keep my score at 2.
We thank the reviewer for the follow-up
-
The MMVP interval in context. The interval has to be read against the benchmark. MMVP consists of 150 image pairs chosen because CLIP cannot tell them apart, each differing in one fine visual detail, and a pair counts as correct only if both of its images are answered correctly; the MMVP paper reports GPT-4V at 38.7 and Gemini at 40.7 pair accuracy, against 95.7 for humans. Under this metric, none of the compared training-free methods improves Qwen2.5-VL-3B at all: VAF, VCD, ILVAD and VHR land between -2.7 and 0 points relative to the baseline (Tab. 1; table below). SRF is the only method above the baseline (+5.3), and the mean over eleven calibration draws (+2.7) still exceeds every competitor's best. On 150 pairs, however, no gain of this magnitude can reach p < 0.05 in a single run: given how many answers change on this benchmark, exact McNemar requires a net gain of about +8 points, half as much again as the largest gain in Tab. 1. The interval therefore measures the resolution of the benchmark, not the reliability of the method, and the appropriate evidence on MMVP is replication. That evidence is now in place: all eleven calibration draws are above the model without SRF (sign test p = 0.001) at pair level and significant at image level (p = 0.02); each of the one-at-a-time hyper-parameter settings on LLaVA MME and Qwen MMVP is at above the model without SRF ; and SRF improves both paper models on MMVP, MME and MMHal-Bench.
MMVP on Qwen2.5-VL-3B-Instruct (Tab. 1). Every compared method is at or below the baseline in pair accuracy; SRF is the only one above it, with no extra decoding pass.
Method +pass MMVP Pair MMVP Img Pair vs baseline Baseline (no intervention) -- 40.0 67.7 -- VAF (CVPR'25) 0 40.0 69.0 0.0 VCD (CVPR'24) +1 37.3 65.7 -2.7 ILVAD (ICML'26) +1 38.7 68.0 -1.3 VHR (ACL'25) +2 38.7 68.3 -1.3 SRF (ours) 0 45.3 70.0 +5.3 -
Most existing training-free methods focus on a single intervention, relying primarily on the VLM's internal signals. However, re-weighting existing attention may reinforce irrelevant regions when the model fails to prioritize the queried evidence. SRF introduces a complementary two-stage design: an external semantic signal identifies relevant regions, while encoder-side foveation preserves their details and decoder-side re-weighting prioritizes them. Our ablations demonstrate the benefit of combining both stages.
We believe our contribution introduces a promising direction for improving visual grounding through lightweight external semantic guidance, without retraining or scaling the VLM. This is particularly relevant for smaller, resource-constrained VLMs, where better use of existing visual evidence may offer a practical alternative to increasing model capacity. We hope the reviewer will consider this broader methodological contribution alongside the additional evidence when reassessing the work.
Response 1/2
We thank the reviewer for the careful and constructive review. The answers are below and also in the paper (all changes in Blue)
For free-form prompts that name several objects, we also now extended g(x) to a benchmark-agnostic multi-target extractor. The prompt is part-of-speech tagged and chunked into noun phrases with NLTK [3]. We also checked on a new dataset GQA, whose free-form questions involve several objects and spatial relations and the merged maps cover the named objects and SRF with this extractor. Check **Figure 9 in appendix **
[1] J.~M.~Wolfe. What can 1 million trials tell us about visual search? \emph{Psychological Science} [2] S.~Liu, K.~Zheng and W.~Chen. Paying more attention to image: A training-free method for alleviating hallucination in LVLMs. In \emph{ECCV}, 2024. [3] Xie et al. ILVAD: Mitigating hallucination in LVLMs via inter-layer visual attention discrepancy. ICML 2026. [4] Z. Sun et al. Aligning large multimodal models with factually augmented RLHF (MMHal-Bench). Findings of ACL 2024.