- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why vision-language models can still hallucinate even when their attention focuses on relevant visual regions. The authors propose a token-level candidate response analysis based on local visual interventions and identify Competitive Evidence Imbalance, where tokens inside relevant regions may support competing answers and incorrect responses contain fewer inhibitory support tokens. Based on this observation, the paper introduces SPEAR, a training-free decoding method that selects suppression-prioritized visual evidence and performs KL-constrained candidate probability recalibration. Experiments across multiple LVLMs and hallucination benchmarks show improvements in object hallucination mitigation, visual perception, and multimodal reasoning.
Critical strengths
-
The paper investigates an important failure mode of LVLMs beyond the common observation that models ignore visual information. The analysis of competing evidence within attended regions provides a useful direction for understanding hallucination.
-
SPEAR is a carefully designed training-free intervention method with reasonable computational constraints. The matched-count ablation in Table 5 helps separate the effect of suppression prioritization from simply selecting more evidence tokens.
Critical weaknesses
-
The paper shows that incorrect answers contain fewer inhibitory support tokens and more rival-favoring tokens, but this is still a correlation between token responses and model decisions. Stronger causal validation is needed, such as directly adding/removing inhibitory support tokens and measuring whether hallucination rates change accordingly.
-
The interpretation of Γ as token-level evidence is based on a local directional derivative around the current hidden state. It remains unclear whether this first-order response reliably predicts finite interventions during actual decoding, especially when multiple tokens are selected and recalibration changes subsequent generation states.
-
SPEAR relies on several manually chosen heuristics, including top-8 candidate selection, attention-based region proposal, fixed 5%/10% role retention rates, and KL budget κ=0.0025. Although sensitivity studies are provided, the paper does not analyze whether these choices transfer across architectures, tasks, and generation settings.
-
The method assumes that correcting hallucination mainly requires increasing suppression of competing candidates. However, some hallucinations may arise from missing visual evidence, language priors, or incorrect object grounding rather than insufficient rival suppression. The paper would benefit from a failure-case analysis showing when SPEAR fails and whether its gains are concentrated on specific hallucination types.
-
Table 14 shows SPEAR increases decode time from 26.9 ms/token to 71.0 ms/token on Qwen3-VL-2B and from 31.4 ms/token to 89.1 ms/token on LLaVA-OneVision-1.5-4B. The paper should better discuss the accuracy-efficiency trade-off and compare against other inference-time intervention methods under similar latency constraints.
Minor comments:
- The notation around diagnostic analysis and online inference is somewhat difficult to follow because the paper uses different candidate sets and response definitions; a clearer separation earlier would help.
- Some qualitative examples are useful for intuition but do not provide strong evidence beyond selected successful cases.
Questions for authors
pls see weakness
4 comments
Q4. What failure patterns remain after calibration?
Response 4. Thank you for suggesting a failure analysis. We use local visual-token responses to study how visual evidence shapes competition between candidate answers. Tokens within the same region may directly support an answer or suppress its competitors. These responses guide SPEAR's training-free selection of evidence for the current candidates. SPEAR prioritizes suppressive evidence and recalibrates candidate probabilities under a KL constraint.
We paired greedy and SPEAR captions for the same CHAIR-500 images and checked each description against the image. The model correctly identified the main objects and produced incorrect details about their relations, parts, or markings. Three LLaVA-OneVision-1.5-4B caption pairs illustrate this pattern.
- Spatial relation. Both captions describe a truck carrying a concrete structure that stands behind it (COCO 376677).
- Part recognition. Both captions describe the blue insert in a urinal as a seat (COCO 34830).
- New reading error. SPEAR adds the bus number 2703. The number visible in the image is 1208 (COCO 363927).
Recalibration draws on cues already encoded by the model and covered by the attention proposal. Missing or misread cues limit the evidence for separating candidates. Strong language priors may also persist under the bounded update. The paired captions show how these errors appear in complete outputs. Figure 5(b) reports gains over greedy decoding across eight MMHal-Bench question categories.
Q5. What quality does SPEAR achieve at comparable decoding latency?
Response 5. On Qwen3-VL-2B COCO captioning, each SPEAR step batches gradients for the top eight candidates, scores regional evidence, and updates probabilities under a KL budget. The implementation reuses the KV cache. We calibrated every 1, 2, or 4 steps starting at step 1. Skipped steps decode from the model's current distribution.
We measured inter-token latency on 1,000 COCO images. We evaluated caption quality on CHAIR-500 with up to 256 new tokens per caption.
| Method | Inter-token latency ↓ (ms/token) | CHAIR-S ↓ (%) | CHAIR-I ↓ (%) |
|---|---|---|---|
| Greedy | 27.0 | 39.80 | 12.18 |
| LVLMs-Saliency (Qwen adaptation) | 694.3 | 37.20 | 8.52 |
| Full SPEAR | 71.0 | 30.80 | 5.52 |
| SPEAR every 2 steps | 53.42 | 32.60 | 6.25 |
| SPEAR every 4 steps | 41.33 | 33.80 | 7.42 |
| VCD (greedy adaptation) | 55.01 | 40.20 | 9.27 |
| VCD (native sampling) | 54.7 | 38.40 | 8.58 |
Full SPEAR takes 2.63× as long per token as greedy and 89.8% less time per token than Qwen-adapted SGRS+LocoRE (71.0 vs 694.3 ms/token). Both methods use gradients. Saliency retains its sampling procedure, and SPEAR selects greedily after calibration. On CHAIR-500, full SPEAR lowers CHAIR-S/I by 6.40/3.00 points relative to Saliency.
Every-two-step SPEAR is close to greedy VCD in latency (53.42 vs 55.01 ms/token). In the CHAIR evaluation it has lower rates than both reported VCD variants, with increases of 1.80/0.73 points over full SPEAR. Every-four-step calibration lowers latency by a further 22.6% relative to every-two-step calibration and raises CHAIR-S/I by 1.20/1.17 points, for total increases of 3.00/1.90 points over full SPEAR. Calibration frequency thus provides a direct way to trade decoding time for caption quality. The two-step schedule stays closer to full SPEAR's CHAIR scores at latency comparable to VCD.
Minor comments. Revised Section 3 distinguishes offline reference–rival probes in annotated boxes from online comparisons among top-eight candidates within attention proposals. Equation (4) gives the local margin response ΔΓ, and Section 3.2 defines the evidence-selection score D. Response 4 includes persistent and newly introduced errors.
Dear Authors,
Thank you for your detailed and thoughtful response. I have a few additional suggestions. There still appears to be some room in the main paper to show your muscle, and I encourage you to make full use of the available space. It would also be helpful to clearly mark the revised sections so that reviewers and the AC can easily identify the changes.
Since my current rating is already positive, I will keep it unchanged for now and follow the discussions and feedback from other reviewers and authors' paper revisions.
Good Luck.
Dear Reviewer PqJE,
Thank you for taking the time to read our response and share these suggestions. If time permits, we’ll consider using the remaining space in the main paper to expand on the points raised during the discussion and present the most relevant supporting results. We’ll also clearly mark the revised sections so that the reviewers and the Area Chair can easily find the changes.
Best regards,
The Authors
Thank you for taking the time to review our paper and for your thoughtful questions and suggestions. We respond to each point below.
Response 1. We used initially correct Qwen3-VL-2B answers from 1,000 fixed TextVQA_GT_bbox questions. We assigned roles at the first answer position of the unmasked run, set attention logits for the selected in-box visual keys to −∞ before softmax, and regenerated complete answers. Regional shares are from Qwen's correct diagnostic cohort (Table 9).
Masking role II caused the largest drop in final-answer score. The IV and III+IV masks removed more tokens and yielded smaller drops. Q3 tests suppression priority at a matched selection budget.
Response 2. On TextVQA_GT_bbox, we fixed the prefix, candidates, and intervention layer. Single-token probes interpolated each head output toward the selected token's value, oh′=(1−ϵ)oh+ϵvhj. Joint probes shifted attention toward or away from up to four role-II tokens while preserving regional attention mass and out-of-region weights. We compared the first-order prediction from ΔΓ with the recomputed gold–rival logit-margin change.
Sign agreement was 93.33–100% at ε=0.1 and fell to 75.00–86.67% at ε=0.3. Joint probes gave 94.12–100% agreement. The results support directional prediction at a fixed prefix. SPEAR uses Γ from the unmodified state to recalibrate probabilities (conditional KL budget κ=0.0025, step cap ηₘₐₓ=1). Final-answer scores assess the decoded sequence.
Response 3. Table 3 uses shared selection and calibration settings across seven backbones on POPE-COCO, CHAIR, MME, and AMBER under greedy decoding, with architecture-specific layer windows. We compared two selectors on POPE-COCO adversarial (3,000 questions) and CHAIR-500 (up to 256 new tokens). Both target full SPEAR's quota per step and candidate pair and share its candidate, region, and calibration settings. Full SPEAR retains 5% of each support role and 10% of each suppression role.
Full SPEAR improves all four metrics over both controls on both backbones. CHAIR-I falls by 1.89/1.17 points versus equal rates and by 3.00/3.59 versus direct-support replacement on Qwen/LLaVA. Replacing suppression with more direct support further worsens all four metrics.
On Qwen CHAIR-500, we exchanged lower-ranked support tokens for the highest-ranked unused suppression tokens at a fixed total count. Percentages denote requested shares of the original quota.
All reallocations reduced object recall by 3.07–3.38 percentage points and raised both CHAIR rates. At 50% reallocation, CHAIR-S/I increased by 6.40/5.04 points. Default retention (5% support, 10% suppression) achieved the highest recall and lowest hallucination rates among these settings.