acceptodds
Under review as a conference paper at ICLR 2027

Attending Yet Hallucinating: Inhibitory Evidence Deficits in Vision-language Models

Abstract

Large Vision-Language Models (LVLMs) combine visual understanding with language generation, yet hallucinated responses continue to limit their reliability. Recent studies show that these errors persist even when attention concentrates on relevant visual regions. This raises a question: **why does attending to the right location still lead to an incorrect answer?** We investigate how visual tokens within these regions affect competing answer candidates. Our analysis reveals **Competitive Evidence Imbalance**: (1) Visual tokens within relevant regions provide conflicting evidence, with a substantial share favoring competing answers over the correct one. (2) Tokens favoring the correct answer support it either directly or indirectly by suppressing competing answers. We call the latter **Inhibitory Support Tokens**, whose average count and regional proportion are markedly lower in hallucinated answers despite substantial remaining direct support. Motivated by this finding, we introduce Suppression-prioritized Evidence for Answer Recalibration (SPEAR), a training-free inference method. SPEAR proposes a relevant region from attention, prioritizes inhibitory support tokens while retaining direct support and opposing effects, and aggregates their signed responses to calibrate candidate probabilities under a bounded distributional change. Experiments show improved object hallucination mitigation, compositional discrimination, open-ended answer quality, and aggregate visual perception. These results highlight the value of using visual evidence to suppress competing answers as well as support plausible ones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal hallucination, large vision-language models, training-free inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PqJEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why vision-language models can still hallucinate even when their attention focuses on relevant visual regions. The authors propose a token-level candidate response analysis based on local visual interventions and identify Competitive Evidence Imbalance, where tokens inside relevant regions may support competing answers and incorrect responses contain fewer inhibitory support tokens. Based on this observation, the paper introduces SPEAR, a training-free decoding method that selects suppression-prioritized visual evidence and performs KL-constrained candidate probability recalibration. Experiments across multiple LVLMs and hallucination benchmarks show improvements in object hallucination mitigation, visual perception, and multimodal reasoning.

Critical strengths
  1. The paper investigates an important failure mode of LVLMs beyond the common observation that models ignore visual information. The analysis of competing evidence within attended regions provides a useful direction for understanding hallucination.

  2. SPEAR is a carefully designed training-free intervention method with reasonable computational constraints. The matched-count ablation in Table 5 helps separate the effect of suppression prioritization from simply selecting more evidence tokens.

Critical weaknesses
  1. The paper shows that incorrect answers contain fewer inhibitory support tokens and more rival-favoring tokens, but this is still a correlation between token responses and model decisions. Stronger causal validation is needed, such as directly adding/removing inhibitory support tokens and measuring whether hallucination rates change accordingly.

  2. The interpretation of Γ as token-level evidence is based on a local directional derivative around the current hidden state. It remains unclear whether this first-order response reliably predicts finite interventions during actual decoding, especially when multiple tokens are selected and recalibration changes subsequent generation states.

  3. SPEAR relies on several manually chosen heuristics, including top-8 candidate selection, attention-based region proposal, fixed 5%/10% role retention rates, and KL budget κ=0.0025. Although sensitivity studies are provided, the paper does not analyze whether these choices transfer across architectures, tasks, and generation settings.

  4. The method assumes that correcting hallucination mainly requires increasing suppression of competing candidates. However, some hallucinations may arise from missing visual evidence, language priors, or incorrect object grounding rather than insufficient rival suppression. The paper would benefit from a failure-case analysis showing when SPEAR fails and whether its gains are concentrated on specific hallucination types.

  5. Table 14 shows SPEAR increases decode time from 26.9 ms/token to 71.0 ms/token on Qwen3-VL-2B and from 31.4 ms/token to 89.1 ms/token on LLaVA-OneVision-1.5-4B. The paper should better discuss the accuracy-efficiency trade-off and compare against other inference-time intervention methods under similar latency constraints.

Minor comments:

  • The notation around diagnostic analysis and online inference is somewhat difficult to follow because the paper uses different candidate sets and response definitions; a clearer separation earlier would help.
  • Some qualitative examples are useful for intuition but do not provide strong evidence beyond selected successful cases.
Questions for authors

pls see weakness

4 comments

Thank you for taking the time to review our paper and for your thoughtful questions and suggestions. We respond to each point below.

Q1. Does removing inhibitory support affect complete answers?

Response 1. We used initially correct Qwen3-VL-2B answers from 1,000 fixed TextVQA_GT_bbox questions. We assigned roles at the first answer position of the unmasked run, set attention logits for the selected in-box visual keys to −∞ before softmax, and regenerated complete answers. Regional shares are from Qwen's correct diagnostic cohort (Table 9).

RoleRegional share (%)Mean masked (Qwen)VQA score (Qwen)Drop (pp)
Unmasked—0100.00%0.00
I (direct support)57.14———
II (inhibitory support)13.195.9483.44%16.56
III6.232.7198.94%1.06
IV23.4510.3597.16%2.84
III + IV29.6813.0796.40%3.60

Masking role II caused the largest drop in final-answer score. The IV and III+IV masks removed more tokens and yielded smaller drops. Q3 tests suppression priority at a matched selection budget.

Q2. Does the local response predict finite single- and multi-token interventions?

Response 2. On TextVQA_GT_bbox, we fixed the prefix, candidates, and intervention layer. Single-token probes interpolated each head output toward the selected token's value, oh′=(1−ϵ)oh+ϵvhjo'_h=(1-\epsilon)o_h+\epsilon v_{hj}. Joint probes shifted attention toward or away from up to four role-II tokens while preserving regional attention mass and out-of-region weights. We compared the first-order prediction from ΔΓ with the recomputed gold–rival logit-margin change.

Modelε=0.01ε=0.1ε=0.3Joint ↑Joint ↓
Qwen3-VL-2B100%100%75.00%100%96.83%
Ministral-3-3B100%100%85.00%100%94.12%
LLaVA-OneVision-1.5-4B100%93.33%86.67%96.10%96.10%

Sign agreement was 93.33–100% at ε=0.1 and fell to 75.00–86.67% at ε=0.3. Joint probes gave 94.12–100% agreement. The results support directional prediction at a fixed prefix. SPEAR uses Γ from the unmodified state to recalibrate probabilities (conditional KL budget κ=0.0025, step cap ηₘₐₓ=1). Final-answer scores assess the decoded sequence.

Q3. Do settings transfer, and what does suppression contribute at a matched token count?

Response 3. Table 3 uses shared selection and calibration settings across seven backbones on POPE-COCO, CHAIR, MME, and AMBER under greedy decoding, with architecture-specific layer windows. We compared two selectors on POPE-COCO adversarial (3,000 questions) and CHAIR-500 (up to 256 new tokens). Both target full SPEAR's quota per step and candidate pair and share its candidate, region, and calibration settings. Full SPEAR retains 5% of each support role and 10% of each suppression role.

  • Equal rate uses one retention rate for all four roles and adjusts integer quotas to match full SPEAR's count. Within each role, it ranks tokens by the magnitude of DD.
  • Direct-support replacement keeps the full method's direct-support selections, removes both suppression roles, and refills those slots with the highest-ranked unused direct-support tokens, subject to availability.
ModelMethodPOPE Acc/F1 ↑CHAIR-S/I ↓
QwenGreedy76.18/79.5239.80/12.18
QwenFull SPEAR87.13/86.9230.80/5.52
Qwen\textcolor[RGB]{48,102,183}{\text{Equal rate}}85.6/85.732.6/7.41
Qwen\textcolor[RGB]{135,70,162}{\text{Direct-support replacement}}82.4/82.934.8/8.52
LLaVAGreedy83.63/83.0032.00/7.77
LLaVAFull SPEAR88.07/87.4226.20/5.06
LLaVA\textcolor[RGB]{48,102,183}{\text{Equal rate}}86.6/86.527.76/6.23
LLaVA\textcolor[RGB]{135,70,162}{\text{Direct-support replacement}}85.4/85.629.8/8.65

Full SPEAR improves all four metrics over both controls on both backbones. CHAIR-I falls by 1.89/1.17 points versus equal rates and by 3.00/3.59 versus direct-support replacement on Qwen/LLaVA. Replacing suppression with more direct support further worsens all four metrics.

On Qwen CHAIR-500, we exchanged lower-ranked support tokens for the highest-ranked unused suppression tokens at a fixed total count. Percentages denote requested shares of the original quota.

Quota reallocatedCHAIR-S ↓CHAIR-I ↓Object recall ↑
Full SPEAR30.805.5275.88%
10%31.205.7272.50%
30%33.406.3472.81%
50%37.2010.5672.50%

All reallocations reduced object recall by 3.07–3.38 percentage points and raised both CHAIR rates. At 50% reallocation, CHAIR-S/I increased by 6.40/5.04 points. Default retention (5% support, 10% suppression) achieved the highest recall and lowest hallucination rates among these settings.

Q4. What failure patterns remain after calibration?

Response 4. Thank you for suggesting a failure analysis. We use local visual-token responses to study how visual evidence shapes competition between candidate answers. Tokens within the same region may directly support an answer or suppress its competitors. These responses guide SPEAR's training-free selection of evidence for the current candidates. SPEAR prioritizes suppressive evidence and recalibrates candidate probabilities under a KL constraint.

We paired greedy and SPEAR captions for the same CHAIR-500 images and checked each description against the image. The model correctly identified the main objects and produced incorrect details about their relations, parts, or markings. Three LLaVA-OneVision-1.5-4B caption pairs illustrate this pattern.

  • Spatial relation. Both captions describe a truck carrying a concrete structure that stands behind it (COCO 376677).
  • Part recognition. Both captions describe the blue insert in a urinal as a seat (COCO 34830).
  • New reading error. SPEAR adds the bus number 2703. The number visible in the image is 1208 (COCO 363927).

Recalibration draws on cues already encoded by the model and covered by the attention proposal. Missing or misread cues limit the evidence for separating candidates. Strong language priors may also persist under the bounded update. The paired captions show how these errors appear in complete outputs. Figure 5(b) reports gains over greedy decoding across eight MMHal-Bench question categories.

Q5. What quality does SPEAR achieve at comparable decoding latency?

Response 5. On Qwen3-VL-2B COCO captioning, each SPEAR step batches gradients for the top eight candidates, scores regional evidence, and updates probabilities under a KL budget. The implementation reuses the KV cache. We calibrated every 1, 2, or 4 steps starting at step 1. Skipped steps decode from the model's current distribution.

We measured inter-token latency on 1,000 COCO images. We evaluated caption quality on CHAIR-500 with up to 256 new tokens per caption.

MethodInter-token latency ↓ (ms/token)CHAIR-S ↓ (%)CHAIR-I ↓ (%)
Greedy27.039.8012.18
LVLMs-Saliency (Qwen adaptation)694.337.208.52
Full SPEAR71.030.805.52
SPEAR every 2 steps53.4232.606.25
SPEAR every 4 steps41.3333.807.42
VCD (greedy adaptation)55.0140.209.27
VCD (native sampling)54.738.408.58

Full SPEAR takes 2.63× as long per token as greedy and 89.8% less time per token than Qwen-adapted SGRS+LocoRE (71.0 vs 694.3 ms/token). Both methods use gradients. Saliency retains its sampling procedure, and SPEAR selects greedily after calibration. On CHAIR-500, full SPEAR lowers CHAIR-S/I by 6.40/3.00 points relative to Saliency.

Every-two-step SPEAR is close to greedy VCD in latency (53.42 vs 55.01 ms/token). In the CHAIR evaluation it has lower rates than both reported VCD variants, with increases of 1.80/0.73 points over full SPEAR. Every-four-step calibration lowers latency by a further 22.6% relative to every-two-step calibration and raises CHAIR-S/I by 1.20/1.17 points, for total increases of 3.00/1.90 points over full SPEAR. Calibration frequency thus provides a direct way to trade decoding time for caption quality. The two-step schedule stays closer to full SPEAR's CHAIR scores at latency comparable to VCD.

Minor comments. Revised Section 3 distinguishes offline reference–rival probes in annotated boxes from online comparisons among top-eight candidates within attention proposals. Equation (4) gives the local margin response ΔΓ, and Section 3.2 defines the evidence-selection score D. Response 4 includes persistent and newly introduced errors.

Dear Authors,

Thank you for your detailed and thoughtful response. I have a few additional suggestions. There still appears to be some room in the main paper to show your muscle, and I encourage you to make full use of the available space. It would also be helpful to clearly mark the revised sections so that reviewers and the AC can easily identify the changes.

Since my current rating is already positive, I will keep it unchanged for now and follow the discussions and feedback from other reviewers and authors' paper revisions.

Good Luck.

Dear Reviewer PqJE,

Thank you for taking the time to read our response and share these suggestions. If time permits, we’ll consider using the remaining space in the main paper to expand on the points raised during the discussion and present the most relevant supporting results. We’ll also clearly mark the revised sections so that the reviewers and the Area Chair can easily find the changes.

Best regards,
The Authors

Reviewer_DGh1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper aims to explain why LVLMs still produce wrong responses even when their attention concentrates on the answer-relevant region, by examining how each visual token affects the probabilities of candidate tokens. The authors define a response score Γ\Gamma to measure this effect and report two observations from their TextVQA analysis: tokens within the region provide conflicting evidence for the correct answer and its competitor, and inhibitory support tokens, which indirectly support the correct answer by suppressing the competitor, drop sharply in incorrect answers. Building on this, they propose SPEAR, a training-free decoding method that prioritizes suppressive evidence during token selection and recalibrates candidate probabilities within a KL divergence.

Critical strengths
  • Starting from the observation that hallucinations occur even when the model attends well to the relevant image region, the paper argues that visual attention alone is insufficient to distinguish hallucinated outputs and proposes a new way to measure the effect of each visual token on the output token.
  • The experiments cover a broad scope (model families, evaluation benchmarks, etc.), and the analysis is conducted in detail.
Critical weaknesses
  • Hard to follow. Some terms are used before they are defined (e.g., "candidate-centered score" in the introduction), key analysis results are placed in the appendix (e.g., Section 2.3), and the notation used in the equations cannot be understood from the main text alone. Together, these made it difficult to follow the storyline.
    • In particular, the values defined in each equation are not explained in terms of what they mean; the text mostly repeats definitions. As a result, it was hard to understand why each quantity is defined and how it is used in the analysis or the method.
    • The authors appear to have conducted several in-depth analyses, but the presentation does not do them justice.
  • The analysis in Figure 3 may reflect a distributional difference between model-favored and model-unfavored candidates, rather than a difference between correct and incorrect answers.
    • The paper treats g as the correct answer and r as the incorrect one. Under the view above, however, g is the model-favored candidate in the first row and the model-unfavored candidate in the second row.
    • Γ(g)\Gamma(g) can be rewritten as Γ(g)=πr⋅ΔΓ(g,r)+∑k≠g,rπk(∇zg−∇zk)\Gamma(g) = \pi_r \cdot \Delta\Gamma(g,r) + \sum_{k \neq g,r} \pi_k(\nabla z_g - \nabla z_k) This means that how often Γ(g)\Gamma(g) and ΔΓ\Delta\Gamma share the same sign depends on how large πr\pi_r is.
      • When πr\pi_r is large, Γ(g)≈ΔΓ(g,r)\Gamma(g) \approx \Delta\Gamma(g,r), so the two values would share the same sign in most cases.
    • In the model-favored case, πr\pi_r is necessarily small, whereas in the model-unfavored case, πr\pi_r is necessarily large. This may be what produces the result in Figure 3. The fact that role III decreases along with role II also seems to support this interpretation.
    • Therefore, this result cannot be regarded as a characteristic of hallucination.
Questions for authors

Please refer to the weaknesses section. It would be nice to provide additional experiments related to the second point.

4 comments

Thank you for taking the time to review our paper and for your thoughtful questions and suggestions. We respond to each point below.

Q1. Candidate-centered score and connection to SPEAR

At a fixed decoding step, let CC be the natural top-8 candidate set, zc(t)z_c(t) the logit of candidate cc under perturbation tt, and πk\pi_k the initial softmax probability of kk within CC. The candidate-centered score is mc(t)=zc(t)−∑k∈Cπkzk(t)m_c(t)=z_c(t)-\sum_{k\in C}\pi_k z_k(t). The weights stay fixed at their initial values. Subtracting the weighted mean cancels common logit shifts and measures how cc moves relative to the baseline candidate distribution.

Let oo be an attention head's output, the attention-weighted sum of token value vectors. To probe visual token jj with value vjv_j, we shift the output via o(t)=o+t(vj−o)o(t)=o+t(v_j-o), defining the directional response Γ(c)=mc′(0)\Gamma(c)=m'_c(0). For c∈Cc\in C, Γ(c)\Gamma(c) equals the derivative at t=0t=0 of log⁡qc(t)\log q_c(t), where qc(t)q_c(t) is the softmax probability over CC after perturbation. For reference candidate gg and rival rr, the pairwise response ΔΓ(g,r)=Γ(g)−Γ(r)\Delta\Gamma(g,r)=\Gamma(g)-\Gamma(r) cancels the shared center and equals the initial derivative of the logit margin zg(t)−zr(t)z_g(t)-z_r(t) and the log probability ratio. Thus, Γ(g)\Gamma(g) tracks whether a visual token lifts gg relative to the active pool, and ΔΓ(g,r)\Delta\Gamma(g,r) tracks separation of gg from rr.

The Introduction defines the centered score and logit margin at first use. Section 2.2 explains what Eqs. (1)–(4) measure. Section 2.3 explains how Jnet=Jtarget−JbaseJ_{\mathrm{net}}=J_{\mathrm{target}}-J_{\mathrm{base}} compares the margin responses under target and original attention weights with fixed regional mass and outside weights. It summarizes the interventions, with curves in Appendix A.3. Sections 3 and 3.2 distinguish annotated diagnosis from annotation-free inference. SPEAR proposes regions using attention and selects visual evidence using normalized differences DD. We marked the revised passages in blue.

Q2. Candidate ranking, sign agreement, and Role II/III shares

The reviewer's decomposition is exact for g,r∈Cg,r\in C. To infer sign agreement, we also need the token's directional responses. Let ac=zc′(0)a_c=z'_c(0), d=ΔΓ(g,r)=ag−ard=\Delta\Gamma(g,r)=a_g-a_r, and R=∑k∈C∖g,rπk(ag−ak)R=\sum_{k\in C\setminus\\{g,r\\}}\pi_k(a_g-a_k). Then Γ(g)=πrd+R,Γ(g)−d=(πr−1)d+R.\Gamma(g)=\pi_r d+R, \qquad \Gamma(g)-d=(\pi_r-1)d+R. Rank gives the order of probabilities. A top-ranked rival can lead narrowly or hold most of the probability mass. Approximation depends on πr\pi_r, the size of dd, and RR. For d≠0d\ne0, the signs agree exactly when πr+R/d>0\pi_r+R/d>0. With bounded logit derivatives, πr→1\pi_r\to1 makes the absolute error tend to zero. At a fixed high πr\pi_r, a small dd can leave RR dominant.

Consider two hypothetical cases with identical candidate probabilities. Fix πg=0.1\pi_g=0.1, πr=0.8\pi_r=0.8, and combined weight 0.10.1 across the remaining six candidates, with ag=0a_g=0 and ak=0.2a_k=0.2 for each of those six, giving R=−0.02R=-0.02. Setting ar=−0.01a_r=-0.01 yields d=0.01d=0.01 and Γ(g)=0.8(0.01)−0.02=−0.012\Gamma(g)=0.8(0.01)-0.02=-0.012, giving opposite signs (Role II). Keeping the exact same weights and setting ar=−0.10a_r=-0.10 yields d=0.10d=0.10 and Γ(g)=0.8(0.10)−0.02=0.06\Gamma(g)=0.8(0.10)-0.02=0.06, giving matching positive signs (Role I).

Role III requires d<0d<0 and R>πr∣d∣R>\pi_r|d|. Its lower share in incorrect answers is consistent with a weighting effect. Explaining that decline requires knowing how dd, RR, and πr\pi_r vary together. We test whether removing Role II tokens affects complete answers, whether perturbing them from the same starting state changes logits and probabilities as predicted, and whether their lower density in incorrect answers recurs in equal-size samples.

Experiment 1: Masking Role II during answer generation. Using initially correct Qwen3-VL-2B predictions from the fixed 1,000-question TextVQA_GT_bbox pool, we assigned token roles at the first unmasked answer position, set attention logits of selected in-box visual keys to −∞-\infty before softmax, and regenerated full answers. Regional shares describe the original correct diagnostic cohorts. Mask counts and scores are from Qwen.

RoleQwen (%)Ministral (%)LLaVA (%)Mean masked (Qwen)VQA score (Qwen)
Unmasked———0100.00%
I57.1443.7541.27——
II13.1913.7613.695.9483.44%
III6.238.049.692.7198.94%
IV23.4534.4535.3510.3597.16%
III+IV29.6842.4945.0413.0796.40%

Role II masking lowers VQA from 100.00% to 83.44%. Masking the larger IV and III+IV sets produces smaller declines. The small declines from masking III or IV may reflect loss of visual context or attention redistribution during generation. These results establish the contribution of Role II evidence to complete answers in this cohort. Experiment 2 measures its effect on candidate probabilities.

Experiment 2. Does Role II produce the predicted probability changes? We use Qwen3-VL-2B in FP32 on the fixed 1,000-question TextVQA_GT_bbox subset. At the first answer prediction, each intervention starts from the same unmodified state for that question, with the same baseline candidate probabilities and role labels. We fix g,rg,r and CC for measurement and probe each in-box visual token from Roles I–IV separately. For token jj, we add ϵ(vj−o)/28\epsilon(v_j-o)/28 to every attention head's output across all 28 language-model layers and rerun the forward pass. We use ϵ=0.01\epsilon=0.01 and check sensitivity at ϵ=0.001\epsilon=0.001.

We measure the changes in the candidate logits and their margin, then recompute conditional probabilities using softmax over CC. We use a log-probability tolerance of 10−610^{-6} and evaluate conditions involving both candidates' probabilities on questions with g,r∈Cg,r\in C. We compute match rates within each question and average across questions equally. The table reports results at ϵ=0.01\epsilon=0.01.

RoleExpected responseFull role match (%)Margin sign match (%)Near-zero gold response (%)
IGold rises and margin rises64.7699.9835.23
IIGold shows no clear increase, rival falls, and margin rises99.5399.53—
IIIGold rises and margin falls48.0499.7551.45
IVGold shows no clear increase and margin falls99.9999.99—

Role II meets its full criteria at a question-averaged rate of 99.53%. In matching responses, the rival probability falls and the gold-to-rival probability ratio rises. When both probabilities fall, the rival's relative decrease is larger. For Roles I and III, near-zero gold responses account for much of the gap between full role match and margin-sign agreement. Role IV reaches 99.99% full role match. These forward passes show that perturbing visual tokens from the same starting state produces the predicted changes in candidate competition.

Experiment 3. Does the lower Role II density in incorrect answers persist under equal sample sizes and forward-pass verification? On the same FP32 benchmark, we keep each question's original candidate weights and baseline role labels. We divide each count by the total number of in-box visual tokens for that question, then average questions equally, including those with zero density. For each of three random seeds, we draw 100 correct and 100 incorrect analyzable questions without replacement within each draw. Draws may overlap.

SeedCorrect (%)Incorrect (%)Correct minus incorrect (pp)
4211.326.734.60
4315.756.958.80
4415.636.938.69
Mean14.236.877.36

Incorrect answers have lower Role II density in all three equal-size draws, with an average gap of 7.36 percentage points (14.23% vs. 6.87%). In a separate analysis of questions with g,r∈Cg,r\in C, we count which baseline Role II tokens meet the full response criteria in the forward pass at ϵ=0.01\epsilon=0.01. The denominator remains all in-box visual tokens for each question.

Role II density criterionCorrectIncorrect
Baseline Role II label12.75%7.61%
Baseline Role II with the full observed Role II response12.74%7.61%

The density gap persists when we count only baseline Role II tokens that produce the full predicted response (12.74% vs. 7.61%). The lower density of tokens that suppress the rival and improve the gold candidate's relative position in incorrect answers motivates SPEAR's priority on suppressive evidence.

Dear Reviewer DGh1,

Thank you for your thoughtful comments on our paper. With one day left in the discussion period, we’d be grateful to hear your thoughts on the clarifications we provided. If you have any remaining questions or concerns, please let us know. We’d be happy to address them before the discussion closes.

Best regards,
The Authors

Thank you for the detailed response and the additional experiments.

Q1. I have checked the revised paper, and I think it reads more easily than before.

Q2. The point of my original review concerns the paper's central claim that "Role II (inhibitory support) tokens are deficient in hallucinated answers." As I understand it, the analysis captures a difference based on whether the reference candidate g is model-favored or model-unfavored, rather than whether the answer is hallucinated, and I expected clarification on this point. The following issues remain unresolved in the current response and additional experiments.

  1. The authors acknowledged the decomposition in my review and presented the sign-agreement condition π_r + R/d > 0. However, this condition directly shows that as π_r grows, the region of (d, R) that yields Role II shrinks. The numerical example shows only a single point where Role II can exist at π_r = 0.8; it does not show that the frequency difference between the two groups is not explained by differences in π_r.
  2. The remark that "rank does not determine the size of the probability" is correct, but then the actual distribution of π_r at the analyzed position should be shown for both groups. In hallucination settings, models are usually highly confident in the wrong answer, so I believe a large π_r is closer to the default assumption.
  3. Regarding the decline of Role III, the authors acknowledged that it "is consistent with a weighting effect." Since Role II and Role III disappear through the same mechanism (sign disagreement between Γ(g) and ΔΓ), there is no basis for explaining Role III by a π_r effect while claiming that Role II is not.
  4. None of the three additional experiments controls for π_r. Experiment 1 is conducted only on the correct group, so it provides no information about the cause of the difference between the groups, and since the comparison groups are only the roles that favor r (III, IV), a comparison against Role I is missing. Experiment 2 shows that the derivative predicts finite perturbations well, which is unrelated to why the sign comes out the way it does. Experiment 3 matches the sample sizes, but the π_r confound operates within each question, so it is unrelated to sample size.
  5. The Role II proportion for incorrect answers was 0.62% in the original cohort but rose by more than 10x to 6.87–7.61% in the new cohort, and there is no explanation for this difference. If this is the result of excluding cases with extreme π_r due to the g, r ∈ C restriction, it would rather be evidence supporting my concern. If I have misunderstood, I would appreciate a clarification.

To resolve this, please conduct the same analysis with the roles swapped. That is, in each group, set r (the generated/favored token) as the reference and g as the competitor, apply the same four-role classification, and report the Role II proportion.

  • If the hypothesis I proposed is correct: in the incorrect group, taking the hallucinated token r as the reference will raise the "Role II" proportion to the level of g in the correct group, and in the correct group, taking the unfavored token r as the reference will lower it. That is, the pattern follows favored/unfavored rather than correctness.
  • If the paper's claim is correct: the hallucinated token should not receive abundant inhibitory support even when taken as the reference, and the difference in Role II proportion should be separated from favored/unfavored and follow correctness.

Additionally, the following results would also be helpful.

  • How the probability of the generated token at the analyzed position (π_r or top-1 probability) is distributed for the correct/incorrect groups.
  • Binning π_r (or top-1 probability) and comparing the Role II proportion of correct vs. incorrect within the same bin.

Without such results, it is difficult to interpret the decline of Role II as a characteristic of hallucination, and since this is directly involved in the paper's central claim and the motivation of SPEAR, I keep my current score.

Reviewer_nQALOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper set out to analyze the phenomenon of Large Vision Language Models hallucinating information even though they allocated attention on the right regions. The authors proposed gradient-based metric, Γ\Gamma, measuring how a visual token locally affects a candidate's prediction score. They proposed a classification of visual tokens within the relevant region into 4 different types: supporting the true token directly, supporting the true token indirectly through suppressing other false candidate tokens (inhibitory support), supporting both true and false tokens but favor false tokens more, and supporting false tokens directly. They found that among those, inhibitory support tokens are much more present in true responses than hallucinated ones. Building on this insight, they introduced the training-free intervention method SPEAR. SPEAR first proposes the relevant region through pooling tokens with the highest mid-to-late layers attention weights, create a set of candidate tokens for each of the 4 role within the relevant region. It then aggregates the signed evidence signals of each token against the current top candidate and recalibrate the candidate tokens' probabilities with KL-divergence. SPEAR achieves state-of-the-art results in Hallucination Mitigation across multiple benchmarks and LVLMs.

Critical strengths
  • Extensive experimental results: The paper conducted extensive comparative experiments on multiple Hallucination benchmarks on a variety of LVLM architectures. The results found SPEAR to have competitive results across many prediction tasks and models, proving the effectiveness of the proposed method
  • Interesting framework to analyze a known Hallucination phenomenon: Framing LVLM grounding as a "which predictions do this token support and suppress" with gradients is an interesting framework for analyzing visual grounding in hallucinatory cases in contrast with other traditional attention weights approaches. The 4 roles classification system for a token's evidence is intuitive.
Critical weaknesses
  • W1 Confusing and hard to follow writing. The authors use many unconventional key terms without formal definitions, or only defined after it's been used. The wording is overly complicated and hard to follow. Many central terms appear before they are defined, mathematically or in text, and some are never defined at all. I believe the paper in this current state is not ready to be published, and require major revisions in writing. To name a few:
    "Current read" is used in the Introduction (line 47) and Section 2.2 (line 136) without direct explanation. It is then called "complete read" (line 146) without any notice. I believe this term is used as the attention head's output, but no such explanation was provided. "Candidates" are used in place of tokens, without any previous clarifications. "Candidate ranking" and "candidate sensitivity" appear before the candidate set C is defined. "Candidate sensitivity" is never formally defined. "Candidate-centered score's local change" appears before the centered score Equation (1). It is never described plainly as a directional derivative of the centered logit. "Advantage" and "mean reference/competitor margin" (Introduction, Section 2.3) are never formally defined. Equation (4) implies the margin is z_g − z_r. The phrases "gold" and "rival" are used for true tokens and competing false tokens, but without any elaboration. The word "event" is used throughout (line 129,140,178,199) and was not elaborated on its meaning in the paper. I assume it to be a sample in this context. The constant use of unconventional jargon without inline explanation makes the paper difficult to comprehend and evaluate their technical correctness efficiently.

  • W2 Unusually high gains in MME performance: The authors reported the MME results of Qwen 3-VL-2B. The results in this table seems quite abnormal due to high improvements made by VLM hallucination mitigation methods over the greedy baselines. Many previous existing works have run experiments on MME for multiple LVLMs (LLaVA-1.5-7B, Qwen-VL) and the gap between methods like VCD, OPERA... are usually small. For example, the MME results in [1] was VCD 648.3, Regular 618.3 for Qwen-VL, a 5% improvement; OPERA 592.33, DeCO 611.6 compared with Regular 565.33 [2] with LLaVA-1.5-7B, about 5% and 8% improvements ; HALC 579.58 compared with Regular 562.50 for LLaVA-1.5-7B, improved by 3% [3]. Even though these are older and less capable models, the patterns established are a modest gain (around 5-10%) compared with regular decoding for all inference time method due to their softer effects on the model's logits and other components. So the big gap between VLM Decoding methods and Greedy Decoding (from 1333.75 to 1981.03 for VCD, more than 48 % improvement) seems hard to believe, and signify risk in potential errors in evaluation protocol. I would like the author to clarify the input prompt, the generation settings, evaluation detail, and the dataset split they used for this experiment. If the implementation details are correct, then explanation for the unusually large performance gap is needed.

[1] Zou, Xin, et al. "Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models." Proceedings of the 42nd International Conference on Machine Learning, vol. 267, PMLR, 2025, pp. 80873–99, proceedings.mlr.press/v267/zou25e.html. [2] Yin, Jianghao, et al. "Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models." The Fourteenth International Conference on Learning Representations, 2026, openreview.net/forum?id=YtWZdwEG5K. [3] Wan, Zifu, et al. "Only: One-layer intervention sufficiently mitigates hallucinations in large vision-language models." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

Questions for authors
  • Q1 Further implementation details and explanation for MME results: As specified in W2, the current results in MME seem anomalous and signify potential evaluation errors somewhere. If not then explanations are needed on why the author's reported results are substantially different from previous works.
  • Q2 Candidate gradients Γ\Gamma novelty with other approaches: The idea of fusing a token's attention with their gradients seem an established paradigm to evaluate a token's contributions towards a prediction [4],[5]. I would like further analysis on the proposed candidate gradients Γ\Gamma on how they differ from these approaches in terms of intuition and mathematical properties. Can these existing approaches be used in place of Γ\Gamma in the main method or in analysis experiments like Figure 3. An ablation results for this would be appreciated.
  • Q3 Does improved performance in CHAIR lead to degraded textual quality: A known phenomenon in Hallucination Mitigation is that Mitigation methods modifying LVLM's output logits can suppress the mentions of hallucinated objects efficiently, but coming with a cost of textual diversity and quality with shorter length [6],[7]. I would like to know the generation quality metrics of SPEAR compared with other methods (object recall, average length, LLM-assisted evaluation) to make sure there's a good balance between hallucination mitigation and text quality
  • Q4 Qualitative samples of candidate gradients: I would like the authors to elaborate on what the image and text of the 4 different types of errors look like? Are there any established patterns for these types of errors (object type, size, color...) correlated to the candidate gradients values?

[4] Chefer, Hila, Shir Gur, and Lior Wolf. "Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers." 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021.
[5] Zhang, Xiaofeng, et al. "Hallucination Begins Where Saliency Drops." The Fourteenth International Conference on Learning Representations, 2026, openreview.net/forum?id=sjnErRHXf3.
[6] Xing, Yun, et al. "Mitigating object hallucination via concentric causal attention." Advances in neural information processing systems 37 (2024): 92012-92035.
[7] Min, Kyungmin, et al. "Mitigating hallucinations in large vision-language models via summary-guided decoding." Findings of the Association for Computational Linguistics: NAACL 2025. 2025.

5 comments

Thank you for taking the time to review our paper and for your thoughtful questions. Your comments helped us identify where the terminology and evaluation setup needed clearer explanations. We revised the manuscript and marked the wording changes in blue.

W1. Clarifying the terminology and the flow of the paper

  • Introduction, lines 44–48, and Section 2.1, line 101. A candidate is a possible next token. A logit margin is the difference between two candidate logits. The current read is the attention head’s output, the weighted mixture of its token value vectors. An event is the prediction at an evaluated answer position.
  • Section 2.2, lines 105–106 and 132–158. We rank possible next tokens by their original logits and take the top eight as the natural candidate set CC. A candidate’s centered logit mcm_c is its logit minus the weighted mean over CC, using the initial candidate probabilities as frozen weights. This centering removes shifts shared by all candidates in CC. Candidate sensitivity is the gradient of mcm_c with respect to the head output oo. The score Γj(c)\Gamma_j(c) is the directional derivative of mcm_c as oo moves toward visual token jj’s value vjv_j. For c∈Cc\in C, it equals the local derivative of the candidate’s conditional log probability.
  • Sections 2.2–2.3, lines 154–158 and 180–189. The gold token gg is compatible with the reference answer, and the rival rr is its designated competitor. The advantage of gg over rr is the margin zg−zrz_g-z_r. The mean reference–competitor margin averages this quantity over events. The response difference ΔΓj(g,r)=Γj(g)−Γj(r)\Delta\Gamma_j(g,r)=\Gamma_j(g)-\Gamma_j(r) measures how the perturbation changes that margin locally. In Section 2.3, Jtarget−JbaseJ_{\rm target}-J_{\rm base} measures the local margin response to redistributing a fixed amount of attention within the relevant region.
  • Sections 3–3.2, lines 236–238 and 257–263. The diagnostic analysis uses the reference and rival in an annotated region. During decoding, SPEAR proposes a region from attention, compares each top-eight challenger with the model’s current winner, and selects evidence using the layer-normalized response difference Dj(a,b)D_j(a,b). It then recalibrates candidate probabilities under a KL constraint.

In practical terms, Γ\Gamma asks what happens to each candidate when the head gives more weight to one visual token. ΔΓ\Delta\Gamma asks which candidate gains relative to the other. The two signs together distinguish direct support from support through stronger suppression of a rival.

Q1 / W2. Why was the original MME gap so large, and how was MME evaluated?

Thank you for flagging the discrepancy. We corrected the MME baselines in Tables 3 and 4. We reviewed the original MME baseline evaluation code and found that Thinking mode was enabled for both Qwen3-VL-2B and Qwen3-VL-4B. In this configuration, intermediate reasoning and the final answer shared a single generation token budget. If the model reached the token limit during reasoning before producing a recognizable “yes” or “no,” the official MME scorer counted the item as incorrect. Such truncation may have lowered the original baseline scores.

Updated MME scores from Table 4, Qwen3-VL-2B

MethodPerceptionCognitionTotal
Greedy1355.81469.371825.18
Beam search1342.70464.581807.28
Nucleus sampling1362.53472.431834.96
VCD1455.31525.721981.03
SPEAR1526.30508.212034.51

With the corrected Greedy score of 1825.18, the VCD gap falls from 48.53% to 8.54%. The SPEAR–Greedy gap in the revised table is 209.33 points (11.47%), with Perception accounting for 170.49 points. The comparisons cited in the review cover four hallucination subtasks with a maximum of 800 points. Table 4 covers all 14 MME subtasks with a maximum of 2,800 points. MemVR, Table 8 · DMAS, Table 1 · ONLY, Table 3

Evaluation protocol. We also ran a separate check on the complete official MME release, covering 2,374 questions across 10 perception and four cognition subtasks. We used Qwen3-VL-2B-Instruct in BF16, its native image processor and chat template, the original questions ending in “Please answer yes or no,” and a cap of 32 new tokens. Greedy, the VCD adaptation, and SPEAR all used greedy token selection in this check. The official scorer sums question accuracy and paired-image accuracy for each subtask, with paired credit requiring both questions on the image to be correct. SPEAR achieved the highest total score in this separate check. The scores above are from the revised Table 4. MME scoring protocol

Q2. How does Γ\Gamma relate to existing gradient attribution methods, and can their scores substitute for it?

Γ\Gamma builds on gradient-based attribution. Our contribution is to define an explicit intervention for each visual token in a relevant region, decompose its effect on candidate competition, and use that decomposition in decoding. For a head output o=AVo=AV, the intervention shifts the current mixture toward vjv_j through o(t)=(1−t)o+tvjo(t)=(1-t)o+tv_j. Then Γj(c)=⟨∇omc,vj−o⟩\Gamma_j(c)=\langle\nabla_o m_c,v_j-o\rangle measures the signed response of candidate cc to that shift. For a candidate in the fixed set CC, this is the local derivative of its conditional log probability. The difference ΔΓj(g,r)\Delta\Gamma_j(g,r) gives the local derivative of the pair’s logit margin and log probability ratio. SPEAR uses these signed effects to select visual evidence and recalibrate candidate probabilities.

  • Chefer et al., Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers (ICCV 2021), propagate positive gradient-weighted attention contributions across layers to construct relevance maps. Paper
  • Zhang et al., Hallucination Begins Where Saliency Drops (ICLR 2026), take the absolute attention–gradient product to measure the strength of candidate saliency. Paper
  • Cheng et al., Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating (ICML 2026), use signed directional effects to estimate the impact of gating the visual or textual route as a whole. Paper

What can be substituted. Chefer et al.’s [1] relevance maps retain positive contributions, and LVLMs-Saliency [2] uses absolute attention–gradient products to retain magnitudes. Both scores help locate influential tokens. In an absolute-value map, equally strong increases and decreases receive the same score. Causal Route Gating [3] retains the sign of a route’s effect, with each score combining the effects of multiple tokens under a whole-route intervention. Assigning a role with Γ\Gamma requires the signed response of each candidate when the head output shifts toward an individual visual token. Role II illustrates why both details matter. Its reference score shows no clear increase, and stronger suppression of the rival widens the reference’s margin. Direct substitution with the final scores above would discard the response direction or the token-level detail required to identify this role. The forward perturbation experiment below tests whether the assigned roles match the measured effects of the specified intervention.

Forward perturbation test. On TextVQA_GT_bbox, we ran Qwen3-VL-2B in FP32 and fixed the prefix, candidate set, reference–rival pair, and baseline roles at the first answer prediction. Starting from the same unperturbed state each time, we shifted the head outputs toward one visual token in the annotated region, adding ϵ(vj−o)/28\epsilon(v_j-o)/28 to each attention head across all 28 language-model layers. We reran the forward computation and measured the change in the reference–rival logit margin and the conditional probabilities obtained from the new logits. The table reports ϵ=0.01\epsilon=0.01, with a 10−610^{-6} tolerance on finite log responses. We also checked ϵ=0.001\epsilon=0.001 for step-size sensitivity. Rates are averaged within each question and then equally across questions. Joint conditional-probability checks use questions with both candidates in CC.

Q2. Forward perturbation results (continued)

RoleExpected effectFull role matchMargin direction match
IReference rises and margin widens64.76%99.98%
IIReference shows no clear rise, rival falls, and margin widens99.53%99.53%
IIIReference rises and margin narrows48.04%99.75%
IVReference shows no clear rise and margin narrows99.99%99.99%

The measured margin moves in the predicted direction in over 99.5% of cases for every role, on the question-averaged measure. Role II meets its full prediction 99.53% of the time. When both probabilities fall in a matching Role II intervention, the rival falls more in proportional terms, so the reference gains relative to it. For Roles I and III, 35.23% and 51.45% of reference responses, respectively, fall within the near-zero tolerance, showing that the reference’s increase may fall below the threshold for a full role match even when the margin moves in the predicted direction. The test validates the local competition effects at the first answer step.

References

[1] Hila Chefer, Shir Gur, and Lior Wolf. “Generic Attention-Model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers.” ICCV, 2021, pp. 397–406. Paper

[2] Xiaofeng Zhang et al. “Hallucination Begins Where Saliency Drops.” ICLR, 2026. Paper

[3] Zhe Cheng, Wenyu Chen, Fode Zhang, and Dehuan Shen. “Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating.” ICML, 2026, pp. 18828–18860. Paper

Q3. Does reducing hallucination affect caption coverage or text quality?

We evaluated captions on the fixed CHAIR-500 set from COCO val2014 using the prompt “Please describe this image in detail.” Greedy and SPEAR used the same images, BF16, greedy token selection, and a cap of 256 new tokens with early stopping at EOS.

ModelMethodObject recall ↑Mean wordsCHAIR-S ↓CHAIR-I ↓
Qwen3-VL-2BGreedy67.65%206.0339.8012.18
Qwen3-VL-2BSPEAR70.90%206.2730.805.52
Qwen3-VL-2BVCD68.78%205.4738.408.58
Qwen3-VL-2BDeCo64.70%210.11436.806.61
LLaVA-OV-1.5-4BGreedy66.73%147.0232.007.77
LLaVA-OV-1.5-4BSPEAR71.72%144.8726.205.06
LLaVA-OV-1.5-4BVCD71.06%144.57028.36.83
LLaVA-OV-1.5-4BDeCo67.32%171.21027.05.37

Relative to Greedy, SPEAR raises object recall by 3.25 percentage points on Qwen and 4.99 points on LLaVA while lowering both CHAIR rates. Mean length changes by +0.24 and −2.15 words, respectively. In this comparison, lower hallucination rates coincide with greater object coverage and similar caption length.

Q4. What do the four visual-token roles look like?

The paper’s four-way classification applies specifically to the effects of individual visual tokens on competing answer candidates. By “four different types of errors” in images and text, do you mean the four visual-token roles in Table 1?

Let gg be the reference-compatible candidate and rr its rival. Γj(c)\Gamma_j(c) measures how the centered logit for candidate cc changes locally when the attention-head output shifts toward visual token jj. For cc in the fixed candidate set CC, this is the directional derivative of its conditional log probability. The response difference ΔΓj(g,r)=Γj(g)−Γj(r)\Delta\Gamma_j(g,r)=\Gamma_j(g)-\Gamma_j(r) measures how the margin zg−zrz_g-z_r changes locally. Positive values favor gg, and negative values favor rr. Table 1 uses the signs of Γj(g)\Gamma_j(g) and ΔΓj(g,r)\Delta\Gamma_j(g,r), with a numerical tolerance, to define four roles.

  • I. Direct support. The reference score rises and its margin widens.
  • II. Inhibitory support. The reference score shows no clear rise. Stronger rival suppression widens the margin.
  • III. Greater rival gain. The reference score rises. The rival gains more, narrowing the margin.
  • IV. Rival preference. The reference score shows no clear rise and its margin narrows.

Figure 3 compares the four roles across correct and incorrect answers. Incorrect answers have a lower in-region share of Role II, with direct support also present. This observation motivates suppression-prioritized evidence selection in SPEAR. During decoding, SPEAR uses the normalized, aggregated response difference Dj(a,b)D_j(a,b) to select evidence for each candidate pair. Testing whether the four roles vary with object category, size, color, or spatial relations requires annotations for those attributes.

Comments on the paper

1 comment

We thank the reviewers for their helpful comments. We have revised the manuscript to clarify the definitions and reasoning and added validation and ablation experiments on inhibitory support. The main updates are summarized below.

  1. Clearer motivation, definitions, and mathematical explanations (Introduction and Sections 2–3). We clarify the research question of why hallucinations persist when attention concentrates on answer-relevant regions. We define key terms and symbols at first use, including next-token candidates, attention-head outputs, analysis events, reference and rival tokens, and the logit margin. Section 2.2 explains why candidate scores are centered, how the visual-token perturbation is defined, and what the signs of Γ and ΔΓ mean. Detailed derivations and implementation formulas are placed in the appendix to keep the main presentation focused.

  2. Additional validation of inhibitory support (Section 2.3, Figure 4, and Appendix A.5). We add three complementary analyses. Forward perturbations confirm the predicted rival suppression and margin increase for Role II at a question-averaged rate of 99.53%. Across three draws with equal numbers of correct and incorrect questions, mean Role II density is 14.23% versus 6.87%. On initially correct answers, masking Role II visual tokens and regenerating complete answers reduces the VQA score from 100% to 83.44%. These results connect the observed role distribution with measurable effects on candidate competition and complete answers.

  3. Additional ablations on evidence selection and the balance between support and suppression (Section 4.4, Tables 7–8). Table 7 compares SPEAR with equal-rate selection and a direct-support replacement control that fills suppression slots with additional support tokens. SPEAR lowers CHAIR-I by 3.00 and 3.59 percentage points relative to replacement on Qwen and LLaVA, respectively. Table 8 examines reallocating 10%, 30%, and 50% of the selection quota from support to suppression at a fixed total token count. These settings increase hallucination rates and reduce object recall, supporting the importance of retaining both types of evidence.

Main-text revisions are highlighted in blue. Newly added references and their corresponding in-text citations are highlighted in green.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.