acceptodds
Under review as a conference paper at ICLR 2027

Attention at Rest Stays at Rest: Breaking Visual Inertia to Mitigate Relation Hallucinations

Abstract

While multimodal large language models demonstrate strong entity-level perception, faithfully grounding relational interactions between objects remains a persistent challenge. Although conventional visual grounding techniques attempt to resolve hallucinations by amplifying visual attention, strengthening overall visual signals fails to reliably correct relational errors. Tracing visual attention in relation descriptions reveals that correct responses tend to dynamically shift focus across regions, whereas hallucinated responses often linger on previously dominant evidence, exhibiting an undesirable visual inertia. Further analysis shows that relation-prediction performance steadily deteriorates as more previous-step visual attention is carried into the current decoding step. We therefore introduce Inertia-aware Visual Excitation (IVE), an MLLM decoding method that dynamically recalibrates visual values using token-level attention history. By contrasting current attention against recent moving averages, IVE separates emergent tokens with rising relevance from persistently dominant inertia tokens, selectively reinforcing newly needed evidence while mildly attenuating contributions from repeatedly attended regions. Across three MLLMs and decoding strategies, IVE reduces relation hallucinations while preserving broader multimodal performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Relation Hallucination Mitigation, Attention Activeness, Visual Inertia

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VfK5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies relation hallucinations in multimodal large language models. The authors observe that hallucinated relation descriptions tend to retain similar visual attention distributions across decoding positions, whereas correct descriptions exhibit larger changes. They further show that increasing previous-step attention carryover degrades relation prediction more than an entropy-matched current-step control. Motivated by these observations, the paper proposes Inertia-aware Visual Excitation (IVE), which compares each visual token's current attention with its recent history, reinforces tokens with rising attention, and attenuates repeatedly dominant tokens. The intervention rescales visual values from fixed original values without parameter updates. Experiments across three 7B backbones show consistent improvements on relation benchmarks, with additional evaluations of general multimodal performance and inference cost. I find the method useful, although the connection between the proposed inertia mechanism and the main performance gains needs further clarification.

Critical strengths
  1. The paper addresses a meaningful problem. Correctly recognizing the entities in an image does not guarantee correct prediction of their interactions. The focus on how visual evidence evolves during generation is a useful complement to methods that primarily increase visual attention strength.

  2. The method is clear and relatively lightweight. IVE combines a token-specific recent reference with attenuation based on prior selection frequency. Reconstructing values from fixed originals avoids cumulative scaling, and tokens can recover their original contribution or become eligible for reinforcement as their attention changes. The reported runtimes correspond to approximately 6-12.4% overhead, with less than 2% additional GPU memory.

  3. The empirical gains are consistent across different backbones. IVE improves Reefknot Y/N and MCQ results on LLaVA-1.5, InstructBLIP, and Qwen2.5-VL, and improves Hal-Eval Relation accuracy by about five percentage points on each backbone. The MMRA result and the shared-configuration evaluation provide additional evidence of transfer beyond one model or setting.

  4. The appendix contains useful controls. In addition to the entropy-matched carryover analysis, the paper includes component ablations, comparisons of selection signals under matched token budgets and scaling factors, random perturbation controls, and comparisons with temporal-attention methods. These controls make the empirical argument more convincing than attention visualizations alone.

Critical weaknesses
  1. The source of the history-based gains is unclear. IVE already uses input-text history for the first prediction, which can decide Y/N or MCQ answers before any generated-token history exists. Could the authors ablate input history, generated history, and their combination to isolate the contribution of decoding history?

  2. The advantage over the closest baselines needs stronger support. In Table 12, IVE leads the strongest temporal baseline by only 0.64-1.44 Rscore points, with validation-tuned IVE configurations compared against baseline defaults. Could the authors use the same validation tuning budget and report paired confidence intervals for these margins?

  3. The carryover experiment does not isolate naturally occurring inertia. Mixing previous attention can also overwrite evidence needed by the current query, which entropy matching does not control. Please distinguish the demonstrated effect of induced carryover from the proposed explanation of naturally occurring relation hallucinations.

  4. Accuracy and F1 may conceal an error trade-off. If POPE uses a balanced split and both metrics come from the same prediction pool, Table 19's LLaVA-1.5 Random results imply a precision drop from about 92.1% to 87.2% despite higher F1. Could the authors report class counts, precision, recall, and the Yes ratio from the existing predictions?

  5. Evidence for open-ended relation correction is limited. The main results concern relation discrimination, while the IVE response in Figure 14 still describes a player swinging a bat without clear support from the image. Could the authors clarify which claims this example corrects and qualify the conclusions about open-ended relational faithfulness?

Questions for authors

Please refer to weaknesses above.

7 comments
AuthorsOpenReview ↗

Response (Part I)

We thank the reviewer for the careful assessment and for recognizing IVE's usefulness. Below, we address each concern with clarifications and additional evidence.

1. Input History Retains Most Short-Answer Gains, while Generation History Improves Descriptions{\color{#ff7f00}\text{1. Input History Retains Most Short-Answer Gains, while Generation History Improves Descriptions}}

We thank the reviewer for suggesting this ablation. IVE already uses input-text attention history for its first output prediction. Input history alone retains most of the short-answer gains, while generation history further improves relation descriptions.

Input-only freezes prompt history statistics while selecting and modulating tokens using current attention. Generation-only accumulates generated-token query history and starts modulation at the third output prediction. All retain full text context and original backbone-specific settings without retuning.

(a) Input history retains most short-answer gains.

We use the complete Reefknot Y/N and MCQ evaluation sets.

Table 1. Short-answer performance (R_score, %). In both tables, Δ = full IVE − input history only (percentage points).

ConditionLLaVA-1.5 Y/NLLaVA-1.5 MCQInstructBLIP Y/NInstructBLIP MCQQwen2.5-VL Y/NQwen2.5-VL MCQ
Vanilla64.3943.7056.2839.9472.0744.91
Input history only66.6845.4258.7442.1374.3746.89
Generation history only65.0744.6357.1941.1172.6445.93
Full IVE67.2145.9959.0942.9474.7647.50
Δ [95% simultaneous CI]+0.52 [+0.07, +0.98]+0.56 [−0.20, +1.33]+0.35 [−0.07, +0.78]+0.81 [+0.19, +1.42]+0.39 [−0.12, +0.90]+0.60 [+0.07, +1.15]

Generation history adds 0.35–0.81 percentage points across the six settings, with statistical support in three.

(b) Generation history further improves relation descriptions.

We reuse the original 2,000 LLaVA-1.5 relation-description inputs and GPT-4o scoring protocol.

Table 2. Relation-description outcomes. Rates are percentages, with response counts in parentheses.

ConditionCorrect target relation ↑Unsupported target relation ↓Unanswered / ambiguous
Vanilla61.80 (1,236)27.40 (548)10.80 (216)
Input history only64.15 (1,283)25.40 (508)10.45 (209)
Generation history only63.95 (1,279)25.30 (506)10.75 (215)
Full IVE65.90 (1,318)23.85 (477)10.25 (205)
Δ [95% simultaneous CI]+1.75 [+0.45, +3.08]−1.55 [−2.84, −0.30]−0.20 [−1.01, +0.60]

Compared with input history alone, full IVE improves target-relation accuracy and reduces the unsupported-target rate, with statistical support for both. The revised manuscript summarizes this ablation in Section 4.3 and provides both result tables in Appendix C.6.

2. IVE Outperforms Both Temporal Baselines under Equal Tuning Budgets{\color{#ff7f00}\text{2. IVE Outperforms Both Temporal Baselines under Equal Tuning Budgets}}

We thank the reviewer for highlighting tuning fairness and uncertainty. With equal candidate budgets, IVE retains statistically supported gains over ADAPT-TF and ANL in all 12 comparisons.

Each method receives 12 candidates per backbone, including IVE's coefficient/layer choices and default/inherited settings. Before testing, each selects and freezes one configuration for both formats by mean Y/N and MCQ R_score on shared validation data without test image/scene overlap. ANL's detector remains fixed.

Intervals use paired bootstrap clustered by source image/scene, jointly corrected across this table and all applicable original baseline comparisons on Reefknot, Hal-Eval Relation, and MMRA.

Table 3. Equal-budget comparison on Reefknot (R_score, %). Differences are in percentage points.

MethodLLaVA-1.5 Y/NLLaVA-1.5 MCQInstructBLIP Y/NInstructBLIP MCQQwen2.5-VL Y/NQwen2.5-VL MCQ
ADAPT-TF66.2145.2857.9642.3174.0647.06
ANL66.4845.6658.0742.4974.1946.88
IVE67.3446.1759.0343.2674.9847.63
IVE − ADAPT-TF [95% simultaneous CI]+1.13 [0.74, 1.52]+0.89 [0.45, 1.33]+1.07 [0.70, 1.44]+0.95 [0.53, 1.37]+0.92 [0.56, 1.28]+0.57 [0.18, 0.96]
IVE − ANL [95% simultaneous CI]+0.86 [0.51, 1.21]+0.51 [0.11, 0.91]+0.96 [0.61, 1.31]+0.77 [0.37, 1.17]+0.79 [0.45, 1.13]+0.75 [0.35, 1.15]

Tuning improves both baselines, while IVE retains its advantage. The revised manuscript summarizes this comparison in Sections 4.1–4.2, with the protocol in Appendix B and results and uncertainty estimates in Appendices C.1–C.2.

AuthorsOpenReview ↗

Response (Part II)

3. Forced Attention Persistence Impairs Relation Prediction More than Matched Controls{\color{#ff7f00}\text{3. Forced Attention Persistence Impairs Relation Prediction More than Matched Controls}}

We thank the reviewer for highlighting this distinction. Forcing earlier attention to persist incurs additional relation-prediction loss relative to the tested controls. In natural responses (Figure 3), unsupported relation descriptions tend to retain earlier visual focus.

(a) Matched controls preserve higher relation-prediction scores.

Shuffled-history and current-derived controls approximately match attention concentration and change using per-head entropy and Jensen–Shannon (JS) divergence from their current attention. Across all inputs, including those with unmatched positions, they exceed Carryover by 2.67 and 3.39 percentage points (Table 4).

Table 4. Jointly matched controls (LLaVA-1.5/Reefknot Y/N, λ = 0.5). Coverage is over valid head/query positions. ± denotes between-run SD.

ConditionEntropy error (×10⁻³)JS error (×10⁻³)Joint coverageR_score ↑ΔR vs. Carryover [95% CI]
Vanilla———64.39+9.89 [8.97, 10.80]
Carryover0 (reference)0 (reference)—54.50Reference
Original entropy-matched control0.04712.46721.58%59.74+5.24 [4.31, 6.20]
Shuffled history + joint matching3.3871.68794.44%57.17 ± 0.35+2.67 [2.39, 2.95]
Current-derived control + joint matching2.7812.09794.09%57.89 ± 0.40+3.39 [3.09, 3.72]

(b) Matched controls better preserve correct-answer preference.

To compare downstream effects, we measure attention-output and hidden-state changes alongside correct-answer preference. All three interventions begin at the same layer during the first answer prediction, from the same unmodified prefix, cache, and attention history.

Table 5. Prediction and representation changes from the same initial state. Δm is the change in the correct-vs-incorrect log-probability margin. Δm and both representation metrics are relative to Vanilla.

ConditionAnswer-margin change ΔmΔm difference vs. Carryover [95% CI]Attention-output change (relative L2)Hidden-state cosine similarity
Carryover−0.484Reference8.52%0.99505
Shuffled history + joint matching−0.319+0.165 [0.151, 0.180]8.63%0.99511
Current-derived control + joint matching−0.253+0.231 [0.215, 0.248]8.42%0.99494

In the sample means, shuffled history produces a larger attention-output change than Carryover, while the current-derived control produces a larger change in hidden-state direction. Both controls nevertheless preserve a stronger preference for the correct answer. The revised manuscript summarizes the matched-control findings in Section 2.3 and reports both comparisons in Appendix A.5.

AuthorsOpenReview ↗

Response (Part III)

4. IVE Improves POPE Accuracy, F1, and Recall, with Lower Precision{\color{#ff7f00}\text{4. IVE Improves POPE Accuracy, F1, and Recall, with Lower Precision}}

We thank the reviewer for highlighting the precision–recall trade-off. Compared with Vanilla, IVE improves accuracy and F1 and misses fewer objects that are actually present, while precision decreases.

In this supplementary evaluation, all five methods use LLaVA-1.5 on the same 3,000 COCO POPE Random questions (500 images, 1,500 Yes and 1,500 No labels). Each method's metrics are computed from the same set of predictions.

Table 6. Complete metrics on COCO POPE Random (%).

MethodAccuracy ↑F1 ↑Precision ↑Recall ↑Yes ratio
Vanilla83.2381.2991.9372.8739.63
VCD86.2786.2986.1286.4750.20
PAI88.3088.8784.7193.4755.17
MemVR89.0789.1288.7189.5350.47
IVE90.5390.9587.1295.1354.60

5. IVE Reduces Relation Hallucinations, and the Incorrect Figure 14 Example Is Removed{\color{#ff7f00}\text{5. IVE Reduces Relation Hallucinations, and the Incorrect Figure 14 Example Is Removed}}

We thank the reviewer for pointing out the error in the original Figure 14. IVE's description of swinging is unsupported by the image, so we have removed this example from the revised manuscript.

IVE improves relation descriptions across the full VQA split.

Using LLaVA-1.5, we evaluated all five methods on the full 4,870-input Reefknot VQA split, replacing relation-completion prompts with requests to freely describe relations between specified entities. GPT-4o evaluated responses with method identities hidden.

Table 7. Open-ended relation descriptions of specified entity pairs. Rates are percentages. Δ = IVE − Vanilla (percentage points).

MethodCorrect target relation ↑Unsupported target relation ↓Unanswered / ambiguousAdditional relation hallucination ↓
Vanilla61.8727.5610.5715.79
VCD63.4726.3210.2114.15
PAI65.1724.859.9812.83
MemVR63.7825.2810.9412.26
IVE66.1023.5310.3711.11
Δ: IVE − Vanilla+4.23−4.02−0.21−4.68

Compared with Vanilla, IVE improves target-relation accuracy and reduces both the unsupported-target rate and the proportion of responses with unsupported relations beyond the target. This evaluation is now reported in Section 4.2, with the scoring protocol in Appendix B of the revised manuscript.

Reviewer_y1oSOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies relation hallucinations in multimodal large language models, where models correctly recognize entities but incorrectly describe their relationships. The authors observe that incorrect relation descriptions often exhibit more persistent visual attention across decoding steps. Through previous-step attention carryover and entropy-matched controls, they further show that forcing previous attention patterns to persist can impair relation prediction.

The authors propose Inertia-aware Visual Excitation (IVE), a decoding method that requires no training. IVE compares each visual token’s current attention with its recent history, reinforcing emergent tokens with rising attention while mildly attenuating persistently dominant inertia tokens with limited relative growth.

Across three multimodal models, IVE improves relation prediction on Reefknot and Hal-Eval. The results of the main experiment and various other experiments demonstrate IVE’s ability to reduce relation hallucinations at acceptable computational cost while preserving performance on the other evaluated tasks.

Critical strengths
  1. An insightful empirical observation motivates a simple and effective method. The authors observe that incorrect relation descriptions often retain attention on previously dominant visual regions. Attention carryover experiments and entropy-matched controls provide further evidence for the relevance of this pattern. Based on these findings, IVE uses attention history to reinforce emergent visual tokens and mildly attenuate persistently dominant ones. The clear connection between the analysis and method design provides both research insight and practical value.

  2. Consistent improvements across relation evaluation settings with low inference overhead. IVE outperforms Vanilla and the compared methods on Reefknot Y/N, Reefknot MCQ, and Hal-Eval Relation across all three models. Results on MMRA also show improvements in a multi-image setting. The method requires no training and introduces only small increases in runtime and GPU memory usage.

Critical weaknesses
  1. Evaluation on newer models is missing. The experiments use LLaVA-1.5, InstructBLIP, and Qwen2.5-VL, but do not include newer models , such as Qwen3-VL-8B-Instruct and LLaVA-OneVision-2-8B-Instruct. Although the results are consistent across the three evaluated models, it remains unclear whether the observed visual inertia and the benefits of IVE also hold for these newer multimodal models.

  2. Quantitative evidence for reducing relation hallucinations in open-ended generation remains limited. The paper demonstrates IVE’s effects on open-ended descriptions through attention analysis on COCO captions, CHAIR evaluation, and qualitative examples. However, it does not directly quantify differences between Vanilla and IVE in relation correctness in these descriptions. An evaluation using the existing annotation rules would more directly connect the experimental evidence to the paper’s focus on relation hallucinations.

Questions for authors
  1. How much do input history and generation history each contribute to the gains? For short-answer tasks such as Y/N and MCQ, do the improvements mainly come from the history accumulated over input-text queries or from history updates during generation? I suggest adding an ablation study to separate these contributions and clarify the main source of improvement.
  2. Can IVE’s reduction of relation hallucinations in open-ended descriptions be quantified directly? As discussed in Weakness 2.
  3. Do visual inertia and the benefits of IVE extend to newer models? As discussed in Weakness 1, attention analysis and relation evaluation on at least one newer model would help establish the scope of the findings and the method.
5 comments
AuthorsOpenReview ↗

Response (Part I)

We thank the reviewer for the careful assessment and for recognizing the connection between our empirical observations and IVE's design. Below, we address each concern with clarifications and additional evidence.

1. IVE Improves Relation Prediction on Three Recent Models{\color{#ff7f00}\text{1. IVE Improves Relation Prediction on Three Recent Models}}

We thank the reviewer for suggesting newer models. IVE improves relation prediction on all three. In natural Vanilla responses, visual focus also persists more strongly in unsupported than correct descriptions.

(a) A unified configuration improves relation prediction. We use the same IVE configuration for Qwen3-VL-8B-Instruct, LLaVA-OneVision-2-8B-Instruct, and InternVL3.5-8B. IVE scores highest on all 12 metrics among the compared methods (Table 1).

Table 1. Relation prediction (%). Y/N and MCQ report Reefknot R_score, and Hal-Eval reports Relation Accuracy / F1.

MethodQwen3-VL Y/NQwen3-VL MCQQwen3-VL Hal-EvalOneVision-2 Y/NOneVision-2 MCQOneVision-2 Hal-EvalInternVL3.5 Y/NInternVL3.5 MCQInternVL3.5 Hal-Eval
Vanilla78.6352.7372.38 / 69.9176.1454.2269.53 / 68.1777.5251.8673.21 / 71.44
VCD78.9152.9673.09 / 70.8676.3854.4070.37 / 69.1277.6952.0973.94 / 72.16
PAI79.0352.8573.83 / 71.9276.8354.1470.94 / 69.7678.2652.1974.51 / 73.09
MemVR79.1152.5874.12 / 71.7677.0354.8371.46 / 70.0278.1452.4074.79 / 73.38
IVE80.1154.1974.52 / 72.2878.0156.0372.11 / 70.2678.7853.5375.52 / 74.14

(b) Unsupported descriptions retain earlier visual focus. We analyze natural Vanilla responses to 300 shared inputs, randomly selected in advance from the original 2,000 relation-description inputs. Length adjustment uses the same responses as the unadjusted comparison.

Table 2. Mean consecutive JS distance of visual attention. Δ = correct − unsupported, with 95% simultaneous CIs.

ModelValid responses (correct / unsupported)CorrectUnsupportedUnadjusted ΔLength-adjusted Δ
Qwen3-VL195 / 770.17820.1446+0.0336 [+0.0133, +0.0536]+0.0302 [+0.0095, +0.0508]
OneVision-2206 / 660.19600.1574+0.0386 [+0.0168, +0.0608]+0.0304 [+0.0092, +0.0511]
InternVL3.5215 / 610.18350.1457+0.0378 [+0.0188, +0.0568]+0.0293 [+0.0102, +0.0481]

Across all three models, unsupported responses have smaller consecutive JS distances, indicating stronger retention of the previous step's visual focus. This difference persists after length adjustment. We report the new-model results in Section 4.2 and the attention analysis in Appendix A.4 of the revised manuscript.

2. IVE Reduces Relation Hallucinations in Open-Ended Descriptions{\color{#ff7f00}\text{2. IVE Reduces Relation Hallucinations in Open-Ended Descriptions}}

We thank the reviewer for suggesting a direct evaluation of generated relations. In open-ended descriptions of specified entity pairs, IVE reduces unsupported target relations and improves target-relation accuracy over Vanilla.

Using LLaVA-1.5, all five methods share the full set of 4,870 Reefknot VQA inputs. We replace the fill-in prompts with requests to freely describe the specified entity pairs' relations. GPT-4o evaluates responses with method identities hidden.

Table 3. Generated relation descriptions. Rates are percentages over all inputs. Δ = IVE − Vanilla, in percentage points.

MethodCorrect target relation ↑Unsupported target relation ↓Unanswered / ambiguousAdditional relation hallucination ↓
Vanilla61.8727.5610.5715.79
VCD63.4726.3210.2114.15
PAI65.1724.859.9812.83
MemVR63.7825.2810.9412.26
IVE66.1023.5310.3711.11
Δ: IVE − Vanilla+4.23−4.02−0.21−4.68

IVE also reduces the proportion of responses containing unsupported relations beyond the requested target. This evaluation is now reported in Section 4.2, with the scoring protocol in Appendix B of the revised manuscript.

AuthorsOpenReview ↗

Response (Part II)

3. Input History Retains Most Short-Answer Gains, while Generation History Improves Descriptions{\color{#ff7f00}\text{3. Input History Retains Most Short-Answer Gains, while Generation History Improves Descriptions}}

We thank the reviewer for suggesting this ablation. Input history alone retains most of IVE's short-answer gains, while generation history further improves relation descriptions.

Input-only freezes prompt history statistics while selecting and modulating tokens using current attention. Generation-only accumulates generated-token query history and starts modulation at the third output prediction. All retain full text context and original backbone-specific settings without retuning.

(a) Input history retains most short-answer gains. On the complete Reefknot Y/N and MCQ evaluation sets, adding generation history yields a further 0.35–0.81 percentage points, with statistical support in three of six settings.

Table 4. Short-answer performance (R_score, %). In both tables, Δ = full IVE − input history only (percentage points).

ConditionLLaVA-1.5 Y/NLLaVA-1.5 MCQInstructBLIP Y/NInstructBLIP MCQQwen2.5-VL Y/NQwen2.5-VL MCQ
Vanilla64.3943.7056.2839.9472.0744.91
Input history only66.6845.4258.7442.1374.3746.89
Generation history only65.0744.6357.1941.1172.6445.93
Full IVE67.2145.9959.0942.9474.7647.50
Δ [95% simultaneous CI]+0.52 [+0.07, +0.98]+0.56 [−0.20, +1.33]+0.35 [−0.07, +0.78]+0.81 [+0.19, +1.42]+0.39 [−0.12, +0.90]+0.60 [+0.07, +1.15]

(b) Generation history further improves relation descriptions. We reuse the original 2,000 LLaVA-1.5 relation-description inputs and GPT-4o scoring protocol. Compared with input history alone, full IVE improves target-relation accuracy and reduces the unsupported-target rate, with statistical support for both.

Table 5. Relation-description outcomes. Rates are percentages, with response counts in parentheses.

ConditionCorrect target relation ↑Unsupported target relation ↓Unanswered / ambiguous
Vanilla61.80 (1,236)27.40 (548)10.80 (216)
Input history only64.15 (1,283)25.40 (508)10.45 (209)
Generation history only63.95 (1,279)25.30 (506)10.75 (215)
Full IVE65.90 (1,318)23.85 (477)10.25 (205)
Δ [95% simultaneous CI]+1.75 [+0.45, +3.08]−1.55 [−2.84, −0.30]−0.20 [−1.01, +0.60]

The revised manuscript summarizes this ablation in Section 4.3 and provides both result tables in Appendix C.6.

AuthorsOpenReview ↗

We Would Appreciate Your Feedback on Our Responses

Dear Reviewer y1oS,

Thank you again for your time and thoughtful feedback. We have provided additional experiments and clarifications in our responses and revised manuscript.

When you have a chance, we would appreciate your feedback on our latest responses, particularly whether any concerns remain. We would be happy to clarify any unresolved points.

Best regards,
The authors

Reviewer_iAG2OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper attributes a subset of relation hallucinations in MLLMs to “visual inertia,” namely persistent reuse of previously dominant visual evidence across autoregressive decoding. It proposes Inertia-aware Visual Excitation (IVE), a training-free decoding intervention that detects visual tokens whose attention rises relative to a moving historical reference and reinforces them, while attenuating frequently dominant tokens. The paper evaluates IVE across three MLLMs on relation-hallucination benchmarks, object-hallucination benchmarks, and selected broader multimodal tasks.

Critical strengths

The paper targets an important distinction that is often blurred in hallucination work: correct recognition of the entities does not guarantee correct grounding of their relation. The framing in Figure 1 is effective in separating object hallucination from relation hallucination, and in conveying why simply “attending to the image more” may be inadequate for relational reasoning.

The empirical motivation is reasonably well developed. Figure 2 demonstrates that increasing visual-attention amplification produces a monotonic improvement on POPE but a non-monotonic pattern on Reefknot. This is a useful observation because it motivates studying the temporal allocation of visual evidence rather than only its aggregate strength.

The proposed mechanism is lightweight and technically concrete. Equations (4) to (10) specify how IVE builds an EMA reference, distinguishes emergent and inertia tokens, and applies modulation directly to cached visual values. The use of fixed original values in Equation (10), rather than repeatedly multiplying already modified values, is a sensible design choice that avoids uncontrolled accumulation over decoding steps.

Critical weaknesses
  1. The evidence supports an association between attention persistence and error, but does not fully establish "visual inertia" as the causal mechanism of relation hallucination. The central motivation in Section 2 relies on contrasting correct and hallucinated generations in Figure 3. However, these groups differ not only in correctness but potentially in output length, linguistic content, confidence, prompt-response interaction, and semantic stage of decoding. The analysis further retains only responses with at least 50 valid output tokens and excludes ambiguous answers and correct answers containing additional hallucinations. This produces a cleaner visualization, but may introduce selection effects that are correlated with both output style and attention trajectory. The result is informative but should be framed as a correlational behavioral signature rather than direct evidence that persistence causes hallucination. The intervention in Equation (3) moves closer to causal evidence, but it still changes an internal attention distribution in a potentially off-manifold manner. Matching entropy controls one property of the distribution, but does not establish that carryover and the reference condition are equivalent with respect to head-wise allocation, visual-to-text attention mass, layer-specific effects, or downstream representation geometry. The substantial degradation in Table 1 is compelling evidence that forcibly propagating previous attention is harmful, but it does not by itself prove that naturally occurring persistence is the principal source of relation hallucinations.

  2. The attention-displacement metric in Equations (1) and (2) is not fully justified as a spatial metric. Equation (2) uses a one-dimensional Wasserstein distance over flattened row-major patch indices, with ground cost |a-b|. This cost treats patches at the end and beginning of adjacent rows as far apart, despite being spatial neighbors in the image, while some patches separated along the flattened order may not correspond to meaningful two-dimensional distances. Consequently, the absolute magnitude and even some relative changes in W_idx are partly artifacts of the arbitrary flattening. The paper should report the primary analyses using a two-dimensional patch-coordinate transport cost, such as Euclidean or Manhattan distance over patch coordinates, and demonstrate that the correct-versus-hallucinated separation in Figure 3 and the carryover trends in Table 1 survive that choice. The supplementary Jensen-Shannon comparison is helpful because it removes dependence on patch ordering, but it measures distributional change rather than spatial displacement. The two measurements answer different questions and should be separated more carefully in the main argument.

  3. The main empirical results lack uncertainty estimates and statistical testing for the deterministic benchmark comparisons. Tables 2, 3, 4, and 5 report single values, despite many differences among baselines being relatively small. The sampling results in Table 6 include standard deviations across five seeds, which is good, but analogous uncertainty is absent for the main greedy-decoding results. Since benchmark samples are finite and relation-hallucination metrics are aggregate rates, the paper should provide confidence intervals or paired bootstrap tests for IVE versus the strongest baseline on each backbone and benchmark. This is particularly important for claims of consistent superiority across a large number of comparisons.

In addition, the paper states that IVE configurations are selected using held-out Reefknot validation data, but it is not equally clear whether competing methods are evaluated with their published defaults, independently tuned validation configurations, or an equivalent search budget. This is especially relevant for baselines whose behavior can be sensitive to decoding parameters or intervention strength. The reproduced RVE baseline deserves particularly detailed implementation and configuration information.

Questions for authors
  1. Can the authors provide a causal analysis at matched generated prefixes? For example, teacher-force identical output prefixes under Vanilla and IVE, then compare visual attention, selected token sets, and relation-token logits. This would separate the direct effect of IVE from downstream differences caused by decoding different earlier tokens.
  2. Does the main conclusion in Figure 3 remain under a two-dimensional spatial ground metric over patch coordinates rather than the flattened-index Wasserstein cost in Equation (2)? Please report this for the correct-versus-hallucinated comparison and the carryover experiment.
  3. What is the empirical distribution of St,jS_{t,j}, ∣Et∣|\mathcal E_t|, and ∣It∣|\mathcal I_t| across backbones and tasks? How often does an inertia token become emergent later? Since emergent multipliers are always capped at 1.101.10 at the chosen threshold, would a simpler binary selector based on EMA-relative increase perform similarly?
  4. Please clarify exactly how value recalibration is implemented for each architecture. Which KV tensors are changed in LLaVA-1.5, InstructBLIP, and Qwen2.5-VL? Are values modified at cross-attention layers, self-attention layers, or both? How are visual token identities aligned across layers?
  5. Can the authors report paired confidence intervals or bootstrap significance tests for IVE versus the strongest baseline in Tables 2 to 4? Also, were all baselines given comparable validation-time tuning budgets, especially methods with intervention-strength hyperparameters?
  6. How does IVE behave on tasks where sustained visual focus is desirable, such as OCR, counting, fine-grained attributes, or referring expression tasks? Evidence here would clarify whether IVE genuinely preserves general capabilities rather than only improving the selected benchmarks.
  7. For the manually annotated motivation analysis underlying Figure 3, who performed annotation, what quality-control or adjudication procedure was used, and what was the inter-annotator agreement? How sensitive is the result to including short, ambiguous, and target-correct-but-otherwise-hallucinated responses?
9 comments
AuthorsOpenReview ↗

Response (Part I)

We thank the reviewer for examining alternative explanations for our attention findings and the fairness of IVE’s evaluation. Below, we address these concerns with additional evidence and clarifications.

1. Unsupported Responses Show Stronger Attention Persistence after Adjustment and Broader Inclusion{\color{#ff7f00}\text{1. Unsupported Responses Show Stronger Attention Persistence after Adjustment and Broader Inclusion}}

We thank the reviewer for highlighting these potential confounds. After adjusting for measured response characteristics and relaxing selection rules, unsupported relation descriptions still retain earlier visual focus more strongly than correct descriptions.

(a) Attention persistence remains after adjustment. We adjust for response length, relation position/span, linguistic features, and predictive entropy. Each row uses the same responses before and after adjustment.

Table 1. Group differences before and after adjustment. Groups are fully correct/unsupported responses (row 1) and responses with supported/unsupported first explicit target-relation assertions (row 2). Δ = correct/supported − unsupported.

Mean displacementGroup sizesUnadjusted group meansUnadjusted Δ [95% CI]Adjusted Δ [95% CI]
All pairs, first 50 positions468 / 48251.16 / 41.769.40 [7.87, 10.96]6.56 [4.84, 8.27]
Consecutive pairs, target-relation span496 / 42544.39 / 37.416.98 [5.61, 8.41]4.38 [2.91, 5.84]

During relation expression, the pattern fits our hypothesis that persistent focus may hinder attention to relevant evidence. Separately, sampled continuations sharing the image, entity pair, and full prompt yield mean paired Δ = 5.12 (95% CI [2.10, 8.08]) across 67/200 prompts.

(b) The association remains under broader inclusion. Table 2 reintroduces short responses and target-correct responses with additional hallucinations from the original 2,800-response pool. We measure mean consecutive displacement over up to the first 50 valid positions, requiring at least two.

Table 2. Group differences under broader inclusion rules.

Inclusion ruleGroup sizes: correct / unsupportedCorrect meanUnsupported meanΔ [95% CI]
Original criteria949 / 56245.7036.858.85 [7.71, 9.97]
Relax length restriction only1,342 / 79044.7235.918.81 [7.84, 9.76]
Relax additional-hallucination restriction only1,227 / 56245.1036.858.25 [7.16, 9.33]
Relax both restrictions1,726 / 79044.1835.918.27 [7.36, 9.16]
First 20 positions after relaxing both restrictions1,548 / 71642.1835.127.06 [6.09, 8.05]

The final row changes both window and sample, using the first 20 positions of responses with at least 20 valid tokens.

We report unanswered/ambiguous responses separately because their target-relation support is indeterminate: 261/274 (95.26%) have measurable displacement (median 36.29, IQR [29.32, 42.94]). We have added the response-adjustment and broader-inclusion analyses to Appendix A.2 of the revised manuscript.

AuthorsOpenReview ↗

Response (Part II)

2. Forced Attention Persistence Impairs Relation Prediction More than Matched Controls{\color{#ff7f00}\text{2. Forced Attention Persistence Impairs Relation Prediction More than Matched Controls}}

We thank the reviewer for highlighting these intervention effects. Forcing earlier attention to persist incurs additional relation-prediction loss relative to the tested controls.

(a) Matched controls preserve higher relation-prediction scores.

Shuffled-history and current-derived controls approximately match attention concentration and change using per-head entropy and Jensen–Shannon (JS) divergence from their current attention. Across all inputs, including those with unmatched positions, they exceed Carryover by 2.67 and 3.39 percentage points (Table 3).

Table 3. Jointly matched controls (LLaVA-1.5/Reefknot Y/N, λ = 0.5). Coverage is over valid head/query positions. ± denotes between-run SD.

ConditionEntropy error (×10⁻³)JS error (×10⁻³)Joint coverageR_score ↑ΔR vs. Carryover [95% CI]
Vanilla———64.39+9.89 [8.97, 10.80]
Carryover0 (reference)0 (reference)—54.50Reference
Original entropy-matched control0.04712.46721.58%59.74+5.24 [4.31, 6.20]
Shuffled history + joint matching3.3871.68794.44%57.17 ± 0.35+2.67 [2.39, 2.95]
Current-derived control + joint matching2.7812.09794.09%57.89 ± 0.40+3.39 [3.09, 3.72]

(b) Matched controls better preserve correct-answer preference.

Similar attention changes can have different downstream effects. We compare changes in attention outputs and hidden states with changes in correct-answer preference. All three interventions begin from the same unmodified prefix and internal state, at the same layer during the first answer prediction.

Table 4. Prediction and representation changes from the same initial state. Δm is the change in the correct-vs-incorrect log-probability margin. Δm and both representation metrics are relative to Vanilla.

ConditionAnswer-margin change ΔmΔm difference vs. Carryover [95% CI]Attention-output change (relative L2)Hidden-state cosine similarity
Carryover−0.484Reference8.52%0.99505
Shuffled history + joint matching−0.319+0.165 [0.151, 0.180]8.63%0.99511
Current-derived control + joint matching−0.253+0.231 [0.215, 0.248]8.42%0.99494

In the sample means, shuffled history produces a larger attention-output change than Carryover, while the current-derived control produces a larger change in hidden-state direction. Both controls nevertheless preserve a stronger preference for the correct answer. The revised manuscript summarizes the matched-control findings in Section 2.3 and reports both comparisons in Appendix A.5.

AuthorsOpenReview ↗

Response (Part III)

3. 2D Spatial Distances Also Show Stronger Attention Persistence in Unsupported Responses{\color{#ff7f00}\text{3. 2D Spatial Distances Also Show Stronger Attention Persistence in Unsupported Responses}}

We thank the reviewer for suggesting a 2D spatial cost. With distances based on actual patch coordinates, unsupported responses still retain earlier visual focus more strongly than correct responses, and stronger Carryover further reinforces attention persistence.

We recompute distances on the same samples and attention trajectories using full 2D optimal transport with normalized Manhattan costs between patches. This measures spatial displacement, whereas JS distance measures changes in attention allocation without using patch geometry.

(a) Unsupported responses retain earlier visual focus.

We reuse Figure 3's 500 correct and 500 target-unsupported responses, comparing all distinct position pairs and consecutive pairs within the first 50 valid output positions.

Table 5. Group differences across metrics. Δ = correct − unsupported.

Position pairsMetricCorrect meanUnsupported meanΔ [95% CI]
All pairs1D index distance51.2441.83+9.40 [7.89, 10.90]
All pairs2D spatial distance0.10240.0854+0.0169 [0.0146, 0.0194]
All pairsJS distance0.23090.2042+0.0267 [0.0236, 0.0299]
Consecutive pairs1D index distance44.6335.88+8.74 [7.32, 10.16]
Consecutive pairs2D spatial distance0.08230.0680+0.0143 [0.0122, 0.0165]
Consecutive pairsJS distance0.19560.1745+0.0212 [0.0185, 0.0238]

The smaller mean consecutive-step 2D displacement in unsupported responses shows that their visual attention shifts less across the image as successive tokens are generated.

(b) Stronger Carryover reinforces attention persistence.

We reuse the Carryover and entropy-matched reference trajectories for a fixed diagnostic subset of 1,000 inputs from 500 images, selected independently of correctness or displacement. All three attention metrics use this subset, while R_score retains the original full-evaluation-set results.

Table 6. Consecutive attention changes across Carryover strengths. C = Carryover, Ref = original entropy-matched reference. λ = 0 is the shared Vanilla baseline.

λ1D: C / Ref2D: C / Ref2D Δ: Ref − C [95% CI]JS: C / RefR_score: C / Ref
041.52 / 41.520.0919 / 0.09190 (shared baseline)0.1930 / 0.193064.39 / 64.39
0.2535.41 / 38.130.0784 / 0.0843+0.0059 [0.0047, 0.0070]0.1697 / 0.182659.61 / 62.27
0.5028.73 / 34.620.0649 / 0.0760+0.0111 [0.0097, 0.0124]0.1425 / 0.164554.50 / 59.74
0.7523.64 / 31.170.0565 / 0.0679+0.0115 [0.0101, 0.0127]0.1285 / 0.152850.71 / 57.55
1.0019.24 / 28.840.0482 / 0.0649+0.0166 [0.0151, 0.0182]0.1046 / 0.139947.68 / 55.86

As Carryover strength increases, mean consecutive-step 2D displacement decreases from the Vanilla baseline and remains below the entropy-matched reference at every nonzero strength. JS distance preserves both the group ordering in Table 5 and the strength trend in Table 6. The revised manuscript uses 2D distances in Sections 2.2–2.3 and reports the metric comparisons in Appendices A.2 and A.5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the AC and all three reviewers for their careful evaluation and constructive feedback. During the rebuttal, we provided additional experiments and analyses, summarized in 23 distinct result tables, and incorporated the key findings and clarifications into the revised manuscript. In the discussion, Reviewer y1oS stated that the rebuttal had addressed their main concerns, raised their rating from 3 to 4, and recommended acceptance.

Below we summarize the discussion, including positive follow-up feedback from two reviewers:

  • y1oS confirmed that the rebuttal had “addressed my main concerns” and recommended acceptance. Their assessment recognized both the contribution and its experimental support: “the idea is insightful and the contribution is meaningful, with sufficient experimental support.”
  • VfK5 stated that “The history ablation and equal-budget comparisons largely address my concerns.” We also responded to the subsequent causal-interpretation question by explicitly distinguishing observation from intervention and adding a natural-state attenuation experiment. We regret that we have not yet had the opportunity for further discussion with the reviewer following this response.
  • We regret that iAG2 has not yet provided follow-up feedback. We have provided detailed responses to all of their concerns, supported by additional experiments, analyses, and corresponding manuscript revisions.

Core contribution. Object recognition alone does not ensure correct relation descriptions. We identify an association between unsupported descriptions and persistent visual focus, and test forced persistence through controlled interventions. These findings motivate IVE, a training-free method that uses attention history to reinforce newly relevant visual contributions and conditionally attenuate persistently dominant ones. Experiments show that its benefits extend from relation prediction to freely generated descriptions of specified entity pairs and transfer to recent backbones.

The table below summarizes how the additional evidence addresses the main concerns and strengthens support for IVE’s use of attention history to reduce relation hallucinations.

Key concernReviewersOur response
Relation hallucinations in open-ended generationy1oS, VfK5IVE reduces hallucinations while improving correctness in freely generated descriptions of specified entity pairs. Evaluation on the full Reefknot VQA set extends its benefits beyond constrained answer formats.
Generalization to recent modelsy1oSIVE outperforms the compared methods on the tested recent backbones using a shared configuration. The motivating attention pattern also remains observable, supporting the applicability of both the method and its empirical motivation.
Fair comparison and statistical reliabilityiAG2, VfK5IVE retains statistically supported gains over ADAPT-TF and ANL when all three methods receive equal tuning budgets, strengthening the evidence for its comparative effectiveness.
Contributions of input and generation historiesy1oS, VfK5Input history preserves most short-answer gains, while generation history further improves relation descriptions, supporting IVE’s use of both sources of attention history.
Reliability of the attention observationiAG2The association between persistent visual focus and unsupported descriptions holds across complementary checks of response characteristics, sample selection, and spatial measurement. Blinded reannotation further supports the reliability of the grouping.
Intervention evidence and causal interpretationiAG2, VfK5From identical natural states without imposed Carryover, history-guided attenuation reduces hallucinations more than matched attenuation of other tokens. This supports IVE’s use of history to select effective intervention targets.

We believe these additions substantively address the reviewers’ main concerns and strengthen the empirical support for IVE’s use of attention history to select attenuation targets and its effectiveness in reducing relation hallucinations. We would be grateful for the AC’s consideration of the expanded evidence and positive reviewer feedback, and sincerely thank the AC and reviewers for their time and constructive suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.