- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies relation hallucinations in multimodal large language models. The authors observe that hallucinated relation descriptions tend to retain similar visual attention distributions across decoding positions, whereas correct descriptions exhibit larger changes. They further show that increasing previous-step attention carryover degrades relation prediction more than an entropy-matched current-step control. Motivated by these observations, the paper proposes Inertia-aware Visual Excitation (IVE), which compares each visual token's current attention with its recent history, reinforces tokens with rising attention, and attenuates repeatedly dominant tokens. The intervention rescales visual values from fixed original values without parameter updates. Experiments across three 7B backbones show consistent improvements on relation benchmarks, with additional evaluations of general multimodal performance and inference cost. I find the method useful, although the connection between the proposed inertia mechanism and the main performance gains needs further clarification.
Critical strengths
-
The paper addresses a meaningful problem. Correctly recognizing the entities in an image does not guarantee correct prediction of their interactions. The focus on how visual evidence evolves during generation is a useful complement to methods that primarily increase visual attention strength.
-
The method is clear and relatively lightweight. IVE combines a token-specific recent reference with attenuation based on prior selection frequency. Reconstructing values from fixed originals avoids cumulative scaling, and tokens can recover their original contribution or become eligible for reinforcement as their attention changes. The reported runtimes correspond to approximately 6-12.4% overhead, with less than 2% additional GPU memory.
-
The empirical gains are consistent across different backbones. IVE improves Reefknot Y/N and MCQ results on LLaVA-1.5, InstructBLIP, and Qwen2.5-VL, and improves Hal-Eval Relation accuracy by about five percentage points on each backbone. The MMRA result and the shared-configuration evaluation provide additional evidence of transfer beyond one model or setting.
-
The appendix contains useful controls. In addition to the entropy-matched carryover analysis, the paper includes component ablations, comparisons of selection signals under matched token budgets and scaling factors, random perturbation controls, and comparisons with temporal-attention methods. These controls make the empirical argument more convincing than attention visualizations alone.
Critical weaknesses
-
The source of the history-based gains is unclear. IVE already uses input-text history for the first prediction, which can decide Y/N or MCQ answers before any generated-token history exists. Could the authors ablate input history, generated history, and their combination to isolate the contribution of decoding history?
-
The advantage over the closest baselines needs stronger support. In Table 12, IVE leads the strongest temporal baseline by only 0.64-1.44 Rscore points, with validation-tuned IVE configurations compared against baseline defaults. Could the authors use the same validation tuning budget and report paired confidence intervals for these margins?
-
The carryover experiment does not isolate naturally occurring inertia. Mixing previous attention can also overwrite evidence needed by the current query, which entropy matching does not control. Please distinguish the demonstrated effect of induced carryover from the proposed explanation of naturally occurring relation hallucinations.
-
Accuracy and F1 may conceal an error trade-off. If POPE uses a balanced split and both metrics come from the same prediction pool, Table 19's LLaVA-1.5 Random results imply a precision drop from about 92.1% to 87.2% despite higher F1. Could the authors report class counts, precision, recall, and the Yes ratio from the existing predictions?
-
Evidence for open-ended relation correction is limited. The main results concern relation discrimination, while the IVE response in Figure 14 still describes a player swinging a bat without clear support from the image. Could the authors clarify which claims this example corrects and qualify the conclusions about open-ended relational faithfulness?
Questions for authors
Please refer to weaknesses above.
7 comments
Response (Part II)
We thank the reviewer for highlighting this distinction. Forcing earlier attention to persist incurs additional relation-prediction loss relative to the tested controls. In natural responses (Figure 3), unsupported relation descriptions tend to retain earlier visual focus.
(a) Matched controls preserve higher relation-prediction scores.
Shuffled-history and current-derived controls approximately match attention concentration and change using per-head entropy and Jensen–Shannon (JS) divergence from their current attention. Across all inputs, including those with unmatched positions, they exceed Carryover by 2.67 and 3.39 percentage points (Table 4).
Table 4. Jointly matched controls (LLaVA-1.5/Reefknot Y/N, λ = 0.5). Coverage is over valid head/query positions. ± denotes between-run SD.
| Condition | Entropy error (×10⁻³) | JS error (×10⁻³) | Joint coverage | R_score ↑ | ΔR vs. Carryover [95% CI] |
|---|---|---|---|---|---|
| Vanilla | — | — | — | 64.39 | +9.89 [8.97, 10.80] |
| Carryover | 0 (reference) | 0 (reference) | — | 54.50 | Reference |
| Original entropy-matched control | 0.047 | 12.467 | 21.58% | 59.74 | +5.24 [4.31, 6.20] |
| Shuffled history + joint matching | 3.387 | 1.687 | 94.44% | 57.17 ± 0.35 | +2.67 [2.39, 2.95] |
| Current-derived control + joint matching | 2.781 | 2.097 | 94.09% | 57.89 ± 0.40 | +3.39 [3.09, 3.72] |
(b) Matched controls better preserve correct-answer preference.
To compare downstream effects, we measure attention-output and hidden-state changes alongside correct-answer preference. All three interventions begin at the same layer during the first answer prediction, from the same unmodified prefix, cache, and attention history.
Table 5. Prediction and representation changes from the same initial state. Δm is the change in the correct-vs-incorrect log-probability margin. Δm and both representation metrics are relative to Vanilla.
| Condition | Answer-margin change Δm | Δm difference vs. Carryover [95% CI] | Attention-output change (relative L2) | Hidden-state cosine similarity |
|---|---|---|---|---|
| Carryover | −0.484 | Reference | 8.52% | 0.99505 |
| Shuffled history + joint matching | −0.319 | +0.165 [0.151, 0.180] | 8.63% | 0.99511 |
| Current-derived control + joint matching | −0.253 | +0.231 [0.215, 0.248] | 8.42% | 0.99494 |
In the sample means, shuffled history produces a larger attention-output change than Carryover, while the current-derived control produces a larger change in hidden-state direction. Both controls nevertheless preserve a stronger preference for the correct answer. The revised manuscript summarizes the matched-control findings in Section 2.3 and reports both comparisons in Appendix A.5.
Response (Part III)
We thank the reviewer for highlighting the precision–recall trade-off. Compared with Vanilla, IVE improves accuracy and F1 and misses fewer objects that are actually present, while precision decreases.
In this supplementary evaluation, all five methods use LLaVA-1.5 on the same 3,000 COCO POPE Random questions (500 images, 1,500 Yes and 1,500 No labels). Each method's metrics are computed from the same set of predictions.
Table 6. Complete metrics on COCO POPE Random (%).
| Method | Accuracy ↑ | F1 ↑ | Precision ↑ | Recall ↑ | Yes ratio |
|---|---|---|---|---|---|
| Vanilla | 83.23 | 81.29 | 91.93 | 72.87 | 39.63 |
| VCD | 86.27 | 86.29 | 86.12 | 86.47 | 50.20 |
| PAI | 88.30 | 88.87 | 84.71 | 93.47 | 55.17 |
| MemVR | 89.07 | 89.12 | 88.71 | 89.53 | 50.47 |
| IVE | 90.53 | 90.95 | 87.12 | 95.13 | 54.60 |
We thank the reviewer for pointing out the error in the original Figure 14. IVE's description of swinging is unsupported by the image, so we have removed this example from the revised manuscript.
IVE improves relation descriptions across the full VQA split.
Using LLaVA-1.5, we evaluated all five methods on the full 4,870-input Reefknot VQA split, replacing relation-completion prompts with requests to freely describe relations between specified entities. GPT-4o evaluated responses with method identities hidden.
Table 7. Open-ended relation descriptions of specified entity pairs. Rates are percentages. Δ = IVE − Vanilla (percentage points).
| Method | Correct target relation ↑ | Unsupported target relation ↓ | Unanswered / ambiguous | Additional relation hallucination ↓ |
|---|---|---|---|---|
| Vanilla | 61.87 | 27.56 | 10.57 | 15.79 |
| VCD | 63.47 | 26.32 | 10.21 | 14.15 |
| PAI | 65.17 | 24.85 | 9.98 | 12.83 |
| MemVR | 63.78 | 25.28 | 10.94 | 12.26 |
| IVE | 66.10 | 23.53 | 10.37 | 11.11 |
| Δ: IVE − Vanilla | +4.23 | −4.02 | −0.21 | −4.68 |
Compared with Vanilla, IVE improves target-relation accuracy and reduces both the unsupported-target rate and the proportion of responses with unsupported relations beyond the target. This evaluation is now reported in Section 4.2, with the scoring protocol in Appendix B of the revised manuscript.
Response (Part I)
We thank the reviewer for the careful assessment and for recognizing IVE's usefulness. Below, we address each concern with clarifications and additional evidence.
1. Input History Retains Most Short-Answer Gains, while Generation History Improves Descriptions
We thank the reviewer for suggesting this ablation. IVE already uses input-text attention history for its first output prediction. Input history alone retains most of the short-answer gains, while generation history further improves relation descriptions.
Input-only freezes prompt history statistics while selecting and modulating tokens using current attention. Generation-only accumulates generated-token query history and starts modulation at the third output prediction. All retain full text context and original backbone-specific settings without retuning.
(a) Input history retains most short-answer gains.
We use the complete Reefknot Y/N and MCQ evaluation sets.
Table 1. Short-answer performance (R_score, %). In both tables, Δ = full IVE − input history only (percentage points).
Generation history adds 0.35–0.81 percentage points across the six settings, with statistical support in three.
(b) Generation history further improves relation descriptions.
We reuse the original 2,000 LLaVA-1.5 relation-description inputs and GPT-4o scoring protocol.
Table 2. Relation-description outcomes. Rates are percentages, with response counts in parentheses.
Compared with input history alone, full IVE improves target-relation accuracy and reduces the unsupported-target rate, with statistical support for both. The revised manuscript summarizes this ablation in Section 4.3 and provides both result tables in Appendix C.6.
2. IVE Outperforms Both Temporal Baselines under Equal Tuning Budgets
We thank the reviewer for highlighting tuning fairness and uncertainty. With equal candidate budgets, IVE retains statistically supported gains over ADAPT-TF and ANL in all 12 comparisons.
Each method receives 12 candidates per backbone, including IVE's coefficient/layer choices and default/inherited settings. Before testing, each selects and freezes one configuration for both formats by mean Y/N and MCQ R_score on shared validation data without test image/scene overlap. ANL's detector remains fixed.
Intervals use paired bootstrap clustered by source image/scene, jointly corrected across this table and all applicable original baseline comparisons on Reefknot, Hal-Eval Relation, and MMRA.
Table 3. Equal-budget comparison on Reefknot (R_score, %). Differences are in percentage points.
Tuning improves both baselines, while IVE retains its advantage. The revised manuscript summarizes this comparison in Sections 4.1–4.2, with the protocol in Appendix B and results and uncertainty estimates in Appendices C.1–C.2.