- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies hallucination of MLLMs. The authors find that attention sink tokens behave differently across attention heads: unfaithful heads allocate more perceptual attention to sink positions, while their sink contributions are less semantically aligned with the input than those of faithful heads. To this end, the authors propose Head-Selective Adaptive Reweighting for Perceptual Sinks (HARPS), a training-free method that selectively reweights attention to perceptual sink tokens in faithful heads and adapts the layer-wise intervention to each input-prompt pair. Experiments on four MLLMs show improvement.
Critical strengths
- Overall, the paper is well-written and easy to follow.
- Experiments show improvement on four MLLMs and several benchmarks.
Critical weaknesses
Limited Novelty
- The recognition mechanism of head faithfulness is of limited novelty. The authors adopt the contrastive influence of modular attribution and intervention [1], which first proposed head-level attribution.
- Perceptual sink detection and attention reweighing are both existing technologies. The authors simply adopt existing methods. Another critical question is, only augmenting sink attention on non-hallucination-related heads is a straightforward method if previous studies have found that some heads are strongly related to halluicination and that sink reweighing can improve grounding?
- Adaptive layer-wise reweighing is just a heuristic and hyper-parameter tuning.
Incremental Improvement
-
The contribution of head selectivity reflected by performance improvement is little. Compared with random head selection, faithful head selection only brings an improvement of 0.56%, meaning only 30 more correct answers within the 5302 questions of AVHBench.
-
The contribution of reweighing is limited. Compared with constant strength reweighing, the adaptive mechanism only brings an improvement of 0.66%, meaning only 35 more correct answers within the 5302 questions of AVHBench.
-
If most improvement comes from simple fixed reweighing, why do we need a complicated adaptive structure containing MDS, layer-wise sink profiles and modality routing? What would happen if the authors only used example-specific sink distribution, prompt-dependent modality routing, random or uniform routing weights, simple per-layer tuning, etc.?
-
The ablation studies on head attribution, perceptual sink reweighing, and adaptive modality routing are missing.
Questions for authors
Please see the weaknesses above.
8 comments
Author Response (2/3)
Q4: Compared with random head selection, faithful head selection only brings an improvement of 0.56%, meaning only 30 more correct answers within the 5302 questions of AVHBench.
We would like to correct a factual misunderstanding in this comment. The 0.56 percentage-point difference reported in Table 3 was measured on a 300-example development subset, not the full AVHBench benchmark of 5,302 examples. Therefore, interpreting this difference as only 30 additional correct predictions on the full benchmark is incorrect. The original ablation was intended to guide our method design without making design choices based on downstream test results.
We further evaluate faithful-head selection on the full AVHBench benchmark (5,302 examples), using the same faithful-head set as on the development subset and comparing against random head selection with different head counts across three seeds:
| Head Selection | Seed 1 | Seed 2 | Seed 3 | Mean ± Std |
|---|---|---|---|---|
| Random 50 | 76.4 | 74.3 | 74.7 | 75.1 ± 1.1 |
| Random 150 | 74.9 | 74.6 | 77.2 | 75.6 ± 1.4 |
| Random 300 | 74.3 | 73.4 | 74.2 | 74.0 ± 0.5 |
| Random 500 | 75.1 | 74.3 | 73.7 | 74.4 ± 0.7 |
| Random 508 | 75.0 | 75.1 | 74.3 | 74.8 ± 0.4 |
| Faithful 508 (Ours) | — | — | — | 77.6 |
On the full benchmark, faithful-head selection achieves 77.6% accuracy, outperforming random selection across different head counts (74.0%–75.6%). In particular, randomly selecting 500 heads achieves only 74.4% on average, a difference of 3.2 percentage points. These results provide stronger evidence for the effectiveness of faithful-head selection than the development-set comparison cited in the review.
Q5: Compared with constant strength reweighing, the adaptive mechanism only brings an improvement of 0.66%, meaning only 35 more correct answers within the 5302 questions of AVHBench. What would happen if the authors only used example-specific sink distribution, prompt-dependent modality routing, random or uniform routing weights, simple per-layer tuning, etc.?
As in Q4, we would like to clarify that the 0.66 percentage-point improvement reported in Table 4 was measured on a 300-example development subset, not the full AVHBench benchmark of 5,302 examples. Therefore, the reviewer's calculation of 35 additional correct predictions on the full benchmark is incorrect.
We further evaluate adaptive reweighting on the full AVHBench benchmark (5,302 examples), using the same method configuration as on the development subset. We also include uniform and random routing as alternative routing baselines:
| Configuration | AVHBench Accuracy (%) |
|---|---|
| Greedy baseline | 75.2 |
| Constant | 76.0 |
| Example-specific profiles + uniform routing | 76.1 |
| Example-specific profiles + random routing | 75.8 ± 0.31 |
| Full adaptive HARPS (Ours) | 77.6 |
On the full benchmark, adaptive HARPS improves over constant-strength reweighting by 1.6 percentage points, compared with 0.66 points on the development subset. The uniform and random routing baselines achieve 76.1% and 75.8%, respectively, further demonstrating the benefit of prompt-dependent modality routing.
Author Response (3/3)
Q6: The ablation studies on head attribution, perceptual sink reweighing, and adaptive modality routing are missing.
We would like to clarify that ablation studies for these components are already presented in Section 4.3 and Appendix G, and we have further extended the evaluation on the full AVHBench benchmark, as discussed in our responses to Q4 and Q5.
Head attribution. To evaluate the contribution of head faithfulness estimation, Table 3 includes an intervention that boosts perceptual sinks in all attention heads, without using head attribution for selection. This achieves 76.00% accuracy, compared with 77.67% when selectively boosting sinks in faithful heads. The table also compares random head selection, boosting sinks in unfaithful heads, and suppressing unfaithful heads. Appendix G (Table 8) further examines how different combinations of proxy corpora affect head attribution.
Perceptual sink reweighting. Table 3 directly evaluates different sink reweighting strategies. Boosting sinks in faithful heads improves accuracy from 73.00% with greedy decoding to 77.67%, whereas boosting sinks in unfaithful heads achieves only 73.33%. These comparisons examine both the effectiveness of sink reweighting and the importance of selecting appropriate heads for intervention.
Adaptive modality routing. In HARPS, modality routing is specifically designed to combine the three modality-specific layer-wise sink profiles (Eqs. 3–4). Without these profiles, there are no modality-specific quantities for the routing weights to act on, so routing cannot be meaningfully evaluated as a standalone intervention. We therefore evaluate its contribution by retaining the example-specific sink profiles while replacing prompt-dependent routing with uniform or random routing. On the full AVHBench benchmark (5,302 examples), HARPS achieves 77.6% accuracy, compared with 76.1% for uniform routing and 75.8% for random routing. Table 4 and Appendix G (Table 7) additionally compare constant-strength and different adaptive reweighting configurations.
Together, these experiments evaluate the individual design choices behind HARPS and provide direct evidence for the contributions of head attribution, sink reweighting, and adaptive modality routing.
We appreciate the reviewer's time and consideration. We hope our responses have clarified the important misunderstandings regarding our contributions, methodology, and experimental results. In particular, our full-benchmark evaluations provide stronger evidence for the effectiveness of both head-selective intervention and adaptive reweighting. We hope these clarifications and results provide a more accurate basis for evaluating our work, and that the reviewer will reconsider their assessment accordingly.
Author Response (1/3)
We sincerely thank the reviewer for their evaluation and feedback. We appreciate the recognition of the clarity of our manuscript and the consistent improvements across four multimodal LLMs and diverse benchmarks. We also noticed several important misunderstandings regarding our methodology and experimental results in the review, which we would like to clarify below.
Q1: The recognition mechanism of head faithfulness is of limited novelty. The authors adopt the contrastive influence of modular attribution and intervention [1], which first proposed head-level attribution.
We would like to clarify a misunderstanding regarding our novelty. We do not claim head-level attribution itself as a novel contribution. As explicitly stated in Section 3.1, we adopt the established contrastive influence criterion from Yang et al. (2025a) as part of our analysis.
Head attribution serves as a tool in our framework, rather than the contribution we claim. While prior work establishes how to identify hallucination-related attention heads, it does not investigate how perceptual sinks behave differently across these heads or how this relationship can inform more effective interventions. Our work addresses this previously unexplored connection through a systematic analysis of head-dependent sink behavior (Fig. 1 and Appendix H), leading to the head-selective intervention proposed in HARPS.
Our contribution therefore goes beyond applying an existing attribution technique: it lies in the new mechanistic understanding and intervention strategy enabled by that analysis.
Q2: The authors simply adopt existing methods and only augmenting sink attention on non-hallucination-related heads is a straightforward method if previous studies have found that some heads are strongly related to halluicination and that sink reweighing can improve grounding?
We believe this characterization overlooks the key empirical finding that motivates our method. While head attribution and perceptual sink reweighting have been studied independently, their combination does not directly establish which heads should receive stronger sink attention, or whether sink tokens provide equally reliable perceptual information across different heads. To the best of our knowledge, this relationship has not been systematically investigated in prior work.
Our analysis (Fig. 1 and Appendix H) reveals a counterintuitive finding: unfaithful heads allocate more attention to perceptual sinks, yet their sink contributions are substantially less semantically aligned with the input than those of faithful heads. This demonstrates that stronger sink attention does not necessarily imply better perceptual grounding and provides the motivation for selectively strengthening sink attention in faithful heads.
Table 3 further supports this design choice: boosting sink attention in faithful heads achieves 77.67% accuracy, compared with 76.00% when boosting all heads and 73.33% when boosting unfaithful heads. We further extend our head-selection ablation to the full AVHBench benchmark (5,302 examples), where faithful-head selection achieves 77.6% accuracy, compared with 74.4% on average for randomly selecting 500 heads, a larger difference of 3.2 percentage points. See our response to Q4 below.
These findings demonstrate that the effectiveness of sink reweighting depends on which heads are targeted, rather than following directly from combining the two existing techniques.
Q3: Adaptive layer-wise reweighing is just a heuristic and hyper-parameter tuning.
We would like to point out a factual error in this comment: our adaptive layer-wise reweighting does not involve any layer-specific hyperparameter tuning. As detailed in Section 3.3 (Eqs. 3–4), the reweighting schedule is computed from the input-specific distributions of audio, visual, and audio-visual sink tokens, combined with prompt-dependent modality routing weights. These quantities are computed for each input-prompt pair, without training or layer-specific optimization. The only performance-tuned hyperparameter is the global scaling factor γbase, which is selected on a held-out development set and kept fixed across downstream benchmarks for each model.
Fig. 3 illustrates how the resulting layer-wise schedules vary across different input-prompt pairs. Moreover, our full AVHBench evaluation (5,302 examples) shows that adaptive reweighting achieves 77.6% accuracy, compared with 76.0% for constant-strength reweighting (see our response to Q5 below). The layer-wise variation is therefore determined by input characteristics and sink distributions, not manually tuned layer-specific hyperparameters, and its effectiveness is supported by our empirical results.