acceptodds
Under review as a conference paper at ICLR 2027

HARPS: Mitigating Multimodal Hallucinations via Head-Selective and Adaptive Perceptual Sink Reweighting

24 views

Abstract

Multimodal large language models increasingly reason over multiple perceptual modalities, yet hallucination remains a persistent challenge. Recent work has shown that attention sink tokens can concentrate high-level perceptual and cross-modal information, motivating their use as inference-time intervention targets. We find, however, that these sink tokens behave differently across attention heads: unfaithful heads allocate more perceptual attention to sink positions, while their sink contributions are less semantically aligned with the input than those of faithful heads. Motivated by this finding, we propose Head-Selective Adaptive Reweighting for Perceptual Sinks (HARPS), a training-free method that selectively reweights attention to perceptual sink tokens in faithful heads and adapts the layer-wise intervention to each input-prompt pair. Across four multimodal LLMs spanning different families, architectures, and scales, HARPS consistently improves both hallucination-focused and general multimodal reasoning benchmarks, while remaining substantially more efficient than other baseline training-free methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal LLMs, Hallucination, Multimodal Attention Sinks, Faithful Heads

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_4A34OpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies hallucination of MLLMs. The authors find that attention sink tokens behave differently across attention heads: unfaithful heads allocate more perceptual attention to sink positions, while their sink contributions are less semantically aligned with the input than those of faithful heads. To this end, the authors propose Head-Selective Adaptive Reweighting for Perceptual Sinks (HARPS), a training-free method that selectively reweights attention to perceptual sink tokens in faithful heads and adapts the layer-wise intervention to each input-prompt pair. Experiments on four MLLMs show improvement.

Critical strengths
  1. Overall, the paper is well-written and easy to follow.
  2. Experiments show improvement on four MLLMs and several benchmarks.
Critical weaknesses

Limited Novelty

  1. The recognition mechanism of head faithfulness is of limited novelty. The authors adopt the contrastive influence of modular attribution and intervention [1], which first proposed head-level attribution.
  2. Perceptual sink detection and attention reweighing are both existing technologies. The authors simply adopt existing methods. Another critical question is, only augmenting sink attention on non-hallucination-related heads is a straightforward method if previous studies have found that some heads are strongly related to halluicination and that sink reweighing can improve grounding?
  3. Adaptive layer-wise reweighing is just a heuristic and hyper-parameter tuning.

Incremental Improvement

  1. The contribution of head selectivity reflected by performance improvement is little. Compared with random head selection, faithful head selection only brings an improvement of 0.56%, meaning only 30 more correct answers within the 5302 questions of AVHBench.

  2. The contribution of reweighing is limited. Compared with constant strength reweighing, the adaptive mechanism only brings an improvement of 0.66%, meaning only 35 more correct answers within the 5302 questions of AVHBench.

  3. If most improvement comes from simple fixed reweighing, why do we need a complicated adaptive structure containing MDS, layer-wise sink profiles and modality routing? What would happen if the authors only used example-specific sink distribution, prompt-dependent modality routing, random or uniform routing weights, simple per-layer tuning, etc.?

  4. The ablation studies on head attribution, perceptual sink reweighing, and adaptive modality routing are missing.

Questions for authors

Please see the weaknesses above.

8 comments
AuthorsOpenReview ↗

Author Response (1/3)

We sincerely thank the reviewer for their evaluation and feedback. We appreciate the recognition of the clarity of our manuscript and the consistent improvements across four multimodal LLMs and diverse benchmarks. We also noticed several important misunderstandings regarding our methodology and experimental results in the review, which we would like to clarify below.

Q1: The recognition mechanism of head faithfulness is of limited novelty. The authors adopt the contrastive influence of modular attribution and intervention [1], which first proposed head-level attribution.

We would like to clarify a misunderstanding regarding our novelty. We do not claim head-level attribution itself as a novel contribution. As explicitly stated in Section 3.1, we adopt the established contrastive influence criterion from Yang et al. (2025a) as part of our analysis.

Head attribution serves as a tool in our framework, rather than the contribution we claim. While prior work establishes how to identify hallucination-related attention heads, it does not investigate how perceptual sinks behave differently across these heads or how this relationship can inform more effective interventions. Our work addresses this previously unexplored connection through a systematic analysis of head-dependent sink behavior (Fig. 1 and Appendix H), leading to the head-selective intervention proposed in HARPS.

Our contribution therefore goes beyond applying an existing attribution technique: it lies in the new mechanistic understanding and intervention strategy enabled by that analysis.


Q2: The authors simply adopt existing methods and only augmenting sink attention on non-hallucination-related heads is a straightforward method if previous studies have found that some heads are strongly related to halluicination and that sink reweighing can improve grounding?

We believe this characterization overlooks the key empirical finding that motivates our method. While head attribution and perceptual sink reweighting have been studied independently, their combination does not directly establish which heads should receive stronger sink attention, or whether sink tokens provide equally reliable perceptual information across different heads. To the best of our knowledge, this relationship has not been systematically investigated in prior work.

Our analysis (Fig. 1 and Appendix H) reveals a counterintuitive finding: unfaithful heads allocate more attention to perceptual sinks, yet their sink contributions are substantially less semantically aligned with the input than those of faithful heads. This demonstrates that stronger sink attention does not necessarily imply better perceptual grounding and provides the motivation for selectively strengthening sink attention in faithful heads.

Table 3 further supports this design choice: boosting sink attention in faithful heads achieves 77.67% accuracy, compared with 76.00% when boosting all heads and 73.33% when boosting unfaithful heads. We further extend our head-selection ablation to the full AVHBench benchmark (5,302 examples), where faithful-head selection achieves 77.6% accuracy, compared with 74.4% on average for randomly selecting 500 heads, a larger difference of 3.2 percentage points. See our response to Q4 below.

These findings demonstrate that the effectiveness of sink reweighting depends on which heads are targeted, rather than following directly from combining the two existing techniques.

Q3: Adaptive layer-wise reweighing is just a heuristic and hyper-parameter tuning.

We would like to point out a factual error in this comment: our adaptive layer-wise reweighting does not involve any layer-specific hyperparameter tuning. As detailed in Section 3.3 (Eqs. 3–4), the reweighting schedule is computed from the input-specific distributions of audio, visual, and audio-visual sink tokens, combined with prompt-dependent modality routing weights. These quantities are computed for each input-prompt pair, without training or layer-specific optimization. The only performance-tuned hyperparameter is the global scaling factor γbase\gamma_{\mathrm{base}}, which is selected on a held-out development set and kept fixed across downstream benchmarks for each model.

Fig. 3 illustrates how the resulting layer-wise schedules vary across different input-prompt pairs. Moreover, our full AVHBench evaluation (5,302 examples) shows that adaptive reweighting achieves 77.6% accuracy, compared with 76.0% for constant-strength reweighting (see our response to Q5 below). The layer-wise variation is therefore determined by input characteristics and sink distributions, not manually tuned layer-specific hyperparameters, and its effectiveness is supported by our empirical results.

AuthorsOpenReview ↗

Author Response (2/3)

Q4: Compared with random head selection, faithful head selection only brings an improvement of 0.56%, meaning only 30 more correct answers within the 5302 questions of AVHBench.

We would like to correct a factual misunderstanding in this comment. The 0.56 percentage-point difference reported in Table 3 was measured on a 300-example development subset, not the full AVHBench benchmark of 5,302 examples. Therefore, interpreting this difference as only 30 additional correct predictions on the full benchmark is incorrect. The original ablation was intended to guide our method design without making design choices based on downstream test results.

We further evaluate faithful-head selection on the full AVHBench benchmark (5,302 examples), using the same faithful-head set as on the development subset and comparing against random head selection with different head counts across three seeds:

Head SelectionSeed 1Seed 2Seed 3Mean ± Std
Random 5076.474.374.775.1 ± 1.1
Random 15074.974.677.275.6 ± 1.4
Random 30074.373.474.274.0 ± 0.5
Random 50075.174.373.774.4 ± 0.7
Random 50875.075.174.374.8 ± 0.4
Faithful 508 (Ours)———77.6

On the full benchmark, faithful-head selection achieves 77.6% accuracy, outperforming random selection across different head counts (74.0%–75.6%). In particular, randomly selecting 500 heads achieves only 74.4% on average, a difference of 3.2 percentage points. These results provide stronger evidence for the effectiveness of faithful-head selection than the development-set comparison cited in the review.

Q5: Compared with constant strength reweighing, the adaptive mechanism only brings an improvement of 0.66%, meaning only 35 more correct answers within the 5302 questions of AVHBench. What would happen if the authors only used example-specific sink distribution, prompt-dependent modality routing, random or uniform routing weights, simple per-layer tuning, etc.?

As in Q4, we would like to clarify that the 0.66 percentage-point improvement reported in Table 4 was measured on a 300-example development subset, not the full AVHBench benchmark of 5,302 examples. Therefore, the reviewer's calculation of 35 additional correct predictions on the full benchmark is incorrect.

We further evaluate adaptive reweighting on the full AVHBench benchmark (5,302 examples), using the same method configuration as on the development subset. We also include uniform and random routing as alternative routing baselines:

ConfigurationAVHBench Accuracy (%)
Greedy baseline75.2
Constant γbase\gamma_{\mathrm{base}}76.0
Example-specific profiles + uniform routing76.1
Example-specific profiles + random routing75.8 ± 0.31
Full adaptive HARPS (Ours)77.6

On the full benchmark, adaptive HARPS improves over constant-strength reweighting by 1.6 percentage points, compared with 0.66 points on the development subset. The uniform and random routing baselines achieve 76.1% and 75.8%, respectively, further demonstrating the benefit of prompt-dependent modality routing.

AuthorsOpenReview ↗

Author Response (3/3)

Q6: The ablation studies on head attribution, perceptual sink reweighing, and adaptive modality routing are missing.

We would like to clarify that ablation studies for these components are already presented in Section 4.3 and Appendix G, and we have further extended the evaluation on the full AVHBench benchmark, as discussed in our responses to Q4 and Q5.

Head attribution. To evaluate the contribution of head faithfulness estimation, Table 3 includes an intervention that boosts perceptual sinks in all attention heads, without using head attribution for selection. This achieves 76.00% accuracy, compared with 77.67% when selectively boosting sinks in faithful heads. The table also compares random head selection, boosting sinks in unfaithful heads, and suppressing unfaithful heads. Appendix G (Table 8) further examines how different combinations of proxy corpora affect head attribution.

Perceptual sink reweighting. Table 3 directly evaluates different sink reweighting strategies. Boosting sinks in faithful heads improves accuracy from 73.00% with greedy decoding to 77.67%, whereas boosting sinks in unfaithful heads achieves only 73.33%. These comparisons examine both the effectiveness of sink reweighting and the importance of selecting appropriate heads for intervention.

Adaptive modality routing. In HARPS, modality routing is specifically designed to combine the three modality-specific layer-wise sink profiles (Eqs. 3–4). Without these profiles, there are no modality-specific quantities for the routing weights to act on, so routing cannot be meaningfully evaluated as a standalone intervention. We therefore evaluate its contribution by retaining the example-specific sink profiles while replacing prompt-dependent routing with uniform or random routing. On the full AVHBench benchmark (5,302 examples), HARPS achieves 77.6% accuracy, compared with 76.1% for uniform routing and 75.8% for random routing. Table 4 and Appendix G (Table 7) additionally compare constant-strength and different adaptive reweighting configurations.

Together, these experiments evaluate the individual design choices behind HARPS and provide direct evidence for the contributions of head attribution, sink reweighting, and adaptive modality routing.


We appreciate the reviewer's time and consideration. We hope our responses have clarified the important misunderstandings regarding our contributions, methodology, and experimental results. In particular, our full-benchmark evaluations provide stronger evidence for the effectiveness of both head-selective intervention and adaptive reweighting. We hope these clarifications and results provide a more accurate basis for evaluating our work, and that the reviewer will reconsider their assessment accordingly.

Reviewer_39e1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HARPS, a training-free method for mitigating hallucinations in multimodal LLMs. The method uses intervention-based attribution on proxy corpora to partition attention heads into faithful and unfaithful groups, then selectively increases attention to perceptual sink tokens in the faithful heads. It further adjusts the layer-wise intervention strength using example-specific sink distributions and prompt-dependent modality routing.

The evaluation covers four models, multiple modality configurations, general multimodal benchmarks, and open-ended captioning with CHAIR metrics. The paper also reports ablations on head selection and adaptive reweighting, together with inference latency measurements.

Critical strengths
  1. The paper provides a concrete motivation for head-selective sink intervention.

    The combination of head attribution and semantic probing suggests that different heads can extract different information from the same sink tokens. This motivates accounting for head-level differences when intervening on sinks. The proposed connection is specific and technically meaningful, even though it builds on established attribution and sink-intervention techniques.

  2. The complete method shows reasonably consistent benefits across models and tasks.

    The evaluation includes multiple model scales and modality configurations, and reports both general task performance and open-ended hallucination metrics. Selecting hyperparameters on a development set and keeping them fixed for subsequent evaluations strengthens the main empirical evidence. Computing the schedule during initialization also provides a clear efficiency motivation compared with approaches that require separate original and calibrated forward passes at every decoding step.

Critical weaknesses
  1. The independent contribution of the key new components is not yet sufficiently established.

    The central ablations are conducted primarily on one model and a 300-example development set. In Table 3, faithful-head intervention improves over random-head intervention by 0.56 percentage points; in Table 4, the full adaptive strategy improves over constant-strength intervention by approximately 0.66 percentage points. Without uncertainty estimates and evaluation on independent test data, it is difficult to determine whether these differences are stable and generalizable.

  2. The head taxonomy and comparisons with closely related interventions require stronger validation.

    Many attribution scores are close to zero, while the partition uses a zero threshold and labels a head unfaithful if its score is positive on any proxy corpus. The stability of this classification under calibration-data resampling remains unclear. A comparison with a complete head-level intervention baseline would also help establish the contribution more precisely.

Questions for authors
  1. In the current ablation experiments, faithful-head intervention outperforms random-head intervention by only approximately 0.56 percentage points, while the adaptive strategy improves over constant-strength intervention by approximately 0.66 percentage points. Could you repeat these comparisons on a larger, independent evaluation set and perform size-matched random-head selection across multiple random seeds, reporting mean performance, standard deviations, and paired confidence intervals for the performance differences to assess whether these small gains are robust?

  2. How stable are the selected head sets and downstream results under resampling of the calibration examples? How is the accuracy of the grounded/hallucinated token labels used for attribution validated?

4 comments
AuthorsOpenReview ↗

Author Response (1/2)

We sincerely thank the reviewer for their detailed evaluation and valuable feedback. We appreciate their recognition of our motivation for head-selective intervention, consistent improvements across diverse models and benchmarks, and careful experimental design. We address the reviewer's questions below.

Q1: Can you repeat the comparison on a larger, independent evaluation set and perform size-matched random-head selection across multiple random seeds

Faithful-head selection vs. random-head selection

We would like to clarify two points about Table 3. First, the original ablation was conducted on a 300-example development subset to guide our method design, rather than to establish the final performance of faithful-head selection.

Second, the random baseline selects the same number of heads as the faithful heads identified by our method. Specifically, our offline attribution identifies 508 faithful heads out of 784, so the random baseline also selects 508 heads. This selection size comes directly from our faithfulness analysis, and under this setting, most randomly selected heads are expected to be faithful.

We further evaluate random-head selection on the full AVHBench benchmark (5,302 examples), using different head counts and three random seeds. We use the same faithful-head set as on the dev set, without re-selecting heads for the full benchmark.

Head SelectionSeed 1Seed 2Seed 3Mean ± Std
Random 5076.474.374.775.1 ± 1.1
Random 15074.974.677.275.6 ± 1.4
Random 30074.373.474.274.0 ± 0.5
Random 50075.174.373.774.4 ± 0.7
Random 50875.075.174.374.8 ± 0.4
Faithful 508 (Ours)———77.6

Faithful-head selection achieves 77.6% accuracy, outperforming random selection across different head counts (74.0%–75.6%). Even randomly selecting 500 heads achieves only 74.4% on average, supporting the importance of identifying faithful heads rather than simply intervening on more heads.


Adaptive reweighting vs. constant-strength intervention

The 0.66 percentage-point improvement reported in Table 4 was measured on the 300-example development subset used to guide our method design. We further evaluate adaptive reweighting on the full AVHBench benchmark (5,302 examples), using the same method configuration as on the development set. We also include uniform and random routing as alternative routing baselines:

ConfigurationAVHBench Accuracy (%)
Greedy baseline75.2
Constant γbase\gamma_{\mathrm{base}}76.0
Example-specific profiles + uniform routing76.1
Example-specific profiles + random routing75.8 ± 0.31
Full adaptive HARPS (Ours)77.6

On the full benchmark, adaptive HARPS improves over constant reweighting by 1.6 percentage points, compared with 0.66 points on the development subset. Uniform and random routing achieve 76.1% and 75.8%, respectively, further supporting the contribution of prompt-dependent modality routing.

Unlike random-head selection, the constant and full-adaptive configurations do not involve random sampling, so the multiple-seed comparison requested for random-head selection does not directly apply here.

AuthorsOpenReview ↗

Author Response (2/2)

Q2: How stable are the selected head sets and downstream results under resampling of the calibration examples?

Stability of faithful-head selection under proxy-data resampling

Our original method uses the full set of examples from four diverse proxy corpora to estimate head faithfulness, rather than relying on randomly sampled subsets. Our goal is to obtain reliable and representative faithfulness estimates by using comprehensive proxy data.

To assess sensitivity to calibration-data sampling, we repeat head attribution using two different 50% subsets of the proxy data. We examine how many of the original 508 faithful heads are recovered and evaluate the resulting head selections on both the development set and the full AVHBench benchmark.

Head SelectionOriginal Faithful Heads Recovered (out of 508)Dev Accuracy (%)Full AVHBench Accuracy (%)
Full proxy data (Original)508 (100%)78.377.6
Resampled proxy data (Seed 1)416 (81.9%)77.376.8
Resampled proxy data (Seed 2)429 (84.4%)73.675.6
Greedy baseline—73.075.2

The two resampled subsets recover 81.9% and 84.4% of the original faithful heads, respectively. On the full AVHBench benchmark (5,302 examples), both resulting head selections outperform the greedy baseline (75.2%), achieving 76.8% and 75.6% accuracy. Using the full proxy data yields the highest accuracy of 77.6%, supporting our choice to leverage comprehensive proxy data for more reliable head faithfulness estimation.

Stability of hyperparameter calibration across development subsets

The only performance-tuned hyperparameter in HARPS is the global scaling factor γbase\gamma_{\mathrm{base}}. To assess its sensitivity to development-set sampling, we repeat calibration on two resampled development subsets, evaluating the same candidate values as in our original experiment.

Development Setγ=2\gamma=2γ=4\gamma=4γ=6\gamma=6γ=8\gamma=8γ=10\gamma=10γ=12\gamma=12BaselineBest γ\gamma
Original dev set76.6777.6777.0078.3377.0075.3373.008
Resampled dev set 176.0076.6776.6777.0076.0076.6775.678
Resampled dev set 273.3376.3375.6776.6775.0076.0072.008

All three development subsets consistently select γbase=8\gamma_{\mathrm{base}}=8 as the best-performing value. Although the absolute accuracies differ across subsets, the selected hyperparameter remains unchanged, demonstrating consistent calibration across these development samples.

Q3: How is the accuracy of the grounded/hallucinated token labels used for attribution validated?

Our grounded/hallucinated token labels are deterministically derived from the ground-truth annotations of four established datasets: ActivityNet, YouTube-VOS, LibriSpeech, and AudioSet. As detailed in Section 3.1 and Appendix A.1, we constrain model generation to each dataset's predefined label vocabulary and classify generated labels as grounded or hallucinated based on their membership in the ground-truth label set. This procedure requires no additional annotation, heuristic matching, or LLM-based judgment. The resulting labels are therefore reproducible and directly determined by the original dataset annotations.


We appreciate the reviewer's suggestions for additional experiments and will incorporate these evaluations into the revised manuscript. We hope our responses have addressed the reviewer's questions and clarified our key findings. We would be grateful if the reviewer could reconsider their rating in light of these responses.

Thank you for the detailed response. The clarifications and additional experiments have addressed some of my concerns. However, the contribution still appears to lie primarily in adapting and integrating existing mechanisms, and the conceptual and technical advances remain insufficiently clear. My concerns about methodological novelty therefore persist, and I have decided to maintain my original score.

We appreciate the reviewer's follow-up and acknowledgment of our additional experiments and clarifications.

We would like to further clarify the conceptual and technical contributions of HARPS. Our conceptual contribution is the previously unexplored finding that unfaithful heads allocate more attention to perceptual sinks, yet their sink contributions are substantially less semantically aligned with the input than those of faithful heads. As the reviewer recognized in the original review, this connection is specific and technically meaningful.

Our technical contribution is the intervention motivated by this finding: selectively strengthening perceptual sink attention in faithful heads, together with input-adaptive layer-wise reweighting. These design choices are supported by the ablations and full-benchmark results presented in our responses.

We would welcome more specific feedback on why these contributions are considered insufficiently novel, and leave their assessment to the Area Chair in light of the evidence and discussion presented.

Reviewer_sASbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HARPS, a training-free method for mitigating hallucinations in multimodal large language models through head-selective and adaptive perceptual sink reweighting. The authors first identify faithful and unfaithful attention heads using intervention-based attribution and observe that faithful heads extract more semantically relevant information from perceptual sink tokens, despite allocating less attention to them than unfaithful heads. Based on this finding, HARPS selectively amplifies attention to perceptual sink tokens in faithful heads while leaving unfaithful heads unchanged. The method further introduces an adaptive layer-wise reweighting strategy based on input-specific sink distributions and prompt-dependent modality requirements.

The authors evaluate HARPS on four multimodal models ranging from 3B to 30B parameters, covering audio-visual, image-audio, audio-only, and visual-only benchmarks. Experimental results show consistent improvements in hallucination mitigation and general multimodal reasoning compared with the base models and existing training-free methods.

Critical strengths

1、The empirical analysis reveals a non-trivial difference between faithful and unfaithful heads in how they use perceptual sink tokens. This observation is relevant because it provides a concrete basis for questioning uniform sink intervention across all heads.

2、Experimental validation is relatively comprehensive. The method is evaluated across more than one multimodal models, multiple architectures and scales, and both hallucination-focused and general multimodal benchmarks, which provides reasonably broad evidence about its robustness.

Critical weaknesses
  1. The novelty over prior head-selective and sink-based intervention methods is not novel. HARPS combines hallucination-related head attribution with perceptual sink reweighting, both of which have close precedents. The paper would need a clearer demonstration that its head-dependent sink analysis leads to a substantively new mechanism rather than an incremental combination of existing techniques.

  2. The evidence for the necessity of faithful-head selection is weak. In the head-selection ablation, boosting faithful heads only slightly outperforms boosting a random subset of heads. This makes it unclear whether the proposed head taxonomy is a critical component of the method.

  3. The adaptive reweighting component is only partially validated. The full adaptive strategy yields only a modest gain over constant reweighting, and the contributions of example-specific sink distributions and prompt-dependent routing are not separately isolated. This weakens the claim that the adaptive design is essential.

Questions for authors
  1. Table 3 reports only one random-head baseline. What is the mean and variance over multiple random head subsets matched to the faithful-head set? This is important for determining whether the proposed head selection is meaningfully better than random selection.

  2. In Table 4, example-specific sink profiles and prompt-dependent modality routing are introduced together. What are their individual contributions when ablated separately?

  3. How sensitive is the faithful/unfaithful head partition to small changes around the zero attribution threshold? Since many attribution scores are close to zero, this could materially affect the resulting head set and the conclusions drawn from it.

4 comments
AuthorsOpenReview ↗

Author Response (1/2)

We thank the reviewer for the thoughtful feedback. We appreciate the recognition of our empirical findings and comprehensive evaluation across diverse models and benchmarks. We also appreciate the opportunity to clarify our contributions and provide further evidence for our key design choices.

Novelty beyond existing head attribution and sink reweighting

We appreciate the opportunity to clarify our novelty. While prior work has studied hallucination-related attention heads and perceptual sinks separately, our contribution lies in the previously unexplored interaction between them and the intervention strategy it motivates. Our analysis (Fig. 1 and Appendix H) reveals a counterintuitive finding: unfaithful heads allocate substantially more attention to perceptual sinks, yet their sink contributions are much less semantically aligned with the input than those of faithful heads. This shows that simply increasing attention to sink tokens does not necessarily lead to better perceptual grounding.

Based on this finding, HARPS selectively strengthens sink attention in faithful heads while leaving unfaithful heads unchanged. Table 3 supports this design choice: boosting sinks in faithful heads achieves 77.67% accuracy, outperforming both boosting sinks in all heads (76.00%) and suppressing unfaithful heads (73.00%).


Q1: Table 3 reports only one random-head baseline. What is the mean and variance over multiple random head subsets matched to the faithful-head set? This is important for determining whether the proposed head selection is meaningfully better than random selection.

We thank the reviewer for this suggestion. We would like to clarify two points about Table 3.

First, the original ablation was conducted on a 300-example development subset to guide our method design, rather than to evaluate the final performance of faithful-head selection. This allowed us to compare different intervention strategies without making design choices based on downstream test results.

Second, the random baseline in Table 3 selects the same number of heads as the faithful heads identified by our method. Specifically, our offline attribution identifies 508 faithful heads out of 784, so the random baseline also selects 508 heads. This selection size comes directly from our faithfulness analysis, and under this setting, most randomly selected heads are expected to be faithful.

Following the reviewer's suggestion, we extend the random-head ablation to different head counts and three random seeds, evaluated on the full AVHBench benchmark (5,302 examples). We use the same faithful-head set as on the dev set, without re-selecting heads for the full benchmark.

Head SelectionSeed 1Seed 2Seed 3Mean ± Std
Random 5076.474.374.775.1 ± 1.1
Random 15074.974.677.275.6 ± 1.4
Random 30074.373.474.274.0 ± 0.5
Random 50075.174.373.774.4 ± 0.7
Random 50875.075.174.374.8 ± 0.4
Faithful 508 (Ours)———77.6

Faithful-head selection achieves 77.6% accuracy, outperforming random selection across different head counts (74.0%–75.6%). Even randomly selecting 500 heads achieves only 74.4% on average, showing that the improvement comes from selecting faithful heads rather than simply intervening on more heads.

AuthorsOpenReview ↗

Author Response (2/2)

Q2: In Table 4, example-specific sink profiles and prompt-dependent modality routing are introduced together. What are their individual contributions when ablated separately?

We thank the reviewer for this question. In HARPS, modality routing is specifically designed to combine the three modality-specific layer-wise sink profiles (Eqs. 3–4). The routing weights are applied to these profiles, so without the profiles, there would be no modality-specific quantities for the routing weights to operate on. This is why our ablation retains the example-specific sink profiles while varying the routing strategy.

As in our response to the previous question, we report ablation results on the full AVHBench benchmark (5,302 examples). We further include uniform and random routing as alternative routing baselines:

ConfigurationAVHBench Accuracy (%)
Greedy baseline75.2
Constant γbase\gamma_{\mathrm{base}}76.0
Example-specific profiles + uniform routing76.1
Example-specific profiles + random routing75.8 ± 0.31
Full adaptive HARPS (Ours)77.6

Full HARPS achieves 77.6% accuracy, outperforming constant reweighting (76.0%), uniform routing (76.1%), and random routing (75.8%). Uniformly combining the example-specific profiles provides only a small improvement over constant reweighting, whereas combining them with prompt-dependent modality routing yields a substantially larger gain.

Q3: How sensitive is the faithful/unfaithful head partition to small changes around the zero attribution threshold? Since many attribution scores are close to zero, this could materially affect the resulting head set and the conclusions drawn from it.

We thank the reviewer for raising this question. The zero threshold is motivated by the semantics of our contrastive attribution criterion (Eq. 1), rather than selected through downstream hyperparameter tuning. A positive attribution score indicates that a head contributes more strongly to hallucinated than grounded generation. We therefore use zero as the natural decision boundary for identifying unfaithful heads.

To examine the sensitivity of this choice, we vary the attribution threshold δ\delta and evaluate the resulting faithful-head selections on the full AVHBench benchmark (5,302 examples):

Attribution Threshold δ\deltaAVHBench Accuracy (%)
Greedy baseline75.2
−0.001-0.00177.0
−0.0005-0.000577.5
−0.0001-0.000176.8
00 (Ours)77.6
+0.0001+0.000176.9
+0.0005+0.000577.1
+0.001+0.00176.9

Although changing the threshold alters the selected faithful-head sets, downstream performance remains consistently above the greedy baseline across all tested thresholds. Accuracy ranges from 76.8% to 77.6%, with a maximum difference of 0.8 percentage points from our original setting. The original zero threshold also achieves the highest accuracy, although its selection is based on the attribution criterion rather than downstream performance. These results indicate that the effectiveness of HARPS does not depend critically on the precise placement of the attribution threshold.


We sincerely appreciate the reviewer's thoughtful feedback and constructive suggestions, particularly the additional experiments proposed to strengthen our empirical evaluation. We believe these suggestions will make our study more comprehensive, and we will incorporate the new experimental results into the revised manuscript. We hope our responses have helped clarify the concerns raised and provided stronger evidence for our key design choices. We would be grateful if the reviewer could reconsider their assessment in light of these clarifications and results.

Thank you for your detailed explanation. The additional experiments are substantial. Most of my concerns are fulfilled. However, looking through the novelty concerns from other reviwers, I remain the novelty concerns about this work. I can't ignore that this work has a limited contribution in this field.

HARPS did a set of experiments for validation, which are solid. I definitely acknowledge the effectiveness of HARPS. However, for a conference paper, we expect a new insight, rather a combination of several technics (head-level hallucination attribution & sink reweighting). I choose to hold the score.

We appreciate the reviewer's follow-up and acknowledgment that our additional experiments have addressed most of their concerns.

Regarding novelty, we would like to emphasize that our contribution is not simply a combination of existing techniques, but the new insight that motivates their integration. As the reviewer recognized in the original review, our analysis reveals a non-trivial difference in how faithful and unfaithful heads use perceptual sinks. This finding is not established by prior head attribution or sink reweighting methods and directly motivates our head-selective intervention.

We have carefully addressed the reviewer's experimental concerns, and we appreciate their acknowledgment that the results are substantial and the effectiveness of HARPS is well supported. We would welcome a more specific discussion of why the head-dependent sink finding itself is considered insufficient as a novel contribution. We leave the assessment of this remaining point to the Area Chair in light of the evidence and discussion presented.

Reviewer_6HDiOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HARPS, a training‑free inference‑time intervention to mitigate multimodal hallucinations in audio‑visual LLMs. The method selectively amplifies attention to sink tokens only within faithful heads and adapts reweighting strength per input‑prompt pair. The authors validate HARPS across four multimodal models of varying scales on diverse hallucination‑focused and general multimodal benchmarks, with thorough ablations verifying each design choice, showing consistent performance gains and much lower inference overhead compared to competing training‑free baselines.

Critical strengths

Instead of purely heuristic decoding tweaks, HARPS is built on a concrete empirical finding: faithful and unfaithful attention heads produce drastically different semantic interpretations when reading the same perceptual sink token. The authors perform targeted interpretability probing to validate this head‑level difference.

HARPS demonstrates consistent improvements across four different multimodal models (3B‑30B, dense and MoE) on a broad suite of hallucination‑specific and general multimodal benchmarks. Comprehensive ablations validate each core design decision.

Critical weaknesses

The faithful/unfaithful head partition is pre‑computed offline using four fixed proxy datasets. There is no guarantee this head grouping generalizes to unseen modalities, new tasks, or domains dissimilar from those four corpora. While ablations test combinations of proxy corpora, the paper does not evaluate performance when applying this pre‑computed head set to substantially out‑of‑distribution inputs. Additionally, the sink‑detection threshold τ and intervention strength γ_base are tuned per individual model. This creates deployment risk: for a brand‑new multimodal model, practitioners would need extra offline attribution work and hyper‑parameter sweeping to run HARPS. This limits zero‑shot plug‑and‑play applicability, a key expected benefit for inference‑time intervention methods.

Though overall benchmark numbers are positive, the paper lacks systematic investigation of scenarios where HARPS could hurt performance. It does not quantify the fraction of test examples where applying HARPS degrades output quality, nor characterize what input properties trigger such failures. Qualitative examples only show successful cases. Without understanding failure triggers, we cannot fully assess real‑world reliability, and users have no guidance on when HARPS should or should not be enabled.

Questions for authors
  1. The faithful‑head partition is derived offline from four fixed proxy corpora. Can you report results where you directly apply this pre‑computed faithful‑head set to a task / domain substantially different from your four proxy corpora, without re‑running the head‑ablation attribution step?

  2. Sink‑detection threshold τ and γ_base are tuned separately for each evaluated model. For a completely new unseen multimodal LLM, can HARPS deliver competitive performance using fixed, pre‑defined τ and γ_base with no per‑model sweeping / offline head‑ablation computation? If not, this substantially reduces the practical plug‑and‑play value of your inference‑time method.

  3. Can you quantify the fraction of test instances across main benchmarks where HARPS degrades performance relative to the greedy baseline, and characterise the common input properties of these failure cases? We need to verify whether your attention modification introduces meaningful failure modes that are not visible only from aggregate benchmark scores.

  4. In your current setup, the faithful‑head set is computed once offline for each model before evaluation. Does the faithful‑unfaithful head grouping remain stable across different prompt styles, input lengths, or heavily‑noised multimodal inputs? If head faithfulness shifts strongly under these conditions, using a static pre‑computed head set during inference may become mis‑aligned at test‑time.

2 comments
AuthorsOpenReview ↗

Author Response (1/2)

We thank the reviewer for the thoughtful and encouraging feedback. We appreciate the recognition of our mechanistic findings, consistent improvements across diverse models, and comprehensive ablation studies. We answer the questions below.

Q1: Can you report results where you directly apply this pre‑computed faithful‑head set to a task / domain substantially different from your four proxy corpora, without re‑running the head‑ablation attribution step?

We would like to clarify that our current evaluation setting already follows the protocol suggested by the reviewer. For each model, head attribution is performed only once offline using four proxy corpora (ActivityNet, YouTube-VOS, LibriSpeech, and AudioSet), covering visual-only and audio-only inputs. The resulting faithful-head set is kept fixed across all downstream evaluations, without re-running attribution or performing task-specific adaptation.

The downstream benchmarks differ substantially from these proxy corpora in both task formulation and modality requirements. While attribution uses constrained label generation on unimodal inputs, our evaluations include open-ended audio-visual captioning on VGGSounder (Table 2) and diverse multimodal QA and reasoning tasks requiring joint audio-visual or image-audio understanding (Table 1). These results demonstrate cross-task and cross-modal transfer using the same pre-computed head partition.


Q2: Can HARPS generalize to unseen MLLMs without model-specific head attribution or hyperparameter tuning?

We thank the reviewer for raising this practical concern. HARPS requires one-time model-specific preparation for a previously unseen MLLM, but no model training or downstream task-specific adaptation.

Model-specific initialization. The sink-detection threshold τ\tau is determined from the activation distribution of the underlying language model backbone, following prior work on massive activations, rather than tuned on downstream benchmark performance (Appendix B) [1,2]. MLLMs sharing the same language model backbone may therefore reuse the sink configuration. Similarly, the faithful-head partition is model-specific because different language model backbones may have different layer configurations, numbers of attention heads, and internal representations. Both sink detection and head attribution are performed only once per model, with their results reused across downstream tasks.

Hyperparameter calibration. The only performance-tuned hyperparameter is the global scaling factor γbase\gamma_{\mathrm{base}}, selected on a held-out 300-example development set and fixed across downstream evaluations. To examine cross-model transfer, we apply γQwen\gamma_{\mathrm{Qwen}}, selected for Qwen2.5-Omni-7B, to VideoLLaMA2-7B without further tuning:

ModelBaselineγQwen\gamma_{\mathrm{Qwen}}Optimal γ\gamma
Qwen2.5-Omni-7B85.587.1 (+1.6)87.1 (+1.6)
VideoLLaMA2-7B77.477.7 (+0.3)78.3 (+0.9)

All results are on CMM. The transferred γQwen\gamma_{\mathrm{Qwen}} still improves performance on VideoLLaMA2-7B, while model-specific calibration achieves a larger gain. Such calibration is needed only once per model, not for individual downstream tasks.

[1] Kang et al. (2025). See What You Are Told: Visual Attention Sink in Large Multimodal Models. ICLR 2025.

[2] Sun et al. (2024). Massive Activations in Large Language Models. COLM 2024.

AuthorsOpenReview ↗

Author Response (2/2)

Q3: Can you quantify the fraction of test instances across main benchmarks where HARPS degrades performance relative to the greedy baseline, and characterise the common input properties of these failure cases?

We thank the reviewer for this suggestion. We further analyze cases where HARPS changes a correct greedy prediction into an incorrect one across different models and benchmarks. We report the regression rate, defined as the number of such cases divided by the total number of test examples in each benchmark.

ModelOmniBenchAVHBenchCMMAV-SpeakerBenchWorldSenseVideo-MMEOverall
Qwen2.5-Omni-3B4.47%3.24%2.38%3.54%2.93%1.50%2.98% (473)
Qwen2.5-Omni-7B4.55%1.94%2.58%3.83%2.24%1.67%2.50% (397)
Qwen3-Omni-30B1.84%1.34%3.00%1.16%2.71%1.83%1.93% (307)
VideoLLaMA2-7B0.79%1.15%0.79%0.87%0.95%0.72%0.94% (150)

The overall rates are aggregated across six benchmarks, covering 15,881 test examples per model. The numbers in parentheses indicate the total number of regression cases. Overall, these cases account for only 0.94%–2.98% of the evaluated examples, while HARPS consistently improves aggregate performance over greedy decoding.

We also manually inspected regression cases for each of the four models across different benchmarks. These cases involve diverse visual and audio content and question types, and we did not identify any consistent pattern associated with particular input characteristics. Together with the low regression rates, these observations suggest that HARPS does not introduce any apparent systematic failure modes.


Q4: Does the offline-estimated head faithfulness remain reliable across different input conditions at inference time?

Our goal is to estimate the overall faithfulness of each attention head once offline and use these estimates across downstream tasks. While the faithfulness of individual heads on a particular test example is not directly known at inference time, we aim to make our offline estimates as representative as possible. This motivates our use of four diverse audio and visual proxy corpora, rather than relying on a single data distribution. We expect that estimates capturing general head behavior should transfer to different downstream inputs. Our evaluations support this: the same pre-computed head selection consistently improves performance across benchmarks with different audio-visual content, prompt formulations, and reasoning tasks, including AV-SpeakerBench, WorldSense, OmniBench, Video-MME, CMM, and VGGSounder (Tables 1–2).

HARPS also accounts for differences between individual inputs through adaptive attention reweighting. Rather than applying the same reweighting strength to every example based solely on offline head estimates, HARPS adjusts the layer-wise intervention strength using the current input's perceptual sink distributions and prompt-dependent modality routing (Eqs. 3–4). This allows the intervention to respond to input-specific characteristics without re-estimating head faithfulness. Table 4 further shows that this adaptive strategy improves performance over constant-strength reweighting.


We appreciate the reviewer's thoughtful feedback and valuable suggestions. We hope our responses have addressed your questions and strengthened your confidence in our work. We would be grateful for your support during the discussion phase.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Updates

We sincerely thank all reviewers for their constructive feedback and discussions. We appreciate the reviewers' acknowledgment of our additional experiments and clarifications, which have now been incorporated into the revised manuscript. Below, we summarize the key updates and their corresponding locations.

1. Held-out evaluation (Reviewers 4A34, 39e1). We extend the original 300-example development-set ablations to 5,002 held-out AVHBench examples, with all configurations fixed before evaluation. The improvements from faithful-head selection and adaptive reweighting persist on held-out data.

Section 4.3, Tables 3–4.

2. Multi-seed random-head ablations (Reviewers sASb, 39e1, 4A34). We evaluate random subsets of different sizes across three seeds. HARPS achieves 77.6% accuracy on full AVHBench, compared with 74.8 ± 0.4% for size-matched random selection.

Section 4.3, Figure 4(a).

3. Adaptive component ablations (Reviewers sASb, 4A34). We separately assess example-specific sink profiles and prompt-dependent routing. HARPS achieves 77.6%, compared with 76.8% using fixed profiles, 76.1% using uniform routing, and 75.8% using random routing.

Section 4.3, Table 4.

4. Attribution-threshold sensitivity (Reviewers sASb, 39e1). Across thresholds from −0.001 to 0.001, accuracy remains between 76.8% and 77.6%, consistently exceeding the 75.2% greedy baseline.

Section 4.3, Figure 4(b).

5. Proxy-data resampling (Reviewer 39e1). Two independent 50% resamples recover 81.9% and 84.4% of the original faithful heads, achieving 76.8% and 75.6% accuracy, respectively, versus 77.6% with the full proxy data.

Section 4.3, Table 5.

6. Reweighting-strength stability (Reviewer 39e1). The original and two resampled development subsets all select the same global reweighting factor, γ_base = 8.

Section 4.3, Table 6.

7. Non-sink semantic alignment control (Reviewer 4A34). The faithful–unfaithful semantic alignment gap is substantially larger at sink than non-sink positions (0.44 vs. 0.07 for audio; 0.31 vs. 0.05 for vision). We also clarify the limits of the OV-circuit probe in establishing causality.

Section 4.3, “Semantic alignment at perceptual sinks.”

8. Regression and failure analysis (Reviewer 6HDi). Across six benchmarks and four models (15,881 examples per model), we quantify cases where HARPS changes a correct prediction into an incorrect one. Overall regression rates range from 0.94% to 2.98%, supplemented by qualitative inspection of failure cases.

Appendix I, Table 12.

We have also clarified the evaluation protocol and the distinction between one-time model-specific preparation and downstream task-specific tuning.

We thank the reviewers again for their valuable suggestions. We believe these additions strengthen the empirical evidence, clarify our methodological contributions, and provide a more comprehensive assessment of HARPS. We hope the revised manuscript will be helpful to the reviewers and Area Chair in their final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.