HARPS: Mitigating Multimodal Hallucinations via Head-Selective and Adaptive Perceptual Sink Reweighting

Multimodal large language models increasingly reason over multiple perceptual modalities, yet hallucination remains a persistent challenge. Recent work has shown that attention sink tokens can concentrate high-level perceptual and cross-modal information, motivating their use as inference-time intervention targets. We find, however, that these sink tokens behave differently across attention heads: unfaithful heads allocate more perceptual attention to sink positions, while their sink contributions are less semantically aligned with the input than those of faithful heads. Motivated by this finding, we propose Head-Selective Adaptive Reweighting for Perceptual Sinks (HARPS), a training-free method that selectively reweights attention to perceptual sink tokens in faithful heads and adapts the layer-wise intervention to each input-prompt pair. Across four multimodal LLMs spanning different families, architectures, and scales, HARPS consistently improves both hallucination-focused and general multimodal reasoning benchmarks, while remaining substantially more efficient than other baseline training-free methods.