acceptodds
Desk rejected at ICLR 2027

From Attention to Semantics: Detecting and Mitigating Sound Event Hallucinations in Large Audio-Language Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Audio-Language Models (LALMs) excel at diverse audio tasks, yet remain prone to hallucinating sound events absent from the input. Prior work links such hallucinations to insufficient audio attention. However, we find that real and hallucinated events can receive comparable audio attention mass while attending to different audio positions. To examine whether the attended representations support the generated events, we adapt LatentLens to retrieve neighboring contextualized text representations. The retrieved contexts reveal semantic support for real events, but a semantic mismatch between hallucinated events and their attended audio representations. Directly masking the corresponding audio regions can suppress hallucinated events at the cost of real content. We are motivated by these findings to introduce an explainable and training-free detect-then-mitigate framework for effectively hallucination detection and mitigation that aggregates semantic support and applies differential audio masking to protect regions associated with events predicted to be supported. Extensive experiments on three LALMs and two datasets demonstrate that our method significantly improves hallucination detection and hallucination mitigation performance.

Primary area:
interpretability and explainable AI
Keywords:
large audio-language models, mechanistic interpretability, hallucination detection, hallucination mitigation

Related papers

Discussion (0)

Sign in to comment.