From Attention Weights to Audio Writes: Mitigating Hallucinations in Large Audio Language Models

Large audio language models (LALMs) may describe sound events or attributes that are absent from the input recording, resulting in hallucinations. Existing attention interventions mitigate these errors by sustaining attention to audio, but attention weights alone do not determine the vector contributions of audio positions to the attention output. To address this limitation, we introduce the Audio Relative Write Ratio (ARWR), which measures the norm of the vector update from audio positions relative to the sum of update norms from audio and other positions. By incorporating value aggregation and output projection, ARWR provides a more accurate characterization of the contribution of audio positions to the attention output. Building on this diagnostic, we propose ARWR-Cal, a framework for hallucination mitigation through residual calibration of the final hidden state. At each generation step, a lightweight low-rank calibrator uses the source vectors and final hidden state to predict a residual adjustment. ARWR controls the permitted adjustment magnitude. Experiments show ARWR outperforms audio attention in hallucination diagnosis, while ARWR-Cal improves average accuracy by 8.2 percentage points over the strongest baseline across three datasets and three LALMs. Code is available at https://anonymous.4open.science/r/ARWR-Cal.