- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ARWR, which measures the relative strength of audio writes to the attention output using vector norms after value aggregation and output projection, instead of attention weights alone. ARWR-Cal uses this ratio to cap a residual correction to the final hidden state, predicted by a small calibrator on top of a frozen backbone. The method is evaluated on three backbones and three datasets, and ARWR is also tested as a hallucination-risk signal on Clotho-AQA.
Critical strengths
S1. Gains across datasets and backbones. ARWR-Cal has the best accuracy and F1 on the Adversarial and Popular conditions of AudioHallucination for all three backbones (Table 1). On HalluAudio it wins all nine backbone–domain combinations, by 11.7–27.3 points over the strongest baseline in each. It is also best on Clotho-AQA for every backbone, though the margins there are small. For me, the HalluAudio numbers are the strongest part of the paper.
S2. The diagnostic claim is checked with numbers. ARWR accounts for value aggregation and output projection, so it can pick up contributions that attention weights miss, and the paper checks whether this actually matters. On Qwen2-Audio answers from Clotho-AQA, ARWR gets an AUROC of 0.7431 vs. 0.6295 for audio attention mass, and the 95% CI of the paired difference stays above zero (§5.4, Table 3). The two extra samples in Appendix C.2 agree.
Critical weaknesses
W1. The smaller gains under Random are not explained. I put a lot of weight on this condition. Popular and Adversarial target errors driven by object frequency and co-occurrence, while Random checks whether the model can judge presence or absence from the audio when negatives are not picked by either criterion. ARWR-Cal allows larger corrections when ARWR is low, so I want to know how its effect changes across these question types.
In Table 1, ARWR-Cal improves Random accuracy over Base by 8.1–14.1 points but is still 9.2–9.6 points behind AHA, the best baseline there. Its gain over Base is smallest under Random on all three backbones. Most other methods do best under Random. ARWR-Cal does worst there on every backbone. The authors mention the dependence on negative sampling (§5.2) and the trade-off between correction and preserving correct answers (§5.3). Per-condition accuracy and F1 are reported, and Appendix D has a few Clotho-AQA examples of fixed and broken answers. What I could not find is a paired before/after analysis for each AudioHallucination condition. Without it, I can't tell whether the Random gap comes from fixing fewer errors, breaking more correct answers, or gains concentrated in some question types. That makes it hard to judge how far the method's benefit extends.
W2. The ablations mix up ARWR-based budget allocation with correction size. In Eq. 6 the adaptive budget lies in , but the fixed-budget control (§5.3, Table 2) uses the lowest value, . Since the budget directly scales the correction in Eq. 10, this comparison changes both input-dependent adaptation and the size of the correction at once. The other two ablations don't resolve this. w/o Source Features keeps the ARWR budget, and w/o Adaptive Budget keeps the source features. What is missing is a calibrator that uses only final hidden-state features with a fixed budget, trained with the same supervision. Without it, it is hard to see what ARWR and the source features add over plain calibration.
This matters for Random too. On Qwen2-Audio, the fixed-budget variant gets 77.4% Random accuracy vs. 70.0% for the full model (Table 2). SALMONN-7B doesn't show this (65.0% for the full model vs. 64.1% for the fixed budget, Appendix Table 5), so I'm not saying adaptive budgets hurt Random in general. I just think the effect of the budget on correction and preservation should be checked across backbones.
I see real value in the cross-dataset gains and in ARWR as a diagnostic. But the Random pattern and the added value of the ARWR budget over simpler calibration aren't explained yet, so for now I lean toward weak reject.
Questions for authors
Q1. The smaller gains over Base under Random are the main reason I'm not more positive about this paper. What drives this pattern? I'd like the authors to look into it with targeted analyses, plus additional experiments if needed. Some places to start: how predictions change after calibration, how results vary by question and sound type, and whether the calibration-data composition or the budget settings matter. These are just suggestions. Please test the explanations you find most plausible and report what you find, including results that don't support them, any trade-offs, and what is still unclear. It would also help to keep what the evidence shows separate from what is still a hypothesis. If this is addressed well, I'd be quite willing to raise my score.
Q2. How are the three AudioHallucination conditions used in calibrator training, and do ARWR-Cal and AHA train on the same data? Since there is one calibrator per backbone–dataset pair, I'd like to know the number of calibration-pool samples per condition, whether any weighting is used, and how duplicate questions across conditions are handled. I'm also not sure whether AHA's "existing training partitions" contain the same audio and questions as ARWR-Cal's training data. Actual training sample counts for both methods would clear this up.
Q3. Does the ARWR-guided budget still help against a well-chosen fixed budget, e.g. one tuned on the validation set or matched to the adaptive budget's mean? A fixed-budget calibrator using only final hidden-state features, trained on the same data with the same objective, would also show what ARWR and the source features bring.
Q4. Do ARWR and the applied correction size behave differently under Random? The ARWR distribution and the actual correction magnitudes per condition, linked to corrected errors and to correct answers that flip to wrong, would help locate the issue. On a separate note, Appendix B.3 counts only false Yes answers as hallucinations. Among answers predicted as Yes, can ARWR separate correct from incorrect ones? That would show whether it tracks correctness or just answer polarity.