- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether large audio-language models can identify an incorrect factual claim when it is mixed with several correct claims about the same audio. The authors introduce FactMix-Bench, which contains paired queries that differ in exactly one factual claim, constructed from the existing MECAT-Caption dataset. They also construct FactMix-Align, a preference dataset used to fine-tune Qwen3-Omni with standard DPO. The reported pair accuracy improves from 54.33% to 76.83%, while performance on general audio understanding benchmarks is largely preserved.
Critical strengths
The paired evaluation design is a useful aspect of the paper. By changing only one factual claim while keeping the other claims fixed, the benchmark provides a controlled way to test whether models can distinguish supported from unsupported descriptions. The reported improvement after DPO is also good, although further analysis is needed to understand what drives it.
Critical weaknesses
1. The paper does not establish its central claim about mixed-fact robustness.
The main motivation is that correct surrounding facts can cause models to overlook an incorrect fact. However, the experiments never compare mixed-fact queries against queries containing only the same incorrect fact. Without this baseline, it is unclear whether the failures are caused by the surrounding correct facts or simply by the model's inability to verify the target fact. This is a critical issue because the claimed failure mode is the main motivation and contribution of the paper.
2. Limited novelty and insufficient comparison with prior audio hallucination research.
The overall framework closely resembles existing audio hallucination research, including counterfactual example construction, factual grounding evaluation, and hallucination mitigation through negative supervision or preference alignment. Several relevant studies are missing from the related work. For example, Kuan et al. (2024, 2025a) have studied object hallucination and fine-grained factual errors involving sound existence, attributes, and temporal order, while Kuan and Lee (2025b) proposed LISTEN, which mitigates audio hallucinations using synthesized negative samples. Audio Hallucination Attacks (Seth et al., 2026) also examines query-based hallucination failures and mitigation. Moreover, although AHA (Chen et al., 2026) is cited, the paper does not adequately distinguish its approach from existing counterfactual preference alignment methods.
The main distinction appears to be placing one incorrect fact among several correct facts, but the paper does not establish why this constitutes a substantially different challenge from existing hallucination settings. The proposed training also relies on standard DPO, without comparisons against simpler SFT or existing hallucination mitigation approaches. As a result, the novelty and added value of the proposed framework remain unclear.
References
- Kuan et al. (2024). Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models. Interspeech 2024.
- Kuan et al. (2025a). Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning. ICASSP 2025.
- Kuan and Lee (2025b). Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples. Interspeech 2025.
- Seth et al. (2026). Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models.
- Chen et al. (2026). AHA: Aligning Large Audio-Language Models for Reasoning Hallucinations via Counterfactual Hard Negatives. Findings of ACL 2026. (Already cited.)
Questions for authors
Q1. How does performance change when the same incorrect target fact is presented alone versus surrounded by several correct facts? This comparison seems essential to support the paper's central claim that the surrounding facts cause models to overlook the error.
Q2. What distinguishes FactMix from prior audio hallucination and counterfactual alignment work, particularly Kuan et al. (2024, 2025), AHA, and Audio Hallucination Attacks? Can the authors explain what existing benchmarks or methods fail to capture?
Q3. Would standard SFT or training on isolated factual verification examples achieve similar improvements? Without these baselines, how can the gains be attributed specifically to mixed-fact alignment?
I was urged by Program chairs to give another review so I am revising my review of the paper here
Summary:
This paper studies whether large audio-language models (LALMs) catch a single false claim when it sits among several true claims about the same audio. They introduce FactMix-Bench, built from the MECAT-Caption test split, with 27,976 paired questions over 5,000 recordings. Each pair has a Grounded Query and a Mixed-Fact Query that changes exactly one target fact to a plausible but unsupported alternative. Target facts fall into five categories (source, content, attribute, temporal, relation). The pipeline uses GPT-5.5 to structure the annotations into facts, Qwen3-14B to propose alternatives and build the query pairs, and Qwen3-Omni to check the alternatives against the audio. The authors also build FactMix-Align, about 124K DPO preference instances from the MECAT training split which uses the same single-fact substitution scheme. DPO on Qwen3-Omni-30B-A3B raises overall pair accuracy from 54.33 to 76.83, which beats every evaluated model.
Strengths:
Weaknesses:
The same model generates the benchmark and is evaluated on it. Qwen3-Omni is the audio verifier that decides which alternatives are kept (only those it calls Contradicted) and it is also the base model that is trained and evaluated.
There is no human validation of the benchmark. Facts are extracted by GPT-5.5 from MECAT captions that were themselves model-generated, alternatives come from Qwen3-14B and verification is done by Qwen3-Omni and Qwen3-14B.
Reporting is thin and only pair accuracy is given. There is no question-level accuracy, no Grounded vs Mixed breakdown, and no acceptance/rejection rates.
No variance or significance testing. The improvements on MMAU/MMAR/MMSU are probably within noise.
There is no control over error severity and only multiple-choice evaluation (no open-ended evaluation). Real queries can contain partially correct claims which is not accounted for.
Novelty is incremental. The idea is close to AHA (counterfactual hard negatives plus DPO) and to minimal-pair or counterfactual benchmarks in vision-language work.
The perturbation scheme closely follows ARO and SugarCrepe in vision-language and CompA in audio, none of which are discussed in the related works section.
Questions:
My rating for the paper is unchanged as of 11/14/2026.