acceptodds
Under review as a conference paper at ICLR 2027

FactMix: Evaluating and Improving Mixed-Fact Robustness in Large Audio-Language Models

Abstract

Large Audio-Language Models (LALMs) achieve strong performance across speech, environmental sound, music, and audio reasoning tasks, yet they can overlook a single incorrect factual claim when it appears together with several correct claims in the same query. To study this failure, we introduce FactMix-Bench, a diagnostic benchmark built from paired Grounded Queries and Mixed-Fact Queries. Each pair shares the same audio and supporting facts, while the Mixed-Fact Query changes exactly one target fact to a plausible but unsupported alternative. FactMix-Bench covers source, content, attribute, temporal, and relation information, with 55,952 multiple-choice questions forming 27,976 pairs over 5,000 recordings. We further construct FactMix-Align, a preference dataset using the same controlled factual changes, with directionally balanced ordered perturbations to reduce training bias. Direct Preference Optimization with FactMix-Align improves Qwen3-Omni pair accuracy from 54.33 to 76.83, achieving the best overall performance among the evaluated models, while performance on established public audio benchmarks is largely preserved or modestly improved. These results show that strong general audio understanding does not necessarily imply reliable verification of multiple factual claims. The data and code associated with FactMix-Bench and FactMix-Align will be publicly released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Audio-Language Models, Mixed-Fact Robustness, Factual Verification, Audio Understanding, Preference Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CjtqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether large audio-language models can identify an incorrect factual claim when it is mixed with several correct claims about the same audio. The authors introduce FactMix-Bench, which contains paired queries that differ in exactly one factual claim, constructed from the existing MECAT-Caption dataset. They also construct FactMix-Align, a preference dataset used to fine-tune Qwen3-Omni with standard DPO. The reported pair accuracy improves from 54.33% to 76.83%, while performance on general audio understanding benchmarks is largely preserved.

Critical strengths

The paired evaluation design is a useful aspect of the paper. By changing only one factual claim while keeping the other claims fixed, the benchmark provides a controlled way to test whether models can distinguish supported from unsupported descriptions. The reported improvement after DPO is also good, although further analysis is needed to understand what drives it.

Critical weaknesses

1. The paper does not establish its central claim about mixed-fact robustness.

The main motivation is that correct surrounding facts can cause models to overlook an incorrect fact. However, the experiments never compare mixed-fact queries against queries containing only the same incorrect fact. Without this baseline, it is unclear whether the failures are caused by the surrounding correct facts or simply by the model's inability to verify the target fact. This is a critical issue because the claimed failure mode is the main motivation and contribution of the paper.

2. Limited novelty and insufficient comparison with prior audio hallucination research.

The overall framework closely resembles existing audio hallucination research, including counterfactual example construction, factual grounding evaluation, and hallucination mitigation through negative supervision or preference alignment. Several relevant studies are missing from the related work. For example, Kuan et al. (2024, 2025a) have studied object hallucination and fine-grained factual errors involving sound existence, attributes, and temporal order, while Kuan and Lee (2025b) proposed LISTEN, which mitigates audio hallucinations using synthesized negative samples. Audio Hallucination Attacks (Seth et al., 2026) also examines query-based hallucination failures and mitigation. Moreover, although AHA (Chen et al., 2026) is cited, the paper does not adequately distinguish its approach from existing counterfactual preference alignment methods.

The main distinction appears to be placing one incorrect fact among several correct facts, but the paper does not establish why this constitutes a substantially different challenge from existing hallucination settings. The proposed training also relies on standard DPO, without comparisons against simpler SFT or existing hallucination mitigation approaches. As a result, the novelty and added value of the proposed framework remain unclear.

References

  • Kuan et al. (2024). Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models. Interspeech 2024.
  • Kuan et al. (2025a). Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning. ICASSP 2025.
  • Kuan and Lee (2025b). Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples. Interspeech 2025.
  • Seth et al. (2026). Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models.
  • Chen et al. (2026). AHA: Aligning Large Audio-Language Models for Reasoning Hallucinations via Counterfactual Hard Negatives. Findings of ACL 2026. (Already cited.)
Questions for authors

Q1. How does performance change when the same incorrect target fact is presented alone versus surrounded by several correct facts? This comparison seems essential to support the paper's central claim that the surrounding facts cause models to overlook the error.

Q2. What distinguishes FactMix from prior audio hallucination and counterfactual alignment work, particularly Kuan et al. (2024, 2025), AHA, and Audio Hallucination Attacks? Can the authors explain what existing benchmarks or methods fail to capture?

Q3. Would standard SFT or training on isolated factual verification examples achieve similar improvements? Without these baselines, how can the gains be attributed specifically to mixed-fact alignment?

Reviewer_7FwGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper builds FactMix-Bench, a set of 27,976 Grounded/Mixed-Fact query pairs over 5,000 MECAT-Caption test recordings. In each pair exactly one target fact is swapped for an audio-contradicted alternative. It also builds FactMix-Align, a 123,860-instance DPO dataset from the MECAT training split with directional balancing for ordered properties. DPO on Qwen3-Omni raises pair accuracy from 54.33 to 76.83, and public audio benchmarks stay roughly unchanged.

Critical strengths

The paired construction and the PairAcc metric are well designed. Requiring both members of a pair to be correct removes always-accept and always-reject shortcuts. The paper is large in scale and covers five categories, with category-level reporting that reveals very low Temporal scores. Directional balancing of ordered properties in the training data is a sensible precaution against a plausible shortcut, though its effect is not ablated. The authors ran every evaluation themselves rather than copying leaderboard numbers.

Critical weaknesses
  1. The core causal claim has no control. The paper argues that surrounding correct claims cause the model to overlook the wrong one. However, there is no condition that asks about the target fact alone, and no condition that varies the number of supporting facts. Without that comparison, the failure could just be weak perception of the property. This is the single most important missing experiment.
  2. The benchmark is filtered by the model it evaluates. Qwen3-Omni is the verifier that keeps only CONTRADICTED alternatives, and Qwen3-Omni is also the base model being evaluated and aligned. The paper doesn't say which variant does the verifying, but if it is the same model, every retained item is one the Qwen3-Omni family can already reject in isolation. This selection bias complicates cross-model comparison and means the aligned gain may partly surface knowledge the filter already guaranteed. No human audit of verifier precision is reported.
  3. The gains may be in-distribution. Training and test share the source corpus (MECAT), the fact-structuring model (GPT-5.5), the query generator and pair checker (Qwen3-14B), and therefore the same generation pipeline. The 22.5-point gain could partly reflect familiarity with that pipeline's query style. The authors should evaluate on human-written mixed-fact queries, on queries from a different generator, or on related existing benchmarks such as AHa-Bench and HalluAudio. The model should be evaluated on those after DPO in any case, since they are the closest relevant tests.
  4. The claimed format transfer is weak. The MCQ options closely mirror the preferred DPO responses. This is a small surface change, not real evidence of cross-format generalization.
  5. Answer biases and chance levels are not reported. Questions mix two, three and four choices, and a model with a strong Yes or No bias can score well on one member of each pair. The authors should report each model's Yes/No bias, option-level confusion, and chance-level PairAcc for each choice cardinality. This matters most for Temporal, where several models score only 12 to 15% (12.57, 14.20 and 15.41).
  6. Baselines are missing. ◦ There is no audio-ablated (text-only) baseline to measure language priors. ◦ There is no comparison with other alignment methods trained on the same data, such as SFT on the preferred responses or AHA-style hard negatives. Audio-Aware Decoding, a training-free inference-time method, would be a useful additional baseline. ◦ Qwen3.5-Omni and Audio Flamingo Next are cited but not evaluated.
  7. No variance is reported. No standard deviations or confidence intervals appear anywhere, and the paper doesn't say how many runs underlie each number. On MMAU-TestMini (1,000 items), the 1.1-point change is about 11 questions and is within noise, so it shouldn't be called an improvement without a test. On larger sets such as MMAU-Test the changes may be real, and a significance test would settle it.
  8. Novelty over AHA is modest. AHA already uses counterfactual hard negatives with preference alignment, and the main difference here is that the error sits in the query. The setting is close to false-premise or presupposition QA. That line of work, in both text and vision-language models, isn't discussed.
  9. Relation has only 537 pairs, about 20 times fewer than Attribute (10,400), which gives a 95% CI of roughly ±4 points. Comparisons across categories therefore carry unequal precision.
Questions for authors

• What is PairAcc on single-fact versions of the same targets? • What is the human-verified precision of the CONTRADICTED filter on a random sample? • How does the aligned model do on AHa-Bench and HalluAudio? • Does the gain survive when queries come from a different LLM with different phrasing?

1 comment
Reviewer_7FwGOpenReview ↗

I was urged by Program chairs to give another review so I am revising my review of the paper here

Summary:

This paper studies whether large audio-language models (LALMs) catch a single false claim when it sits among several true claims about the same audio. They introduce FactMix-Bench, built from the MECAT-Caption test split, with 27,976 paired questions over 5,000 recordings. Each pair has a Grounded Query and a Mixed-Fact Query that changes exactly one target fact to a plausible but unsupported alternative. Target facts fall into five categories (source, content, attribute, temporal, relation). The pipeline uses GPT-5.5 to structure the annotations into facts, Qwen3-14B to propose alternatives and build the query pairs, and Qwen3-Omni to check the alternatives against the audio. The authors also build FactMix-Align, about 124K DPO preference instances from the MECAT training split which uses the same single-fact substitution scheme. DPO on Qwen3-Omni-30B-A3B raises overall pair accuracy from 54.33 to 76.83, which beats every evaluated model.

Strengths:

  1. The positioning against prior work is precise. The paper separates its setting from AHa-Bench, HalluAudio, AHA and VoxParadox by where the conflict sits.
  2. The metric is well-designed since pair accuracy cannot be gamed by always accepting or always rejecting.
  3. The benchmark has about 56K questions across five fact categories, and the evaluation covers seven models, both open (Qwen2.5/3-Omni, AF3, Step-Audio-2) and closed (GPT-4o Audio, Gemini-3.1-Pro).
  4. Directional balancing for ordered properties targets a plausible shortcut.

Weaknesses:

  1. The same model generates the benchmark and is evaluated on it. Qwen3-Omni is the audio verifier that decides which alternatives are kept (only those it calls Contradicted) and it is also the base model that is trained and evaluated.

  2. There is no human validation of the benchmark. Facts are extracted by GPT-5.5 from MECAT captions that were themselves model-generated, alternatives come from Qwen3-14B and verification is done by Qwen3-Omni and Qwen3-14B.

  3. Reporting is thin and only pair accuracy is given. There is no question-level accuracy, no Grounded vs Mixed breakdown, and no acceptance/rejection rates.

  4. No variance or significance testing. The improvements on MMAU/MMAR/MMSU are probably within noise.

  5. There is no control over error severity and only multiple-choice evaluation (no open-ended evaluation). Real queries can contain partially correct claims which is not accounted for.

  6. Novelty is incremental. The idea is close to AHA (counterfactual hard negatives plus DPO) and to minimal-pair or counterfactual benchmarks in vision-language work.

  7. The perturbation scheme closely follows ARO and SugarCrepe in vision-language and CompA in audio, none of which are discussed in the related works section.

Questions:

  1. What are the no-audio baseline scores?
  2. How do the results split between Grounded and Mixed questions, and between acceptance and rejection, especially for temporal?
  3. How does the aligned model do on AHa-Bench or HalluAudio?
  4. What happens if directional balancing is removed?

My rating for the paper is unchanged as of 11/14/2026.

Reviewer_aBeQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FactMix, a framework for evaluating and improving the factual robustness of Large Audio-Language Models (LALMs), addressing a critical failure mode in which models overlook a single incorrect factual claim embedded among multiple correct claims about an audio recording. The authors propose FactMix-Bench, a controlled benchmark comprising 55,952 multiple-choice questions organized into 27,976 paired grounded and mixed-fact queries across five factual dimensions: source, content, attribute, temporal, and relation. Each query pair differs by exactly one factual claim, enabling systematic evaluation of a model's ability to identify subtle factual inconsistencies. To mitigate this weakness, the authors further introduce FactMix-Align, a preference-alignment dataset containing 123,860 instances generated through controlled counterfactual perturbations and directional balancing to reduce shortcut learning. Using Direct Preference Optimization (DPO) to fine-tune Qwen3-Omni, the proposed approach improves overall paired accuracy from 54.33% to 76.83%, with particularly substantial gains in temporal reasoning (+41.41 percentage points), while largely preserving general audio-understanding performance.

Critical strengths

The paper's strongest contribution is its controlled experimental design. By constructing paired queries that differ in exactly one factual claim, FactMix-Bench provides a more targeted evaluation of compositional factual consistency than conventional audio-understanding benchmarks.

Another strength is its empirical improvements: Qwen3-Omni's paired accuracy increases from 54.33% to 76.83%, with a particularly large 41.41-percentage-point improvement on temporal queries. The authors additionally evaluate several competitive models and demonstrate that performance on general audio benchmarks is largely preserved.

Critical weaknesses

There are several critical weaknesses I hope authors can address during the rebuttal:

Potential benchmark reliability issues. The benchmark uses GPT-5.5 to structure facts, Qwen3-14B to generate counterfactuals, and Qwen3-Omni to verify whether alternatives contradict the audio. However, these stages rely heavily on automated model judgments, incorrect annotations or ambiguous counterfactuals could affect evaluation reliability. The paper does not provide sufficiently convincing human-validation evidence.

Insufficient ablation studies. The paper does not establish how much of the improvement comes from controlled counterfactual generation, directional balancing, or DPO itself. I have not found the comparisons with standard supervised fine-tuning, randomly generated negative examples, and alternative preference-alignment strategies.

Questions for authors

Can we add more human validation? For example you can manually verify a statistically meaningful subset of grounded and mixed-fact pairs, reporting annotation accuracy and inter-annotator agreement.

Did you compare DPO against SFT and evaluate the effects of directional balancing, counterfactual quality, and different negative-sampling strategies.

Did you demonstrate robustness improvements on independently constructed hallucination benchmarks and other models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.