acceptodds
Under review as a conference paper at ICLR 2027

AVID: Benchmarking Audio-Visual Inconsistency Understanding for Omni-Modal Language Models

Abstract

Omni-modal large language models demonstrate strong content understanding in tasks such as captioning and question answering, yet their ability to understand fine-grained audio-visual inconsistencies remains largely unevaluated. To bridge this gap, we present AVID, a benchmark for audio-visual inconsistency understanding comprising 11.2K long-form videos with 39.4K annotated inconsistency events and 78.7K derived segment clips, with source-video-disjoint training and test splits. It supports detection, temporal grounding, classification, and reasoning across eight fine-grained inconsistency categories, including temporal shifts, semantic contradictions, and environmental conflicts. Compared with prior benchmarks focused on aligned events, forgery detection, or controlled multiple-choice diagnostics, it evaluates the localization and explanation of multiple inconsistencies in long videos, spanning low-level physical mismatches and high-level semantic and contextual conflicts. The benchmark is constructed through a scalable pipeline that combines content-type-aware temporal segmentation with an agentic framework for category selection and diverse audio-visual conflict injection. Modality ablations and processed-consistent controls show that strong performance requires cross-modal comparison and cannot be explained by manipulation artifacts. Extensive experiments and analyses of open- and closed-source models reveal persistent limitations in fine-grained inconsistency understanding. Using the training split, we develop AVID-Qwen, which improves over its base model across detection, temporal grounding, classification, and reasoning. Adding examples from the training split to general training data improves inconsistency detection while retaining general omni-modal capabilities.

Primary area:
datasets and benchmarks
Keywords:
Audio-Visual Inconsistency, Multimodal Benchmark, Omni-Modal Language Models, Temporal Grounding, Cross-Modal Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jrB8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AVID benchmarks omni-modal language models on detecting, classifying, temporally localizing, and explaining audio-visual inconsistencies. Built from LongVALE through audio interventions and human filtering, it covers eight inconsistency categories across 11.2K videos. The authors evaluate segment and full-video settings and provide a fine-tuned baseline, AVID-Qwen.

Critical strengths

The paper addresses a useful gap in omni-model evaluation by combining inconsistency detection, classification, localization, and explanation, particularly for multiple events in long videos. The construction pipeline is systematic, with human filtering, annotation-agreement studies, and source-video-disjoint splits. Modality ablations and consistent-TTS controls provide meaningful evidence that the task requires audiovisual information. The broad model evaluation, training ablations, general-mixture experiment, and DAVE transfer further demonstrate the dataset's challenge and training utility.

Critical weaknesses
  1. AVID's added capability over the closest existing audio-visual conflict benchmarks needs clearer positioning. The relationship between AVID and AV-ConfuseBench is not discussed. AVID's eight-category taxonomy and multi-event setting are valuable additions. A direct comparison of conflict types, reasoning requirements, and representative model failures would clarify which previously untested abilities AVID evaluates and how it extends existing benchmarks.

  2. The long-video evaluation needs a clearer reference point for its central contribution of multi-event localization and explanation. Section 3.5 describes the short clips as containing self-contained evidence, while the full-video task requires finding multiple such events. Tables 2 and 5 establish substantial difficulty and the benefit of full-video supervision, but do not compare full-video inference with a simple same-backbone window-based detection and interval-merging baseline. This comparison would help distinguish limitations in local inconsistency recognition from temporal search and handling long inputs, and better contextualize the full-video results. It would also provide a useful practical reference for future benchmark users.

  3. As a supplementary comparison, we suggest including representative current closed-source omni models, such as Gemini 3.8 Flash, Qwen3.5-Omni, and Qwen3.8-Omni-Flash.

Questions for authors

Please refer to the Critical Weaknesses above.

Reviewer_Nt8cOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether omni-modal language models can identify, localize, and explain conflicts between audio and visual information. To facilitate systematic evaluation, the authors introduce AVID, a benchmark constructed from existing videos through an agent-guided pipeline that selects and injects different types of audio-visual inconsistencies. The dataset contains 11,200 full videos, 39,361 annotated inconsistency events, and 78,722 short clips, covering eight inconsistency categories. It provides evaluation tasks at both the segment and full-video levels, including inconsistency detection, classification, temporal localization, and reasoning.

The authors benchmark several open-source and proprietary omni-modal models to examine their performance across these tasks. They also develop AVID-Qwen by applying a two-stage fine-tuning strategy to Qwen3-Omni. Experimental results indicate that existing models face challenges in temporal localization and reasoning, while AVID-Qwen achieves notable improvements over its base model, particularly in temporal grounding and text-based reasoning metrics.

Critical strengths
  1. A Meaningful Benchmark for an Underexplored Audio-Visual Understanding Task

The paper addresses an important but relatively underexplored aspect of omni-modal understanding: identifying and explaining inconsistencies between audio and visual information. Unlike conventional audio-visual benchmarks that primarily focus on aligned content, AVID provides a systematic evaluation of cross-modal conflicts across eight inconsistency categories. Its combination of segment-level and full-video tasks enables a more comprehensive assessment of detection, classification, temporal grounding, and reasoning. This is a valuable contribution, as it broadens the evaluation scope of existing omni-modal benchmarks.

  1. Scalable Benchmark Construction and Comprehensive Experimental Validation

The proposed agent-driven construction pipeline combines segment-aware strategy planning, specialized inconsistency injection, and human-in-the-loop quality control, enabling the creation of a large-scale dataset with detailed temporal annotations and explanations. The authors also evaluate multiple open-source and closed-source models and introduce AVID-Qwen as a fine-tuned baseline. These experiments reveal limitations in existing models and demonstrate measurable improvements through task-specific training. Together, the dataset construction framework and experimental analysis provide a useful foundation for future research on audio-visual inconsistency understanding.

Critical weaknesses
  1. Potential Unimodal Shortcuts Introduced by Synthetic Audio

The paper constructs audio-visual inconsistencies through TTS-based speech replacement, pitch modification, and background audio manipulation. These operations may introduce acoustic artifacts that models could exploit to detect inconsistencies without genuinely understanding the relationships between audio and visual content. Although the authors emphasize that AVID aims to evaluate cross-modal reasoning rather than synthesis artifact detection, the absence of audio-only and video-only control experiments makes it difficult to rule out such shortcuts. It would be helpful if the authors could include unimodal ablation studies and acoustic artifact analyses to further validate whether the benchmark genuinely assesses audio-visual inconsistency understanding.

  1. Limited Evaluation of Generalization Beyond Synthetic Inconsistencies

AVID primarily constructs inconsistent samples using five predefined audio injection methods. While this approach enables scalable and controlled data generation, it may not fully capture the complexity and diversity of audio-visual inconsistencies encountered in real-world scenarios. Furthermore, AVID-Qwen is trained and evaluated on data generated through the same construction pipeline, leaving its generalization to naturally occurring or unseen types of audio-visual conflicts insufficiently explored. The authors are encouraged to consider evaluations on real-world datasets or cross-dataset experiments to further assess the representativeness of the benchmark and the generalization capabilities of the model.

  1. Limited Evaluation of Evidence-Grounded Reasoning Capabilities

The paper primarily evaluates reasoning quality using BLEU-4, ROUGE-L, and METEOR, which focus on textual similarity between generated explanations and reference answers rather than directly assessing factual correctness or grounding in audio-visual evidence. Moreover, AVID-Qwen is fine-tuned using relatively structured response templates, suggesting that the substantial improvements in reasoning metrics may partly reflect alignment with reference answer formats rather than genuine improvements in cross-modal reasoning. Although the paper includes human validation of annotation quality, it lacks independent human evaluation or LLM-as-a-Judge assessment of model-generated explanations. It would be beneficial to include evidence-grounded evaluations of explanation correctness to better substantiate the reported improvements in Fine-Grained Reasoning and Dense Reasoning.

  1. Potential Selection Bias in the Evaluation Protocol

The paper evaluates classification, temporal grounding, and reasoning only on correctly detected inconsistencies (True Positives). Consequently, different models may be evaluated on subsets with varying sample sizes and difficulty distributions, which could affect the comparability of their results and may not fully reflect end-to-end performance. For example, Video-SALMONN 2-7B rarely predicts inconsistencies but still achieves 50.0% conditional classification accuracy. This raises the question of whether the current evaluation metrics adequately capture the models' overall classification capabilities. It would be helpful if the authors could clarify the evaluation denominators, report the number of True Positives for each model, and consider including end-to-end evaluations covering all ground-truth inconsistent samples to further strengthen the reliability of the experimental conclusions.

Questions for authors

1.Potential Unimodal Shortcuts How do the authors ensure that models are identifying genuine audio-visual inconsistencies rather than acoustic artifacts introduced by the synthetic injection process? Have audio-only or video-only evaluations been conducted to determine whether the inconsistencies can be detected without cross-modal reasoning?

  1. Generalization Beyond Synthetic Inconsistencies Given that AVID-Qwen is trained and evaluated on data generated through the same construction pipeline, how well does the model generalize to naturally occurring inconsistencies or unseen injection methods? Is there any additional cross-dataset or out-of-distribution evaluation that could help establish whether the improvements extend beyond the predefined synthetic patterns? the predefined synthetic patterns?

  2. Evaluation of Reasoning Quality Since reasoning performance is primarily measured using text similarity metrics, how do the authors distinguish improvements in factual, evidence-grounded reasoning from improvements in response formatting or similarity to reference explanations? Have the model-generated explanations been independently assessed by human evaluators or multimodal LLM judges?

Reviewer_r7c1OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AVID, a benchmark for audio-visual inconsistency understanding in omni-modal LLMs. It starts from LongVALE source videos. A segmentation step based on Silero VAD and Haar-cascade face detection labels each segment as Active Speaker, Voiceover or Scenic. A Gemini 3.1 Pro planning agent then chooses segments and one of 8 inconsistency categories, and five audio-only injectors create the conflict: time offset, TTS replacement with voice cloning, pitch/formant shift, a volume envelope, and background replacement via Demucs. Video frames are never modified. The result is 11.2K full videos containing 39.4K inconsistency events, with three inconsistent videos for every consistent one, and 78.7K segments split evenly between the two classes. Training and test splits share no source videos. The tasks are detection, 8-way classification, temporal grounding and free-form reasoning.

Nine omni-models are evaluated. The best zero-shot results both come from Gemini 3.1 Pro: 69.7% segment-level detection accuracy and 26.2% full-video mIoU. AVID-Qwen, a two-stage LoRA fine-tune of Qwen3-Omni, raises grounding to 36.1% mIoU and also improves the n-gram reasoning scores. To support validity, the authors show that models detect inconsistencies better with audio-visual input (61–70%) than with audio alone (53–56%), and that passing 100 consistent clips through the TTS pipeline raises the false-positive rate by only 0.5 and 2.5 points for Gemini 3.1 Pro and AVID-Qwen. To show training utility, they add 20K AVID segments to a general training mixture, which lifts AVID detection accuracy from 51.5% to 57.0% without hurting general benchmarks, and they report that AVID-Qwen's overall score on DAVE rises from 37.3 to 47.9.

Critical strengths

S1. A large, well-structured benchmark that fills a real gap. Existing audio-visual grounding benchmarks assume the modalities are consistent. Deepfake datasets target authenticity, and DAVE uses controlled multiple-choice questions. AVID instead asks models to detect, localize, categorize and explain multiple inconsistencies in long videos (avg. 235 s). It uses a taxonomy conditioned on segment class, train/test splits disjoint at the source-video level, and exhaustive human QC of the inconsistent test videos (1,250 → 1,171; App. A.4). All manipulation is restricted to audio, which removes visual-editing artifacts as a shortcut. This is the main reason the paper could merit acceptance: if its validity holds up, it is a reusable resource that the field currently lacks.

S2. The training split has demonstrated utility beyond its own test set. Adding 20K AVID segments to a 390K general mixture raises AVID detection from 51.5% to 57.0% while keeping DailyOmni and WorldSense at or above the base model (Table 4). AVID-Qwen also improves on the external DAVE benchmark without DAVE-specific tuning (37.3 → 47.9 overall; Table 5). This matters because it suggests the supervision teaches something transferable, not only AVID's output format.

Critical weaknesses

W1. The benchmark's validity as a measure of perceivable cross-modal inconsistency is not established.

  • There is no human performance baseline. The paper's principle is that "a human viewer should be able to determine consistency and identify the category" (§3.4). Yet no blind human detection, classification or grounding accuracy is reported on the test set. Agreement κ is reported on 300 videos, but that is not accuracy. The App. E "human evaluation" rates reasoning text against reference text without the annotators seeing the video. Without a human ceiling, one cannot tell whether a 69.7% detection score or 13.9% low-level classification accuracy (Table 17) reflects model weakness, imperceptible injections or label noise.
  • The negatives are unverified, and the clean-clip false-positive rate is high. Consistent items are "consistent by construction" and are not human-checked (App. A.4). Yet the strongest models flag 36.0% (Gemini 3.1 Pro) and 31.5% (AVID-Qwen) of original, unprocessed clips as inconsistent (Table 3). Real YouTube videos often carry background music or narration that would satisfy the paper's own definitions of BACKGROUND CONFLICT or EMOTION MISMATCH. A substantial share of these "false positives" may therefore be label errors in the negatives, and that directly affects detection accuracy and FPR.
  • The artifact controls are narrow. The processed-consistent control covers 100 clips and only the TTS path. The Demucs separation-and-remixing pipeline (Semantic and Background injectors), the FFmpeg pitch/volume filters and the AAC re-encode are not controlled. The control is also run on a zero-shot model, which has no reason to exploit artifacts, and on AVID-Qwen with only 100 clips. Audio-only accuracy is also above chance (55.8% and 53.3% on a balanced set), which is expected for VOLUME FLUCTUATION, where the envelope goes from 1.0 to 0.01, and for pitch-shifted VOICE IDENTITY. The paper's conclusion that AVID "cannot be explained by manipulation artifacts" is therefore stronger than the evidence.

W2. Several headline metrics do not measure the capabilities claimed.

  • Full-video detection. Three quarters of the full-video test set is inconsistent (1,171 of 1,561 videos), so a model that always answers "Yes" scores 75%. Qwen3-Omni (75.3%) and AVID-Qwen (78.2%) are essentially at that baseline, yet §5.2 presents full-video detection as a strength. Balanced accuracy or F1 should be reported.
  • Reasoning. Explanations are scored with BLEU/ROUGE/METEOR against references whose provenance is not stated. They appear to be the planner's pre-injection descriptions (the "inconsistency_understanding" field, generated by Gemini 3.1 Pro), which describe the intended conflict rather than verified audio. AVID-Qwen is trained to emit a fixed template ("The video shows … However, the audio contains …, creating a strong mismatch"; Fig. 4), so the headline result that its segment-level and full-video BLEU-4 scores are 2.8 and 10.9 times those of the base model largely reflects format matching. The human study (App. E) shows only a weak upward trend with large spread: explanations rated 5 have a mean BLEU-4 of 4.41 (s.d. 5.95), against 1.97 (s.d. 1.64) for those rated 2. It also does not report human scores per model.
  • Learning the low-level categories. After fine-tuning on 68K segments, AVID-Qwen reaches only 13.9% classification on the low-level sync categories (lip sync, temporal shift), against 53.6% for Gemini 3.1 Pro (Table 17). Its gains are concentrated in the high-level categories, which can partly be solved from audio content alone. This weakens the claim that AVID supervision teaches "fine-grained" cross-modal inconsistency understanding.
Questions for authors

Clarifications that could change my assessment:

  1. What is human performance on a blind, random test sample that includes consistent items: detection accuracy, category accuracy and boundary accuracy at both segment and full-video level? Were the annotators in the 300-video κ study blind to the label? If so, what was their accuracy?
  2. On a random sample of consistent test items, what fraction do humans judge inconsistent under the 8 category definitions? How much of the 31–36% clean-clip FPR is label noise?
  3. Do consistent items go through the same audio processing as injected ones (Demucs separation and remixing, and re-encoding to 192 kbps AAC at 48 kHz)? Can you report a processed-consistent control for each injector path, evaluated with AVID-Qwen, plus audio-only accuracy per category?
  4. How exactly are the ground-truth reasoning references produced: the planner's descriptions, post-injection verification, or human writing? Can you report the App. E human scores per model?
  5. Can you report full-video detection as balanced accuracy or F1 alongside the majority-class baseline?
  6. On DAVE, the base model's 10.1% on sound absence looks below chance for a multiple-choice task. What is the chance level per sub-task, and how are answers distributed before and after AVID training? I want to rule out that the gain mainly reflects a prior shift toward "absent/No" answers.

Presentation: Several figures are hard to read because their text is too small at normal zoom, for example the axis labels in Figure 3(b), the prompt and answer templates in Figure 4, and the labels, axes and legends in all three panels of Figure 5. This matters for understanding the paper. Figure 4 shows the response template that AVID-Qwen is trained to produce, and Figure 5 is the main evidence for the analysis in §5.4, so readers currently cannot check either from the figures. Please enlarge the fonts, give dense panels more space or move them to the appendix at full width, and report the underlying values in a table wherever a plot cannot be made legible.

Suggestions: Add a specialized A/V synchronization detector (e.g., SyncNet-style) as a reference for the temporal-shift and lip-sync categories; add a small set of naturally occurring inconsistencies (dubbing or desync errors) to test ecological validity; position the work against AVHBench (ICLR 2025), which already evaluates audio-visual mismatch and hallucination in AV-LLMs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.