acceptodds
Under review as a conference paper at ICLR 2027

Auditing Audio–Visual Integration in Omni-Modal Models

Abstract

Video can change speech recognition without showing whether a model uses the visible articulation. We evaluate eight omni-modal models using silent word recognition, speech in noise, repeated and permuted frames, and cross-dubs that hold the auditory waveform fixed across visual donors. The six open-weight models recognise silent command words at near-chance accuracy and show little average movement toward the donor's word, although their output distributions vary across donors. The two closed-weight models benefit from native video in selected noisy conditions. A follow-up tests one of these models with matched waveforms across native, reprocessed and dubbed clips and finds no clear benefit or preference for compatible video. A masker-only tail changes its audio-only accuracy by about eleven percentage points. A specialist audiovisual recogniser shows donor-directed effects on the same files; its benefit survives reprocessing and depends on the donor's word. We report word and onset outcomes separately, including transcripts that the onset rule cannot classify. Exploratory probes recover label information after linear nuisance adjustment, but the tested interventions do not establish how that information affects recognition. These results distinguish changes caused by adding video from evidence that a model uses visual speech, while leaving their architectural and training causes unresolved.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio-visual, omni-modals

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_A7usOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This study examines eight general-purpose models capable of processing audio and video inputs, distinguishing between “changes in responses when video is added” and “the use of lip shape and articulation information from the video for recognition.” The study conducted experiments on silent word recognition, recognition under noise, frame repetition and sequence permutation, and video substitution while keeping the audio waveform fixed. For the six models with publicly available weights, silent word recognition performance was nearly at the chance level, and the average effect of responses being pulled toward the word in the video was small. In contrast, the dedicated Auto-AVSR model demonstrated a visually derived effect from the same stimulus, indicating that multiple control experiments are necessary for evaluating general-purpose models.

Critical strengths
  • The experimental design effectively controls for major confounding factors. The identity of the decoded audio waveforms across different video substitution conditions is verified using hash values, ensuring that videos of the same speaker are used. This makes it difficult to mistake changes in the audio or differences between speakers for the effects of visual speech cues.
  • The interpretation takes into account even unfavorable results. The study reports that the advantages observed in the initial experiments with Gemini Flash were not replicated in follow-up experiments, and that the final segment consisting solely of noise affects the accuracy rate for audio-only data; however, it does not pinpoint a single cause for these findings.
Critical weaknesses
  • The scope of the study and the input conditions are limited. The main experiment involved four English words and seven speakers, and generalizability to longer utterances or a broader vocabulary has not been confirmed. In particular, Gemini operates at a nominal 1 fps, making it impossible to verify which frames were actually selected. Consequently, it is not possible to sufficiently distinguish between the system’s ability to utilize visual information and the lack of input sampling required to observe articulation.
  • Significant unresolved issues remain regarding the results of the closed model. The initial experiments and replication studies on Flash differed in terms of noise, presentation methods, and the dates on which the API was executed. The replication study was limited to 96 utterances from three speakers, and the Pro version did not include any video substitution experiments or replication studies. For this reason, it is difficult to draw definitive conclusions regarding the use of video in the closed model.
Questions for authors
  • Would the effect remain minimal when evaluating longer utterances or a broader vocabulary under conditions where the number, timing, and resolution of the actual input phonetic frames can be verified, even when these correspond to the speech in the video?
  • Regarding the discrepancy between the initial Flash experiment and the replication study, can the cause be isolated by re-evaluating the same utterances, the same noise, and the same presentation conditions within the same API execution period?
  • If a general-purpose model is also made to perform free transcription, will it reach the same conclusion as the forced-choice method? Is it possible to further separate the dependence on options and prompts from the underutilization of speech information?
  • On page 9, there are unresolved references (“the temporal architectures of ? or ?.”), which requires correction.
Reviewer_wBY4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The question of the paper was whether omni-modal models actually use visual speech, or just change their answers when video shows up. Several models were tested on silent word recognition, speech in noise, repeated and permuted frames, and cross-dubs that hold the audio fixed while swapping the visible word.

Critical strengths
  • I think the cross-dubbing experiment is one of the strongest parts of the paper, this because keeping the audio the same and changing only the visible word, it can be tested whether the model actually uses visual speech information. This is much more convincing than simply comparing audio-only and audiovisual performance.
  • I also like that the authors include Auto-AVSR as a reference, since the lack of lipreading effects could simply mean that the videos don't contain enough useful visual speech information.
Critical weaknesses

Rephrasing my concerns here:

  • I was a bit confused by some of the model results, because several of them basically predict the same word all the time. One model always says place, another always says set, and a third does almost the same whenever GRID frames are present. The authors do mention this, but with only four possible answers, around 25% accuracy is what you would expect anyway if you always pick the same option. So I'm not sure what these numbers really tell us about visual speech recognition. It could be that the models dont understand the task, or that someting in the input format is affecting them. This also makes the comparison across the eight models harder, since they may not be failing for the same reason.

  • The results in one of the experiments suggest that adding video helps, but in the bridge experiment this improvement is no longer really there. I understand the experiments are different and the confidence intervals overlap, so this is not necessarily a contradiction, and the authors explain that only 45 of the 96 tokens overlap. Still, I was left wondering where the improvement actually comes from: is it the visual speech information, or something about how the experiment was set up? A more direct comparison would help here, becuase at the moment it is hard to tell.

  • I also noticed that some results change quite a lot with relatively small changes in the setup. Adding a masker-only tail shifts the audio-only accuracy by about 0.11, and switching syllable to word in the prompt changes the audiovisual effect from -0.018 to -0.150 in the quiet condition. That is a large diference for something that is not really about the visual speech itself. The authors discuss this, but it made me wonder about the other experiments too. Would the cross-dubbing results look the same with a diferent prompt? Possibly, but I think it would be worth checking. Otherwise it is hard to know whether the models are actually using the video or just reacting to how the input is given.

Questions for authors
  1. Since some models seem to predict almost the same word every time, could the authors report how much their answers actually vary, for example using output entropy or the proportion of different predictions? If possible, it would also be useful to see the accuracy for cases where the models don't simply repeat the same answer. This would help clarify whether the near-chance results reflect poor visual speech recognition or difficulties with the task itself.
  2. It is explained that the 96-token bridge experiment is not a direct replication of Flash's earlier improvement. Would it be possible to compare the results using exactly the same clips, noise conditions and input durations?
  3. Given how much the results change when using syllable instead of word (appen A.4), I'm wondering whether the main cross-dubbing findings are also sensitive to this choice. Do the conclusions about which visual word influences the prediction remain the same with the alternative prompt?
  4. I noticed that Table 3 reports improvements of +0.099 and +0.111 for Pro at -6 dB and 0 dB, while Sec.4.2 reports +0.120 and +0.063 for 300 paired clips. Are these results based on different subsets, with Table 3 using talker 7 and Sec.4.2 using the larger set?
  5. I also noticed that the experimental coverage differs across models; e.g., Pro is only evaluated with visual controls for talker 7 and is not included in cross-dubbing, while Gemma 3n does not have matched GRID controls. Could the authors explain the reason for the differences there?
Reviewer_BzvNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper examines the impact of visual modality in several AV models on a speech recognition task (AVSR). Authors perform several tests, including: speech recognition in noisy conditions, silent speech recognition, etc.

Critical strengths

I'm not sure I can find any real strengths. The authors tested several models and performed a number of tests that probably took time and required resources - but I can't find a good reason on why should these benchmarks be useful to broader research community ... The tested models will definitely be outdated in 6-12 months. For the paper to be accepted at ICLR, it needs to have contributions worth reading in at lest a couple of years from now.

Critical weaknesses

GENERAL: English needs improvements. Paper is far from being readable - it is not easy to follow or understand the real contributions from the noise.

ABSTRACT: Abstract is not clear or informative at all - what’s this paper dealing with? What are the main takeaways? Why is this paper worth my time reading?! I don’t need a random “summary” with no context whatsoever. If I was not the assigned reviewer I would have never continued reading this paper after reading the abstract.

INTRODUCTION: Introduce the problem and the goal of the paper at the beginning! What are we dealing with?! I don’t need to be solving a puzzle and “wait” till halfway through the Introduction to just figure out what this paper is all about.

You start with “Does a model use visible speech, or does adding video simply change its answer?” - and I do not have a clue of what THE MODEL is supposed to do? The next sentence “Audiovisual accuracy alone cannot resolve this question.” still does not solve the question of WHAT audiovisual task we are speaking about - accuracy of WHAT is this sentence referring to?! … I’m not going to paste the following sentences here but this pattern of speaking about some not well-defined audiovisual problem continues for a while (it can be inferred it is a speech related problem but not if it deals with speech recognition, speech synthesis, or something else). Reading halfway through the Introduction section made me feel like the character in Kafka’s “The Trial” - Yes, there is the MODEL / TRIAL happening and dealing with the PROBLEM / CRIME but I really can’t figure out what exactly is the PROBLEM / CRIME we are speaking about.

Please start with something like “This paper deals with audio-visual speech recognition - i.e. a task of …” or “Audio-visual speech recognition is important because of this this this and this, but here is the problem …”.

CONTRBUTION section is also a word salad: I can’t understand why the contributions matter, why are they better or what do they add with respect to what’s already there? The whole premise sounds like a project for a graduate course (“we did this benchmarks to get familiar with the models and find a best one for our purpose”), without clear thesis on why should this be important for the broader research community.

RELATED WORK this reads like an AI overview with little context. Like with a lot of other parts of this paper, it requires rewriting to make it reader-friendly and give context why does it matter with respect to the problem and contributions of the paper - right now it is just a random collection of (not well balanced) summaries.

RESULTS: really not sure why does this matter at all - the whole results section feels like random tables displayed to show numbers and occupy space. Please be clear on WHY WHOULD ANYBODY CARE ABOUT THESE NUMBERS in >1-2 years from now?

Questions for authors

Authors should definitely spend time explaining why this work is a contribution with broader research impact that can inform future work, and not just an effort on ranking a couple of current (but soon to be outdated) models for purpose of a current (graduate course) project. The authors need to formulate a clear thesis on why is the paper important for the broader research community.

The whole paper requires a lot of rewriting in order to make it clear and easy to follow. Right now it is a mix of AI slop and poorly written details with little effort to emphasize what really matters. Throw half of technical word salads to appendix an use space to explain: why does it matter; why is it better than what is already there; why nobody did it before; and why will I need this in the future?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.