- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This study examines eight general-purpose models capable of processing audio and video inputs, distinguishing between “changes in responses when video is added” and “the use of lip shape and articulation information from the video for recognition.” The study conducted experiments on silent word recognition, recognition under noise, frame repetition and sequence permutation, and video substitution while keeping the audio waveform fixed. For the six models with publicly available weights, silent word recognition performance was nearly at the chance level, and the average effect of responses being pulled toward the word in the video was small. In contrast, the dedicated Auto-AVSR model demonstrated a visually derived effect from the same stimulus, indicating that multiple control experiments are necessary for evaluating general-purpose models.
Critical strengths
- The experimental design effectively controls for major confounding factors. The identity of the decoded audio waveforms across different video substitution conditions is verified using hash values, ensuring that videos of the same speaker are used. This makes it difficult to mistake changes in the audio or differences between speakers for the effects of visual speech cues.
- The interpretation takes into account even unfavorable results. The study reports that the advantages observed in the initial experiments with Gemini Flash were not replicated in follow-up experiments, and that the final segment consisting solely of noise affects the accuracy rate for audio-only data; however, it does not pinpoint a single cause for these findings.
Critical weaknesses
- The scope of the study and the input conditions are limited. The main experiment involved four English words and seven speakers, and generalizability to longer utterances or a broader vocabulary has not been confirmed. In particular, Gemini operates at a nominal 1 fps, making it impossible to verify which frames were actually selected. Consequently, it is not possible to sufficiently distinguish between the system’s ability to utilize visual information and the lack of input sampling required to observe articulation.
- Significant unresolved issues remain regarding the results of the closed model. The initial experiments and replication studies on Flash differed in terms of noise, presentation methods, and the dates on which the API was executed. The replication study was limited to 96 utterances from three speakers, and the Pro version did not include any video substitution experiments or replication studies. For this reason, it is difficult to draw definitive conclusions regarding the use of video in the closed model.
Questions for authors
- Would the effect remain minimal when evaluating longer utterances or a broader vocabulary under conditions where the number, timing, and resolution of the actual input phonetic frames can be verified, even when these correspond to the speech in the video?
- Regarding the discrepancy between the initial Flash experiment and the replication study, can the cause be isolated by re-evaluating the same utterances, the same noise, and the same presentation conditions within the same API execution period?
- If a general-purpose model is also made to perform free transcription, will it reach the same conclusion as the forced-choice method? Is it possible to further separate the dependence on options and prompts from the underutilization of speech information?
- On page 9, there are unresolved references (“the temporal architectures of ? or ?.”), which requires correction.