- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper is interested in examining whether omni-modal LLMs can understand user intent from video clips with speech, without text questions. The model needs to reply with assistance, clarification, answer, or silence. The authors created a benchmark of 1400 generated video clips, covering 5 categories and 14 subtasks. They also trained an internal omni model using GRPO on a spoken QA dataset. The authors tested a handful of models on their benchmark and observed varying performance across models and across tasks. They conclude that current omni-modal LLMs do not yet reliably understand user intents spoken in video.
Critical strengths
- The focus on spoken intention understanding in video appears to be novel and well-motivated. The task is real and is a jump above current video benchmarks in the practical side.
- The benchmark design is clean and isolates different failure modes. The evaluation is well-designed as well, e.g., intent and content are scored separately and as a result getting the request wrong is distinguished from getting the specific facts wrong.
- The new training datasets (omniintent-train-24k) are useful resources.
Critical weaknesses
- The benchmark heavily relies on synthetic video and speech. The quality of the benchmark could be questionable. The paper contains minimum information on how data quality is checked. In the main text at least, I just found one sentence describing that media quality is checked. As the benchmark is the major contribution, a more detailed description on how the video is vetted could be important.
- The evaluator is a GPT-4o that judges replies against a textual description of the scene, and it has never seen the video. For this to work again we need to assume that the generated video is of high quality and no gap exists between the script and the video. A subset evaluated by humans and a LLM-human agreement metric would be helpful here.
- The training rewards' design needs more consideration. For example, the reward scores "refusing to answer" as 0. Wouldn't this penalize models that choose to stay silent, which is a category being tested.
- The evaluation could have been more complete: for example, silence is measured as accuracy on true-silence examples, what about false silence rate? i.e., models choosing to stay silent even when response is required? Examples of how models fail would be helpful too.
Questions for authors
- Section 5.2 mentions that "Results are available for the two Qwen models, while the remaining open-source evaluations are incomplete." Moreover, section 5.3 mentions that Table 2 summarizes results on the "current" diagnostic evaluation subset. What does "current" here mean? In general, how should one read Table 2? Any numbers that should not be read as of now?
- How do the authors deal with ambiguous cases? Are all videos used in the experiment containing clear, single intent?
- Which models are used for generating the data, i.e., writing the scripts, generating the speech and video?
- Any more details could be given on the internal base model? Why not train an open source model?