- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces Omni Demand Understanding benchmark, which evaluates whether multimodal models are able to identify valid user requests and are able to understand the underlying intent from audio, visual, as well as conversational context. The authors construct ODU-Bench, which contains 2078 synthetic as well as human recorded interactions, and they evaluate models across 5 dimensions. They test 14 multimodal models and their ablations show weaknesses such as high false trigger rates and weaknesses in contextual reasoning.
Critical strengths
-
I feel the paper addresses an important aspect of multimodal interactions, which is currently a bit under explored. Instead of just evaluating whether the response of a model is correct or not, the paper focuses on evaluating whether a model actually understands when a user has made a request or not, and what the user actually intends.
-
The benchmark covers a wide set of interactions, including indirect requests, as well as requests not directed to the assistant, and they have tested 14 models, which is almost all the models available.
-
I liked the ablations done in the paper, including Figure 4, which shows that models perform much better in recovering information which has been explicitly stated in speech rather than information which requires visual, acoustic, or dialogue history.
Critical weaknesses
-
The paper does not specify which audio and video generation models were used to generate the videos in the benchmark and the synthetic interactions. Also since these synthetic interactions make up the majority of the benchmark (1801 out of 2078 interactions), these missing details about the implementation limit the reproducibility. This also makes it difficult to assess any potential generation specific artifacts.
-
The paper does not state the quantitative human verification results of the generated synthetic interactions. Although appendix C.4 describes that there was human verification of the generated interactions, it does not mention the inter-annotator agreement or the rejection rate, for example how many samples were rejected from the initial pool of generated samples, and it also does not tell us the portion of samples which required correction. Because the benchmark relies a lot on synthetic generated media, these statistics are important so that we can assess the reliability and the consistency of its annotations.
-
The paper's aggregate score gives 60% weightage to intent recovery, which is M2, and only 15% weightage to demand detection, which is M1. I feel this may under represent false trigger failures, which are important for real world assistants, for example Seed 2.0 Lite scores a 67.9% overall Audio-Visual score, but it has a 82.2% false trigger rate in Table 3. So the paper does not properly show whether model rankings remain same under changed or different metric weights. The authors should show how the model rankings change if we give different weightage to the 5 evaluation metrics. Also, the weightage of these 5 evaluation metrics seem to be heuristic instead of empirically justified.
-
The paper shows that there are 969 multi-turn conversations but it does not have any ablation which compares the performance with and without the previous conversation history. Without this comparison it is hard to quantify how much the models improve or degrade when we give them the conversational history and the context.
-
The paper does not provide any sort of distribution of the number of turns or the average number of turns per conversation. It would be helpful to see how does the model performance vary in accordance to the number of turns per conversation.
Questions for authors
I would like the authors to tell in detail which audio and video generation models they use to construct the synthetic interactions in the benchmark. And what protocol they followed for qualitative and quantitative assessment.
-
The author should also mention the statistics regarding their selection criteria of the videos generated by the video generation model . What was the inter-annotator agreement, how many humans rated it, what was the rating process, and what was the rejection rate, what were the qualifications of the humans who rated it.
-
The authors should show how would the model rankings in table 3 change if a greater weightage is assigned to demand detection and the false trigger behavior.
-
It would be helpful if the authors could evaluate the multi-turn scenarios with and without the previous conversation history so that we can measure its contribution in regards to demand understanding.
-
It would be helpful if the authors could report the distribution of the number of turns in the multi-turn scenarios. And the mean and median number of turns in these multi-turn conversations.