acceptodds
Under review as a conference paper at ICLR 2027

Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction

Abstract

Natural audio-visual interaction is emerging as an important interface for AI assistants, allowing users to communicate through speech and vision rather than carefully composed text prompts. However, existing benchmarks of interactive capabilities still focus primarily on response quality, leaving a more fundamental question underexplored: can a model correctly infer the user's underlying demand from a complex multimodal interaction? Real-world user demands are often underspecified in speech and must be inferred from multimodal cues and dialogue history. This inference is further complicated by ambiguous or disfluent expression and noisy acoustic environments. Conversely, request-like speech may not constitute a demand to the assistant, leading to false triggers. We establish Omni Demand Understanding (ODU) as a distinct multimodal contextual inference problem: given an interaction stream, a model must detect whether a user demand is present and infer the user's intent from multimodal and conversational context. ODU evaluates this capability along five dimensions, covering both single-turn and multi-turn interactions. We construct ODU-Bench using a challenge-driven taxonomy, taxonomy-guided agentic video generation, and human-recorded interactions, followed by media-grounded annotation and human verification. We evaluate 14 native audio and audio-visual MLLMs. Even Gemini 3.1 Pro, the strongest model by audio-visual aggregate score, recovers only 44.7% of the key contextual information needed to interpret user demands. Moreover, 11 of the 14 models exhibit false-trigger rates above 50% on non-demand scenarios. These results reveal a systematic capability gap in current MLLMs' ability to infer contextual user demands. We hope ODU can establish the evaluation of a previously underexplored yet essential capability in multimodal interaction: correctly understanding user demands before generating an appropriate response. See demos at https://odubench.github.io

Primary area:
datasets and benchmarks
Keywords:
Multimodal Interaction, Audio-Visual Understanding, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_r7tgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces Omni Demand Understanding benchmark, which evaluates whether multimodal models are able to identify valid user requests and are able to understand the underlying intent from audio, visual, as well as conversational context. The authors construct ODU-Bench, which contains 2078 synthetic as well as human recorded interactions, and they evaluate models across 5 dimensions. They test 14 multimodal models and their ablations show weaknesses such as high false trigger rates and weaknesses in contextual reasoning.

Critical strengths
  1. I feel the paper addresses an important aspect of multimodal interactions, which is currently a bit under explored. Instead of just evaluating whether the response of a model is correct or not, the paper focuses on evaluating whether a model actually understands when a user has made a request or not, and what the user actually intends.

  2. The benchmark covers a wide set of interactions, including indirect requests, as well as requests not directed to the assistant, and they have tested 14 models, which is almost all the models available.

  3. I liked the ablations done in the paper, including Figure 4, which shows that models perform much better in recovering information which has been explicitly stated in speech rather than information which requires visual, acoustic, or dialogue history.

Critical weaknesses
  1. The paper does not specify which audio and video generation models were used to generate the videos in the benchmark and the synthetic interactions. Also since these synthetic interactions make up the majority of the benchmark (1801 out of 2078 interactions), these missing details about the implementation limit the reproducibility. This also makes it difficult to assess any potential generation specific artifacts.

  2. The paper does not state the quantitative human verification results of the generated synthetic interactions. Although appendix C.4 describes that there was human verification of the generated interactions, it does not mention the inter-annotator agreement or the rejection rate, for example how many samples were rejected from the initial pool of generated samples, and it also does not tell us the portion of samples which required correction. Because the benchmark relies a lot on synthetic generated media, these statistics are important so that we can assess the reliability and the consistency of its annotations.

  3. The paper's aggregate score gives 60% weightage to intent recovery, which is M2, and only 15% weightage to demand detection, which is M1. I feel this may under represent false trigger failures, which are important for real world assistants, for example Seed 2.0 Lite scores a 67.9% overall Audio-Visual score, but it has a 82.2% false trigger rate in Table 3. So the paper does not properly show whether model rankings remain same under changed or different metric weights. The authors should show how the model rankings change if we give different weightage to the 5 evaluation metrics. Also, the weightage of these 5 evaluation metrics seem to be heuristic instead of empirically justified.

  4. The paper shows that there are 969 multi-turn conversations but it does not have any ablation which compares the performance with and without the previous conversation history. Without this comparison it is hard to quantify how much the models improve or degrade when we give them the conversational history and the context.

  5. The paper does not provide any sort of distribution of the number of turns or the average number of turns per conversation. It would be helpful to see how does the model performance vary in accordance to the number of turns per conversation.

Questions for authors

I would like the authors to tell in detail which audio and video generation models they use to construct the synthetic interactions in the benchmark. And what protocol they followed for qualitative and quantitative assessment.

  1. The author should also mention the statistics regarding their selection criteria of the videos generated by the video generation model . What was the inter-annotator agreement, how many humans rated it, what was the rating process, and what was the rejection rate, what were the qualifications of the humans who rated it.

  2. The authors should show how would the model rankings in table 3 change if a greater weightage is assigned to demand detection and the false trigger behavior.

  3. It would be helpful if the authors could evaluate the multi-turn scenarios with and without the previous conversation history so that we can measure its contribution in regards to demand understanding.

  4. It would be helpful if the authors could report the distribution of the number of turns in the multi-turn scenarios. And the mean and median number of turns in these multi-turn conversations.

Reviewer_3mUYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors introduce ODU-Bench, designed to jointly evaluate whether a valid demand is directed to an assistant and infer the user's latent intent from multimodal & conversational context. The benchmark comprises 2K human-verified scenes across audio-only and audio–visual interactions, including single-turn and multi-turn scenarios, combining synthetic media with human-recorded interactions. Authors perform survey over 14 native multimodal models on the proposed benchmark across five dimensions, showing substantial gaps in contextual intent recovery and false-trigger suppression.

Critical strengths
  • Gap identification and corresponding contribution. The paper explicitly separates demand understanding from response generation -- which makes a useful distinction since understanding what the user wants and whether assistance is requested are often assumed rather than evaluated explicitly.

  • Well-designed taxonomy with broad capability coverage. The proposed taxonomy covers modality dependency, contextual disambiguation, expression forms, acoustic conditions, etc. with a separate taxonomy for no-demand scenarios.

  • Explicit no-demand evaluation seems particularly important due to the subjective nature of the task and also unique, because conventional assistant benchmarks primarily focus on answering valid requests only.

  • Empirical analysis is broad and thorough. Evaluation across 14 models establish the difficulty of the proposed benchmark. Abaltion breakdowns makes the capability gap clear as well. The blinded response comparison further supports the practical value of correct demand understanding.

Critical weaknesses
  • Heavy reliance on synthetic & scripted interactions. Majority of the scenes are synthetically structured, and the human-recorded subset also follows generated scripts -- while human verification and the source comparison are useful, generalization to spontaneous real-world interactions remains unclear.

  • The big assumption seems to be that the user's context and intent have groundable 1:1 mapping, whereas in reality similar contexts may lead to multiple varying intents, depending on personalization signals, etc. Defining precision and recall on a task with this level of subjectivity would need more careful design.

  • Limited human validation of evaluation reliability. Human verification is described but inter-annotator agreement and a human performance baseline are not reported -- as in the above which are particularly important for ambiguous implicit demands. The cross-judge audit preserves rankings for three models, but does not necessarily establish agreement with human judgments.

Questions for authors

See above

Reviewer_cb3nOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates omni demand understanding as a distinct multimodal contextual inference problem, establishing it as an explicit evaluation target rather than leaving it implicit within overall response generation. To address this, the authors construct a challenging benchmark spanning diverse demand and no-demand scenarios through a combination of taxonomy-guided agentic generation and human-recorded interactions. Utilizing this benchmark, the work uncovers a systematic capability gap, demonstrating that current multimodal large language models consistently struggle with contextual user-intent inference when it requires integrating multimodal and conversational evidence beyond the explicit spoken request.

Critical strengths

This paper introduces Omni-Demand Understanding (ODU) as a novel multimodal task, backed by the comprehensive ODU-Bench and a structured five-dimension evaluation framework. By testing 14 state-of-the-art models, the study reveals that current systems systematically struggle with integrating multimodal evidence for contextual reasoning, despite having strong perception capabilities. Ultimately, A/B testing proves that accurate demand understanding significantly improves downstream assistant responses, highlighting the framework's practical value.

Critical weaknesses
  1. The ODU-Bench dataset relies too heavily on synthetic data, featuring only 277 real human interactions out of 2,078 total scenes. This small sample size of real data raises concerns about whether LLM-generated media can accurately capture real-world audio-visual noise, especially since the paper does not evaluate the reliability of these synthetic scenes.
  2. For the core semantic evaluation metric (M2: Structured Intent and Required Context), the paper relies on an LLM (Qwen3.6-Flash) to compare predictions against reference key points. While common in large-scale evaluations, ICLR reviewers frequently raise strict concerns regarding the stability and potential bias of LLM judges.
Questions for authors

Why is the layout for Section 5.1 left-aligned instead of fully justified?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.