acceptodds
Withdrawn from ICLR 2027

Can Omni-modal LLMs Understand Spoken Intentions?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Omni-modal large language models (Omni-modal LLMs) can answer questions about audiovisual content, yet useful assistance also requires understanding what help the user intends to obtain. This paper introduces \ourbench for evaluating spoken intention understanding through contextual responses. The central requirement is to determine the valid request, retain its constraints, and respond according to the available evidence. We accordingly organize the benchmark around request grounding, request and context selection, assistance need assessment, request integration, and responding under uncertainty. Beyond direct requests, it examines indirect help seeking, revised requirements, ambiguous goals, and false premises. Paired scenarios vary request and evidence conditions to test whether models adjust their responses appropriately. A scenario-driven pipeline constructs scripts, generates audiovisual interactions, and defines response rubrics for language, intent, and content. Recorded videos with synthesized questions provide complementary spoken QA evaluation and training data. We further develop \ourmodel through reinforcement learning to examine transfer from explicit spoken QA to complex request conditions. Evaluation reveals substantial variation across models and tasks. Post-training improves instruct-mode performance, while uncertainty handling and selective responding remain weak. These findings establish \ourbench as a diagnostic resource for assessing spoken intention understanding and identifying priorities for model improvement.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Omni-modal LLM, Spoken Intention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PS8EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is interested in examining whether omni-modal LLMs can understand user intent from video clips with speech, without text questions. The model needs to reply with assistance, clarification, answer, or silence. The authors created a benchmark of 1400 generated video clips, covering 5 categories and 14 subtasks. They also trained an internal omni model using GRPO on a spoken QA dataset. The authors tested a handful of models on their benchmark and observed varying performance across models and across tasks. They conclude that current omni-modal LLMs do not yet reliably understand user intents spoken in video.

Critical strengths
  1. The focus on spoken intention understanding in video appears to be novel and well-motivated. The task is real and is a jump above current video benchmarks in the practical side.
  2. The benchmark design is clean and isolates different failure modes. The evaluation is well-designed as well, e.g., intent and content are scored separately and as a result getting the request wrong is distinguished from getting the specific facts wrong.
  3. The new training datasets (omniintent-train-24k) are useful resources.
Critical weaknesses
  1. The benchmark heavily relies on synthetic video and speech. The quality of the benchmark could be questionable. The paper contains minimum information on how data quality is checked. In the main text at least, I just found one sentence describing that media quality is checked. As the benchmark is the major contribution, a more detailed description on how the video is vetted could be important.
  2. The evaluator is a GPT-4o that judges replies against a textual description of the scene, and it has never seen the video. For this to work again we need to assume that the generated video is of high quality and no gap exists between the script and the video. A subset evaluated by humans and a LLM-human agreement metric would be helpful here.
  3. The training rewards' design needs more consideration. For example, the reward scores "refusing to answer" as 0. Wouldn't this penalize models that choose to stay silent, which is a category being tested.
  4. The evaluation could have been more complete: for example, silence is measured as accuracy on true-silence examples, what about false silence rate? i.e., models choosing to stay silent even when response is required? Examples of how models fail would be helpful too.
Questions for authors
  1. Section 5.2 mentions that "Results are available for the two Qwen models, while the remaining open-source evaluations are incomplete." Moreover, section 5.3 mentions that Table 2 summarizes results on the "current" diagnostic evaluation subset. What does "current" here mean? In general, how should one read Table 2? Any numbers that should not be read as of now?
  2. How do the authors deal with ambiguous cases? Are all videos used in the experiment containing clear, single intent?
  3. Which models are used for generating the data, i.e., writing the scripts, generating the speech and video?
  4. Any more details could be given on the internal base model? Why not train an open source model?
Reviewer_yy9yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces OmniIntent-Bench to evaluate whether omni-modal LLMs can identify valid user requests and respond appropriately, including clarification or silence. The benchmark comprises 1,400 generated audiovisual clips across 14 diagnostic subtasks and approximately 500 recorded videos with synthesized spoken questions. The authors also construct a 24K-sample spoken-QA training set and apply GRPO to an internal model to study transfer to complex request conditions. Results show improved instruct-mode performance, but no clear advantage over general-data RL in thinking mode, with uncertainty handling and selective responding remaining challenging.

Critical strengths

The main contribution is OmniIntent-Bench, which evaluates whether omni-modal models identify the user’s current valid request and decide how—or whether—to respond. Its 14 diagnostic subtasks systematically cover request selection, revision, integration, and uncertainty handling, including cases requiring clarification or silence. This fine-grained, response-oriented design offers a useful framework for diagnosing interaction failures beyond audiovisual answer correctness.

Critical weaknesses

1. The benchmark’s distinct diagnostic contribution is not sufficiently established.

The proposed taxonomy addresses relevant interaction behaviors, but the comparison with existing benchmarks is primarily based on differences in task coverage. The paper provides limited empirical evidence showing what additional diagnostic information OmniIntent-Bench offers beyond existing evaluations. The concern is not that every subtask must be entirely new, but that the paper’s main contribution depends on demonstrating that its task decomposition and paired-scenario design reliably isolate meaningful differences in model capabilities.

In particular, Section 3.2 and Appendix A.2 state that paired variants share a scenario design but need not be visually identical. Differences in model responses may therefore reflect both the intended changes in request conditions and incidental changes introduced during generation. This is particularly important for conditional responding, request selection, and uncertainty handling, where the appropriate response depends on specific contextual cues. The reported aggregate scores do not establish whether models respond correctly to the relevant condition changes within each scenario pair.

Controlled pairs that hold irrelevant context fixed while varying only the relevant request or evidence cues would provide stronger support for the diagnostic claims. Pair-level results should show whether models respond correctly under both conditions, rather than merely changing their outputs, and whether their decisions remain stable under irrelevant variations. Input ablations on tasks requiring visual evidence would further help distinguish multimodal request understanding from reliance on linguistic cues alone. Without such validation, the benchmark offers a potentially useful collection of interaction tasks, but its ability to isolate the intended capabilities and provide distinct diagnostic insights remains insufficiently demonstrated.

2. The evaluation protocol and post-training results leave important conclusions insufficiently validated.

GPT-4o judges responses using textual annotations and rubrics without inspecting the audiovisual input. A text-based judge is not inherently inappropriate, but its reliability depends on whether these annotations accurately and sufficiently represent the evidence available to the evaluated models. The paper does not report independent human evaluation based on the original clips or human–judge agreement to validate the resulting scores. Consequently, the impact of annotation omissions, ambiguous interaction cues, and differences in rubric interpretation remains unquantified. This limits confidence in the reported model comparisons and diagnostic findings. Separately, establishing whether models actually use the relevant audiovisual evidence requires input-level controls; it cannot be inferred from response scores alone.

The post-training results support a benefit from spoken-QA training in instruct mode, but do not establish an additional benefit from difficulty filtering. OmniIntent-RL improves over general-data RL from 53.9 to 66.6 in instruct mode, yet the unfiltered audio-QA setup also achieves 66.6. In thinking mode, OmniIntent-RL scores 66.3, compared with 66.6 for general-data RL and 66.8 for the unfiltered setup. Without uncertainty estimates or repeated runs, these small differences should not be interpreted as meaningful improvements or regressions. The supported conclusion is therefore narrower: spoken-QA training improves instruct-mode performance in the reported experiment, while its benefit in thinking mode and the contribution of filtering remain unestablished.

Furthermore, the policy backbone is identified only as an internal model, limiting reproducibility and leaving the applicability of these findings to other backbones uncertain. Overall, the key concern is not the absence of a new RL algorithm or uniformly positive results, but insufficient validation of the benchmark’s diagnostic reliability and limited evidence isolating the contributions of the proposed training choices.

Questions for authors

1. Benchmark validity and controlled evaluation

Could you provide results on controlled counterfactual pairs in which irrelevant scene details and context are held fixed while the request condition changes? In particular, how often do models correctly switch between answering, seeking clarification, and remaining silent when required, while staying consistent under irrelevant changes? Such results would help establish whether the benchmark measures sensitivity to the intended request conditions rather than artifacts of independently generated scenarios.

2. Reliability of the automatic evaluation

Have you conducted an independent human evaluation in which annotators directly inspect the audiovisual clips and judge the model responses? If so, please report agreement between human judgments and GPT-4o, as well as inter-annotator agreement, preferably broken down by task category. If not, how do you verify that the textual scene descriptions and rubrics provide sufficient evidence for reliable judgments, especially for addressee tracking, implicit requests, and visual premise verification?

3. Interpretation of the RL results and reproducibility

Could you clarify the exact policy backbone and whether the general-data RL, unfiltered audio-QA RL, and filtered training configurations use comparable training budgets? The current results show no overall improvement for OmniIntent-RL over general-data RL in thinking mode. How do you interpret the contribution of difficulty filtering, given the comparable or slightly higher scores of the unfiltered setup? Results from repeated runs would help determine whether the observed differences are reliable.

4. Dataset construction and release details

Appendix A.1 states that the final retained evaluation counts remain to be confirmed, whereas the main paper reports 1,400 diagnostic clips and approximately 500 spoken QA clips. Could you clarify the final retained counts and the exact evaluation set used for the reported results? Please also describe the quality-control and sample-rejection procedures, and clarify the permissions and privacy safeguards for videos collected from WeChat Channels, particularly for the planned public release.

Reviewer_QvF7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OmniIntent-Bench to evaluate whether omni-modal LLMs can understand spoken intentions and respond appropriately to different request conditions. The benchmark consists of 1,400 generated diagnostic clips across 14 subtasks across and approximately 500 spoken QA clips constructed from recorded videos. The benchmark is organized around request grounding, request and context selection, assistance need assessment, request integration, and responding under uncertainty. The authors also construct OmniIntent-Train-24K and develop OmniIntent-RL using GRPO to investigate whether training on explicit spoken QA can improve performance on more complex request conditions.

Critical strengths

The benchmark addresses a meaningful problem in spoken intention understanding. It covers different aspects of user requests, such as implicit requests, request revisions, clarification, and false premises, as well as when the model should remain silent. The paired scenario design is also useful for examining how models respond to different request conditions.

Critical weaknesses
  1. In Section 5.2 (Lines 429–430), the authors state that “Results are available for the two Qwen models, while the remaining open-source evaluations are incomplete.” However, Table 2 already reports results for these models. This inconsistency raises concerns about the care taken in preparing the paper and makes it unclear whether the reported evaluations are complete. The authors should clarify this discrepancy.
  2. The benefits of the proposed training dataset OmniIntent-Train-24K and OmniIntent-RL model are not clear. The overall score of OmniIntent-RL (Instruct) is 66.6, which is the same as AQ RL w/o filtering (Instruct). The overall score of OmniIntent-RL (Thinking) is 66.3, lower than General RL (Thinking) (66.6) and AQ RL w/o filtering (Thinking) (66.8). This makes the benefit of the proposed difficulty-based data selection unclear. Moreover, since OmniIntent-Train-24K is mixed with general multimodal data during training, it is hard to isolate its contribution to the final results. More analysis is needed to show what the proposed dataset and training strategy contribute beyond the existing baselines.
  3. The abstract does not clearly convey the full scope of the paper. For example, it does not mention the size of OmniIntent-Bench or clearly present the benchmark's task categorization. It also omits OmniIntent-Train-24K, another contribution of the paper. Including these details would help readers understand the main contributions more quickly.
Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.