- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces OmniAssistBench for evaluating omni-modal models in assistant-style video interactions. The benchmark contains 300 videos covering perception, visual instruction following, proactive responses, and multi-turn process tracking. To enable offline evaluation, the authors derive procedural priors from source videos and constrain the expected interaction to a predefined path. They also include three longer, manually recorded scenarios. Evaluation uses open-ended responses assessed by an LLM judge.
Critical strengths
-
The benchmark addresses relevant assistant capabilities that are not fully captured by conventional video QA, including identifying the intended addressee, following visual instructions, and maintaining ongoing tasks across interruptions.
-
The scenarios are carefully constructed, with partially completed steps, visually similar distractors, and concurrent activities. These details require models to track the user’s situation beyond recognizing the main content of a clip.
-
The modality and teacher-forcing ablations also provide useful analysis beyond model rankings. The finding that background speech can interfere with an earlier instruction is particularly relevant to assistants operating in natural environments.
Critical weaknesses
-
I think the fixed-path construction cannot fully resolve the interaction-path problem motivating the paper. A prescribed procedure constrains the expected response, but the model may still recommend an incorrect step or ask the user to repeat an action, while subsequent clips follow the recorded trajectory regardless. For example, the model might ask the user to show an object from another angle before giving further instructions, but the next clip already shows the user completing the next step. Later turns would then involve observations that do not follow from the model’s guidance. The benchmark can evaluate responses along a recorded trajectory, but whether this predicts successful interactive assistance remains unclear. The teacher-forcing ablation addresses errors in response history rather than this mismatch.
-
Proactive responses appear to be evaluated at predefined clip boundaries. The paper does not explain well how these boundaries are selected or how they affect performance. Deciding whether to respond after a completed clip differs from responding at the appropriate moment in a continuous stream. Testing shifted boundaries or fixed-duration segments would help establish the robustness of this evaluation.
-
The scoring protocol lacks validation against human judgments. Correlation across LLM judges shows consistency, but does not establish that their scores reflect answer quality. The appendix reports that Gemini scores higher with audio-only input on one task because its audiovisual responses receive formatting penalties. Human validation and separate reporting of content and presentation scores would help determine whether these penalties actually affect the conclusions.
Questions for authors
-
How are subsequent turns evaluated when a model’s earlier guidance conflicts with the prerecorded action? Is there evidence that performance under this offline protocol predicts performance in live interactions? Can you do some further experiments about this?
-
How are clip boundaries determined for proactive-response tasks, and how sensitive are the results to shifting these boundaries or using fixed-duration segments?
-
What is the agreement between automatic scores and human ratings? Would the model rankings change if content quality and presentation style were evaluated separately?
3 comments
3. Human-score validation, content/style separation, and Gemini modality effects
Thank you for highlighting the need for human validation. We conducted an additional study in which two human annotators independently evaluated 60 responses from Gemini-3-Pro and Qwen3-Omni-Instruct on 30 QAs, covering all task subclasses except the real-world cases. The annotators followed the same 0–5 scoring criteria as our LLM judge. The overall Pearson correlation between LLM scores and averaged human ratings is 0.797, closely matching the inter-human correlation of 0.796. The correlations remain strong for both Gemini-3-Pro (0.767) and Qwen3-Omni-Instruct (0.781). Moreover, both human and LLM evaluations produce the same ranking between these two models. We have incorporated these results into the revised paper, please refer to Appendix D.2 for detailed information.
Content-focused evaluation and Gemini modality comparison. Thank you for this suggestion. We have conducted an additional evaluation that separates content quality from presentation-related factors. Specifically, we retain the original holistic scores and independently re-evaluate saved model responses using a three-level content-focused rubric, without penalizing formatting, conversational style, or harmless redundancy.
As shown in Appendix D.5 of the revised paper, the ranking of four representative models remains unchanged under the new rubric: Gemini-3-Pro MiMo Qwen3-Omni MiniCPM-o-4.5. This ranking is also preserved when excluding mandatory silence tasks. We further observe strong correlations between the original and content-focused scores (--). These results indicate that although presentation requirements may affect individual scores, the main differences between models are not solely caused by formatting penalties.
We specifically re-evaluated Gemini-3-Pro on the 20 Context-aware Response questions under both audio-only and audiovisual settings. The resulting content scores are 1.60 and 1.45 (out of 2), respectively. Although the audio-only setting retains a slightly higher average score, the paired difference is not statistically significant (). Further inspection shows that the observed difference involves not only presentation style but also individual content errors and an incorrect silence decision in the audiovisual setting.
We have revised the discussion in Appendix D.1 to clarify that formatting may influence individual holistic scores, but the observed audio-only advantage does not establish a consistent performance difference between the two input modalities.
Polite follow-up: request for reviewers’ feedback on our responses!
Hi, we sincerely thank you very much for these constructive comments and evaluation of our manuscript. We would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.
Thank you again for dedicating your time to reviewing our paper.
1. Fixed-path mismatch and evidence for live interactions
We agree that predefined priors cannot eliminate the mismatch between model guidance and subsequent prerecorded observations. Our motivation is to address two challenges in constructing offline interaction benchmarks: the scarcity of real human-assistant interaction recordings and the ambiguity caused by multiple valid procedures. We therefore propose a reverse-engineering pipeline to construct multi-turn tasks from Internet videos, while providing predefined priors to specify the intended procedure and establish a common basis for evaluation. We also agree that teacher forcing addresses response-history effects rather than environment-transition mismatch. Thus, our benchmark evaluates guidance along recorded trajectories, while its ability to predict live task success or recovery remains unverified. We have revised the Introduction section to clarify this distinction.
2. Sensitivity to proactive-response clip boundaries
We thank the reviewer for this valuable suggestion. In our benchmark, clip boundaries are determined during annotation to divide videos into sequential interaction turns, rather than being explicitly aligned with the exact onset of proactive events. To investigate the sensitivity of evaluation results to boundary placement, we conducted an additional experiment on all 32 proactive-response videos, comprising 133 QA instances. We perturbed the internal clip boundaries by up to 10 seconds while preserving the number of interaction turns and the total video duration. Importantly, the first clip of each video remained unchanged to preserve the initial user instruction. Consequently, 96 QA instances had their observation boundaries actually modified, and we focused our analysis exclusively on these affected instances. The average score of Qwen3-Omni changed only from 2.427/5 (48.54/100) under the original boundaries to 2.448/5 (48.96/100) under the perturbed boundaries, corresponding to a negligible difference of +0.021/5 (+0.42/100). Furthermore, the paired scores exhibited a strong Pearson correlation (r = 0.826). A paired t-test revealed no statistically significant difference (p = 0.804). These results suggest that small perturbations of predefined clip boundaries do not significantly affect the overall evaluation scores, supporting the robustness of our discrete multi-turn evaluation protocol. Nevertheless, we acknowledge that this experiment does not establish response-timing accuracy in fully continuous streaming scenarios.