acceptodds
Under review as a conference paper at ICLR 2027

OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs

Abstract

Recent omni-modal large language models (Omni-LLMs) show great potential as real-time video assistants,which continuously perceive environments and guide users to achieve specific goals. Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help. Evaluating this is rather challenging, as the model's unpredictable response dynamically changes the user's subsequent actions, which static offline datasets cannot accommodate. To address this bottleneck, we introduce . To solve the issue of diverging interaction paths (where the same user goal can be achieved through various methods), we provide models with predefined priors derived from the source video, requiring them to guide users along the exact same routes. Since real interaction videos are rare, we construct the dataset by reverse-engineering existing Internet videos. We deduce logical user goals and segment the videos into multi-turn clips to simulate continuous interactions. This rigorous pipeline required over 1000 expert person-hours to build the dataset. Results show that the proprietary Gemini-3-Pro reaches 66.4 out of the max point of 100, while the open-source Qwen3-Omni-Instruct achieves 51.2. Although current models generally understand user inputs, they frequently provide incorrect or incomplete answers. Specifically, they struggle with visual prompts (e.g., hand gestures), fail to maintain historical context during multi-turn interactions, and fail to delay response until the target event. Results indicate substantial room for improvement before models can become reliable assistants.

Primary area:
datasets and benchmarks
Keywords:
Omni-modal Large Language Models, Interactive Understanding, Video Assistant, Benchmark Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DHBgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces OmniAssistBench for evaluating omni-modal models in assistant-style video interactions. The benchmark contains 300 videos covering perception, visual instruction following, proactive responses, and multi-turn process tracking. To enable offline evaluation, the authors derive procedural priors from source videos and constrain the expected interaction to a predefined path. They also include three longer, manually recorded scenarios. Evaluation uses open-ended responses assessed by an LLM judge.

Critical strengths
  1. The benchmark addresses relevant assistant capabilities that are not fully captured by conventional video QA, including identifying the intended addressee, following visual instructions, and maintaining ongoing tasks across interruptions.

  2. The scenarios are carefully constructed, with partially completed steps, visually similar distractors, and concurrent activities. These details require models to track the user’s situation beyond recognizing the main content of a clip.

  3. The modality and teacher-forcing ablations also provide useful analysis beyond model rankings. The finding that background speech can interfere with an earlier instruction is particularly relevant to assistants operating in natural environments.

Critical weaknesses
  1. I think the fixed-path construction cannot fully resolve the interaction-path problem motivating the paper. A prescribed procedure constrains the expected response, but the model may still recommend an incorrect step or ask the user to repeat an action, while subsequent clips follow the recorded trajectory regardless. For example, the model might ask the user to show an object from another angle before giving further instructions, but the next clip already shows the user completing the next step. Later turns would then involve observations that do not follow from the model’s guidance. The benchmark can evaluate responses along a recorded trajectory, but whether this predicts successful interactive assistance remains unclear. The teacher-forcing ablation addresses errors in response history rather than this mismatch.

  2. Proactive responses appear to be evaluated at predefined clip boundaries. The paper does not explain well how these boundaries are selected or how they affect performance. Deciding whether to respond after a completed clip differs from responding at the appropriate moment in a continuous stream. Testing shifted boundaries or fixed-duration segments would help establish the robustness of this evaluation.

  3. The scoring protocol lacks validation against human judgments. Correlation across LLM judges shows consistency, but does not establish that their scores reflect answer quality. The appendix reports that Gemini scores higher with audio-only input on one task because its audiovisual responses receive formatting penalties. Human validation and separate reporting of content and presentation scores would help determine whether these penalties actually affect the conclusions.

Questions for authors
  1. How are subsequent turns evaluated when a model’s earlier guidance conflicts with the prerecorded action? Is there evidence that performance under this offline protocol predicts performance in live interactions? Can you do some further experiments about this?

  2. How are clip boundaries determined for proactive-response tasks, and how sensitive are the results to shifting these boundaries or using fixed-duration segments?

  3. What is the agreement between automatic scores and human ratings? Would the model rankings change if content quality and presentation style were evaluated separately?

3 comments

1. Fixed-path mismatch and evidence for live interactions

We agree that predefined priors cannot eliminate the mismatch between model guidance and subsequent prerecorded observations. Our motivation is to address two challenges in constructing offline interaction benchmarks: the scarcity of real human-assistant interaction recordings and the ambiguity caused by multiple valid procedures. We therefore propose a reverse-engineering pipeline to construct multi-turn tasks from Internet videos, while providing predefined priors to specify the intended procedure and establish a common basis for evaluation. We also agree that teacher forcing addresses response-history effects rather than environment-transition mismatch. Thus, our benchmark evaluates guidance along recorded trajectories, while its ability to predict live task success or recovery remains unverified. We have revised the Introduction section to clarify this distinction.

 

2. Sensitivity to proactive-response clip boundaries

We thank the reviewer for this valuable suggestion. In our benchmark, clip boundaries are determined during annotation to divide videos into sequential interaction turns, rather than being explicitly aligned with the exact onset of proactive events. To investigate the sensitivity of evaluation results to boundary placement, we conducted an additional experiment on all 32 proactive-response videos, comprising 133 QA instances. We perturbed the internal clip boundaries by up to 10 seconds while preserving the number of interaction turns and the total video duration. Importantly, the first clip of each video remained unchanged to preserve the initial user instruction. Consequently, 96 QA instances had their observation boundaries actually modified, and we focused our analysis exclusively on these affected instances. The average score of Qwen3-Omni changed only from 2.427/5 (48.54/100) under the original boundaries to 2.448/5 (48.96/100) under the perturbed boundaries, corresponding to a negligible difference of +0.021/5 (+0.42/100). Furthermore, the paired scores exhibited a strong Pearson correlation (r = 0.826). A paired t-test revealed no statistically significant difference (p = 0.804). These results suggest that small perturbations of predefined clip boundaries do not significantly affect the overall evaluation scores, supporting the robustness of our discrete multi-turn evaluation protocol. Nevertheless, we acknowledge that this experiment does not establish response-timing accuracy in fully continuous streaming scenarios.

3. Human-score validation, content/style separation, and Gemini modality effects

Thank you for highlighting the need for human validation. We conducted an additional study in which two human annotators independently evaluated 60 responses from Gemini-3-Pro and Qwen3-Omni-Instruct on 30 QAs, covering all task subclasses except the real-world cases. The annotators followed the same 0–5 scoring criteria as our LLM judge. The overall Pearson correlation between LLM scores and averaged human ratings is 0.797, closely matching the inter-human correlation of 0.796. The correlations remain strong for both Gemini-3-Pro (0.767) and Qwen3-Omni-Instruct (0.781). Moreover, both human and LLM evaluations produce the same ranking between these two models. We have incorporated these results into the revised paper, please refer to Appendix D.2 for detailed information.

Content-focused evaluation and Gemini modality comparison. Thank you for this suggestion. We have conducted an additional evaluation that separates content quality from presentation-related factors. Specifically, we retain the original holistic scores and independently re-evaluate saved model responses using a three-level content-focused rubric, without penalizing formatting, conversational style, or harmless redundancy.

As shown in Appendix D.5 of the revised paper, the ranking of four representative models remains unchanged under the new rubric: Gemini-3-Pro >> MiMo >> Qwen3-Omni >> MiniCPM-o-4.5. This ranking is also preserved when excluding mandatory silence tasks. We further observe strong correlations between the original and content-focused scores (ρ=0.779\rho=0.779--0.8960.896). These results indicate that although presentation requirements may affect individual scores, the main differences between models are not solely caused by formatting penalties.

We specifically re-evaluated Gemini-3-Pro on the 20 Context-aware Response questions under both audio-only and audiovisual settings. The resulting content scores are 1.60 and 1.45 (out of 2), respectively. Although the audio-only setting retains a slightly higher average score, the paired difference is not statistically significant (p=0.317p=0.317). Further inspection shows that the observed difference involves not only presentation style but also individual content errors and an incorrect silence decision in the audiovisual setting.

We have revised the discussion in Appendix D.1 to clarify that formatting may influence individual holistic scores, but the observed audio-only advantage does not establish a consistent performance difference between the two input modalities.

AuthorsOpenReview ↗

Polite follow-up: request for reviewers’ feedback on our responses!

Hi, we sincerely thank you very much for these constructive comments and evaluation of our manuscript. We would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.

Thank you again for dedicating your time to reviewing our paper.

Reviewer_Sjc9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OmniAssistBench, a novel benchmark including Basic-tier and Advanced-tier framework to evaluate the ability of state-of-the-art models. The authors construct the dataset by reverse-engineering existing videos to multi-turn clips to simulate real interactions. Four-step data construction pipeline including Scenario design and video collecting, QA design, video editing and user prompt embedding, and quality refinement, ensures the whole data construction process. Extensive experiments including task evaluation and 4 ablation studies indicate the room for model improvement.

Critical strengths

1.The annotation of OmniAssistBench is time-consuming and subtle. It includes over 1000 expert hours for data construction. 4-step novel pipeline provides a comprehensive process of construction OmniAssistBench. They focus on the details that may be ignored by the current MLLMs. 2. This work evaluates different Omni-LLMs including 5 closed-source models and 6 open-source models and presents performance on its designed tasks. 3. Critical ablation studies indicate the core function of modalities and generalizability of the selection of the judge model, proving insights and the strong application of OmniAssistBench.

Critical weaknesses
  1. The authors provide the four-stage pipeline of constructing OmniAssistBench. However, the authors don’t provide a detailed description of acquiring videos from Youtube.
  2. The authors give the source of existing datasets, but the proportion of selecting existing datasets is not given.
  3. Qualitative analysis for the representative failure cases for models is not provided, such as struggling with visual prompts, maintaining historical context during multi-turn interactions, and delay response.
  4. Although the paper includes 3 home-filmed real-world scenarios, its evaluation primarily relies on predefined paths and offline segments. It remains unclear how well the benchmark reflects model behavior in open-ended, real-time interactions.
Questions for authors
  1. How do the authors select the raw videos from Youtube? Please provide a detailed process and selecting and processing the raw videos. Are the raw videos from Youtube consented? Please clarify the licensing applicable to the use of these videos.
  2. Please provide the information about the existing dataset that are added to OmniAssistBench, including percentage information and so on.
  3. Qualitative analysis is required to acquire a clear understanding of the failures mentioned in the abstract.
  4. The authors say that they have predefined the paths for the questions and answers, how about the real-application of the model ability on open-ended and real-time open videos?
  5. In Page 7, Evaluated Models, it is mentioned that “For models except the Gemini family, videos are sampled at a rate of 1 frame per second (FPS).” What the FPS of the Gemini family for videos in this work? How is evaluation comparability ensured?
  6. Minor issues: a)Please pay attention to the correctness of some sentences: i.“While the former focus on fundamental perception tasksthat are critical” in 3 Task Design ii.“Base on this classification, we distill …” in 3 Task Design b)The uniformness of OmniassistBench versus OmniassisBench (in 2.1 Data Format and 4.1 Evaluated Models) should be focused. c)“Our evaluation suite comprises four closed-source models … alongside five open-source models” in 4.1 Evaluated Model. It should be five and six.
6 comments

 

2. Proportions from existing datasets

Thank you for your suggestion. Among the 300 videos in OmniAssistBench, 54 (18.0%) are sourced from existing public datasets, contributing 54 QA pairs (7.9% of the total 685 QA pairs). All these videos belong to the Basic tier, while the remaining 246 videos are collected directly from YouTube or filmed by ourselves. Importantly, we only use the raw video footage from existing datasets, without reusing their original annotations or captions. These videos undergo the same annotation pipeline as those collected from YouTube. We have add this information in the "cenario design & video collecting" part of Section 2.2 in the revised paper.

 

3. Representative qualitative failure cases

Thank you for the suggestion. We have added three qualitative examples in Appendix C, illustrating failures in gesture-based prompt following, event-triggered responses, and multi-task tracking. Specifically, the multi-task tracking example shows that Qwen3-Omni fails to correctly update the progress of several concurrent tasks across turns, even though task names and action descriptions are explicitly provided in the video. We have also revised the experimental analysis (The "Advanced Tier tasks" part in Section 4.2) to distinguish two challenges in maintaining historical context: (1) limited context capacity and FIFO truncation, which may remove earlier task information, and (2) difficulties in utilizing available observations to maintain and update task states. We acknowledge that these factors may coexist, and therefore avoid attributing individual failures solely to the model's intrinsic memory capability.

 

4. Applicability to real-time, open-ended interactions

We acknowledge that our offline protocol does not fully reproduce open-ended, real-time interactions. Our intention is to approximate multi-turn online video assistance while enabling scalable and reproducible evaluation. Specifically, we divide videos into sequential clips and append spoken user questions to mimic voice-based conversations. Models must combine evolving visual observations, user goals, and interaction history to provide appropriate guidance. Since real human-assistant interaction recordings are scarce, our reverse-engineering pipeline constructs such tasks from existing Internet videos. While the benchmark evaluates important capabilities required for real-world assistance, its ability to predict live interaction success remains unverified. We have revised the Introduction to clarify this design motivation and evaluation scope.

 

5. Gemini frame sampling and comparability

Thank you for pointing out this missing detail. For the Gemini family, we upload the complete video files through Google's official Files API without any external frame sampling or preprocessing. According to the official Gemini API documentation, the default video processing samples frames at 1 FPS. For other models, we uniformly sample videos at 1 FPS while preserving the original audio tracks. Therefore, the nominal video sampling rates are consistent across models. However, we acknowledge that internal video encoding and audio processing may differ across model implementations, as we follow their respective official recommended configurations. We have added these details and the corresponding documentation reference to the Evaluated Models section to clarify the sampling settings and evaluation comparability.

 

6. Writing errors and model-count corrections

Thank you for pointing out these writing and consistency issues. We have corrected the two sentences in Section 3 (Task Design), standardized the benchmark name as OmniAssistBench throughout the manuscript, and corrected the model counts in Section 4.1 (Evaluated Models) to five closed-source and six open-source models.

1. YouTube video collection, processing, and licensing

Thank you for your suggestion. Our video collection follows a capability-driven, plot-guided process, consisting of the following steps:

  1. Task Objective Definition. For each task category, experts first define the specific capability to be evaluated. For example, the Identity Recognition task evaluates whether models can associate a person's identity (e.g., name or occupation) with their visual appearance, and recall this information when the person appears later. Also, we want the identity information to be presented in audio, so the model needs to combine visual and audio information to answer the question.

  2. Plot Template Design. Experts then design challenging plot templates that can effectively evaluate the target capability. For Identity Recognition, an ideal video contains multiple people who introduce their names only once at the beginning, requiring models to remember the association between names and appearances. Videos with only one person or names continuously displayed on screen would be less challenging. These templates guide video selection but are not strict requirements, as other plots satisfying the same evaluation objective are also acceptable.

  3. Video Search. Experts collaborate with LLMs to identify video topics likely to contain the desired plots, and search YouTube using relevant keywords. For example, interviews are likely to contain introductions of multiple people. We therefore search for different types of interviews (e.g., news interviews and street interviews), while also exploring other possible scenarios, such as multi-person vlogs and athlete introductions in sports videos, to improve scene diversity.

  4. Manual Screening. Experts watch candidate videos and determine whether challenging QA pairs aligned with the original task objective can be constructed. For example, Identity Recognition videos should contain sufficient distractors and require models to remember previously introduced identities, rather than simply reading names from the current frame. We also require key visual cues to remain clearly recognizable for at least one second, without severe camera shaking or other quality issues. Questions must be answerable using video content and common knowledge, without specialized domain knowledge.

  5. Video Processing and Annotation. After selection, experts design task-specific QA pairs and edit the videos to preserve the necessary evidence. For Identity Recognition, this includes retaining the introduction of each person's identity and the visual observations needed for later identification, while masking continuously displayed names when necessary. For multi-turn tasks, videos are divided into sequential clips, with boundaries generally placed at natural breaks rather than strictly following procedural steps. A clip may contain multiple steps or only part of a step, requiring models to infer task progress from the observations. We also balance clip lengths and remove irrelevant content while preserving task continuity. Finally, user questions are converted into speech and embedded into the videos.

  6. Video Rights. We thank the reviewer for raising this important concern. OmniAssistBench is developed for non-commercial academic research, with the goal of evaluating the interactive capabilities of omni-modal models. Rather than simply redistributing existing videos, we process source footage into structured, multi-turn interaction scenarios through temporal segmentation, task-specific annotation, and the integration of user queries and contextual information. These research-specific adaptations provide additional value for evaluating model capabilities and serve a different research purpose from the original videos. We acknowledge that publicly accessible videos may still be subject to copyright restrictions, and we respect the intellectual property rights of the original creators. We will further clarify these considerations in the revised paper. Specifically, we plan to review the licensing status of the source videos, provide appropriate source attribution, and avoid redistributing video files where the necessary permissions cannot be established. In such cases, we will consider releasing our original annotations, evaluation protocols, and source references instead.

We have expanded the Data Collection and Video Editing sections to clarify these procedures. The central principle of our pipeline is to select videos based on whether they support meaningful and challenging evaluations of the target capabilities, rather than requiring them to strictly match predefined plot templates.

AuthorsOpenReview ↗

Polite follow-up: request for reviewers’ feedback on our responses!

Hi, we sincerely thank you very much for these constructive comments and evaluation of our manuscript. We would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.

Thank you again for dedicating your time to reviewing our paper.

Reviewer_P19qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, authors argued that static video benchmark cannot evaluate the capability of models under assistant-style interactions, i.e., the user reacted to the model/assistant’s response and model/assistant follow-ups with the user’s feedback. In this way, authors proposed OmniAssistantBench which adopts fixed iteration trajectories between the user and model /assistant with a video and covers 16 sub-tasks ranging from basic interactions to long-horizon tasks (derived from realistic user scenarios). To collect data, authors first collect videos from Internet, infer the user’s potential/possible intents when interacting with an assistant with the video, and convert the video into multi-turn user-assistant interactions. Specifically, they leveraged a pipeline to segment videos, and added prompt as voice, subtitles, hand gesture, and handwritings back to the videos. The model/agent would respond with open-ended responses which would be scored by an LLM judge from the perspectives of correctness, completeness, redundancy, and compliance with silence requirements.

With this pipeline, the research evaluated 11 Omni-LLMs with their datasets, performed ablation studies on multiple dimensions, including modality of input data, judge models, video resolution, etc. This benchmark somehow investigated the limitations of existing models in instruction following based on hand gesture, long-term memory, and multi-turn follow-ups.

Critical strengths
  1. Rather than common Video QA tasks, this work focuses on multi-turn interactions between the user and model/assistant. The evaluation addresses the goal (potential intent) of the user with such video, multi-turn context and proactive response.

  2. This work uses fixed interaction paths to ensure the reproducible evaluation of user-model/assistant interactions. The evaluation allows the assistant to return with open-ended responses rather than multiple choices which were frequently adopted by existing benchmarks. This work also includes audio, gesture and handwriting prompts to further enrich the evaluation.

  3. A wide range of models have been evaluated while ablation studies have been conducted, covering multiple dimensions including modalities of input data, judge models, video resolution and etc.

Critical weaknesses

Fixed path evaluation ensures reproducibility. However if the model/assistant would not respond or act as the way of prefixed turns in a fixed path, the evaluation would make no sense. A human-bot setup (using an agent to simulate a user under a specific goal) would be able to fix this issue.

It looks the goal of this work is to evaluate models rather than model+harness, however the evaluation result looks depend on the harness. For example, the benchmark leveraged a FIFO context truncation strategy to compact context in multi-turn evaluation. Though this strategy has been used to evaluate all models in a fair setting. I am not sure if any model would favor this strategy, as the strategy itself is quite straightforward and simple. Furthermore, it looks the evaluation would adjust the frame sampling rate of a video subject to different models.

Even with a single pass inference for LLM-as-a-Judge, the benchmark should score each response from multiple perspectives - just like the angles it defined: accuracy, style, timing, and format compliance, (and run multiple times with rationales in one pass). Then human annotators could screen these evaluation results for better understanding or calibration.

Questions for authors

Please address the three weakness points above.

6 comments

1. Fixed trajectories and interactive validity

We agree that fixed-path evaluation cannot fully assess open-ended adaptation or error recovery. Our goal is to approximate key aspects of online assistant-style interactions through a reproducible offline protocol, rather than replace fully closed-loop evaluation. We preserve spoken user questions, evolving visual observations, and multi-turn dependencies, where models must generate appropriate guidance based on the current task state and interaction history. Since the same goal may be achieved through different valid procedures, predefined priors specify the intended route to reduce ambiguity in evaluating model responses. We have revised the Introduction section to clarify this motivation and the limitations of fixed trajectories, whose scores do not directly measure live task success.

2. Evaluation harness, FIFO truncation, and frame sampling

We thank the reviewer for raising this important concern regarding the influence of the evaluation harness. We agree that context management can affect model performance, especially in long multi-turn interactions. We would like to clarify our FIFO implementation, quantify its association with evaluation scores, and discuss its implications for the benchmark.

1. Clarification of FIFO implementation. Our FIFO strategy does not discard entire historical videos at once. At each turn, the model receives the current audiovisual segment together with the retained interaction history, including previous video segments and model responses. When the input exceeds the supported context budget, we progressively remove approximately 10 seconds of the oldest audiovisual content and recheck the input length. This process continues until the input fits the context budget. If an entire historical video segment is removed, its corresponding assistant response is also discarded. Since user questions are generally located within the final 3–5 seconds of each segment, this strategy preferentially preserves recent user instructions while retaining as much preceding context as possible.

2. Potential loss of task priors. We acknowledge that FIFO can remove important task priors. In our dataset, text-based priors are rendered as images and embedded at the beginning of the initial video, rather than provided as separate text inputs. Consequently, once truncation reaches the beginning of that video, text priors might be influenced. In contrast, video-based priors typically span longer durations and may be partially preserved after truncation. Thus, FIFO does not necessarily remove the complete task prior whenever it is triggered, but it introduces a genuine risk of losing early task information. We agree that errors caused by externally removed information should not be attributed solely to a model's intrinsic memory or reasoning limitations.

3. Quantitative analysis of FIFO and performance.

To examine the potential influence of FIFO on evaluation results, we analyzed all multi-turn QA instances for the three models with accessible FIFO records: MiniCPM-o-4.5, MiMo-V2-Omni, and Qwen3-Omni. We consider a QA to be FIFO-exposed if truncation occurred at the current turn or any preceding turn. On the original 0–5 scoring scale, FIFO-exposed QAs achieved lower average scores than non-exposed QAs: 1.860 vs. 2.563 for MiniCPM-o-4.5, 2.267 vs. 2.790 for MiMo-V2-Omni, and 2.322 vs. 2.778 for Qwen3-Omni.FIFO exposure is closely associated with longer interaction histories and later turns, while questions of later turns might naturally be more difficult. To partially account for differences in task difficulty, we further conducted a fixed-effects regression on 455 common multi-turn QAs (1,365 model–QA observations), controlling for QA-specific and model-specific effects. Importantly, 119 of these QAs exhibited different FIFO exposure states across the three models, providing within-QA comparisons that allow FIFO-related associations to be distinguished from shared QA characteristics. The adjusted coefficient is -0.310 (95% CI: [-0.647, 0.028], p = 0.072), with standard errors clustered by source video. While this analysis does not establish a statistically significant independent association, it cannot fully identify the performance loss potentially shared by models that all experience FIFO on the same QA. Therefore, our results reveal a consistent descriptive association between FIFO exposure and lower scores, but do not establish how much of the degradation is causally attributable to FIFO.

2. Evaluation harness, FIFO truncation, and frame sampling

4. Implications for benchmark validity.

We agree that the evaluation harness, including its context-management strategy, can influence the measured performance. However, handling continuously growing audiovisual histories under finite context budgets is itself a fundamental challenge for real-time multimodal assistants. OmniAssistBench intentionally includes long videos and sustained multi-turn interactions to capture this practical challenge, rather than restricting evaluation to short inputs that always fit within a model's context window.

In our experiments, some interaction histories exceeded the supported context lengths of existing models. To enable evaluation under these constraints, we adopted FIFO as a simple, transparent, and reproducible baseline strategy that progressively removes the oldest audiovisual content while preserving as much recent information as possible. FIFO might not be optimal or equally beneficial for all models. In particular, it may remove task priors required for subsequent responses, thereby affecting evaluation outcomes.

Nevertheless, this limitation does not invalidate the benchmark's objective of evaluating end-to-end interactive performance under finite context budgets. The reported results characterize model performance under a specified context-management protocol, rather than isolating intrinsic memory or reasoning capabilities independently of the inference harness. The limitations exposed by FIFO further highlight the need for more effective long-term context management, such as selective preservation of task-critical information, history summarization, or retrieval-based memory. We believe OmniAssistBench provides a useful evaluation setting for studying such improvements.

We have revised the "Context Management" part in Section 4.1 to explain the implementation of FIFO.

Frame-sampling comparability. Thank you for pointing out this missing detail. For the Gemini family, we upload the complete video files through Google's official Files API without any external frame sampling or preprocessing. According to the official Gemini API documentation, the default video processing samples frames at 1 FPS. For other models, we uniformly sample videos at 1 FPS while preserving the original audio tracks. Therefore, the nominal video sampling rates are consistent across models. However, we acknowledge that internal video encoding and audio processing may differ across model implementations, as we follow their respective official recommended configurations. We have added these details and the corresponding documentation reference to the Evaluated Models section to clarify the sampling settings and evaluation comparability.

Comments on the paper

1 comment

We sincerely thank the Area Chair and all reviewers for your careful evaluation and constructive feedback on our submission, OmniAssistBench (Submission #6602).

We are encouraged that the reviewers recognized several strengths of our work, including:

  • The novel focus on assistant-style, multi-turn interactions beyond conventional video QA (P19q, DHBg);
  • The reproducible evaluation framework based on predefined interaction trajectories and open-ended responses (P19q);
  • The carefully designed scenarios covering diverse interactive capabilities, including audio, visual instructions, and long-term task tracking (P19q, Sjc9, DHBg);
  • The systematic data construction pipeline and substantial annotation efforts (Sjc9, DHBg);
  • The extensive evaluation of diverse omni-modal models and informative ablation studies (P19q, Sjc9, DHBg).

The reviewers also raised several important concerns, which we have carefully addressed in our rebuttal and manuscript revision. We summarize the major topics and our responses below:

  1. Fixed-path evaluation and its applicability to real-world interactions (P19q, Sjc9, DHBg). The reviewers questioned whether predefined trajectories adequately reflect open-ended, real-time interactions, particularly when model guidance diverges from subsequent recorded actions. We revised the Introduction section of the paper to clarify that our benchmark aims to provide a reproducible offline approximation of assistant-style interactions rather than fully closed-loop evaluation, and explicitly discussed its scope and limitations.

  2. Evaluation harness and video processing settings (P19q, Sjc9). The reviewers raised concerns about the influence of FIFO context truncation and differences in video sampling across models. We clarified the FIFO implementation and conducted additional quantitative analyses of its relationship with model performance, while acknowledging the potential influence of context management. We also clarified that Gemini processes uploaded videos through its official API, with a default sampling rate of 1 FPS, consistent with the nominal sampling rate used for other models.

  3. Reliability and validity of LLM-based evaluation (P19q, DHBg). The reviewers requested human validation and a clearer distinction between content quality and presentation-related scoring factors. We conducted an additional human evaluation, obtaining a Pearson correlation of 0.797 between LLM and human scores, comparable to inter-human agreement (0.796). We also introduced content-focused re-evaluation, which preserved the ranking of representative models, and further analyzed Gemini's audio-only versus audiovisual performance.

  4. Robustness of proactive-response evaluation (DHBg). The reviewer questioned whether predefined clip boundaries influence proactive-response scores. We conducted a boundary perturbation experiment covering 133 QA instances, including 96 with modified observation boundaries, and found that the perturbations had negligible influence on average scores, with strong correlation between the original and perturbed results.

  5. Dataset construction, source composition, and video licensing (Sjc9). The reviewer requested additional details about YouTube video selection, the proportion of videos sourced from existing datasets, and copyright considerations. We expanded the description of our video collection and processing pipeline, clarified the dataset composition, and provided additional discussion of video licensing and copyright issues.

  6. Qualitative failure analysis and presentation issues (Sjc9). The reviewer requested representative examples to better understand model limitations. We added qualitative failure cases in Appendix C involving visual instruction following, event-triggered responses, and multi-task tracking, and refined our discussion of historical context limitations. We also corrected the writing errors and inconsistencies identified by the reviewer.

We are grateful for the reviewers' insightful suggestions, which have helped us improve both the methodological clarity and empirical support of our work. In response, we have made substantial revisions throughout the manuscript, including clarifications of the benchmark design and evaluation protocol, additional ablation and validation experiments, expanded qualitative analyses, and more explicit discussions of limitations.

We sincerely thank the Area Chair and reviewers again for their time and consideration. We hope that our responses and revisions adequately address the concerns and help clarify the contributions and significance of OmniAssistBench.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.