- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
In this paper, Omni models are realtime speech-vision interactive models for the user to chat with, over a shared visual scene (e.g. canvas, phone camera). Existing benchmarks use playback of user speech and video regardless of model output. The proposed OmniGym provides a full-duplex evaluation where the simulated user responds to the model output. Although this user reaction is implemented via simple keyword matching, this marks the first step to closed-loop evaluation of Omni models. The authors propose a full pipeline and formulate a DSL to characterize each evaluation case. The DSL defines the reproducible test case for Omni models while being adaptive to the simulation run. Task success is evaluated by a text model. Other metrics include response latency and interruption responsiveness. Using the defined pipeline, the authors construct a benchmark of 200 such cases from some image Q&A data, adding challenges such as side-talk, speech-video offset, pauses, interruptions, and more. Results show that Gemini-3.8-Live is overall stronger. The paper publishes the source code but I have not tested it.
Critical strengths
- The paper's motivation to build closed-loop evaluation for Omni models is innovative, reasonable, and well presented. OmniGym is the first close-loop full-duplex benchmark according to my review. (The field of speech-vision model evaluation is still relatively small, while more related work exists in realtime speech model evaluation.)
- Proposing a DSL to characterize eval cases is elegant and functioning. It offers the right amount of complexity. It gives different models the same inherent task setup while allowing simulation runs to diverge depending on model output and timing. It implements a fair and flexible evaluation.
- The design and experimentation show attention to details from hands-on practitioners (e.g. "reference answers used for scoring are kept separate from these triggers"), strengthening the validity of experiments.
Critical weaknesses
- "Final intent" and "final-intent correctness" are unclear to me. What are their definition? How does OmniGym observe whether the model correctly identified the "final intent"? This point is made significant by the sentence in Conclusion: "Our benchmark results show that models can resolve user intent while failing to satisfy interaction requirements." To make readers understand this insight, the authors must clarify "final intent".
- Needs a demo page to include examples from the 200 benchmark cases. Attach the models' performance in video and in metrics. The current issues are:
- Missing quantitative presentation of the video-audio degradation.
- Missing illustration of failure cases. How/Why did some models fail?
- Sporadic issues with citation format, although without breach of anonymization. For examples,
- L49 needs to move the parenthesis. More cases exist.
- L15 shouldn't cite the current paper.
- The proposed DSL is limited to using keyword matching to implement the simulated user. This still has a significant gap from a general full-duplex evaluation. This point is already mentioned in the paper's Limitation section. Despite this limitation, as the first proposal to full-duplex evaluation of Omni models, OmnyGym is still innovative and valuable.
Questions for authors
- What is "final intent"?
- Will the authors fix the citation format according to https://iclr.cc/Conferences/2026/AuthorGuide ?
- Benchmark construction: what is the source of the image QA data? Are agents used during the construction? L441 in Section 3 mentioned an agent but Section 2 needs a proper definition.
- Will the authors present related work in speech-only full-duplex closed-loop evaluation methods? I saw many powered by other realtime agents.
- L40-43 sentence shuffled. What is the originally intended text?
Minor points:
- "Since severity groups contain different videos, their differences remain descriptive rather than isolated effects of interference intensity" is problematic. If the data size is large and selection of severity is independently sampled w.r.t. video content, then this claim is incorrect, or needs to be specifically clarified.
- L46 Consider changing "Such a setting" to "An ideal setting" for clarity.
- L195 Consider mentioning each of the three symbols in text, to establish correspondence.
Lastly,
- How do the authors envision high-quality large-scale multi-round benchmark construction with user reaction beyond simple error clarification?
5 comments
Q2, Q4, Q5. Related Work, Presentation Issues
We will correct the citation formatting, revise the unclear sentences (including L40–43), and clarify the notation in the revised paper. We will expand the related work to include speech-only closed-loop benchmarks, such as Full-Duplex-Bench-v2 [1] and τ-Voice [2]. The fundamental distinction is that OmniGym moves beyond conversational evaluation toward system-level closed-loop evaluation, capturing the coupled dynamics of user behavior, model responses, and environmental conditions, rather than evaluating dialogue interactions in isolation. Regarding the shuffled sentence in L40–43, the intended meaning is that recent omni models enable real-time multimodal applications, such as visual assistance, translation, and meeting agents. The confusion mainly stems from misplaced citations and formatting issues, which we will correct in the revised paper.
Reference
[1] Guan-Ting Lin, Shih-Yun Shan Kuan, Jiatong Shi, Kai-Wei Chang, Siddhant Arora, Shinji Watanabe and Hung-yi Lee. Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examiner. arXiv:2510.07838, 2026. https://arxiv.org/abs/2510.07838
[2] Soham Ray, Keshav Dhandhania, Victor Barres, and Karthik Narasimhan. τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains. arXiv:2603.13686, 2026. https://arxiv.org/abs/2603.13686
Minor Comments
Minor comment (1). To address the concerns regarding isolating the effect of interference intensity, we conduct new experiments expanding the original 200 scenarios across all three distractor levels (Light, Medium, and Heavy), increasing our benchmark to 600 test cases (3×). Furthermore, we perform three repeated runs for each test case to assess result consistency. Please refer to General Response 2 for detailed experimental results.
Minor comment (2) (3). We thank the reviewer for these constructive suggestions regarding wording clarity and symbol definitions, and we will address them carefully in the revised paper.
Minor comment (4). To construct a high-quality, large-scale multi-round benchmark with user reactions beyond simple error clarification, we envision combining our controllable synthesis pipeline with real-world trace mining. OmniGym currently synthesizes interaction traces conditioned on background videos, question-answer pairs, and target distractors. While this ensures controllability and diversity, synthetic interactions may not fully capture real-world conversational patterns. Future work could mine real human-model interaction traces to guide scenario generation, enabling more realistic combinations of interruptions, intent changes, and distractors while preserving controllability and reproducibility. Building on these mined patterns, future multi-round benchmarks could model diverse user reactions to ensure high benchmark quality at scale.
W1 Q1: I still do not fully understand the split of desiderata here, and consequently one of your main results. Is there related work that uses this evaluation perspective? If not, can you explain more about intent handling VS task completion, as well as how they are detected by OmniGym?
W2 OK.
Q3 is well addressed. Will your answer be added to Section 2?
Finally, I notice the manuscript has not been updated since your rebuttal answer on Nov 13. Please upload your revision soon, because the presentation issues are serious. It will be the reviewer's responsibility to check it again before deciding to maintain the original rating of 4.
Many thanks to Reviewer RAte for providing a detailed review and insightful suggestions to improve the clarity of our paper.
W1, Q1. Clarification of Final Intent and other metrics
Please kindly refer to General Response 1, where we provide detailed definitions and examples to clarify the evaluation metrics, including "final-intent correctness," which we will rename "Intent Handling" in the revised paper for greater clarity.
The "Intent Handling" refers to the user's active request after any corrections, clarifications, or changes in intent, rather than simply the last utterance. OmniGym does not directly observe the model's internal intent recognition; instead, it evaluates whether the recorded response satisfies the predefined task-specific criteria, including justified clarification or uncertainty. This distinction explains our conclusion: a model may correctly address the user's intent while still violating interaction requirements, such as interruption handling or ignoring third-party speech. We will clarify this distinction in the revised paper.
W2. Quantitative Degradation and Failure Analysis
Quantitative audio and visual degradation. We use the following parameters for matched tasks at three severity levels.
Table 5. Degradation configurations across severity levels.
Each row applies to its corresponding degradation mechanism. Audio interference includes competing speech, rain, fan noise, pink noise, and brown noise. Video sampling rates specify the frames supplied to the model.
Failure illustration. We will provide a demo page in the appendix with representative benchmark cases. For example, in a Qwen3.8 side-talk recovery case, the model correctly answers both visual questions and also audibly says "You're welcome" in response to unrelated third-party speech. The case passes intent handling, while failing interaction correctness and joint task success because the task requires ignoring that side conversation. This example exposes a weakness in multi-party interaction: the model may respond to unrelated third-party speech.
W3, Q3. Benchmark Construction and Validation
Our visual material is chosen from StreamingBench. Their original audio is removed, and user and distractor speech is constructed separately. We write conversational questions for each OmniGym scenario, using the original StreamingBench QA as a reference where relevant. Each DSL is associated with expected responses in the source video and the interaction requirements. The corresponding video content provides evidence for evaluation.
We use GPT-6-Astra for AI-assisted scenario construction. Based on source-video evidence and distractor-specific rules, it drafts and revises conversational questions, follow-up branches, expected responses, and scoring criteria. AI-assisted review checks natural English phrasing and consistency between user intent and expected outcomes. Sampled human listening informs revisions to synthesized speech, including hesitation cues and pauses. Automated checks verify DSL schema validity, event references, and acyclic dependencies. All scenarios are additionally reviewed manually for naturalness, interaction logic, and oracle correctness.
W4. DSL Generality and Future Extensions
Our current implementation uses predefined keyword-based guards to ensure reproducible interactions. However, the DSL is not inherently limited to keyword matching; it supports response-dependent events and programmable interaction policies. We acknowledge the limitations of lexical matching and envision extending OmniGym with semantic triggering and stateful user agents to support richer multi-round interactions, including goal changes, clarifications, corrections, and context recovery.