acceptodds
Under review as a conference paper at ICLR 2027

OmniGym: A Closed-Loop Evaluation Framework for Full-Duplex Omni Models

Abstract

Real-time omni models support continuous audio-visual interaction, where users can interrupt, revise requests, and resume earlier tasks. Evaluating these capabilities requires more than replaying predefined input trajectories: full-duplex models should be exercised inside evolving interactions, where simulated user behavior adapts to model responses while perceptual and runtime conditions change over time. We introduce OmniGym, a programmable framework for closed-loop evaluation of full-duplex omni models. OmniGym represents each test case as an executable interaction episode, using an event-driven user simulator and a controllable environment emulator to realize adaptive user behavior and environment shifts along a shared timeline. A timeline DSL captures the event dependencies and evaluation objectives of each episode, allowing the same interaction logic to be reproducibly executed across models and conditions. Using OmniGym, we construct 200 scenarios spanning seven categories of interaction and perceptual distractors and their combinations, and evaluate four systems from the Qwen, Gemini, and GLM families. Our evaluation reveals substantial variation across distractor types and severity groups: different models excel under different interaction and perceptual conditions, while exhibiting markedly different trade-offs among semantic correctness, interaction behavior, and response and interruption latency. These results show that full-duplex capability cannot be characterized by a single metric or fixed interaction trajectory, motivating closed-loop evaluation that exercises how models behave as users and environments evolve.

Primary area:
datasets and benchmarks
Keywords:
Omni Model, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_RAteOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, Omni models are realtime speech-vision interactive models for the user to chat with, over a shared visual scene (e.g. canvas, phone camera). Existing benchmarks use playback of user speech and video regardless of model output. The proposed OmniGym provides a full-duplex evaluation where the simulated user responds to the model output. Although this user reaction is implemented via simple keyword matching, this marks the first step to closed-loop evaluation of Omni models. The authors propose a full pipeline and formulate a DSL to characterize each evaluation case. The DSL defines the reproducible test case for Omni models while being adaptive to the simulation run. Task success is evaluated by a text model. Other metrics include response latency and interruption responsiveness. Using the defined pipeline, the authors construct a benchmark of 200 such cases from some image Q&A data, adding challenges such as side-talk, speech-video offset, pauses, interruptions, and more. Results show that Gemini-3.8-Live is overall stronger. The paper publishes the source code but I have not tested it.

Critical strengths
  • The paper's motivation to build closed-loop evaluation for Omni models is innovative, reasonable, and well presented. OmniGym is the first close-loop full-duplex benchmark according to my review. (The field of speech-vision model evaluation is still relatively small, while more related work exists in realtime speech model evaluation.)
  • Proposing a DSL to characterize eval cases is elegant and functioning. It offers the right amount of complexity. It gives different models the same inherent task setup while allowing simulation runs to diverge depending on model output and timing. It implements a fair and flexible evaluation.
  • The design and experimentation show attention to details from hands-on practitioners (e.g. "reference answers used for scoring are kept separate from these triggers"), strengthening the validity of experiments.
Critical weaknesses
  • "Final intent" and "final-intent correctness" are unclear to me. What are their definition? How does OmniGym observe whether the model correctly identified the "final intent"? This point is made significant by the sentence in Conclusion: "Our benchmark results show that models can resolve user intent while failing to satisfy interaction requirements." To make readers understand this insight, the authors must clarify "final intent".
  • Needs a demo page to include examples from the 200 benchmark cases. Attach the models' performance in video and in metrics. The current issues are:
    • Missing quantitative presentation of the video-audio degradation.
    • Missing illustration of failure cases. How/Why did some models fail?
  • Sporadic issues with citation format, although without breach of anonymization. For examples,
    • L49 needs to move the parenthesis. More cases exist.
    • L15 shouldn't cite the current paper.
  • The proposed DSL is limited to using keyword matching to implement the simulated user. This still has a significant gap from a general full-duplex evaluation. This point is already mentioned in the paper's Limitation section. Despite this limitation, as the first proposal to full-duplex evaluation of Omni models, OmnyGym is still innovative and valuable.
Questions for authors
  • What is "final intent"?
  • Will the authors fix the citation format according to https://iclr.cc/Conferences/2026/AuthorGuide ?
  • Benchmark construction: what is the source of the image QA data? Are agents used during the construction? L441 in Section 3 mentioned an agent but Section 2 needs a proper definition.
  • Will the authors present related work in speech-only full-duplex closed-loop evaluation methods? I saw many powered by other realtime agents.
  • L40-43 sentence shuffled. What is the originally intended text?

Minor points:

  • "Since severity groups contain different videos, their differences remain descriptive rather than isolated effects of interference intensity" is problematic. If the data size is large and selection of severity is independently sampled w.r.t. video content, then this claim is incorrect, or needs to be specifically clarified.
  • L46 Consider changing "Such a setting" to "An ideal setting" for clarity.
  • L195 Consider mentioning each of the three symbols in text, to establish correspondence.

Lastly,

  • How do the authors envision high-quality large-scale multi-round benchmark construction with user reaction beyond simple error clarification?
5 comments

Many thanks to Reviewer RAte for providing a detailed review and insightful suggestions to improve the clarity of our paper.

W1, Q1. Clarification of Final Intent and other metrics

Please kindly refer to General Response 1, where we provide detailed definitions and examples to clarify the evaluation metrics, including "final-intent correctness," which we will rename "Intent Handling" in the revised paper for greater clarity.

The "Intent Handling" refers to the user's active request after any corrections, clarifications, or changes in intent, rather than simply the last utterance. OmniGym does not directly observe the model's internal intent recognition; instead, it evaluates whether the recorded response satisfies the predefined task-specific criteria, including justified clarification or uncertainty. This distinction explains our conclusion: a model may correctly address the user's intent while still violating interaction requirements, such as interruption handling or ignoring third-party speech. We will clarify this distinction in the revised paper.

W2. Quantitative Degradation and Failure Analysis

Quantitative audio and visual degradation. We use the following parameters for matched tasks at three severity levels.

Table 5. Degradation configurations across severity levels.

Degradation parameterLightMediumHigh
Noise / competing-speech SNR (dB)20124
Target audio bitrate (kbps)16106
Video width (pixels)640320160
Video sampling rate (fps)10.50.2
Local blur radius (pixels)31024
Pixelation block-size parameter (pixels)82464

Each row applies to its corresponding degradation mechanism. Audio interference includes competing speech, rain, fan noise, pink noise, and brown noise. Video sampling rates specify the frames supplied to the model.

Failure illustration. We will provide a demo page in the appendix with representative benchmark cases. For example, in a Qwen3.8 side-talk recovery case, the model correctly answers both visual questions and also audibly says "You're welcome" in response to unrelated third-party speech. The case passes intent handling, while failing interaction correctness and joint task success because the task requires ignoring that side conversation. This example exposes a weakness in multi-party interaction: the model may respond to unrelated third-party speech.

W3, Q3. Benchmark Construction and Validation

Our visual material is chosen from StreamingBench. Their original audio is removed, and user and distractor speech is constructed separately. We write conversational questions for each OmniGym scenario, using the original StreamingBench QA as a reference where relevant. Each DSL is associated with expected responses in the source video and the interaction requirements. The corresponding video content provides evidence for evaluation.

We use GPT-6-Astra for AI-assisted scenario construction. Based on source-video evidence and distractor-specific rules, it drafts and revises conversational questions, follow-up branches, expected responses, and scoring criteria. AI-assisted review checks natural English phrasing and consistency between user intent and expected outcomes. Sampled human listening informs revisions to synthesized speech, including hesitation cues and pauses. Automated checks verify DSL schema validity, event references, and acyclic dependencies. All scenarios are additionally reviewed manually for naturalness, interaction logic, and oracle correctness.

W4. DSL Generality and Future Extensions

Our current implementation uses predefined keyword-based guards to ensure reproducible interactions. However, the DSL is not inherently limited to keyword matching; it supports response-dependent events and programmable interaction policies. We acknowledge the limitations of lexical matching and envision extending OmniGym with semantic triggering and stateful user agents to support richer multi-round interactions, including goal changes, clarifications, corrections, and context recovery.

Q2, Q4, Q5. Related Work, Presentation Issues

We will correct the citation formatting, revise the unclear sentences (including L40–43), and clarify the notation in the revised paper. We will expand the related work to include speech-only closed-loop benchmarks, such as Full-Duplex-Bench-v2 [1] and τ-Voice [2]. The fundamental distinction is that OmniGym moves beyond conversational evaluation toward system-level closed-loop evaluation, capturing the coupled dynamics of user behavior, model responses, and environmental conditions, rather than evaluating dialogue interactions in isolation. Regarding the shuffled sentence in L40–43, the intended meaning is that recent omni models enable real-time multimodal applications, such as visual assistance, translation, and meeting agents. The confusion mainly stems from misplaced citations and formatting issues, which we will correct in the revised paper.

Reference

[1] Guan-Ting Lin, Shih-Yun Shan Kuan, Jiatong Shi, Kai-Wei Chang, Siddhant Arora, Shinji Watanabe and Hung-yi Lee. Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examiner. arXiv:2510.07838, 2026. https://arxiv.org/abs/2510.07838

[2] Soham Ray, Keshav Dhandhania, Victor Barres, and Karthik Narasimhan. τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains. arXiv:2603.13686, 2026. https://arxiv.org/abs/2603.13686

Minor Comments

Minor comment (1). To address the concerns regarding isolating the effect of interference intensity, we conduct new experiments expanding the original 200 scenarios across all three distractor levels (Light, Medium, and Heavy), increasing our benchmark to 600 test cases (3×). Furthermore, we perform three repeated runs for each test case to assess result consistency. Please refer to General Response 2 for detailed experimental results.

Minor comment (2) (3). We thank the reviewer for these constructive suggestions regarding wording clarity and symbol definitions, and we will address them carefully in the revised paper.

Minor comment (4). To construct a high-quality, large-scale multi-round benchmark with user reactions beyond simple error clarification, we envision combining our controllable synthesis pipeline with real-world trace mining. OmniGym currently synthesizes interaction traces conditioned on background videos, question-answer pairs, and target distractors. While this ensures controllability and diversity, synthetic interactions may not fully capture real-world conversational patterns. Future work could mine real human-model interaction traces to guide scenario generation, enabling more realistic combinations of interruptions, intent changes, and distractors while preserving controllability and reproducibility. Building on these mined patterns, future multi-round benchmarks could model diverse user reactions to ensure high benchmark quality at scale.

W1 Q1: I still do not fully understand the split of desiderata here, and consequently one of your main results. Is there related work that uses this evaluation perspective? If not, can you explain more about intent handling VS task completion, as well as how they are detected by OmniGym?

W2 OK.

Q3 is well addressed. Will your answer be added to Section 2?

Finally, I notice the manuscript has not been updated since your rebuttal answer on Nov 13. Please upload your revision soon, because the presentation issues are serious. It will be the reviewer's responsibility to check it again before deciding to maintain the original rating of 4.

Reviewer_Z2gBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OmniGym is a programmable and closed-loop evaluation framework for real-time full-duplex multimodal models (e.g., Gemini-3.8-Live). The authors construct a benchmark of 200 scenarios covering seven types of interaction and perceptual distractors, and evaluate four models.

Critical strengths

It's true that evaluating real-time multimodal interaction requires more than conventional question-answering benchmarks, particularly for interruptions and the changing user intentions.

The proposed framework is reasonably comprehensive for different conditions. The experimental results reveal interesting (but not very convincing) differences among current full-duplex models.

Critical weaknesses

Most importantly, the claimed advantage of closed-loop evaluation is not demonstrated. The main contribution is a programmable framework for adaptive interaction, yet the experiments only compare different models within OmniGym. No comparison with conventional fixed-trajectory evaluation is provided. Therefore, it remains unclear whether the proposed closed-loop mechanism reveals additional model failures or provides more meaningful evaluation results than existing approaches.

Also, the light, medium, and heavy distractor groups contain different videos/scenarios rather than different perturbation levels applied to identical cases. Consequently, performance differences cannot be attributed to distractor severity. This is particularly problematic given the non-monotonic results in Table 1, where some models perform better under heavier interference, which the authors acknowledged.

The benchmark scenarios and user interaction traces are synthetically generated, without reported validation against real human validations. Therefore, the realism of the constructed interactions and reliability of the evaluation remain insufficiently established.

Questions for authors

Please provide some insight on how to make the benchmark really usable and grounded in more realistic settings.

4 comments

Many thanks to Reviewer Z2gB for providing a detailed review and insightful suggestions to improve the clarity of our paper.

W1. Necessity and Advantages of Closed-loop Evaluation

Please refer to the General Response 3, where we clarify the motivation for closed-loop evaluation and provide an empirical comparison across four models. Under fixed replay, interaction timing alignment drops to 52.22%, 27.78%, and 2.22% for Qwen3.8, Gemini, and GLM, respectively, compared with 100% under closed-loop execution. These results demonstrate the importance of adapting user interactions to actual model behavior rather than relying on predetermined trajectories.

W2. Benchmark expansion and Distractor Severity Levels

Please kindly refer to General Response 1, where we provide further details on our expanded matched-scenario evaluation. To address the original severity confounding, we expanded the benchmark from 200 to 600 test cases (3×), with each original scenario evaluated at all three distractor severity levels (Light, Medium, and Heavy). All three severity variants share the same source video and underlying task, differing only in distractor severity, thereby eliminating source-video differences as a confounding factor.

Under these matched conditions, overall joint task success decreases consistently from light to heavy across all four models:

Table 4. Overall joint task success (%) across severity levels.

ModelLightMediumHeavy
Qwen3.5-Omni63.6756.3337.17
Qwen3.8-Omni6954.8347.33
Gemini3.8-Live73.569.8356.67
GLM-Realtime32.6721.6720.67

At the category level, performance also generally trends downward as severity increases, though a few minor local reversals remain. For example, in visual degradation, scores decrease consistently except on Gemini3.8-Live, where Heavy is slightly higher than Medium (78.57% vs. 77.38%); this occurs because stronger degradation shifts the model in some cases from making an incorrect guess to explicitly acknowledging unclear visual evidence, which satisfies intent handling and slightly raises joint success. Meanwhile, in categories such as audio degradation, background noise inherently has a relatively weak impact on outcomes in multimodal scenarios, resulting in largely flat performance across severity levels with minor fluctuations. Overall, controlling scenario composition establishes clear aggregate downward trends with severity and enables more meaningful within-task comparisons. Please refer to General Response 2 for detailed experimental results.

W3. Scenario Realism and Evaluation Reliability

We appreciate the reviewer's concerns regarding scenario realism and evaluation reliability, which we address in detail in General Response 1. Specifically, we manually reviewed all 200 original benchmark scenarios for interaction naturalness, logical consistency, and oracle correctness, finding no issues. We further inspected 240 recorded interaction traces across models and distractor categories, observing 99.2% agreement between human judgments and AI-assisted task-success scores, with minor discrepancies limited to borderline cases involving justified uncertainty under degraded perceptual conditions.

GPT-6-Astra evaluates intent handling and semantic interaction requirements against predefined criteria using visual evidence, transcripts, and recorded responses, combined with timing checks to determine task success. We will clarify these procedures in the revised paper and explore incorporating real human–model interaction traces to further improve scenario realism.

Q. Real-world Grounding and Practical Utility of OmniGym

We envision two promising directions for OmniGym.

(1) Real-world trace mining and augmentation. OmniGym currently synthesizes interaction traces conditioned on background videos, question–answer pairs, and target distractors. While this ensures controllability and diversity, synthetic interactions may not fully capture real-world conversational patterns. Future work could mine interaction patterns from real human–model interaction traces to guide scenario generation, enabling more realistic combinations of interruptions, intent changes, and distractors while preserving controllability and reproducibility.

(2) RL training for full-duplex omni models. Recent work such as SpeechGym [1] demonstrates the potential of training voice agents through simulated multi-turn interactions. Beyond evaluation, OmniGym could potentially serve as an interactive RL environment, leveraging its event-driven user simulator, controllable environment emulator, and fine-grained reward scoring to provide feedback on task success, interruption handling, turn-taking, recovery, and latency. This would enable omni models to optimize their behavior through reproducible live interactions rather than static input–output examples.

Reference

[1] Fan, J., et al. SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning. arXiv:2608.26432, 2026. https://arxiv.org/abs/2608.26432

Thank you for the detailed rebuttal and additional experiments. The comparison with fixed-replay evaluation and the expanded matched-scenario severity analysis help address my main concerns. I appreciate the authors' efforts and have no further comments.

Thank you again for your time and constructive feedback. We have uploaded our revised manuscript for your reference.

Reviewer_SF1zOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents OmniGym, a framework for closed-loop evaluation of full-duplex omni models. Each test case is an executable episode written in a timeline DSL. It specifies user intent, user and environment events (some triggered by the model’s responses), perceptual conditions, an outcome oracle and metrics. An offline compiler freezes the media assets and builds an event DAG. A runtime user simulator and an environment emulator then run the episode against a live model with time-aligned telemetry. The authors build 200 scenarios across seven distractor types (four interaction, three perceptual, plus mixed) and three severity tiers. They evaluate four hosted systems (Qwen3-Omni-Flash-Realtime, Qwen3.8-Omni-Flash-Realtime, Gemini-3.8-Live, GLM-Realtime). They report that joint task success (Gemini 73.5%, Qwen3 62.0%, Qwen3.8 59.0%, GLM 37.0%) falls below final-intent correctness, that robustness varies by distractor type, and that response latency does not predict interruption latency.

Critical strengths
  • The runtime design is careful. It separates source time, planned delivery, actual send, receipt and simulated playback time, and it measures interruption at audible playback. Fixed seeds and compile-time rejection of cycles support reproducible scheduling.
  • The distractor taxonomy is sensible, and the mixed scenarios probe compounding effects.
  • The reported numbers are internally consistent. I re-derived each overall success rate from Table 1 and the Figure 5 cell sizes, and they match.
  • Some observations are interesting: interruption and response latency are dissociated, and three of four systems solve 0 of 15 side-talk scenarios.
  • The authors admit that severity tiers use different videos and that interruption numbers reflect the client.
Critical weaknesses
  • The primary metric is unvalidated. Task success comes from “AI-assisted review”. The paper gives no judge model or prompt and no human agreement. Scenarios are LLM-generated and checked only for schema validity, resolvable references and acyclicity, with no check of naturalness or oracle correctness and no human baseline.
  • The central claim is untested. The paper argues that closed-loop evaluation is necessary, but there is no open-loop replay of the same scenarios to show that rankings or failure modes differ. The “adaptive” user is a first-match decision tree on text cues, and when no guard fires, no follow-up occurs. A model that clarifies in different words can therefore fail because of a guard miss. The miss rate is not reported.
  • The sample is too small for the claims. Per-category cells hold 3–10 scenarios, so one scenario moves a cell by 10–33 points. There are no repeats, confidence intervals or paired tests. Several comparisons are single-scenario differences: heavy referential inconsistency (7/9 vs 6/9), medium visual degradation (8/9 vs 7/9) and heavy visual degradation (9/9 vs 8/9). The Qwen3 vs Qwen3.8 overall gap is 6 of 200 scenarios (z ≈ 0.6). Severity tiers use different videos, so severity effects are confounded with content. Some non-monotonicity, such as Qwen3 on visual degradation (70 → 89 → 100%), is attributed to rubric-permitted uncertainty answers, which points to oracle design.
  • The metric is partly tautological. Joint task success includes final-intent correctness, so it must be lower by construction. The size of the gap depends on how strict the interaction rubric is. “Neutral passes” inflate interaction correctness. The 0/15 side-talk result needs failure analysis, because it could be a floor effect or a harness strictness issue.
Questions for authors

Which model and prompt produce the task-success judgment, and what is the agreement with human labels on a stratified sample? How do model rankings and failure types change if the same scenarios are replayed open-loop? How often does a response-triggered guard fail to fire, and how many failures are attributable to this?

3 comments

Many thanks to Reviewer SF1z for providing a detailed review and insightful suggestions to improve the clarity of our paper.

W1, Q1. Scenario Validity and Judge Model

We appreciate the reviewer's concerns regarding scenario validity and evaluation reliability. As detailed in General Response 1, we manually reviewed all 200 original scenarios for naturalness, logical consistency, and oracle correctness, finding no issues. We also manually evaluated 240 interaction traces, achieving 99.2% agreement with AI-assisted task-success judgments. We clarify the evaluation metrics and will provide the judge model (GPT-6-Astra), evaluation prompt, and scoring criteria in the revised appendix. We acknowledge that further validation with real human–model interactions remains an important future direction.

W2, Q2, Q3. Necessity of Closed-loop Evaluation and Guard Reliability

(1) Closed-loop necessity and comparison. Please refer to the General Response 3, where we explain the coupled dynamics of user behavior, model responses, and environmental conditions, and provide an closed-loop evaluation v.s. open loop replay comparison demonstrating substantial differences in interaction timing alignment.

(2) Guard reliability. Our predefined decision-tree design ensures reproducibility while supporting response-dependent interactions. Guard matching controls subsequent user actions but does not directly determine task-success scores. We manually reviewed 108 recorded branch decisions and found no false triggers. Future work could replace keyword matching with lightweight, low-latency semantic recognition to support richer interactions while preserving reproducibility and real-time responsiveness.

W3. Benchmark expanding and Statistical Reliability

(1) Sample size and statistical reliability. Please refer to General Response 2, where we address concerns about sample size and severity confounding. We expanded the benchmark from 200 to 600 test cases (3×) across three severity levels (Light, Medium, and Heavy), and conducted three repeated runs per case. For cases with determinate judgments across all runs, joint task success varied by at most 3.83 percentage points within each model, indicating generally stable results.

(2) Severity confounding. In the original paper, the success rate does not decrease as visual degradation increase because Light, Medium, and Heavy corresponded to different scenarios, so the results are affected by the inherent differences among the scenarios themselves. In the expanded experiment, each parent contributes all three severity variants using the same source video and final task goal. Please refer to General Response 2 for detailed experimental results.

W4. Failure Analysis

(1) Metric calculation and neutral passes. Please kindly refer to General Response 1, where we clarify the definitions and calculation of our evaluation metrics. Some perceptual-degradation scenarios do not involve specific interaction behaviors, such as interruptions or side-talk, and therefore receive neutral passes for those requirements. However, joint task success requires both correct intent handling and interaction correctness, ensuring that a neutral interaction pass alone does not count as overall success.

(2) Side-talk recovery.

To better analyze the causes of side-talk failures, we report the side-talk recovery results across three evaluation metrics in Table 3.

Table 3. Side-talk recovery outcomes

ModelJoint task successIntent handlingInteraction correctness
Qwen3.5-Omni0.0067.412.22
Qwen3.8-Omni0.0071.850.00
Gemini3.8-Live13.3380.7414.81
GLM-Realtime0.0027.410.00

For Qwen3.5-Omni, Qwen3.8-Omni, and GLM-Realtime, side-talk success remains 0/45 after expansion, while Gemini3.8-Live achieves 13.33%. Failure analysis shows that models often respond to unrelated third-party speech despite correctly answering the visual questions. For example, one model answers both questions correctly but responds "You're welcome" to an unrelated speaker. This reflects a limitation of systems primarily designed for one-to-one human–AI interaction. In particular, Qwen employs a cascaded architecture combining semantic VAD and Qwen3.8-Omni, rather than native full-duplex interaction. While semantic VAD detects speech and turn boundaries, it struggles to determine whether an utterance is directed at the assistant. These findings highlight addressee recognition as a key challenge in multi-party real-time interaction.

Thank you again for your time and constructive feedback. We have uploaded our revised manuscript for your reference.

thanks for your response, most of my concerns are addressed, I have raised my score to 3.

Comments on the paper

4 comments

General Response 3. Necessity of Closed-loop Evaluation

(1) Closed-loop necessity. Our goal is to extend fixed trajectory evaluation with adaptive interactions and composable distractors. Real-world interactions involve coupled dynamics among model responses, user behavior, and environmental conditions. As illustrated by Artic [1], environmental fluctuations can affect model responses and, consequently, reshape user–model interaction dynamics, while stochastic model outputs further introduce variability in both content and timing. OmniGym combines a User Simulator and an Environment Emulator to adapt interactions to actual model behavior and expose diverse failure modes within the same task.

(2) Closed-loop vs. open-loop replay comparison. We compare open-loop replay and closed-loop execution on the same scenarios across four models. We define interaction timing alignment rate as the proportion of runs in which simulated user actions satisfy timing requirements relative to actual model responses, such as interrupting during model speech or issuing follow-ups after response completion. Specifically, we use the recorded trajectories from Qwen3.5 as the open-loop replay reference. Under open-loop replay, alignment rates are 52.22%, 27.78%, and 2.22% for Qwen3.8, Gemini, and GLM, respectively, compared with 100% under closed-loop execution. These results show that open-loop replay may misalign intended interactions with actual model behavior, and it cannot adapt to diverse model, user, and environment dynamics.

Reference

[1] Wu, Jiangkai, et al. "Artic: AI-oriented Real-time Communication for MLLM Video Assistant." Proceedings of the ACM SIGCOMM 2026 Conference. 2026.

General Response 2. Benchmark expansion and statistical reliability

(1) Benchmark expansion. To address the reviewers' concerns regarding sample size and isolating the effect of interference intensity, we conducted new experiments expanding the original 200 scenarios across three distractor levels (Light, Medium, and Heavy), increasing our benchmark to 600 test cases (3×).

(2) Statistical reliability.

Furthermore, to address the concerns on statistical reliability, we performed three repeated runs for each test case to assess result consistency. Across all three runs, the joint task success rates for Qwen3.5-Omni, Qwen3.8-Omni, Gemini3.8-Live, and GLM-Realtime are 52.17%–52.67%, 55.83%–58.83%, 66.33%–67.17%, and 23.00%–26.83%, respectively. These results remain stable across runs with no overlap between the score ranges of different models, confirming reliable relative rankings.

(3) Severity confounding.

In the original manuscript, the success rate does not decrease as visual degradation increase because Light, Medium, and Heavy corresponded to different scenarios, so the results are affected by the inherent differences among the scenarios themselves.

In the expanded experiment, each parent contributes all three severity variants using the same source video and final task goal. Table 2 reports joint task success by distractor category, model, and severity. Each category contains the same parent tasks at every severity.

Table 2. Joint task success by distractor category and severity

Distractor categoryQwen3.5-Omni LQwen3.5-Omni MQwen3.5-Omni HQwen3.8-Omni LQwen3.8-Omni MQwen3.8-Omni HGemini3.8-Live LGemini3.8-Live MGemini3.8-Live HGLM-Realtime LGLM-Realtime MGLM-Realtime H
Turn-taking ambiguity86.2183.914.6090.8089.6647.1380.4668.9710.3419.546.902.30
Intent evolution76.9235.908.9788.4614.1010.2679.4989.7478.2143.590.000.00
Referential inconsistency16.0516.059.8823.4619.7518.5264.2060.4955.568.643.704.94
Side-talk recovery0.000.000.000.000.000.0017.7817.784.440.000.000.00
Audio degradation86.2183.9183.9187.3691.9591.9586.2183.9180.4647.1349.4347.13
Visual degradation86.9080.9566.6794.0577.3873.8188.1077.3878.5754.7635.7133.33
Temporal degradation83.9186.2181.6188.5186.2185.0691.9588.5187.3652.8755.1756.32
Mixed25.4915.697.8429.417.847.8439.2233.3321.579.800.000.00
Overall63.6756.3337.1769.0054.8347.3373.5069.8356.6732.6721.6720.67

With task composition held constant across severity levels, overall joint success, averaged over three runs, decreases monotonically from Light to Heavy for every model: 63.67% to 37.17% for Qwen3.5-Omni, 69.00% to 47.33% for Qwen3.8-Omni, 73.50% to 56.67% for Gemini3.8-Live, and 32.67% to 20.67% for GLM-Realtime. At the category level, performance also generally decreases with severity, most notably in turn-taking ambiguity, visual degradation, and mixed distractors. We observe that a few minor local reversals remain. For instance, in visual degradation, Gemini3.8-Live scores slightly higher on Heavy than Medium (78.57% vs. 77.38%) because heavier degradation shifts some incorrect guesses toward justified uncertainty that satisfies intent handling, whereas categories like audio degradation remain largely flat due to the weaker impact of noise in multimodal scenarios. We will include these expanded results and analyses in the revised paper.

AuthorsOpenReview ↗

General Response

We thank all the reviewers for the constructive feedback. This post intends to address the common concerns regarding the submission for all the reviewers.

General Response 1. Scenario validity and judge model and evaluation metrics

(1) Scenario validity.

We manually reviewed all original benchmark scenarios (200/200) for interaction naturalness, logical consistency, and oracle correctness, finding no issues. We further inspected approximately 10% (240/2400) of recorded interaction traces across models and distractor categories, observing 99.2% agreement between human judgments and AI-assisted task-success scores. The few disagreements occurred in borderline visual-degradation cases where human reviewers found the content recognizable and rated Intent Handling as false, while the judge model rated it as true by accepting the model's claim of being unable to see clearly as justified uncertainty.

(2) Judge model.

In our current AI-assisted evaluation, GPT-6-Astra assesses intent handling and semantic interaction requirements against task-specific criteria, using visual evidence, transcripts, and responses retained in playback. These judgments are combined with recorded timing checks to determine interaction correctness and joint task success. Each judgment includes supporting evidence and a rationale. We will document these validation procedures and provide the judge model version, evaluation prompt, and scoring criteria in the appendix of revised paper.

(3) Evaluation metrics.

For clarity, we will rename final-intent to Intent Handling in the revised paper. The revised evaluation metrics are detailed as follows:

Intent Handling (final-intent correctness): measures whether the response satisfies the user's final active request and its constraints, including justified clarification or uncertainty permitted by the task criterion. For example, when an object is visually unrecognizable and the model honestly states, "I can't see it clearly," it passes Intent Handling through justified uncertainty, allowing the user to improve the evidence (e.g., by adjusting the camera angle).

Interaction Correctness: measures whether the model responds at the appropriate time and to the appropriate speaker, including respecting turn boundaries, following instructions to wait, handling interruptions correctly, and avoiding responses to third-party side-talk.

Joint Task Success: requires both correct Intent Handling and Interaction Correctness.

Table 1 shows the metric results under distractors. As Qwen3-Omni from the original paper has been retired from Alibaba Cloud Model Studio and its API is no longer accessible, we replace it with Qwen3.5-Omni and evaluate Qwen3.5-Omni, Qwen3.8-Omni, Gemini3.8-Live, and GLM-Realtime in the response and revised paper.

Table 1. Overall Performance

ModelJoint task successIntent handlingInteraction correctness
Qwen3.5-Omni52.3970.1763.28
Qwen3.8-Omni57.0674.5665.22
Gemini3.8-Live66.6780.5677.67
GLM-Realtime25.0055.1737.06

As shown in Table 1, Gemini3.8-Live achieves the highest overall performance at 66.67% joint task success, followed by Qwen3.8-Omni (57.06%), Qwen3.5-Omni (52.39%), and GLM-Realtime (25.00%). Reporting Intent Handling and Interaction Correctness alongside Joint Task Success allows us to examine request handling and interaction behavior separately, while ensuring that passing only one metric does not count as overall success.

We thank all reviewers for being positive about our paper and for their constructive feedback. We have uploaded a revised manuscript (with all substantive changes highlighted in red). Specifically, we resolved presentation issues across the manuscript, clarified our metric definitions, judge validation, and differences from fixed-trajectory replay (Sec. 1, Sec. 3, Appendix B), expanded the benchmark to 600 severity-matched test cases with multi-run evaluations (Sec. 2.4, Sec. 3), added failure case studies and guard reliability analysis (Sec. 3.2, Sec. 4, Appendix C), and strengthened the future work and related work discussions (Sec. 4, Sec. 5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.