acceptodds
Under review as a conference paper at ICLR 2027

OmniDuet-Bench: Do Omni-Modal Models Act on What They See in Full-Duplex Audiovisual Conversation?

Abstract

Real-time omni-modal models can now generate speech while receiving streaming audio and video, yet no benchmark systematically evaluates whether they respond to the speech, facial expressions, and actions in their audiovisual input with appropriate timing and content: full-duplex benchmarks evaluate only spoken interaction, and video understanding benchmarks assess comprehension of video content rather than how a model, as a conversational participant, responds to the user on screen. We present OmniDuet-Bench, which evaluates full-duplex audiovisual conversation along three dimensions: Interaction Capacity, Visual Perception, and Cross-Modal Integration. The benchmark comprises 707 human-annotated task instances curated from natural two-person video calls and organized into six streaming tasks. Visual Perception and Cross-Modal Integration tasks share the same expression and action events and are complemented by matched audiovisual and audio-only conditions; together, these designs dissociate the perception of a visual cue from its use in conversational behavior. We evaluate seven real-time omni-modal models and find that no model leads on all three dimensions: models often fail to hold the floor and to take the turn promptly when the user yields it, and even after identifying an expression or action correctly, they often fail to adapt their responses to it. Data and evaluation code will be released.

Primary area:
datasets and benchmarks
Keywords:
Full-Duplex Conversation, Audiovisual Benchmark, Omni-modal Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HfQvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a full-duplex audio and video interaction benchmark based on real two-person video calls. It tests not only the model's ability to understand user expressions and actions but also its ability to adjust its dialogue behavior at the appropriate time based on this visual information. It also reveals a common "perception-to-interaction gap" in current omni-modal models: understanding something does not guarantee a correct response. The core idea of ​​this paper can be summarized as: shifting the focus from evaluating whether the model "understands what is happening" to evaluating whether the model "can take appropriate interactive actions in real time based on what is happening."

Critical strengths
  1. The problem is worth studying. Most existing video benchmarks focus on whether a model understands what is happening, rather than whether it responds appropriately during an ongoing conversation. I like the distinction between visual recognition and actual conversational behavior. Evaluating both on the same events, together with audio-only comparisons, is a useful way to investigate this gap.
  2. The evaluation protocol is carefully designed. The authors evaluate the audio actually produced by the models and separate timing rules from content judgments. The detailed prompts, scoring rubrics, and judge agreement study in the appendix are also appreciated.
Critical weaknesses
  1. The benchmark seems rather small, especially for its core visual interaction tasks. There are only 707 task instances across six tasks, with just 99 expression events and 68 action events shared between recognition and response tasks. Some categories contain very few examples, such as only five contempt samples. Compared with related benchmarks such as OmniMMI and SocialOmni, the coverage is fairly limited. I understand that collecting natural audiovisual interactions is challenging, but I would expect a larger and more diverse evaluation to support the paper's broad conclusions about omni-modal interaction.
  2. The evaluation setting is still somewhat limited compared with real-world interaction. Each sample is constructed around a single predefined event. In real conversations, expressions, gestures, and interruptions occur continuously, and the interpretation of one event often depends on previous interactions. The current setup does not really test whether models can maintain and update conversational states across multiple events over longer periods. I think this limits how much we can conclude about their full-duplex interaction capabilities in practice.
Questions for authors
  1. Why is the benchmark relatively small? Given the size of Seamless Interaction, what limits the benchmark to 707 task instances, particularly only 99 expression events and 68 action events? How does this scale compare with related benchmarks, and could the core visual interaction tasks be expanded?
  2. Could the authors clarify the data source? Which subsets of Seamless Interaction were used, and what is the proportion of Naturalistic versus Improvised interactions? The paper describes natural remote video calls, whereas the original dataset also includes in-person recordings. How were these settings handled?
  3. What motivates the different trailing-silence settings? Appendix D.2 adds 3 seconds of trailing silence for audio-only runs but none for audiovisual runs, while recognition tasks use 8 seconds. Why are these settings different, and could they affect the comparison between conditions?
  4. Have the authors considered multi-event evaluation? Real conversations involve sequences of interdependent events, where previous expressions, actions, and responses influence subsequent interactions. Would it be possible to extend the benchmark to evaluate long-horizon conversational state tracking and responses across multiple events?
7 comments

Sorry for the incomplete AI-use disclosure. I am not very familiar with this specific area, so I used AI to help understand the paper and polish my review. However, the main concerns are my own:

  1. The benchmark is too small, especially the core tasks.
  2. Multi-event continuous interaction is worth considering as a future extension.

The first is my primary concern. Since scaling up the core tasks is difficult during the rebuttal phase, I sincerely advice the authors can expand the dataset in their next revision, as it would make the work much more convincing.

Thank you for your constructive feedback and for recognizing our paired evaluation design and protocol. We have expanded the dataset and run additional control experiments to address each of your questions below.

W1 & Q1: Why the Benchmark Is Compact, Scale Comparison with Related Benchmarks, and Dataset Expansion (167 → 342 Visual Events)

Open-source recordings of real two-person audiovisual conversation are rare due to privacy constraints. Among related benchmarks, OmniMMI (1,121 videos, 2,290 QA pairs) tests streaming video QA rather than live spoken dialogue, while SocialOmni also evaluates live interactive generation and has a similarly compact generation split (209 instances). In addition, T3/T4 and T5/T6 share the same video clips to compare visual perception with spoken interaction. Each visual cue must therefore cause a real change in the other speaker's reply (CHANGED). Otherwise, a model could reply appropriately from the audio alone. Across the entire Seamless dev and test splits (115.64 hours, 2,145 dialogues, 12,756 initial segments), we manually reviewed 3,254 candidate clips (2,454 expressions and 800 actions) after structural filtering. We rejected 94.9% of candidates, including over 78% where the visual cue did not change the speaker's reply. Negative expressions such as Contempt and Anger are also naturally rare in everyday conversation. This filtering yielded 167 visual events across 235 speakers.

To enlarge the benchmark and balance rare classes, we annotated additional clips from the Seamless train split (3,893.35 hours, 62,606 dialogues) under the same three-annotator blind review protocol. This expands the visual events to 342 (2.05x, 684 paired instances, and 1,057 total instances), which we will release in full:

Task & CategoryOriginal (dev+test)Added (train)Expanded TotalShare (%)
T3/T4 Expression Events (Shared Clips)99+103202100.0%
Happiness22+143617.8%
Surprise & Fear21+143517.3%
Disgust19+163517.3%
Sadness20+143416.8%
Anger12+213316.3%
Contempt5+242914.4%
T5/T6 Action Events (Shared Clips)68+72140100.0%
Function: Continue26+224834.3%
Function: Stop21+264733.6%
Function: Modify21+244532.1%
Form: Nod19+82719.3%
Form: Head shake12+122417.1%
Form: Stop hand11+122316.4%
Form: Shrug10+132316.4%
Form: Pointing8+142215.7%
Form: Wave8+132115.0%
Total Visual Events167+175342
Naturalistic5910916849.1%
Improvised1086617450.9%

Results on the expanded 342 visual events are shown below. Aside from a minor rank swap between Gemini 3.1 and 3.8, the model tiers, the perception-to-interaction gap, and the lack of consistent visual gains in T4/T6 all remain unchanged:

T3/T4: 99 original and 202 expanded expression events. T5/T6: 68 original and 140 expanded action events.

ModelSetT3 F1T4 AVT4 AOT5 GroundedT6 AVT6 AO
Qwen3.5-Omni-PlusOriginal33.171.470.447.951.849.7
Expanded34.868.969.150.647.248.9
Gemini 3.1 Flash LiveOriginal27.560.965.644.734.333.9
Expanded31.858.463.939.433.632.8
Gemini 3.8 LiveOriginal29.859.164.132.831.238.4
Expanded28.960.263.536.530.435.7
GPT-Realtime-2.1Original35.641.850.932.514.818.5
Expanded39.837.948.234.113.517.2
MiniCPM-o 4.5Original19.744.915.310.829.317.3
Expanded18.542.613.814.326.818.9
VITA-1.5Original24.547.753.05.732.435.4
Expanded27.948.654.46.428.735.1

Q2: Source Subsets (Naturalistic vs. Improvised) and In-Person Recording vs. Video-Call Streaming Setup

We checked the official Seamless Interaction metadata for all samples. The initial 707 instances come from the official dev and test splits and cover both the Naturalistic and Improvised subsets. The Improvised subset uses scenario prompts to elicit conflict, skepticism, or stop gestures that rarely appear in casual small talk. Our expanded set adds a high proportion of Naturalistic clips from train, bringing the visual events to a balanced 49.1% vs. 50.9% ratio:

TaskOriginal totalDevTestAdded (train)
T1 Interruption Handling17964115—
T2 Turn Entry1949896—
T3 / T4 Expressions (Shared Clips)994752+103
T5 / T6 Actions (Shared Clips)681355+72
Unique Visual Events (T3–T6)16760107+175
All Task Instances (T1–T6)707282425+350
TaskSetNaturalisticImprovised
T1 Interruption HandlingOriginal88 (49.2%)91 (50.8%)
Added (train)——
Expanded88 (49.2%)91 (50.8%)
T2 Turn EntryOriginal162 (83.5%)32 (16.5%)
Added (train)——
Expanded162 (83.5%)32 (16.5%)
T3 / T4 Expressions (Shared Clips)Original46 (46.5%)53 (53.5%)
Added (train)5647
Expanded102 (50.5%)100 (49.5%)
T5 / T6 Actions (Shared Clips)Original13 (19.1%)55 (80.9%)
Added (train)5319
Expanded66 (47.1%)74 (52.9%)
Unique Visual Events (T3–T6)Original59 (35.3%)108 (64.7%)
Added (train)10966
Expanded168 (49.1%)174 (50.9%)
All Task Instances (T1–T6)Original368 (52.1%)339 (47.9%)
Added (train)218132
Expanded586 (55.4%)471 (44.6%)

Regarding the video-call setting, Seamless Interaction records each speaker with a dedicated single-person front-facing camera and an isolated microphone track. Only that single speaker appears in frame facing the lens, and the partner is never in view. The camera angle and audio isolation therefore match a local video-call stream. During evaluation, the model takes the role of the conversation partner over a live streaming API, receiving this single-person front-facing audiovisual stream and replying in real time. Both the input stream and the interaction setup thus correspond to a two-person video call.

Q3: Motivation for Trailing-Silence Settings and Symmetric AV (+3s) vs. A_ONLY (+3s) Control Experiment

In the recognition tasks (T3/T5), the model must wait until the whole clip ends before answering the question, so we append 8 s of silence at the end to trigger turn-end detection and collect the full answer. In the live interaction tasks (T4/T6), the visual event happens in the middle of the clip and is already followed by natural dialogue (up to 30 s in T4 and 10 s in T6). Moreover, the scored observation window for both AV and A_ONLY ends strictly at t_end, when the last natural audio packet is sent. The 3 s of silence added after t_end in A_ONLY to flush the audio buffer therefore falls completely outside the scored window and does not give A_ONLY extra response time.

To rule out any boundary effect from silence after t_end, we ran a symmetric control across all 6 models on all 99 T4 events and 68 T6 events. We added the exact same 3 s of trailing silence and 4 s wait to AV (AV +3s) and compared it with AV (0s) and A_ONLY (+3s):

Symmetric Δ\Delta: AV (+3s) minus AO (+3s). Original Δ\Delta: AV (0s) minus AO (+3s).

ModelTaskAV (0 s)AV (+3 s)AO (+3 s)Δ\Delta (symmetric)Δ\Delta (original)
Qwen3.5-Omni-PlusT471.471.870.4+1.4+1.0
T651.851.249.7+1.5+2.1
Gemini 3.1 Flash LiveT460.961.365.6-4.3-4.7
T634.334.633.9+0.7+0.4
Gemini 3.8 LiveT459.158.864.1-5.3-5.0
T631.231.738.4-6.7-7.2
GPT-Realtime-2.1T441.842.350.9-8.6-9.1
T614.814.518.5-4.0-3.7
MiniCPM-o 4.5T444.945.215.3+29.9+29.6
T629.328.917.3+11.6+12.0
VITA-1.5T447.748.153.0-4.9-5.3
T632.432.135.4-3.3-3.0

Because the scored window ends strictly at t_end, adding +3s trailing silence to AV changes T4 and T6 scores by only 0.3–0.6 points. All model rankings and AV - AO differences remain unchanged.

Reviewer_wDmmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

OmniDuet-Bench asks whether real-time omni-modal models respond to the words, facial expressions, and actions of the person on a video call, with appropriate timing and content. It has 707 human-annotated instances from natural two-person calls, organized into Interaction Capacity, Visual Perception, and Cross-Modal Integration. Perception and response tasks share expression and action events, and the response tasks also run in a matched audio-only condition.

Critical strengths
  1. Localizing the failure is the contribution, and the design actually localizes it.

  2. Turn-taking has a human reference, and the original speaker is not treated as an oracle on every subskill. Once a model is speaking, several models stop within 1.5 s at rates close to the original speaker, with median latencies of about 0.41–0.44 s against 0.40 s. The problem is mainly failing to hold the floor and failing to take the turn.

Critical weaknesses
  1. The judge belongs to the same model family as a leading evaluated system, and cross-modal integration is the strongest claim.

  2. The decisive cells are small, and every model is run once.

  3. The headline finding may be partly an input-fidelity artifact. Video is streamed at 1 fps in one-second causal chunks, so a model may not see a brief expression or fast gesture well enough to use it.

Questions for authors

1.Please rescore T4 and T6 with a non-Qwen judge, at least for Qwen3.5, Gemini 3.1 Flash, and GPT-Realtime-2.1.

  1. Can the headline T1 score be stop quality conditional on the model already speaking, with “not speaking at interruption time” reported separately?

3.On some action events, appropriate responses are more common after an incorrect recognition than after a correct one. Please check that T5’s strict grounded F1 is not defining “correct” so narrowly that the gap is inflated.

  1. How sensitive are T3 and T4 to the 1 fps video rate? Please report T3/T4 for at least one model at a higher frame rate. If recognition and response both rise, part of the gap is an input-fidelity limit.
4 comments

Thank you for your thorough feedback and for recognizing our problem formulation, human baselines, and modality ablations. We have run additional experiments to address each of your questions below.

W1 & Q1: Full Re-evaluation with an Independent Non-Qwen Judge (DeepSeek-V4-Pro)

In addition to the 200-sample cross-judge check with GPT and GLM in Appendix D.3, we added DeepSeek-V4-Pro as an independent blind judge that shares no model family with any evaluated model. Using the same outputs, rubrics, and anonymized setup, we re-evaluated all 6 models across the full T4 (AV/AO), T5, and T6 (AV/AO) benchmark (2,412 evaluations in total):

Judges: Qwen3.8-Max and DeepSeek-V4-Pro. Δ\Delta = AV minus AO.

ModelJudgeT4 AVT4 AOT4 Δ\DeltaT5 Grounded
Qwen3.5-Omni-PlusQwen71.470.4+1.047.9
DeepSeek68.269.5-1.351.6
Gemini 3.1 Flash LiveQwen60.965.6-4.744.7
DeepSeek65.363.8+1.540.9
Gemini 3.8 LiveQwen59.164.1-5.032.8
DeepSeek62.870.6-7.839.4
GPT-Realtime-2.1Qwen41.850.9-9.132.5
DeepSeek38.648.2-9.637.8
MiniCPM-o 4.5Qwen44.915.3+29.610.8
DeepSeek49.213.9+35.314.7
VITA-1.5Qwen47.753.0-5.35.7
DeepSeek54.157.6-3.58.4
ModelJudgeT6 AVT6 AOT6 Δ\Delta
Qwen3.5-Omni-PlusQwen51.849.7+2.1
DeepSeek48.146.8+1.3
Gemini 3.1 Flash LiveQwen34.333.9+0.4
DeepSeek38.939.7-0.8
Gemini 3.8 LiveQwen31.238.4-7.2
DeepSeek29.535.8-6.3
GPT-Realtime-2.1Qwen14.818.5-3.7
DeepSeek17.921.4-3.5
MiniCPM-o 4.5Qwen29.317.3+12.0
DeepSeek26.419.1+7.3
VITA-1.5Qwen32.435.4-3.0
DeepSeek30.234.8-4.6

With DeepSeek-V4-Pro as the judge, individual scores shift by 3.0–6.6 points and a few near-zero modality gaps flip sign due to slight differences in scoring strictness. Even so, Qwen3.5-Omni-Plus still ranks first on T4, T5, and T6, and the overall tier ordering across all 6 models is unchanged. Apart from MiniCPM-o 4.5 (which has high silence rates in audio-only mode), AV - AO gaps remain near zero or negative, confirming that our main rankings and conclusions are not driven by same-family judge bias.

Q2: Disentangling Floor-Holding (Speaking Coverage) from Conditional Interruption Handling in T1

The main metric scores 0 when the model is not speaking at the interruption moment teventt_{\text{event}}, because we can only test whether a model stops upon interruption if it is speaking at that moment. We agree that separating floor-holding rate from conditional interruption performance (when the model is already speaking at teventt_{\text{event}}) makes each model's bottleneck much clearer. We break down T1 below:

ModelFloor-Holding RateCond. Timely Stop RateUncond. Timely Stop RateCond. Median Stop Latency
Original speaker (Human)99.2% (129/130)80.6% (104/129)80.0% (104/130)0.40s
Qwen3.5-Omni-Plus44.6% (58/130)86.2% (50/58)38.5% (50/130)0.43s
Gemini 3.1 Flash Live42.3% (55/130)54.5% (30/55)23.1% (30/130)1.04s
Gemini 3.8 Live34.6% (45/130)93.3% (42/45)32.3% (42/130)0.44s
GPT-Realtime-2.119.2% (25/130)80.0% (20/25)15.4% (20/130)0.41s
MiniCPM-o 4.554.6% (71/130)21.1% (15/71)11.5% (15/130)3.93s
VITA-1.555.4% (72/130)25.0% (18/72)13.8% (18/130)3.36s
ModelCond. Interruption Score (P)Uncond. Interruption Score (P)Backchannel Robustness (N1)T1 Overall
Original speaker (Human)74.674.090.382.2
Qwen3.5-Omni-Plus65.929.424.527.0
Gemini 3.1 Flash Live53.222.524.023.2
Gemini 3.8 Live62.121.512.817.1
GPT-Realtime-2.171.213.77.110.4
MiniCPM-o 4.531.717.341.829.6
VITA-1.528.215.620.418.0

This breakdown shows two different failure modes. First, commercial APIs and Qwen3.5 already stop almost as fast as humans once they are speaking. When speaking at teventt_{\text{event}}, Qwen3.5, Gemini 3.8, and GPT-Realtime reach conditional timely stop rates of 80.0%–93.3%, median latencies of 0.41–0.44 s, and conditional scores of 62.1–71.2. Their lower unconditional scores come from generating short replies that end before teventt_{\text{event}} and from stopping too easily on brief backchannels (N1). Second, open-source models (MiniCPM-o 4.5 and VITA-1.5) struggle to stop when interrupted. They hold the floor more often and score higher on N1 because they tend to keep talking, but when truly interrupted they stop in time on only 21.1% and 25.0% of clips, with median latencies of 3.93 s and 3.36 s. We will add this table to the main text to show both conditional and unconditional metrics.

Q3: Decomposing T5 into Action-Only, Function-Only, and Grounded Metrics & Explaining the T5/T6 Conditional Rate Reversal

T5 uses the Grounded joint metric because there are only three function labels (Continue / Stop / Modify), so a model can guess the function label from audio context or by chance. At the same time, our T5 action-description score uses a lenient 0–2 semantic match rather than exact wording (for example, raises an open hand gets a full score of 2 for stop gesture). Combining action matching with function classification ensures that the model actually saw the gesture. Below we break T5 into Action-Only, Function-Only, and Grounded metrics and list the corresponding T6 appropriate rates:

ModelT5 criterionMatch / accuracyF1T6 app. if correctT6 app. if wrong
Qwen3.5-Omni-PlusAction-Only63.2% (43/68)55.8% (24/43)40.0% (10/25)
Function-Only55.9% (38/68)51.455.3% (21/38)43.3% (13/30)
Grounded41.2% (28/68)47.964.3% (18/28)40.0% (16/40)
Gemini 3.1 Flash LiveAction-Only42.6% (29/68)31.0% (9/29)33.3% (13/39)
Function-Only57.4% (39/68)55.933.3% (13/39)31.0% (9/29)
Grounded33.8% (23/68)44.734.8% (8/23)31.1% (14/45)
Gemini 3.8 LiveAction-Only36.8% (25/68)20.0% (5/25)27.9% (12/43)
Function-Only54.4% (37/68)51.824.3% (9/37)25.8% (8/31)
Grounded23.5% (16/68)32.812.5% (2/16)28.8% (15/52)
GPT-Realtime-2.1Action-Only38.2% (26/68)7.7% (2/26)14.3% (6/42)
Function-Only57.4% (39/68)54.710.3% (4/39)13.8% (4/29)
Grounded22.1% (15/68)32.56.7% (1/15)13.2% (7/53)
MiniCPM-o 4.5Action-Only36.8% (25/68)28.0% (7/25)30.2% (13/43)
Function-Only33.8% (23/68)30.230.4% (7/23)28.9% (13/45)
Grounded8.8% (6/68)10.833.3% (2/6)29.0% (18/62)
VITA-1.5Action-Only27.9% (19/68)15.8% (3/19)22.4% (11/49)
Function-Only32.4% (22/68)26.218.2% (4/22)21.7% (10/46)
Grounded4.4% (3/68)5.70.0% (0/3)21.5% (14/65)

This table shows that the reversal in Table 2 is not caused by the Grounded metric being too strict. Even under Action-Only or Function-Only correctness, Gemini 3.8, GPT-Realtime, and VITA-1.5 still do not score higher on T6 when recognition is correct than when it is wrong. Instead, the reversal happens because different action types have opposite difficulty in T5 and T6. Large Stop gestures (such as raising a hand) are easy to recognize in T5, but in T6 they require the model to stop talking within 1.5 s from a silent visual cue, where models almost always fail (in Appendix Table 16, Gemini 3.8 and GPT-Realtime score only 20.2 and 6.0 on T6 Stop). By contrast, small Continue gestures (such as nodding) are easier to miss in T5, but in T6 a model can score well just by continuing its speech (37.5 and 18.3 on T6 Continue). As a result, the T5-correct group contains more Stop gestures that fail in T6, while the T5-wrong group contains more Continue clips that succeed in T6 by default.

W3 & Q4: Sensitivity to 1 FPS Video Input Rate (1 FPS vs. 2 FPS Ablation)

Every annotated expression window in T3/T4 lasts 1.0 s, and action events in T5/T6 have a median duration of 4.34 s (minimum 1.25 s). Sampling at 1 FPS therefore captures at least one key frame in every target event window. In addition, the Google Live API (Gemini 3.1 / 3.8) caps video input at 1 FPS, so using 1 FPS keeps the comparison fair across models. For the other 4 models that accept higher frame rates, we ran a 1 FPS vs. 2 FPS ablation on T3 and T4 (99 paired expression events):

ModelMetric1 FPS2 FPSΔ\Delta
Qwen3.5-Omni-PlusT3 Macro-F133.131.2-1.9
T4 AV71.470.1-1.3
GPT-Realtime-2.1T3 Macro-F135.640.4+4.8
T4 AV41.843.2+1.4
MiniCPM-o 4.5T3 Macro-F119.720.6+0.9
T4 AV44.944.1-0.8
VITA-1.5T3 Macro-F124.522.8-1.7
T4 AV47.746.5-1.2

Raising the frame rate to 2 FPS does not generally improve performance, with only GPT-Realtime-2.1 showing a small gain. Further checks show that the bottleneck lies in each model's server-side visual processing rather than our 1 FPS input protocol. For example, VITA-1.5 uniformly samples a fixed 4 frames per turn on the server regardless of the client frame rate. We also ran a 2 FPS synthetic digit-flash test (0.5 s flash duration) on Qwen3.5-Omni-Plus. It read out 4/5 digits flashed at integer seconds (t = X.0s) but 0/5 digits flashed at half seconds (t = X.5s), showing that its streaming server internally samples video at roughly 1 Hz.

W2: Small Decisive Cells (Dataset Expansion 167 → 342 Events) and Single-Run Evaluation (3-Run Mean ± Std)

To address both cell sample sizes and single-run evaluation, we expanded the visual events by 2.05$\times$ and ran 3 independent trials across all models:

1. Dataset Expansion (167 → 342 Visual Events, 1,057 Total Instances) We annotated additional clips from the Seamless train split (3,893.35 hours, 62,606 dialogues) under the same three-annotator blind review protocol. This expands the visual events to 342 (2.05x, 684 paired instances, and 1,057 total instances), which we will release in full:

Task & CategoryOriginal (dev+test)Added (train)Expanded TotalShare (%)
T3/T4 Expression Events (Shared Clips)99+103202100.0%
Happiness22+143617.8%
Surprise & Fear21+143517.3%
Disgust19+163517.3%
Sadness20+143416.8%
Anger12+213316.3%
Contempt5+242914.4%
T5/T6 Action Events (Shared Clips)68+72140100.0%
Function: Continue26+224834.3%
Function: Stop21+264733.6%
Function: Modify21+244532.1%
Form: Nod19+82719.3%
Form: Head shake12+122417.1%
Form: Stop hand11+122316.4%
Form: Shrug10+132316.4%
Form: Pointing8+142215.7%
Form: Wave8+132115.0%
Total Visual Events167+175342
Naturalistic5910916849.1%
Improvised1086617450.9%

Results on the expanded 342 visual events are shown below. Aside from a minor rank swap between Gemini 3.1 and 3.8, the model tiers, the perception-to-interaction gap, and the lack of consistent visual gains in T4/T6 all remain unchanged:

T3/T4: 99 original and 202 expanded expression events. T5/T6: 68 original and 140 expanded action events.

ModelSetT3 F1T4 AVT4 AOT5 GroundedT6 AVT6 AO
Qwen3.5-Omni-PlusOriginal33.171.470.447.951.849.7
Expanded34.868.969.150.647.248.9
Gemini 3.1 Flash LiveOriginal27.560.965.644.734.333.9
Expanded31.858.463.939.433.632.8
Gemini 3.8 LiveOriginal29.859.164.132.831.238.4
Expanded28.960.263.536.530.435.7
GPT-Realtime-2.1Original35.641.850.932.514.818.5
Expanded39.837.948.234.113.517.2
MiniCPM-o 4.5Original19.744.915.310.829.317.3
Expanded18.542.613.814.326.818.9
VITA-1.5Original24.547.753.05.732.435.4
Expanded27.948.654.46.428.735.1

2. Multi-Run Stability (Mean ± Std over 3 Independent Runs) We ran all 6 models 3 independent times on T1–T6:

ModelRunT1 AVT2 AVT3 Macro-F1T5 Grounded
Qwen3.5-Omni-PlusPaper27.039.733.147.9
3-Run27.4±0.844.6±4.534.6±1.850.1±2.8
Gemini 3.1 Flash LivePaper23.261.027.544.7
3-Run20.8±2.561.8±1.229.4±2.141.5±3.2
Gemini 3.8 LivePaper17.158.629.832.8
3-Run15.3±1.959.1±0.928.6±1.437.1±4.2
GPT-Realtime-2.1Paper10.457.735.632.5
3-Run8.8±1.757.8±0.837.5±2.134.4±1.9
MiniCPM-o 4.5Paper29.655.019.710.8
3-Run28.4±1.356.1±1.518.9±1.012.1±1.4
VITA-1.5Paper18.023.724.55.7
3-Run18.7±0.922.8±1.125.8±1.55.2±0.8
ModelRunT4 AVT4 AOT6 AVT6 AO
Qwen3.5-Omni-PlusPaper71.470.451.849.7
3-Run69.2±2.268.7±1.746.8±4.849.2±0.9
Gemini 3.1 Flash LivePaper60.965.634.333.9
3-Run59.6±1.564.2±1.434.2±0.832.9±1.1
Gemini 3.8 LivePaper59.164.131.238.4
3-Run59.1±0.963.8±0.830.6±1.035.4±2.9
GPT-Realtime-2.1Paper41.850.914.818.5
3-Run37.9±3.848.2±2.613.1±1.816.4±2.1
MiniCPM-o 4.5Paper44.915.329.317.3
3-Run45.7±1.414.2±1.327.9±1.618.4±1.3
VITA-1.5Paper47.753.032.435.4
3-Run46.8±1.354.1±1.233.1±1.234.6±1.0

Standard deviations across the 3 runs are mostly 0.8–2.5 points, with a few timing-sensitive metrics around 4.2–4.8 points. The overall model rankings, the gap between perception and interaction, and the lack of visual gains in T4/T6 all hold across runs.

Reviewer_QRUJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OmniDuet-Bench, a benchmark for evaluating whether real-time omni-modal models can appropriately incorporate audiovisual cues into full-duplex conversational behavior. The authors argue that existing full-duplex dialogue benchmarks primarily focus on audio-based turn-taking, while streaming video understanding benchmarks assess visual comprehension without evaluating how visual information influences conversational responses.

Critical strengths
  1. Important and timely research problem.
  2. Well-motivated decomposition of conversational capabilities.
  3. Thoughtful paired-task and modality-ablation design.
  4. Naturalistic conversational data and human annotation.
Critical weaknesses
  1. The dataset size is small, particularly regarding the core visual interaction component. Although the paper claims a total of 707 samples, the distinct events used to study the recognition of—and response to—facial expressions and gestures amount to only 99 expressions and 68 gestures. Such a limited test set is highly susceptible to the influence of individual samples, annotation discrepancies, and model stochasticity. Given that this study aims to compare the performance of seven different models in complex social interactions, the statistical robustness is insufficient.

  2. Each sample is processed only once, failing to adequately verify stability. As noted in Appendix D.1, each model is tested on each sample only once under any given condition. However, real-time speech models inherently involve a degree of randomness, compounded by variations in APIs, Voice Activity Detection (VAD), and dialogue generation strategies. Running the same video three times could yield entirely different model responses. I suggest that the authors incorporate repeated experiments, confidence intervals, and statistical significance testing for the performance differences observed between the "AV" (audio-visual) and "audio-only" conditions.

  3. Visual input is limited to 1 FPS, which may lead to distorted evaluation. The paper processes all videos at a rate of one frame per second, feeding them into the models in one-second increments. Yet, facial expressions and rapid gestures often occur within much shorter timeframes. Consequently, it remains unclear whether a model’s failure to recognize an action stems from a lack of visual understanding or from the loss of critical cues due to the low sampling rate. The authors have not sufficiently distinguished between these possibilities through ablation studies on frame rates.

  4. The causal explanation for the "perception-to-interaction gap" lacks rigor. Tasks T3 and T5 explicitly require the recognition of expressions or gestures, whereas T4 and T6 involve standard conversational generation—each employing different prompts and objectives. Therefore, the fact that an AI might succeed in a recognition task but underperform in a conversational task does not, by itself, prove that it correctly perceived an action yet failed to translate that perception into an appropriate behavior. Strictly speaking, this merely demonstrates an inconsistency in performance across different tasks; further experimentation is required to substantiate the proposed mechanism behind the so-called "perception-to-interaction gap."

Questions for authors

Please refer to the weaknesses.

4 comments

Thank you for your constructive feedback and for recognizing our evaluation design. We have expanded the dataset and run additional control experiments to address each of your questions below.

W1: Dataset Size, Class Imbalance, and Statistical Robustness

Open-source recordings of real two-person audiovisual conversation are rare due to privacy constraints, and related streaming benchmarks are similarly compact (e.g., SocialOmni has 209 generation instances). In addition, T3/T4 and T5/T6 share the same video clips to compare visual perception with spoken interaction. Each visual cue must therefore cause a real change in the other speaker's reply (CHANGED). Otherwise, a model could respond appropriately from the audio alone. Across the entire Seamless dev and test splits (115.64 hours, 2,145 dialogues, 12,756 initial segments), we manually reviewed 3,254 candidate clips (2,454 expressions and 800 actions) after structural filtering. We rejected 94.9% of candidates, including over 78% where the visual cue did not change the speaker's reply. Negative expressions such as Contempt and Anger are also naturally rare in everyday conversation. This filtering yielded 167 visual events across 235 speakers.

To enlarge the benchmark and balance rare classes, we annotated additional clips from the Seamless train split (3,893.35 hours, 62,606 dialogues) under the same three-annotator blind review protocol. This expands the visual events to 342 (2.05x, 684 paired instances, and 1,057 total instances), which we will release in full:

Task & CategoryOriginal (dev+test)Added (train)Expanded TotalShare (%)
T3/T4 Expression Events (Shared Clips)99+103202100.0%
Happiness22+143617.8%
Surprise & Fear21+143517.3%
Disgust19+163517.3%
Sadness20+143416.8%
Anger12+213316.3%
Contempt5+242914.4%
T5/T6 Action Events (Shared Clips)68+72140100.0%
Function: Continue26+224834.3%
Function: Stop21+264733.6%
Function: Modify21+244532.1%
Form: Nod19+82719.3%
Form: Head shake12+122417.1%
Form: Stop hand11+122316.4%
Form: Shrug10+132316.4%
Form: Pointing8+142215.7%
Form: Wave8+132115.0%
Total Visual Events167+175342
Naturalistic5910916849.1%
Improvised1086617450.9%

Results on the expanded 342 visual events are shown below. Aside from a minor rank swap between Gemini 3.1 and 3.8, the model tiers, the perception-to-interaction gap, and the lack of consistent visual gains in T4/T6 all remain unchanged:

T3/T4: 99 original and 202 expanded expression events. T5/T6: 68 original and 140 expanded action events.

ModelSetT3 F1T4 AVT4 AOT5 GroundedT6 AVT6 AO
Qwen3.5-Omni-PlusOriginal33.171.470.447.951.849.7
Expanded34.868.969.150.647.248.9
Gemini 3.1 Flash LiveOriginal27.560.965.644.734.333.9
Expanded31.858.463.939.433.632.8
Gemini 3.8 LiveOriginal29.859.164.132.831.238.4
Expanded28.960.263.536.530.435.7
GPT-Realtime-2.1Original35.641.850.932.514.818.5
Expanded39.837.948.234.113.517.2
MiniCPM-o 4.5Original19.744.915.310.829.317.3
Expanded18.542.613.814.326.818.9
VITA-1.5Original24.547.753.05.732.435.4
Expanded27.948.654.46.428.735.1

W2: Single-Run Evaluation and Multi-Run Stability (Mean ± Std over 3 Runs)

To measure run-to-run variance in live streaming inference, we ran all 6 models 3 independent times on T1–T6. The original single-run scores and the 3-run Mean ± Std are compared below:

ModelRunT1 AVT2 AVT3 Macro-F1T5 Grounded
Qwen3.5-Omni-PlusPaper27.039.733.147.9
3-Run27.4±0.844.6±4.534.6±1.850.1±2.8
Gemini 3.1 Flash LivePaper23.261.027.544.7
3-Run20.8±2.561.8±1.229.4±2.141.5±3.2
Gemini 3.8 LivePaper17.158.629.832.8
3-Run15.3±1.959.1±0.928.6±1.437.1±4.2
GPT-Realtime-2.1Paper10.457.735.632.5
3-Run8.8±1.757.8±0.837.5±2.134.4±1.9
MiniCPM-o 4.5Paper29.655.019.710.8
3-Run28.4±1.356.1±1.518.9±1.012.1±1.4
VITA-1.5Paper18.023.724.55.7
3-Run18.7±0.922.8±1.125.8±1.55.2±0.8
ModelRunT4 AVT4 AOT6 AVT6 AO
Qwen3.5-Omni-PlusPaper71.470.451.849.7
3-Run69.2±2.268.7±1.746.8±4.849.2±0.9
Gemini 3.1 Flash LivePaper60.965.634.333.9
3-Run59.6±1.564.2±1.434.2±0.832.9±1.1
Gemini 3.8 LivePaper59.164.131.238.4
3-Run59.1±0.963.8±0.830.6±1.035.4±2.9
GPT-Realtime-2.1Paper41.850.914.818.5
3-Run37.9±3.848.2±2.613.1±1.816.4±2.1
MiniCPM-o 4.5Paper44.915.329.317.3
3-Run45.7±1.414.2±1.327.9±1.618.4±1.3
VITA-1.5Paper47.753.032.435.4
3-Run46.8±1.354.1±1.233.1±1.234.6±1.0

Standard deviations across the 3 runs are mostly 0.8–2.5 points, with a few timing-sensitive metrics around 4.2–4.8 points. The overall model rankings, the gap between perception and interaction, and the lack of visual gains in T4/T6 all hold across runs.

W3: 1 FPS Input Sampling and 1 vs. 2 FPS Ablation

Thank you for raising this point. Every annotated expression window in T3/T4 lasts 1.0 s, and action events in T5/T6 have a median duration of 4.34 s (minimum 1.25 s). Sampling at 1 FPS therefore captures at least one key frame in every target event window. In addition, the Google Live API (Gemini 3.1 / 3.8) caps video input at 1 FPS, so using 1 FPS keeps the comparison fair across models. For the other 4 models that accept higher frame rates, we ran a 1 FPS vs. 2 FPS ablation on T3 and T4 (99 paired expression events):

ModelMetric1 FPS2 FPSΔ\Delta
Qwen3.5-Omni-PlusT3 Macro-F133.131.2-1.9
T4 AV71.470.1-1.3
GPT-Realtime-2.1T3 Macro-F135.640.4+4.8
T4 AV41.843.2+1.4
MiniCPM-o 4.5T3 Macro-F119.720.6+0.9
T4 AV44.944.1-0.8
VITA-1.5T3 Macro-F124.522.8-1.7
T4 AV47.746.5-1.2

Raising the frame rate to 2 FPS does not generally improve performance, with only GPT-Realtime-2.1 showing a small gain. Further checks show that the bottleneck lies in each model's server-side visual processing rather than our 1 FPS input protocol. For example, VITA-1.5 uniformly samples a fixed 4 frames per turn on the server regardless of the client frame rate. We also ran a 2 FPS synthetic digit-flash test (0.5 s flash duration) on Qwen3.5-Omni-Plus. It read out 4/5 digits flashed at integer seconds (t = X.0s) but 0/5 digits flashed at half seconds (t = X.5s), showing that its streaming server internally samples video at roughly 1 Hz.

W4: Interpretation of the Perception-to-Interaction Gap

We evaluate T3/T5 (explicit recognition) and T4/T6 (live conversation) in separate sessions on the same video clips to keep the conversation natural while allowing a controlled comparison. In a real video call, a user will not pause every time they frown or raise a hand to ask the model to classify their expression or gesture first. Doing so would break the natural turn-taking flow and give the model an explicit hint. Testing the same clip separately in T3/T5 checks whether the model can recognize the visual cue at all, which rules out blurry video or missing visual knowledge. In T4/T6, the system prompt explicitly instructs the model to use the user's visible expressions and actions during conversation. If a model recognizes the cue when directly asked in T3/T5, but ignores the video or fails to adjust its reply during live conversation in T4/T6, this shows that it cannot proactively use visual cues during spoken interaction. That is the core limitation captured by our perception-to-interaction gap. Below we provide the full 2×22 \times 2 counts corresponding to Table 2 (99 T3/T4 events and 68 T5/T6 events):

ModelTasks (recognition / response)Correct / goodCorrect / poor (gap)Wrong / goodBoth wrong
Qwen3.5-Omni-PlusT3/T436 (80.0%)9 (20.0%)33 (61.1%)21
T5/T618 (64.3%)10 (35.7%)16 (40.0%)24
Gemini 3.1 Flash LiveT3/T424 (61.5%)15 (38.5%)30 (50.0%)30
T5/T68 (34.8%)15 (65.2%)14 (31.1%)31
Gemini 3.8 LiveT3/T423 (59.0%)16 (41.0%)25 (41.7%)35
T5/T62 (12.5%)14 (87.5%)15 (28.8%)37
GPT-Realtime-2.1T3/T418 (39.1%)28 (60.9%)17 (32.1%)36
T5/T61 (6.7%)14 (93.3%)7 (13.2%)46
MiniCPM-o 4.5T3/T415 (41.7%)21 (58.3%)26 (41.3%)37
T5/T62 (33.3%)4 (66.7%)18 (29.0%)44
VITA-1.5T3/T411 (31.4%)24 (68.6%)16 (25.0%)48
T5/T60 (0.0%)3 (100.0%)14 (21.5%)51

Many cues that models recognize correctly in T3/T5 still fail to trigger appropriate replies in T4/T6. At the same time, some T3-misclassified clips still get acceptable T4 replies from tone of voice and spoken context alone. This matches the near-zero AV - AO differences in T4/T6 and shows that explicit perception metrics cannot replace live interaction evaluation.

We are truly sorry for submitting our response so late and for keeping you waiting. We completely understand why the delay was disappointing and gave the impression that we were not planning to respond, and we should have posted a brief update earlier to let you know we were working on it.

In fact, we took your comments very seriously from the start. Throughout this period, we have been working on expanding the dataset, completing the manual screening and blind annotations, and running the additional evaluations. Expanding this dataset turned out to be much more difficult and time-consuming than we expected, which is why it took us so long to finish and post our response.

We really appreciate your thoughtful feedback and would very much welcome the chance to discuss your concerns and our new results with you. Please let us know if you have any further questions, and we will reply promptly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.