- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a full-duplex audio and video interaction benchmark based on real two-person video calls. It tests not only the model's ability to understand user expressions and actions but also its ability to adjust its dialogue behavior at the appropriate time based on this visual information. It also reveals a common "perception-to-interaction gap" in current omni-modal models: understanding something does not guarantee a correct response. The core idea of this paper can be summarized as: shifting the focus from evaluating whether the model "understands what is happening" to evaluating whether the model "can take appropriate interactive actions in real time based on what is happening."
Critical strengths
- The problem is worth studying. Most existing video benchmarks focus on whether a model understands what is happening, rather than whether it responds appropriately during an ongoing conversation. I like the distinction between visual recognition and actual conversational behavior. Evaluating both on the same events, together with audio-only comparisons, is a useful way to investigate this gap.
- The evaluation protocol is carefully designed. The authors evaluate the audio actually produced by the models and separate timing rules from content judgments. The detailed prompts, scoring rubrics, and judge agreement study in the appendix are also appreciated.
Critical weaknesses
- The benchmark seems rather small, especially for its core visual interaction tasks. There are only 707 task instances across six tasks, with just 99 expression events and 68 action events shared between recognition and response tasks. Some categories contain very few examples, such as only five contempt samples. Compared with related benchmarks such as OmniMMI and SocialOmni, the coverage is fairly limited. I understand that collecting natural audiovisual interactions is challenging, but I would expect a larger and more diverse evaluation to support the paper's broad conclusions about omni-modal interaction.
- The evaluation setting is still somewhat limited compared with real-world interaction. Each sample is constructed around a single predefined event. In real conversations, expressions, gestures, and interruptions occur continuously, and the interpretation of one event often depends on previous interactions. The current setup does not really test whether models can maintain and update conversational states across multiple events over longer periods. I think this limits how much we can conclude about their full-duplex interaction capabilities in practice.
Questions for authors
- Why is the benchmark relatively small? Given the size of Seamless Interaction, what limits the benchmark to 707 task instances, particularly only 99 expression events and 68 action events? How does this scale compare with related benchmarks, and could the core visual interaction tasks be expanded?
- Could the authors clarify the data source? Which subsets of Seamless Interaction were used, and what is the proportion of Naturalistic versus Improvised interactions? The paper describes natural remote video calls, whereas the original dataset also includes in-person recordings. How were these settings handled?
- What motivates the different trailing-silence settings? Appendix D.2 adds 3 seconds of trailing silence for audio-only runs but none for audiovisual runs, while recognition tasks use 8 seconds. Why are these settings different, and could they affect the comparison between conditions?
- Have the authors considered multi-event evaluation? Real conversations involve sequences of interdependent events, where previous expressions, actions, and responses influence subsequent interactions. Would it be possible to extend the benchmark to evaluate long-horizon conversational state tracking and responses across multiple events?
7 comments
Thank you for your constructive feedback and for recognizing our paired evaluation design and protocol. We have expanded the dataset and run additional control experiments to address each of your questions below.
W1 & Q1: Why the Benchmark Is Compact, Scale Comparison with Related Benchmarks, and Dataset Expansion (167 → 342 Visual Events)
Open-source recordings of real two-person audiovisual conversation are rare due to privacy constraints. Among related benchmarks, OmniMMI (1,121 videos, 2,290 QA pairs) tests streaming video QA rather than live spoken dialogue, while SocialOmni also evaluates live interactive generation and has a similarly compact generation split (209 instances). In addition, T3/T4 and T5/T6 share the same video clips to compare visual perception with spoken interaction. Each visual cue must therefore cause a real change in the other speaker's reply (CHANGED). Otherwise, a model could reply appropriately from the audio alone. Across the entire Seamless dev and test splits (115.64 hours, 2,145 dialogues, 12,756 initial segments), we manually reviewed 3,254 candidate clips (2,454 expressions and 800 actions) after structural filtering. We rejected 94.9% of candidates, including over 78% where the visual cue did not change the speaker's reply. Negative expressions such as Contempt and Anger are also naturally rare in everyday conversation. This filtering yielded 167 visual events across 235 speakers.
To enlarge the benchmark and balance rare classes, we annotated additional clips from the Seamless train split (3,893.35 hours, 62,606 dialogues) under the same three-annotator blind review protocol. This expands the visual events to 342 (2.05x, 684 paired instances, and 1,057 total instances), which we will release in full:
| Task & Category | Original (dev+test) | Added (train) | Expanded Total | Share (%) |
|---|---|---|---|---|
| T3/T4 Expression Events (Shared Clips) | 99 | +103 | 202 | 100.0% |
| Happiness | 22 | +14 | 36 | 17.8% |
| Surprise & Fear | 21 | +14 | 35 | 17.3% |
| Disgust | 19 | +16 | 35 | 17.3% |
| Sadness | 20 | +14 | 34 | 16.8% |
| Anger | 12 | +21 | 33 | 16.3% |
| Contempt | 5 | +24 | 29 | 14.4% |
| T5/T6 Action Events (Shared Clips) | 68 | +72 | 140 | 100.0% |
| Function: Continue | 26 | +22 | 48 | 34.3% |
| Function: Stop | 21 | +26 | 47 | 33.6% |
| Function: Modify | 21 | +24 | 45 | 32.1% |
| Form: Nod | 19 | +8 | 27 | 19.3% |
| Form: Head shake | 12 | +12 | 24 | 17.1% |
| Form: Stop hand | 11 | +12 | 23 | 16.4% |
| Form: Shrug | 10 | +13 | 23 | 16.4% |
| Form: Pointing | 8 | +14 | 22 | 15.7% |
| Form: Wave | 8 | +13 | 21 | 15.0% |
| Total Visual Events | 167 | +175 | 342 | |
| Naturalistic | 59 | 109 | 168 | 49.1% |
| Improvised | 108 | 66 | 174 | 50.9% |
Results on the expanded 342 visual events are shown below. Aside from a minor rank swap between Gemini 3.1 and 3.8, the model tiers, the perception-to-interaction gap, and the lack of consistent visual gains in T4/T6 all remain unchanged:
T3/T4: 99 original and 202 expanded expression events. T5/T6: 68 original and 140 expanded action events.
| Model | Set | T3 F1 | T4 AV | T4 AO | T5 Grounded | T6 AV | T6 AO |
|---|---|---|---|---|---|---|---|
| Qwen3.5-Omni-Plus | Original | 33.1 | 71.4 | 70.4 | 47.9 | 51.8 | 49.7 |
| Expanded | 34.8 | 68.9 | 69.1 | 50.6 | 47.2 | 48.9 | |
| Gemini 3.1 Flash Live | Original | 27.5 | 60.9 | 65.6 | 44.7 | 34.3 | 33.9 |
| Expanded | 31.8 | 58.4 | 63.9 | 39.4 | 33.6 | 32.8 | |
| Gemini 3.8 Live | Original | 29.8 | 59.1 | 64.1 | 32.8 | 31.2 | 38.4 |
| Expanded | 28.9 | 60.2 | 63.5 | 36.5 | 30.4 | 35.7 | |
| GPT-Realtime-2.1 | Original | 35.6 | 41.8 | 50.9 | 32.5 | 14.8 | 18.5 |
| Expanded | 39.8 | 37.9 | 48.2 | 34.1 | 13.5 | 17.2 | |
| MiniCPM-o 4.5 | Original | 19.7 | 44.9 | 15.3 | 10.8 | 29.3 | 17.3 |
| Expanded | 18.5 | 42.6 | 13.8 | 14.3 | 26.8 | 18.9 | |
| VITA-1.5 | Original | 24.5 | 47.7 | 53.0 | 5.7 | 32.4 | 35.4 |
| Expanded | 27.9 | 48.6 | 54.4 | 6.4 | 28.7 | 35.1 |
Q2: Source Subsets (Naturalistic vs. Improvised) and In-Person Recording vs. Video-Call Streaming Setup
We checked the official Seamless Interaction metadata for all samples. The initial 707 instances come from the official dev and test splits and cover both the Naturalistic and Improvised subsets. The Improvised subset uses scenario prompts to elicit conflict, skepticism, or stop gestures that rarely appear in casual small talk. Our expanded set adds a high proportion of Naturalistic clips from train, bringing the visual events to a balanced 49.1% vs. 50.9% ratio:
| Task | Original total | Dev | Test | Added (train) |
|---|---|---|---|---|
| T1 Interruption Handling | 179 | 64 | 115 | — |
| T2 Turn Entry | 194 | 98 | 96 | — |
| T3 / T4 Expressions (Shared Clips) | 99 | 47 | 52 | +103 |
| T5 / T6 Actions (Shared Clips) | 68 | 13 | 55 | +72 |
| Unique Visual Events (T3–T6) | 167 | 60 | 107 | +175 |
| All Task Instances (T1–T6) | 707 | 282 | 425 | +350 |
| Task | Set | Naturalistic | Improvised |
|---|---|---|---|
| T1 Interruption Handling | Original | 88 (49.2%) | 91 (50.8%) |
| Added (train) | — | — | |
| Expanded | 88 (49.2%) | 91 (50.8%) | |
| T2 Turn Entry | Original | 162 (83.5%) | 32 (16.5%) |
| Added (train) | — | — | |
| Expanded | 162 (83.5%) | 32 (16.5%) | |
| T3 / T4 Expressions (Shared Clips) | Original | 46 (46.5%) | 53 (53.5%) |
| Added (train) | 56 | 47 | |
| Expanded | 102 (50.5%) | 100 (49.5%) | |
| T5 / T6 Actions (Shared Clips) | Original | 13 (19.1%) | 55 (80.9%) |
| Added (train) | 53 | 19 | |
| Expanded | 66 (47.1%) | 74 (52.9%) | |
| Unique Visual Events (T3–T6) | Original | 59 (35.3%) | 108 (64.7%) |
| Added (train) | 109 | 66 | |
| Expanded | 168 (49.1%) | 174 (50.9%) | |
| All Task Instances (T1–T6) | Original | 368 (52.1%) | 339 (47.9%) |
| Added (train) | 218 | 132 | |
| Expanded | 586 (55.4%) | 471 (44.6%) |
Regarding the video-call setting, Seamless Interaction records each speaker with a dedicated single-person front-facing camera and an isolated microphone track. Only that single speaker appears in frame facing the lens, and the partner is never in view. The camera angle and audio isolation therefore match a local video-call stream. During evaluation, the model takes the role of the conversation partner over a live streaming API, receiving this single-person front-facing audiovisual stream and replying in real time. Both the input stream and the interaction setup thus correspond to a two-person video call.
Q3: Motivation for Trailing-Silence Settings and Symmetric AV (+3s) vs. A_ONLY (+3s) Control Experiment
In the recognition tasks (T3/T5), the model must wait until the whole clip ends before answering the question, so we append 8 s of silence at the end to trigger turn-end detection and collect the full answer. In the live interaction tasks (T4/T6), the visual event happens in the middle of the clip and is already followed by natural dialogue (up to 30 s in T4 and 10 s in T6). Moreover, the scored observation window for both AV and A_ONLY ends strictly at t_end, when the last natural audio packet is sent. The 3 s of silence added after t_end in A_ONLY to flush the audio buffer therefore falls completely outside the scored window and does not give A_ONLY extra response time.
To rule out any boundary effect from silence after t_end, we ran a symmetric control across all 6 models on all 99 T4 events and 68 T6 events. We added the exact same 3 s of trailing silence and 4 s wait to AV (AV +3s) and compared it with AV (0s) and A_ONLY (+3s):
Symmetric : AV (+3s) minus AO (+3s). Original : AV (0s) minus AO (+3s).
| Model | Task | AV (0 s) | AV (+3 s) | AO (+3 s) | (symmetric) | (original) |
|---|---|---|---|---|---|---|
Qwen3.5-Omni-Plus | T4 | 71.4 | 71.8 | 70.4 | +1.4 | +1.0 |
| T6 | 51.8 | 51.2 | 49.7 | +1.5 | +2.1 | |
Gemini 3.1 Flash Live | T4 | 60.9 | 61.3 | 65.6 | -4.3 | -4.7 |
| T6 | 34.3 | 34.6 | 33.9 | +0.7 | +0.4 | |
Gemini 3.8 Live | T4 | 59.1 | 58.8 | 64.1 | -5.3 | -5.0 |
| T6 | 31.2 | 31.7 | 38.4 | -6.7 | -7.2 | |
GPT-Realtime-2.1 | T4 | 41.8 | 42.3 | 50.9 | -8.6 | -9.1 |
| T6 | 14.8 | 14.5 | 18.5 | -4.0 | -3.7 | |
MiniCPM-o 4.5 | T4 | 44.9 | 45.2 | 15.3 | +29.9 | +29.6 |
| T6 | 29.3 | 28.9 | 17.3 | +11.6 | +12.0 | |
VITA-1.5 | T4 | 47.7 | 48.1 | 53.0 | -4.9 | -5.3 |
| T6 | 32.4 | 32.1 | 35.4 | -3.3 | -3.0 |
Because the scored window ends strictly at t_end, adding +3s trailing silence to AV changes T4 and T6 scores by only 0.3–0.6 points. All model rankings and AV - AO differences remain unchanged.
Sorry for the incomplete AI-use disclosure. I am not very familiar with this specific area, so I used AI to help understand the paper and polish my review. However, the main concerns are my own:
The first is my primary concern. Since scaling up the core tasks is difficult during the rebuttal phase, I sincerely advice the authors can expand the dataset in their next revision, as it would make the work much more convincing.