acceptodds
Under review as a conference paper at ICLR 2027

DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams

Abstract

Recent advances in multimodal large language models (MLLMs) have substan- tially improved spoken interaction and video understanding. Existing online and streaming benchmarks primarily assess causal multimodal perception through timestamped question answering, whereas real-world interaction requires mod- els to continuously identify task-relevant multimodal cues and decide whether, when, and how to respond. This leaves open how well current models can sus- tain real-time interaction throughout an evolving multimodal stream. To address this gap, we introduce DIVE, a benchmark for evaluating full-duplex interaction with visual engagement under continuous and realistic multimodal streams. Our benchmark organizes six tasks into three complementary interaction scenarios. (1) Proactive Intervention evaluates whether models can identify emerging vi- sual or semantic cues and intervene at the appropriate moment. (2) Continuous Engagement evaluates sustained participation through continuous visual tracking and visually grounded query interaction. (3) Interaction Coordination evaluates how models coordinate listening and speaking under overlapping and multi-party interactions. DIVE contains 525 audio/video samples with fine-grained human annotations and precise temporal metadata, covering six task-oriented interaction settings grounded in realistic scenarios. Its modality-aware design enables evalu- ation of both full-duplex models and modality-specific streaming models. We hope DIVE facilitates progress toward more natural and capable real-time multimodal interactive systems

Primary area:
datasets and benchmarks
Keywords:
full-duplex multimodal interaction, real-time multimodal interaction, audio-visual understanding, streaming multimodal models, multimodal large language models, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SGWoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper builds the DIVE, a benchmark containing 525 evaluation instances across six tasks for continuous audio-visual interaction. The topic is relevant, and tasks such as addressee tracking and response adaptation have practical value. However, the submission does not establish a sufficiently clear research gap, substantive incremental contribution, or convincing empirical justification for the proposed benchmark. The paper mainly describes a collection of tasks and reports model scores, while the connection between the motivation, benchmark design, and resulting scientific insights remains weak.

Critical strengths
  • The benchmark addresses meaningful interaction behaviors, particularly repeated action counting, response adaptation, and addressee-dependent participation.
Critical weaknesses
  • Unclear motivation and limited novelty. Existing benchmarks such as OmniInteract and Omni-DuplexEval already evaluate continuous interaction, proactive responses, and interruption handling. The paper does not clearly identify what these evaluations miss or why addressing that gap requires DIVE’s particular task taxonomy. Assembling additional scenarios is potentially useful, but the paper provides limited evidence of a substantive advance in evaluation methodology or scientific understanding.
  • Weak presentation and supporting evidence. The paper lacks a clear progression from existing limitations to design requirements and experimental validation. Figures illustrate intended interactions rather than demonstrating distinctive challenges or actual failures.
  • Unverified visual and full-duplex requirements. No controlled experiments establish whether visual input or simultaneous perception is necessary. Some updates can be understood from speech alone. Prerecorded interruptions may also arrive before slower models begin speaking, making actual overlap uncertain.
  • Insufficient validation and reproducibility. Annotation agreement, human validation of the judge, essential dataset statistics, inference settings, and aggregation procedures are missing.
Questions for authors

See weakness.

4 comments

Dear Reviewer SGWo,

Thank you again for your detailed feedback. In our rebuttal, we have provided a task-level comparison with OmniInteract and Omni-DuplexEval, controlled visual-input ablations, and additional analyses of the interaction failures revealed by DIVE. We have also reported human validation of the judge and temporal robustness analyses, alongside clarifications of the dataset statistics, inference settings, aggregation procedures, and ORA evaluation setup.

We would greatly appreciate it if you could review our responses and the supporting results, particularly Tables R6 and R7, and let us know which concerns remain. Your feedback would help us understand whether further evidence or clarification is needed.

If these additions address your concerns sufficiently to change your assessment, we would be grateful if you could update your evaluation and rating accordingly.

Thank you for your time and consideration.

Thanks for the rebuttal, which make comparisions clear. However, the paper is still poorly written. It fails to clearly explain the limitations of existing work in the main paper, the motivation for this study, or its contributions. The writing lacks a logical structure, and the claims are insufficiently supported. The figures and tables do not effectively convey the motivation or demonstrate the advantages of the proposed approach. Merely establishing a benchmark is too limited a contribution.

AuthorsOpenReview ↗

Request for specific remaining concerns

Dear Reviewer SGWo,

Thank you for your reply. We are glad that the rebuttal made the comparison with OmniInteract and Omni-DuplexEval clear, as the lack of a clear research gap relative to these benchmarks was the first weakness listed in your review.

Your reply does not mention your other two substantive concerns: whether visual input and simultaneous perception are necessary (W3), and validation and reproducibility (W4). We addressed these with a black-frame ablation (Table R6), human validation of the judge (93.3% agreement, κ = 0.844), annotator variability, window-sensitivity analysis, and bootstrap confidence intervals. Could you confirm whether these concerns are now resolved? If not, please tell us which result you find insufficient.

Your remaining comments concern the writing, but they are stated in general terms, and we cannot determine which parts of the paper they refer to. We are very willing to revise, and specific pointers would help greatly:

Limitations of existing work. Section 1, Section 2.2, and Table 1 discuss prior benchmarks. Which limitation is unclear, or which prior work is described inaccurately? We will move Table R7 into Section 2.2. Would that address this point? Motivation and contributions. Which part of the motivation do you find unconvincing, and which stated contribution is unclear? Logical structure. Which sections or transitions are hard to follow? Unsupported claims. Could you list the specific claims you consider insufficiently supported? For each one, we will either point to the supporting evidence or revise the claim. Figures and tables. Which figures or tables fail to convey the motivation? We will replace the illustrative figures with actual model failures, for example Gemini-3.1-Flash-Live on AVAT (quality 88.4 but ESR 25.0) and the black-frame ablation on VTI and CAT. Would this address your concern? Contribution of a benchmark. The paper is submitted under the datasets and benchmarks primary area, where a benchmark is the intended contribution. Beyond the tasks themselves, DIVE contributes paired-condition designs (ORA, AVAT), a joint evaluation of timing and content (ESR, FIR), and findings that existing benchmarks do not show: proprietary models fail mainly on timing while open models fail mainly on content, and proactive intervention degrades sharply when the trigger is spoken rather than visual (ESR 51.2–65.3 on VTI vs. 18.0–19.2 on SPI). What additional contribution would you expect from a benchmark paper in this area?

Finally, your review rates Soundness as 1 (“significant concerns … that do not appear easy to address”). If your remaining concerns are about presentation, could you clarify which soundness concerns still stand?

We appreciate your time and look forward to your specific feedback while the discussion is still open.

Best regards, The authors of Submission 9379

Dear Reviewer SGWo,

We have uploaded a revised manuscript that restructures the introduction around the limitations of existing benchmarks, the resulting design requirements, and their experimental validation, and moves the comparison with OmniInteract and Omni-DuplexEval and the validation analyses into the paper. We would still welcome specific pointers to any section, claim, figure, or table you find unclear or unsupported, and we will address each one while the discussion is open.

The authors

Reviewer_GghHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DIVE is a benchmark for full-duplex audio-visual interaction in continuous streams. It contains 525 evaluation instances from 413 base scenarios across six tasks, covering proactive intervention, continuous engagement, and interaction coordination. The evaluation separates participation decisions, response timing, and content quality. Tests of proprietary and open models show that low average latency does not necessarily imply consistent compliance with response windows. The highest reported window response rate, averaged equally across the six tasks, is 53.77%.

Critical strengths

The benchmark addresses a useful gap between understanding a stream and participating appropriately in it. Separating whether, when, and how well a model responds makes failures visible that a content-only score would miss. The six tasks cover several forms of sustained interaction, and the comparison includes both proprietary and open models.

Critical weaknesses
  1. The content-quality comparisons need validation against human judgments. Section 3.5 uses GPT-5.6 Terra as a binary judge, and Appendix A supplies detailed task-specific prompts, but the manuscript does not report agreement with human ratings. Providing the prompts supports reproducibility; it does not establish the reliability of the resulting scores. This leaves uncertainty about an important part of the benchmark evaluation.

  2. The uncertainty in the temporal evaluation is not quantified. Appendix B describes four independent annotators, gives the mean of their CAT timestamps, and defines task-specific response windows. However, it does not report annotation variability or sensitivity to boundary choices. Confidence intervals for the reported comparisons are also absent. Without these checks, it is difficult to assess whether smaller differences between models are robust. Uncertainty estimates should account for multiple events within a video and the paired variants derived from shared base scenarios.

I lean toward weak rejection because the benchmark's value depends on the reliability of these measurements. Human validation of the content judge and a focused analysis of temporal uncertainty could change my assessment.

Questions for authors
  1. Report agreement between GPT-5.6 Terra and human judgments on a representative subset, including whether disagreements affect the model comparisons.
  2. Report the variability of the four annotators' temporal labels and a sensitivity analysis for the response windows. Uncertainty estimates should account for repeated events and paired scenario variants.
  3. Clarify the treatment of missing responses in the quality score. Section 3.5 and Eq. (5) exclude events without an assigned response, whereas the AVAT and CAT prompts in Appendix A specify zero scores for missing responses. Explain whether these zeros enter Table 4 or are excluded during aggregation. These descriptions may refer to different processing stages, but the relationship is currently unclear.
3 comments

Dear Reviewer GghH,

Thank you again for your constructive feedback. Our rebuttal now includes human validation of the judge (93.3% agreement), temporal annotation variability, window-sensitivity analysis, and confidence intervals accounting for shared base scenarios. We have also clarified how missing responses are handled.

We would greatly appreciate it if you could review these additional results and let us know whether they address your concerns. If so, we would be grateful if you could reconsider your assessment and rating in light of this evidence. If any concerns remain, we would welcome the opportunity to address them.

Thank you for your time and consideration.

Dear Reviewer GghH,

We are following up briefly while the discussion period is still open. Your review noted that “human validation of the content judge and a focused analysis of temporal uncertainty could change my assessment.” We have provided both:

Human validation of the judge: The judge agrees with adjudicated human labels on 93.3% of 240 responses (κ = 0.844), close to the agreement between the two human annotators (κ = 0.907). Temporal uncertainty: We report annotator variability for each task and the sensitivity of the results to window boundaries. Rankings stay stable when the windows are tightened (Kendall τ ≥ 0.78 for WRR and ≥ 0.96 for ESR). We also report 95% cluster-bootstrap confidence intervals over base scenarios, which account for repeated events and paired variants. Missing responses: In Table 4, missed events score 0 and remain in the denominator. We will correct Eq. (5) to match. We would greatly appreciate it if you could let us know whether these analyses address your concerns and, if so, consider updating your assessment.

Thank you for your time.

Dear Reviewer GghH,

Thank you again for raising your score. As promised, we have incorporated the human validation of the judge, the temporal-uncertainty analyses, the confidence intervals, and the corrected Eq. (5) into the revised manuscript, and improved the writing and presentation. We would be grateful if you would take a look and consider whether a further update to your assessment is warranted.

The authors

Reviewer_KeSEOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DIVE, it's a benchmark for full duplex audio-visual interaction in continuous streams. Its 525 instances cover proactive intervention, continuous engagement, and interaction coordination. Six tasks were included in the dataset. The evaluation wants to seperately assess whether a model responds, whether it responds in the correct time window, and whether the response is correct. The benchmark shows that low latency does not guarentee the right window selection or the right response. The models could become overly eager for the ones with low laency.

Critical strengths
  1. The benchmark includes useful interaction settings that are less covered by prior benchmarks, like audio-visual addressee tracking. These tasks test whether a model can maintain an evolving interaction state and determine when it is being addressed.

  2. The evaluation separates response decisions, temporal alignment, and content quality. Although a joint metric is still needed, it's good that this decomposition reveals these failure modes. It's meaningful to see that fast responses would miss the valid window. This was not obvious in the prior works that relied heavily on latency and response quality.

Critical weaknesses
  1. The response quality is averaged only over responses that were assigned to an event, and missed events do not lower that score. I'm a bit skeptical of this, since streaming models might miss the current response and end up respond at the wrong timing. Reporting decision, timing, and quality separately makes it difficult to compare overall interaction success and can make a selective model look better on quality.

  2. DIVE overlaps quite a bit with OmniInteract in task design. I think the added tasks are useful, like continuous query interaction and addressee tracking, but the paper does not clearly show what new conclusion we can derive from DIVE over OmniInteract. This makes the benchmark novelty less convincing.

Questions for authors
  1. Can you provide a direct task and metric level comparison with OmniInteract and explain what capability or empirical conclusion DIVE captures that OmniInteract does not?

  2. Can you report a joint event level success rate that requires a response to occur in the valid window and pass the content judgment, while also penalizing false interventions? This would make overall interaction performance easier to interpret.

7 comments
AuthorsOpenReview ↗

Response to Reviewer KeSE

We thank the reviewer for the careful reading and for recognizing the value of addressee tracking and of separating participation, timing, and content. We address both concerns below; all new results will be added to the revision.

W1 / Q2: Joint event-level success and false interventions.

We thank the reviewer for raising this point, which also made us notice that Eq. (5) misdescribes our implementation. In the code used for Table 4, an event without an assigned response receives a score of 0 and remains in the denominator, consistent with the judge prompts in Appendix A; Eq. (5) incorrectly restricts the average to events with an assigned response. We will correct Eq. (5) in the revision. Missed events therefore already lower the quality scores in Table 4. We agree, however, with the reviewer's broader point: quality does not account for timing, and WRR does not account for content, so reporting them separately makes overall interaction success hard to compare. We therefore add a joint metric and report false interventions:

  • Event Success Rate (ESR): an event counts as successful only if its first assigned response is on time (zjz_j = On-Time) and judged correct. The denominator is all response-required events, so missed, early, late, and incorrect responses all count as failures.
  • False Intervention Rate (FIR): the fraction of a model's responses that are not assigned to any response-required event, i.e., responses produced when no response is expected.

Table R1: Event Success Rate (%) ↑

ModelVTISPICQIORAAVATCATAvg
Native Audio-Visual Models
Aero-Realtime-4B1.63.32.10.40.31.51.53
MiniCPM-o 4.53.56.73.01.70.53.23.10
DuplexOmni3.63.53.81.63.53.03.17
Qwen3-Omni-30B-A3B-Instruct5.26.75.62.76.53.14.97
GPT-Realtime-2.165.318.433.930.939.628.736.13
Gemini-3.1-Flash-Live57.219.230.325.625.030.331.27
Qwen3.5-Omni-Plus-Realtime51.218.027.926.029.223.729.33
Streaming Visual Models with Timestamp-Aligned Text
StreamingVLM3.33.85.13.37.07.75.03
Mage-VL6.84.05.75.214.012.17.97
JoyAI-VL-Interaction7.45.85.18.98.010.87.67
MOSS-VL-Realtime4.13.13.53.92.97.24.12

Table R2: False Intervention Rate (%) ↓

ModelVTISPICQIORAAVATCATAvg
Native Audio-Visual Models
Aero-Realtime-4B8.59.07.58.39.68.58.57
MiniCPM-o 4.520.320.213.317.923.416.118.53
DuplexOmni15.914.818.519.922.425.819.55
Qwen3-Omni-30B-A3B-Instruct18.917.020.720.518.921.719.62
GPT-Realtime-2.12.83.23.02.73.12.12.82
Gemini-3.1-Flash-Live4.25.35.24.14.73.54.50
Qwen3.5-Omni-Plus-Realtime4.93.13.74.74.43.84.10
Streaming Visual Models with Timestamp-Aligned Text
StreamingVLM16.921.217.520.416.322.119.07
Mage-VL13.310.512.813.611.213.712.52
JoyAI-VL-Interaction22.417.416.316.423.621.719.63
MOSS-VL-Realtime15.821.315.318.222.619.618.77
AuthorsOpenReview ↗

Response to Reviewer KeSE

We highlight three findings.

  1. For proprietary models, the bottleneck is timing rather than content. Dividing ESR by WRR gives the probability that an on-time response is correct. For the three proprietary models this averages 72–83%, well above their conditional quality over all assigned responses (47–56%). On SPI, for example, 91–92% of their on-time responses are correct, compared with 54–61% of all their responses. Responses outside the valid window are therefore disproportionately wrong: when these models respond at the right moment, they mostly respond correctly, and their scores are limited mainly by when they respond (WRR 41–54%).
  2. For open models, being on time is largely hollow. Only 11–34% of their on-time responses are correct, i.e., 66–89% of the responses that WRR counts as timely fail the content judgment (vs. 17–28% for proprietary models). As a result, the gap between the best open model and the weakest proprietary model grows from about 2× under conditional quality (24.23 vs. 47.07) to about 4× under ESR (7.97 vs. 29.33). False interventions are likewise concentrated in open models (FIR 8.6–19.6%, vs. 2.8–4.5% for proprietary models).
  3. Quality alone can overrate a model whose correct responses are mistimed. Gemini-3.1-Flash-Live has the highest quality (56.38) but responds on time to fewer events than GPT-Realtime-2.1 (WRR 40.58 vs. 53.77), so its ESR is lower (31.27 vs. 36.13). The effect is largest on AVAT: Gemini responds correctly to 88.4% of events, but only 25.7% of events receive an on-time response, giving an ESR of 25.0, so most of its correct AVAT responses arrive outside the valid window. This is why a joint metric is needed: the decomposition shows that Gemini's content is strong, while ESR reflects that much of it is not delivered at the right moment.

Dear Reviewer KeSE,

Thank you again for your thoughtful feedback and for recognizing the value of DIVE’s interaction settings and evaluation framework. Following your suggestions, we have added Event Success Rate (ESR), which requires responses to be both timely and correct, together with False Intervention Rate (FIR). We have also clarified that missed events already receive zero scores in the reported quality results.

Regarding the comparison with OmniInteract, please also see Table R7 in our response to Reviewer SGWo, which provides a direct task-level comparison. Our accompanying analysis explains the additional capabilities evaluated by DIVE and the distinct timing and content failure patterns revealed by the new results.

We would greatly appreciate your feedback on these additions. If they resolve your concerns, we would be grateful if you could reconsider your overall assessment and rating accordingly. Please let us know if any aspects would benefit from further clarification.

Thank you for your time and consideration.

Comments on the paper

2 comments

Thank you for the detailed response. In my review I said that human validation of the content judge and an analysis of temporal uncertainty could change my assessment, and both are now provided. The 93.3% agreement with adjudicated human labels (κ = 0.844), close to the human–human agreement, addresses my main concern about the LLM judge. The window-sensitivity results and the scenario-clustered confidence intervals also make the model comparisons more convincing. Thank you as well for clarifying how missed events are scored; please make sure Eq. (5) is corrected in the revision. I am raising my score to 3. Writing and presentation still need work, and the new analyses should be moved into the paper rather than left only in the response.

Dear Reviewer GghH,

Thank you for carefully reviewing the additional analyses and updating your assessment. Your feedback has helped us strengthen the validation of DIVE’s evaluation, and we are glad to know that these additions have addressed your main concerns.

We commit to incorporating these analyses into the revised manuscript and its appendices. Specifically, we will:

  1. Include the reliability analyses and clarify the evaluation procedures;
  2. Correct Eq. (5) and ensure consistent descriptions of the scoring procedure throughout the paper;
  3. Reorganize our writing to improve clarity and consistency.

We plan to upload the revised manuscript soon.

Thank you again for your detailed feedback and updated assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.