- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper builds the DIVE, a benchmark containing 525 evaluation instances across six tasks for continuous audio-visual interaction. The topic is relevant, and tasks such as addressee tracking and response adaptation have practical value. However, the submission does not establish a sufficiently clear research gap, substantive incremental contribution, or convincing empirical justification for the proposed benchmark. The paper mainly describes a collection of tasks and reports model scores, while the connection between the motivation, benchmark design, and resulting scientific insights remains weak.
Critical strengths
- The benchmark addresses meaningful interaction behaviors, particularly repeated action counting, response adaptation, and addressee-dependent participation.
Critical weaknesses
- Unclear motivation and limited novelty. Existing benchmarks such as OmniInteract and Omni-DuplexEval already evaluate continuous interaction, proactive responses, and interruption handling. The paper does not clearly identify what these evaluations miss or why addressing that gap requires DIVE’s particular task taxonomy. Assembling additional scenarios is potentially useful, but the paper provides limited evidence of a substantive advance in evaluation methodology or scientific understanding.
- Weak presentation and supporting evidence. The paper lacks a clear progression from existing limitations to design requirements and experimental validation. Figures illustrate intended interactions rather than demonstrating distinctive challenges or actual failures.
- Unverified visual and full-duplex requirements. No controlled experiments establish whether visual input or simultaneous perception is necessary. Some updates can be understood from speech alone. Prerecorded interruptions may also arrive before slower models begin speaking, making actual overlap uncertain.
- Insufficient validation and reproducibility. Annotation agreement, human validation of the judge, essential dataset statistics, inference settings, and aggregation procedures are missing.
Questions for authors
See weakness.
4 comments
Thanks for the rebuttal, which make comparisions clear. However, the paper is still poorly written. It fails to clearly explain the limitations of existing work in the main paper, the motivation for this study, or its contributions. The writing lacks a logical structure, and the claims are insufficiently supported. The figures and tables do not effectively convey the motivation or demonstrate the advantages of the proposed approach. Merely establishing a benchmark is too limited a contribution.
Request for specific remaining concerns
Dear Reviewer SGWo,
Thank you for your reply. We are glad that the rebuttal made the comparison with OmniInteract and Omni-DuplexEval clear, as the lack of a clear research gap relative to these benchmarks was the first weakness listed in your review.
Your reply does not mention your other two substantive concerns: whether visual input and simultaneous perception are necessary (W3), and validation and reproducibility (W4). We addressed these with a black-frame ablation (Table R6), human validation of the judge (93.3% agreement, κ = 0.844), annotator variability, window-sensitivity analysis, and bootstrap confidence intervals. Could you confirm whether these concerns are now resolved? If not, please tell us which result you find insufficient.
Your remaining comments concern the writing, but they are stated in general terms, and we cannot determine which parts of the paper they refer to. We are very willing to revise, and specific pointers would help greatly:
Limitations of existing work. Section 1, Section 2.2, and Table 1 discuss prior benchmarks. Which limitation is unclear, or which prior work is described inaccurately? We will move Table R7 into Section 2.2. Would that address this point? Motivation and contributions. Which part of the motivation do you find unconvincing, and which stated contribution is unclear? Logical structure. Which sections or transitions are hard to follow? Unsupported claims. Could you list the specific claims you consider insufficiently supported? For each one, we will either point to the supporting evidence or revise the claim. Figures and tables. Which figures or tables fail to convey the motivation? We will replace the illustrative figures with actual model failures, for example Gemini-3.1-Flash-Live on AVAT (quality 88.4 but ESR 25.0) and the black-frame ablation on VTI and CAT. Would this address your concern? Contribution of a benchmark. The paper is submitted under the datasets and benchmarks primary area, where a benchmark is the intended contribution. Beyond the tasks themselves, DIVE contributes paired-condition designs (ORA, AVAT), a joint evaluation of timing and content (ESR, FIR), and findings that existing benchmarks do not show: proprietary models fail mainly on timing while open models fail mainly on content, and proactive intervention degrades sharply when the trigger is spoken rather than visual (ESR 51.2–65.3 on VTI vs. 18.0–19.2 on SPI). What additional contribution would you expect from a benchmark paper in this area?
Finally, your review rates Soundness as 1 (“significant concerns … that do not appear easy to address”). If your remaining concerns are about presentation, could you clarify which soundness concerns still stand?
We appreciate your time and look forward to your specific feedback while the discussion is still open.
Best regards, The authors of Submission 9379
Dear Reviewer SGWo,
We have uploaded a revised manuscript that restructures the introduction around the limitations of existing benchmarks, the resulting design requirements, and their experimental validation, and moves the comparison with OmniInteract and Omni-DuplexEval and the validation analyses into the paper. We would still welcome specific pointers to any section, claim, figure, or table you find unclear or unsupported, and we will address each one while the discussion is open.
The authors
Dear Reviewer SGWo,
Thank you again for your detailed feedback. In our rebuttal, we have provided a task-level comparison with OmniInteract and Omni-DuplexEval, controlled visual-input ablations, and additional analyses of the interaction failures revealed by DIVE. We have also reported human validation of the judge and temporal robustness analyses, alongside clarifications of the dataset statistics, inference settings, aggregation procedures, and ORA evaluation setup.
We would greatly appreciate it if you could review our responses and the supporting results, particularly Tables R6 and R7, and let us know which concerns remain. Your feedback would help us understand whether further evidence or clarification is needed.
If these additions address your concerns sufficiently to change your assessment, we would be grateful if you could update your evaluation and rating accordingly.
Thank you for your time and consideration.