acceptodds
Desk rejected at ICLR 2027

D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Spoken dialogue is a primary source of information in videos; therefore, accurately identifying who spoke what and when is essential for deep video understanding. We introduce D-ORCA, a dialogue-centric omni-modal large language model optimized for robust audio-visual captioning. We further curate DVD, a large-scale, high-quality bilingual dataset comprising nearly 40,000 multi-party dialogue videos for training and 2000 videos for evaluation in English and Mandarin, addressing a critical gap in the open-source ecosystem. To ensure fine-grained captioning accuracy, we adopt group relative policy optimization with three novel reward functions that assess speaker attribution accuracy, global speech content accuracy, and sentence-level temporal boundary alignment. These rewards are derived from evaluation metrics widely used in speech processing and, to our knowledge, are applied for the first time as reinforcement learning objectives for audio-visual captioning. Extensive experiments demonstrate that D-ORCA substantially outperforms existing open-source models in speaker identification, speech recognition, and temporal grounding. Notably, despite having only 8 billion parameters, D-ORCA achieves performance competitive with Qwen3-Omni across several general-purpose audio-visual understanding benchmarks. Demos are available at https://d-orca-llm.github.io/https://d-orca-llm.github.io/, and our code, data, and checkpoints will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
dialogue-centric video, audio-visual captioning, video understanding

Related papers

Discussion (0)

Sign in to comment.