acceptodds
Under review as a conference paper at ICLR 2027

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

Abstract

We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, a good reply often needs to account for the user's surroundings, facial expressions, and nearby objects, and such responses can be expressed in many different ways, making keyword matching unreliable for evaluating reply quality. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-Visual Dialogue

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_u2aQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OmniVChat combines a dialogue synthesis engine, a 2,800-instance audio-visual benchmark, and rubric-based RL. Training Qwen3-Omni on synthetic dialogues raises scores from 0.465 to 0.652 on the synthetic benchmark and from 0.402 to 0.632 on 360 human-recorded dialogues.

Critical strengths
  • S1. Useful evaluation setting. Spoken queries are embedded in audio-visual input, and 17 subcategories cover grounding, dialogue state, and memory (§3). This targets capabilities missed by separate text-question evaluation.
  • S2. Evidence of transfer. Gains on the held-out recordings, from 0.402 to 0.632 (Table 1), make the synthesis approach more convincing than synthetic-only improvements.
Critical weaknesses
  • W1. The gains lack an independent quality check. The same judge supplies training rewards and evaluation scores. Appendix D.3 compares judges only on fixed Gemini replies, not base-versus-trained outputs, leaving reward-judge preferences as an unresolved explanation for the gains.
  • W2. The data engine is insufficiently reproducible. Section 2 and Appendix C describe agent roles but omit the agent/renderer model identities and numerical repair limits. Section 9 promises benchmark and RL-code release, without committing to the synthesis engine. These omissions limit reuse of a central contribution.
Questions for authors
  • Clarification (W1): Do available independent-judge or human ratings preserve the base-to-trained gains?
  • Clarification (W2): Which agent/rendering models and repair limits were used, and which synthesis artifacts will be released?
6 comments
AuthorsOpenReview ↗

Response to Reviewer u2aQ (1/3): Independent validation of the training gains

Thank you for raising these concerns about evaluation and reproducibility. We appreciate your feedback.

Concern (Weakness 1; Question 1): Training rewards and evaluation scores use the same judge, and the previous comparison evaluated fixed Gemini replies. Independent judges or human ratings are needed to verify the Base-to-trained gains.

To check whether the gains hold with other evaluators, we added a six-judge comparison of Base and Full OmniVChat-RL and blind pairwise evaluation by three human raters.

Table 7 reports correctness scores before and after training, evaluated by six models using the same rubric and identity information. Δ is Full OmniVChat-RL minus Base; Gain is Δ/Base.

JudgeBench BaseBench Full OmniVChat-RLΔGainBench-Human BaseBench-Human Full OmniVChat-RLΔGain
gpt-5.6-terra0.4660.637+0.17136.7%0.4080.628+0.22054.0%
gpt-5.6-luna0.4650.628+0.16435.2%0.4000.601+0.20150.4%
gpt-5.6-sol0.4780.656+0.17737.1%0.4020.640+0.23859.2%
qwen3.6-flash0.4790.665+0.18638.9%0.4640.648+0.18439.6%
qwen3.7-plus0.4690.654+0.18539.5%0.4260.645+0.21951.5%
qwen3.7-max0.4650.652+0.18740.2%0.4020.632+0.23057.2%

All six judges assign higher correctness scores to Full OmniVChat-RL on both benchmarks, supporting gains across evaluation models.

Table 8 reports 120 dialogues, ten per single-turn subcategory, shared by four pairwise comparisons. Full OmniVChat-RL is compared separately with Base, No Efficiency Term, No Style Term, and No Efficiency or Style Terms. For each pair, three human raters view the original audio and video and two randomly labeled replies: one from Full OmniVChat-RL and one from the other model in the pair.

Raters assess three criteria:

  • Content quality: Correctness, grounding in the audio-visual input, relevance, coverage of the user's needs, and helpfulness.
  • Efficiency: Expressing useful information more concisely while preserving content quality and respecting requests for detail.
  • Style: Following the shared response policy, including natural spoken language, fluent grammar, and an appropriate level of detail.
Comparator to Full OmniVChat-RLCriterionFull OmniVChat-RL preferredComparator preferredTieSplitIncomplete
BaseContent quality59361942
No Efficiency TermEfficiency9212646
No Style TermStyle50243772
No Efficiency or Style TermsEfficiency9511662
No Efficiency or Style TermsStyle91101342
  • Better content quality. Full OmniVChat-RL is preferred over Base in 49.2% of dialogues, compared with 30.0% for Base.
  • Higher reply efficiency. Full OmniVChat-RL is preferred over No Efficiency Term by 76.7% vs. 10.0%, and over No Efficiency or Style Terms by 79.2% vs. 9.2%.
  • Stronger Style compliance. Full OmniVChat-RL is preferred over No Style Term by 41.7% vs. 20.0%, and over No Efficiency or Style Terms by 75.8% vs. 8.3%.

Manuscript changes: We replaced the earlier agreement analysis with a direct comparison of Base and Full OmniVChat-RL in Section E.4 (Table 7). We added the human protocol, separate criteria, and complete outcomes in Section E.5 (Table 8). The full response policy, rubric-grading prompt, and seven Style criteria are provided in Section E.6.

AuthorsOpenReview ↗

Response to Reviewer u2aQ (2/3): Data engine settings and release plans

Concern (Weakness 2; Question 2): Reuse of the synthesis engine requires concrete agent and renderer identities, numerical repair limits, and a synthesis-artifact release commitment.

To make the data engine easier to reproduce, we added settings for both single-turn and multi-turn synthesis. Table 4 lists them below.

ModuleParameterSingle-turnMulti-turn
DescriptionCorpus passages per sample33
Text generation modulesMaximum output tokens per call8,1928,192
Video caption / Q-AMaximum output tokens per call4,0964,096
Global controlMaximum control-loop steps20120; MRR: 240
PlanningPlan-review retries—2
PlanningPlan updates—3
Script / prompt reviewRewrites after review22 per turn
Video generationOutput resolution1080p1080p
Video generationMaximum clip duration15 s15 s per turn
Video feedbackSeed-based regenerations22 per turn
Video feedbackScript / prompt rewrites11 per turn
Video Q-AFollow-up questions per rendering33
Reply reviewReply refinements11 per turn

Table 5 gives mean calls and estimated USD costs per dialogue on 120 dialogue pairs, ten pairs per single-turn subcategory:

StageDirect Generation callsOmniVChat-Studio callsDirect Generation USDOmniVChat-Studio USD
Joint script and reply1.017—0.02—
Description—1.567—0.10
Planning—1.000—0.13
Script writing—1.175—0.15
Format repair—0.000—0.00
Validation review—0.117—0.01
Script review—1.058—0.11
Video planning—1.017—0.10
Video feedback—1.767—0.32
Reply planning—1.000—0.20
Reply writing—1.067—0.19
Reply review—1.067—0.16
Text calls, total1.01710.8330.021.47
Multimodal review—3.225—0.08
Video generation1.0001.5084.206.43
Total2.01715.5674.217.98

Regeneration and rewrite limits apply per script or multi-turn round; rewriting resets regeneration and question counters. Exhausted video or script repair budgets terminate the sample. Reply review permits one refinement before retaining the current reply, subject to final completeness checks. Single-turn rendering is set to 1080p for the reported comparison. Dialogue lengths follow subcategory configurations: MRR uses 6–10 turns, MDCC 2–5, and MVMS, MIST, and MTIC 2–3.

We added references to the language model and video generation model in Section C.3. In the paired comparison, OmniVChat-Studio costs USD 7.98 per dialogue, versus USD 4.21 for Direct Generation.

For security reasons, we cannot directly release OmniVChat-Studio's source code. However, we will release two agent skills: one to help users build OmniVChat-Studio and one to automatically add new subcategories. We describe this plan in Section 7.

In Section C.3.2, Figure 8 reports the seven-criterion quality comparison. Selected paired frames appear for Initial Connectivity Confirmation (Figure 9), Emotion Recognition (Figure 10), and Descriptive Reference (Figure 11), with both judges' preferences on the five video-quality criteria. Section C.3.4 gives the Direct Generation and video, script, and reply-comparison prompts.

Manuscript changes: We added the settings and stopping rules in Section C.3.1 (Table 4), model references in Section C.3, call counts and costs in Section C.3.3 (Table 5), and the two-skill release plan in Section 7.

AuthorsOpenReview ↗

Response to Reviewer u2aQ (3/3): Additional manuscript revisions

Other revisions: We added an SFT baseline and reward comparisons, including No Efficiency or Style Terms, in Section 4.3.1 (Table 1). SFT's lower score on recorded dialogues suggests that patterns learned from synthetic data may not transfer as well. Keeping Style improves style and efficiency without an observed loss in answer quality in this comparison. Section E.3 (Figure 14) shows the reward comparisons' training curves and checkpoint choices.

We added comparisons of native audio/video, ASR, ASR plus silent video, and joint audio-visual captions in Section 4.3.2 (Tables 2–3). The results suggest better use of native cues, partial text-to-native transfer, and less cross-modal interference in MSA; ER remains strongest with ASR. We revised the Abstract, Introduction (Section 1), and Conclusion (Section 5) to explain these findings and completed all 17 subcategory definitions in Section D.3.

Section B.2 now discusses MultiVox, AV-EMO-Reasoning, and All-in-One's character-aware movie understanding. Its closing paragraph clarifies the contributions of native dialogue data and an extensible pipeline for training and evaluation.

Reviewer_iDxDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Targeting the task of Native Audio-Visual Dialogue, this paper constructs OmniVChat, a unified framework covering dialogue data synthesis, capability evaluation, and reinforcement learning training. Specifically, the paper proposes OmniVChat-Studio, a synthetic data pipeline incorporating generation, post-hoc review, and validation mechanisms; OmniVChat-Bench, a hierarchical rubric-based benchmark containing 2800 synthetic samples; and OmniVChat-RL, a post-training scheme combining rubric, format, efficiency, and style rewards. The main contribution of this work lies in combining controllable audio-visual dialogue synthesis, explicit hierarchical evaluation, and multi-objective reinforcement learning training, while providing empirical support on both the 2800 sample synthetic test set and a 360 sample human-recorded single-turn test set that synthetic dialogue data improves model performance on human-recorded inputs. Ablation studies also reveal the trade-off between response correctness and conciseness. Overall, this appears to be a promising systematic work.

Critical strengths
  1. The paper focuses on specific interaction problems in native audio-visual dialogue and makes them quantifiably evaluable through 17 subcategories and hierarchical gated rubrics. In terms of data construction, OmniVChat-Studio adopts a post-hoc observation, validation, and repair pipeline based on the actually generated video visuals, and maintains the initial scene constraints during the repair process, effectively mitigating the text-visual/audio-visual inconsistency issues common in synthetic data.

  2. After RL training on the synthetic data, the model not only achieves improvements on the synthetic test set, but also demonstrates clear positive transfer on the 360 sample human recorded single-turn test set.

Critical weaknesses
  1. The paper currently seems to lack an SFT baseline on the same training data, as well as a rubric-only RL baseline that removes both efficiency and style rewards simultaneously. Therefore, it cannot yet sufficiently distinguish the respective contributions of the synthetic data, the training paradigm, and the full reward combination. Furthermore, the training reward and test evaluation share the same rubric mechanism and judge model. Although I noticed that the appendix validates the scoring consistency of different judges on the same set of model responses, the paper does not directly verify the performance gains before and after training via independent judges, external benchmarks, or blind human evaluation, making it difficult to confirm whether sacrificing part of the task quality score in exchange for briefer responses (by adding the efficiency reward) aligns with real human preferences.

  2. Some rubric scoring items bind objective audio-visual understanding with specific response styles, and dimensions such as model identity and physical limitations also seem to rely more on system instruction settings than on audio-visual fusion capabilities. In addition, the human-recorded test set only contains Chinese single-turn samples recorded according to guidelines, and the multi-turn evaluation also adopts an offline scoring approach with fixed historical context. Furthermore, I believe it would be beneficial to supplement information in the paper regarding the underlying model versions of each component, the synthesis pass rate, the generation cost per sample, and the multi-turn historical reference responses.

Questions for authors
  1. On the same 5600 synthetic training samples, how do a standard SFT model and an RL model using only the rubric reward perform? Furthermore, if a simplified data generation pipeline is adopted (for example, without post-hoc review by the Reviewer) or the input modality is changed (transcribed text+video?), how much difference would there be in performance?
  2. If evaluated using other judge models not involved in the training process, tested on external public audio-visual dialogue benchmarks, or assessed via a small-scale blind human evaluation comparing base model vs full RL vs RL without efficiency reward, do the training-induced improvements and the efficiency-quality trade-off still hold?
  3. Please provide additional details on the specific model versions called by each module of OmniVChat-Studio, the first-pass acceptance rate, the repair discard rate, the average generation cost per data item, and the human review workload. Meanwhile, do the authors plan to release the historical reference responses used in the multi-turn evaluation to support full reproducibility?
  4. Merely as a suggestion, core audio-visual question answering correctness and proactively providing additional visual information / social expression style could be reported separately in the evaluation results to measure the model's audio-visual perception capabilities more clearly.
  5. It is suggested to supplement discussions on related works such as MultiVox (EMNLP 2025), AV-EMO-Reasoning (2025), and All in One (2026) to more accurately define the contributions of this paper.
11 comments
AuthorsOpenReview ↗

Response to Reviewer iDxD (1/8): SFT baseline and reward comparisons

Thank you for the detailed comments and helpful suggestions. They helped us improve the paper.

Concern (Weakness 1; Question 1): The original comparison lacks same-data SFT and an RL baseline removing both efficiency and style.

To compare the training methods and rewards, we added SFT trained on the same data and No Efficiency or Style Terms to Table 1. SFT uses the same OmniVChat-Bench-Train data and LoRA settings as Full OmniVChat-RL. It runs for 1,000 steps, and we select step 720 because it has the highest Subcategory Mean on the development set. The table below shows all scores for Base and the trained models.

ModelAHERMSAMEADSLPSingleMultiMeanHumanREStyle
Base0.1860.6940.3880.5730.4510.4500.5080.4650.4025.750.710
SFT0.7730.8010.7870.7450.6630.7340.7190.7320.64311.630.650
Full OmniVChat-RL0.6670.6510.7340.7130.5730.6640.6380.6520.63218.380.992
No Efficiency Term0.7330.8130.8100.7460.5860.7100.6770.6970.6917.120.987
No Style Term0.6720.7190.7580.7300.5600.6740.6450.6610.63213.700.788
No Efficiency or Style Terms0.7010.8800.7960.7710.5770.7130.6640.6950.6425.060.641

SFT may overfit more easily than RL. Its score drops from 0.732 on Bench to 0.643 on Human, a drop of 0.089. This suggests that patterns learned from synthetic data may not work as well in recorded conversations.

Keeping the Style reward improves style and efficiency while preserving answer quality in this comparison. No Efficiency Term scores higher than No Efficiency or Style Terms on Bench Mean (0.697 vs. 0.695), Human (0.691 vs. 0.642), RE (7.12 vs. 5.06), and Style (0.987 vs. 0.641). Answer scores also increase, so we see no loss in answer quality in this comparison.

Manuscript changes: We added these two models to Section 4.3.1 (Table 1), where we explain SFT's lower score on real recordings and the benefit of keeping Style. We added training curves for the model without efficiency or Style rewards and explain how we chose its checkpoint in Section E.3 (Figure 14).

AuthorsOpenReview ↗

Response to Reviewer iDxD (2/8): Independent judges and blind human preferences

Concern (Weakness 1; Question 2): Sharing the training judge and rubric with evaluation may favor the trained model, while the efficiency-quality tradeoff requires evidence from independent evaluators and human preferences.

To check whether the gains hold with other evaluators, we added a six-judge comparison of Base and Full OmniVChat-RL and blind pairwise evaluation by three human raters.

Table 7 reports correctness scores before and after training, evaluated by six models using the same rubric and identity information. Δ is Full OmniVChat-RL minus Base; Gain is Δ/Base.

JudgeBench BaseBench Full OmniVChat-RLΔGainBench-Human BaseBench-Human Full OmniVChat-RLΔGain
gpt-5.6-terra0.4660.637+0.17136.7%0.4080.628+0.22054.0%
gpt-5.6-luna0.4650.628+0.16435.2%0.4000.601+0.20150.4%
gpt-5.6-sol0.4780.656+0.17737.1%0.4020.640+0.23859.2%
qwen3.6-flash0.4790.665+0.18638.9%0.4640.648+0.18439.6%
qwen3.7-plus0.4690.654+0.18539.5%0.4260.645+0.21951.5%
qwen3.7-max0.4650.652+0.18740.2%0.4020.632+0.23057.2%

All six judges assign higher correctness scores to Full OmniVChat-RL on both benchmarks, supporting gains across evaluation models.

Table 8 reports 120 dialogues, ten per single-turn subcategory, shared by four pairwise comparisons. Full OmniVChat-RL is compared separately with Base, No Efficiency Term, No Style Term, and No Efficiency or Style Terms. For each pair, three human raters view the original audio and video and two randomly labeled replies: one from Full OmniVChat-RL and one from the other model in the pair.

Raters assess three criteria:

  • Content quality: Correctness, grounding in the audio-visual input, relevance, coverage of the user's needs, and helpfulness.
  • Efficiency: Expressing useful information more concisely while preserving content quality and respecting requests for detail.
  • Style: Following the shared response policy, including natural spoken language, fluent grammar, and an appropriate level of detail.
Comparator to Full OmniVChat-RLCriterionFull OmniVChat-RL preferredComparator preferredTieSplitIncomplete
BaseContent quality59361942
No Efficiency TermEfficiency9212646
No Style TermStyle50243772
No Efficiency or Style TermsEfficiency9511662
No Efficiency or Style TermsStyle91101342
  • Better content quality. Full OmniVChat-RL is preferred over Base in 49.2% of dialogues, compared with 30.0% for Base.
  • Higher reply efficiency. Full OmniVChat-RL is preferred over No Efficiency Term by 76.7% vs. 10.0%, and over No Efficiency or Style Terms by 79.2% vs. 9.2%.
  • Stronger Style compliance. Full OmniVChat-RL is preferred over No Style Term by 41.7% vs. 20.0%, and over No Efficiency or Style Terms by 75.8% vs. 8.3%.

Manuscript changes: We replaced the earlier agreement analysis with a direct comparison of Base and Full OmniVChat-RL in Section E.4 (Table 7). We added the human protocol, separate criteria, and complete outcomes in Section E.5 (Table 8). The full response policy, rubric-grading prompt, and seven Style criteria are provided in Section E.6.

AuthorsOpenReview ↗

Response to Reviewer iDxD (3/8): What the benchmark measures and what changes with the input

Concern (Weakness 2; Questions 1 and 4): Some tasks and rubrics mix perception with dialogue behavior or style, and changing the input format may reveal different capabilities.

To see how the input affects the answer, we compared replies to the same 2,800 Bench dialogues before and after training under four input formats, using the same rubric.

B = Base; F = Full OmniVChat-RL. FB uses F's weights to answer from B's ASR transcripts or joint audio-visual captions. B and F generate their own conversions. N = native audio/video; T = ASR; VT = T plus silent video; J = joint audio-visual captions. P_T = percentage with max(N, VT, J) > T: the share of dialogues where at least one input with information beyond ASR produces a higher-scoring reply than ASR alone.

Table 2: Bench results (Section 4.3.2).

CategoryB:NB:TB:VTB:JB:P_TF:NF:TF:VTF:JF:P_TFB:TFB:VTFB:JFB:P_T
DSLP0.4510.4020.4380.50039.2%0.5730.4360.5660.59441.1%0.4390.5580.58341.2%
MEA0.5730.2840.5250.57666.8%0.7130.2090.6340.61475.6%0.2190.6470.58975.8%
MSA0.3880.4350.2970.37131.8%0.7340.7110.7000.74836.7%0.7060.6990.72137.1%
AH0.1860.1140.1260.15634.3%0.6670.2330.5750.44475.5%0.2390.5760.41475.5%
ER0.6940.7140.6260.69433.3%0.6510.7260.6430.70326.7%0.7010.6230.68832.0%
Mean0.4650.3880.4300.48045.9%0.6520.4410.6150.61555.6%0.4440.6160.60056.1%

Table 3 shows N−J, the native-versus-text interaction gap; VT−T, the effect of adding video; and N−VT, native speech's benefit over ASR. N:F−B is the native score gain after RL. J:FB−B indirectly reflects improved text replies with fixed captions; J:F−FB may partly reflect perceptual gains through changed captions. P_T:F−B is the change in P_T. Values are differences on a percentage scale, not relative growth.

Table 3: score differences (Section 4.3.2).

CategoryB:N−JB:VT−TB:N−VTF:N−JF:VT−TF:N−VTN:F−BJ:FB−BJ:F−FBP_T:F−B
DSLP−4.9%3.6%1.3%−2.1%13.0%0.7%12.2%8.3%1.1%1.9%
MEA−0.3%24.1%4.8%9.9%42.5%7.9%14.0%1.3%2.5%8.8%
MSA1.7%−13.8%9.1%−1.4%−1.1%3.4%34.6%35.0%2.7%4.9%
AH3.0%1.2%6.0%22.3%34.2%9.2%48.1%25.8%3.0%41.2%
ER0.0%−8.8%6.8%−5.2%−8.3%0.8%−4.3%−0.6%1.5%−6.6%

What the results show:

  • DSLP: N and J both improve, and their gap shrinks from 0.049 to 0.021. This suggests better text replies and better use of those abilities in native dialogue.
  • MEA: After training, N exceeds J by 0.099, and P_T rises from 66.8% to 75.6%, suggesting better use of audio-visual cues.
  • MSA: Adding video lowers the score by 0.138 before training and only 0.011 afterward. This suggests less cross-modal interference.
  • AH: N rises from 0.186 to 0.667, and P_T from 34.3% to 75.5%, showing large gains with native input.
  • ER: T remains the highest-scoring format, while N falls from 0.694 to 0.651. This suggests more reliance on speech content when judging emotion.

Additional context: When we designed the benchmark, we focused more on how people use audio and video to interact in daily life than on whether every question needs both to answer it. Our experiments show that these tasks are still worth training and testing. Some questions can be answered with just one modality, but models can still struggle when given both: the extra input can get in the way, and the model may not fully use the skills it already has with text.

We think the results above may be because the model has not seen enough native audio-visual training data, so its existing skills have not fully carried over. This is where OmniVChat-Studio can help: it creates training data for these tasks.

Manuscript changes: We added input comparisons in Section 4.3.2 (Tables 2–3) and separate human content judgments in Section E.5 (Table 8).

Reviewer_KaiNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies dialogue in which the user supplies audio and video directly and the model produces a text reply. It introduces OmniVChat-Studio, an agent-based synthesis pipeline with planning, audiovisual rendering, verification, and rubric construction; OmniVChat-Bench, with 2,800 synthetic dialogues spanning 17 subcategories; and an RL recipe combining correctness, reply efficiency, and style rewards. Training Qwen3-Omni-Instruct on 5,600 synthetic dialogues improves its reported benchmark score from 0.465 to 0.652 and its score on 360 human-recorded Chinese dialogues from 0.402 to 0.632. The evaluation also compares released models and removes individual reward terms to characterize the correctness/efficiency/style trade-off.

Critical strengths

The paper measures transfer to actual recordings and exposes meaningful reward trade-offs. The data engine tackles a practical bottleneck with explicit grounding checks. The pipeline, tiered rubric, reward, and data splits are described clearly, with useful construction examples and supplementary audit details.

Critical weaknesses

The paper showed insufficient evidence for multimodal necessity and the justification of pipeline complexity. It does not sufficiently establish how much of the benchmark requires joint audio–visual understanding beyond transcribed speech and straightforward visual descriptions. Although the synthesis pipeline checks visual necessity qualitatively, controlled modality ablations are missing. Some appendix examples reinforce this concern, the core criteria can be satisfied without visual cue/paralinguistic/facial expression/emotion evidence. Moreover, references and rubrics for the human-recorded set are generated from corrected captions and transcripts, making a caption-and-transcript baseline particularly informative. The existing ASR-cascade baseline does not isolate these factors under matched conditions.

The current ablations mainly examine reward components. Controlled experiments should quantify the performance drop when removing visual input or replacing audio with transcripts, and report the proportion of examples that genuinely require both modalities to satisfy the core correctness criteria. Separately, the benefits of the elaborate data-generation pipeline should be isolated through comparison with a simpler generation baseline.

Questions for authors
  1. Per-category comparisons using transcript-only, transcript-plus-video, and automatically generated captions plus transcripts would be helpful. Could the authors report these results by task category? Can a text-based agent prompted to respond in a friendly and empathetic manner satisfy most rubric criteria using only ASR transcripts? How does this baseline compare with native audio–visual models?

  2. What proportion of benchmark examples require both modalities to satisfy the core correctness criteria? Specifically, which criteria require nonverbal acoustic cues or visual evidence beyond what is available in the transcript?

  3. What measurable benefits does the multi-agent data-generation pipeline provide over a simpler generation baseline in terms of data quality, multimodal necessity, and downstream performance, considering its cost?

8 comments
AuthorsOpenReview ↗

Response to Reviewer KaiN (1/5): Input-format comparisons across ability categories

Thank you for the careful review. Your questions helped us see where we needed stronger evidence.

Concern (Weaknesses 1 and 2; Question 1): The input comparison needs matched controls by category, including friendly, empathetic transcript-only replies, transcript-plus-video, and captions/transcripts.

Following your suggestion, we added a comparison across input formats. We tested the same 2,800 Bench dialogues before and after training, using the same rubric for all four formats.

B = Base; F = Full OmniVChat-RL. FB uses F's weights to answer from B's ASR transcripts or joint audio-visual captions. B and F generate their own conversions. N = native audio/video; T = ASR; VT = T plus silent video; J = joint audio-visual captions. P_T = percentage with max(N, VT, J) > T: the share of dialogues where at least one input with information beyond ASR produces a higher-scoring reply than ASR alone.

Table 2: Bench results (Section 4.3.2).

CategoryB:NB:TB:VTB:JB:P_TF:NF:TF:VTF:JF:P_TFB:TFB:VTFB:JFB:P_T
DSLP0.4510.4020.4380.50039.2%0.5730.4360.5660.59441.1%0.4390.5580.58341.2%
MEA0.5730.2840.5250.57666.8%0.7130.2090.6340.61475.6%0.2190.6470.58975.8%
MSA0.3880.4350.2970.37131.8%0.7340.7110.7000.74836.7%0.7060.6990.72137.1%
AH0.1860.1140.1260.15634.3%0.6670.2330.5750.44475.5%0.2390.5760.41475.5%
ER0.6940.7140.6260.69433.3%0.6510.7260.6430.70326.7%0.7010.6230.68832.0%
Mean0.4650.3880.4300.48045.9%0.6520.4410.6150.61555.6%0.4440.6160.60056.1%

Table 3 shows N−J, the native-versus-text interaction gap; VT−T, the effect of adding video; and N−VT, native speech's benefit over ASR. N:F−B is the native score gain after RL. J:FB−B indirectly reflects improved text replies with fixed captions; J:F−FB may partly reflect perceptual gains through changed captions. P_T:F−B is the change in P_T. Values are differences on a percentage scale, not relative growth.

Table 3: score differences (Section 4.3.2).

CategoryB:N−JB:VT−TB:N−VTF:N−JF:VT−TF:N−VTN:F−BJ:FB−BJ:F−FBP_T:F−B
DSLP−4.9%3.6%1.3%−2.1%13.0%0.7%12.2%8.3%1.1%1.9%
MEA−0.3%24.1%4.8%9.9%42.5%7.9%14.0%1.3%2.5%8.8%
MSA1.7%−13.8%9.1%−1.4%−1.1%3.4%34.6%35.0%2.7%4.9%
AH3.0%1.2%6.0%22.3%34.2%9.2%48.1%25.8%3.0%41.2%
ER0.0%−8.8%6.8%−5.2%−8.3%0.8%−4.3%−0.6%1.5%−6.6%

What the results show:

  • DSLP: N and J both improve, and their gap shrinks from 0.049 to 0.021. This suggests better text replies and better use of those abilities in native dialogue.
  • MEA: After training, N exceeds J by 0.099, and P_T rises from 66.8% to 75.6%, suggesting better use of audio-visual cues.
  • MSA: Adding video lowers the score by 0.138 before training and only 0.011 afterward. This suggests less cross-modal interference.
  • AH: N rises from 0.186 to 0.667, and P_T from 34.3% to 75.5%, showing large gains with native input.
  • ER: T remains the highest-scoring format, while N falls from 0.694 to 0.651. This suggests more reliance on speech content when judging emotion.

Manuscript changes: We added these comparisons and analyses in Section 4.3.2 (Tables 2–3).

AuthorsOpenReview ↗

Response to Reviewer KaiN (2/5): Benchmark scope and evidence beyond transcripts

Concern (Weaknesses 1 and 2; Question 2): Some examples appear answerable without visual or nonverbal acoustic cues. The criteria that require these cues and the share of dialogues requiring both modalities remain unclear.

To address the proportion question, we focus on how often information beyond ASR helps the model answer better. The table below repeats only the P_T columns from Table 2.

P_T is the percentage of dialogues where at least one input format scores higher than ASR alone: native audio/video, ASR with silent video, or joint audio-visual captions. B = Base; F = Full OmniVChat-RL; FB uses F's weights to answer from B's ASR transcripts or joint captions.

CategoryB:P_TF:P_TFB:P_T
DSLP39.2%41.1%41.2%
MEA66.8%75.6%75.8%
MSA31.8%36.7%37.1%
AH34.3%75.5%75.5%
ER33.3%26.7%32.0%
Mean45.9%55.6%56.1%

P_T rises in four of the five categories after RL. The reported Mean increases from 45.9% to 55.6%, suggesting that training helps the model benefit from information beyond the transcript more often.

  • AH shows the largest increase, from 34.3% to 75.5%. After training, extra information helps the model give a better answer in about three quarters of these dialogues.
  • MEA rises from 66.8% to 75.6%. Information beyond ASR already helps in many dialogues before training, and this benefit becomes more frequent after RL.
  • DSLP and MSA show smaller increases: 39.2% to 41.1% and 31.8% to 36.7%, respectively. Extra information helps more often, although the change is less pronounced than in AH.
  • ER falls from 33.3% to 26.7%. This matches the input-format results in our preceding reply, where ASR remains the highest-scoring format.

Using Base's text conversions gives a similar pattern: FB also exceeds Base in four categories, with a Mean P_T of 56.1%. This suggests that the trained model's replies contribute to the improvement, alongside changes in the text conversions.

Additional context: When we designed the benchmark, we focused more on how people use audio and video to interact in daily life than on whether every question needs both to answer it. Our experiments show that these tasks are still worth training and testing. Some questions can be answered with just one modality, but models can still struggle when given both: the extra input can get in the way, and the model may not fully use the skills it already has with text.

We think the results above may be because the model has not seen enough native audio-visual training data, so its existing skills have not fully carried over. This is where OmniVChat-Studio can help: it creates training data for these tasks.

Manuscript changes: In Section 4.3.2, we added P_T to Table 2, its training changes to Table 3, and the category-level analysis.

AuthorsOpenReview ↗

Response to Reviewer KaiN (3/5): Direct Generation comparison: complete quality results

Concern (Weaknesses 1 and 2; Question 3): The multi-agent pipeline needs comparison with a simpler generator to assess data quality, multimodal necessity, downstream performance, and cost.

To compare the quality of the two pipelines, we evaluated 120 dialogue pairs, ten per single-turn subcategory, sharing corpus passages and task requirements. Direct Generation writes a script and reference reply in one language-model call, then renders the unchanged script. Figure 8 reports the following percentages. Qwen denotes Qwen3.5-Omni-Plus; Gemini denotes Gemini-3.7-Flash. For each judge, S/D/T/U mean OmniVChat-Studio preferred, Direct Generation preferred, tie, and unjudgeable.

CriterionQwen SQwen DQwen TQwen UGemini SGemini DGemini TGemini U
Device Viewpoint81.714.24.20.072.53.324.20.0
Everyday Capture Style74.29.216.70.076.78.315.00.0
Visual Consistency55.07.536.70.843.37.549.20.0
Dialogue Naturalness62.515.022.50.069.210.820.00.0
Fewer AI-Like Traces75.08.310.06.746.78.345.00.0
Script Alignment8.35.885.80.014.25.080.80.0
Reply Preference22.520.057.50.026.75.068.30.0

Two judges assess randomly labeled A/B outputs. The first five criteria assess video quality, Script Alignment checks subcategory conformity, and Reply Preference evaluates each reply in its own video's user context. Both judges favor OmniVChat-Studio on all seven criteria, with the clearest differences in video quality. Most script and reply comparisons are ties.

The generation and comparison prompts are provided in Section C.3.4.

Manuscript changes: We added the paired baseline in Section C.3 and quality results in Section C.3.2 (Figure 8). In Section C.3.2, we also added selected video comparisons for connection confirmation in Figure 9, emotion recognition in Figure 10, and descriptive reference in Figure 11. These show frames from both pipelines and both judges' choices on the five video criteria.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.