- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OmniVChat combines a dialogue synthesis engine, a 2,800-instance audio-visual benchmark, and rubric-based RL. Training Qwen3-Omni on synthetic dialogues raises scores from 0.465 to 0.652 on the synthetic benchmark and from 0.402 to 0.632 on 360 human-recorded dialogues.
Critical strengths
- S1. Useful evaluation setting. Spoken queries are embedded in audio-visual input, and 17 subcategories cover grounding, dialogue state, and memory (§3). This targets capabilities missed by separate text-question evaluation.
- S2. Evidence of transfer. Gains on the held-out recordings, from 0.402 to 0.632 (Table 1), make the synthesis approach more convincing than synthetic-only improvements.
Critical weaknesses
- W1. The gains lack an independent quality check. The same judge supplies training rewards and evaluation scores. Appendix D.3 compares judges only on fixed Gemini replies, not base-versus-trained outputs, leaving reward-judge preferences as an unresolved explanation for the gains.
- W2. The data engine is insufficiently reproducible. Section 2 and Appendix C describe agent roles but omit the agent/renderer model identities and numerical repair limits. Section 9 promises benchmark and RL-code release, without committing to the synthesis engine. These omissions limit reuse of a central contribution.
Questions for authors
- Clarification (W1): Do available independent-judge or human ratings preserve the base-to-trained gains?
- Clarification (W2): Which agent/rendering models and repair limits were used, and which synthesis artifacts will be released?
6 comments
Response to Reviewer u2aQ (2/3): Data engine settings and release plans
Concern (Weakness 2; Question 2): Reuse of the synthesis engine requires concrete agent and renderer identities, numerical repair limits, and a synthesis-artifact release commitment.
To make the data engine easier to reproduce, we added settings for both single-turn and multi-turn synthesis. Table 4 lists them below.
| Module | Parameter | Single-turn | Multi-turn |
|---|---|---|---|
| Description | Corpus passages per sample | 3 | 3 |
| Text generation modules | Maximum output tokens per call | 8,192 | 8,192 |
| Video caption / Q-A | Maximum output tokens per call | 4,096 | 4,096 |
| Global control | Maximum control-loop steps | 20 | 120; MRR: 240 |
| Planning | Plan-review retries | — | 2 |
| Planning | Plan updates | — | 3 |
| Script / prompt review | Rewrites after review | 2 | 2 per turn |
| Video generation | Output resolution | 1080p | 1080p |
| Video generation | Maximum clip duration | 15 s | 15 s per turn |
| Video feedback | Seed-based regenerations | 2 | 2 per turn |
| Video feedback | Script / prompt rewrites | 1 | 1 per turn |
| Video Q-A | Follow-up questions per rendering | 3 | 3 |
| Reply review | Reply refinements | 1 | 1 per turn |
Table 5 gives mean calls and estimated USD costs per dialogue on 120 dialogue pairs, ten pairs per single-turn subcategory:
| Stage | Direct Generation calls | OmniVChat-Studio calls | Direct Generation USD | OmniVChat-Studio USD |
|---|---|---|---|---|
| Joint script and reply | 1.017 | — | 0.02 | — |
| Description | — | 1.567 | — | 0.10 |
| Planning | — | 1.000 | — | 0.13 |
| Script writing | — | 1.175 | — | 0.15 |
| Format repair | — | 0.000 | — | 0.00 |
| Validation review | — | 0.117 | — | 0.01 |
| Script review | — | 1.058 | — | 0.11 |
| Video planning | — | 1.017 | — | 0.10 |
| Video feedback | — | 1.767 | — | 0.32 |
| Reply planning | — | 1.000 | — | 0.20 |
| Reply writing | — | 1.067 | — | 0.19 |
| Reply review | — | 1.067 | — | 0.16 |
| Text calls, total | 1.017 | 10.833 | 0.02 | 1.47 |
| Multimodal review | — | 3.225 | — | 0.08 |
| Video generation | 1.000 | 1.508 | 4.20 | 6.43 |
| Total | 2.017 | 15.567 | 4.21 | 7.98 |
Regeneration and rewrite limits apply per script or multi-turn round; rewriting resets regeneration and question counters. Exhausted video or script repair budgets terminate the sample. Reply review permits one refinement before retaining the current reply, subject to final completeness checks. Single-turn rendering is set to 1080p for the reported comparison. Dialogue lengths follow subcategory configurations: MRR uses 6–10 turns, MDCC 2–5, and MVMS, MIST, and MTIC 2–3.
We added references to the language model and video generation model in Section C.3. In the paired comparison, OmniVChat-Studio costs USD 7.98 per dialogue, versus USD 4.21 for Direct Generation.
For security reasons, we cannot directly release OmniVChat-Studio's source code. However, we will release two agent skills: one to help users build OmniVChat-Studio and one to automatically add new subcategories. We describe this plan in Section 7.
In Section C.3.2, Figure 8 reports the seven-criterion quality comparison. Selected paired frames appear for Initial Connectivity Confirmation (Figure 9), Emotion Recognition (Figure 10), and Descriptive Reference (Figure 11), with both judges' preferences on the five video-quality criteria. Section C.3.4 gives the Direct Generation and video, script, and reply-comparison prompts.
Manuscript changes: We added the settings and stopping rules in Section C.3.1 (Table 4), model references in Section C.3, call counts and costs in Section C.3.3 (Table 5), and the two-skill release plan in Section 7.
Response to Reviewer u2aQ (3/3): Additional manuscript revisions
Other revisions: We added an SFT baseline and reward comparisons, including No Efficiency or Style Terms, in Section 4.3.1 (Table 1). SFT's lower score on recorded dialogues suggests that patterns learned from synthetic data may not transfer as well. Keeping Style improves style and efficiency without an observed loss in answer quality in this comparison. Section E.3 (Figure 14) shows the reward comparisons' training curves and checkpoint choices.
We added comparisons of native audio/video, ASR, ASR plus silent video, and joint audio-visual captions in Section 4.3.2 (Tables 2–3). The results suggest better use of native cues, partial text-to-native transfer, and less cross-modal interference in MSA; ER remains strongest with ASR. We revised the Abstract, Introduction (Section 1), and Conclusion (Section 5) to explain these findings and completed all 17 subcategory definitions in Section D.3.
Section B.2 now discusses MultiVox, AV-EMO-Reasoning, and All-in-One's character-aware movie understanding. Its closing paragraph clarifies the contributions of native dialogue data and an extensible pipeline for training and evaluation.
Response to Reviewer u2aQ (1/3): Independent validation of the training gains
Thank you for raising these concerns about evaluation and reproducibility. We appreciate your feedback.
Concern (Weakness 1; Question 1): Training rewards and evaluation scores use the same judge, and the previous comparison evaluated fixed Gemini replies. Independent judges or human ratings are needed to verify the Base-to-trained gains.
To check whether the gains hold with other evaluators, we added a six-judge comparison of Base and Full OmniVChat-RL and blind pairwise evaluation by three human raters.
Table 7 reports correctness scores before and after training, evaluated by six models using the same rubric and identity information. Δ is Full OmniVChat-RL minus Base; Gain is Δ/Base.
All six judges assign higher correctness scores to Full OmniVChat-RL on both benchmarks, supporting gains across evaluation models.
Table 8 reports 120 dialogues, ten per single-turn subcategory, shared by four pairwise comparisons. Full OmniVChat-RL is compared separately with Base, No Efficiency Term, No Style Term, and No Efficiency or Style Terms. For each pair, three human raters view the original audio and video and two randomly labeled replies: one from Full OmniVChat-RL and one from the other model in the pair.
Raters assess three criteria:
Manuscript changes: We replaced the earlier agreement analysis with a direct comparison of Base and Full OmniVChat-RL in Section E.4 (Table 7). We added the human protocol, separate criteria, and complete outcomes in Section E.5 (Table 8). The full response policy, rubric-grading prompt, and seven Style criteria are provided in Section E.6.