- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces ORAV Bench, a benchmark for evaluating audio-video generation conditioned on multiple image, video, and audio references. It contains 380 instances covering 9 semantic roles and 30 reference compositions. The authors also propose a reference-aware pairwise evaluation protocol that uses structured visual and auditory evidence to evaluate reference fidelity and task fulfillment. Experiments on five frontier generation systems reveal different strengths across reference compositions and identify reference substitution as a common failure mode.
Critical strengths
- The paper studies an important and timely problem in audio-video generation. I agree that reference similarity alone is insufficient to evaluate whether a model correctly combines information from multiple references. The benchmark covers diverse reference types and compositions, which could be useful for evaluating current generation systems.
- The authors propose a carefully designed reference-aware pairwise evaluator. The ablation in Appendix A.9 shows that the structured protocol improves human agreement by 7.45 percentage points over a simplified prompt using the same model and evidence. I think this is a meaningful result that supports the usefulness of the proposed evaluation method.
Critical weaknesses
- The novelty over existing benchmarks is not sufficiently clear. MultiRef-Compass already evaluates reference consistency, compositional binding, and instruction following using a sophisticated MLLM-based evaluator, with a comparable dataset size (350 vs. 380 instances). While ORAV covers more diverse reference types, I would like to see a more direct comparision with existing evaluation methods to justify its novelty. Without this comparison, I am not fully convinced that the proposed evaluation method provides substantial improvements over prior work.
- The reliability of the proposed evaluation protocol is not sufficiently validated. Although the paper formalizes positive requirements and negative constraints, these semantic reference contracts are not explicitly annotated, and the judge must infer them from the instructions and references. This makes the evaluation heavily dependent on the judge's interpretation. I think the benchmark would be substantially more reliable if these constraints were explicitly annotated, potentially using MLLMs followed by human verification. Some constraints could also be evaluated more reliably and efficiently using specialized tools, such as object detectors or ASR. Moreover, only 38 instances are evaluated by human annotators, and the paper does not report human-human agreement or judge-human agreement across different reference types. Since the proposed evaluator is one of the main contributions, I think stronger validation of its reliability is important.
Questions for authors
Here are my main questions and concerns. I would be happy to change my opinion if the major concerns are resolved.
Q1: Could the authors clarify the main novelty compared with MultiRef-Compass, especially regarding the evaluation methodology? It would be helpful to compare the proposed judge with existing MLLM-based evaluators on the same instances and human annotations. This would help understand whether the proposed evaluation protocol provides meaningful improvements over existing methods.
Q2: Why are the positive requirements and negative constraints not explicitly annotated? I think using a strong MLLM to generate these annotations followed by human verification should be feasible. Have the authors considered this approach? It would also be interesting to compare evaluation results with and without explicit annotations, to understand how much the reliability depends on the judge's own interpretation.
Q3: Could the authors report human-human agreement and provide a breakdown of judge-human agreement across different reference types? This would help understand how reliable the evaluator is, particularly for motion and audio references. Since the current human validation only covers 38 instances, I think this is important to support the reliability of the proposed benchmark.
7 comments
Part 2/3: Response to Reviewer RvD9
Q2. Reference-Contract Interpretation & Specialized Tools
1. On-the-fly reference-contract interpretation. Thank you for this suggestion. We have considered MLLM-assisted contract annotation with human verification. While pre-extracting explicit requirement lists is effective for verifying isolated visual or lexical clauses—as explored in MultiRef-Compass—we intentionally couple on-the-fly contract interpretation with pairwise preference reasoning for three reasons:
- Contextual cross-reference interactions and weights. In open-ended multimodal compositions, what should be transferred, adapted, or excluded from one reference depends relationally on co-occurring references and the target scene, such as permitted pose, lighting, or timing adaptations when binding motion and speech references to a new subject and environment. Exhaustively pre-specifying every cross-reference interaction and fixed aggregation weights ahead of time is challenging, as static lists can miss relational nuances or over-constrain valid variations.
- Resolving task-dependent pairwise trade-offs. Candidates on omni-reference tasks rarely succeed or fail cleanly on the same requirement; one candidate may better preserve camera framing or attire while another more faithfully captures defining motion mechanics or avoids source-arena leakage, as in Fig. 3. Even with an explicit list of constraints, determining the winner still requires reasoning about each discrepancy's magnitude, scope, and consequence for the complete instructed event. As the Q1 comparison with MultiRef-Compass illustrates, scoring pre-extracted checklists under fixed weights frequently collapses competing partial fulfillments into score ties—especially after rejudging—limiting strict human agreement to 45.42%–47.51% compared with 81.43% for ORAV Judge. Conversely, our coupled protocol achieves 85.46% non-tie human agreement overall and 92.17% on comparisons decided at the usability-veto stage.
- Direct generalizability to new natural-language tasks. Interpreting contracts on the fly from the instruction and reference media allows ORAV Judge to be applied directly to new open-ended prompts and reference combinations without a separate manual contract-annotation stage, while remaining compatible with human-verified contracts as an optional audit resource. We will clarify this design rationale in the revision.
2. Role of specialized tools. We agree that specialized tools offer efficient, reproducible measurements where direct signal extraction applies. Indeed, Table 3 and Appendices A.3–A.5 already incorporate 15 specialized pretrained tools across all 1,852 candidates, covering ASR word/character error rates (Whisper), speaker similarity (ECAPA-TDNN), recording replay (waveform cross-correlation), localized object, subject, face, and scene affinity (GroundingDINO, SAM2.1, DINO, CLIP, ArcFace), and motion trajectories (CoTracker3). As shown in Appendix A.4, however, localized detector and segmentation affinities alone do not significantly improve correspondence with overall task preferences, with all paired 95% CIs including zero, because specialized tools verify isolated presence or words rather than whether the intended subject performs the referenced motion in the requested scene. Combining ORAV's pairwise judge with our 15-tool diagnostic suite provides both contextual binding reliability and efficient factor-level measurements.
Part 3/3: Response to Reviewer RvD9
Q3. Stratified Human–Human & Judge–Human Agreement
Across the expanded 76-task human validation set of 729 candidate pairs and 10 annotators using 2,000 whole-task bootstrap resamples, task-macro agreement conditional on non-tied judgments is as follows:
- Overall (76 tasks, 729 pairs): Human–human agreement is 81.80% [78.37, 85.20], and judge–human agreement is 85.46% [82.80, 88.14] at 92.04% AB/BA pair coverage.
- Video-reference tasks (
Motion,Camera,VFX— 55 tasks, 519 pairs): Human–human agreement is 79.89% [76.17, 83.79], and judge–human agreement is 85.76% [82.63, 88.62]. - Audio-reference tasks (
Speech,Music— 22 tasks, 209 pairs): Human–human agreement is 84.09% [79.59, 88.93], and judge–human agreement is 86.88% [82.47, 91.24].
The breakdown across all nine reference roles is summarized below:
| Reference Role | Audited Tasks | Pairs | Judge–Human Agreement %, 95% CI | Coverage % | Order Conflicts % |
|---|---|---|---|---|---|
| Subject | 72 | 689 | 85.08 [82.43, 87.75] | 92.02 | 7.40 |
| Prop | 28 | 276 | 84.00 [79.30, 88.28] | 93.12 | 6.88 |
| Scene | 37 | 370 | 83.13 [78.90, 86.94] | 93.78 | 6.22 |
| Style | 9 | 90 | 89.50 [82.81, 95.43] | 93.33 | 6.67 |
| Motion | 46 | 429 | 85.96 [82.59, 89.23] | 91.61 | 7.69 |
| Camera | 7 | 70 | 88.71 [80.00, 96.28] | 94.29 | 5.71 |
| VFX | 6 | 53 | 83.03 [74.45, 92.19] | 96.23 | 3.77 |
| Speech | 16 | 160 | 87.92 [82.06, 93.21] | 88.75 | 10.62 |
| Music | 6 | 49 | 84.10 [79.63, 88.08] | 91.84 | 4.08 |
Across all reference types—including Motion (85.96%), Speech (87.92%), and Music (84.10%)—ORAV Judge maintains consistent reliability on par with inter-annotator agreement. We will incorporate this expanded 76-task validation and the MultiRef-Compass comparison into the revised manuscript.
Part 1/3: Response to Reviewer RvD9
We appreciate the reviewer for recognizing the importance of reference-aware audiovisual evaluation, the diversity of ORAV Bench, and the controlled ablation in Appendix A.9, as well as for raising constructive questions. Below, we address Q1–Q3 on a unified, expanded 76-task human validation set spanning 729 candidate pairs and 10 annotators, covering all nine semantic roles and all 30 reference compositions.
Q1. Comparison with MultiRef-Compass
MultiRef-Compass is valuable concurrent work (arXiv, July 2026), which we discussed in Section 2.2 and compared in Table 1. To clarify our contributions and directly address your question, we provide both a methodological comparison and a head-to-head empirical evaluation on the same 76 audited tasks below.
1. Benchmark and methodological distinction:
C + D + Avs.C + A): MultiRef-Compass focuses primarily on static appearance references (subject,object,scene) alongside audio-video coherence. ORAV extends multi-reference conditioning to jointly compose dynamic video references (motion,camera_movement,vfx) with appearance and audio (speech,music). Because video and speech references carry conflicting source content—such as original performers, scenes, or spoken words—that must be excluded while transferring only the assigned attribute, ORAV directly evaluates selective extraction, target binding, and reference substitution.2. Direct head-to-head comparison on the same 76 audited tasks and human annotations: Following your suggestion, we evaluated MultiRef-Compass's MLLM judge using
Gemini-3.1-Pro-Preview(mediumreasoning) in both first-pass and cross-candidate rejudging modes, alongside a vanilla pairwise MLLM judge, our simplified protocol ablation, and the full ORAV Judge on the exact same 76 tasks and 729 candidate pairs with 2,000 whole-task bootstrap resamples:On shared eligible comparisons across the 76 tasks, ORAV Judge outperforms MultiRef-Compass by +31.93 pp in strict agreement, with a 95% CI of [+25.69,+38.01] pp and +34.28 pp after rejudging, and by +18.41 pp in non-tie agreement, with a 95% CI of [+12.03,+25.29] pp and +14.85 pp after rejudging. Under identical prepared evidence, the full protocol also improves over the simplified prompt in both non-tie agreement by +4.28 pp [+1.21,+7.87] and AB/BA pair coverage (92.04% vs. 79.84%). A primary reason for MultiRef-Compass's lower strict agreement is its high predicted tie rate on candidate pairs where human annotators express clear directional preferences—35.60% in the first pass and 45.12% after rejudging—causing strict agreement to drop from 47.51% to 45.42% despite higher conditional non-tie accuracy. In addition, fixed-weight aggregation over satisfied appearance or speech checklist items frequently masks unrequested source-scene copying or simplified motion phases.