acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Audio-Video Generation from Multimodal Contexts

Abstract

Audio-video generation using heterogeneous multimodal references has emerged as a new challenge, requiring both compositional control over generation and grounded understanding of multimodal context. In this paper, we introduce ORAV Bench for Omni Reference Audio-Video Generation, comprising 380 task instances with 2-10 references, 9 semantic roles, and 30 role compositions. Instructions specify the relationships among references; the media supply the identities, dynamics, and audio characteristics to be realized. To evaluate these open-ended outputs, we develop a reference-aware pairwise protocol that prepares visual and auditory evidence, compares the intended contribution of each reference, and checks the overall verdict in both presentation orders. On held-out instances, it achieves 86.08% effective agreement with human judgments. Across 5 frontier systems, overall rankings conceal distinct strengths across reference compositions. A recurring failure is to reproduce unintended source content in place of the requested result, despite closely resembling a reference. Reproducible pointwise diagnostics of quality, reference affinity, and speech reveal distinct dimensions of model behavior. ORAV thus offers a benchmark for tracking progress toward controllable, compositional, and reference-faithful audio-video generation.

Primary area:
datasets and benchmarks
Keywords:
multimodal reference conditioning, audio-video generation, pairwise evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RvD9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces ORAV Bench, a benchmark for evaluating audio-video generation conditioned on multiple image, video, and audio references. It contains 380 instances covering 9 semantic roles and 30 reference compositions. The authors also propose a reference-aware pairwise evaluation protocol that uses structured visual and auditory evidence to evaluate reference fidelity and task fulfillment. Experiments on five frontier generation systems reveal different strengths across reference compositions and identify reference substitution as a common failure mode.

Critical strengths
  1. The paper studies an important and timely problem in audio-video generation. I agree that reference similarity alone is insufficient to evaluate whether a model correctly combines information from multiple references. The benchmark covers diverse reference types and compositions, which could be useful for evaluating current generation systems.
  2. The authors propose a carefully designed reference-aware pairwise evaluator. The ablation in Appendix A.9 shows that the structured protocol improves human agreement by 7.45 percentage points over a simplified prompt using the same model and evidence. I think this is a meaningful result that supports the usefulness of the proposed evaluation method.
Critical weaknesses
  1. The novelty over existing benchmarks is not sufficiently clear. MultiRef-Compass already evaluates reference consistency, compositional binding, and instruction following using a sophisticated MLLM-based evaluator, with a comparable dataset size (350 vs. 380 instances). While ORAV covers more diverse reference types, I would like to see a more direct comparision with existing evaluation methods to justify its novelty. Without this comparison, I am not fully convinced that the proposed evaluation method provides substantial improvements over prior work.
  2. The reliability of the proposed evaluation protocol is not sufficiently validated. Although the paper formalizes positive requirements and negative constraints, these semantic reference contracts are not explicitly annotated, and the judge must infer them from the instructions and references. This makes the evaluation heavily dependent on the judge's interpretation. I think the benchmark would be substantially more reliable if these constraints were explicitly annotated, potentially using MLLMs followed by human verification. Some constraints could also be evaluated more reliably and efficiently using specialized tools, such as object detectors or ASR. Moreover, only 38 instances are evaluated by human annotators, and the paper does not report human-human agreement or judge-human agreement across different reference types. Since the proposed evaluator is one of the main contributions, I think stronger validation of its reliability is important.
Questions for authors

Here are my main questions and concerns. I would be happy to change my opinion if the major concerns are resolved.

Q1: Could the authors clarify the main novelty compared with MultiRef-Compass, especially regarding the evaluation methodology? It would be helpful to compare the proposed judge with existing MLLM-based evaluators on the same instances and human annotations. This would help understand whether the proposed evaluation protocol provides meaningful improvements over existing methods.

Q2: Why are the positive requirements and negative constraints not explicitly annotated? I think using a strong MLLM to generate these annotations followed by human verification should be feasible. Have the authors considered this approach? It would also be interesting to compare evaluation results with and without explicit annotations, to understand how much the reliability depends on the judge's own interpretation.

Q3: Could the authors report human-human agreement and provide a breakdown of judge-human agreement across different reference types? This would help understand how reliable the evaluator is, particularly for motion and audio references. Since the current human validation only covers 38 instances, I think this is important to support the reliability of the proposed benchmark.

7 comments
AuthorsOpenReview ↗

Part 1/3: Response to Reviewer RvD9

We appreciate the reviewer for recognizing the importance of reference-aware audiovisual evaluation, the diversity of ORAV Bench, and the controlled ablation in Appendix A.9, as well as for raising constructive questions. Below, we address Q1–Q3 on a unified, expanded 76-task human validation set spanning 729 candidate pairs and 10 annotators, covering all nine semantic roles and all 30 reference compositions.

Q1. Comparison with MultiRef-Compass

MultiRef-Compass is valuable concurrent work (arXiv, July 2026), which we discussed in Section 2.2 and compared in Table 1. To clarify our contributions and directly address your question, we provide both a methodological comparison and a head-to-head empirical evaluation on the same 76 audited tasks below.

1. Benchmark and methodological distinction:

  • Selective attribute transfer across dynamic references (C + D + A vs. C + A): MultiRef-Compass focuses primarily on static appearance references (subject, object, scene) alongside audio-video coherence. ORAV extends multi-reference conditioning to jointly compose dynamic video references (motion, camera_movement, vfx) with appearance and audio (speech, music). Because video and speech references carry conflicting source content—such as original performers, scenes, or spoken words—that must be excluded while transferring only the assigned attribute, ORAV directly evaluates selective extraction, target binding, and reference substitution.
  • Evidence-grounded pairwise reasoning vs. decomposed pointwise checklists: MultiRef-Compass decomposes evaluation into pointwise MLLM checklists and sub-metrics aggregated with fixed weights alongside an optional cross-candidate rejudging pass, which works well for verifying whether individual static attributes are present. On omni-reference tasks, however, quantized checklist scores frequently tie, under-resolve continuous motion phases—which ORAV captures via 8 fps chronological panels and bidirectional blind voice comparison—and cannot weigh task-dependent trade-offs between competing partial failures or enforce a whole-artifact usability gate against severe source-content intrusion.

2. Direct head-to-head comparison on the same 76 audited tasks and human annotations: Following your suggestion, we evaluated MultiRef-Compass's MLLM judge using Gemini-3.1-Pro-Preview (medium reasoning) in both first-pass and cross-candidate rejudging modes, alongside a vanilla pairwise MLLM judge, our simplified protocol ablation, and the full ORAV Judge on the exact same 76 tasks and 729 candidate pairs with 2,000 whole-task bootstrap resamples:

EvaluatorBackboneStrict Agreement (w/ Ties) %Non-Tie Agreement %
MultiRef-Compass (First-Pass)Gemini-3.1-Pro47.51 [41.98, 53.27]68.22 [61.67, 74.39]
MultiRef-Compass (+ Rejudge)Gemini-3.1-Pro45.42 [39.79, 51.05]73.16 [66.59, 79.14]
Vanilla Pairwise JudgeGemini-3.1-Pro74.61 [69.94, 78.97]78.42 [73.86, 82.74]
Simplified ProtocolGemini-3.1 + Astra78.47 [74.30, 82.60]82.47 [78.21, 86.43]
ORAV Judge (Full Protocol)Gemini-3.1 + Astra81.43 [78.47, 84.26]85.46 [82.80, 88.14]

On shared eligible comparisons across the 76 tasks, ORAV Judge outperforms MultiRef-Compass by +31.93+31.93 pp in strict agreement, with a 95% CI of [+25.69,+38.01][+25.69, +38.01] pp and +34.28+34.28 pp after rejudging, and by +18.41+18.41 pp in non-tie agreement, with a 95% CI of [+12.03,+25.29][+12.03, +25.29] pp and +14.85+14.85 pp after rejudging. Under identical prepared evidence, the full protocol also improves over the simplified prompt in both non-tie agreement by +4.28+4.28 pp [+1.21,+7.87][+1.21, +7.87] and AB/BA pair coverage (92.04% vs. 79.84%). A primary reason for MultiRef-Compass's lower strict agreement is its high predicted tie rate on candidate pairs where human annotators express clear directional preferences—35.60% in the first pass and 45.12% after rejudging—causing strict agreement to drop from 47.51% to 45.42% despite higher conditional non-tie accuracy. In addition, fixed-weight aggregation over satisfied appearance or speech checklist items frequently masks unrequested source-scene copying or simplified motion phases.

AuthorsOpenReview ↗

Part 2/3: Response to Reviewer RvD9

Q2. Reference-Contract Interpretation & Specialized Tools

1. On-the-fly reference-contract interpretation. Thank you for this suggestion. We have considered MLLM-assisted contract annotation with human verification. While pre-extracting explicit requirement lists is effective for verifying isolated visual or lexical clauses—as explored in MultiRef-Compass—we intentionally couple on-the-fly contract interpretation with pairwise preference reasoning for three reasons:

  • Contextual cross-reference interactions and weights. In open-ended multimodal compositions, what should be transferred, adapted, or excluded from one reference depends relationally on co-occurring references and the target scene, such as permitted pose, lighting, or timing adaptations when binding motion and speech references to a new subject and environment. Exhaustively pre-specifying every cross-reference interaction and fixed aggregation weights ahead of time is challenging, as static lists can miss relational nuances or over-constrain valid variations.
  • Resolving task-dependent pairwise trade-offs. Candidates on omni-reference tasks rarely succeed or fail cleanly on the same requirement; one candidate may better preserve camera framing or attire while another more faithfully captures defining motion mechanics or avoids source-arena leakage, as in Fig. 3. Even with an explicit list of constraints, determining the winner still requires reasoning about each discrepancy's magnitude, scope, and consequence for the complete instructed event. As the Q1 comparison with MultiRef-Compass illustrates, scoring pre-extracted checklists under fixed weights frequently collapses competing partial fulfillments into score ties—especially after rejudging—limiting strict human agreement to 45.42%–47.51% compared with 81.43% for ORAV Judge. Conversely, our coupled protocol achieves 85.46% non-tie human agreement overall and 92.17% on comparisons decided at the usability-veto stage.
  • Direct generalizability to new natural-language tasks. Interpreting contracts on the fly from the instruction and reference media allows ORAV Judge to be applied directly to new open-ended prompts and reference combinations without a separate manual contract-annotation stage, while remaining compatible with human-verified contracts as an optional audit resource. We will clarify this design rationale in the revision.

2. Role of specialized tools. We agree that specialized tools offer efficient, reproducible measurements where direct signal extraction applies. Indeed, Table 3 and Appendices A.3–A.5 already incorporate 15 specialized pretrained tools across all 1,852 candidates, covering ASR word/character error rates (Whisper), speaker similarity (ECAPA-TDNN), recording replay (waveform cross-correlation), localized object, subject, face, and scene affinity (GroundingDINO, SAM2.1, DINO, CLIP, ArcFace), and motion trajectories (CoTracker3). As shown in Appendix A.4, however, localized detector and segmentation affinities alone do not significantly improve correspondence with overall task preferences, with all paired 95% CIs including zero, because specialized tools verify isolated presence or words rather than whether the intended subject performs the referenced motion in the requested scene. Combining ORAV's pairwise judge with our 15-tool diagnostic suite provides both contextual binding reliability and efficient factor-level measurements.

AuthorsOpenReview ↗

Part 3/3: Response to Reviewer RvD9

Q3. Stratified Human–Human & Judge–Human Agreement

Across the expanded 76-task human validation set of 729 candidate pairs and 10 annotators using 2,000 whole-task bootstrap resamples, task-macro agreement conditional on non-tied judgments is as follows:

  • Overall (76 tasks, 729 pairs): Human–human agreement is 81.80% [78.37, 85.20], and judge–human agreement is 85.46% [82.80, 88.14] at 92.04% AB/BA pair coverage.
  • Video-reference tasks (Motion, Camera, VFX — 55 tasks, 519 pairs): Human–human agreement is 79.89% [76.17, 83.79], and judge–human agreement is 85.76% [82.63, 88.62].
  • Audio-reference tasks (Speech, Music — 22 tasks, 209 pairs): Human–human agreement is 84.09% [79.59, 88.93], and judge–human agreement is 86.88% [82.47, 91.24].

The breakdown across all nine reference roles is summarized below:

Reference RoleAudited TasksPairsJudge–Human Agreement %, 95% CICoverage %Order Conflicts %
Subject7268985.08 [82.43, 87.75]92.027.40
Prop2827684.00 [79.30, 88.28]93.126.88
Scene3737083.13 [78.90, 86.94]93.786.22
Style99089.50 [82.81, 95.43]93.336.67
Motion4642985.96 [82.59, 89.23]91.617.69
Camera77088.71 [80.00, 96.28]94.295.71
VFX65383.03 [74.45, 92.19]96.233.77
Speech1616087.92 [82.06, 93.21]88.7510.62
Music64984.10 [79.63, 88.08]91.844.08

Across all reference types—including Motion (85.96%), Speech (87.92%), and Music (84.10%)—ORAV Judge maintains consistent reliability on par with inter-annotator agreement. We will incorporate this expanded 76-task validation and the MultiRef-Compass comparison into the revised manuscript.

Reviewer_TgdbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ORAV Bench for evaluating audio-video generation from heterogeneous image, video, and audio references. It contains 380 tasks spanning nine semantic roles and 30 role compositions, with semantic reference contracts specifying which source attributes should be transferred to which output targets. The authors propose a reference-aware pairwise evaluation protocol grounded in visual and auditory evidence and evaluate five generation systems. The results reveal composition-dependent strengths and recurring failures in which unintended source content replaces the requested output, highlighting the distinction between reference resemblance and successful task fulfillment.

Critical strengths

1, Well-motivated and informative evaluation target. The benchmark evaluates selective attribute transfer and correct binding across heterogeneous references, going beyond perceptual quality and source similarity. Its analysis reveals composition-dependent system strengths and unintended source copying, demonstrating the diagnostic value of this evaluation target. 2, Structured, evidence-grounded evaluation. The protocol links reference-specific visual and auditory observations to overall task fulfillment, providing an interpretable basis for pairwise preferences. A controlled comparison using the same judge and prepared evidence supports the value of the structured protocol over a simplified judging prompt.

Critical weaknesses

1, The human validation relies on only five expert annotators and 38 task instances, raising concerns about its representativeness. 2, Although Appendix A.9 provides evaluator comparisons, it does not establish whether the main rankings remain stable when replacing the judge or listener within the same evaluation protocol. This leaves the robustness of the findings to evaluator choice insufficiently verified.

Questions for authors

1, Do you have human validation results on a larger sample, or could you justify the choice of five annotators and a 10% task subset? 2, Have you tested whether rankings remain stable with alternative judge or listener models?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Tgdb

We appreciate the reviewer for recognizing the diagnostic value of our benchmark and the controlled validation supporting our structured evaluation protocol. We respond to both questions below.

Q1. Expanded 76-Task Human Validation

In the submitted paper, five expert annotators independently performed blind rankings on a randomly selected 10% held-out subset of 38 tasks and 356 candidate pairs because each ORAV task requires comparing up to five 10–15 s audiovisual outputs against multiple heterogeneous image, video, and audio references, with all bootstrap intervals clustered at the whole-task level rather than treating candidate pairs as independent samples.

To strengthen representativeness—especially for less frequent reference roles—we expanded the annotator pool from five to 10 annotators and doubled the validation set to 76 tasks, covering 20% of the benchmark and 729 candidate pairs. This achieves complete coverage of all nine semantic roles and all 30 reference compositions while maintaining strict separation from evaluator development.

Reporting task-macro agreement conditional on non-tied judgments across 2,000 whole-task bootstrap resamples, human–human agreement is 81.80% [78.37, 85.20] and evaluator–human agreement is 85.46% [82.80, 88.14] across the 76 tasks with 92.04% AB/BA pair coverage. This closely matches the submitted 38-task result of 86.08% [81.29, 90.59] and remains consistently strong across every reference role:

Reference RoleAudited TasksPairsEvaluator–Human Agreement %, 95% CICoverage %Order Conflicts %
Subject7268985.08 [82.43, 87.75]92.027.40
Prop2827684.00 [79.30, 88.28]93.126.88
Scene3737083.13 [78.90, 86.94]93.786.22
Style99089.50 [82.81, 95.43]93.336.67
Motion4642985.96 [82.59, 89.23]91.617.69
Camera77088.71 [80.00, 96.28]94.295.71
VFX65383.03 [74.45, 92.19]96.233.77
Speech1616087.92 [82.06, 93.21]88.7510.62
Music64984.10 [79.63, 88.08]91.844.08

Q2. Robustness to Alternative Evaluator Models

To test whether our main findings depend on the specific foundation models used for listening (Gemini-3.1-Pro-Preview) and judging (GPT-6-Astra), we repeated the evaluation on the complete 380-task benchmark using Gemini 4 Argon as the blind listener and GPT-6.1-Sol as the pairwise judge under the identical evaluation protocol and medium reasoning-effort setting.

The resulting pairwise judgments achieve 96.53% agreement with those of the original evaluator across the benchmark. Refitting the Davidson–Bradley–Terry model on the shared-delivery tasks with 2,000 whole-task bootstrap resamples preserves the exact five-system ranking and closely reproduces both the strength estimates and expected win rates:

RankSystemOriginal Strength [95% CI]Win Rate %Alt. Evaluator Strength [95% CI]Win Rate %
1Seedance-2.5+0.70 [+0.56,+0.85]+0.70\ [+0.56, +0.85]68.38+0.68 [+0.55,+0.82]+0.68\ [+0.55, +0.82]67.83
2Seedance-2.0+0.48 [+0.36,+0.59]+0.48\ [+0.36, +0.59]62.69+0.45 [+0.33,+0.56]+0.45\ [+0.33, +0.56]61.87
3MiniMax-H3+0.15 [+0.02,+0.28]+0.15\ [+0.02, +0.28]54.00+0.09 [−0.04,+0.22]+0.09\ [-0.04, +0.22]52.46
4Wan3.0-Video−0.56 [−0.70,−0.42]-0.56\ [-0.70, -0.42]35.04−0.53 [−0.67,−0.39]-0.53\ [-0.67, -0.39]35.82
5Kling-v3-Omni−0.77 [−0.93,−0.63]-0.77\ [-0.93, -0.63]29.90−0.68 [−0.84,−0.54]-0.68\ [-0.84, -0.54]32.02

Eight of the nine semantic-role leaders also remain identical: Seedance-2.5 leads Subject, Prop, Scene, Motion, VFX, and Music, while MiniMax-H3 leads Speech and Camera. Only Style, where the top two systems originally differed by less than 2 percentage points, shifts narrowly between Seedance-2.5 and Seedance-2.0. These results confirm that ORAV's rankings and diagnostic findings are highly robust to evaluator model choice.

Thanks for the author's response, I will maintain my positive score.

Thank you very much for taking the time to read our response and for your continued support of our work. Your constructive suggestions on expanding the human validation and verifying consistency across alternative evaluator models have helped us further corroborate our findings and improve the clarity of the paper. We truly appreciate your thoughtful feedback throughout the review process.

Reviewer_oDZJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ORAV Bench for audio-video generation conditioned on references. In this dataset, each instruction is intended to specify what information a reference contributes, where that information is bound, and which source content must not be copied. This work proposes a reference-aware pairwise judge that uses timestamped visual panels and independent textual listening observations, evaluates both presentation orders, and aggregates valid output with a Davidson–Bradley–Terry model. On 38 held-out samples, this work achieves 86.08% effective agreement with expert preferences, compared with 71.82% for a vanilla multimodal judge. This work evaluates five commercial models, the results show that Seedance-2.5 ranks first. One of the main claims is that current generation models frequently reproduce unrequested source content, such as the original performer, scene, or recording, instead of performing the requested composition.

Critical strengths
  1. This paper distinguishes reproducing source content from extracting and binding reference attributes as instructed. For example, a motion reference should not automatically determine the performer's identity, and a voice reference should not automatically determine the spoken words. Relative to the benchmarks listed in Table 1, jointly composing motion, camera movement, and visual effects with appearance and audio attributes is a meaningful extension of coverage.

  2. This work checks compatibility among scenes, props, and actions; specify bindings between multiple subjects, speech content, and some permitted timing adjustments; and use repeated human review to determine inclusion.

Critical weaknesses
  1. The evaluator lacks validation stratified by task category and judgment dimension. Sec 4.4 and Tab 13 report aggregate human agreement, while Fig 4 presents system-level win–tie–loss profiles. However, the paper does not report agreement, coverage, or order-conflict rates separately across semantic roles, reference compositions, or judgment dimensions such as reference matching, binding, instruction following, and usability. The role- and composition-conditioned results in Tab 2 and Fig 6 characterize generator performance, but do not establish the reliability of the evaluator within those subsets. The qualitative examples also do not constitute a systematic analysis of ORAV's own errors. A stratified human audit, with sample counts, confidence intervals, and an analysis of evaluator–human disagreements, would directly address this concern.

  2. Fig 4 raises a specific question: on the same held-out set, the human win-rate profile favors Seedance-2.0 over Seedance-2.5, whereas the ORAV profile favors Seedance-2.5. This warrants checking whether errors concentrate on comparisons that determine the ordering of the leading systems.

  3. Usability vetoes determine 27% of resolved pairwise comparisons, making this mechanism consequential for the ranking. However, the boundary at which source-content intrusion makes an output unusable is largely interpreted by the judge from the instruction. The targets, requirements, and exclusions of reference contracts are also not supplied as independent annotations. Agreement on overall preferences does not establish that these intermediate judgments and their explanations are individually correct.

  4. The task formulation emphasizes audiovisual temporal coherence, and Fig 1 includes lip synchronization. Appendix A.7 explicitly states that fine-grained audiovisual synchronization lies outside the validated scope. The sensitivity of separate visual panels and textual auditory observations to lip-sync errors, action–sound misalignment, and incorrect audiovisual event binding remains unclear. This limitation should be more prominent in the main text, and the scope should be tested using controlled soundtrack shifts, speaker swaps, or audiovisual event mismatches.

  5. The subject role appears in 362/380 tasks, whereas camera, VFX, and music appear in only 27, 22, and 20 tasks, respectively. Several compositions in Figure 6 contain only 9–13 shared-delivery tasks. The top three estimates in the camera column of Table 2 differ by less than one percentage point, yet intervals are not provided for these subgroup results. Therefore, claims about which system leads a particular role or composition have uneven evidential support. This paper also does not establish whether the random sample of 38 validation tasks adequately covers the less frequent roles.

Questions for authors
  1. As mentioned in the weaknesses, a stratified human audit, with sample counts, confidence intervals, and an analysis of evaluator–human disagreements, would directly address the concern about validation of the evaluator.

  2. Why is original-sound retention enabled for speech tasks requiring new utterances? Are there other supported configurations that preserve the ability to generate new speech? How do the main ranking and reference-substitution conclusions change when native audio support and adapted support are reported separately?

  3. Which roles and compositions are represented in the 38 held-out tasks? Could this paper provide paired confidence intervals for subgroup comparisons and test the key composition differences with controls that keep the assets and scene fixed while adding, removing, or swapping one reference constraint?

6 comments
AuthorsOpenReview ↗

Part 1/4: Response to Reviewer oDZJ

We thank the reviewer for recognizing the distinction between reproducing source content and selectively extracting and binding reference attributes, the expanded compositional coverage in Table 1, and the care taken in curating compatible multi-reference tasks. We respond to each concern and question below.

Q1. Stratified Validation & Disagreement Audit

We expanded the human audit from 38 to 76 tasks—20% of the benchmark, spanning 729 candidate pairs with blind rankings from 10 annotators—increasing coverage of underrepresented compositions so that all nine semantic roles and all 30 reference compositions are represented while maintaining separation from evaluator development.

Agreement, Coverage, and Uncertainty. We report task-macro agreement conditional on both judgments being non-tied, with 95% confidence intervals from 2,000 whole-task bootstrap resamples preserving all within-task comparisons. Human–human agreement is 81.80% [78.37, 85.20], and evaluator–human agreement is 85.46% [82.80, 88.14]. Across all 729 candidate pairs, 92.04% yield a consistent AB/BA decision (Coverage) and 7.41% have inconsistent outcomes across presentation orders (Order Conflicts).

Role-Conditioned Validation

Reference RoleAudited TasksPairsEvaluator–Human Agreement %, 95% CICoverage %Order Conflicts %
Subject7268985.08 [82.43, 87.75]92.027.40
Prop2827684.00 [79.30, 88.28]93.126.88
Scene3737083.13 [78.90, 86.94]93.786.22
Style99089.50 [82.81, 95.43]93.336.67
Motion4642985.96 [82.59, 89.23]91.617.69
Camera77088.71 [80.00, 96.28]94.295.71
VFX65383.03 [74.45, 92.19]96.233.77
Speech1616087.92 [82.06, 93.21]88.7510.62
Music64984.10 [79.63, 88.08]91.844.08

Role subsets overlap and include co-occurring requirements; these figures measure agreement on overall rankings within each subset rather than isolated accuracy on a single role.

Composition-Complexity Validation

Stratifying into mutually exclusive groups by the number of distinct reference roles gives:

Distinct RolesAudited TasksPairsEvaluator–Human Agreement %, 95% CICoverage %Order Conflicts %
21817183.90 [77.45, 89.55]88.899.94
34138888.16 [85.03, 91.07]93.046.44
41717080.58 [74.26, 86.19]92.947.06

Agreement exceeds 80% across every role and composition stratum. Speech tasks exhibit a moderately higher order-conflict rate of 10.62%, reflecting closer trade-offs between vocal likeness and visual acting, which our AB/BA gate conservatively isolates as ties for ranking rather than forcing directional errors.

Evaluator–Human Disagreement Analysis. Non-tied disagreements involve 167 distinct candidate pairs across 61 tasks. Categorizing pairs using all valid human labels before tie filtering, 80.16% of disagreement pairs with multiple human judgments involve opposing human directional preferences, 10.32% involve human tie-versus-direction splits, and only 9.52% show unanimous human directional preferences that the evaluator contradicts. Across all evaluable pairs with unanimous multi-annotator directional preferences, the evaluator agrees on 94.67%. Qualitative review of the unanimous-disagreement cases shows that they primarily reflect competing severity trade-offs when candidates fail different constraints—such as preserving complex dance or skating mechanics while leaking a source arena versus preserving the target setting while simplifying the motion—or stricter weighting of fine prop orientation at panel resolution, rather than hallucinated observations. We will include this analysis in the revision.

AuthorsOpenReview ↗

Part 2/4: Response to Reviewer oDZJ

Q2. Original-Sound Retention & Native Audio

Enabling original-sound retention stems from a capability limitation of Kling-v3-Omni itself: the model does not support generating new audio when a reference video is provided as input, and its interface offers no separate audio-reference channel. Disabling original-sound retention yields silent outputs rather than newly synthesized speech. To include Kling-v3-Omni on video-reference tasks, we use a compatibility adapter, documented in Appendix A.8, that packages the audio reference as a video-input soundtrack and retains original sound on speech tasks.

Reporting native and adapted audio configurations separately leaves our main conclusions unchanged:

  • Visual-reference subset (Table 5): On the 259 shared-delivery tasks without speech or music references, the top-three ordering of Seedance-2.5 > Seedance-2.0 > MiniMax-H3 is unchanged, while Kling-v3-Omni moves from fifth to fourth.
  • Native-audio systems only: Refitting Davidson–Bradley–Terry across the four systems with native audio-video generation preserves the exact overall ranking: Seedance-2.5 at 64.49%, Seedance-2.0 at 57.53%, MiniMax-H3 at 50.34%, and Wan3.0-Video at 27.65%. On speech-reference tasks, MiniMax-H3 likewise retains the lead at 66.87% [59.62, 73.59], ahead of Seedance-2.5 at 58.08% [50.66, 65.57], Seedance-2.0 at 48.22% [40.48, 55.48], and Wan3.0-Video at 26.83% [20.18, 34.00].
  • Reference substitution: Importantly, we do not interpret Kling-v3-Omni's 79/79 retained soundtracks as generative reference substitution, as noted in Section 5.3 and Appendices A.5–A.6. Across the four native systems alone, 93.97% of veto explanations still cite reference substitution; Wan3.0-Video independently replays the reference recording in 9/78 assessed speech outputs, and its unusability rate increases from 12.0% without video references to 55.9% with video references due to frequent reproduction of source performers and scenes. We will clarify and report this separation explicitly in the revision.
AuthorsOpenReview ↗

Part 3/4: Response to Reviewer oDZJ

Q3. Subgroup Uncertainty & Matched Controls

Held-out coverage. The original 38-task validation set was a random 10% sample covering all nine reference roles and 16 of the 30 composition signatures. As detailed in Q1, our expanded 76-task set covers all nine reference roles and all 30 composition signatures.

Subgroup uncertainty. We added 95% bootstrap confidence intervals for all nine role-conditioned expected win rates (%) in Table 2, using 2,000 whole-task resamples shared across systems to preserve pairing:

Image References

SystemSubjectPropSceneStyle
Seedance-2.568.92 [65.58, 72.01]64.07 [58.80, 69.05]68.61 [64.43, 72.55]65.48 [55.35, 75.31]
Seedance-2.062.82 [59.66, 65.85]61.01 [56.03, 65.99]61.85 [57.55, 66.10]63.54 [54.87, 71.98]
MiniMax-H353.39 [49.99, 56.87]56.40 [51.04, 62.05]56.87 [51.70, 61.89]57.51 [48.58, 66.33]
Wan3.0-Video35.50 [31.83, 39.05]38.90 [33.08, 44.57]34.71 [29.78, 39.78]36.47 [26.08, 47.38]
Kling-v3-Omni29.37 [25.94, 32.91]29.63 [24.12, 35.39]27.96 [22.76, 33.37]27.00 [18.56, 36.10]

Video References

SystemMotionCameraVFX
Seedance-2.571.01 [66.86, 75.37]65.38 [52.40, 77.40]82.63 [71.50, 91.70]
Seedance-2.065.70 [61.86, 69.63]64.90 [55.76, 74.04]71.36 [63.70, 78.98]
MiniMax-H342.37 [38.61, 46.35]65.87 [54.81, 77.40]58.28 [44.18, 73.35]
Wan3.0-Video30.06 [25.28, 34.82]25.00 [17.31, 33.65]30.00 [18.63, 41.29]
Kling-v3-Omni40.86 [36.38, 45.53]28.85 [15.87, 43.75]7.73 [2.94, 13.59]

Audio References

SystemSpeechMusic
Seedance-2.568.04 [62.19, 73.80]76.56 [63.32, 89.21]
Seedance-2.059.95 [53.74, 66.17]50.98 [37.76, 63.53]
MiniMax-H373.58 [68.18, 79.06]39.85 [25.28, 54.58]
Wan3.0-Video40.90 [34.42, 47.13]40.42 [23.07, 58.70]
Kling-v3-Omni7.53 [3.47, 11.94]42.19 [25.64, 59.48]

For the Camera subgroup, paired whole-task bootstrap 95% intervals for win-rate differences confirm that the top three systems are not statistically separated:

SubgroupComparisonWin-Rate Difference (pp), 95% CI
CameraMiniMax-H3 − Seedance-2.5+0.48 [−15.40,+17.31]+0.48\ [-15.40, +17.31]
CameraMiniMax-H3 − Seedance-2.0+0.96 [−17.31,+18.75]+0.96\ [-17.31, +18.75]
CameraSeedance-2.5 − Seedance-2.0+0.48 [−14.90,+16.35]+0.48\ [-14.90, +16.35]

Inspection of the Camera judge rationales shows that these closely clustered scores reflect partial rather than saturated camera fidelity: while the leading systems capture basic forward/backward travel, rationales repeatedly identify omitted trajectory phases, simplified rotation or banking, discontinuities, and apparent camera matches inside leaked reference scenery. Thus, the Camera results highlight shared room for improvement rather than a definitive winner.

Benchmark scope and matched composition controls. ORAV Bench prioritizes asset diversity across its 380 tasks and 1,322 distinct reference assets without cross-task reuse, while providing a natural foundation for controlled single-constraint extensions. To examine the key composition contrast in Section 5.2, we constructed Subject + Speech counterparts for all tasks in the Subject + Motion + Speech subset by removing the motion-reference requirement while holding the remaining assets and scene fixed, and evaluated all five systems under the same ranking protocol:

Matched ConditionSeedance-2.5 (%)MiniMax-H3 (%)Difference: Seedance − MiniMax
Subject + Motion + Speech73.453.6+19.8+19.8 pp
Subject + Speech80.287.8−7.6-7.6 pp

Removing the motion-reference requirement shifts the Seedance–MiniMax difference from +19.8+19.8 to −7.6-7.6 pp, a −27.4-27.4 pp change. The point estimates reproduce the ordering reversal under matched assets and scenes, consistent with the composition-dependent pattern in Section 5.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.