acceptodds
Under review as a conference paper at ICLR 2027

OmniVBench: A Benchmark and Large-Scale Dataset for Omni Reference-to-Video Generation

Abstract

Omni reference-to-video (R2V) generation is emerging as a general paradigm that enables flexible video customization from diverse and compositional visual references. However, existing benchmarks fall short of these emerging capabilities: their test cases cover only a limited range of reference types and compositions, and their evaluation protocols largely assess holistic reference consistency, overlooking whether reference factors are properly preserved, disentangled, and routed. Meanwhile, the high cost of constructing omni R2V training data makes suitable training resources scarce. To address these gaps, we introduce OmniVBench and the Omni-R2V Dataset, establishing a shared foundation for evaluating and training omni R2V models. OmniVBench expands R2V evaluation across broader reference types, richer reference compositions, and more flexible instruction-driven customization, covering 7 task families and 18 fine-grained tasks spanning content, motion, style, structure, narrative, and multi-reference settings. Complementing this task design, we introduce factor-grounded evaluation with 12,172 case-specific checklist items, explicitly assessing whether intended reference factors are faithfully preserved, correctly disentangled and bound to their targets, and realized according to the instruction. We further introduce the Omni-R2V Dataset, bringing industrial-grade training resources for diverse R2V tasks to the broader research community. Drawing primarily on a large-scale corpus of professional video footage, it comprises 340K processed training samples spanning diverse reference types and multi-reference compositions. Specifically, we develop task-specific pipelines for reference-target pair construction, offering a practical and scalable recipe for omni R2V data construction. Extensive evaluation of advanced open- and closed-source R2V models reveals substantial performance variation across diverse tasks and distinct capability patterns across evaluation dimensions, highlighting OmniVBench’s fine-grained evaluation capability.

Primary area:
datasets and benchmarks
Keywords:
Reference-to-Video Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tiegOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents OmniVBench, a benchmark for omni reference-to-video generation. It has 813 cases spanning 7 task families and 18 sub-tasks, covering content, motion, style, structure, narrative and multi-reference settings.

Evaluation is factor-grounded. A VLM selects the criteria that apply to each case and instantiates them as manually verified checklists. A VLM judge then scores these for reference fidelity and instruction realization, the latter split into disentanglement/routing and target compliance. Video quality comes from off-the-shelf predictors.

The paper also introduces Omni-R2V, about 340K training samples built mainly from an in-house professional video corpus through cross-pair matching and inverse construction. Eleven open and closed models are benchmarked. The protocol is validated with a human correlation study and a comparison against holistic scoring.

Critical strengths
  1. Breadth of the task space. The benchmark covers structure references (greybox, line art, storyboards), narrative references, camera motion and cross-aspect compositions, which existing R2V benchmarks largely lack. Cases are designed so that the reference information cannot be read off the text, and the grounding cases include distractors, so the benchmark probes reference understanding rather than prompt following. Together with the evaluation of eleven current systems, this coverage is the main reason the paper would be useful to the community.
  2. A transparent, case-level protocol. Criteria, prompts, rubrics and evidence presentation are fully documented. Checklists are fixed per case and human-verified before scoring, so every model faces the same questions. That makes the protocol auditable and extensible, and there is initial evidence that it tracks human judgment better than holistic scoring.
Critical weaknesses
  1. The dataset contribution is asserted rather than demonstrated.

    a. No model is trained or fine-tuned on Omni-R2V. The claims of an "industrial-grade training resource" and a "practical and scalable recipe" are therefore unsupported, and the only quality evidence is a small pass/fail audit.

    b. The data are heavily skewed toward narrative samples.

    c. The claimed reusable recipe relies on an in-house source corpus, an internal camera-controllable video generator, and closed APIs such as GPT Image 2.0 and Gemini 3.1 Pro, while the instruction-generation rules and filtering prompts are not provided. Please release these rules and prompts and indicate open substitutes for the internal and closed components.

    d. The dataset is half of the stated contribution. I would expect a fine-tuning study on an open model against existing R2V data at a matched budget, and a sub-task breakdown showing which benchmark tasks the training data cover.

  2. Judge validity and ranking reliability are under-established, which undercuts the central claim of fine-grained evaluation.

    a. A single proprietary VLM activates the criteria, writes the checklists, scores the outputs and helps construct the cases, while a model from the same vendor is being evaluated.

    b. The human validation is small and missing basic details.

    c. The per-model inference protocol is not described.

    d. Before the reported differences can be trusted, the paper needs a second judge and a fuller human study with agreement statistics and per-family results (Table 5).

Questions for authors
  1. Has any model been trained on Omni-R2V? Please compare it against the same model trained on existing R2V data at matched compute, on OmniVBench and on an external benchmark.

  2. Human study (Sec. 5.2, A.6):

    a. How many models and outputs were rated?

    b. Is correlation computed per sample or per system? What are the per-family correlations?

    c. How was VQ rated, given that A.6 describes only RF and IR?

  3. Judge robustness (Sec. 3.3.2). How stable are the rankings under a judge from another model family and across repeated runs, particularly for Gemini Omni?

  4. What durations, resolutions and number of generations per case were used?

  5. Dataset scope (A.1.1). Which benchmark sub-tasks appear in the training data? How are training and benchmark data de-duplicated?

Reviewer_9NvWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OmniVBench, a benchmark for omni reference-to-video (R2V) generation, together with the Omni-R2V Dataset, a 340K-sample training resource. OmniVBench organizes the R2V task space into seven task families and eighteen sub-tasks spanning content, motion, style, structure, and narrative references as well as multi-content and cross-aspect compositions, comprising 813 evaluation cases. Its central methodological idea is factor-grounded evaluation: each case is decomposed into case-specific checklist items (12,172 in total) that assess Reference Fidelity (RF), Instruction Realization (IR, split into disentanglement/routing and target compliance), and Video Quality (VQ), with criteria activated per sample by a VLM and scored by a VLM judge against detailed rubrics. The authors evaluate eleven open- and closed-source R2V models, validate the automatic metric against human judgments on 100 cases (Pearson 0.78-0.86), and show that the checklist protocol correlates better with humans than holistic scoring. They also describe construction pipelines for the Omni-R2V dataset (339,570 processed samples built primarily from professional footage via cross-pair matching and inverse construction).

Critical strengths
  1. Timely, well-scoped problem with honest positioning. The move from isolated reference control to compositional "omni" R2V is real, and Tables 1-2 document the coverage gap against existing benchmarks and datasets concretely rather than by assertion.
  2. The task taxonomy is genuinely comprehensive and thoughtfully decomposed. Object references split into holistic identity / material / pattern / part; character references into single-view, multi-view, customized, and attribute-grounded (with distractors) settings; camera motion into primitives and compound moves; 43 styles across 16 categories; structure into greybox / line art / rough storyboard; narrative into story / multi-panel storyboard / preceding shot. The attribute-grounded and compositional-grounding sub-tasks are well-designed capability probes that go beyond holistic similarity.
  3. Factor-grounded evaluation is a real methodological contribution, and the paper shows it is informative rather than decorative. The RF / IR-D / IR-E decomposition separates fidelity from disentanglement-and-routing from target compliance, and Table 4 demonstrates the split carries signal: Vidu-Q2-Pro scores 84.69 IRDR but 56.59 IRTC on narrative, while Happy Horse 1.0 shows the opposite pattern on structure (75.13 vs 97.15). Such dissociations are exactly what a holistic score would hide.
  4. Validation effort is above average for a benchmark paper: human correlation on 100 cases spanning all 18 sub-tasks (RF 0.81/0.77, IR 0.78/0.74, VQ 0.86/0.82 Pearson/Spearman) plus a holistic-vs-checklist ablation (RF Spearman 0.67 -> 0.77), which directly supports the central design claim.
Critical weaknesses
  1. Judge-model conflict of interest, unaddressed. The criterion activator and the RF/IR judge are Gemini, and Gemini Omni (Google DeepMind) is among the evaluated models; Gemini Omni obtains the highest or near-highest IRDR in most columns of Table 4 (96-98 range). The paper reports no control with a different-family judge, no inter-judge agreement, and no discussion of self-preference risk. The human-correlation study mitigates but does not settle this, since 100 cases cannot exclude a systematic bias toward the judge's own lineage. This is the most serious design concern.
  2. Fine-grained claims rest on very small per-sub-task samples without any error analysis. 813 cases across 18 sub-tasks leaves 19-54 cases each (Greybox 19, Preceding-Shot 25, Object 33, Style 43); e.g., Seedance 2.0's 100.00 style IRDR rests on ~43 cases. No confidence intervals, significance tests, or per-task variances are reported anywhere, so the paper's "clear gaps" and "distinct capability patterns" claims are not statistically supported at this N, and several rankings in Tables 3-4 are within noise.
  3. VQ is generic and its aggregation is unjustified. The three VQ components are off-the-shelf predictors (DOVER++ technical branch, Aesthetic Predictor V2.5, UnifiedReward 2.0 physics) with no R2V-specific adaptation, equal-weighted and averaged together with RF and IR into each task score. This mixes reference control (the benchmark's stated purpose) with generic video quality; the paper itself shows the two dissociate (Vidu-Q2-Pro competitive in VQ but weaker in RF). The weighting is never justified and no reference-control-only leaderboard is provided.
  4. No human baseline or upper bound on the benchmark itself. The human study only correlates annotators with the automatic metric; nothing anchors the absolute RF/IR levels (e.g., what a careful human editor would score on these cases), leaving "how far are current models" partially unanswerable.
  5. Criterion activation is performed by a VLM with no measured accuracy. Activation errors silently change which criteria are scored per case, yet no activation-agreement statistic against human activation and no sensitivity analysis of final scores to activation errors are reported.
  6. Motion fidelity is judged from sparse frame evidence. Temporally aligned references are presented as ten matched reference-output frame pairs, and non-aligned references as 8-20 frames sampled at 2 FPS; RF-M01/RF-M02 require judging action stages, rhythm, trajectory, and compound camera paths from this evidence. No sensitivity analysis of sampling density for fast motions is provided.
Questions for authors
  1. Can you report results with a different-family judge (e.g., Claude or Qwen-VL) or quantify inter-judge agreement and any self-preference effect for Gemini Omni? This is the single result that could most change my score.
  2. With 19-54 cases per sub-task, what are the confidence intervals on the Table 3-4 rankings, and which reported gaps survive paired significance testing?
  3. How were the equal RF/IR/VQ weights in the per-task score chosen, and what does the ranking look like on a reference-control-only leaderboard (VQ excluded)?
  4. Do the three annotators' raw scores suggest a near-ceiling on these cases (e.g., RF around 90+)? If so, why do the best models sit at 70-75?
  5. How sensitive are RF-M01/RF-M02 to the number of sampled frame pairs for fast actions and compound camera moves?
Reviewer_86k4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces OmniVBench for reference-to-video generation, covering 813 cases across seven task families and 18 subtasks. It uses case-specific checklists to assess reference fidelity (RF) and instruction realization (IR), alongside automatic video-quality metrics. The paper also introduces Omni-R2V, approximately 340K training examples constructed through task-specific pipelines. Experiments compare eleven generators and assess agreement with human judgments.

Critical strengths
  1. The task coverage extends beyond content preservation to motion, style, structure, narrative, and combinations of references.

  2. Separating reference fidelity from factor routing and target compliance provides more informative feedback than a single similarity score.

  3. The human evaluation is encouraging. It improves RF/IR Spearman correlation from 0.67/0.69 to 0.77/0.74. Table A.6 also includes a training-data quality audit.

Critical weaknesses
  1. My main concern is that the human evaluation is coarser than the claimed diagnostic capability. The 100-case study evaluates overall reference fidelity, instruction realization, and video quality. The benchmark makes finer claims about individual factors and routing. Tables 5–6 report neither criterion-level agreement nor per-family correlations or annotator agreement. Overall correlation does not establish that the detailed diagnoses are reliable.

  2. The temporal evaluation uses limited evidence. Some samples use only ten reference–output frame pairs for motion and temporally aligned structure references. This can miss brief events or motion between sampled frames. The paper needs to establish how sensitive motion and structural judgments are to this sampling choice.

  3. The generator comparison is difficult to reproduce precisely. It lists models but does not fully specify output durations, resolutions, generation counts, or handling of failed requests and incompatible reference inputs. Table 3 also lacks uncertainty estimates. Differences such as 72.68 versus 72.41 are therefore difficult to interpret as stable rankings.

  4. The training dataset is a major contribution, but its learning benefits remain untested. Table A.6 supports sample quality, yet no training-utility experiment is reported. The value of the different construction pipelines also remains unclear. The dataset is promising, but its usefulness for learning broad reference control is less established than its scale and coverage.

Questions for authors

As shown in the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.