- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents OmniVBench, a benchmark for omni reference-to-video generation. It has 813 cases spanning 7 task families and 18 sub-tasks, covering content, motion, style, structure, narrative and multi-reference settings.
Evaluation is factor-grounded. A VLM selects the criteria that apply to each case and instantiates them as manually verified checklists. A VLM judge then scores these for reference fidelity and instruction realization, the latter split into disentanglement/routing and target compliance. Video quality comes from off-the-shelf predictors.
The paper also introduces Omni-R2V, about 340K training samples built mainly from an in-house professional video corpus through cross-pair matching and inverse construction. Eleven open and closed models are benchmarked. The protocol is validated with a human correlation study and a comparison against holistic scoring.
Critical strengths
- Breadth of the task space. The benchmark covers structure references (greybox, line art, storyboards), narrative references, camera motion and cross-aspect compositions, which existing R2V benchmarks largely lack. Cases are designed so that the reference information cannot be read off the text, and the grounding cases include distractors, so the benchmark probes reference understanding rather than prompt following. Together with the evaluation of eleven current systems, this coverage is the main reason the paper would be useful to the community.
- A transparent, case-level protocol. Criteria, prompts, rubrics and evidence presentation are fully documented. Checklists are fixed per case and human-verified before scoring, so every model faces the same questions. That makes the protocol auditable and extensible, and there is initial evidence that it tracks human judgment better than holistic scoring.
Critical weaknesses
-
The dataset contribution is asserted rather than demonstrated.
a. No model is trained or fine-tuned on Omni-R2V. The claims of an "industrial-grade training resource" and a "practical and scalable recipe" are therefore unsupported, and the only quality evidence is a small pass/fail audit.
b. The data are heavily skewed toward narrative samples.
c. The claimed reusable recipe relies on an in-house source corpus, an internal camera-controllable video generator, and closed APIs such as GPT Image 2.0 and Gemini 3.1 Pro, while the instruction-generation rules and filtering prompts are not provided. Please release these rules and prompts and indicate open substitutes for the internal and closed components.
d. The dataset is half of the stated contribution. I would expect a fine-tuning study on an open model against existing R2V data at a matched budget, and a sub-task breakdown showing which benchmark tasks the training data cover.
-
Judge validity and ranking reliability are under-established, which undercuts the central claim of fine-grained evaluation.
a. A single proprietary VLM activates the criteria, writes the checklists, scores the outputs and helps construct the cases, while a model from the same vendor is being evaluated.
b. The human validation is small and missing basic details.
c. The per-model inference protocol is not described.
d. Before the reported differences can be trusted, the paper needs a second judge and a fuller human study with agreement statistics and per-family results (Table 5).
Questions for authors
-
Has any model been trained on Omni-R2V? Please compare it against the same model trained on existing R2V data at matched compute, on OmniVBench and on an external benchmark.
-
Human study (Sec. 5.2, A.6):
a. How many models and outputs were rated?
b. Is correlation computed per sample or per system? What are the per-family correlations?
c. How was VQ rated, given that A.6 describes only RF and IR?
-
Judge robustness (Sec. 3.3.2). How stable are the rankings under a judge from another model family and across repeated runs, particularly for Gemini Omni?
-
What durations, resolutions and number of generations per case were used?
-
Dataset scope (A.1.1). Which benchmark sub-tasks appear in the training data? How are training and benchmark data de-duplicated?