- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents OmniEdit-Bench, a multi-dimensional benchmark for evaluating instruction-based video editing models. The benchmark organizes editing tasks into five tracks, each with finer-grained sub-categories intended to capture video-specific editing requirements beyond simple spatial manipulations inherited from image editing. The authors also propose a four-dimensional evaluation framework measuring, paired with an accuracy-aware penalty mechanism that scales the scores of the other three dimensions by the normalized accuracy score. This mechanism is designed to prevent visually plausible but instruction-inconsistent edits from receiving inflated scores.
Critical strengths
-
The benchmark explicitly decomposes video editing along spatial, temporal, audio, reference, and reasoning axes, which moves beyond the image-editing-derived spatial focus of many early IVE benchmarks.
-
The accuracy-aware penalty mechanism is a sensible contribution that addresses a well-known flaw in many video editing metrics: high visual similarity scores can mask poor instruction following. By treating accuracy as a gating factor that modulates preservation, realism, and consistency scores, the scoring framework better aligns with the core goal of instruction-based editing.
-
The paper provides reasonably rigorous validation of its VLM-based evaluation pipeline, including human alignment analysis across four dimensions, inter-annotator agreement statistics, cross-VLM robustness tests, and bootstrap confidence intervals.
Critical weaknesses
-
As a benchmark paper intended to serve the research community, OmniEdit-Bench evaluates an extremely narrow set of models, and heavily favors closed-source commercial APIs over the open-source methods that dominate academic research. Only a small number of open-source models are included, many of which are lower-resolution (480P) that are not representative of current state-of-the-art open IVE methods. Notably, the benchmark entirely omits many widely referenced open-source models and methods that are standard baselines in the field. This severely limits the utility of the benchmark for the academic community, as researchers cannot use it to benchmark their own open-source methods against a meaningful set of baselines.
-
The field of instruction-guided video editing evaluation is already very crowded, with numerous prior benchmarks including VE-Bench, EditBoard, IVEBench, OpenVE-3M, VIE-Bench, and RISE-Bench, among many others. Against this backdrop, OmniEdit-Bench does not advance the state of the art in either depth or breadth.
-
While the paper positions its five-track taxonomy as comprehensive, most of these dimensions have already been explored in prior work. Spatial editing at attribute/subject/global levels, reference-conditioned editing, and reasoning-based editing all have dedicated benchmarks or sections in earlier work.
-
The benchmark is surprisingly shallow in terms of diagnostic granularity. With only 100 samples for the Audio track and 50 for the Reasoning track, sample sizes are too small for statistically robust stratified analysis. Unlike recent benchmarks that use atomic checklist items for fine-grained failure diagnosis, OmniEdit-Bench relies on coarse 1–5 Likert scores per dimension, which provide very little actionable insight into specific failure modes. Overall, the benchmark feels like a consolidation of existing evaluation ideas rather than a meaningful advance in evaluation methodology.
-
The evaluation pipeline relies entirely on subjective VLM ratings, with no supporting automated objective metrics. Prior benchmarks typically combine VLM-based semantic evaluation with quantitative automated metrics to provide a more complete and reproducible assessment. The absence of any objective metrics makes the results less verifiable and more sensitive to prompt engineering and VLM biases, and reduces the benchmark’s value for fine-grained technical diagnosis.
-
The paper provides only high-level descriptions of data sourcing, instruction generation, and curation procedures. Details such as inter-rater agreement during instruction curation, exclusion criteria for source videos, and the full distribution of task difficulty are missing. While the authors state that data and code will be released upon publication, the heavy reliance on closed-source commercial APIs for both model outputs and evaluation means that full reproduction of results is practically impossible for independent researchers.
-
The Temporal and Reasoning tracks are framed as the paper’s most distinctive contributions, but their task designs remain relatively simple. For example, the Reasoning track focuses on basic physical/causal changes and simple hypothetical scenarios, without exploring more complex multi-step reasoning, compositional edits, or cross-modal reasoning. Similarly, the Temporal track does not include fine-grained evaluation of long-range temporal coherence or motion trajectory accuracy. As a result, these tracks do not meaningfully advance evaluation beyond what has already been shown in prior work.
Questions for authors
Refer to weaknesses part.