acceptodds
Withdrawn from ICLR 2027

OmniEdit-Bench: A Comprehensive Benchmark for Instruction-based Video Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Instruction-based video editing (IVE) is an emerging field with rich application scenarios, yet evaluating the editing models remains a significant challenge. Existing evaluation benchmarks for IVE task suffer from two fundamental limitations. First, task coverage is narrow and largely inherited from image editing, focusing on frame-level spatial manipulations while neglecting video-specific dimensions. Second, current evaluation metrics fail to capture instruction fidelity, allowing models to achieve high scores despite incorrect edits due to the strong visual prior of the original video. To this end, we introduce a comprehensive and clearly structured benchmark for IVE task. Our benchmark systematically decomposes editing tasks along multiple video-specific dimensions, including spatial, temporal, audio, and reference-based editing, going beyond conventional frame-level formulations. Moreover, it explicitly distinguishes between explicit and implicit instructions, incorporating reasoning-based scenarios to better reflect real-world editing requirements. This design enables a more complete and fine-grained evaluation of model capabilities across diverse editing dimensions. Furthermore, we propose a systematic evaluation framework that decomposes editing quality into four complementary dimensions: accuracy, preservation, realism, and consistency with both human judges and state-of-the-art vision language model (VLM). To emphasize the central role of instruction fidelity, we further introduce an accuracy-aware penalty mechanism, where the scores of other dimensions are conditioned on accuracy. This design prevents misleading high scores for visually plausible yet incorrect edits. We conducted experiments evaluating current prominent instruction-based video editing models, comprising both open-source and commercial models. Experimental results reveal that current models remain far from satisfactory in instruction-based video editing. OmniEdit-Bench provides a comprehensive and reliable testbed for instruction-based video editing, offering valuable insights into current model limitations and guiding future research in this direction.

Primary area:
datasets and benchmarks
Keywords:
video editing, benchmark, generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_u2KrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents OmniEdit-Bench, a multi-dimensional benchmark for evaluating instruction-based video editing models. The benchmark organizes editing tasks into five tracks, each with finer-grained sub-categories intended to capture video-specific editing requirements beyond simple spatial manipulations inherited from image editing. The authors also propose a four-dimensional evaluation framework measuring, paired with an accuracy-aware penalty mechanism that scales the scores of the other three dimensions by the normalized accuracy score. This mechanism is designed to prevent visually plausible but instruction-inconsistent edits from receiving inflated scores.

Critical strengths
  1. The benchmark explicitly decomposes video editing along spatial, temporal, audio, reference, and reasoning axes, which moves beyond the image-editing-derived spatial focus of many early IVE benchmarks.

  2. The accuracy-aware penalty mechanism is a sensible contribution that addresses a well-known flaw in many video editing metrics: high visual similarity scores can mask poor instruction following. By treating accuracy as a gating factor that modulates preservation, realism, and consistency scores, the scoring framework better aligns with the core goal of instruction-based editing.

  3. The paper provides reasonably rigorous validation of its VLM-based evaluation pipeline, including human alignment analysis across four dimensions, inter-annotator agreement statistics, cross-VLM robustness tests, and bootstrap confidence intervals.

Critical weaknesses
  1. As a benchmark paper intended to serve the research community, OmniEdit-Bench evaluates an extremely narrow set of models, and heavily favors closed-source commercial APIs over the open-source methods that dominate academic research. Only a small number of open-source models are included, many of which are lower-resolution (480P) that are not representative of current state-of-the-art open IVE methods. Notably, the benchmark entirely omits many widely referenced open-source models and methods that are standard baselines in the field. This severely limits the utility of the benchmark for the academic community, as researchers cannot use it to benchmark their own open-source methods against a meaningful set of baselines.

  2. The field of instruction-guided video editing evaluation is already very crowded, with numerous prior benchmarks including VE-Bench, EditBoard, IVEBench, OpenVE-3M, VIE-Bench, and RISE-Bench, among many others. Against this backdrop, OmniEdit-Bench does not advance the state of the art in either depth or breadth.

  3. While the paper positions its five-track taxonomy as comprehensive, most of these dimensions have already been explored in prior work. Spatial editing at attribute/subject/global levels, reference-conditioned editing, and reasoning-based editing all have dedicated benchmarks or sections in earlier work.

  4. The benchmark is surprisingly shallow in terms of diagnostic granularity. With only 100 samples for the Audio track and 50 for the Reasoning track, sample sizes are too small for statistically robust stratified analysis. Unlike recent benchmarks that use atomic checklist items for fine-grained failure diagnosis, OmniEdit-Bench relies on coarse 1–5 Likert scores per dimension, which provide very little actionable insight into specific failure modes. Overall, the benchmark feels like a consolidation of existing evaluation ideas rather than a meaningful advance in evaluation methodology.

  5. The evaluation pipeline relies entirely on subjective VLM ratings, with no supporting automated objective metrics. Prior benchmarks typically combine VLM-based semantic evaluation with quantitative automated metrics to provide a more complete and reproducible assessment. The absence of any objective metrics makes the results less verifiable and more sensitive to prompt engineering and VLM biases, and reduces the benchmark’s value for fine-grained technical diagnosis.

  6. The paper provides only high-level descriptions of data sourcing, instruction generation, and curation procedures. Details such as inter-rater agreement during instruction curation, exclusion criteria for source videos, and the full distribution of task difficulty are missing. While the authors state that data and code will be released upon publication, the heavy reliance on closed-source commercial APIs for both model outputs and evaluation means that full reproduction of results is practically impossible for independent researchers.

  7. The Temporal and Reasoning tracks are framed as the paper’s most distinctive contributions, but their task designs remain relatively simple. For example, the Reasoning track focuses on basic physical/causal changes and simple hypothetical scenarios, without exploring more complex multi-step reasoning, compositional edits, or cross-modal reasoning. Similarly, the Temporal track does not include fine-grained evaluation of long-range temporal coherence or motion trajectory accuracy. As a result, these tracks do not meaningfully advance evaluation beyond what has already been shown in prior work.

Questions for authors

Refer to weaknesses part.

Reviewer_MzH5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OmniEdit-Bench, a benchmark for instruction-based video editing covering five tracks: spatial, temporal, audio, reference-based, and reasoning. It evaluates editing quality along four dimensions—accuracy, preservation, realism, and consistency—and proposes an accuracy-aware penalty that scales the contributions of the latter three dimensions by instruction-following accuracy. The paper evaluates several commercial and open-source models and reports human-alignment experiments to assess the reliability of its VLM-based evaluator.

Critical strengths
  1. Broad task coverage. The benchmark extends beyond spatial editing to include temporal manipulation, audio editing, reference-conditioned editing, and implicit instructions. This scope is useful for examining capabilities that are not adequately characterized by frame-level visual quality alone.

  2. Relevant evaluation objective. The paper identifies an important concern: visually plausible outputs that preserve the source video may still fail to execute the requested edit. Explicitly prioritizing instruction accuracy is a reasonable direction for evaluating editing systems.

Critical weaknesses
  1. The reasoning-track results do not isolate reasoning ability from editing and generation ability. The paper interprets low reasoning-track scores as evidence that models struggle to infer implicit instructions. However, an unsuccessful output does not establish that the model failed to understand or reason about the intended transformation; the model may infer the correct target but fail to generate it. The paper does not provide paired evaluations using implicit and explicit instructions for the same source video and target edit, which would help distinguish these failure modes. Consequently, the results demonstrate difficulty on the selected tasks but do not adequately support the stronger interpretation of a reasoning-specific bottleneck.

  2. Inconsistencies in the evaluation protocol and score definitions hinder interpretation and reproducibility. The main text identifies Gemini-3.1-Pro as the evaluator, whereas Appendix D describes Gemini-2.5-Pro. Section 2.2 defines the component scores on a 1–5 scale, while Appendix C specifies a 0–5 scale for human ratings and Appendix F permits accuracy scores of zero. This distinction affects the accuracy-aware penalty, since zero accuracy eliminates the other dimensions’ contributions whereas a minimum accuracy of one does not. Additionally, Table 9 reports bootstrap means ranging from 3.57 to 4.18 without clearly identifying the model, aggregation, or score transformation that connects these values to the substantially lower scores in Table 1. These ambiguities leave the actual evaluation procedure and the interpretation of the uncertainty estimates unclear.

  3. The human-alignment results contain statistical inconsistencies that undermine the validation of the automatic evaluator. Table 4 contains inconsistencies between the reported human-score means and MAEs, notably in the Realism group with a VLM score of 5 and the Accuracy group with a VLM score of 3. Under the stated scoring bounds and grouping procedure, these statistics cannot simultaneously hold, and the discrepancies exceed rounding error. Because the model comparisons and capability analyses rely on this evaluator, the authors should clarify the calculations, provide corrected statistics and auditable scoring data, and establish that the reliability claims and main conclusions remain supported.

Questions for authors

See Weaknesses

Reviewer_DKsPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OmniEdit-Bench, a benchmark covering spatial, temporal, audio, reference, and reasoning aspects of video editing. It evaluates accuracy, preservation, realism, and consistency, with accuracy scaling the contributions of the other dimensions. Experiments compare different editing models and include human evaluation, comparisons across VLM judges, and scoring ablations.

Critical strengths
  1. The benchmark covers a broad range of video editing tasks, including temporal, audio, and reasoning-based editing.

  2. The evaluation of both commercial and open-source models provides useful insights into current video editing capabilities.

Critical weaknesses
  1. The benchmark relies heavily on a general-purpose VLM to assess complex video editing tasks. However, it is unclear whether the VLM can reliably identify fine-grained motion changes, action order, speed, and temporal consistency errors. Prior studies such as TOMATO (ICLR 2025) and FPS-Bench (CVPR 2026) have shown significant limitations of VLMs in understanding fine-grained video dynamics. Have the authors tested whether the same VLM produces consistent scores across repeated evaluations of the same video?

  2. Existing benchmarks such as IVEBench (ICLR 2026) already provide comprehensive editing task categories and evaluate video using multiple metrics. The proposed evaluation method mainly combines four VLM scores with an accuracy-aware penalty. There is no direct comparison with existing video editing evaluators to demonstrate the advantage of the proposed evaluation method.

Questions for authors
  1. Section 2.2 defines scores in the range of 1 to 5, while Appendix C uses 0 to 5. The main text also uses Gemini-3.1-Pro, while Appendix D reports Gemini-2.5-Pro. Could the authors clarify the actual evaluation settings, the conversion to the 100-point scale, and how the overall scores are calculated when some tracks are unsupported?

  2. Can you report how reliably the judge detects errors in action order, speed, and audio-visual synchronization? How much do scores and model rankings vary across repeated evaluations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.