acceptodds
Under review as a conference paper at ICLR 2027

RVEBench: A Comprehensive Benchmark for Reference-Guided Video Editing

Abstract

Reference-guided video editing is evolving toward complex workflows that combine natural-language instructions with multiple visual references. However, existing benchmarks do not systematically construct challenge-specific cases targeting advanced capabilities and therefore fail to expose capability-specific failures. To address this gap, we introduce RVEBench, a comprehensive and diagnostic benchmark comprising 400 human-calibrated cases and 5,672 fine-grained checklist questions, with six challenging scenarios spanning Spatial Perception, Temporal Modeling, Semantic Reasoning, and Symbolic Fidelity. We further introduce RVE-Eval, a five-dimensional framework that combines sample-specific checklists and rubrics to distinguish edit execution, reference grounding, source preservation, perceptual quality, and physical or logical plausibility. Experiments on ten configurations spanning seven representative models reveal several fundamental limitations. Current systems often produce perceptually polished videos that remain physically or logically implausible. Challenging scenarios expose distinct capability-specific failure patterns. As editing requirements accumulate, instruction execution remains stable, but unintended changes and quality degradation become increasingly severe.

Primary area:
datasets and benchmarks
Keywords:
reference-guided video editing, video editing benchmarks, multimodal evaluation, video generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HmXEOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces RVEBench, a benchmark for reference-guided video editing comprising 400 human-calibrated cases with one to four editing requirements per case and 5,672 fine-grained checklist questions. The benchmark includes Basic Editing and six challenging scenarios targeting spatial perception, temporal modeling, semantic reasoning, and symbolic fidelity. The authors further propose RVE-Eval, a five-dimensional evaluation framework covering Instruction Following (IF), Reference Consistency (RC), Video Consistency (VC), Video Quality (VQ), and Video Realism (VR). The framework combines sample-specific checklists for the first three dimensions with source-anchored rubric scores for the latter two. Experiments on ten configurations spanning seven video editing models examine capability-specific failures, performance under increasing edit counts, the effects of prompt enhancement, and alternative multi-reference editing strategies.

Critical strengths
  1. Important and practically relevant evaluation setting. Reference-guided video editing increasingly requires models to coordinate multiple visual references and editing instructions. The benchmark addresses this need with challenging scenarios involving ambiguous targets, inconspicuous entities, rapid motion, multiple shots, world knowledge, and text fidelity.

  2. Structured and interpretable evaluation framework. Separating instruction execution, reference fidelity, source preservation, perceptual quality, and scene plausibility provides a more informative characterization of editing performance than a single overall score. Fixed, sample-specific checklists also support consistent comparisons across model outputs.

  3. Meaningful human calibration and validation efforts. The authors conduct human review of benchmark cases and checklists, evaluate agreement with expert preferences, and provide repeated-evaluation and cross-evaluator analyses. These efforts strengthen the empirical foundation of the proposed evaluation framework.

Critical weaknesses
  1. The coverage of the sparse checklists has not been sufficiently validated. The evaluation limits IF and RC to at most three questions per edit and VC to at most five questions per case, prioritizing a subset of key requirements. For multi-target, multi-shot, and complex-motion editing, this design may overlook errors outside the preselected checks. The existing validation primarily measures dimension-level ranking agreement and does not directly assess whether the checklists adequately cover important failure types.

  2. The experimental coverage of the multi-reference strategy comparison is limited. The conclusion that sequential editing and reference-image concatenation cannot reliably substitute for native multi-reference editing is based solely on experiments with Kling-3.0-Omni, which is insufficient to establish its generalizability across models. Additional experiments with Wan3.0 and Seedance would help further substantiate this conclusion.

Questions for authors

See Weakness

2 comments

We sincerely thank the reviewer for the positive assessment and the constructive suggestions. We are glad that the reviewer finds the evaluation setting important and practical, the five-dimensional framework structured and interpretable, and the human calibration and validation meaningful. Below we address each concern in turn.


W1: Coverage of the sparse checklists

We thank the reviewer for raising this point. We limit the number of checklist questions so that low-value questions do not dilute the weight of key requirements in the checklist score. To directly measure whether the checklists miss important errors in actual model outputs, we conducted a human coverage audit on the same 60 stratified cases and five configurations used in the Human Agreement Analysis (Section 4.3).

  • Protocol. For each case, annotators reviewed the source video, editing instructions, reference images, and the edited videos from all five configurations, together with the IF, RC, and VC checklists. For each edited video and each dimension, they identified every error that is not captured by any checklist question, i.e., an error that no question in the checklist checks and that the checklist score therefore cannot reflect. For each missed error, they recorded:
    • its dimension (IF, RC, or VC);
    • its severity: major if the error alone constitutes a core failure of the dimension; minor otherwise.
  • Annotators. Each case was audited independently by two of three expert annotators who were not involved in dataset construction or calibration, and disagreements were resolved by the remaining annotator.

Table R1. Checklist coverage audit on the 60 cases × 5 configurations. A unit is an edit in one edited video for IF and RC, and one edited video for VC. "Miss rate" is the percentage of units with at least one missed error.

DimensionMiss rate (any)Miss rate (major)Miss rate (minor only)Missed errors per unit
Instruction Following2.34%0.00%2.34%0.0234
Reference Consistency1.08%0.00%1.08%0.0108
Video Consistency5.00%0.00%5.00%0.0500

The audit shows that the checklists capture the large majority of errors in the edited videos, and all missed errors are minor. These missed errors are long-tail side effects outside the edit targets and hard to anticipate when constructing the checklists. For example, the VC checklist asks whether the layout of the background scene is preserved, while a pedestrian absent from the source video appears in the background of the edited video. Covering such deviations would require many speculative questions, which would in turn dilute the weight of key requirements.

We have added the checklist coverage audit (Table R1) to Appendix C.2 and its annotation protocol to Appendix H in the revised version (highlighted in blue).

W2: Generality of the multi-reference strategy comparison

We thank the reviewer for this suggestion. We repeat the comparison of joint, sequential, and grid-based editing on Wan3.0-Video, using the cases with two to four edits as in Figure 5(d) and excluding those blocked by provider-side safety filters (246 cases in total). On this subset, Kling-3.0-Omni shows the same trends in Instruction Following and Reference Consistency as in Figure 5(d).

Table R2. Comparison of multi-reference editing strategies on Wan3.0-Video over the 246 cases completed under all three strategies. The number of cases for each edit count is given in parentheses. The best score for each edit count is shown in bold.

EditsStrategyIFRCVCVQVR
2 (144)Sequential83.083.389.976.566.7
Joint89.288.591.178.163.5
Grid88.186.791.975.365.9
3 (69)Sequential86.886.589.968.166.3
Joint90.289.591.772.768.0
Grid87.886.391.268.768.6
4 (33)Sequential92.786.384.973.571.2
Joint95.688.083.474.561.4
Grid91.286.290.281.365.9

On Wan3.0-Video, joint editing again achieves the highest Instruction Following and Reference Consistency at every edit count. Across two to four edits, sequential editing trails it by 2.9–6.2 and 1.7–5.2 points, and the grid-based strategy trails it by 1.1–4.4 and 1.8–3.2 points, respectively. This confirms that neither alternative reliably substitutes for native multi-reference editing in condition alignment.

However, the effects on content preservation and generation quality observed on Kling-3.0-Omni do not generalize to Wan3.0-Video. Sequential editing does not show widening deficits in Video Consistency or Video Quality, and the grid-based strategy does not consistently improve Video Quality or Video Realism. We have therefore removed these conclusions from Section 4.3 and retained only the conclusions on Instruction Following and Reference Consistency, which hold for both models.

Regarding Seedance, its service does not accept reference images or videos that contain real people unless identity verification or prior legal authorization is provided, and it also filters inputs involving copyrighted content. Since the source videos and reference images of RVEBench frequently contain real people and recognizable characters, most cases were rejected by these filters, so a meaningful comparison on Seedance could not be obtained.

We have added the Wan3.0-Video results to Appendix E (Table 11) and revised Section 4.3 accordingly in the revised version (highlighted in blue).


We hope these responses address the reviewer's concerns, and we would be happy to provide further clarification on any remaining questions.

Reviewer_i9o9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces RVEBench, a benchmark for reference-guided video editing with 400 human-calibrated cases and 5k sample-specific checklist questions. It contains 160 Basic Editing cases and 240 Advanced cases covering six challenging scenarios: Target Disambiguation, Inconspicuous Entities, Fast Movement, Multi-Shot Editing, World Knowledge, and Text Fidelity. RVE-Eval assesses model outputs along five dimensions: Instruction Following, Reference Consistency, Video Consistency, Video Quality, and Video Realism. The authors evaluate ten configurations from seven models and further analyze challenging scenarios, edit count, prompt enhancement, and multi-reference editing strategies.

Overall, RVEBench addresses an important problem and presents a clearly organized benchmark with six challenging scenarios and a five-dimensional evaluation framework. My main concerns are the use of Overall Avg as the main aggregate score, the capability-specific interpretation of the Advanced Scenarios, and whether the current validation adequately supports the sample-specific checklist evaluation.

Critical strengths
  1. The benchmark goes beyond a taxonomy based only on editing dimensions. The six challenging scenarios cover target identification, inconspicuous entities, fast movement, multi-shot editing, world knowledge, and text fidelity, all of which are relevant limitations of current reference-guided video editing models.

  2. The five evaluation dimensions provide a useful breakdown of editing performance. In particular, separating Instruction Following, Reference Consistency, and Video Consistency helps distinguish failures in edit execution, reference grounding, and source preservation. Video Quality and Video Realism further separate perceptual quality from physical or logical plausibility.

  3. The experimental section offers more than a leaderboard. The paper analyzes challenging scenarios, editing dimensions, edit count, prompt enhancement, multi-reference editing strategies, and agreement with human judgments. I found several of these observations useful for understanding where current systems fail.

Critical weaknesses
  1. The use of Overall Avg as the main aggregate score is not sufficiently justified. The five dimensions measure different aspects of an editing result, yet Table 2 combines them using an unweighted arithmetic mean without showing that this number reflects overall editing success. Because VC rewards source preservation, while VQ and VR are source-anchored, these dimensions may compensate for poor IF and RC. How well would an unchanged source video score? Such an output would fail the requested edit, but could still retain high VC, VQ, VR. The paper should report this baseline and test whether the model ranking remains stable when failure in IF limits the contribution of the other dimensions.

  2. The comparison between Basic Editing and the Advanced Scenarios does not isolate the named capabilities. Figure 4 compares different sets of cases rather than controlled variants of the same editing task. These subsets may differ in editing dimension, reference and edit counts, video duration, target visibility, occlusion, shot structure, and content domain. The results therefore show different scores and distinct degradation patterns across subsets, but they do not establish that Fast Movement, Multi-Shot Editing, or another named challenge independently causes the observed difference.

  3. The Human Agreement Analysis validates model-level rankings rather than the individual checklist questions used by RVE-Eval. Human annotators compare five configurations pairwise, and the resulting ranking is correlated with the RVE-Eval ranking. This is useful evidence that RVE-Eval broadly orders the selected systems similarly to humans, but it does not establish that each yes/no judgment is correct or that the selected questions cover all important failures. As sample-specific checklist evaluation is a central part of the paper, a direct human evaluation of a representative subset of checklist questions would better support the fine-grained evaluation claims, especially if it reports judgment accuracy and whether important failures were omitted.

  4. Some conclusions in the Further Analysis are stronger than the reported measurements support. IF is evaluated at the individual-edit level, so a stable average score across one to four edits does not show that all requested edits in a case are completed successfully. A four-edit case with one failed edit may still receive a high average score. Reporting the percentage of cases in which all editing requirements are satisfied would be more informative.

  5. The related-work comparison is too narrow for the paper’s contribution claims. Table 1 mainly focuses on reference-guided benchmarks, while several broader video-editing benchmarks are directly relevant to the design of RVE-Eval. For example, CoVEBench also uses fine-grained checklist evaluation, VEFX-Bench separates Instruction Following, Rendering Quality, and Edit Exclusivity, UniEditBench uses multidimensional evaluation. These works do not use exactly the same input setting as RVEBench, but they matter for understanding what is new here.

  6. The criteria for selecting the seven evaluated models are not stated. The paper cites closely related systems such as OmniWeaving and VINO, but does not explain why they were not included or whether their available interfaces were incompatible with RVEBench. At minimum, the inclusion and exclusion criteria should be made explicit.

Questions for authors
  1. What scores does an output identical to the source video receive on the five evaluation dimensions and on Overall Avg? Does the Table 2 ranking remain stable under an aggregation in which failure in Instruction Following limits the contribution of the other dimensions?
  2. Since Basic Editing and the Advanced Scenarios contain different cases, can the authors provide paired cases or an analysis controlling for major differences among the subsets?
  3. How were the five “representative configurations” in Figure 4 selected? Do the same conclusions hold across all ten configurations and with confidence intervals?
  4. Beyond the human calibration of checklist design, have the evaluator’s individual yes/no judgments been directly compared with human answers? If so, what is the item-level agreement, and how often are important failures omitted from the selected questions?
  5. Can the edit-count analysis report the percentage of cases in which all requested editing requirements are satisfied, rather than only average Instruction Following over individual edits?
  6. Beyond the benchmark-level comparison in Table 1, what empirical advantage does RVE-Eval provide over closely related multidimensional or checklist-based evaluation protocols when applied to the same outputs and human judgments?
6 comments

We sincerely thank the reviewer for the thorough and constructive review. We are glad that the reviewer finds the six challenging scenarios relevant, the five-dimensional decomposition useful, and the analyses informative beyond a leaderboard. Below we address each concern in turn.


W1 & Q1: Overall Avg. as a summary score

We thank the reviewer for this insightful point and agree. On all 400 cases of RVEBench, we evaluated the source video itself as the edited output (identity output) with the same RVE-Eval protocol.

Table R1. Scores of the identity output on all 400 cases.

OutputIFRCVCVQVROverall Avg.
Identity output (source video)3.140.1197.2790.6382.2154.67

Despite its low IF and RC, the identity output still obtains an Overall Avg. of 54.67.

Overall Avg. thus cannot measure editing success and should be read only as an auxiliary summary of the five dimensions.

Ranking stability under IF-gated aggregation. To answer the reviewer's question directly, we recomputed all configurations under two aggregations in which IF limits the contribution of the other dimensions:

  • Soft gate: the RC score of each edit and the VC/VQ/VR scores of each case are multiplied by the corresponding IF/100.
  • Hard gate: the RC score of an edit counts only if its IF ≥ 50, and the VC/VQ/VR scores of a case count only if every edit in the case has IF ≥ 50; otherwise they are set to 0.

Table R2. Scores and ranks under different aggregations.

ConfigurationOverall Avg. (Table 2)Soft-gatedHard-gated
MiniMax-H384.66 (#1)77.87 (#1)78.02 (#1)
Bernini77.00 (#2)67.77 (#3)66.53 (#3)
Kling-3.0-Omni76.75 (#3)64.23 (#5)62.87 (#5)
Bernini-R (14B)76.51 (#4)66.95 (#4)64.56 (#4)
Wan2.7-VideoEdit75.78 (#5)69.20 (#2)69.73 (#2)
ViduQ2-Pro68.89 (#6)53.50 (#7)51.35 (#6)
Kling-O168.49 (#7)52.15 (#8)48.25 (#8)
Bernini-R (1.3B)68.04 (#8)54.78 (#6)51.16 (#7)
UniVideo (hidden)49.47 (#9)21.46 (#9)16.81 (#9)
UniVideo (query)48.31 (#10)19.21 (#10)15.27 (#10)
Kendall's τ vs. Table 2 (10 configurations)–0.7330.778

Under both gated aggregations, MiniMax-H3 remains first and the tier structure is unchanged: positions 2–5, 6–8, and 9–10 are occupied by the same configurations as in Table 2. Rank changes occur only within tiers, whose members are already nearly tied under Overall Avg. (within 1.22 points of each other).

On aggregation. We believe that no single aggregation fits all scenarios. IF gating reflects a priority on completing the requested edits; in other applications, however, a visually broken video is as unacceptable as a failed edit, and gating on IF alone would underweight the latter failure. We will therefore extend the evaluation code we release to support user-defined weights for each dimension, as well as soft or hard gating on the score of any chosen dimension, so that users can define an aggregation that matches their own scenario.

We have clarified in the caption of Table 2 that Overall Avg. is an auxiliary summary rather than a measure of editing success, and added the identity-output results and the IF-gated rankings in Table R2 to Appendix B.4 in the revised version (highlighted in blue).

W2 & Q2 & Q3: Basic vs. Advanced comparison and capability-specific interpretation

Q3: Selection of the five configurations and results on all ten. The five configurations in Figure 4 are the top five by Overall Avg. in Table 2: MiniMax-H3, Bernini, Kling-3.0-Omni, Bernini-R (14B), and Wan2.7-VideoEdit. We chose them to characterize the failure patterns of the strongest current systems.

As suggested, we recomputed every scenario-level finding in Section 4.3 over all ten configurations. 95% confidence intervals are obtained by case-level bootstrap with 1,000 resamples, where Basic cases and scenario cases are resampled independently.

Table R3. Score decline of each scenario relative to Basic Editing (points). "Paper" is the value reported in Section 4.3 (top five configurations); the remaining columns use all ten configurations.

ScenarioDimensionPaper (top 5)All 1095% CI (all 10)Configs with a decline
Target DisambiguationIF18.219.6[13.3, 25.7]10 / 10
Inconspicuous EntitiesIF15.319.3[13.0, 26.0]10 / 10
Fast MovementRC17.016.1[11.3, 21.1]10 / 10
Fast MovementVR15.916.8[4.0, 29.2]10 / 10
Fast MovementVC6.413.0[8.4, 17.4]8 / 10
Multi-Shot EditingVC22.638.5[34.3, 42.8]10 / 10
Multi-Shot EditingIF16.016.6[12.4, 20.7]10 / 10
World KnowledgeIF10.510.2[4.3, 16.5]9 / 10
World KnowledgeVR4.97.0[−3.0, 16.8]10 / 10
Text FidelityRC6.06.3[1.1, 12.0]8 / 10

All ten findings keep the same direction over the ten configurations, and the declines are similar to or larger than those reported in the paper. Nine of the ten confidence intervals exclude zero. The exception is the Video Realism decline under World Knowledge: although all ten configurations show a decline, its confidence interval includes zero, so this decline should be regarded as a tendency.

W2 & Q2: Controlled comparison. We thank the reviewer for this point and would like to clarify the scope of our claim. RVEBench is designed to measure how models perform when a given challenge is present in real footage, rather than to isolate the causal effect of each factor. Diagnostic benchmarks commonly evaluate each capability on a dedicated set of samples; for example, VBench constructs a dedicated prompt suite for each evaluation dimension. Following this practice, we build each challenging scenario from real videos in which the targeted challenge is prominent. The findings in Section 4.3 therefore characterize how current models perform in each challenging scenario, rather than attributing each decline to the named challenge alone.

Within this scope, the findings are robust: as shown in Table R3, all of them keep the same direction over all ten configurations, and nine of the ten declines have confidence intervals that exclude zero.

A strictly controlled design would require source videos that differ only in the targeted factor while all other conditions are identical, e.g., with or without similar distractors, a visible or inconspicuous target, slow or fast motion, and a single shot or multiple shots. Since these factors are intrinsic to real footage, varying one while fixing the rest would require multiple recordings of the same content, which is extremely difficult to obtain at scale.

W3 & Q4: Item-level human verification of checklist judgments

We thank the reviewer for this point. We add two analyses on the same 60 cases and five configurations used in Table 3.

(1) Accuracy of individual yes/no judgments. The six annotators of Table 3 independently answered every IF, RC, and VC checklist question for each edited video. Among the items a human answered yes and those a human answered no, we separately compute the proportion on which the evaluator gave the same answer, averaged over annotators. The results are shown in Table R4.

Table R4. Agreement on checklist yes/no judgments.

DimensionHuman–human Fleiss' κEvaluator–human agreement (yes)Evaluator–human agreement (no)
Instruction Following0.8591.38%89.93%
Reference Consistency0.7989.82%87.59%
Video Consistency0.8692.47%90.22%

The high human–human κ indicates that the checklist questions can be answered consistently. In all three dimensions, the evaluator agrees closely with humans on both yes and no items.

(2) Frequency of missed important failures. To measure how often the selected questions miss important failures, we conducted a human coverage audit on the same cases and configurations.

  • Protocol. For each case, annotators reviewed the source video, editing instructions, reference images, and the edited videos from all five configurations, together with the IF, RC, and VC checklists. For each edited video and each dimension, they identified every error that is not captured by any checklist question, i.e., an error that no question in the checklist checks and that the checklist score therefore cannot reflect. For each missed error, they recorded:
    • its dimension (IF, RC, or VC);
    • its severity: major if the error alone constitutes a core failure of the dimension; minor otherwise.
  • Annotators. Each case was audited independently by two of three expert annotators who were not involved in dataset construction or calibration, and disagreements were resolved by the remaining annotator.

Table R5. Checklist coverage audit on the 60 cases × 5 configurations. A unit is an edit in one edited video for IF and RC, and one edited video for VC. "Miss rate" is the percentage of units with at least one missed error.

DimensionMiss rate (any)Miss rate (major)Miss rate (minor only)Missed errors per unit
Instruction Following2.34%0.00%2.34%0.0234
Reference Consistency1.08%0.00%1.08%0.0108
Video Consistency5.00%0.00%5.00%0.0500

The audit shows that the checklists capture the large majority of errors in the edited videos, and all missed errors are minor. These missed errors are long-tail side effects outside the edit targets and hard to anticipate when constructing the checklists. For example, the VC checklist asks whether the layout of the background scene is preserved, while a pedestrian absent from the source video appears in the background of the edited video. Covering such deviations would require many speculative questions, which would dilute the weight of key requirements.

We have added the checklist-level agreement (Table R4) and the checklist coverage audit (Table R5) to Appendix C.1 and C.2, respectively, and their annotation protocols to Appendix H in the revised version (highlighted in blue).

Reviewer_3VH9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RVEBench, a reference-guided video editing benchmark containing 400 cases and 5,672 checklist questions, including foundational editing and six categories of advanced scenarios. It also proposes RVE-Eval, which evaluates results along five dimensions: instruction following, reference consistency, video consistency, video quality, and video realism. The authors evaluate seven models under ten configurations and analyze the benchmark and evaluation method through human comparison, repeated evaluation, and cross-evaluator comparison.

Critical strengths

1.The benchmark is not a simple expansion in the number of tasks. Rather, it designs dedicated scenarios around capabilities such as target identification, fast motion, multi-shot editing, commonsense reasoning, and text fidelity, and uses different evaluation dimensions to distinguish issues such as edit execution, reference matching, and source-video preservation. It therefore has good diagnostic value.

2.The experiments present several useful patterns, such as video realism lagging behind perceptual quality, multiple editing requirements exacerbating damage to source content, and different scenarios exhibiting different capability weaknesses. The authors also report human comparison and multiple stability analyses.

Critical weaknesses

1.The human validation provides useful evidence of rank alignment, but its scope is limited: correlations are computed across only five configurations for each of 60 cases and then averaged, without reporting checklist-level or absolute-score agreement, inter-annotator reliability, or uncertainty. Thus, the study does not yet establish that RVE-Eval’s absolute dimension scores are well calibrated.

2.Table 8 excludes provider-blocked outputs without clearly stating per-model denominators or common-case rankings, model outputs use different resolutions and frame rates, and Figure 3 does not explain whether UniVBench was evaluated with comparably constructed and calibrated sample-specific checklists. These factors may affect score comparability; a common-subset analysis and clearer protocol details, or more qualified claims, would address this concern.

Questions for authors

1.Can you supplement the correspondence between human judgments and RVE-Eval at the checklist-item or absolute-score level, inter-annotator agreement, and confidence intervals for the relevant results? Is the current average rank correlation computed based on five configurations sufficient to support interpretation of absolute scores?

2.For the failed or excluded cases in Table 8, how many cases serve as the denominator for each model’s score? If scores are recomputed only on the common cases completed by all models, do the main rankings in Table 2 remain unchanged?

3.When Figure 3 compares RVEBench with UniVBench, did the UniVBench samples also use per-sample checklist items? If so, how were the checklist items constructed and calibrated? If not, how is the comparability of the evaluation results between the two ensured?

3 comments

We sincerely thank the reviewer for the careful reading and constructive feedback. We are glad that the reviewer finds RVEBench to have good diagnostic value and the experimental findings useful. Below we address each concern in turn.


W1 & Q1: Item-level / absolute-score agreement, and inter-annotator reliability

We thank the reviewer for this suggestion. Building on the ranking agreement in Table 3, we add three analyses: the inter-annotator reliability of the human annotations, item-level agreement on checklist judgments, and absolute-score agreement across all five dimensions.

(1) Inter-annotator reliability of the Table 3 annotations. Based on the pairwise comparisons made by the six annotators in Table 3, we compute Fleiss' κ for each dimension. κ ranges from 0.69 to 0.83 across the five dimensions, indicating that the human annotations in Table 3 are reliable.

Table R1. Fleiss' κ of the pairwise annotations in Table 3.

DimensionIFRCVCVQVR
Fleiss' κ0.830.780.810.720.69

(2) Item-level agreement on checklist judgments. On the 60 cases and five configurations used in Table 3, the six annotators of Table 3 independently answered yes or no to each checklist question. Among the items a human answered yes and those a human answered no, we separately compute the proportion on which the evaluator gave the same answer, averaged over annotators.

Table R2. Agreement on checklist yes/no judgments.

DimensionHuman–human Fleiss' κEvaluator–human agreement (yes)Evaluator–human agreement (no)
Instruction Following0.8591.38%89.93%
Reference Consistency0.7989.82%87.59%
Video Consistency0.8692.47%90.22%

The high human–human κ indicates that the checklist questions can be answered consistently. In all three dimensions, the evaluator agrees closely with humans on both yes and no items.

(3) Absolute-score agreement. We compute absolute human scores on the same 60 cases and five configurations using the same formulas as RVE-Eval: for IF, RC, and VC, the human answers from (2) are scored with Eq. 2; for VQ and VR, annotators rate the source and edited videos from 1 to 5 under the same rubrics (Appendix K.2), converted with Eq. 1. We compare human and RVE-Eval scores using MAE, with annotator-vs-rest MAE as the human reference.

Table R3. Absolute-score agreement. "Config-level |Δ|" is the mean absolute difference between human and RVE-Eval scores over the five configurations.

DimensionMAE (evaluator–human)MAE (human–human)Config-level |Δ|
Instruction Following5.143.951.25
Reference Consistency5.584.221.43
Video Consistency4.713.471.29
Video Quality11.178.952.62
Video Realism12.199.762.86

RVE-Eval's agreement with humans is comparable to that between human annotators across all five dimensions, indicating that its absolute scores are reasonably well calibrated.

In summary, RVE-Eval agrees with human judgments not only in model rankings but also in individual checklist judgments and absolute scores across all five dimensions. We have added Fleiss' κ to Table 3, the checklist-level and absolute-score agreement to Appendix C, and the annotation protocol to Appendix H in the revised version (highlighted in blue).

W2 & Q2: Denominators for excluded outputs and rankings on the common subset

A small number of outputs were blocked by provider-side safety filters (Table 8 in the original submission, now Table 14) and could not be scored. Each configuration's score is therefore computed over the cases it successfully generated:

ConfigurationScored cases
Wan2.7-VideoEdit393 / 400
ViduQ2-Pro394 / 400
MiniMax-H3394 / 400
All other configurations400 / 400

To remove any effect of differing denominators, we recomputed all results on the common subset of 385 cases that every configuration completed successfully (Basic: 158 cases; Advanced: 227 cases):

Table R4. Results of all 10 configurations on the common subset (385 cases). Values in parentheses under Overall Avg. denote the change relative to Table 2; Rank denotes the Overall Avg. rank on the full benchmark → on the common subset.

MethodBasic IFRCVCVQVRAvg.Adv. IFRCVCVQVRAvg.Overall Avg.Rank
Kling-3.0-Omni87.2378.1588.2881.4368.3080.6868.1973.7282.0379.1965.4273.7176.95 (+0.20)3 → 3
Wan2.7-VideoEdit95.1885.3689.2170.1562.4580.4781.2679.0278.4764.9857.6772.2875.97 (+0.19)5 → 5
ViduQ2-Pro81.6477.2277.0673.4765.4074.9662.2372.4261.9369.3857.7864.7569.33 (+0.44)6 → 6
Kling-O177.1070.8682.8374.6364.8274.0555.6362.3972.2671.5559.4364.2568.76 (+0.27)7 → 7
MiniMax-H391.7487.8289.6584.5571.6885.0989.4287.2187.8387.6369.8284.3884.72 (+0.06)1 → 1
Bernini92.1584.0384.9972.6367.3580.2376.6773.5883.3275.9963.4074.5977.32 (+0.32)2 → 2
Bernini-R (14B)89.0181.5587.4067.1961.9277.4275.9070.8591.4376.9562.7475.5776.71 (+0.20)4 → 4
Bernini-R (1.3B)82.7174.2477.7863.2458.7671.3562.8963.3982.6963.1854.9965.4368.35 (+0.31)8 → 8
UniVideo (hidden)34.9830.7077.8262.0860.5553.2326.9232.9361.7057.4957.7547.3649.86 (+0.39)9 → 9
UniVideo (query)28.9424.5582.1062.9261.9252.0924.4029.1565.2758.8853.9346.3348.69 (+0.38)10 → 10

The main results remain unchanged. On the common subset, the Overall Avg. ranking is identical to that of Table 2 (Kendall's τ = 1.00), as are the rankings by Basic Avg. and Advanced Avg. The maximum absolute change in any configuration's Overall Avg. is 0.44 points, and all main findings in Section 4 continue to hold. We have clarified the denominators and added the common-subset results to Appendix I (Table 15) in the revised version (highlighted in blue).

W2: Differences in output resolution and frame rate

  • Frame rate. The RVE-Eval evaluator uniformly samples every video at 2 FPS, regardless of the model that produced it (Appendix I). Differences in native output frame rate (16–30 FPS) are therefore removed at the evaluator's input, and all configurations are judged on the same temporal sampling.
  • Resolution. Each model is run at its officially recommended output setting (Table 7 in the original submission, now Table 13). Scores thus reflect each model's performance under its actual usage conditions, which is what practitioners experience when deploying these systems.

W2 & Q3: Comparability of the UniVBench comparison (Figure 3)

Yes. The UniVBench samples were processed with exactly the same pipeline as RVEBench, rather than reusing UniVBench's original evaluation:

  1. Checklist construction. Sample-specific checklists for IF, RC, and VC were generated with the same MLLMs, the same dimension-specific prompts (Appendix K.1), and the same per-edit and per-case question limits.
  2. Calibration. The resulting samples and checklists went through the same criteria-guided human calibration as RVEBench (Section 3.2).
  3. Evaluation. Outputs were scored with the identical RVE-Eval protocol: the same evaluator, inference configuration, checklist-based scoring (Eq. 2), and rubric scoring (Eq. 1).

Because the checklist construction, calibration, and evaluation are held fixed, the score gap in Figure 3 reflects differences in case difficulty rather than differences in how the two benchmarks are evaluated. We have clarified this in Section 3.4 of the revised version (highlighted in blue).

Ethics concerns: provenance, licensing, privacy, takedown, and usage restrictions

We thank the reviewer for raising these important points and fully agree that they must be addressed before release. We commit to the following measures:

  1. Provenance and licensing. Each source video and reference image will be released with its source, URL, and license information.
  2. Release scope. Openly licensed media (e.g., from Pexels), our synthesized images, and all annotations and checklists will be released directly. Content whose redistribution is restricted, such as YouTube and Bilibili videos and web-searched images, will be released only as URLs together with download and processing scripts.
  3. Privacy screening. All media were screened for sensitive content and privacy risks during collection, and the screening criteria and procedure will be documented with the release.
  4. Takedown process. We will provide a dedicated contact email on the dataset page. Any rights holder or depicted individual can request the removal of an item through this channel.
  5. Usage restrictions and abuse reporting. The dataset will be distributed through gated access, with users agreeing to the terms before download. The annotations are licensed for non-commercial research use only. The terms explicitly prohibit using the data to create deceptive, defamatory, or non-consensual content involving real people. The same contact channel serves as an abuse feedback mechanism, and access will be revoked for users found to violate these terms.

We have updated the Ethics Statement accordingly in the revised version (highlighted in blue).


We hope these responses address the reviewer's concerns, and we would be happy to provide further clarification on any remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Revision Summary

We sincerely thank all reviewers for their time and constructive feedback. We are encouraged that the reviewers find the challenging scenarios relevant and of diagnostic value (3VH9, i9o9, HmXE), the five-dimensional evaluation framework useful and interpretable (3VH9, i9o9, HmXE), the analyses informative beyond a leaderboard (3VH9, i9o9), and the human calibration and validation meaningful (HmXE).

We have uploaded a revised version of the paper, with all changes highlighted in blue. The main revisions are summarized below.

1. Fine-grained human validation of RVE-Eval (3VH9, i9o9, HmXE).

  • Added Fleiss' κ of the human annotations to Table 3.
  • Added Appendix C with three analyses on the 60 cases and five configurations of the Human Agreement Analysis: item-level agreement on checklist judgments (Appendix C.1, Table 7), a human audit of checklist coverage (Appendix C.2, Table 8), and absolute-score agreement across all five dimensions (Appendix C.3, Table 9).
  • Added the corresponding annotation protocols to Appendix H.

2. Score aggregation (i9o9).

  • Clarified in the caption of Table 2 that Overall Avg. is an auxiliary summary rather than a measure of editing success.
  • Added the scores of the identity output and the rankings under IF-gated aggregation to Appendix B.4 (Table 6).

3. Increasing edit counts (i9o9).

  • Clarified in the Abstract, Introduction, and Section 4.3 that the stable instruction execution refers to individual edits.
  • Added the case-level success rate under increasing edit counts to Appendix D (Table 10).

4. Multi-reference editing strategies (HmXE).

  • Added the comparison of joint, sequential, and grid-based editing on Wan3.0-Video to Appendix E (Table 11).
  • Revised Section 4.3 to retain only the conclusions that hold for both Kling-3.0-Omni and Wan3.0-Video.

5. Evaluation protocol and comparability (3VH9).

  • Clarified in Appendix I that each configuration is scored over its successfully generated cases, and added the results on the common subset of 385 cases (Table 15).
  • Clarified in Section 3.4 that UniVBench samples undergo the same checklist generation, human calibration, and RVE-Eval scoring as RVEBench.

6. Related work (i9o9).

  • Added a discussion of CoVEBench, VEFX-Bench, and UniEditBench to Section 2.

7. Ethics Statement (3VH9).

  • Specified the provenance and licensing information, release scope, privacy screening, takedown process, and usage restrictions for the dataset release.

We address each reviewer's comments in detail in the individual responses. We thank the reviewers again and are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.