- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces RVEBench, a benchmark for reference-guided video editing comprising 400 human-calibrated cases with one to four editing requirements per case and 5,672 fine-grained checklist questions. The benchmark includes Basic Editing and six challenging scenarios targeting spatial perception, temporal modeling, semantic reasoning, and symbolic fidelity. The authors further propose RVE-Eval, a five-dimensional evaluation framework covering Instruction Following (IF), Reference Consistency (RC), Video Consistency (VC), Video Quality (VQ), and Video Realism (VR). The framework combines sample-specific checklists for the first three dimensions with source-anchored rubric scores for the latter two. Experiments on ten configurations spanning seven video editing models examine capability-specific failures, performance under increasing edit counts, the effects of prompt enhancement, and alternative multi-reference editing strategies.
Critical strengths
-
Important and practically relevant evaluation setting. Reference-guided video editing increasingly requires models to coordinate multiple visual references and editing instructions. The benchmark addresses this need with challenging scenarios involving ambiguous targets, inconspicuous entities, rapid motion, multiple shots, world knowledge, and text fidelity.
-
Structured and interpretable evaluation framework. Separating instruction execution, reference fidelity, source preservation, perceptual quality, and scene plausibility provides a more informative characterization of editing performance than a single overall score. Fixed, sample-specific checklists also support consistent comparisons across model outputs.
-
Meaningful human calibration and validation efforts. The authors conduct human review of benchmark cases and checklists, evaluate agreement with expert preferences, and provide repeated-evaluation and cross-evaluator analyses. These efforts strengthen the empirical foundation of the proposed evaluation framework.
Critical weaknesses
-
The coverage of the sparse checklists has not been sufficiently validated. The evaluation limits IF and RC to at most three questions per edit and VC to at most five questions per case, prioritizing a subset of key requirements. For multi-target, multi-shot, and complex-motion editing, this design may overlook errors outside the preselected checks. The existing validation primarily measures dimension-level ranking agreement and does not directly assess whether the checklists adequately cover important failure types.
-
The experimental coverage of the multi-reference strategy comparison is limited. The conclusion that sequential editing and reference-image concatenation cannot reliably substitute for native multi-reference editing is based solely on experiments with Kling-3.0-Omni, which is insufficient to establish its generalizability across models. Additional experiments with Wan3.0 and Seedance would help further substantiate this conclusion.
Questions for authors
See Weakness
2 comments
W2: Generality of the multi-reference strategy comparison
We thank the reviewer for this suggestion. We repeat the comparison of joint, sequential, and grid-based editing on Wan3.0-Video, using the cases with two to four edits as in Figure 5(d) and excluding those blocked by provider-side safety filters (246 cases in total). On this subset, Kling-3.0-Omni shows the same trends in Instruction Following and Reference Consistency as in Figure 5(d).
Table R2. Comparison of multi-reference editing strategies on Wan3.0-Video over the 246 cases completed under all three strategies. The number of cases for each edit count is given in parentheses. The best score for each edit count is shown in bold.
| Edits | Strategy | IF | RC | VC | VQ | VR |
|---|---|---|---|---|---|---|
| 2 (144) | Sequential | 83.0 | 83.3 | 89.9 | 76.5 | 66.7 |
| Joint | 89.2 | 88.5 | 91.1 | 78.1 | 63.5 | |
| Grid | 88.1 | 86.7 | 91.9 | 75.3 | 65.9 | |
| 3 (69) | Sequential | 86.8 | 86.5 | 89.9 | 68.1 | 66.3 |
| Joint | 90.2 | 89.5 | 91.7 | 72.7 | 68.0 | |
| Grid | 87.8 | 86.3 | 91.2 | 68.7 | 68.6 | |
| 4 (33) | Sequential | 92.7 | 86.3 | 84.9 | 73.5 | 71.2 |
| Joint | 95.6 | 88.0 | 83.4 | 74.5 | 61.4 | |
| Grid | 91.2 | 86.2 | 90.2 | 81.3 | 65.9 |
On Wan3.0-Video, joint editing again achieves the highest Instruction Following and Reference Consistency at every edit count. Across two to four edits, sequential editing trails it by 2.9–6.2 and 1.7–5.2 points, and the grid-based strategy trails it by 1.1–4.4 and 1.8–3.2 points, respectively. This confirms that neither alternative reliably substitutes for native multi-reference editing in condition alignment.
However, the effects on content preservation and generation quality observed on Kling-3.0-Omni do not generalize to Wan3.0-Video. Sequential editing does not show widening deficits in Video Consistency or Video Quality, and the grid-based strategy does not consistently improve Video Quality or Video Realism. We have therefore removed these conclusions from Section 4.3 and retained only the conclusions on Instruction Following and Reference Consistency, which hold for both models.
Regarding Seedance, its service does not accept reference images or videos that contain real people unless identity verification or prior legal authorization is provided, and it also filters inputs involving copyrighted content. Since the source videos and reference images of RVEBench frequently contain real people and recognizable characters, most cases were rejected by these filters, so a meaningful comparison on Seedance could not be obtained.
We have added the Wan3.0-Video results to Appendix E (Table 11) and revised Section 4.3 accordingly in the revised version (highlighted in blue).
We hope these responses address the reviewer's concerns, and we would be happy to provide further clarification on any remaining questions.
We sincerely thank the reviewer for the positive assessment and the constructive suggestions. We are glad that the reviewer finds the evaluation setting important and practical, the five-dimensional framework structured and interpretable, and the human calibration and validation meaningful. Below we address each concern in turn.
W1: Coverage of the sparse checklists
We thank the reviewer for raising this point. We limit the number of checklist questions so that low-value questions do not dilute the weight of key requirements in the checklist score. To directly measure whether the checklists miss important errors in actual model outputs, we conducted a human coverage audit on the same 60 stratified cases and five configurations used in the Human Agreement Analysis (Section 4.3).
Table R1. Checklist coverage audit on the 60 cases × 5 configurations. A unit is an edit in one edited video for IF and RC, and one edited video for VC. "Miss rate" is the percentage of units with at least one missed error.
The audit shows that the checklists capture the large majority of errors in the edited videos, and all missed errors are minor. These missed errors are long-tail side effects outside the edit targets and hard to anticipate when constructing the checklists. For example, the VC checklist asks whether the layout of the background scene is preserved, while a pedestrian absent from the source video appears in the background of the edited video. Covering such deviations would require many speculative questions, which would in turn dilute the weight of key requirements.
We have added the checklist coverage audit (Table R1) to Appendix C.2 and its annotation protocol to Appendix H in the revised version (highlighted in blue).