- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces VEFX-Dataset, VEFX-Reward (a Qwen3-VL-based ordinal-regression reward model), and VEFX-Bench (300 curated video-prompt pairs). It benchmarks 10 commercial/open-source video editing models and shows VEFX-Reward outperforms generic VLM judges and prior reward models.
Critical strengths
The dataset and annotation protocol are solid contributions.
- The three-dimensional decoupled evaluation (IF / RQ / EE) is well motivated and supported by weak inter-dimension correlations; the (IF=1, RQ=4, EE=4) pattern concretely shows why a single scalar is insufficient.
- The annotation protocol is thorough: 25 trained annotators, three-stage calibration, independent 550-sample cross-check with within-1 agreement >91%.
- The 10-model benchmark with per-category heatmaps provides useful practical insights (e.g., RQ is easy, IF separates models).
Critical weaknesses
-
The title claims "Video Editing AND Visual Effects," but VFX is never separately treated. "Visual Effect Editing" is merely one of nine task categories. There is no VFX-specific evaluation protocol, no VFX-specific metrics (e.g., compositing seam quality, particle physics plausibility), and no discussion of how VFX assessment differs from generic editing. The title overpromises; either the VFX component should be substantively developed or the title should be narrowed to "video editing."
-
Instruction complexity is not characterized; are these deployment-realistic or toy tasks? Instructions are generated by Gemini 3 Flash, but the paper does not analyze their complexity distribution: How many are single-step vs. multi-step? How many involve conditional logic, spatial constraints, or temporal sequencing? The task-difficulty heatmap (Table A12) shows "model performance" varies, but this conflates instruction difficulty with model capability. Without an explicit complexity taxonomy or human difficulty rating of the instructions themselves, it is unclear whether the benchmark reflects real production requirements or mostly simple single-attribute edits.
-
Gemini-3.1-Pro (zero-shot, no fine-tuning on this data) nearly matches VEFX-Reward-32B — is the difference significant? Overall SRCC: Gemini-3.1-Pro 0.752 vs. VEFX-Reward-32B 0.780, a gap of only 0.028. No significance test is reported for this comparison. On IF specifically, Gemini-3.1-Pro reaches 0.731 vs. 0.754. This raises the question: what is the marginal value of training a dedicated reward model on 4,200 samples when a strong general-purpose VLM with the same rubric prompt achieves nearly the same alignment? The paper should report bootstrap confidence intervals or paired tests, and discuss under what conditions the dedicated model is irreplaceable (e.g., cost, latency, consistency, deployability) rather than implying the gap is large.
-
Providing edited outputs — what exactly is the benefit beyond reward-model training? Table 1 positions "edited outputs" as a key differentiator. The stated benefit is enabling reward-model training, but this creates a circularity: the benchmark's value proposition depends on the reward model, which depends on the edited outputs, which come from specific models. If the reward model's advantage over a strong VLM judge is marginal, then the practical benefit of curating edited outputs + human labels (expensive) over simply using a VLM judge directly (cheap) is not clearly established. The paper should articulate the concrete deployment scenario where VEFX-Reward is necessary and a prompted VLM is insufficient.
-
Does the paper actually solve the problems it identifies in Sec. 1? The introduction identifies two bottlenecks: (a) lack of large-scale human-annotated complete editing triplets, and (b) reliance on generic VLM judges. On (a): 5,049 examples with 4,200 for training is useful but not "large-scale" by current standards. On (b): if Gemini-3.1-Pro already achieves 0.752 SRCC zero-shot, the "reliance on generic VLM judges" may not be the bottleneck the paper claims. The gap between problem framing and solution strength should be honestly discussed.
Questions for authors
please answer the questions above.
7 comments
Re: Critical Weaknesses 2
Re: Critical Weaknesses 2. Instruction complexity is not characterized; are these deployment-realistic or toy tasks? Instructions are generated by Gemini 3 Flash, but the paper does not analyze their complexity distribution: How many are single-step vs. multi-step? How many involve conditional logic, spatial constraints, or temporal sequencing? The task-difficulty heatmap (Table A12) shows "model performance" varies, but this conflates instruction difficulty with model capability. Without an explicit complexity taxonomy or human difficulty rating of the instructions themselves, it is unclear whether the benchmark reflects real production requirements or mostly simple single-attribute edits.
We thank the reviewer for highlighting the distinction between instruction complexity and task difficulty.
We agree that instruction complexity and empirical task difficulty are distinct concepts. However, the practical difficulty of an editing task inherently depends on the capabilities of the available editing models. Accordingly, Table A12 characterizes the empirical, model-relative difficulty of different editing categories through their average human-rated instruction-following (IF) scores. For example, Camera Angle Editing achieves a mean human IF score of 1.76, compared with 2.87 for Style Editing, indicating that current models struggle more with camera-angle editing.
To directly address the concern of instruction complexity, we conducted an additional instruction-level analysis using Qwen to annotate four non-mutually-exclusive dimensions: Multi-operation (multiple distinct editing operations), Conditional (explicit conditional dependencies), Spatial (spatial relationships or constraints), and Temporal (timing or sequencing constraints).
| Multi-operation (%) | Conditional (%) | Spatial (%) | Temporal (%) | At least one (%) |
|---|---|---|---|---|
| 15.5 | 8.2 | 82.4 | 55.6 | 90.5 |
Notably, 90.5% of instructions exhibit at least one of these four properties, with spatial (82.4%) and temporal (55.6%) constraints being particularly prevalent. Although most instructions involve a single primary editing operation, they frequently incorporate additional spatial or temporal requirements. These findings demonstrate that instruction complexity extends beyond the number of requested editing operations.
The annotation prompt specifies the following criteria:
Task: Given a video editing instruction, assign four independent binary labels indicating whether the instruction contains the following properties:
Multi-operation: Assign 1 if the instruction requests two or more distinct editing operations (e.g., removing an object and changing the background). Constraints that merely specify how to perform a single operation, such as preserving the original lighting, do not count as additional operations.
Conditional: Assign 1 if the requested edit depends on an explicitly stated condition (e.g., "if," "when," "only if," or "unless"). The condition must determine whether or when an edit should occur. Do not infer implicit conditions from the video content.
Spatial: Assign 1 if the instruction explicitly specifies spatial relationships, positions, orientations, or relative placements that the edit must satisfy (e.g., "move the car to the left of the tree" or "keep the subject centered"). Simple references to objects without spatial constraints do not count.
Temporal: Assign 1 if the instruction explicitly specifies timing, duration, temporal ordering, or the segment/event during which an edit should occur (e.g., "after three seconds," "before the person enters," or "during the final scene"). Ordinary action descriptions without explicit temporal requirements do not count.
Rules:
- Labels are independent and may overlap.
- Label only explicitly stated requirements; do not infer implicit constraints.
- When ambiguous, assign 0.
- Return only a valid JSON object with integer values.
Re: Critical Weaknesses 3
Re: Critical Weaknesses 3. Gemini-3.1-Pro (zero-shot, no fine-tuning on this data) nearly matches VEFX-Reward-32B — is the difference significant? Overall SRCC: Gemini-3.1-Pro 0.752 vs. VEFX-Reward-32B 0.780, a gap of only 0.028. No significance test is reported for this comparison. On IF specifically, Gemini-3.1-Pro reaches 0.731 vs. 0.754. This raises the question: what is the marginal value of training a dedicated reward model on 4,200 samples when a strong general-purpose VLM with the same rubric prompt achieves nearly the same alignment? The paper should report bootstrap confidence intervals or paired tests, and discuss under what conditions the dedicated model is irreplaceable (e.g., cost, latency, consistency, deployability) rather than implying the gap is large.
We agree that Gemini-3.1-Pro is a strong zero-shot evaluator and that the Overall SRCC improvement (+0.028) is modest. However, VEFX-Reward-32B shows a substantially larger improvement in Rendering Quality (RQ):
| Metric | Gemini-3.1-Pro [95% CI] | VEFX-Reward-32B [95% CI] | Difference |
|---|---|---|---|
| IF | 0.731 [0.702, 0.762] | 0.754 [0.735, 0.790] | +0.023 |
| RQ | 0.518 [0.499, 0.534] | 0.681 [0.658, 0.698] | +0.163 |
| EE | 0.681 [0.667, 0.701] | 0.717 [0.701, 0.735] | +0.036 |
| Overall | 0.752 [0.743, 0.765] | 0.780 [0.760, 0.802] | +0.028 |
VEFX-Reward-4B achieves a competitive Overall SRCC of 0.760, requiring only 0.45 seconds and 11.9 GB peak GPU memory per edit on an H100. At $4.39/GPU-hour, this corresponds to approximately $0.55 per 1,000 edits in active GPU time, compared with an estimated $10–14 for Gemini-3.1-Pro at standard API rates (assuming 2,000–4,000 input and 500 output tokens). These illustrative estimates exclude local serving overhead and depend on actual API token usage.
Moreover, VEFX-Reward supports local deployment with fixed weights and preprocessing, enabling reproducible evaluation without dependence on commercial API versions. Thus, its primary practical value lies in competitive human alignment, low recurring inference costs, and reproducibility, particularly for high-volume evaluation.
Re: Critical Weaknesses 1
Re: Critical Weaknesses 1: The title claims "Video Editing AND Visual Effects," but VFX is never separately treated. "Visual Effect Editing" is merely one of nine task categories. There is no VFX-specific evaluation protocol, no VFX-specific metrics (e.g., compositing seam quality, particle physics plausibility), and no discussion of how VFX assessment differs from generic editing. The title overpromises; either the VFX component should be substantively developed or the title should be narrowed to "video editing."
Thanks for the suggestion. We agree that the original title could imply a standalone VFX evaluation protocol. We have therefore revised it to “VEFX-Bench: A Benchmark for Instruction-Guided Video Editing.”
Visual Effect Editing remains one of nine task categories, covering 33 of 300 benchmark inputs, including background changes, decorations, and event effects. These edits are evaluated using the shared IF/RQ/EE rubric rather than dedicated VFX-specific metrics. We have clarified this scope in Sections 1 and 3 of the revised manuscript.