acceptodds
Under review as a conference paper at ICLR 2027

VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects

Abstract

As AI-assisted video creation becomes increasingly practical, instruction-guided video editing has become essential for refining generated or captured footage to meet professional requirements. Yet the field still lacks both a large-scale human-annotated dataset with complete editing examples and a dedicated reward model for automated assessment. Existing resources are limited by small scale, missing edited outputs, or the absence of human quality labels, while current evaluation often relies on expensive manual inspection or generic vision-language model judges that are not specialized for editing quality. We introduce VEFX-Dataset, a human-annotated dataset containing 5,049 video editing examples spanning 9 major editing categories and 32 subcategories, each labeled along three decoupled dimensions: Instruction Following, Rendering Quality, and Edit Exclusivity. Building on this dataset, we train VEFX-Reward, the first reward model designed specifically for video editing quality assessment. VEFX-Reward jointly processes the source video, the editing instruction, and the edited video, and predicts per-dimension quality scores via ordinal regression. We further release VEFX-Bench, a benchmark of 300 curated video-prompt pairs for a holistic evaluation of video editing models. Experiments show that VEFX-Reward consistently outperforms generic VLM judges and prior reward models on both standard video quality metrics and group-wise preference evaluation, making it an effective tool for benchmarking, model selection, and future reward-driven optimization. We will open-source the dataset, benchmark, and reward model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
video editing, reward model, video quality assessment, instruction-guided video editing, human-annotated benchmark, multimodal evaluation, preference modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BeDrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VEFX-Dataset, VEFX-Reward (a Qwen3-VL-based ordinal-regression reward model), and VEFX-Bench (300 curated video-prompt pairs). It benchmarks 10 commercial/open-source video editing models and shows VEFX-Reward outperforms generic VLM judges and prior reward models.

Critical strengths

The dataset and annotation protocol are solid contributions.

  • The three-dimensional decoupled evaluation (IF / RQ / EE) is well motivated and supported by weak inter-dimension correlations; the (IF=1, RQ=4, EE=4) pattern concretely shows why a single scalar is insufficient.
  • The annotation protocol is thorough: 25 trained annotators, three-stage calibration, independent 550-sample cross-check with within-1 agreement >91%.
  • The 10-model benchmark with per-category heatmaps provides useful practical insights (e.g., RQ is easy, IF separates models).
Critical weaknesses
  • The title claims "Video Editing AND Visual Effects," but VFX is never separately treated. "Visual Effect Editing" is merely one of nine task categories. There is no VFX-specific evaluation protocol, no VFX-specific metrics (e.g., compositing seam quality, particle physics plausibility), and no discussion of how VFX assessment differs from generic editing. The title overpromises; either the VFX component should be substantively developed or the title should be narrowed to "video editing."

  • Instruction complexity is not characterized; are these deployment-realistic or toy tasks? Instructions are generated by Gemini 3 Flash, but the paper does not analyze their complexity distribution: How many are single-step vs. multi-step? How many involve conditional logic, spatial constraints, or temporal sequencing? The task-difficulty heatmap (Table A12) shows "model performance" varies, but this conflates instruction difficulty with model capability. Without an explicit complexity taxonomy or human difficulty rating of the instructions themselves, it is unclear whether the benchmark reflects real production requirements or mostly simple single-attribute edits.

  • Gemini-3.1-Pro (zero-shot, no fine-tuning on this data) nearly matches VEFX-Reward-32B — is the difference significant? Overall SRCC: Gemini-3.1-Pro 0.752 vs. VEFX-Reward-32B 0.780, a gap of only 0.028. No significance test is reported for this comparison. On IF specifically, Gemini-3.1-Pro reaches 0.731 vs. 0.754. This raises the question: what is the marginal value of training a dedicated reward model on 4,200 samples when a strong general-purpose VLM with the same rubric prompt achieves nearly the same alignment? The paper should report bootstrap confidence intervals or paired tests, and discuss under what conditions the dedicated model is irreplaceable (e.g., cost, latency, consistency, deployability) rather than implying the gap is large.

  • Providing edited outputs — what exactly is the benefit beyond reward-model training? Table 1 positions "edited outputs" as a key differentiator. The stated benefit is enabling reward-model training, but this creates a circularity: the benchmark's value proposition depends on the reward model, which depends on the edited outputs, which come from specific models. If the reward model's advantage over a strong VLM judge is marginal, then the practical benefit of curating edited outputs + human labels (expensive) over simply using a VLM judge directly (cheap) is not clearly established. The paper should articulate the concrete deployment scenario where VEFX-Reward is necessary and a prompted VLM is insufficient.

  • Does the paper actually solve the problems it identifies in Sec. 1? The introduction identifies two bottlenecks: (a) lack of large-scale human-annotated complete editing triplets, and (b) reliance on generic VLM judges. On (a): 5,049 examples with 4,200 for training is useful but not "large-scale" by current standards. On (b): if Gemini-3.1-Pro already achieves 0.752 SRCC zero-shot, the "reliance on generic VLM judges" may not be the bottleneck the paper claims. The gap between problem framing and solution strength should be honestly discussed.

Questions for authors

please answer the questions above.

7 comments
AuthorsOpenReview ↗

Re: Critical Weaknesses 1

Re: Critical Weaknesses 1: The title claims "Video Editing AND Visual Effects," but VFX is never separately treated. "Visual Effect Editing" is merely one of nine task categories. There is no VFX-specific evaluation protocol, no VFX-specific metrics (e.g., compositing seam quality, particle physics plausibility), and no discussion of how VFX assessment differs from generic editing. The title overpromises; either the VFX component should be substantively developed or the title should be narrowed to "video editing."

Thanks for the suggestion. We agree that the original title could imply a standalone VFX evaluation protocol. We have therefore revised it to “VEFX-Bench: A Benchmark for Instruction-Guided Video Editing.”

Visual Effect Editing remains one of nine task categories, covering 33 of 300 benchmark inputs, including background changes, decorations, and event effects. These edits are evaluated using the shared IF/RQ/EE rubric rather than dedicated VFX-specific metrics. We have clarified this scope in Sections 1 and 3 of the revised manuscript.

AuthorsOpenReview ↗

Re: Critical Weaknesses 2

Re: Critical Weaknesses 2. Instruction complexity is not characterized; are these deployment-realistic or toy tasks? Instructions are generated by Gemini 3 Flash, but the paper does not analyze their complexity distribution: How many are single-step vs. multi-step? How many involve conditional logic, spatial constraints, or temporal sequencing? The task-difficulty heatmap (Table A12) shows "model performance" varies, but this conflates instruction difficulty with model capability. Without an explicit complexity taxonomy or human difficulty rating of the instructions themselves, it is unclear whether the benchmark reflects real production requirements or mostly simple single-attribute edits.

We thank the reviewer for highlighting the distinction between instruction complexity and task difficulty.

We agree that instruction complexity and empirical task difficulty are distinct concepts. However, the practical difficulty of an editing task inherently depends on the capabilities of the available editing models. Accordingly, Table A12 characterizes the empirical, model-relative difficulty of different editing categories through their average human-rated instruction-following (IF) scores. For example, Camera Angle Editing achieves a mean human IF score of 1.76, compared with 2.87 for Style Editing, indicating that current models struggle more with camera-angle editing.

To directly address the concern of instruction complexity, we conducted an additional instruction-level analysis using Qwen to annotate four non-mutually-exclusive dimensions: Multi-operation (multiple distinct editing operations), Conditional (explicit conditional dependencies), Spatial (spatial relationships or constraints), and Temporal (timing or sequencing constraints).

Multi-operation (%)Conditional (%)Spatial (%)Temporal (%)At least one (%)
15.58.282.455.690.5

Notably, 90.5% of instructions exhibit at least one of these four properties, with spatial (82.4%) and temporal (55.6%) constraints being particularly prevalent. Although most instructions involve a single primary editing operation, they frequently incorporate additional spatial or temporal requirements. These findings demonstrate that instruction complexity extends beyond the number of requested editing operations.

The annotation prompt specifies the following criteria:

Task: Given a video editing instruction, assign four independent binary labels indicating whether the instruction contains the following properties:

  1. Multi-operation: Assign 1 if the instruction requests two or more distinct editing operations (e.g., removing an object and changing the background). Constraints that merely specify how to perform a single operation, such as preserving the original lighting, do not count as additional operations.

  2. Conditional: Assign 1 if the requested edit depends on an explicitly stated condition (e.g., "if," "when," "only if," or "unless"). The condition must determine whether or when an edit should occur. Do not infer implicit conditions from the video content.

  3. Spatial: Assign 1 if the instruction explicitly specifies spatial relationships, positions, orientations, or relative placements that the edit must satisfy (e.g., "move the car to the left of the tree" or "keep the subject centered"). Simple references to objects without spatial constraints do not count.

  4. Temporal: Assign 1 if the instruction explicitly specifies timing, duration, temporal ordering, or the segment/event during which an edit should occur (e.g., "after three seconds," "before the person enters," or "during the final scene"). Ordinary action descriptions without explicit temporal requirements do not count.

Rules:

  • Labels are independent and may overlap.
  • Label only explicitly stated requirements; do not infer implicit constraints.
  • When ambiguous, assign 0.
  • Return only a valid JSON object with integer values.
AuthorsOpenReview ↗

Re: Critical Weaknesses 3

Re: Critical Weaknesses 3. Gemini-3.1-Pro (zero-shot, no fine-tuning on this data) nearly matches VEFX-Reward-32B — is the difference significant? Overall SRCC: Gemini-3.1-Pro 0.752 vs. VEFX-Reward-32B 0.780, a gap of only 0.028. No significance test is reported for this comparison. On IF specifically, Gemini-3.1-Pro reaches 0.731 vs. 0.754. This raises the question: what is the marginal value of training a dedicated reward model on 4,200 samples when a strong general-purpose VLM with the same rubric prompt achieves nearly the same alignment? The paper should report bootstrap confidence intervals or paired tests, and discuss under what conditions the dedicated model is irreplaceable (e.g., cost, latency, consistency, deployability) rather than implying the gap is large.

We agree that Gemini-3.1-Pro is a strong zero-shot evaluator and that the Overall SRCC improvement (+0.028) is modest. However, VEFX-Reward-32B shows a substantially larger improvement in Rendering Quality (RQ):

MetricGemini-3.1-Pro [95% CI]VEFX-Reward-32B [95% CI]Difference
IF0.731 [0.702, 0.762]0.754 [0.735, 0.790]+0.023
RQ0.518 [0.499, 0.534]0.681 [0.658, 0.698]+0.163
EE0.681 [0.667, 0.701]0.717 [0.701, 0.735]+0.036
Overall0.752 [0.743, 0.765]0.780 [0.760, 0.802]+0.028

VEFX-Reward-4B achieves a competitive Overall SRCC of 0.760, requiring only 0.45 seconds and 11.9 GB peak GPU memory per edit on an H100. At $4.39/GPU-hour, this corresponds to approximately $0.55 per 1,000 edits in active GPU time, compared with an estimated $10–14 for Gemini-3.1-Pro at standard API rates (assuming 2,000–4,000 input and 500 output tokens). These illustrative estimates exclude local serving overhead and depend on actual API token usage.

Moreover, VEFX-Reward supports local deployment with fixed weights and preprocessing, enabling reproducible evaluation without dependence on commercial API versions. Thus, its primary practical value lies in competitive human alignment, low recurring inference costs, and reproducibility, particularly for high-volume evaluation.

Reviewer_wduNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VEFX-Dataset, with 5,049 human-scored video editing examples, and VEFX-Reward, a Qwen3-VL-based model that predicts instruction following, rendering quality, and edit exclusivity. It also presents a 300-case benchmark and evaluates ten video editing systems.

Critical strengths
  • The dataset provides editing triplets with separate human scores for three aspects of editing quality. The annotation protocol includes multiple raters and an independent consistency check.

  • The experiments include a same-backbone, same-data SFT baseline, evaluation on unseen editing systems, and human validation of the overall score. These provide useful evidence for the evaluator’s reliability.

Critical weaknesses
  • The reported metrics are promising, but it is difficult to judge the actual video quality and whether the scores are reasonable from static examples alone. Figures A1–A7 illustrate human annotations without showing the corresponding model predictions. Full-video examples with both predicted and human scores, including cases where they disagree, would help assess the evaluator’s strengths and failures.

  • The usefulness of VEFX-Reward for improving video editing models remains unverified. The paper reports agreement with human scores, but no post training experiment with the trained reward model. Such an experiment would help establish its practical value for model optimization.

Questions for authors
  • Are the full evaluation videos available in the supplementary material? A webpage showing the source and edited videos alongside model scores and available human scores would make the results easier to inspect.
6 comments

We thank the reviewer for these constructive suggestions.

We provide an anonymous website (https://vefxreward.github.io/) featuring representative full-video examples with human annotations and VEFX-Reward predictions, including cases where model predictions disagree with human judgments.

We agree that downstream validation would strengthen the paper. We are currently conducting RL training with VEFX-Reward, Best-of-N selection, and user studies to assess its effectiveness in improving video editing quality.

We expect preliminary results within the next 24 hours and will share the findings as soon as they become available.

AuthorsOpenReview ↗

Re: Critical Weakness 2

Re: Critical Weakness 2: "The usefulness of VEFX-Reward for improving video editing models remains unverified. The paper reports agreement with human scores, but no post training experiment with the trained reward model. Such an experiment would help establish its practical value for model optimization."

To answer this question thoroughly, we conducted two experiments: (1) Best-of-K selection with human evaluation, and (2) using VEFX-Reward to train a better video editor.

(1) Best-of-K selection with human evaluation

Setup. For each source video and editing instruction, we sample eight candidate edits from Kiwi-Edit and score them with VEFX-Reward-32B. For each dimension (IF, RQ, and EE), we select the highest- and lowest-scoring edits according to that dimension. Human raters compare the selected pair on the same dimension and indicate which edit is better or whether the two are tied. We collect 40 judgments per dimension.

Results. Across the three dimensions, raters preferred the top-ranked edit in 86 of 112 non-tied judgments (76.8%). They preferred the bottom-ranked edit in 26 judgments and reported eight ties. The table reports the top-ranked edit's preference rate among non-tied judgments.

DimensionJudgmentsTop-ranked preferredTieBottom-ranked preferredTop-ranked preference (excluding ties)
IF40304683.3%
RQ402801270.0%
EE40284877.8%
Pooled1208682676.8%

These results show that selecting the top-ranked edit from eight Kiwi-Edit samples usually yields an output that humans prefer to the bottom-ranked edit on the selected dimension.

AuthorsOpenReview ↗

Re: Critical Weakness 2 (Continue)

(2) Using VEFX-Reward to train a better video editor

Complete results and visualizations are available at https://vefxreward.github.io/experiments/kiwi_rl.html.

Setup.

  • Editor. Kiwi-Edit 5B an open-source video editor.
  • Training data. 512 (source video, instruction) pairs from the VEFX-Dataset training split, with sources disjoint from the benchmark. For each pair, the base editor samples K=8 candidate edits.
  • Reward-LoRA. Following reward-ranked fine-tuning (RAFT), the frozen VEFX-Reward-4B scores all candidates (GeoAgg of IF/RQ/EE), and the top-1 candidate is used as the fine-tuning target. Training uses the editor's own flow-matching objective, with LoRA (rank 64).
  • Control (Random-LoRA). It is trained on a randomly chosen candidate from the same pool, with everything else identical: initialization, data order, timesteps, noise, and hyperparameters. The two models therefore differ only in which sample is used as the target.
  • Primary comparison. Reward-LoRA vs. Random-LoRA, specified in advance because it isolates the effect of the reward signal. The comparison with the untouched base editor is reported for reference.
  • Hyperparameters. The learning rate and checkpoint were selected on a separate 32-input dev split, using a rule that treats both models identically. The selected setting is lr 5e-5, 500 steps, and batch size 8.
  • Evaluation. 100 held-out VEFX-Bench prompts covering all 9 categories, each edited with 4 random seeds (one sample per seed). Neither evaluator is used during training:
    • VEFX-Reward-32B;
    • an independent zero-shot judge (Qwen3-VL-32B-Instruct)

Results (GeoAgg, 1–4, higher is better; 95% bootstrap CIs over prompts):

EvaluatorBaseRandom-LoRAReward-LoRAReward − Random [95% CI]Reward − Base [95% CI]Win rate vs. Random
VEFX-Reward-32B2.4062.3442.440+0.096 [+0.066, +0.128]+0.034 [−0.003, +0.075]69.8%
Zero-shot Qwen3-VL-32B judge2.5192.4612.579+0.118 [+0.074, +0.166]+0.060 [−0.008, +0.133]68.8%

Reward − Random is the primary, pre-specified test of whether VEFX-Reward is a useful training signal. It is significant under both evaluators (p < 10⁻⁵). Reward − Base is shown for reference only.

Per-dimension gains of Reward-LoRA:

IFRQEE
vs. Random-LoRA+0.05+0.15+0.19
vs. Base−0.04 (n.s.)+0.11+0.28

Analysis.

  1. The reward's ranking transfers into the model.
    • Reward-LoRA significantly outperforms the matched Random-LoRA control under both evaluators, by about +0.1 GeoAgg with a paired win rate of about 70%.
    • The two models differ only in the targets selected by VEFX-Reward, so the improvement can be attributed to the reward signal.
    • On the dev split, Reward-LoRA outperforms Random-LoRA at every checkpoint (12/12), and the gap widens as training proceeds.
  2. For reference, Reward-LoRA also improves over the base editor. Fine-tuning on randomly chosen self-generated samples degrades the editor (−0.06). Reward-based selection more than compensates for this drift: Reward-LoRA scores +0.03 / +0.06 above Base (p ≈ 0.09; the 95% CIs include zero).
  3. The gains concentrate on render quality and edit exclusivity (+0.11 RQ and +0.28 EE vs. Base), the dimensions on which the base model is weakest. Instruction following shows no significant change.

Conclusion. With VEFX-Reward as the only reward signal, even a single round of offline reward-ranked fine-tuning yields a significantly better editor than an identically trained control. This holds under evaluators that were never used for training. VEFX-Reward is thus useful not only for evaluation but also as a training signal for video editing models.

Limitations.

  • Scope. Due to the limited rebuttal period, we fine-tuned only one editor (Kiwi-Edit) and performed only a light hyperparameter search (two learning rates on a small dev split).
  • Evaluation difficulty. Evaluating generated videos is notoriously difficult, and no automatic metric fully captures edit quality. We therefore rely on two complementary evaluators: VEFX-Reward-32B and an independent zero-shot VLM judge. We also provide an interactive web report (link) showing the Source / Base / Random-LoRA / Reward-LoRA test outputs side by side, together with the training curves, so that readers can inspect the results qualitatively.
  • Single reward signal. VEFX-Reward was the only reward signal, and it already yields clear improvements on its own. Prior work generally finds that combining multiple complementary rewards (e.g., aesthetic, motion-quality, and text-alignment rewards) outperforms any single signal. We therefore recommend that future work use VEFX-Reward together with additional reward signals.
Reviewer_ytpZOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a dataset, a learned evaluator, and a benchmark for assessing generic video editing and visual effects. VEFX-Dataset contains 5,049 source-video, instruction, and edited-video examples with human ratings for Instruction Following, Rendering Quality, and Edit Exclusivity. These dimensions distinguish successful execution of the requested change from visual quality and preservation of content that should remain unchanged. VEFX-Reward builds on Qwen3-VL at 4B and 32B scales, jointly processes the source and edited videos with the instruction, and uses three reward tokens with a shared ordinal prediction head. VEFX-Bench then provides 300 fixed video-instruction cases spanning nine major categories and 32 subcategories for comparing editing systems. Experiments report stronger agreement with human ratings than the evaluated generic judges and prior reward models, including gains over a textual supervised-fine-tuning baseline using the same backbone and training data. Additional studies examine unseen editors and validate a weighted geometric aggregation rule against holistic human judgments. The contribution is potentially useful for video editing evaluation, but ambiguities in the ordinal objective, data splits, and ranking protocol limit the reproducibility and interpretation of its results.

Critical strengths
  1. Separating instruction following, rendering quality, and edit exclusivity captures meaningful failure modes that a single aesthetic or perceptual score can conceal. An edit may look convincing while failing to execute the instruction or changing unrelated content. The proposed evaluation therefore addresses a concrete need in video editing, and the combination of labeled outputs, a reusable evaluator, and fixed benchmark inputs offers a practical basis for subsequent comparisons.

  2. The dataset construction includes substantive efforts to improve annotation quality. The paper provides scoring rubrics, uses at least two independent ratings per example, revisits large disagreements, and reports a separate 550-example cross-check with six annotators. Splitting by source video and prompt group also addresses a common source of overly optimistic evaluation. These details strengthen the dataset contribution and make its reported human agreement more informative than an unspecified annotation pipeline.

  3. The experiments include relevant controls beyond comparisons with prompted judges. The same-data Qwen3-VL-4B textual SFT baseline reaches an overall SRCC of 0.735, compared with 0.760 for VEFX-Reward, supporting a measurable contribution from the proposed prediction formulation. The unseen-editor study and the independently collected holistic judgments for GeoAgg also provide useful evidence of transfer and aggregation validity, although their scale remains limited.

Critical weaknesses
  1. The probability semantics of the ordinal objective are unclear. Equation 1 trains each threshold only on examples meeting the preceding score condition, which suggests conditional probabilities, while inference sums probabilities of exceeding each threshold to compute the expected score. Conditional probabilities require conversion to cumulative probabilities before this sum is valid. The paper may perform that conversion, but does not explain it. Averaging annotator ratings also produces fractional targets whose treatment under the integer-threshold objective is unspecified. The authors should provide explicit loss and inference pseudocode, including target encoding, before the formulation and reported scores can be independently reproduced.

  2. The validation protocol is insufficiently specified. The paper defines 4,200 training examples and 849 test examples, yet selects checkpoints and several design choices using validation performance without clearly identifying the validation split. Source-video and prompt-group isolation is explicitly described for train versus test, but should also cover validation. The relationship of the 300 benchmark inputs and the two pre-frozen 100-example studies to these groups should be documented. A split manifest and sample counts would resolve this uncertainty; the current description does not establish test-set tuning, but leaves the independence of model selection and final evaluation difficult to verify.

  3. The pairwise accuracy definition inflates its absolute interpretation by assigning full credit whenever human scores tie, irrespective of the predicted ordering. With a human tie fraction of t, even a constant predictor obtains an expected score of 0.5 + 0.5t. Consequently, the reported 0.8723 does not directly measure accuracy on informative, non-tied preferences. This constant contribution does not itself invalidate relative comparisons on the same pairs. However, the paper should report tie rates, accuracy on non-tied pairs, and group-level confidence intervals. Macro-averaging across groups would also reveal whether large groups dominate the current pair-weighted result (Section 5.3).

Questions for authors

None

5 comments
AuthorsOpenReview ↗

Re: Critical Weaknesses 1

Re: Critical Weaknesses 1. The probability semantics of the ordinal objective are unclear. Equation 1 trains each threshold only on examples meeting the preceding score condition, which suggests conditional probabilities, while inference sums probabilities of exceeding each threshold to compute the expected score. Conditional probabilities require conversion to cumulative probabilities before this sum is valid. The paper may perform that conversion, but does not explain it. Averaging annotator ratings also produces fractional targets whose treatment under the integer-threshold objective is unspecified. The authors should provide explicit loss and inference pseudocode, including target encoding, before the formulation and reported scores can be independently reproduced.

Thank you for identifying two details missing from Sec. 4. Equation 1 uses a conditional binary task at each threshold. The expected score at inference uses the marginal exceedance probabilities recovered from those conditional outputs.

For each dimension, let Y∈{1,2,3,4}Y \in \{1,2,3,4\} be the ordinal rating and let xx contain the source video, instruction, and edited video. The sigmoid output qk=σ(zk)q_k = \sigma(z_k) estimates the conditional probability P(Y>k∣Y≥k,x)P(Y>k \mid Y\ge k, x) for k∈{1,2,3}k \in \{1,2,3\}. For example, the second output estimates P(Y>2∣Y>1,x)P(Y>2 \mid Y>1, x). Because the events are nested, we recover each marginal exceedance probability by multiplying the conditional probabilities up to that threshold. The estimated marginals and the resulting expected rating are given by

\qquad \hat{s} = 1 + \sum_{k=1}^{3} \widehat{P}(Y>k \mid x) = 1 + q_1 + q_1q_2 + q_1q_2q_3.$$ The expected rating is a continuous score in $[1,4]$ for each of IF, RQ, and EE. The supplementary implementation takes cumulative products, recovers the four class probabilities from their differences, and computes the same expectation. As described in Sec. 3.2, we average the valid annotator ratings for each dimension. To construct the ordinal training label, we round that mean to the nearest integer in $\{1,2,3,4\}$. For an example with rounded label $y$, the eligibility mask and binary target in Eq. 1 are $$m_k = \mathbf{1}[y \ge k], \qquad t_k = \mathbf{1}[y > k].$$ The binary cross-entropy for threshold $k$ includes only examples with $m_k=1$. We revised Sec. 4 to distinguish the integer training label from the continuous prediction and to state the training and inference equations explicitly. In the loss-function ablation in Appendix D, Table A4, ORD achieves an average validation error of 0.709, compared with 0.741 for direct continuous $L_2$ regression (REG). This comparison supports the choice of the ordinal formulation.
AuthorsOpenReview ↗

Re: Critical Strengths 2

Re: Critical Strengths 2. The validation protocol is insufficiently specified. The paper defines 4,200 training examples and 849 test examples, yet selects checkpoints and several design choices using validation performance without clearly identifying the validation split. Source-video and prompt-group isolation is explicitly described for train versus test, but should also cover validation. The relationship of the 300 benchmark inputs and the two pre-frozen 100-example studies to these groups should be documented. A split manifest and sample counts would resolve this uncertainty; the current description does not establish test-set tuning, but leaves the independence of model selection and final evaluation difficult to verify.

The 4,200 examples reported in the original submission comprised both the training and validation subsets. We used 3,738 examples to update the weights of VEFX-Reward-4B and VEFX-Reward-32B and 462 examples to select their checkpoints by validation loss. The separate 849-example test split was used for final evaluation and did not contribute to weight updates or checkpoint selection. We apologize for omitting this breakdown from the original manuscript. The revised manuscript now reports all three splits explicitly in Secs. 3–5 and Appendix B.

AuthorsOpenReview ↗

Re: Critical Strengths 3

Re: Critical Strengths 3. The pairwise accuracy definition inflates its absolute interpretation by assigning full credit whenever human scores tie, irrespective of the predicted ordering. With a human tie fraction of t, even a constant predictor obtains an expected score of 0.5 + 0.5t. Consequently, the reported 0.8723 does not directly measure accuracy on informative, non-tied preferences. This constant contribution does not itself invalidate relative comparisons on the same pairs. However, the paper should report tie rates, accuracy on non-tied pairs, and group-level confidence intervals. Macro-averaging across groups would also reveal whether large groups dominate the current pair-weighted result (Section 5.3).

Thank you for pointing out how the tie rule affects the interpretation of Table 4. In the original protocol, a pair with equal human scores receives full credit regardless of the predicted order. If the human tie rate is tt, the original pair-weighted accuracy is t+(1−t) PairAccnon-tied,microt+(1-t)\,\mathrm{PairAcc}_{\mathrm{non\text{-}tied,micro}}. A constant predictor scores t+0.5(1−t)t+0.5(1-t) under this rule. The 32B model's original Overall PairAcc of 0.8723 therefore combines the full credit on human ties with its performance on pairs where the human scores differ.

We recomputed PairAcc after excluding human-tied pairs within each source-video and instruction group. Overall uses the arithmetic mean of IF, RQ, and EE for each edit before pairwise comparison. Human ties account for the following shares of candidate pairs.

ScoreIFRQEEOverall
Human tie rate32.73%38.13%32.54%13.67%

For the remaining pairs, a correct predicted order receives 1, a predicted tie receives 0.5, and a reversed order receives 0. Micro gives equal weight to each remaining pair. Macro first computes the mean pair score within each group and then weights each group with at least one remaining pair equally. The brackets show 95% confidence intervals from a cluster bootstrap with 10,000 resamples of the ranking groups. Each resampled group carries all its comparisons.

ModelScoreMicro [95% CI]Macro [95% CI]
EditRewardIF0.7447 [0.7107, 0.7773]0.7266 [0.6851, 0.7661]
EditRewardRQ0.2936 [0.2569, 0.3311]0.2920 [0.2489, 0.3374]
EditRewardEE0.3058 [0.2739, 0.3390]0.3150 [0.2752, 0.3567]
EditRewardOverall0.7589 [0.7313, 0.7845]0.7336 [0.6968, 0.7681]
VE-BenchIF0.6062 [0.5677, 0.6437]0.5685 [0.5215, 0.6160]
VE-BenchRQ0.6972 [0.6535, 0.7396]0.7007 [0.6544, 0.7453]
VE-BenchEE0.5764 [0.5355, 0.6164]0.5772 [0.5308, 0.6224]
VE-BenchOverall0.6121 [0.5785, 0.6444]0.5800 [0.5378, 0.6212]
VEFX-Reward-4BIF0.8692 [0.8441, 0.8934]0.8803 [0.8535, 0.9053]
VEFX-Reward-4BRQ0.8884 [0.8649, 0.9102]0.8898 [0.8597, 0.9178]
VEFX-Reward-4BEE0.8766 [0.8511, 0.8990]0.8699 [0.8444, 0.8939]
VEFX-Reward-4BOverall0.8411 [0.8166, 0.8643]0.8488 [0.8196, 0.8757]
VEFX-Reward-32BIF0.9058 [0.8863, 0.9244]0.8978 [0.8696, 0.9233]
VEFX-Reward-32BRQ0.8563 [0.8284, 0.8831]0.8623 [0.8301, 0.8923]
VEFX-Reward-32BEE0.8808 [0.8581, 0.9024]0.8859 [0.8580, 0.9110]
VEFX-Reward-32BOverall0.8521 [0.8327, 0.8706]0.8554 [0.8302, 0.8791]

Using the 13.67% Overall tie rate, the 32B micro estimate of 0.8521 reconstructs the original 0.8723 to the reported precision. Both VEFX-Reward variants have higher Overall point estimates than the two baselines under both weightings. Equal weighting across groups gives the 32B model an Overall estimate of 0.8554 and preserves the Overall ordering of the four models.

We have added the tie rates, scoring and bootstrap procedures, and the full micro and macro results to Appendix J.4, Table A14, of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.