acceptodds
Withdrawn from ICLR 2027

IsoVGRBench: Video Reward Modeling Without Human Feedback

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning post-trains video generators against reward models, so a generator improves only along the properties that its reward model perceives. Video reward models are trained and evaluated on human preferences between videos that often differ simultaneously on properties such as prompt alignment, visual fidelity, motion, and generator identity. As a result, aggregate accuracy does not reveal which video properties a reward model can reliably distinguish. We construct preference pairs by controlled interventions that change one property of a generated video while holding the rest fixed. This determines both the preferred side and the property under test without per-pair human labels, requiring human validation only once per intervention to confirm perceptibility. We propose ISOVGRBENCH, a benchmark of 8,000 such pairs across eight interventions, and show that current video reward models separate broad quality differences between generators but miss frame order. The best reward model reaches only 60.8% on reordered frames against 100% for humans, and rankings on existing benchmarks do not predict this failure. Intervention pairs can also serve as training data: our ISOVRM-Q2 outperforms VideoReward, which shares its base model but trains on human preferences, by 3.8 points on VBench-2.0 and provides better supervision for video generator post-training. Adding intervention pairs to VideoReward’s human-preference supervision further improves transfer. Because intervention pairs are automatically generated rather than collected, the benchmark and training data can be regenerated as generators evolve.

Primary area:
datasets and benchmarks
Keywords:
Video Reward Modeling, Synthetic Preference, Controllable Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_he9iOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes using controlled interventions on generated videos to construct preference pairs with known labels, avoiding per-pair human annotation. Based on this idea, it introduces ISOVGRBENCH, an 8,000-pair benchmark spanning eight intervention families targeting motion, text alignment, visual quality, and overall quality. Experiments on nine reward models and two VLM judges show that current models can detect broad quality differences but are weak on frame-order sensitivity, with the best reward model reaching only 60.8% on temporally shuffled videos versus 100% for humans. The paper also shows that intervention-generated pairs are useful as training data: reward models trained with this supervision improve transfer on human-labeled benchmarks and yield better downstream video-generator post-training results than a human-preference-trained baseline.

Critical strengths
  1. The intervention-based construction isolates properties more cleanly than existing preference benchmarks, making reward-model behavior much more interpretable.
  2. The paper does not stop at analysis: it also shows that the same synthetic preference data can improve reward-model training and generator post-training, which materially increases the paper’s impact.
Critical weaknesses
  1. The central “frame-order blindness” claim is supported mainly by a highly synthetic intervention without real-world videos.
  2. Temporal Shuffle is a strong diagnostic probe, but it is not obvious that failing on segment-reordered videos implies comparable weakness on realistic temporal-order errors produced by current video generators. This matters because the paper’s main takeaway is framed as a practical limitation of current reward models, while the evidence is strongest for a constructed stress test.
Questions for authors
  1. Could the paper test the “frame-order blindness” claim on real-world videos?
  2. Could the paper provide empirical evidence or a quantitative analysis demonstrating that a reward model’s failure on segment-reordered videos directly correlates with its inability to detect realistic temporal-order errors produced by current state-of-the-art video generators?
2 comments

We thank the reviewer for their positive assessment of the paper and for noting that the construction "isolates properties more cleanly than existing preference benchmarks, making reward-model behavior much more interpretable," and that the training results "materially increase the paper's impact." We ran new experiments for both weaknesses and added them to the revision (Appendix C.5).

W1 & Q1. "The central 'frame-order blindness' claim is supported mainly by a highly synthetic intervention without real-world videos."

The claim holds on real-world videos. We took real camera footage from Kinetics (500 high-motion clips and 500 random clips), cut each into 6 segments and shuffled them exactly as in the benchmark, and scored every model in the paper with the prompt "A video of <action>" (accuracy %, chance is 50; ties count as wrong, as in the paper, so models that often tie score far below 50):

ModelHigh motionRandom
GPT-4.162.461.0
IsoVRM-Q2 (ours)62.061.0
IsoVRM-VR (ours)56.856.8
Qwen3-VL55.854.0
VideoReward54.053.4
Other reward models19.0–50.414.8–51.6

No existing reward model exceeds 54% on real footage. IsoVRM-Q2, trained only on generated video, ties the strongest judge, GPT-4.1.

W2 & Q2. "It is not obvious that failing on segment-reordered videos implies comparable weakness on realistic temporal-order errors produced by current video generators."

Models that fail Temporal Shuffle also fail other temporal errors. Temporal Shuffle tests a necessary condition: a reward that cannot tell a clip from its own reordered version has no basis for penalizing order errors in generated video. Labeling real order errors would need per-item human judgment, which our construction avoids, so we tested whether the same models fail other temporal errors. For each error type, we ranked the 11 existing models (excluding ours, which were trained on shuffled pairs) and compared that ranking with their ranking on Temporal Shuffle. A Spearman ρ near 1 means the same models fail both tests:

Temporal errorρ with Temporal Shuffle
Swapping just two neighboring segments0.85
Frame Hold, the benchmark's stutter intervention (Table 2)0.83

Both have p < 0.01. By contrast, existing temporal benchmarks such as TemporalBench and TempCompass (Figure 4) ask a model to pick the right caption for a single video and never compare two orderings, and their rankings do not track Temporal Shuffle.

Reversal, an order change with no cut. We also played each clip backwards, on the 1,000 Temporal Shuffle anchors and on 1,000 random anchors (numbers below in that order). The three strongest existing models on Temporal Shuffle score at most 56.9%: VideoReward 56.6% and 55.3%, Qwen3-VL 56.9% and 53.6%, GPT-4.1 47.7% and 49.6%. IsoVRM-Q2 reaches 91.9% and 93.3% without ever training on reversed clips, so the frame-order signal is learnable and existing models miss it.

We hope these results strengthen the reviewer's confidence in the paper and would be glad to answer any further questions.

Thank you for the comments. My concerns are addressed. I keep my score unchanged.

Reviewer_TH3fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces IsoVGRBench, a benchmark containing 8,000 video preference pairs constructed through eight controlled interventions without per-pair human labels. Evaluations of nine reward models and two vision-language judges reveal weaknesses in temporal-order understanding that existing benchmarks fail to capture. The authors further use 160,000 intervention-generated pairs to train reward models, demonstrating improvements on existing benchmarks and downstream video generation through GRPO post-training.

Critical strengths
  1. Controlled evaluation. Interventions such as Temporal Shuffle and Frame Hold preserve visual content while modifying temporal properties, enabling more targeted evaluation of reward models.

  2. The benchmark reveals poor temporal-order sensitivity across existing reward models. Increasing the frame budget does not resolve this issue.

  3. Scalable data construction. The approach generates preference pairs without per-pair human annotation, with human validation supporting the constructed labels across eight interventions.

  4. Downstream effectiveness. Models trained on intervention pairs show competitive performance on human-labeled benchmarks and improve human preferences for post-trained video generators.

Critical weaknesses
  1. Incomplete intervention isolation. Regenerating videos may alter properties beyond the target dimension, while generator substitution explicitly changes multiple properties. Confinement is only additionally validated for Motion Re-gen, leaving the isolation of other interventions less clear.

  2. Limited temporal generalization. Temporal Shuffle uses high-motion videos with six reordered segments. It is unclear whether this evaluation reflects sensitivity to subtler temporal errors in naturally generated videos.

  3. Weak transfer to temporal generation. Despite substantial improvements on Temporal Shuffle (Table 8), the post-trained generators show little improvement in Motion Order Understanding: 17.1 for VideoReward, 17.7 for IsoVRM-Q2, and 13.8 for IsoVRM-VR (Table 18).

  4. Uncertain downstream gains. The VBench-2.0 improvement is largely driven by Multi-View Consistency. Removing this dimension reverses the advantage of IsoVRM-VR over VideoReward, while the reported bootstrap interval includes zero.

Questions for authors
  1. Could the authors further validate that regenerated intervention pairs preserve non-target properties, particularly for Attribute Edit?

  2. Does Temporal Shuffle remain effective with less aggressive perturbations or different anchor-selection criteria?

  3. Why do improvements in Temporal Shuffle accuracy not translate into better Motion Order Understanding in downstream generation?

  4. Could the authors report confidence intervals for the human preference results, given the relatively low inter-annotator agreement in Appendix P?

3 comments

We thank the reviewer for recognizing the controlled evaluation, that "the benchmark reveals poor temporal-order sensitivity across existing reward models," the scalable construction, and the "downstream effectiveness" of training on intervention pairs. We add new results for each weakness. On training, a reward trained with zero human labels is competitive with VideoReward, trained on 182k human-labeled pairs (Liu et al., 2025), so on the cost–quality trade-off it outperforms VideoReward, and its training data can scale without annotation. The revised abstract makes this cost comparison explicit.

W1 & Q1. "Incomplete intervention isolation ... Confinement is only additionally validated for Motion Re-gen."

Temporal Shuffle and Frame Hold are confined by construction, since they reuse the anchor's own frames, and we add direct evidence for Attribute Edit. It re-renders with the same generator, seed and sampling settings, changing one attribute of the prompt. Raters judged motion and visual quality as ties in 44 and 43 of 45 Attribute Edit pairs (97.8% and 95.6%). Across 1,000 pairs, the edited clip has less motion (mean frame difference) in 48.6% and is less sharp in 48.3%, and VideoReward's motion- and visual-quality heads prefer the original in 51.2% and 53.0%, so non-target differences carry no consistent signal about the label. Generator substitution gives up confinement by design and carries an overall label (Section 2.2).

W2 & Q2. "Limited temporal generalization ... subtler temporal errors in naturally generated videos."

The finding holds for subtler errors, ordinary anchors and an order change with no cut. We built five new versions of Temporal Shuffle, 1,000 pairs each. Three are milder, cutting the video into 2, 3 or 4 segments instead of 6. One swaps just two neighboring segments out of 6. The last keeps 6 segments but picks anchors at random instead of the highest-motion ones. Accuracy (%):

Model2 segments34One adjacent swap of 6Random anchors
IsoVRM-VR88.687.187.569.986.7
IsoVRM-Q281.784.484.470.285.2
VideoReward56.557.958.759.161.6
GPT-4.159.860.560.554.261.4
Best other reward model53.552.052.951.052.1

Random anchors have less than half the benchmark's motion (median mean frame difference 10.2 vs 24.3). Across the 11 existing models, the ranking on the adjacent swap nearly matches the ranking on Temporal Shuffle (Spearman ρ = 0.85). Playing whole clips backwards, which reorders every frame without a cut, VideoReward and GPT-4.1 score 56.6% and 47.7% while IsoVRM-Q2 reaches 91.9% without ever training on reversal. All of these are now in Appendix C.5 of the revision.

W3 & Q3. "Weak transfer to temporal generation ... Motion Order Understanding."

On this dimension the generator trained with our zero-label reward matches VideoReward's (17.7 vs 17.1), which is the goal. All four runs, the pretrained generator's 36 of 181 included, are within chance of one another; IsoVRM-VR's 25 against VideoReward's 31 is 6 videos (p = 0.4). The dimension also often measures text alignment: in 41 of 101 sampled base-generator videos, the generator never draws one of the two actions, so there is no order to judge. Per-dimension generator metrics mix properties this way (Section 6), so they cannot show whether a reward reads frame order, which is why IsoVGRBench tests the reward directly.

W4 & Q4. "Uncertain downstream gains ... Removing this dimension reverses the advantage of IsoVRM-VR."

The downstream claim is comparable or better quality than VideoReward without any human labels, and VBench-2.0 and the human study support it. Removing Multi-View Consistency alone leaves IsoVRM-VR ahead (50.4 vs 49.9), and we have corrected the Appendix E.3 text that stated otherwise. Human preference, 95% intervals from resampling prompts (80 prompts, 3 passes):

ComparisonWin rate95% interval
IsoVRM-VR over VideoReward57.1%[50.0, 64.2]
IsoVRM-Q2 over VideoReward56.2%[48.8, 63.7]
Both of ours combined over VideoReward56.7%[50.2, 63.1]
IsoVRM-Q2 over IsoVRM-VR54.6%[47.5, 61.7]

Every estimate favors our generators over VideoReward's, consistent with matching it or better. A low κ is expected for closely matched generators; within each pass, choices are coherent (preference cycles in only 5 of 240 pass-prompt combinations, Appendix E.2).

Presentation. Section 2.2 now gives an overview of the eight interventions and their dimensions, and the appendix is reorganized from 19 sections into 5.

We hope the new results address the reviewer's concerns and would appreciate it if the reviewer would consider updating their score.

Since we are approaching the end of the discussion period, we would like to ask the reviewer to kindly review our response regarding the contribution of our paper. We would love to engage in further discussion if the reviewer has any additional questions or concerns. We really appreciate the reviewer's time and contribution in helping us improve the manuscript.

Thank you for authors' efforts on the rebuttal. After carefully reviewing the rebuttal and revisiting the paper, some of my concerns are solved and the following remaining concerns not not fully resolved:

  1. For the temporal-generation transfer, the provided explanation does not demonstrate the improved temporal shuffle accuracy translates into better temporal ordering in generated videos.
  2. The downstream improvements are modest and limited.
Reviewer_M9YwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes IsoVGRBench for video reward models. It creates 8000 preference pairs without per-pair human labels. Authors find that existing reward models perform poorly on temporal changes, especially frame shuffling. They also use 160K synthetic pairs to train reward models and show some improvements on reward benchmarks and video generation post-training.

Critical strengths
  1. The paper studies an important problem in video reward modeling. The controlled intervention idea is useful for finding the weaknesses in benchmark.
  2. The experiments cover many reward models. The results on temporal shuffle are interesting.
  3. Using synthetic preference pairs to train reward models can reduce human annotations.
Critical weaknesses
  1. The main gaol is to improve temporal understanding in video reward models. From Tab18, motion rrder understanding drops from 17.1 to 13.8. The overall vbench improvement mainly comes from multi-view consistency, rather than temporal quality. So, the reviewer has a serious concern about the effectiveness of the proposed method.
  2. The paper says that each intervention changes only one property. But, attribute edit regenarates the whole video, motion and visual quality can also change. The generator replacement also changes several properties at the same time. whether the benchmark can really test one property at a time?
  3. from Tab3, IsoVRM-Q2 performs worse than videoreward on 3/4 human-labeled benchmarks. Also, in human evaluation, the agreement between raters is low (kappa = 0.18). the reviewer has concerns about the reliability and generalization of the results.
Questions for authors

see Weaknesses part

3 comments
AuthorsOpenReview ↗

Clarification that our main contribution is the benchmark and matched performance with zero human data

We thank the reviewer for finding that the paper "studies an important problem," that the controlled intervention idea "is useful," and that the Temporal Shuffle results "are interesting."

IsoVGRBench is a benchmark, and its main finding is that existing video reward models barely notice frame order. The best of them scores 60.8% on Temporal Shuffle, where people score 100%.

The same construction removes the need for human preference labels in training. VideoReward (Liu et al., 2025) required 182k human-labeled pairs. Our 160k training pairs required none: they are built automatically, so more data costs only compute, and they can be regenerated for any new generator (Section 1). Trained only on them, IsoVRM-Q2 is competitive with VideoReward, which starts from the same base model, and as a post-training reward it scores higher on VBench-2.0 (49.8 vs 49.0) and with human raters (56.2%). This is the sense in which it outperforms VideoReward: comparable or better quality at a small fraction of the cost, since it needs no annotation at all (Section 1, contributions). The revised abstract makes the cost explicit.

1. "The main goal is to improve temporal understanding in video reward models."

Our reward models do understand frame order far better than any existing one, including on a test they never trained on. On Temporal Shuffle, IsoVRM-Q2 scores 84.9%, against 60.8% for the best existing reward model (Appendix D.2). As a held-out test, we played each benchmark clip fully backwards, which changes order with no cut to spot and was never part of training: IsoVRM-Q2 picks the forward clip 91.9% of the time, VideoReward 56.6% and GPT-4.1 47.7%; we will release the pairs and scores. The per-dimension VBench-2.0 results (Appendix E.3) measure something else, the video generator after post-training with each reward.

"From Tab18, motion order understanding drops from 17.1 to 13.8. The overall vbench improvement mainly comes from multi-view consistency."

The goal is comparable quality at zero annotation cost, and the VBench-2.0 dimensions show that. The 17.1 to 13.8 drop is 6 of 181 videos, for IsoVRM-VR on one dimension. IsoVRM-Q2, with no human labels, scores 17.7, level with VideoReward's 17.1, and all four runs, the pretrained generator's 19.9 included, are within chance of one another (two-proportion p ≥ 0.12). The metric also often turns on text alignment: in 41 of 101 sampled videos one of the two actions never appears. Generator metrics mix properties this way (Section 6); IsoVGRBench tests the reward directly. Human raters preferred both of our generators over VideoReward's (56.2% and 57.1%, Appendix E.2).

2. "Attribute edit regenerates the whole video, motion and visual quality can also change."

Motion and visual quality do not decide the label: the edit moves them in no consistent direction. Attribute Edit changes one attribute of the prompt and keeps the generator, seed and sampling settings identical. Across 1,000 pairs, the edited clip has less motion in 48.6% and is less sharp in 48.3%, and VideoReward's own motion- and visual-quality heads prefer the original in only 51.2% and 53.0%. Raters also marked motion and visual quality as ties in 44 and 43 of 45 Attribute Edit pairs (97.8% and 95.6%).

"The generator replacement also changes several properties."

Yes, and that is intended. Because a different generator "can simultaneously affect visual quality, motion and text alignment, these pairs receive an overall label and deliberately give up confinement" (Section 2.2). The revised abstract now notes this exception.

3. "IsoVRM-Q2 performs worse than videoreward on 3/4 human-labeled benchmarks."

IsoVRM-Q2 is competitive with a model trained on 182k human labels without using a single one (Table 3). IsoVRM-Q2 is within 2 points on VideoGen-RewardBench and GenAI-Bench, 3.8 ahead on VBench-2.0 and 5.2 behind on MJ-Bench-Video. Adding our pairs to VideoReward's own data (IsoVRM-VR) raises VBench-2.0 from 56.3 to 63.5 and MJ-Bench-Video from 66.4 to 69.0.

"The agreement between raters is low (kappa = 0.18)."

The benchmark itself is validated at 96.4% agreement (347 of 360 pairs, and all 64 controls judged ties; Section 2.3). The κ = 0.18 comes from a separate study, Appendix E.2, comparing three post-trained generators whose VBench-2.0 scores are within 2.5 points, so raters naturally split on individual prompts. Their choices are still coherent: A over B, B over C and C over A occurred in only 5 of 240 cases.

In short, the benchmark exposes a blind spot every existing reward model shares, and the same construction trains a reward competitive with a human-supervised one at zero annotation cost. We hope this clarifies the contribution and would appreciate it if the reviewer would update their score accordingly.

Since we are half way through the discussion period, we wanted to send a reminder to the reviewer. We would be happy to answer any additional questions or engage in further discussion.

To quickly clarify, the main contribution of IsoVRGBench is to synthetically create evaluation to diagnose current reward models, and to create training data that match human quality, rather than claiming we outperform SOTA.

We sincerely thank the reviewer for their time and hope the reviewer could consider updating their scores if all concerns are addressed. Thank you!

Since we are approaching the end of the discussion period, we would like to ask the reviewer to kindly review our response regarding the contribution of our paper. We would love to engage in further discussion if the reviewer has any additional questions or concerns. We really appreciate the reviewer's time and contribution in helping us improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.