- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes using controlled interventions on generated videos to construct preference pairs with known labels, avoiding per-pair human annotation. Based on this idea, it introduces ISOVGRBENCH, an 8,000-pair benchmark spanning eight intervention families targeting motion, text alignment, visual quality, and overall quality. Experiments on nine reward models and two VLM judges show that current models can detect broad quality differences but are weak on frame-order sensitivity, with the best reward model reaching only 60.8% on temporally shuffled videos versus 100% for humans. The paper also shows that intervention-generated pairs are useful as training data: reward models trained with this supervision improve transfer on human-labeled benchmarks and yield better downstream video-generator post-training results than a human-preference-trained baseline.
Critical strengths
- The intervention-based construction isolates properties more cleanly than existing preference benchmarks, making reward-model behavior much more interpretable.
- The paper does not stop at analysis: it also shows that the same synthetic preference data can improve reward-model training and generator post-training, which materially increases the paper’s impact.
Critical weaknesses
- The central “frame-order blindness” claim is supported mainly by a highly synthetic intervention without real-world videos.
- Temporal Shuffle is a strong diagnostic probe, but it is not obvious that failing on segment-reordered videos implies comparable weakness on realistic temporal-order errors produced by current video generators. This matters because the paper’s main takeaway is framed as a practical limitation of current reward models, while the evidence is strongest for a constructed stress test.
Questions for authors
- Could the paper test the “frame-order blindness” claim on real-world videos?
- Could the paper provide empirical evidence or a quantitative analysis demonstrating that a reward model’s failure on segment-reordered videos directly correlates with its inability to detect realistic temporal-order errors produced by current state-of-the-art video generators?
2 comments
Thank you for the comments. My concerns are addressed. I keep my score unchanged.
We thank the reviewer for their positive assessment of the paper and for noting that the construction "isolates properties more cleanly than existing preference benchmarks, making reward-model behavior much more interpretable," and that the training results "materially increase the paper's impact." We ran new experiments for both weaknesses and added them to the revision (Appendix C.5).
W1 & Q1. "The central 'frame-order blindness' claim is supported mainly by a highly synthetic intervention without real-world videos."
The claim holds on real-world videos. We took real camera footage from Kinetics (500 high-motion clips and 500 random clips), cut each into 6 segments and shuffled them exactly as in the benchmark, and scored every model in the paper with the prompt "A video of <action>" (accuracy %, chance is 50; ties count as wrong, as in the paper, so models that often tie score far below 50):
No existing reward model exceeds 54% on real footage. IsoVRM-Q2, trained only on generated video, ties the strongest judge, GPT-4.1.
W2 & Q2. "It is not obvious that failing on segment-reordered videos implies comparable weakness on realistic temporal-order errors produced by current video generators."
Models that fail Temporal Shuffle also fail other temporal errors. Temporal Shuffle tests a necessary condition: a reward that cannot tell a clip from its own reordered version has no basis for penalizing order errors in generated video. Labeling real order errors would need per-item human judgment, which our construction avoids, so we tested whether the same models fail other temporal errors. For each error type, we ranked the 11 existing models (excluding ours, which were trained on shuffled pairs) and compared that ranking with their ranking on Temporal Shuffle. A Spearman ρ near 1 means the same models fail both tests:
Both have p < 0.01. By contrast, existing temporal benchmarks such as TemporalBench and TempCompass (Figure 4) ask a model to pick the right caption for a single video and never compare two orderings, and their rankings do not track Temporal Shuffle.
Reversal, an order change with no cut. We also played each clip backwards, on the 1,000 Temporal Shuffle anchors and on 1,000 random anchors (numbers below in that order). The three strongest existing models on Temporal Shuffle score at most 56.9%: VideoReward 56.6% and 55.3%, Qwen3-VL 56.9% and 53.6%, GPT-4.1 47.7% and 49.6%. IsoVRM-Q2 reaches 91.9% and 93.3% without ever training on reversed clips, so the frame-order signal is learnable and existing models miss it.
We hope these results strengthen the reviewer's confidence in the paper and would be glad to answer any further questions.