- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper first introduces SyncReward-Data-80K, a dataset containing 80K professional synchronization ratings for clips from eight audio–visual generators. A taxonomy-guided construction pipeline covers four sound-event families: Speech, Onset, Instrument, and Ambient. It further constructs SyncReward-Bench, a held-out benchmark of 1K clips, each independently rated by multiple professionals. Using this benchmark, it systematically evaluates specialized synchronization metrics and omni-modal models, revealing their substantial misalignment with human judgments. Finally, it develops SyncReward, a reward model that combines semantic–temporal contrastive pretraining on real clips with human supervision on generated clips. As an evaluation metric, SyncReward achieves the strongest agreement with human ratings among the evaluated methods. As a reward, it improves synchronization through inference-time best-of-N selection and reinforcement learning based post-training.
Critical strengths
- The paper introduces SyncReward-Data-80K, with 80K professional annotations spanning eight audio-visual generators and four sound-event families.
- It builds SyncReward-Bench and systematically benchmarks specialized synchronization metrics, off-the-shelf omni-modal judges, and fine-tuned models.
- It develops SyncReward, which aligns with human judgments and scales to inference-time selection and generative post-training.
Critical weaknesses
- The model is optimized directly toward SyncReward's own score SyncR., so an increase in its own score is to be expected. The effectiveness should be further verified through completed human evaluation.
- Best-of-16 selection and DiffusionNFT post-training are tested only on LTX-2.3. Only one generation model was tried, without verifying whether the reward works on other generators.
Questions for authors
- Has the human evaluation been finished? What are the results, and do they still support the paper's conclusions?
- The inter-annotator agreement is quite low, with Fleiss' kappa reported as only 0.23 in the appendix. Did you filter the human scores? If not, does that mean the annotations do not reliably support your conclusions?