acceptodds
Withdrawn from ICLR 2027

SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent years have witnessed rapid progress in joint audio–visual generation. However, audio–visual synchronization remains a challenge, as generated speech and sound often temporally misalign with visual events. A key bottleneck is the lack of human-annotated data, reliable evaluation benchmarks, and synchronization-aware evaluation models for generated audio–visual outputs. To address these limitations, we conduct a comprehensive study spanning data, benchmarking, and reward modeling. We first introduce SyncReward-Data-80K, a dataset containing 80K professional synchronization ratings for clips from eight audio–visual generators. A taxonomy-guided construction pipeline covers four sound-event families: Speech, Onset, Instrument, and Ambient. We further construct SyncReward-Bench, a held-out benchmark of 1K clips, each independently rated by multiple professionals. Using this benchmark, we systematically evaluate specialized synchronization metrics and omni-models, revealing their substantial misalignment with human judgments. Finally, we develop SyncReward, a reward model with semantic–temporal mixed contrastive Learning on real clips and align with human supervision on generated clips. As an evaluation metric, SyncReward achieves the strongest agreement with human ratings among the evaluated methods. As a reward, it improves synchronization through inference-time best-of- selection and reinforcement learning based post-training. We will release the data, benchmark, model, and code. Additional demos are available at https://anonymous.4open.science/r/SyncReward.

Primary area:
datasets and benchmarks
Keywords:
audio-visual generation, reward model, synchronizaiton, human aligned dataset

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M1rEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper first introduces SyncReward-Data-80K, a dataset containing 80K professional synchronization ratings for clips from eight audio–visual generators. A taxonomy-guided construction pipeline covers four sound-event families: Speech, Onset, Instrument, and Ambient. It further constructs SyncReward-Bench, a held-out benchmark of 1K clips, each independently rated by multiple professionals. Using this benchmark, it systematically evaluates specialized synchronization metrics and omni-modal models, revealing their substantial misalignment with human judgments. Finally, it develops SyncReward, a reward model that combines semantic–temporal contrastive pretraining on real clips with human supervision on generated clips. As an evaluation metric, SyncReward achieves the strongest agreement with human ratings among the evaluated methods. As a reward, it improves synchronization through inference-time best-of-N selection and reinforcement learning based post-training.

Critical strengths
  • The paper introduces SyncReward-Data-80K, with 80K professional annotations spanning eight audio-visual generators and four sound-event families.
  • It builds SyncReward-Bench and systematically benchmarks specialized synchronization metrics, off-the-shelf omni-modal judges, and fine-tuned models.
  • It develops SyncReward, which aligns with human judgments and scales to inference-time selection and generative post-training.
Critical weaknesses
  1. The model is optimized directly toward SyncReward's own score SyncR., so an increase in its own score is to be expected. The effectiveness should be further verified through completed human evaluation.
  2. Best-of-16 selection and DiffusionNFT post-training are tested only on LTX-2.3. Only one generation model was tried, without verifying whether the reward works on other generators.
Questions for authors
  1. Has the human evaluation been finished? What are the results, and do they still support the paper's conclusions?
  2. The inter-annotator agreement is quite low, with Fleiss' kappa reported as only 0.23 in the appendix. Did you filter the human scores? If not, does that mean the annotations do not reliably support your conclusions?
Reviewer_1k8fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SyncReward, a human-aligned reward model for evaluating and improving synchronization in generated audio-visual content. It presents SyncReward-Data-80K, comprising human-rated clips from eight generators across four sound-event families, and SyncReward-Bench, an independently annotated benchmark of 1,000 clips. The model combines semantic-temporal contrastive pretraining on real audio-visual data with human-supervised regression and ranking on generated clips.

Critical strengths

The paper targets synchronization errors in generated audio-visual content, where metrics developed for real videos may not adequately reflect human judgments. The dataset spans eight generators, four sound-event families, and 58 fine-grained categories. The use of five independent annotators, blinded generator identities, and a separately annotated benchmark represents a substantial data-collection effort and provides a useful foundation for studying synchronization beyond speech.

Critical weaknesses

Section 3.1 specifies that training and validation do not share (uid, model) pairs, but this does not necessarily prevent the same source prompt from appearing across splits through different generators. Similarly, the benchmark is described as strictly disjoint without explicitly establishing prompt-level or source-caption-level separation. This leaves potential content overlap unresolved. Prompt-disjoint evaluation and a held-out-generator experiment would make the generalization claims substantially more convincing.

Table 2 states that the reported preference scores are provisional estimates from base-referenced comparisons and will be replaced by a complete all-pairs evaluation. Consequently, the current evidence does not establish the claimed ranking among reward methods as strongly as a completed, directly comparable preference study would. The main text also do not report the evaluation sample sizes, confidence intervals, or variability across training runs. These omissions make it difficult to assess the statistical reliability of the reported gains.

Reward optimization can improve synchronization by sacrificing requested content. Figure 5 explicitly shows an example where post-training removes an instrument specified by the prompt while increasing the synchronization score. This is a substantive limitation: a generator may improve the reward by simplifying the scene or eliminating difficult sound sources instead of synchronizing the intended content. Although the paper acknowledges this trade-off, Table 2 does not quantify prompt fidelity or content preservation, so the net benefit of reward-guided generation remains insufficiently established.

The reliability and interpretation of the human targets need stronger evidence. Section 3.1 assigns the same intermediate label to “partially” and “uncertainly” synchronized clips, conflating perceived synchronization quality with annotation uncertainty. Regression and pairwise ranking subsequently treat aggregated labels as meaningful ordered targets. However, the main text provide no inter-annotator agreement statistics, category-specific disagreement analysis, or human-agreement reference level. Without these, it is difficult to determine how reliably the labels capture synchronization, particularly for ambiguous or weakly time-localized events.

Questions for authors

Are training, validation, and benchmark splits disjoint at the source-prompt or source-caption level, rather than only at the (uid, model) or generated-clip level? How does performance change when evaluating on a completely held-out generator?

Can the authors quantify the synchronization–prompt-fidelity trade-off under reward optimization, including the frequency of omitted sound-producing objects or events, and report whether human preference gains persist when both synchronization and preservation of requested content are evaluated jointly?

Reviewer_o4qROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper targets audio–visual synchronization in jointly generated (not merely real) video+audio clips. The authors contribute three linked artifacts: SyncReward-Data-80K (80K professional sync ratings on outputs from eight AV generators, built with a taxonomy over Speech / Onset / Instrument / Ambient events), SyncReward-Bench (1K held-out, multi-annotator clips for evaluation), and SyncReward (a ~1.6B reward model: Stage 1 semantic–temporal contrastive pretraining on ~2.2M real clips from AudioSet+VGGSound with Perception Encoder AV; Stage 2 frozen encoders + cross-modal Transformer and scalar head trained on the 80K human scores with regression and pairwise ranking).

Critical strengths
  1. Timely, well-scoped problem with a coherent “data → benchmark → model” package. The paper correctly argues that sync failures in joint generation (semantic mismatch, missing/hallucinated events) are not well captured by real-video offset metrics or generic omni judges. The taxonomy-guided prompt construction, multi-generator coverage, and explicit train/bench split (no shared (prompt uid, generator) pairs) make the effort usable as a starting point for the community if released as claimed.

  2. Empirical comparison that includes strong, data-matched LLM baselines. Fine-tuning Qwen2.5-Omni and Qwen3-Omni on the same 80K labels and showing that specialized metrics fail on generated content (Table 1) is informative: it supports the claim that generated-content human labels matter, not only architecture scale.

Critical weaknesses
  1. Core reward claims rest on incomplete human evaluation. Table 2 states that human preference scores are “provisional estimates from base-referenced comparisons” and “will be replaced by complete all-pairs evaluation.” Yet the abstract and conclusion emphasize SyncReward as the strongest optimization signal based largely on these numbers (+11/+12 human preference points). For a paper whose main contribution includes reward-guided generation, treating non-final human metrics as primary evidence is a serious soundness issue and is not a minor fix without rerunning the full human study.

  2. The “human-aligned” benchmark and metric SOTA are tightly coupled to noisy, in-house labels with limited external validity. SyncReward-Bench reports Fleiss’ κ = 0.23 and pairwise annotator Spearman ~0.45–0.50 (exact agreement ~52–55%), while training labels use bias-corrected aggregation and the bench uses simple means—protocol mismatch. SyncReward is trained to predict these labels and evaluated on the same construction pipeline; gains over same-data 30B FT, while real, are incremental and do not establish general evaluation or reward utility. Reward experiments are only on LTX-2.3 with a failure-biased RL prompt pool (excluding “easy” LTX prompts), so improvements may not transfer to other generators or unbiased prompt distributions—undermining the paper’s broad claims about improving AV generation.

Questions for authors

Human preferences (Table 2): What is the exact protocol for “base-referenced comparisons” vs the planned “complete all-pairs” study (number of prompts, raters, pairs per method)? Will the final human results still support the reported margins over SynthSync and other rewards, and can you share them for the camera-ready if accepted?

Label reliability: Given Fleiss’ κ = 0.23 on Bench, what is inter-rater agreement on the 80K training set? Why is bias correction applied for training but not for Bench aggregation, and how sensitive are Table 1 rankings to aggregation choice?

Fair strongest baselines: Did you fine-tune SynthSync (or another sync-specific preference model) on SyncReward-Data-80K with matched compute? If not, why is Qwen-FT considered sufficient as the primary “same data” competitor?

Cross-generator / cross-model generalization: Can you report Best-of-N or post-training results on at least one non-LTX generator, or hold out entire generators from Stage 2 training and evaluate metric performance?

Reward hacking: What fraction of RL/BoN gains show prompt fidelity regressions (e.g., missing instruments, simplified motion) beyond the two qualitative examples? Any automatic check for “sync via content deletion or silent video”?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.