- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates latent reward hacking in video diffusion alignment, where optimising against a fixed latent reward model leads to increasing reward scores but deteriorating generation quality. The authors attribute this phenomenon to distributional escape, in which generated samples move beyond the reward model’s training distribution. To address this issue, they propose CoRe, a co-evolving reward framework that alternates between updating the reward model on current generated samples and optimising the generator. The method incorporates a three-way Bradley–Terry objective, relative reward optimisation, and feature-space regularisation to stabilise training. Experiments on Wan2.1-T2V-1.3B demonstrate improved generation quality and reduced reward hacking compared with fixed-reward and other alignment methods.
Critical strengths
-
The paper identifies an important challenge in latent-space video alignment and provides empirical evidence that fixed reward optimisation can lead to distributional drift and generation collapse. The analysis of reward scores, feature distributions, and generation quality provides useful insights into this failure mode.
-
CoRe combines online reward-model updates with relative reward optimisation and feature-space regularisation. The experiments demonstrate improved training stability and competitive performance on VBench and VBench-2.0, suggesting that the framework can effectively mitigate latent reward hacking.
Critical weaknesses
-
The proposed framework combines several mechanisms, including online reward-model updates, relative reward optimisation, feature-space anchoring, and flow-matching regularisation. Although the paper provides ablations of the generator objective, it remains unclear whether continuously updating the reward model is essential to the observed improvements. In particular, a comparison between a fixed reward model and a co-evolving reward model under the same relative objective and regularisation would help isolate the contribution of co-evolution. Without this comparison, the central methodological claim is not fully established.
-
The experiments are conducted exclusively on Wan2.1-T2V-1.3B, limiting evidence that the proposed framework generalises to larger or architecturally different video diffusion models.
-
The evaluation relies primarily on automatic benchmarks. Although CoRe achieves competitive results, its VBench improvement over the strongest baseline is relatively small (84.92 vs. 84.78), and no statistical significance analysis or direct human preference evaluation is provided. These limitations make it difficult to assess the robustness and practical significance of the reported improvements.
Questions for authors
Q1. Could the authors compare fixed and co-evolving reward models using identical relative reward objectives and regularisation terms? This would help clarify whether the improvements primarily originate from reward-model adaptation or the additional stabilisation mechanisms.
Q2. Could the authors provide a more quantitative analysis of how reward-model reliability changes as the generator distribution shifts, such as preference prediction accuracy on generated samples collected at different training stages? This would strengthen the proposed explanation of latent reward hacking.
Q3. Does CoRe remain effective on larger video diffusion models or different model architectures? Additional experiments would help establish whether the observed stability improvements generalise beyond Wan2.1-T2V-1.3B.
Q4. Given the relatively modest improvement over the strongest VBench baseline, could the authors provide confidence intervals or human preference evaluations to establish whether the improvements are statistically and perceptually meaningful?
We thank the reviewer for the constructive feedback and the helpful suggestions.
W1 & Q1. Is co-evolution itself essential? Following the reviewer's suggestion, we added a direct comparison between a fixed reward model and CORE under identical objectives and regularization: relative reward target, kernel anchor, and flow-matching anchor, with the same warm start and update budget (Tab. R1). The only difference between the two runs is whether the reward head is refit on current generated samples. Under this matched setting. This isolates reward-model adaptation as the key component.
There is also a structural reason why the stabilization terms cannot replace co-evolution. A frozen head already scores generated latents above real ones at step 0 (0.666 vs. 0.551, Tab. 1). The relative hinge is therefore inactive and gives no learning signal. The flow-matching anchor alone leads either to no effective optimization (Fig. 8b) or to collapse (Tab. 1: DD 68.1 → 27.8). We further compare the learning dynamics of the two settings. Under the fixed RM, generated samples drift out of distribution and the reward signal becomes distorted: the generated–real score gap grows to +0.27, and generated latents form a separate high-reward cluster (Fig. 4, 5, 7). Under CORE, the gap stays near zero and the two domains overlap.
Q2. Reward reliability along training. We measure the preference accuracy of the reward model on generated samples collected at different training stages, using 300 human-labeled pairs per checkpoint (Tab. R2). For the fixed RM, accuracy drops from 66.3% at step 0 to 56.7% at step 500, while its mean score on these samples keeps rising (Tab. 1). This directly quantifies the loss of reliability under distributional shift. Under CORE, accuracy stays between 65.0% and 67.7% throughout training.
W2 & Q3. Generalization to larger models. We added experiments on Wan2.1-T2V-14B. CORE improves VBench and VBench-2.0.
W3 & Q4. Significance of the improvement. The baselines in Tab. 2 are post-training methods built on the same Wan2.1-1.3B backbone; we achieved competitive performance improvement against the baselines.The most controlled comparison is therefore against the same pretrained base model, where CORE improves VBench Total by +0.96 (83.96 → 84.92), Semantic by +2.10, and VBench-2.0 Total by +1.73, with Controllability rising by +4.49. We have also added: (i) Statistical significance. We ran tests with multiple random seeds (42, 123, and 2000) and used the official standard prompts for evaluation, mean ± standard deviation across seeds is 84.92 ± 0.12 on VBench and 58.03 ± 0.20 on VBench-2.0. (ii) Human preference evaluation.
Table. Human evaluation (Win / Tie / Lose %, CORE vs. each method). 6 raters, 20 prompts; videos generated with the same prompt and seed, presented in randomized, blind order. Inter-rater agreement (Fleiss' κ) = 0.62.
Table R1. Fixed vs. co-evolving RM under identical objectives (relative target + kernel anchor + flow-matching anchor; same warm start and update budget). Avg. is the mean of six dimensions
Table R2. Preference accuracy (%) of the reward model on generated samples at different training stages (300 human-labeled pairs per checkpoint, same prompt and different seeds; 95% bootstrap CI in parentheses).