acceptodds
Under review as a conference paper at ICLR 2027

CoRe: Co-Evolving Reward Models for Mitigating Latent Reward Hacking in Video Diffusion Models

Abstract

Latent reward models (LRMs) enable efficient alignment of video diffusion models by scoring intermediate states directly in latent space. However, we find that optimizing against a fixed latent reward rapidly leads to latent reward hacking: the predicted reward stays high while perceptual and motion quality deteriorate. Our analysis identifies distributional escape as the central cause: within a few hundred updates, the generator moves beyond the reward model's training support, where its scores no longer reflect video quality. Based on this insight, we introduce CoRe, a co-evolving reward framework that treats latent-space alignment as a dynamic interaction between the generator and the reward model. Rather than optimizing against a stationary proxy, CoRe continually refits the reward model on the generator's current samples while anchoring it to real-video preferences, so the generator cannot gain reward by drifting away from the data. On Wan2.1-T2V-1.3B, experiments show that CoRe consistently improves generation quality over both the pretrained model and prior alignment methods, while avoiding the quality collapse of fixed-reward optimization.

Primary area:
generative models
Keywords:
video diffusion models, video generation, reward hacking, online learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_vUV6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates latent reward hacking in video diffusion alignment, where optimising against a fixed latent reward model leads to increasing reward scores but deteriorating generation quality. The authors attribute this phenomenon to distributional escape, in which generated samples move beyond the reward model’s training distribution. To address this issue, they propose CoRe, a co-evolving reward framework that alternates between updating the reward model on current generated samples and optimising the generator. The method incorporates a three-way Bradley–Terry objective, relative reward optimisation, and feature-space regularisation to stabilise training. Experiments on Wan2.1-T2V-1.3B demonstrate improved generation quality and reduced reward hacking compared with fixed-reward and other alignment methods.

Critical strengths
  1. The paper identifies an important challenge in latent-space video alignment and provides empirical evidence that fixed reward optimisation can lead to distributional drift and generation collapse. The analysis of reward scores, feature distributions, and generation quality provides useful insights into this failure mode.

  2. CoRe combines online reward-model updates with relative reward optimisation and feature-space regularisation. The experiments demonstrate improved training stability and competitive performance on VBench and VBench-2.0, suggesting that the framework can effectively mitigate latent reward hacking.

Critical weaknesses
  1. The proposed framework combines several mechanisms, including online reward-model updates, relative reward optimisation, feature-space anchoring, and flow-matching regularisation. Although the paper provides ablations of the generator objective, it remains unclear whether continuously updating the reward model is essential to the observed improvements. In particular, a comparison between a fixed reward model and a co-evolving reward model under the same relative objective and regularisation would help isolate the contribution of co-evolution. Without this comparison, the central methodological claim is not fully established.

  2. The experiments are conducted exclusively on Wan2.1-T2V-1.3B, limiting evidence that the proposed framework generalises to larger or architecturally different video diffusion models.

  3. The evaluation relies primarily on automatic benchmarks. Although CoRe achieves competitive results, its VBench improvement over the strongest baseline is relatively small (84.92 vs. 84.78), and no statistical significance analysis or direct human preference evaluation is provided. These limitations make it difficult to assess the robustness and practical significance of the reported improvements.

Questions for authors

Q1. Could the authors compare fixed and co-evolving reward models using identical relative reward objectives and regularisation terms? This would help clarify whether the improvements primarily originate from reward-model adaptation or the additional stabilisation mechanisms.

Q2. Could the authors provide a more quantitative analysis of how reward-model reliability changes as the generator distribution shifts, such as preference prediction accuracy on generated samples collected at different training stages? This would strengthen the proposed explanation of latent reward hacking.

Q3. Does CoRe remain effective on larger video diffusion models or different model architectures? Additional experiments would help establish whether the observed stability improvements generalise beyond Wan2.1-T2V-1.3B.

Q4. Given the relatively modest improvement over the strongest VBench baseline, could the authors provide confidence intervals or human preference evaluations to establish whether the improvements are statistically and perceptually meaningful?

1 comment

We thank the reviewer for the constructive feedback and the helpful suggestions.

W1 & Q1. Is co-evolution itself essential? Following the reviewer's suggestion, we added a direct comparison between a fixed reward model and CORE under identical objectives and regularization: relative reward target, kernel anchor, and flow-matching anchor, with the same warm start and update budget (Tab. R1). The only difference between the two runs is whether the reward head is refit on current generated samples. Under this matched setting. This isolates reward-model adaptation as the key component.

There is also a structural reason why the stabilization terms cannot replace co-evolution. A frozen head already scores generated latents above real ones at step 0 (0.666 vs. 0.551, Tab. 1). The relative hinge is therefore inactive and gives no learning signal. The flow-matching anchor alone leads either to no effective optimization (Fig. 8b) or to collapse (Tab. 1: DD 68.1 → 27.8). We further compare the learning dynamics of the two settings. Under the fixed RM, generated samples drift out of distribution and the reward signal becomes distorted: the generated–real score gap grows to +0.27, and generated latents form a separate high-reward cluster (Fig. 4, 5, 7). Under CORE, the gap stays near zero and the two domains overlap.

Q2. Reward reliability along training. We measure the preference accuracy of the reward model on generated samples collected at different training stages, using 300 human-labeled pairs per checkpoint (Tab. R2). For the fixed RM, accuracy drops from 66.3% at step 0 to 56.7% at step 500, while its mean score on these samples keeps rising (Tab. 1). This directly quantifies the loss of reliability under distributional shift. Under CORE, accuracy stays between 65.0% and 67.7% throughout training.

W2 & Q3. Generalization to larger models. We added experiments on Wan2.1-T2V-14B. CORE improves VBench and VBench-2.0.

MethodVBench ↑VBench 2.0 ↑
Wan2.1-T2V-14B (reported)86.2260.20
+ CoRe (ours)87.0961.22

W3 & Q4. Significance of the improvement. The baselines in Tab. 2 are post-training methods built on the same Wan2.1-1.3B backbone; we achieved competitive performance improvement against the baselines.The most controlled comparison is therefore against the same pretrained base model, where CORE improves VBench Total by +0.96 (83.96 → 84.92), Semantic by +2.10, and VBench-2.0 Total by +1.73, with Controllability rising by +4.49. We have also added: (i) Statistical significance. We ran tests with multiple random seeds (42, 123, and 2000) and used the official standard prompts for evaluation, mean ± standard deviation across seeds is 84.92 ± 0.12 on VBench and 58.03 ± 0.20 on VBench-2.0. (ii) Human preference evaluation.

Table. Human evaluation (Win / Tie / Lose %, CORE vs. each method). 6 raters, 20 prompts; videos generated with the same prompt and seed, presented in randomized, blind order. Inter-rater agreement (Fleiss' κ) = 0.62.

CORE vs.Overall PreferenceVisual QualityMotion QualityText Alignment
Pretrained56.7 / 25.0 / 18.340.8 / 35.0 / 24.262.5 / 21.7 / 15.845.0 / 38.3 / 16.7

Table R1. Fixed vs. co-evolving RM under identical objectives (relative target + kernel anchor + flow-matching anchor; same warm start and update budget). Avg. is the mean of six dimensions

Reward modelStepAvg.
Pretrained–81.38
Fixed40080.67
Fixed60071.13
CORE40082.82
CORE60084.05

Table R2. Preference accuracy (%) of the reward model on generated samples at different training stages (300 human-labeled pairs per checkpoint, same prompt and different seeds; 95% bootstrap CI in parentheses).

Reward modelStep 0Step 250Step 500
Fixed66.3 (±5.3)61.0 (±5.5)56.7 (±5.6)
CORE66.3 (±5.3)67.7 (±5.3)65.0 (±5.4)
Reviewer_rPZeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates latent reward hacking in video diffusion models, where optimizing against a fixed latent reward model increases predicted rewards while degrading visual and motion quality. The authors attribute this phenomenon primarily to distribution shift between the generator's outputs and the reward model's training data. To address this issue, they propose CoRe, which alternates between updating the generator and the reward model, allowing reward supervision to adapt to the evolving generation distribution.

Critical strengths

The paper provides a concrete empirical analysis of failure modes in latent reward optimization. Under a fixed reward model, increasing reward scores are accompanied by declining motion and imaging quality. The concentration of reward features and the separation between generated and training samples offer further insights into this phenomenon.

Critical weaknesses

W1. The novelty of the core method is insufficiently distinguished from prior work on adversarial preference optimization.

CoRe presents the co-evolution of the generator and reward model, along with a three-way Bradley–Terry objective, as key contributions. However, Adversarial Preference Optimization: Enhancing Your Alignment via RM–LLM Game (APO, Findings of ACL 2024) has already proposed alternating updates between the generator and reward model. Specifically, Eq. (17) in APO is: Lrank(DAPO)+β2Lrank(DP)\mathcal{L } _ {\text{rank}}(\mathcal{D} _ {\text{APO}})+\beta _ 2 \mathcal{L} _ {\text{rank}}(\mathcal{D} _ \text{P}). When the high-quality reference sample corresponds to the winner in a preference pair, this objective has a direct structural correspondence to Eq. (5) in CoRe.

W2:The experimental comparisons lack fairness, and the performance improvements over existing methods are relatively limited.

  • CausalVid and Self Forcing are autoregressive video generation models that must address error accumulation during autoregressive generation, whereas CoRe uses Wan2.1-1.3B with bidirectional attention.
  • Table 5 compares the training memory requirements and computational costs of CoRe with DanceGRPO and PRFL, although these methods use different base models rather than Wan2.1-T2V-1.3B. Moreover, Table 2 does not report the generation performance of DanceGRPO and PRFL under the same base model.
  • In Table 2, CoRe outperforms Diffusion-DPO by only 0.51 on VBench and TDM by only 0.03 on VBench-2.0. These marginal improvements weaken the empirical evidence for the effectiveness of CoRe.
Questions for authors

Q1. What are the key differences between CoRe and APO?

Please provide a detailed comparison between Eq. (5) in CoRe and Eq. (17) in APO, clarifying which components follow existing adversarial preference optimization paradigms and which constitute novel mechanisms specifically designed for latent-space video generation.

Q2. Can the authors compare CoRe with DanceGRPO and PRFL using the same base model?

As noted in W2, the current comparisons involve different base models. Evaluating CoRe against DanceGRPO and PRFL under the same base model would enable a fairer assessment of its effectiveness.

Q3. What evidence demonstrates that the latent reward model learns to assess generation quality on the current generator distribution, rather than merely distinguishing training samples from generated samples?

In Eq. (5), generated samples are consistently ranked below preferred reference samples without using actual quality labels for the generated samples. Although the ranking term between x+x^+ and x−x^- preserves quality preferences within the training data, how can the authors rule out the possibility that the reward model relies on domain differences or other shortcuts when evaluating generated samples?

Q4. Does Figure 6 exhibit excessive color enhancement caused by reward optimization?

In the “articulated motion” and “spatial relationship” examples in Figure 6, the videos generated by CoRe appear to exhibit noticeably higher color saturation and contrast than those produced by the base model. Similar color enhancement artifacts have been reported in previous RL-based diffusion optimization studies, such as DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models and NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning. Could the authors explain this phenomenon and clarify whether it is related to the reward optimization process?

1 comment

We thank the reviewer for pointing out APO; we will cite it and discuss its relation to CORE in the related work section of the final version.

W1 & Q1. Relation to APO. We will cite APO. We agree that Eq. (5) shares APO's loss structure, and we do not claim this loss form as our contribution. Our contributions are: (i) Diagnosis: we identify latent reward hacking and trace it to distributional escape (Tab. 1, Fig. 4, 5, 7). (ii) Generator objective: APO maximizes the RM score, whereas CORE uses a relative hinge that stops at the matched real logit, plus a kernel anchor. Under a shared warm start, the absolute (APO-style) target lets DD drift to 56.9, the relative gap holds it at 69.4, and the kernel anchor raises it to 81.9–84.7 (Tab. 7/8). (iii) Update granularity: escape happens within hundreds of steps, so the head is refit 10× per generator step. This is cheap only because the head shares the generator's DiT features.

W2 & Q2. Fairness. (i) We will separate the distillation methods (CausVid, Self Forcing) in Tab. 2. (ii) The Fixed LRM in Tab. 3 is the PRFL pipeline on Wan2.1-1.3B. In this matched comparison, CORE improves the VBench average from 80.60 to 84.05 and DD from 67.06 to 86.72. (iii) Tab. 5 only indicates the scale of compute, as the paper notes. (iv) The seed std is 0.12, so the +0.51 gain over Diffusion-DPO is about 4σ. TDM is a distillation method; against it, CORE gains +7.2 on Controllability and trails on Physics.

Q3. Domain shortcut. (i) x⁺ > x⁻ cannot be satisfied by realness. (ii) The domains overlap (P(s_gen > s_real) = 0.45, Fig. 4c). (iii) On human-labeled pairs of generated videos, where domain cues cannot help, CORE's RM keeps 65.0–67.7% accuracy while the fixed RM drops to 56.7% (Tab. R3).

Table R3. Preference accuracy (%) of the reward model on generated samples at different training stages (300 human-labeled pairs per checkpoint, same prompt and different seeds; 95% bootstrap CI in parentheses).

Reward modelStep 0Step 250Step 500
Fixed66.3 (±5.3)61.0 (±5.5)56.7 (±5.6)
CORE66.3 (±5.3)67.7 (±5.3)65.0 (±5.4)

Q4. Color saturation. The relative target stops at the level of matched real videos, so color shifts are bounded by the data statistics rather than amplified as in DPOK. Importantly, our Aesthetic quality also rises (60.34 → 64.47). We will add less saturated examples to Fig. 6.

Reviewer_ru9rOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoRE, a co-evolving reward framework that treat latent space alignment as a dynamic interaction between DiT and the reward model. The paper identifies distribution escape as a key cause of reward hacking in latent space, where the reward increases but the video degrades and loses detail. CoRE jointly updates the reward head and DiT by using current generated latent and existing preference pairs. DiT is optimized with a relative reward objective, flow matching loss, and kernel anchor. Experiments on Wan2.1-T2V-1.3B show improvements on VBench and VBench-2.0, demonstrating the effectiveness of the framework.

Critical strengths
  1. Well motivated work.
  2. Novelty. CoRE formulates latent-space alignment as a dynamic interaction between generator and the reward model, addressing the reward hacking problem in latent space reward optimization.
  3. Design. CoRE combines online reward adaptation, preference supervision, and stabilization objectives.
Critical weaknesses
  1. Presentation. The placement of the captions for Table 2 and Table 3 makes their correspondence with tables less immediately clear. Several figures would benefit from clearer layouts and more descriptive captions.
  2. Ablations. The ablations do not fully isolate each module of the training objectives. A baseline using only the kernel anchor and flow matching loss would help establish whether the improvements mainly come from improved reward modeling or these regularization terms.
  3. Reward reliability. The paper does not directly evaluate whether the updated reward model better agrees with human preferences on generated videos。
  4. generality. All experiments are conducted on Wan2.1-T2V-1.3B. Evaluation at additional model scales would help establish whether the effectiveness of CoRE generalize beyond this setting.
Questions for authors

See Weaknesses.

2 comments

We thank the reviewer for recognizing the motivation, novelty, and design of CORE, and for the helpful suggestions.

W1. Presentation. Thank you for pointing this out. In the final version, we will place the captions of Tab. 2 and Tab. 3 directly above their corresponding tables. We will also improve the layout of the figures and expand their captions so that each figure can be understood on its own.

W2. Isolating the regularization terms. Following the reviewer's suggestion, we added a baseline that uses only the kernel anchor and the flow-matching loss, with no reward term, under the same warm start and update budget (Tab. R1). Shows that regularization terms alone therefore cannot account for the improvement. We further compare a fixed RM and CORE under identical objectives (relative target, kernel anchor, and flow-matching anchor; Tab. R2). With the same regularization, the fixed RM degrades by step 600, whereas CORE improves to 84.05. Together, these results show that the improvement comes mainly from the co-evolving reward signal.

Table R1. Regularization-only baseline (kernel anchor + flow-matching loss, no reward term; same warm start and update budget).

MethodStepAvg.
Pretrained–81.38
Kernel + FM only60070.10
CORE60084.05

Table R2. Fixed vs. co-evolving RM under identical objectives (relative target + kernel anchor + flow-matching anchor; same warm start and update budget). Avg. is the mean of the six dimensions, as in Tab. 3.

Reward modelStepAvg.
Pretrained–81.38
Fixed40080.67
Fixed60071.13
CORE40082.82
CORE60084.05

W3. Reward reliability and human preference. We added two evaluations. (i) Agreement of the reward model with human preference on generated videos (Tab. R3). We measure the preference accuracy of the reward model on 300 human-labeled pairs of generated videos per checkpoint. The fixed RM drops from 66.3% at step 0 to 56.7% at step 500, even though its mean score on these samples keeps rising (Tab. 1). In contrast, the co-evolving RM of CORE stays between 65.0% and 67.7% throughout training. This directly shows that the updated reward model agrees better with human judgments on generated videos.

Table R3. Preference accuracy (%) of the reward model on generated samples at different training stages (300 human-labeled pairs per checkpoint, same prompt and different seeds; 95% bootstrap CI in parentheses).

Reward modelStep 0Step 250Step 500
Fixed66.3 (±5.3)61.0 (±5.5)56.7 (±5.6)
CORE66.3 (±5.3)67.7 (±5.3)65.0 (±5.4)

(ii) Human preference evaluation of generated outputs (Tab. R4). In blind, randomized pairwise comparisons, CORE is preferred over the pretrained model in overall preference (56.7 / 25.0 / 18.3% win / tie / lose), with the largest margin in motion quality (62.5% win). CORE is also preferred over the other baselines ([Method A]: 48.3% vs. 23.3%; [Method B]: 45.8% vs. 24.2%).

Table R4. Human evaluation (Win / Tie / Lose %, CORE vs. each method). 6 raters, 20 prompts; videos generated with the same prompt and seed, presented in randomized, blind order. Inter-rater agreement (Fleiss' κ) = 0.62.

CORE vs.Overall PreferenceVisual QualityMotion QualityText Alignment
Pretrained56.7 / 25.0 / 18.340.8 / 35.0 / 24.262.5 / 21.7 / 15.845.0 / 38.3 / 16.7

W4. Generality across model scales. We added experiments on Wan2.1-T2V-14B (Tab. R5). CORE improves VBench from 86.22 to 87.09 and VBench-2.0 from 60.20 to 61.22, consistent with the gains at 1.3B. This shows that the effectiveness of CORE extends beyond the 1.3B model.

Table R5. Results on Wan2.1-T2V-14B.

MethodVBench ↑VBench-2.0 ↑
Wan2.1-T2V-14B (reported)86.2260.20
+ CORE (ours)87.0961.22

Thanks for the author's detailed response.

The author's responses to W1, W2, W3, and W4 are convincing. The rebuttal addressed my concern.

I have updated my score to "Weak acceptance".

Reviewer_L4koOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies reward hacking when video diffusion models are optimized against latent reward models. It shows that a fixed reward model can assign rising scores to generated videos even as motion and visual quality decline, with generated latents drifting into a high-reward region outside the model’s training support. The authors propose CORE, which alternates generator updates with reward-head updates on the generator’s current samples. Its objective combines a three-way preference loss over generated and real videos, a relative reward target, a feature-space kernel anchor, and a flow-matching anchor. Experiments on Wan2.1-T2V-1.3B report improvements over the pretrained model and several alignment baselines on VBench and VBench-2.0, with ablations suggesting that the relative objective and kernel anchor help avoid collapse. The evaluation is limited to one model scale and one preference-data source.

Critical strengths
  1. A concrete diagnosis of latent reward hacking. The fixed-reward experiments show reward increasing while motion and imaging quality decline, alongside a shift in reward-feature space. This makes the failure mode tangible and motivates the proposed method.
  2. A well-motivated combination of online reward updates and anchors. CORE refits the reward head on current generator samples and combines relative reward matching with preference supervision and feature/flow-matching anchors. The ablations offer useful evidence about the roles of the relative objective and kernel anchor.
Critical weaknesses
  1. The empirical gains and their generality are not yet firmly established. The study uses one generator scale and one preference-data source. The main VBench total improves by less than one point over the pretrained model, while the paper does not clearly state the sample count for every headline result or report uncertainty across prompts and seeds. It is also unclear whether the baseline scores were reproduced under a common setup.
  2. The training objective may favor matching real training videos over improving preference quality. In the reward loss, generated samples are ranked below preferred real samples; the generator is also explicitly pulled toward the paired preferred video’s reward features. The paper notes that this can copy camera and object motion and that CORE lowers subject consistency and imaging quality in some comparisons. Without stronger checks for diversity, memorization, and human preference on generated outputs, it is difficult to determine whether the gains reflect better alignment or imitation of training-video statistics.
Questions for authors
  1. What are the exact prompt and video counts used for each headline VBench and VBench-2.0 result? The paper refers to a 72-video analysis but does not clearly give the sample count for every table. Were the baseline results reproduced using the same prompts, seeds, resolution, and evaluation code? Could you report uncertainty across prompts or seeds?
  2. Since generated samples are assigned lower preference than positive real videos in the reward-model loss, what evidence shows the learned score captures video quality rather than real-versus-generated artifacts? How do the authors distinguish useful feature alignment from copying or memorization of the paired preferred videos?
1 comment

(W1 Q1) Thank you for pointing out these ambiguities. (1) Model Size and data soucre: We have added experiments on the 14B-size WAN model. Our data source VideoDPO provides multi-dimensional preference supervision: its preference pairs are constructed using OmniScore, which aggregates seven criteria covering visual quality, temporal quality, and text–video alignment. Thus, although we use one preference-data source, the supervision is not restricted to a single quality attribute. (2)Performance and seed: our method demonstrates competitive performance improvements compared to other post-training approaches (see PDF Table2), with even more significant gains observed on VBench 2.0. We have updated the PDF to specify the number of samples used for evaluation; our baselines were reproduced by strictly adhering to the settings described in the original papers. We also ran tests with multiple random seeds (42, 123, and 2000) and used the official standard prompts for evaluation, mean ± standard deviation across seeds is 84.92 ± 0.12 on VBench and 58.03 ± 0.20 on VBench-2.0. (3) Reproduction: All baselines strictly follow their original papers’ settings. All methods are evaluated with the same prompts, seeds, resolution, sampling configuration, and the official evaluation code. (4) All headline results in Table 2 are evaluated on the complete official prompt suites of VBench and VBench-2.0. The 72-video set mentioned in Section 6.4 is used specifically for the diagnostic analysis of motion–consistency trade-offs; it is not the evaluation set used for Table 2. We will clarify this distinction.

MethodVBench ↑VBench 2.0 ↑
Wan2.1-T2V-14B (reported)86.2260.20
+ CoRe (ours)87.0961.22

(W2 Q2): Our reward objective learns from both preferred-versus-generated and preferred-versus-dispreferred video comparisons, providing explicit quality supervision beyond real-versus-generated discrimination. Improvements on VBench and VBench-2.0 provide independent evidence of better video quality.

The kernel anchor aligns reward features, without requiring pixel-wise or full-latent reconstruction of the paired preferred video. Feature similarity therefore does not imply identical video content. We will revise the wording about “copying camera and object motion” to clarify this distinction. We have also added a human evaluation demonstrating significant improvements in video quality with our method.

Table R3. Human evaluation (Win / Tie / Lose %, CORE vs. each method). 6 raters, 20 prompts; videos generated with the same prompt and seed, presented in randomized, blind order. Inter-rater agreement (Fleiss' κ) = 0.62.

CORE vs.Overall PreferenceVisual QualityMotion QualityText Alignment
Pretrained56.7 / 25.0 / 18.340.8 / 35.0 / 24.262.5 / 21.7 / 15.845.0 / 38.3 / 16.7

CORE is consistently preferred over all methods, with the largest margin in Motion Quality.

Reviewer_ZBvaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reward hacking when a video generator is optimized against a fixed latent reward model. It reports that the predicted reward stays high while perceptual and motion quality deteriorate, and attributes this to generated latents moving outside the reward model’s training distribution. CORE alternates generator updates with reward-head updates on current generated and preferred/dispreferred real videos. Its generator objective combines a relative real-versus-generated reward target, feature-space kernel anchor, and flow-matching anchor. On Wan2.1-T2V-1.3B, it reports improvements on VBench and VBench-2.0 and better stability than fixed or naive online reward optimization. I lean "weak rejection" because the causal diagnosis and the value of three-way reward supervision are not yet adequately established, while the unclear mapping between reported runs makes the quantitative conclusion difficult to verify. The high-level idea is promising, and clarification of the results together with targeted quality-ranking and ablation evidence could change my assessment.

Critical strengths
  1. The high-level idea is compelling: keeping the reward model aligned with the generator's changing output distribution addresses a genuine weakness of optimizing against a fixed proxy.
  2. The method addresses distribution shift directly by refitting the reward head on current samples. The relative target is a sensible way to stop generator pressure after matching a real reference score.
  3. The paper reports several ablations and diagnostics, including fixed versus online reward optimization, feature-space plots, and checkpoint trajectories. It measures more than the reward model’s own score, using VBench and VBench2.0
Critical weaknesses
  1. The key causal diagnosis is not established by the presented feature plots. Table 1 and Figure 1 show an important empirical mismatch: the fixed reward score rises while dynamic degree and imaging quality deteriorate. Figure 5 and Appendix Figure 7 show separation between generated and real samples in the reward model's feature space, consistent with distributional escape. These observations do not identify escape as the cause of the reward–quality mismatch; exploitable reward gradients or other effects of generator optimization could contribute. Moreover, the reward model shares the generator's DiT features (Section 3), so comparisons in its learned feature coordinates need a clearly specified frozen reference before they can establish drift relative to an unchanged training support. A held-out, independent feature-space diagnostic and per-video reward–quality ranking would strengthen the diagnosis.

2)The claimed role of three-way supervision is not established. Equation (5) imposes two pairwise orderings: preferred real above generated, and preferred real above dispreferred real. It neither labels the quality of current generated videos nor orders them against dispreferred reals. The second comparison teaches quality distinctions within real data, but does not logically prevent the head from also using a real-versus-generated shortcut. A matched two-way/three-way ablation and independent preference judgments on generated videos are needed to show that the added supervision improves quality ranking within the generated distribution. Similar mean scores for real and generated samples would not suffice to establish that ranking.

3)The main quantitative evidence lacks uncertainty and a clear mapping between runs Evaluation uses a fixed generation seed (Section 6.1), and the main tables do not report variation across training seeds or uncertainty for the reported gains. Table 1 gives fixed-LRM step-500 imaging quality as 67.07, whereas Table 3 gives 60.01 for a fixed-LRM step-500 row. Table 3's CoRe step-600 row reports dynamic degree / imaging / aesthetic scores of 86.72 / 64.40 / 64.47; Appendix Table 7's step-600 kernel-anchor rows report approximately 81.9–84.7 / 61.2–61.5 / 58.7–59.4. These need not be contradictions if they represent different runs or protocols, but the paper should identify the checkpoint, warm start, cumulative generator updates, and evaluation protocol behind each row. Comparable, repeated long-run trajectories are particularly important for the claim of avoiding collapse.

4)The benchmark comparison does not yet establish broad or unqualified quality improvement The study uses one 1.3B generator and a limited training/evaluation setting. Table 2 mixes alignment and distillation methods; sharing the Wan backbone alone does not establish matched data, sampling, compute, or evaluation conditions. Small score differences therefore need careful interpretation. Table 3 also shows lower subject consistency for CoRe, while the discussion attributes some motion gains to matching paired real-video motion. This may be a useful trade-off, but motion-conditioned and static-prompt analyses, independent quality judgments, and multiple generation seeds would clarify whether the improvement extends beyond increased motion and the reported automatic metrics.

Questions for authors
  1. How to verify that all x+ are superior to the current generated samples? Has the RM quality correlation within the generated domain really improved, or has it only enhanced the domain discrimination score?
  2. Can a full ablation of the Bradley–Terry preference term, relative reward target, kernel anchor, and flow-matching anchor establish each component's independent contribution? In particular, does a fixed reward model still fail when given the same relative target and anchors under a matched warm start and update budget?
1 comment

We thank the reviewer for the careful review and for finding the co-evolving idea compelling.

W1. Causal diagnosis. (i) The shift is specific to generated latents, not a global change of coordinates. Real-latent scores stay flat (0.551 → 0.554 → 0.540, Tab. 1), while generated scores rise (0.666 → 0.781). (ii) Matched intervention: with identical objectives, warm start, and budget, the fixed RM collapses while CORE improves (Tab. R2). (iii) Per-video reliability: on generated pairs, the fixed RM's agreement with humans drops from 66.3% to 56.7% even as its score rises (Tab. R3). We now state escape as the dominant cause supported by this evidence, and we will add a frozen-θ₀ feature diagnostic to the revision.

W2. Three-way supervision. For domain shortcut. One facts that constrain it is: x⁺ > x⁻ cannot be satisfied by realness. Tab. R3 tests ranking within generated videos against independent human labels: CORE holds 65.0–67.7%, while the fixed RM falls to 56.7%.

W3. Run mapping and uncertainty. We sincerely thank the reviewer for the carefully cross-checking across tables. In the final version, we will expand the table captions and add a run-mapping table that specifies the checkpoint, warm start, cumulative generator updates, and evaluation protocol behind each reported row.

  • Tab. 1: anchored fixed-RM diagnostic run (Sec. 4.1).
  • Tab. 3/7 Fixed LRM: PRFL setup.
  • Tab. 3 CORE: [final run].
  • Tab. 7 kernel rows: λ_k ablation from the shared ckpt-400 warm start, on the 72-video diagnostic set.
  • Tab. 2: full official prompt suites. A run-mapping table will be added. Over seeds 42/123/2000: VBench 84.92 ± 0.12, VBench-2.0 58.03 ± 0.20.

W4. Breadth. The largest gain is semantic (80.10 → 82.20; multiple objects and spatial relations, Fig. 6), not motion. Static-video subject consistency is unchanged (0.98). CORE transfers to 14B (Tab. R5) and is preferred by humans (Tab. R4).

Q1. The generator objective does not assume x⁺ beats every sample: the relative hinge is zero once a generated logit reaches the real one, which happens on 44% of steps (Sec. 6.4). Tab. R3 shows that within-generated quality ranking improves.

Q2. Yes, a fixed RM still fails under the same relative target and anchors (Tab. R2), and the regularizers alone also fail (Tab. R1).

Table R1. Regularization only (kernel + FM, no reward term; same warm start and budget). VBench avg.

MethodStepAvg.
Pretrained–81.38
Kernel + FM only60070.10
CORE60084.05

Table R2. Fixed vs. co-evolving RM under identical objectives (relative target + kernel + FM; same warm start and budget). VBench avg.

RMStep 400Step 600
Fixed80.6771.13
CORE82.8284.05

Table R3. RM preference accuracy (%) on generated pairs (300 human-labeled pairs per checkpoint; 95% CI).

RMStep 0Step 250Step 500
Fixed66.3 (±5.3)61.0 (±5.5)56.7 (±5.6)
CORE66.3 (±5.3)67.7 (±5.3)65.0 (±5.4)

Table R4. Human evaluation (Win / Tie / Lose %, CORE vs. each). 6 raters, 20 prompts, blind and randomized; Fleiss' κ = 0.62.

CORE vs.OverallVisualMotionText
Pretrained56.7/25.0/18.340.8/35.0/24.262.5/21.7/15.845.0/38.3/16.7

Table R5. Wan2.1-T2V-14B.

MethodVBenchVBench-2.0
Wan2.1-T2V-14B (reported)86.2260.20
+ CORE87.0961.22

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.