acceptodds
Under review as a conference paper at ICLR 2027

WoRL: Efficient Reinforcement Learning Framework for Video World Models with Next-Video Reward Modeling

Abstract

While video world models have achieved remarkable progress, reinforcement learning (RL) remains largely underexplored for their post-training. RL offers a promising way to improve these models through self-generated rollouts without real and long videos with interactive signal, yet scaling it in autoregressive video generation remains challenging due to expensive long-video optimization, the lack of history-aware reward models, and limited exposure to accumulated consistency failures during standard rollouts. We introduce **WoRL**, an end-to-end RL framework that addresses these challenges through three components. **(1) Efficient Train–Test-Matched RL.** We eliminate the intra-clip train–test mismatch in autoregressive DiffusionNFT and introduce Parallel Clean-Prefix Attention, context-aware policy scheduling, and distributed execution to scale full-attention RL to large models and long video contexts. **(2) Next-Video Reward Model (NVRM).** NVRM evaluates each generated clip conditioned on its preceding video history, enabling history-aware preference optimization over long horizons. We further train the model with a graph-based data construction pipeline that builds 312.4k video-continuation preference pairs without human expert annotation. **(3) Revisit-Centered Training.** We construct trajectories that revisit observed scenes to expose accumulated inconsistencies and introduce rewind regularization to improve learning under limited generation diversity. WoRL achieves a **9.2× training speedup** over the naive implementation. NVRM reaches **74.4% consistency agreement** and **82.4% physical-plausibility agreement** with blinded human judgments, outperforming HPSv3 and 3D Score on both criteria. Across action-conditioned LingBot-World-Fast-14B and text-conditioned Helios-Distilled-14B, WoRL consistently improves long-term consistency, controllability, and physical plausibility, yielding a **37.5-percentage-point human-preference margin**. The qualitative results are best visualized as videos in the supplementary material.

Primary area:
generative models
Keywords:
video generation, world model, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xSpxOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a efficient RL post-training framework designed for video world models. Traditional RL workflow includes candidate rollout generation, reward computation, and policy gradient optimization, which suffers from prohibitive training latency when applying to long-horizon training. Therefore, the authors systematically optimize each stage. Specifically, they propose Parallel Clean-Prefix Attention to merge policy evaluations across multiple chunks, drastically reducing policy gradient training time and preserving train–test-matched RL. They further design a Context-Aware Policy Scheduling strategy to eliminate KV-cache overhead across stages, alongside a Distributed Execution to circumvent communication delays during reward computation.

Beyond systems-level efficiency, the paper addresses a core modeling limitation: prior approaches evaluate clips independently without history conditioning, leading to severe rollout inconsistencies. To resolve this, the authors introduce the Next-Video Reward Model (NVRM) trained on an automated preference pair synthesis pipeline. By generating real–fake and fake–fake pairs via pre-trained models and annotating them with VLMs, they construct a preference graph. Then, they apply positional debiasing, MFAS, and MDM to resolve preference contradictions and inconsistencies.

Empirical results indicate a 9.2× speedup over baseline. Moreover, experiments demonstrate the efficacy of NVRM and RL strategy across two video world models. It indicates the enhanced alignment between human preference and NVRM in terms of consistency and physical plausibility.

Critical strengths

The paper introduces an human-annotation-free, history-conditioned reward model that effectively addresses the critical issues of long-horizon rollout inconsistency and temporal drift in video world models. This formulation is technically sound and offers insightful perspectives that could inspire future research in this domain. In addition, extensive qualitative and quantitative evaluations substantiate the efficacy of the method.

Critical weaknesses
  1. Compared to prior approaches, this work decomposes a clip into multiple chunks (J blocks in the paper), allowing RL post-training to optimize over multiple chunks concurrently. However, the authors do not evaluate the impact of different values of J on post-training performance. Moreover, Parallel Clean-Prefix Attention appears to provide meaningful speedups only when J>1. The authors should provide further ablation studies and discussion regarding the impact of varying J.

  2. The clarity and organization of the paper should be improved and self-contained, as the text contains multiple confusing explanations. For instance, the method section does not clearly state whether Parallel Clean-Prefix Attention is used exclusively during policy gradient training or also during the rollout stage, creating unnecessary confusion for the reader. Key details are introduced without adequate definitions or are relegated entirely to the appendix (e.g., the metric 'PI camera' and the benchmark 'RevBench').

Given these critical weaknesses, I currently lean towards weak rejection. But I am willing to reconsider and upgrade my score if the authors adequately resolve these issues in their response.

Questions for authors
  1. The experimental section lacks a coherent organization. Different experimental sections are jumbled together. Experiment setups ('Training and evaluation', 'Human-evaluation protocol'), ablation studies ('Component ablations', 'Preference curation'), and baseline comparisons ('Long-horizon quality', 'Qualitative analysis') are formatted at the same hierarchical level without clear distinction. I strongly recommend the authors to thoroughly rewrite and restructure the manuscript to meet the publication standard of ICLR.

  2. Inconsistent terminology should be addressed: Acronyms used in Figure 6, such as 'MFAS' and 'MDM', lack corresponding definitions or explanations in the main text.

  3. In standard RL workflows, rollout generation typically does not involve policy switching; such swapping overhead is expected primarily during policy gradient optimization phase. Could the authors clarify why Context-Aware Policy Scheduling produces a speedup during the rollout phase?

  4. For Table 3, which base model was used for this ablation? Moreover, the authors do not report the results when combining all rewards or multiple rewards, making it hard to assess the impact and necessity of each individual reward.

  5. What are the concrete configurations for 'Noisy-prefix' and 'Clean-prefix' in Table 4, and what specific claim is this experiment designed to substantiate? A more detailed explanation of the motivation and findings behind this ablation is needed.

  6. In Table 5, the authors should consider incorporating camera control for completeness, thereby better indicating the distinct contribution of each reward model.

6 comments
AuthorsOpenReview ↗

Response to Reviewer xSpx (1/2): W1, W2 (Q1) and Q2

We thank the reviewer for highlighting the value of the history-conditioned reward model and the effectiveness of our method. We address other concerns regarding experiment details in turn below.

W1: Compared to prior approaches, this work decomposes a clip into multiple chunks (J blocks in the paper), allowing RL post-training to optimize over multiple chunks concurrently ... provide meaningful speedups only when J>1. The authors should provide further ablation studies and discussion regarding the impact of varying J.

We thank the reviewer for the valuable suggestions. The concern is how the number of J affect post-training performance.

Following your suggestions, we compare J∈{1,4,7}J \in \{1, 4, 7\} using the same experimental setup as the ablation study in the main paper. For each value of JJ, we measure the speedup of PCPA over serial clean-prefix replay on the same workload and report WBench-Navigation Overall to assess post-training quality.

Blocks per clip JJPCPA training speedup ↑WBench-Navigation Overall ↑
Before RL—77.45
11.00×77.79
41.62×78.34
7 (default)2.31×79.12

For LingBot-World-Fast, our default is J=7J=7, corresponding to a roughly 5-second clip. Smaller values of JJ are possible, but at a fixed block duration, they shorten the continuation presented to the reward model. A shorter continuation can make differences between candidates harder to judge, especially when consistency or physical errors take time to emerge. It may also reduce meaningful variation across candidates, making it harder for the RL algorithm to obtain an informative preference signal.

W2: The clarity and organization of the paper should be improved and self-contained, as the text contains multiple confusing explanations. For instance, the method section does not clearly state whether Parallel Clean-Prefix Attention is used exclusively during policy gradient training or also during the rollout stage, creating unnecessary confusion for the reader. Key details are introduced without adequate definitions or are relegated entirely to the appendix (e.g., the metric 'PI camera' and the benchmark 'RevBench').

Q1: The experimental section lacks a coherent organization. Different experimental sections are jumbled together. Experiment... at the same hierarchical level without clear distinction. I strongly recommend the authors to thoroughly rewrite and restructure the manuscript to meet the publication standard of ICLR.

Thanks for the suggestions. The concern is that the main text of the paper is not self-contained enough and experiment section is not well-structured. To address your concern, in the revision with one more page limit, we will move most key definitions from appendix to the main text and reorganize the experiment section to make it more clear. Below we address each sub question one-by-one.

Use of PCPA: PCPA is used only during policy optimization, not during rollout generation. During rollout, the policy generates one block at a time, conditioning each new block on the clean blocks that have already been generated. During optimization, the generated history is already available. PCPA uses that history to evaluate multiple target blocks in parallel while preserving the clean prefix that each block would see during rollout. We will make this distinction explicit in the method section of revision.

Evaluation details and organization: The submitted manuscript provides details on PI Camera in Appendix H.5 and on RevBench in Appendix I.3. In the revision, we will define both when they first appear in the main text and retain their detailed protocols in the appendices. We will also reorganize the experimental section to clearly separate the experimental setup, including the human-evaluation protocol, from the main results, component ablations, and reward evaluation.

Q2: Inconsistent terminology should be addressed: Acronyms used in Figure 6, such as 'MFAS' and 'MDM', lack corresponding definitions or explanations in the main text.

We thanks the reviewer for the reminder. The concern is that existing terminilogy (e.g.,MFAS and MDM) is without clear definition in the main text.

We provide more details here. After collecting preference data within a group, we form it into a directed graph. MFAS (minimum feedback arc set) is used to eliminate the conflict cycle in the preference graph (e.g., A > B, B > C, C > A). MDM (minimum directed multicut) is used to eliminate the ambiguous pairs (e.g., remove preference of A and C if A > B, B > C, A = C).

The details of MFAS (minimum feedback arc set) and MDM (minimum directed multicut) are provided in Appendix D.2 and Appendix H.1 (from Eq.38 to Eq.40). They are classical graph-based algorithm where we apply to filter the preference data.

We will include the reference and the definition in the main text in the revision.

AuthorsOpenReview ↗

Response to Reviewer xSpx (2/2): Q3, Q4, Q5 and Q6

Q3: In standard RL workflows, rollout generation typically does not involve policy switching; such swapping overhead is expected primarily during policy gradient optimization phase. Could the authors clarify why Context-Aware Policy Scheduling produces a speedup during the rollout phase?

Thanks for the observation. The Rollout column in Table 2 includes history KV-cache prefilling, not just autoregressive denoising. Following the streaming generation setup of Self-Forcing and LongLive, we consider prefilling the history KV cache as part of rollout since they are typically detached and gradient-free. The speedup comes from computing the KV cache at a single time for all the candidates sharing the same history. We will add more explanation in the revision.

Q4: For Table 3, which base model was used for this ablation? Moreover, the authors do not report the results when combining all rewards or multiple rewards, making it hard to assess the impact and necessity of each individual reward.

Thanks for the constructive feedback. Table 3 uses LingBot-World-Fast as the base model. Under the same experimental setup, combining all three rewards gives the highest Overall score among the tested configurations (79.12). The original HPSv3 and NVRM entries in Table 3 both include the camera-control reward, so we label those combinations explicitly below.

With Camera held fixed, adding HPSv3 to NVRM raises Quality from 77.99 to 79.43, while adding NVRM to HPSv3 raises Consistency from 84.74 to 86.74 and Physical from 52.73 to 55.12. HPSv3 alone gives the highest Quality score (81.77), but adding Camera to that configuration raises Interaction from 74.32 to 81.11 while lowering Quality to 79.52. This comparison shows what Camera adds when HPSv3 is the only other reward. It does not isolate Camera’s contribution when both HPSv3 and NVRM are present.

Reward configurationOverall ↑Quality ↑Settings ↑Interaction ↑Consistency ↑Physical ↑
LingBot-World-Fast (before RL)77.4579.3492.1579.1684.9551.64
WoRL (HPSv3)77.1481.7791.8574.3284.9852.82
WoRL (HPSv3 + Camera)78.1979.5292.8581.1184.7452.73
WoRL (NVRM + Camera)78.5577.9993.0481.1986.1854.35
WoRL (HPSv3 + NVRM + Camera)79.1279.4393.1581.1786.7455.12

Q5: What are the concrete configurations for 'Noisy-prefix' and 'Clean-prefix' in Table 4, and what specific claim is this experiment designed to substantiate? A more detailed explanation of the motivation and findings behind this ablation is needed.

Switching from noisy-prefix to clean-prefix conditioning raises the RevBench NVRM score from 43.78 to 45.61 in Table 4 and WBench-Navigation Overall from 78.25 to 79.12 in our additional evaluation. This ablation tests whether matching the prefix used during policy optimization to the one used at inference improves performance.

As shown in Figure 3, the noisy-prefix variant adds noise to all blocks in the current clip during optimization, so each target block attends to noisy preceding blocks within that clip. At inference, the model denoises each target block conditioned on clean, previously generated blocks. The clean-prefix variant matches this behavior during optimization by letting each noisy target block attend to clean versions of its predecessors. Parallel Clean-Prefix Attention (Section 3) preserves this conditioning while evaluating multiple target blocks in parallel.

Following your suggestions, we will add more explanation on the ablation study into the revision.

Q6: In Table 5, the authors should consider incorporating camera control for completeness, thereby better indicating the distinct contribution of each reward model.

Table 5 evaluates how closely video-quality rewards agree with human judgments of consistency and physical plausibility. The camera-control reward measures how closely the video follows the requested camera motion, which is outside the scope of that comparison. To assess its contribution, we turn to the WBench-Navigation ablation in Q4 and compare training with HPSv3 alone against training with HPSv3 and the camera-control reward.

Adding the camera-control reward raises Interaction from 74.32 to 81.11, above the base model’s 79.16. Quality drops from 81.77 to 79.52 but remains slightly above the base model’s 79.34.

Reward configurationOverall ↑Quality ↑Settings ↑Interaction ↑Consistency ↑Physical ↑
LingBot-World-Fast (before RL)77.4579.3492.1579.1684.9551.64
WoRL (HPSv3)77.1481.7791.8574.3284.9852.82
WoRL (HPSv3 + Camera)78.1979.5292.8581.1184.7452.73

Thanks for the response. Most of my concerns have been addressed. However, the authors have not yet incorporated the necessary revisions regarding clarity into the main paper.

Reviewer_mHCaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WoRL, a reinforcement learning (RL) post-training framework for autoregressive video world models. It aims to address critical issues in long-horizon video generation, such as entity identity drift, scene state inconsistency, geometric structural instability, and a lack of physical plausibility. To overcome the limitations of existing video-based RL—specifically regarding training efficiency, reward modeling conditioned on historical context, and the accumulation of errors over long sequences—the authors propose three core components:

  1. Efficient, Training-Inference-Aligned RL: The framework introduces "Parallel Clean-Prefix Attention" (PCPA), which enables the parallel evaluation of target video segments while maintaining preceding segments as a "clean" (unmodified) context. Combined with policy scheduling, KV cache reuse, and distributed execution, this approach significantly reduces the computational overhead of RL over long contexts.

  2. Next-Video Reward Model (NVRM): This component consists of a reward model conditioned on historical video content. It evaluates whether a newly generated segment remains consistent with previously established entity identities, scene states, and physical properties. The authors constructed a dataset of 312.4K video-continuation preference samples by leveraging a multimodal model to generate comparisons, subsequently refining the data through bidirectional comparison, cycle removal in preference graphs, and path filtering.

  3. Scene-Revisit-Centric Training Environment: The framework constructs trajectories involving "observing a scene, leaving, and returning," which makes long-horizon state drift explicitly observable. Additionally, the authors introduce "rewind regularization," which incorporates reverse-revisit candidates to create more opportunities for meaningful preference comparisons, particularly when the generated outputs lack sufficient diversity.

Experiments were conducted using the action-conditioned video world model LingBot-World-Fast-14B and the text-conditioned video model Helios-Distilled-14B. The results show that the full system achieves a 9.2× speedup in training compared to a naive "clean-prefix replay" baseline. The NVRM demonstrates an alignment rate with human judgment of 74.4% for consistency and 82.4% for physical plausibility. In overall human preference evaluations, videos generated using the RL-tuned models achieved a 53.1% win rate, compared to 15.6% for the baseline, with 31.3% of cases resulting in a tie. Overall, this paper attempts to integrate long-horizon reward modeling, train-inference consistency, and targeted training trajectories to develop a more comprehensive reinforcement learning framework for video world models.

Critical strengths
  1. Well-defined problem with strong systemic synergy among the three technical modules This paper focuses not on the visual quality of isolated video clips, but rather on how errors propagate during the autoregressive generation process, leading to a degradation in long-range consistency—a problem of clear research significance.

The three modules each address a specific bottleneck: PCPA resolves the mismatch between training-time and inference-time contexts; NVRM provides reward signals for the quality of continuations based on historical dependencies; and "revisiting trajectories" proactively exposes long-term state changes that are difficult to observe during standard forward generation. This synergistic design—spanning optimization algorithms, reward functions, and training environments—offers a more comprehensive approach than simply swapping out reward models or applying existing reinforcement learning (RL) algorithms directly.

  1. Significant improvements in system efficiency, validated across two types of video generation models The paper reports that, using 16 NVIDIA B200 GPUs, the time required for a single optimization update has been reduced from 109.8 minutes to 12.0 minutes. PCPA alone cuts training time from 67.1 minutes to 29.0 minutes, with further reductions achieved through strategic scheduling and system-level optimizations.

Moreover, the experiments encompass both action-conditioned models (LingBot-World-Fast-14B) and text-conditioned models (Helios-Distilled-14B), utilizing benchmarks such as WBench, RevBench, and VBench-Long. This demonstrates that the framework is not limited to a single modality of conditional input and possesses strong potential for practical, real-world application.

Critical weaknesses
  1. The attribution of gains relative to existing video reinforcement learning (RL) methods remains insufficient. While the authors attribute the contributions of WoRL to an optimization mechanism that aligns training with inference, a history-conditioned reward model, and a training environment that facilitates scene revisiting, the experiments do not fully clarify the source of the proposed method's advantages. Specifically, it remains unclear how much of the performance gain stems from the individual new components versus other factors, such as more suitable training data, reward combinations, or the allocated computational budget. The primary experiments focus on comparing a base model against a version trained using WoRL. Although the paper includes some ablation studies—such as those examining the effects of "clean" prefix conditioning and "rewind regularization," as well as comparisons of NVRM against rewards like HPSv3 and 3D Score—it lacks a comprehensive, end-to-end benchmark against related methods (e.g., WorldCompass, Astrolabe) conducted under identical base models and training budgets. Furthermore, since the primary value of PCPA lies in restoring the correct conditional structure and enhancing execution efficiency, it is essential to distinguish between quality improvements resulting from better-aligned training objectives and those resulting from higher computational efficiency (which allows for more frequent or effective model updates). Without this distinction, the causal link between the 9.2x system speedup and the observed gains in video quality remains ambiguous.

  2. The scale of NVRM evaluation and the evidence for reward generalization remain limited. The training preferences for NVRM are primarily generated and filtered by multimodal models. Although the authors employ techniques like double-sequence judgment and graph-based filtering to mitigate bias, these measures cannot entirely eliminate systematic errors in the evaluator models, inherent preference biases, or potential selection biases introduced during the data construction process. Regarding human evaluation, the paper reports on 64 pairs of videos; the alignment rate between the reward model and human judgment is calculated based on a smaller subset (excluding ties). While this serves as preliminary evidence, the statistical uncertainty is significant, making it difficult to conclude that NVRM can reliably measure long-term consistency and physical plausibility. Additionally, the training environment is explicitly designed to emphasize trajectories that involve revisiting scenes. While this design is effective for exposing specific types of long-term consistency errors, it is unclear whether these benefits generalize to scenarios such as long-form videos without explicit scene revisiting, complex object interactions, or broader environmental transitions. For text-conditioned models in particular, the WBench Overall score improved only from 68.79 to 69.63; while other metrics showed improvement, the gains were uneven. The authors should further clarify the applicability boundaries of their method across different task types and in scenarios involving long-duration failure modes.

Questions for authors

Can it be demonstrated, through controlled experiments with strict control over training budgets and data conditions, that the main benefits of WoRL do indeed come from its proposed core mechanism, and not just reward model replacement, scenario revisit data construction, or training process changes?

6 comments
AuthorsOpenReview ↗

Response to Reviewer mHCa (1/2): W1 (Q1)

We thank the reviewer for recognizing how WoRL's components address different bottlenecks and its superior system efficency. Below we clarify the matched comparison with WorldCompass, report a unified component ablation, and distinguish the effect of clean-prefix conditioning on video quality from the effect of PCPA on training speed.

W1: The attribution of gains relative to existing video reinforcement learning (RL) methods remains insufficient. While the authors attribute the contributions of WoRL to an optimization mechanism that aligns training with inference, a history-conditioned reward model, and a training environment that facilitates scene revisiting, the experiments do not fully clarify the source of the proposed method's advantages. Specifically, it remains unclear how much of the performance gain stems from the individual new components versus other factors, such as more suitable training data, reward combinations, or the allocated computational budget. The primary experiments focus on comparing a base model against a version trained using WoRL. Although the paper includes some ablation studies—such as those examining the effects of "clean" prefix conditioning and "rewind regularization," as well as comparisons of NVRM against rewards like HPSv3 and 3D Score—it lacks a comprehensive, end-to-end benchmark against related methods (e.g., WorldCompass, Astrolabe) conducted under identical base models and training budgets. Furthermore, since the primary value of PCPA lies in restoring the correct conditional structure and enhancing execution efficiency, it is essential to distinguish between quality improvements resulting from better-aligned training objectives and those resulting from higher computational efficiency (which allows for more frequent or effective model updates). Without this distinction, the causal link between the 9.2x system speedup and the observed gains in video quality remains ambiguous.

Q1: Can it be demonstrated, through controlled experiments with strict control over training budgets and data conditions, that the main benefits of WoRL do indeed come from its proposed core mechanism, and not just reward model replacement, scenario revisit data construction, or training process changes?

Thank you for your the constructive and comprehensive feedback. We try to address your concern one-by-one below.

Comparison with existing video RL methods. One of the concern is that there is no fair comaprison with existing methods. We want to clarify that Table 1(a) already includes an end-to-end comparison with WorldCompass using the same LingBot-World-Fast base checkpoint, training data and same optimization step. WoRL achieves 79.91 on WBench-Navigation Overall, compared with 78.22 for existing method WorldCompass (+1.69 points).

Detailed ablation on performance gain. Another concern is the contribution of each component in WoRL is not clear given existing ablation study. We want to clarify that on-policy training is only one component of WoRL, not the sole mechanism we expect to drive all the gains. WoRL combines a train-test matched RL algorithm, a history-conditioned reward (NVRM), and a revisit-centric training environment. To clarify the contribution of each component, we further conduct experiments as follows:

SettingOn-policy trainingNVRMRevisit-centric trainingOverall ↑Quality ↑Settings ↑Interaction ↑Consistency ↑Physical ↑
LingBot-World-Fast (base model)———77.4579.3492.1579.1684.9551.64
Ablation baseline✗✗✗77.6979.5792.3280.1184.6151.85
On-policy only✓✗✗77.9179.6892.3180.9284.6851.97
On-policy + revisit-centric training✓✗✓78.1979.5292.8581.1184.7452.73
On-policy + NVRM✓✓✗78.3579.5692.7481.1185.1653.21
Full WoRL (matched ablation setup)✓✓✓79.1279.4393.1581.1786.7455.12

Starting from the on-policy setting (77.91), adding revisit-centric training or NVRM separately raises Overall to 78.19 and 78.35, respectively. Combining both gives the highest score (79.12).

Training alignment versus execution efficiency. The concern is that it's unclear that quality improvements results from better-aligned training objectives and those resulting from higher computational efficiency. We want ot clarify that clean-prefix conditioning changes what history the policy sees during optimization (which enables quality improvement), while PCPA changes how the clean-prefix objective is computed (which improves execution efficiency). The matched noisy-prefix versus clean-prefix ablation tests the effect of conditioning on quality. The 9.2× end-to-end speedup is measured against literal clean-prefix replay.

AuthorsOpenReview ↗

Response to Reviewer mHCa (2/2): W2

W2: The scale of NVRM evaluation and the evidence for reward generalization remain limited. The training preferences for NVRM are primarily generated and filtered by multimodal models. Although the authors employ techniques like double-sequence judgment and graph-based filtering to mitigate bias, these measures cannot entirely eliminate systematic errors in the evaluator models, inherent preference biases, or potential selection biases introduced during the data construction process. Regarding human evaluation, the paper reports on 64 pairs of videos; the alignment rate between the reward model and human judgment is calculated based on a smaller subset (excluding ties). While this serves as preliminary evidence, the statistical uncertainty is significant, making it difficult to conclude that NVRM can reliably measure long-term consistency and physical plausibility. Additionally, the training environment is explicitly designed to emphasize trajectories that involve revisiting scenes. While this design is effective for exposing specific types of long-term consistency errors, it is unclear whether these benefits generalize to scenarios such as long-form videos without explicit scene revisiting, complex object interactions, or broader environmental transitions. For text-conditioned models in particular, the WBench Overall score improved only from 68.79 to 69.63; while other metrics showed improvement, the gains were uneven. The authors should further clarify the applicability boundaries of their method across different task types and in scenarios involving long-duration failure modes.

Evaluation of NVRM. The concern is that the data preprocessing cannot fully eliminates systematic judge bias or selection bias inherited from VLM judgement. We agree with it and thereby need to further verify its effective. Therefore, we also measured how well NVRM aligned with other metrics. WBench validated its metrics against human preferences using 13,515 blind pairwise comparison tasks. So we used the benchmark’s Consistency and Physical scores to assess how well NVRM evaluates consistency and physical plausibility. In this experiment, we used 480 video pairs drawn from official WBench examples for LingBot-World-Fast, Matrix-Game 3.0, and Kling 3.0.

WBench reference metricPairwise agreement with NVRM ↑
Consistency80.8%
Physical84.2%

As the table shows, NVRM agrees closely with WBench on both metrics. While this comparison cannot replace a larger human study, it provides complementary evidence that NVRM can be used to compare consistency and physical plausibility between video continuations.

The ablation in W1 also provides a separate test of NVRM's value as a training reward. With the other components held fixed, adding NVRM improves the resulting policy's WBench-Navigation Consistency and Physical scores, neither of which uses NVRM for evaluation.

We hope this can address you concern regarding effectiveness of NVRM.

Generalization beyond explicit revisits. The concern is whether WoRL can generalize to scenarios such as long-form videos without explicit scene revisiting.

To address this concern, we can provide more details on the evaluation. WBench-Navigation includes multi-turn trajectories without explicit returns, such as progressive movement, repeated actions, and L-shaped paths. On the full split, WoRL improves Overall from 77.45 to 79.91, Consistency from 84.95 to 86.96, and Physical from 51.64 to 56.51. These results show that the gains extend beyond RevBench to a broader navigation benchmark. We also provide qualitative results in our supplementary material without explicit revisiting. For instance, in a07-before.mp4, the subject’s identity drifts over time, whereas in a07-after.mp4, WoRL better preserves the subject’s identity along the same trajectory.

Text-conditioned generation. Thanks for raising this point. WoRL improves Helios’s scores on all WBench-Text sub-metrics, raising Overall from 68.79 to 69.63 (+0.84 points). This gain is comparable to several gaps between neighboring models on the published WBench full-split leaderboard. For example, LTX 2.3 and Cosmos 2.5 differ by 0.6 points, while Cosmos3-Super and LTX 2.3 differ by 1.5 points.

The gain is nonetheless smaller than what we see for LingBot-World-Fast on WBench-Navigation. One possible reason is that WBench-Text has fewer trajectories and prompts involving revisits or complex interactions, where the base model can struggle and failures in consistency and physical plausibility are easier to spot.

Thank you again for the suggestions. We will clarify the scope of these results in the revision.

Thank you to the authors for the additional experiments and detailed response. I would like to ask a follow-up question: why is there a 0.79 difference in the "Overall" score between the full ablation study and the main experiment (79.12 vs. 79.91)?

Reviewer_oc2yOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposed WoRL, an end-to-end reinforcement learning framework for post-training autoregressive video world models. The core idea is to align both policy optimization and reward evaluation with the autoregressive structure of video generation: each new clip should be optimized and evaluated conditional on its generated history, rather than scoring only as isolated video clips. The approach consists of three parts: an efficient train-test-matched RL, a historical conditional reward model NVRM, and a revisit-centric training environment. The WoRL performs experiments on action-conditioned LingBot-World-Fast and text-conditioned Helios-Distilling-14B, it reports the improvements in long-range consistency, physical plausibility, and human preference.

Critical strengths
  1. The method design is highly matched to the problem. NVRM models rewards as "continuation utility after a given history" and avoids scoring only isolated clips, making it more suitable for evaluating long-term consistency and physical plausibility.

  2. The contribution of system efficiency is strong. The paper reports that full system reduces end-to-end update latency from 109.8 minutes of literal replay to 12.0 minutes, achieving approx 9.2× speedup. This has practical implications for post-RL training on long-context, full-attention video world models.

  3. The experiments cover different conditions, which reflects the generalization ability. The paper also validates the method on action-conditioned LingBot-World-Fast and text-conditioned Helios-Distilling-14b. The results show that WBench-Navigation, RevBench, WBench-Text and VBench-Long are all improved to different degrees.

Critical weaknesses
  1. The sample size and protocol details of Human evaluation are still insufficient, which weakens the effectiveness of NVRM. The reward-human agreement of Table 5 in the paper actually only uses 43 consistency comparisons and 34 physical-plausibility after excluding human ties. The effective sample size is small. In addition, the number of annotators involved in annotation, the consistency of annotators, and the specific criteria for "consistency/physical plausibility" are insufficiently explained.

  2. The evaluation metrics are potentially coupled to the training objective, and the independent contributions of NVRM have not been sufficiently isolated. NVRM is the core reward model of WoRL, However, component ablation of Table 4 in turn uses RevBench NVRM scores to measure the effects of clean-prefix conditioning and rewind, which may lead to evaluation bias toward models optimized for NVRM. And it lacks systematic ablation to remove or replace NVRM in the complete WoRL framework to more clearly verify the independent contribution of NVRM to the final performance improvement.

Questions for authors
  1. In Table 5, the consistency of each reward is evaluated for human beings, but the effective samples are slightly less, and more sample size can further confirm the effectiveness of NVRM. At the same time, the authors could introduce more detail about the number of people involved in the human evaluation, and what are the criteria for consistency.

  2. For Table4, since the NVRM method is used for training, it may be difficult to ensure its effectiveness as an evaluation metric, simultaneously. It is suggested to supplement the comparison of subjective and objective indicators such as human evaluation, HPS, and 3D score.

  3. As one of the important components of the paper, the NVRM lacks its separate ablation.

  4. The NVRM has the highest consistency performance in Table 5, but why does the introduction of NVRM as reward also require the reward of HPS, shown in Formula 5? And what is the impact of using three rewards together on the model?

  5. There are few details about the setting of each hyper-parameter in Formula 5. The authors can make more introductions about the content and reasons of parameter design, which will make the paper better understood.

4 comments
AuthorsOpenReview ↗

Response to Reviewer oc2y (1/2): W1 (Q1) and W2 (Q2)

We thank the reviewer for the thoughtful feedback and for highlighting our history-conditioned reward design, the gains in training efficiency, and the generalization ability of our model. Below we address your concern one-by-one.

W1: The sample size and protocol details of Human evaluation are still insufficient, which weakens the effectiveness of NVRM. The reward-human agreement of Table 5 in the paper actually only uses 43 consistency comparisons and 34 physical-plausibility after excluding human ties. The effective sample size is small. In addition, the number of annotators involved in annotation, the consistency of annotators, and the specific criteria for "consistency/physical plausibility" are insufficiently explained.

Q1: In Table 5, the consistency of each reward is evaluated for human beings, but the effective samples are slightly less, and more sample size can further confirm the effectiveness of NVRM. At the same time, the authors could introduce more detail about the number of people involved in the human evaluation, and what are the criteria for consistency.

Thank you for the constructive feedback. We understand that the concern is whether NVRM's human alignment is reliable. Here we provide more details on the human study below. Twelve annotators took part in the human study, and three independently rated each of the 64 matched video pairs. They watched the videos side by side with synchronized playback, randomized display order, and hidden model identities. For consistency, we asked which continuation better preserved the subjects and environment shown earlier. For physical plausibility, we asked which showed more plausible motion, contact, and interactions. Table 5 includes only pairs for which all three annotators preferred the same video, leaving 43 pairs for consistency and 34 for physical plausibility. We will add these details as well as the visualization of the human evaluation interface to the revision.

It is difficult to substantially expand the human study during the rebuttal period. To further address the concern about evaluating NVRM’s effectiveness, we also measured how well NVRM aligned with other metrics. WBench validated its metrics against human preferences using 13,515 blind pairwise comparison tasks. So we used the benchmark’s Consistency and Physical scores to assess how well NVRM evaluates consistency and physical plausibility. In this experiment, we used 480 video pairs drawn from official WBench examples for LingBot-World-Fast, Matrix-Game 3.0, and Kling 3.0.

WBench reference metricPairwise agreement with NVRM ↑
Consistency80.8%
Physical84.2%

While this comparison cannot replace a larger human study, it provides complementary evidence that NVRM ranks video continuations similarly to WBench, whose metrics for consistency and physical plausibility have been shown to align with human preferences in large data scale.

W2: The evaluation metrics are potentially coupled to the training objective, and the independent contributions of NVRM have not been sufficiently isolated. NVRM is the core reward model of WoRL, However, component ablation of Table 4 in turn uses RevBench NVRM scores to measure the effects of clean-prefix conditioning and rewind, which may lead to evaluation bias toward models optimized for NVRM. And it lacks systematic ablation to remove or replace NVRM in the complete WoRL framework to more clearly verify the independent contribution of NVRM to the final performance improvement.

Q2: For Table4, since the NVRM method is used for training, it may be difficult to ensure its effectiveness as an evaluation metric, simultaneously. It is suggested to supplement the comparison of subjective and objective indicators such as human evaluation, HPS, and 3D score.

Thanks for the suggestions. The concern is that using NVRM for the ablation study might have evaluation bias toward models optimized for NVRM. To address this concern, we further evaluate the same two matched comparisons on WBench-Navigation, using the same evaluation protocol and metrics as in Table 1(a). We report the conditioning and rewind comparisons separately to show how each change affects the WBench-Navigation results.

AblationConfigurationWBench-Navigation Overall ↑
Prefix conditioningNoisy-prefix78.25
Prefix conditioningClean-prefix79.12
Rewind regularizationWithout rewind78.85
Rewind regularizationWith rewind79.12

WBench is a well-established benchmark that provides a comprehensive evaluation on the video world model that is independent of how NVRM evaluates the model. This is better to demonstrate how the component contributes to the performance improvement.

The separate question of whether NVRM itself improves the trained model is further addressed in Q3, where we hold HPSv3 and the camera-control reward fixed and compare training with and without NVRM.

AuthorsOpenReview ↗

Response to Reviewer oc2y (2/2): Q3, Q4 and Q5

Q3: As one of the important components of the paper, the NVRM lacks its separate ablation.

The HPSv3 and NVRM columns in Table 3 use the same training setup and both retain the camera-control reward. The column labels indicate which reward is used alongside the camera-control term. We extend this ablation with an all-three-reward configuration trained under the same setup and report the full WBench-Navigation breakdown below.

Reward configurationOverall ↑Quality ↑Settings ↑Interaction ↑Consistency ↑Physical ↑
LingBot-World-Fast (before RL)77.4579.3492.1579.1684.9551.64
WoRL (HPSv3 + Camera)78.1979.5292.8581.1184.7452.73
WoRL (NVRM + Camera)78.5577.9993.0481.1986.1854.35
WoRL (HPSv3 + NVRM + Camera)79.1279.4393.1581.1786.7455.12

With HPSv3 and the camera-control reward held fixed, adding NVRM improves Consistency by 2.00 points, Physical by 2.39 points, and Overall by 0.93 points while achiveing similar score on other metrics. This isolates the marginal contribution of NVRM.

Q4: The NVRM has the highest consistency performance in Table 5, but why does the introduction of NVRM as reward also require the reward of HPS, shown in Formula 5? And what is the impact of using three rewards together on the model?

Thanks for raising this point. The ablation in Q3 shows why we retain HPSv3. NVRM + Camera improves Consistency and Physical over the base model but lowers Quality from 79.34 to 77.99. Adding HPSv3 raises Quality to 79.43 and further improves Consistency and Physical. The three-reward configuration achieves the highest Overall score in this ablation (79.12).

The three terms in Eq. (5) serve complementary purposes. HPSv3 scores clip-level visual quality (Quality) and prompt alignment (Settings). The camera-control reward encourages adherence to the requested camera motion (Interaction). NVRM scores the consistency (Consistency) and physical plausibility (Physical) of a continuation given its generated history.

Q5: There are few details about the setting of each hyper-parameter in Formula 5. The authors can make more introductions about the content and reasons of parameter design, which will make the paper better understood.

Thanks for the suggestions. For each reward term mm, we normalize its raw score as r^m=(rm−μm)/σm\hat r_m = (r_m - \mu_m)/\sigma_m, where μm\mu_m and σm\sigma_m are the mean and standard deviation as shown in Appendix H.4. This is to put the rewards on comparable scales so that a term does not dominate simply because its raw scores have a larger numerical range.

To provide more details, these hyperparameters are estimated before RL from about 5,000 rollout sequences generated by different base models. We collect 1,250 triplet of (text prompt, first image, camera condition), and use Lingbot-World-Fast, HY-WorldPlay, Matrix-Game-3.0 and SANA-WM to generate the video based on the condition.

These statistics remain fixed throughout training. Eq. (5) sums the three normalized reward terms with equal weights since they are already in an approximately same scale.

We will clarify this rationale in the main text and refer readers to the appendix for the implementation details.

The rebuttal provides substantial new experimental evidence, including a 4-way ablation, WBench sub-dimension breakdown, and human-study protocol details. However, the fact that these results were not included in the original submission raises concerns about the completeness of the initial experimental design. Specifically: (1) the NVRM-only ablation, which reveals quality degradation, should have been disclosed from the beginning; (2) the choice to use NVRM as the evaluation metric in Table 4, despite the availability of independent metrics, warrants further explanation; and (3) the human evaluation protocol, including unanimous filtering and sample-size limitations, should have been transparently reported in the main paper. While the rebuttal partially addresses these concerns, the pattern of supplementation suggests that the original submission lacked full transparency. At the same time, given the small effective sample size of the human evaluation and the limited number of annotators, the indirect comparison using WBench alone does not fully dispel concerns about the effectiveness of NVRM.

We thank the reviewer for the follow-up. We agree that the original submission could have described the experimental setup and evaluation protocols more clearly. We have revised the manuscript for completness and highlighted the main changes in blue.

Use of NVRM in the ablations. Reporting training reward scores in targeted ablations is common practice in generative RL, as illustrated by WorldCompass, Astrolabe, DiffusionNFT, and Flow-GRPO. We used NVRM in the original ablations to measure how individual design choices affected the reward being optimized. In the main experiments, we evaluated downstream performance separately on WBench, which does not use NVRM for scoring. Following your suggestions, the revision also reports WBench-Navigation results for the ablations, including the lower Quality score for the NVRM + Camera setting.

Human evaluation and effectiveness of NVRM. We move human evaluation protocol to Sec.6.1. Appendix I.5 reports the annotator counts, evaluation criteria, and rule for retaining only comparisons with unanimous preferences. Figure 13 shows the annotation interface. We recognize that the number of retained comparisons limits the conclusions we can draw about agreement with human judgments. The WBench comparison provides less direct evidence of such agreement than a human study. However, the improvement on this established benchmark shows that using NVRM as a training reward can improve model performance. We designed NVRM to serve as a training reward and did not intend it to replace human-aligned evaluation. We believe the main experiments, the full set of ablations, the analysis of agreement with WBench, and the relatively small human study together support NVRM’s effectiveness as a training reward.

We added these analyses and protocol details in direct response to the reviewer’s questions. We acknowledge that the original submission could have been clearer, but we did not intentionally withhold information. The supplementation is used to provide more evidence to validate existing results and improve our revision based on the reviewers' suggestions. We thank the reviewer again for the constructive feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

All changes are in blue in the revised manuscript:

Sec. 3 — Clarified that PCPA is used during policy optimization, not during rollout generation (xSpx).

Sec. 4 — Defined MFAS and MDM, explained the roles of the three reward terms, and introduced PI Camera with a reference to App. H.5 (oc2y, xSpx).

Sec. 6.1–6.3 — Separated the experimental setup, main results, and ablations. Moved the human-evaluation protocol into the setup and added a pointer to the RevBench protocol in App. I.3 (oc2y, xSpx).

Tab. 3a–b — Added WBench-Navigation results for the clean-prefix and rewind comparisons and a joint ablation of on-policy training, NVRM, and revisit-centric training (oc2y, mHCa, xSpx).

Tab. 3c — Expanded the reward ablation to compare HPSv3 alone, combinations with the camera-control reward, and the full three-reward configuration (oc2y, xSpx).

App. F — Added a 480-pair evaluation of NVRM’s agreement with WBench on Consistency and Physical (oc2y, mHCa). Added a block-count ablation at (J=1,4,7), reporting PCPA training speedup and WBench-Navigation Overall (xSpx).

App. I.5 and Fig. 13 — Added annotator counts, evaluation criteria, the unanimous-preference rule, and a screenshot of the annotation interface (oc2y, mHCa).

App. J — Clarified where the gains are most visible and noted that performance depends on the base model (mHCa).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.