- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a efficient RL post-training framework designed for video world models. Traditional RL workflow includes candidate rollout generation, reward computation, and policy gradient optimization, which suffers from prohibitive training latency when applying to long-horizon training. Therefore, the authors systematically optimize each stage. Specifically, they propose Parallel Clean-Prefix Attention to merge policy evaluations across multiple chunks, drastically reducing policy gradient training time and preserving train–test-matched RL. They further design a Context-Aware Policy Scheduling strategy to eliminate KV-cache overhead across stages, alongside a Distributed Execution to circumvent communication delays during reward computation.
Beyond systems-level efficiency, the paper addresses a core modeling limitation: prior approaches evaluate clips independently without history conditioning, leading to severe rollout inconsistencies. To resolve this, the authors introduce the Next-Video Reward Model (NVRM) trained on an automated preference pair synthesis pipeline. By generating real–fake and fake–fake pairs via pre-trained models and annotating them with VLMs, they construct a preference graph. Then, they apply positional debiasing, MFAS, and MDM to resolve preference contradictions and inconsistencies.
Empirical results indicate a 9.2× speedup over baseline. Moreover, experiments demonstrate the efficacy of NVRM and RL strategy across two video world models. It indicates the enhanced alignment between human preference and NVRM in terms of consistency and physical plausibility.
Critical strengths
The paper introduces an human-annotation-free, history-conditioned reward model that effectively addresses the critical issues of long-horizon rollout inconsistency and temporal drift in video world models. This formulation is technically sound and offers insightful perspectives that could inspire future research in this domain. In addition, extensive qualitative and quantitative evaluations substantiate the efficacy of the method.
Critical weaknesses
-
Compared to prior approaches, this work decomposes a clip into multiple chunks (J blocks in the paper), allowing RL post-training to optimize over multiple chunks concurrently. However, the authors do not evaluate the impact of different values of J on post-training performance. Moreover, Parallel Clean-Prefix Attention appears to provide meaningful speedups only when J>1. The authors should provide further ablation studies and discussion regarding the impact of varying J.
-
The clarity and organization of the paper should be improved and self-contained, as the text contains multiple confusing explanations. For instance, the method section does not clearly state whether Parallel Clean-Prefix Attention is used exclusively during policy gradient training or also during the rollout stage, creating unnecessary confusion for the reader. Key details are introduced without adequate definitions or are relegated entirely to the appendix (e.g., the metric 'PI camera' and the benchmark 'RevBench').
Given these critical weaknesses, I currently lean towards weak rejection. But I am willing to reconsider and upgrade my score if the authors adequately resolve these issues in their response.
Questions for authors
-
The experimental section lacks a coherent organization. Different experimental sections are jumbled together. Experiment setups ('Training and evaluation', 'Human-evaluation protocol'), ablation studies ('Component ablations', 'Preference curation'), and baseline comparisons ('Long-horizon quality', 'Qualitative analysis') are formatted at the same hierarchical level without clear distinction. I strongly recommend the authors to thoroughly rewrite and restructure the manuscript to meet the publication standard of ICLR.
-
Inconsistent terminology should be addressed: Acronyms used in Figure 6, such as 'MFAS' and 'MDM', lack corresponding definitions or explanations in the main text.
-
In standard RL workflows, rollout generation typically does not involve policy switching; such swapping overhead is expected primarily during policy gradient optimization phase. Could the authors clarify why Context-Aware Policy Scheduling produces a speedup during the rollout phase?
-
For Table 3, which base model was used for this ablation? Moreover, the authors do not report the results when combining all rewards or multiple rewards, making it hard to assess the impact and necessity of each individual reward.
-
What are the concrete configurations for 'Noisy-prefix' and 'Clean-prefix' in Table 4, and what specific claim is this experiment designed to substantiate? A more detailed explanation of the motivation and findings behind this ablation is needed.
-
In Table 5, the authors should consider incorporating camera control for completeness, thereby better indicating the distinct contribution of each reward model.
6 comments
Response to Reviewer xSpx (2/2): Q3, Q4, Q5 and Q6
Q3: In standard RL workflows, rollout generation typically does not involve policy switching; such swapping overhead is expected primarily during policy gradient optimization phase. Could the authors clarify why Context-Aware Policy Scheduling produces a speedup during the rollout phase?
Thanks for the observation. The Rollout column in Table 2 includes history KV-cache prefilling, not just autoregressive denoising. Following the streaming generation setup of Self-Forcing and LongLive, we consider prefilling the history KV cache as part of rollout since they are typically detached and gradient-free. The speedup comes from computing the KV cache at a single time for all the candidates sharing the same history. We will add more explanation in the revision.
Q4: For Table 3, which base model was used for this ablation? Moreover, the authors do not report the results when combining all rewards or multiple rewards, making it hard to assess the impact and necessity of each individual reward.
Thanks for the constructive feedback. Table 3 uses LingBot-World-Fast as the base model. Under the same experimental setup, combining all three rewards gives the highest Overall score among the tested configurations (79.12). The original HPSv3 and NVRM entries in Table 3 both include the camera-control reward, so we label those combinations explicitly below.
With Camera held fixed, adding HPSv3 to NVRM raises Quality from 77.99 to 79.43, while adding NVRM to HPSv3 raises Consistency from 84.74 to 86.74 and Physical from 52.73 to 55.12. HPSv3 alone gives the highest Quality score (81.77), but adding Camera to that configuration raises Interaction from 74.32 to 81.11 while lowering Quality to 79.52. This comparison shows what Camera adds when HPSv3 is the only other reward. It does not isolate Camera’s contribution when both HPSv3 and NVRM are present.
| Reward configuration | Overall ↑ | Quality ↑ | Settings ↑ | Interaction ↑ | Consistency ↑ | Physical ↑ |
|---|---|---|---|---|---|---|
| LingBot-World-Fast (before RL) | 77.45 | 79.34 | 92.15 | 79.16 | 84.95 | 51.64 |
| WoRL (HPSv3) | 77.14 | 81.77 | 91.85 | 74.32 | 84.98 | 52.82 |
| WoRL (HPSv3 + Camera) | 78.19 | 79.52 | 92.85 | 81.11 | 84.74 | 52.73 |
| WoRL (NVRM + Camera) | 78.55 | 77.99 | 93.04 | 81.19 | 86.18 | 54.35 |
| WoRL (HPSv3 + NVRM + Camera) | 79.12 | 79.43 | 93.15 | 81.17 | 86.74 | 55.12 |
Q5: What are the concrete configurations for 'Noisy-prefix' and 'Clean-prefix' in Table 4, and what specific claim is this experiment designed to substantiate? A more detailed explanation of the motivation and findings behind this ablation is needed.
Switching from noisy-prefix to clean-prefix conditioning raises the RevBench NVRM score from 43.78 to 45.61 in Table 4 and WBench-Navigation Overall from 78.25 to 79.12 in our additional evaluation. This ablation tests whether matching the prefix used during policy optimization to the one used at inference improves performance.
As shown in Figure 3, the noisy-prefix variant adds noise to all blocks in the current clip during optimization, so each target block attends to noisy preceding blocks within that clip. At inference, the model denoises each target block conditioned on clean, previously generated blocks. The clean-prefix variant matches this behavior during optimization by letting each noisy target block attend to clean versions of its predecessors. Parallel Clean-Prefix Attention (Section 3) preserves this conditioning while evaluating multiple target blocks in parallel.
Following your suggestions, we will add more explanation on the ablation study into the revision.
Q6: In Table 5, the authors should consider incorporating camera control for completeness, thereby better indicating the distinct contribution of each reward model.
Table 5 evaluates how closely video-quality rewards agree with human judgments of consistency and physical plausibility. The camera-control reward measures how closely the video follows the requested camera motion, which is outside the scope of that comparison. To assess its contribution, we turn to the WBench-Navigation ablation in Q4 and compare training with HPSv3 alone against training with HPSv3 and the camera-control reward.
Adding the camera-control reward raises Interaction from 74.32 to 81.11, above the base model’s 79.16. Quality drops from 81.77 to 79.52 but remains slightly above the base model’s 79.34.
| Reward configuration | Overall ↑ | Quality ↑ | Settings ↑ | Interaction ↑ | Consistency ↑ | Physical ↑ |
|---|---|---|---|---|---|---|
| LingBot-World-Fast (before RL) | 77.45 | 79.34 | 92.15 | 79.16 | 84.95 | 51.64 |
| WoRL (HPSv3) | 77.14 | 81.77 | 91.85 | 74.32 | 84.98 | 52.82 |
| WoRL (HPSv3 + Camera) | 78.19 | 79.52 | 92.85 | 81.11 | 84.74 | 52.73 |
Thanks for the response. Most of my concerns have been addressed. However, the authors have not yet incorporated the necessary revisions regarding clarity into the main paper.
Response to Reviewer xSpx (1/2): W1, W2 (Q1) and Q2
We thank the reviewer for highlighting the value of the history-conditioned reward model and the effectiveness of our method. We address other concerns regarding experiment details in turn below.
We thank the reviewer for the valuable suggestions. The concern is how the number of J affect post-training performance.
Following your suggestions, we compare J∈{1,4,7} using the same experimental setup as the ablation study in the main paper. For each value of J, we measure the speedup of PCPA over serial clean-prefix replay on the same workload and report WBench-Navigation Overall to assess post-training quality.
For LingBot-World-Fast, our default is J=7, corresponding to a roughly 5-second clip. Smaller values of J are possible, but at a fixed block duration, they shorten the continuation presented to the reward model. A shorter continuation can make differences between candidates harder to judge, especially when consistency or physical errors take time to emerge. It may also reduce meaningful variation across candidates, making it harder for the RL algorithm to obtain an informative preference signal.
Thanks for the suggestions. The concern is that the main text of the paper is not self-contained enough and experiment section is not well-structured. To address your concern, in the revision with one more page limit, we will move most key definitions from appendix to the main text and reorganize the experiment section to make it more clear. Below we address each sub question one-by-one.
Use of PCPA: PCPA is used only during policy optimization, not during rollout generation. During rollout, the policy generates one block at a time, conditioning each new block on the clean blocks that have already been generated. During optimization, the generated history is already available. PCPA uses that history to evaluate multiple target blocks in parallel while preserving the clean prefix that each block would see during rollout. We will make this distinction explicit in the method section of revision.
Evaluation details and organization: The submitted manuscript provides details on PI Camera in Appendix H.5 and on RevBench in Appendix I.3. In the revision, we will define both when they first appear in the main text and retain their detailed protocols in the appendices. We will also reorganize the experimental section to clearly separate the experimental setup, including the human-evaluation protocol, from the main results, component ablations, and reward evaluation.
We thanks the reviewer for the reminder. The concern is that existing terminilogy (e.g.,MFAS and MDM) is without clear definition in the main text.
We provide more details here. After collecting preference data within a group, we form it into a directed graph. MFAS (minimum feedback arc set) is used to eliminate the conflict cycle in the preference graph (e.g., A > B, B > C, C > A). MDM (minimum directed multicut) is used to eliminate the ambiguous pairs (e.g., remove preference of A and C if A > B, B > C, A = C).
The details of MFAS (minimum feedback arc set) and MDM (minimum directed multicut) are provided in Appendix D.2 and Appendix H.1 (from Eq.38 to Eq.40). They are classical graph-based algorithm where we apply to filter the preference data.
We will include the reference and the definition in the main text in the revision.