- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the World-VLA-Loop framework, which alternately performs video world model training and VLA policy optimization. It constructs a SANS dataset using successful and near-successful trajectories, jointly trains the video generation model and the reward prediction head, and uses the resulting world model for GRPO-based policy post-training; subsequently, it collects trajectories of the updated policy running in the real environment to further optimize the world model. The experiments cover a selection of LIBERO tasks and two real-world manipulation tasks, and conduct ablation studies on the "near-successful" data and reward modeling.
Critical strengths
The improvement of the policy was verified through execution by actual robots. After two rounds of training, the real-environment success rates on the two tasks increased from 13.3% to 50.0% and from 26.7% to 53.3%, respectively. These results provide useful preliminary evidence that training in the learned simulator can improve the policy deployed in reality. The studies on the individual components confirm the effectiveness of the proposed training signal. Removing the "near-success" data or removing the reward supervision signal leads to a decrease in the consistency of the results. In addition, the appendix section also compares other reward models and their corresponding downstream policy performance, providing further evidence beyond the qualitative video examples.
Critical weaknesses
The paper states that closed-loop updating requires deploying the policy to the corresponding environment; then, in real-world scenarios, would this seriously lengthen the entire update cycle? Compared with real-robot RL, how much would the efficiency be improved? In addition, the update of the video model will always lag behind the policy model; in this case, can the video model still provide reliable signals for the policy model?
The reward part lacks detailed introduction. In the paper, both step-level binary rewards and sparse rewards at the end appear; if it is a sparse reward, then how does it mutually promote with the dense video generation; if it is a step-level reward, then how is it set? And how does this reward handle the case of failure recovery, for the advantage-based GRPO algorithm in this paper?
The main claim of the paper is strong action following and physics; however, the experimental evidence provided in this paper (the alignment of whether the terminal state succeeds or not) cannot directly establish fine-grained action following. I think whether ACWM can serve as the criterion for judging policy improvement is precisely action following.
Questions for authors
What are the source and the time-annotation rules of the "ground-truth" reward? Please, for the trajectory of "successful recovery after a grasp failure", show its reward sequence and the calculation process of the GRPO advantage.
Can the boundary of ACWM be tested? How is its action-following capability for some random actions? And has there been qualitative analysis of when the policy will roll out actions outside the distribution of the video model? And what is the proportion of such cases?
5 comments
Response to Reviewer fomE
W3&Q2.
We would like to clarify that the action coverage and generalization ability of our world model mainly come from its action-conditioned pretraining on Maniskill dataset, comprising 35k video–action pairs across 23 tasks. Task-specific SANS fine-tuning then adapts it to the target scene and sharpens its distinction between success and near-success failure. Its action coverage therefore extends beyond the small target-task dataset.
To support this, we provide additional qualitative visualizations in this anonymous link (if videos are squeezed, please widen your browser window, or view directly from the assets repository). These examples condition the world model on an initial scene image with random and out-of-distribution action trajectories. The results show that, despite the stochasticity of these unseen inputs, the world model still faithfully follows the action conditioning and generates physically correct motions aligned with the correct trajectory sequence. This suggests that the pretrained world model provides meaningful action coverage beyond the offline trajectories, while SANS fine-tuning mainly improves its sensitivity to subtle success/near-success boundaries.
For actions arising during policy optimization, most are within the training distribution, given that our world model is trained using the previous policy rollouts. So although our trained world model is capable enough to deal with OOD random action sequences, the main focus of the proposed method is on improving success rate in fine-grained near-success scenarios.
Thank you for the clarification. My main remaining concern is whether the reliability of the world model's updates can be quantified. Since RL relies on sparse terminal rewards, whether the policy improves reliably depends heavily on the accuracy of the world model's predictions and reward estimates conditioned on the actions. This would help assess whether, as the policy continues to evolve, these updates genuinely reduce the model–policy mismatch.
Response to Reviewer fomE
Thank you for your constructive feedback, and for recognizing the real-robot improvements and the evidence supporting our training signals. We will address the reviewer's remaining concerns as follows.
W1.
We thank the reviewer for raising this important point. We explicitly compare the number of real-world rollouts/demonstrations required by our method and physical RL. As shown below, during the initial SFT/world-model adaptation stage, our method requires manually collecting a small number of success and near-success trajectories, typically fewer than 50 near-success trajectories per task. During the RL stage, our method only needs real-robot policy rollouts for SANS dataset augmentation. In contrast, physical online RL would require around 10,240 real-world rollouts under the same GRPO setting as ours: 8 group size × 64 batch size × 20 RL steps, which is far more expensive.
Therefore, although our method does require some real-robot data to train the world model, it substantially reduces the dominant cost of physical RL which needs large-scale online robot rollouts with resets and supervision. In addition, during World-VLA-Loop refinement, near-success data can be collected as a by-product of policy evaluation, which does not introduce much additional effort.
The lag between the world model and the improving policy is precisely what motivates our loop. A policy may 'hack' inaccurate predictions and rewards in the current world model. Physical deployment exposes these errors as failed executions, which are added to SANS dataset. Refining the world model on these trajectories helps it correct the previously missed edge cases, providing better signals for the next policy-training round. Figure 5 illustrates this process: updated-policy failures reveal a model error, and incorporating them supports more reliable subsequent behavior. The loop therefore uses deployment to identify and correct the discrepancies that policy optimization exposes.
W2&Q1.
Our ground-truth rewards are stepwise binary completion labels. During SANS curation, we manually identify the first frame at which the task is completed, denoted by ts. We assign rt=0 for t<ts and rt=1 for t≥ts; trajectories that never complete the task have all-zero labels.
These labels supervise the reward head alongside the video-generation loss. Because the reward head and video generator share the diffusion backbone, reward gradients also shape the visual representation, encouraging it to distinguish successful interactions from visually similar failures. Binary supervision therefore contributes directly to the dense video model.
Our collected successful SFT demonstrations include movements like retrying or failure recovering to find the correct grasp or interaction pose. These training data equips the policy recovery behaviors, which RL can further reinforce. Consider this illustrative pick-and-place example:
During GRPO, the trajectory reward Ri is the thresholded final completion prediction (also a binary value). The group-relative advantage is: Ai=std(R1,…,R8)+ϵRi−Rˉ.
This trajectory-level advantage reinforces the successful action sequence, including recovery. An initial failed grasp does not prevent the rollout from receiving a positive advantage when it ultimately completes the task.