acceptodds
Under review as a conference paper at ICLR 2027

World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policy

Abstract

Reinforcement learning (RL) can refine Vision-Language-Action (VLA) policies beyond behavior cloning, but real-world RL remains expensive due to extensive rollouts, resets, supervision, and safety risks. Action-conditioned video world models offer an option to train in virtual environments, yet they exhibit imprecise action following, particularly on subtle near-success failures. Besides, they lack native reward signals for RL. Computing rewards based on inaccurate visual predictions remain unreliable. We introduce World-VLA-Loop, structured around two foundational designs and a higher-level co-evolving paradigm. We first curate SANS dataset, dedicatedly mixing successful and near-success trajectories to improve action–outcome alignment. Then, we train a state-aware video world model that jointly predicts future frames and binary rewards from diffusion latents. It couples reward estimation to the generator rather than a separate module, and in turn, benefits visual prediction. Since VLA behavior shifts during RL, a fixed simulator can misalign with the updated policy, World-VLA-Loop therefore closes the loop by using the refined world model for iterative VLA post-training while feeding rollouts from each improved policy back to augment and fine-tune the world model. Across simulation and real-robot experiments, World-VLA-Loop substantially improves VLA performance while reducing reliance on costly physical interaction.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, World Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fomEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the World-VLA-Loop framework, which alternately performs video world model training and VLA policy optimization. It constructs a SANS dataset using successful and near-successful trajectories, jointly trains the video generation model and the reward prediction head, and uses the resulting world model for GRPO-based policy post-training; subsequently, it collects trajectories of the updated policy running in the real environment to further optimize the world model. The experiments cover a selection of LIBERO tasks and two real-world manipulation tasks, and conduct ablation studies on the "near-successful" data and reward modeling.

Critical strengths

The improvement of the policy was verified through execution by actual robots. After two rounds of training, the real-environment success rates on the two tasks increased from 13.3% to 50.0% and from 26.7% to 53.3%, respectively. These results provide useful preliminary evidence that training in the learned simulator can improve the policy deployed in reality. The studies on the individual components confirm the effectiveness of the proposed training signal. Removing the "near-success" data or removing the reward supervision signal leads to a decrease in the consistency of the results. In addition, the appendix section also compares other reward models and their corresponding downstream policy performance, providing further evidence beyond the qualitative video examples.

Critical weaknesses

The paper states that closed-loop updating requires deploying the policy to the corresponding environment; then, in real-world scenarios, would this seriously lengthen the entire update cycle? Compared with real-robot RL, how much would the efficiency be improved? In addition, the update of the video model will always lag behind the policy model; in this case, can the video model still provide reliable signals for the policy model?

The reward part lacks detailed introduction. In the paper, both step-level binary rewards and sparse rewards at the end appear; if it is a sparse reward, then how does it mutually promote with the dense video generation; if it is a step-level reward, then how is it set? And how does this reward handle the case of failure recovery, for the advantage-based GRPO algorithm in this paper?

The main claim of the paper is strong action following and physics; however, the experimental evidence provided in this paper (the alignment of whether the terminal state succeeds or not) cannot directly establish fine-grained action following. I think whether ACWM can serve as the criterion for judging policy improvement is precisely action following.

Questions for authors

What are the source and the time-annotation rules of the "ground-truth" reward? Please, for the trajectory of "successful recovery after a grasp failure", show its reward sequence and the calculation process of the GRPO advantage.

Can the boundary of ACWM be tested? How is its action-following capability for some random actions? And has there been qualitative analysis of when the policy will roll out actions outside the distribution of the video model? And what is the proportion of such cases?

5 comments
AuthorsOpenReview ↗

Response to Reviewer fomE

Thank you for your constructive feedback, and for recognizing the real-robot improvements and the evidence supporting our training signals. We will address the reviewer's remaining concerns as follows.

W1.

We thank the reviewer for raising this important point. We explicitly compare the number of real-world rollouts/demonstrations required by our method and physical RL. As shown below, during the initial SFT/world-model adaptation stage, our method requires manually collecting a small number of success and near-success trajectories, typically fewer than 50 near-success trajectories per task. During the RL stage, our method only needs real-robot policy rollouts for SANS dataset augmentation. In contrast, physical online RL would require around 10,240 real-world rollouts under the same GRPO setting as ours: 8 group size × 64 batch size × 20 RL steps, which is far more expensive.

MethodSFTRL
Ours80 (40 success + 40 near-success demonstrations)~88 (SANS augmentation)
Physical RL40 (success demonstrations)~10,240 (massive online rollouts)

Therefore, although our method does require some real-robot data to train the world model, it substantially reduces the dominant cost of physical RL which needs large-scale online robot rollouts with resets and supervision. In addition, during World-VLA-Loop refinement, near-success data can be collected as a by-product of policy evaluation, which does not introduce much additional effort.

The lag between the world model and the improving policy is precisely what motivates our loop. A policy may 'hack' inaccurate predictions and rewards in the current world model. Physical deployment exposes these errors as failed executions, which are added to SANS dataset. Refining the world model on these trajectories helps it correct the previously missed edge cases, providing better signals for the next policy-training round. Figure 5 illustrates this process: updated-policy failures reveal a model error, and incorporating them supports more reliable subsequent behavior. The loop therefore uses deployment to identify and correct the discrepancies that policy optimization exposes.

W2&Q1.

Our ground-truth rewards are stepwise binary completion labels. During SANS curation, we manually identify the first frame at which the task is completed, denoted by tst_s. We assign rt=0r_t=0 for t<tst<t_s and rt=1r_t=1 for t≥tst\geq t_s; trajectories that never complete the task have all-zero labels.

These labels supervise the reward head alongside the video-generation loss. Because the reward head and video generator share the diffusion backbone, reward gradients also shape the visual representation, encouraging it to distinguish successful interactions from visually similar failures. Binary supervision therefore contributes directly to the dense video model.

Our collected successful SFT demonstrations include movements like retrying or failure recovering to find the correct grasp or interaction pose. These training data equips the policy recovery behaviors, which RL can further reinforce. Consider this illustrative pick-and-place example:

EventApproachGrasp ObjectPlace in the Wrong PositionRetry and Regrasp ObjectPlace SuccessfullyTerminal state
Ground-truth reward000011

During GRPO, the trajectory reward RiR_i is the thresholded final completion prediction (also a binary value). The group-relative advantage is: Ai=Ri−Rˉstd⁡(R1,…,R8)+ϵ.A_i=\frac{R_i-\bar R}{\operatorname{std}(R_1,\ldots,R_8)+\epsilon}.

This trajectory-level advantage reinforces the successful action sequence, including recovery. An initial failed grasp does not prevent the rollout from receiving a positive advantage when it ultimately completes the task.

AuthorsOpenReview ↗

Response to Reviewer fomE

W3&Q2.

We would like to clarify that the action coverage and generalization ability of our world model mainly come from its action-conditioned pretraining on Maniskill dataset, comprising 35k video–action pairs across 23 tasks. Task-specific SANS fine-tuning then adapts it to the target scene and sharpens its distinction between success and near-success failure. Its action coverage therefore extends beyond the small target-task dataset.

To support this, we provide additional qualitative visualizations in this anonymous link (if videos are squeezed, please widen your browser window, or view directly from the assets repository). These examples condition the world model on an initial scene image with random and out-of-distribution action trajectories. The results show that, despite the stochasticity of these unseen inputs, the world model still faithfully follows the action conditioning and generates physically correct motions aligned with the correct trajectory sequence. This suggests that the pretrained world model provides meaningful action coverage beyond the offline trajectories, while SANS fine-tuning mainly improves its sensitivity to subtle success/near-success boundaries.

For actions arising during policy optimization, most are within the training distribution, given that our world model is trained using the previous policy rollouts. So although our trained world model is capable enough to deal with OOD random action sequences, the main focus of the proposed method is on improving success rate in fine-grained near-success scenarios.

Thank you for the clarification. My main remaining concern is whether the reliability of the world model's updates can be quantified. Since RL relies on sparse terminal rewards, whether the policy improves reliably depends heavily on the accuracy of the world model's predictions and reward estimates conditioned on the actions. This would help assess whether, as the policy continues to evolve, these updates genuinely reduce the model–policy mismatch.

Reviewer_apL8OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes World-VLA-Loop, a closed-loop framework that alternates between improving an action-conditioned video world model and optimizing a VLA policy within that model. Real-environment rollouts from the updated policy are added to the world model’s training data, helping it capture newly encountered behaviors and failure modes. The framework combines successful and near-success trajectories with joint video prediction and reward learning, and demonstrates policy improvements in simulation and on two real-world manipulation tasks.

Critical strengths

s1. The loop is well motivated: real rollouts from the updated policy are used to improve the world model, which then supports further policy training. This provides a practical way to reduce the gap between generated and real rollouts.

s2. Building the complete system requires substantial engineering effort. The authors bring together world model training, policy optimization, and real-world data collection, and demonstrate that the full loop works on a physical robot.

s3. Including near-success failures is a useful design choice. These examples help the world model distinguish successful execution from attempts that look similar but fail, and the ablation results support their value.

Critical weaknesses

w1. The paper does not clearly explain how the balance between successful and failed rollouts is maintained across training rounds. As the policy improves, failures may become less common. It is unclear whether this change makes the world model more likely to incorrectly predict success.

w2. The paper provides limited analysis of what goes wrong as the world model generates more action chunks. The reported decline in image quality does not explain whether the model still follows the actions correctly or predicts task success reliably. Examples from longer tasks would help establish where the method works and where it fails.

w3. It is unclear whether the rewards measure intermediate progress or only task completion. If they only indicate completion, the method may not fully use the earlier, more accurate parts of generated videos or the moments when failures occur to help the policy learn.

Questions for authors

q1. How do the authors handle changes in the proportion of successful and failed rollouts across training rounds? How much does this proportion affect the world model’s ability to distinguish success from failure?

q2. What goes wrong as the world model generates more action chunks? Does it stop following the actions accurately, lose track of objects, or incorrectly predict success? Could the authors show how these errors develop in longer tasks, including failed attempts?

q3. Do the rewards measure progress during the task, or only whether the task succeeds? If they only measure success, could the method make better use of the earlier, more accurate parts of generated videos and the moments when things go wrong to help the policy learn?

2 comments
AuthorsOpenReview ↗

Response to Reviewer apL8

Thank you for your feedback and for recognizing the motivation of the refinement loop and the value of near-success examples. We address the reviewer's three concerns below.

W1&Q1.

Thank you for raising the question. We maintain an approximately balanced curated dataset as the policy improves. As detailed in our response to W2&Q2 of the reviewer J72Z, cumulative success/failure counts are 40/40 initially, 60/60 after the first collection round, and 80/88 after the second. The training mixture therefore remains balanced even as the policy's natural failure frequency decreases. We continue collecting failure examples, which increasingly capture subtle execution errors. These updated-policy failures are especially valuable: they expose the precise behaviors that the next world-model refinement should learn to represent.

We also evaluated sensitivity to the success/failure ratio on real-world Push Cube:

Success : failureVisual alignment
4 : 380%
1 : 180%
3 : 484%

The results show that when the ratio is kept close to balanced, the world model is not very sensitive to the exact data composition. Together, we hope the maintained failure coverage and ratio ablation could address the concern that policy improvement necessarily produces a success-dominated training set.

W2&Q2.

When the video world model prediction sequence becomes unnecessarily long, especially in the current autoregressive way, we observe that:

  • Robot and object geometry becomes blurred or fragmented, and predicted positions drift. The generated scene then represents the commanded motion and object interaction less accurately.
  • Subsequent chunks condition on these imperfect predictions, propagating errors in object location and contact. This can also distort the apparent task outcome and the predicted reward.
  • The policy receives observations increasingly different from its training distribution and can produce inappropriate actions, further downgrading the overall performance.

Our appendix B.1 quantifies the accompanying degradation: PSNR falls from 23.59 at 200 frames to 20.24 at 300 frames, while SSIM falls from 0.743 to 0.628. Our evaluated LIBERO tasks typically require about 220 frames, while longer tasks can require 300–400 frames. We will continue to work on improving the method by incorporating more advanced autoregressive video model.

W3&Q3.

The reward targets are binary task-completion labels, but the head predicts a continuous scalar at each step. We observe a clear increase in this score as trajectories approach successful completion. This provides an empirical progress-related cue near success; the score is trained to predict completion, rather than a calibrated distance to the goal.

In the reported GRPO implementation, the continuous output is thresholded into a binary success signal. Earlier actions still receive learning signals through the trajectory-level advantage (due to GRPO's model update/learning process), and joint reward supervision trains the world-model representation alongside video prediction. The method therefore already learns from the earlier portions of successful trajectories without requiring manually annotated intermediate subgoals.

Please also refer to our response to W2&Q1 of the reviewer fomE, with a more detailed analysis on how our reward works under GRPO setting.

Thank you for the thoughtful response. I appreciate the authors’ candid and measured discussion of the strengths and limitations of their work. The clarifications are helpful, and I will maintain my positive rating.

Reviewer_J72ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

World-VLA-Loop studies reinforcement learning of VLA policies inside action-conditioned video world models. It introduces SANS, a dataset containing successful and near-success trajectories, and jointly trains video generation and reward prediction using diffusion latents. The learned simulator supports GRPO-based policy optimization, while subsequent physical policy rollouts provide additional data for world-model refinement. Experiments on selected LIBERO tasks and two real-world manipulation tasks demonstrate improvements over the initial supervised policy.

Critical strengths
  • The paper targets an important failure mode of video world models. Hallucinated success under slightly incorrect actions is particularly problematic when generated observations and rewards drive policy optimization. Focusing on near-success failures is well motivated and relevant to precise manipulation.

  • The joint video-and-reward model is a practical design with supporting ablations. Predicting rewards from shared diffusion features simplifies the simulator interface. Table 4 and Appendix B provide evidence that reward supervision benefits outcome prediction and that the integrated reward head can support downstream policy learning.

  • The evaluation includes actual robot deployment. The improvements are not limited to rewards measured inside the learned simulator. Physical evaluations show that some benefits transfer to real execution, and the qualitative analysis openly identifies reward exploitation during the first iteration. This provides useful insight into both the potential and the limitations of the approach.

Critical weaknesses
  • Important method-level comparisons are missing. WMPO [1] and VLA-MBPO [2] are directly relevant baselines for world-model-based VLA reinforcement learning. WMPO already performs on-policy GRPO using pixel-space imagined trajectories, while VLA-MBPO addresses joint visual/reward modeling and compounding errors through chunk-level branched rollouts. I acknowledge that Appendix C.2 evaluates a WMPO-style reward model and replaces the proposed reward head with it during RL. However, this is a component substitution within the authors’ pipeline, not an end-to-end comparison with WMPO. No comparison with VLA-MBPO is provided. The paper therefore does not establish whether its complete framework improves over existing approaches under comparable policy initialization, task data, physical-interaction budgets, and optimization budgets. These comparisons are important for assessing the contribution beyond improvements over SFT.

  • The benefit of closed-loop refinement is not separated from the benefit of additional data. Section 4.3 adds physical rollouts from the improved policy before the next iteration, while reinitializing the world model from the ManiSkill checkpoint and the policy from the base SFT checkpoint. This demonstrates that retraining with an expanded dataset can help, but does not isolate the value of the proposed iterative collection strategy. A matched-data comparison with additional trajectories from the original policy, or another collection strategy, would clarify this point. The total number of physical trajectories collected at each iteration is also insufficiently specified. Appendix A.4 reports compute costs, but without complete interaction accounting and a matched-budget control, the claimed reduction in costly physical interaction remains difficult to assess.

  • The evaluation scope and uncertainty reporting limit the strength of the conclusions. Table 3 presents two selected tasks from each of three LIBERO suites, whereas the oracle numbers are described as averages over entire suites. These are not directly matched evaluations. The real-world study covers only two tasks with 30 physical evaluation rollouts, and the reported results lack confidence intervals or variation across independent training runs. The second iteration improves both tasks by approximately 13.3 percentage points, corresponding to only four additional successes if evaluated over 30 trials. These are encouraging observations, but stronger statistical evidence is needed to establish reproducible closed-loop gains. The omission of long-horizon evaluation due to model drift further limits the claimed broader applicability.

  • Outcome agreement does not sufficiently establish reliable action-conditioned simulation under policy optimization. Table 2 measures success/failure agreement on 50 samples per task, but aggregate accuracy does not reveal the false-positive rate on near-success failures—the central failure mode motivating the paper. Moreover, agreement between the generated video and a reward head sharing the same backbone can coexist with both predictions being wrong about physical execution. The qualitative reward-hacking example makes this concern concrete. The paper should report success/failure confusion statistics, clarify the annotation and held-out evaluation protocol, and assess alignment on trajectories produced by successive policy checkpoints. A size-matched comparison between near-success failures and ordinary failure data would also help establish whether SANS’s benefit specifically comes from near-success examples.

[1] Fangqi Zhu, Zhengyang Yan, Zicong Hong, Quanxin Shou, Xiao Ma, and Song Guo. WMPO: World Model-based Policy Optimization for Vision-Language-Action Models. arXiv preprint arXiv:2511.09515, 2025. https://arxiv.org/abs/2511.09515

[2] Zhilong Zhang, Haoxiang Ren, Yihao Sun, Yifei Sheng, Haonan Wang, Haoxin Lin, Zhichao Wu, Pierre-Luc Bacon, and Yang Yu. Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models. arXiv preprint arXiv:2603.20607, 2026. https://arxiv.org/abs/2603.20607

Questions for authors
  • Are any end-to-end comparisons with WMPO or VLA-MBPO available?

  • How many additional physical trajectories are collected in each iteration, and are they also available to the DPO or other baselines? Is there an existing control that matches the total data budget without iterative collection?

  • Which LIBERO tasks are evaluated, and can oracle results be reported on those same tasks? How many independent training runs underlie the physical results, and are confidence intervals available?

  • How are visual outcomes annotated, and how are evaluation trajectories separated from world-model training data? Are false-positive rates on near-success failures and alignment results for updated-policy rollouts available?

3 comments
AuthorsOpenReview ↗

Response to Reviewer J72Z

Thank you for the constructive feedback. And we are glad that the reviewer find our joint-prediction design and real-robot experiments meaningful. Here we will address the remaining concerns as follows:

W1&Q1.

Our method addresses the fine-grained errors that make imagined rewards unreliable. WMPO uses a separate video-based reward model while our method's joint reward supervision also trains the video generator, making more accurate predictions. In Appendix C.2, our head achieves 78% versus 58% reward alignment on generated trajectories, and 53.3% versus 13.3% physical Push Cube success under the reward substitution. These controlled results have already compared our method with WMPO's core components, specifically demonstrating the practical benefit of our reward design.

While VLA-MBPO jointly predicts observations and rewards and reduces error accumulation through short branched rollouts, our focus is correcting subtle action–outcome errors and refreshing the model with failures exposed by updated policies. What's more, World-VLA-Loop focuses on the refinement loop between robot policy and world model, which could be applied to other methods including WMPO and VLA-MBPO. We will add the discussion in the final manuscript.

W2&Q2.

Thank you for paying attention to detailed data budget. The collection budget per task is:

StageAdded successesAdded near-success failuresCumulative total
Base404080
Iteration 12020120
Iteration 22028168

The initial 40 successes also train the SFT policy. DPO receives the complete dataset of 80 successes and 88 failures. With this equal data budget, our method reaches 53.3% versus DPO’s 33.3% on Push Cube, a 20-percentage-point gain supporting world-model-based policy optimization beyond direct offline preference learning. Besides, we collect another 50 rollouts for world model evaluation.

For 64 groups of eight trajectories over 20 GRPO steps, direct physical RL would require 10,240 human-in-the-loop rollout slots per RL round. Our optimization uses synthetic rollouts for those slots, avoiding the physical execution and resets.

W3&Q3.

We would like to clarify details on our evaluation results. First, the descriptions of evaluated LIBERO tasks could be seen in the Figure 7 and Figure 8, and we will also formally add task descriptions in the supplementary. Second, the oracle row provides a suite-level reference, as stated in the caption, and is not directly compared with our results. Our improvement claim primarily comes from comparing SFT and RL on the same six tasks, yielding gains of 6.1–24.0 percentage points.

Each reported number has 1 train run and 30 physical trials per task. Grouping these into three batches of ten gives mean ± sample SD (%):

CheckpointPlace CupPush Cube
SFT13.3 ± 5.826.7 ± 11.5
Round 136.7 ± 5.840.0 ± 10.0
Round 250.0 ± 10.053.3 ± 5.8

These summarize evaluation-batch variability for one trained policy per condition. Both tasks improve through both rounds and uncertainty level is within accpetable range.

W4&Q4.

Thank you for your detailed feedback. We will answer the questions one by one.

First, visual outcomes are manually annotated for task completion (please also refer to our response to W2&Q1 of the reviewer fomE, with a more detailed clarification on our reward design). Evaluation episodes are excluded from training and checkpoint selection and follow the same environment and policy/teleoperator collection protocol.

Second, We find that for the trained video world model, visual and reward predictions agree on over 96% of evaluated trajectories, supporting their internal consistency. Our Table 2 separately compares each prediction with the physical execution outcome, providing an external correctness check.

Third, SANS dataset targets small pose and contact errors that separate apparent success from actual completion. All observed failures in our collection fall into this category, making these examples directly relevant to the policy’s remaining errors. Our table 4 also shows that including them raises visual alignment from 60%/66% to 92%/90% on the two Object tasks. Only trained with ordinary failures data cannot expose such fine distinction, therefore they are bound to result in a weaker and more inaccurate world model.

Thank you for the clarifications. While the matched-data DPO comparison is helpful, it does not isolate the benefit of closed-loop data collection: this requires comparing the same world-model-based RL pipeline under equal data budgets, with and without iterative policy-guided collection. Similarly, replacing the reward head with a WMPO-style module is a component ablation rather than an end-to-end comparison with WMPO. Finally, splitting 30 evaluation trials from a single training run into three batches does not measure variability across independent training runs, so the claim that uncertainty is “within an acceptable range” remains insufficiently supported. These clarifications therefore do not resolve my main concerns.

Thank you for the follow-up. We will address the three points below.

1. Matched data comparison

Following the reviewer's suggestion, we ran the requested control on the real-world Push Cube task: the same world-model-based RL pipeline with the same total data budget, with and without iterative policy-guided collection.

  • Closed loop (ours): the 80 initial trajectories (40 successful + 40 near-success), plus 88 trajectories (40 successful + 48 failed) collected from the updated policies in iterations 1 and 2.
  • Matched-data control: the same 80 initial trajectories, plus 88 trajectories (40 successful + 48 failed) collected from the original SFT policy, with no iterative collection.

The new experiment trains once on its 168 trajectories with identical initializations, the same GRPO configuration and the same physical evaluation setting. The only difference between the two settings is which policy collected the additional 88 trajectories.

Source of the augmented trajectoriesPush Cube success
Original SFT policy (no iteration)36.7%
Updated policies (closed loop, ours)53.3%

With the same data budget, collecting from the updated policies gives an additional 16.7% improvement. This is consistent with our motivation. Failures of the improved policy expose the specific errors that the current world model mispredicts. These are the cases the next refinement must correct, and the SFT policy's rollouts do not reliably reveal them. We will add this experiment to the final version.

2. Comparison with WMPO

We agree that an end-to-end comparison would be informative, and we carefully examined whether one is feasible. Two issues prevent a fair comparison within the rebuttal period:

  • No shared benchmark. WMPO's simulated experiments use four MimicGen tasks rather than LIBERO, so no published WMPO results exist on our tasks.
  • WMPO's world model uses an OpenSora-based video diffusion backbone pretrained on Open X-Embodiment. Ours is pretrained on our ManiSkill action-video dataset. A fair comparison would require re-pretraining one of the two models on matched data, then adapting its world model and policy to our SANS data and real-robot setup. Otherwise, any difference would be confounded by backbone and pretraining-data effects rather than reflect the methods themselves. This exceeds the time and compute available during the rebuttal period.

For this reason, our Appendix C.2 experiment isolates the main methodological difference from WMPO, holding everything else fixed: a separate reward model versus reward supervision that also trains the video generator. Under this control, our design improves reward alignment and physical Push Cube success rate.

3. Variability across training runs

We agree with the reviewer that splitting 30 trials from one training run into batches measures evaluation variability instead of variability across training runs. Our physical results come from one training run per condition, each evaluated over 30 trials. However, this follows common practice for real-robot evaluation in closely related work. WMPO reports real-robot success over 30 trials per policy (Section 4.6), and π0.5\pi_{0.5} performs 10 evaluations per task for each policy (Appendix B). Neither reports variance across independent training runs. We will state our evaluation protocol explicitly in the final version.

We thank the reviewer again for the detailed engagement. We hope the matched-data experiment, together with the clarifications above, addresses the main concerns. We would be grateful if the reviewer would take them into account in the final assessment.

Comments on the paper

1 comment

Dear Reviewers, 

We’re about halfway through the discussion period. Please read the authors’ rebuttal and assess whether it adequately addresses the key concerns raised in your review. We encourage you to leave a follow-up comment for the authors, indicating which concerns have been resolved and whether any important issues remain or require further clarification. Please update your review and scores as appropriate in light of the rebuttal and discussion.

Thank you for your time and participation. 

Best regards, 

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.