acceptodds
Under review as a conference paper at ICLR 2027

WEAVER, Better, Faster, Longer: An Effective World Model for Robotic Manipulation

Abstract

The potential impacts of world models (WMs, i.e., learned simulators) on robotics are far-reaching—policy evaluation, policy improvement, and test-time planning—all with limited real-world interaction. To unlock these downstream capabilities, a WM needs to jointly satisfy three desiderata: *(i)* fidelity (i.e., producing simulated trajectories that correlate with reality), *(ii)* consistency (i.e., producing simulated trajectories that are coherent over long horizons), and *(iii)* efficiency (i.e., producing simulated trajectories quickly). We propose (World Estimation Across Views for Embodied Reasoning): a WM architecture that simultaneously achieves all three desiderata, providing state-of-the-art results on robotic manipulation tasks. is a multi-view WM trained to predict future latents and reward values via a flow-matching loss. We distill the key design decisions across model architecture, memory, and prediction objectives required to unlock the kinds of long-horizon dynamic manipulation tasks that have confounded prior world modeling approaches. We apply in robotic hardware, demonstrating its effectiveness at policy evaluation ( correlation with real-world success rate), policy improvement (real-world success rate improvement of % on top of the robot foundation model), and test-time planning (real-world success rate improvement of % made possible by upto inference speedup over prior WMs). also demonstrates better performance than prior WMs when evaluated on out-of-distribution scenarios. Videos at https://weaver-world-model.github.io.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mxzQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WEAVER, a multi-view world model for robotic manipulation that combines flow matching, diffusion forcing, memory, and latent reward prediction. The model achieves improved generation quality and inference efficiency compared with existing world models. Real-robot experiments demonstrate its effectiveness in policy evaluation, synthetic-data-based policy improvement, and test-time planning.

Critical strengths

WEAVER achieves substantially better generation quality and faster inference than existing world models, making it more practical for robotic manipulation and planning. The paper goes beyond video generation metrics and demonstrates improvements in policy evaluation, policy fine-tuning, and test-time planning on real hardware.

Critical weaknesses

Although the paper evaluates several design choices, the contributions of sparse memory, diffusion forcing, and joint multi-view prediction are not sufficiently isolated. This makes it difficult to determine which components drive the improvements in long-horizon consistency. The reported end-to-end planning latency appears difficult to reconcile with the claimed control frequency. More details about how planning is integrated with real-robot execution are needed to assess its practical real-time performance.

Questions for authors

Section 5.2.3 uses four candidate action chunks with a horizon of 12 steps, while Appendix A5.4 reports approximately 1.45 seconds of planning latency. How is the claimed 10.3 Hz control frequency achieved? Is planning performed asynchronously with robot execution? Could the authors provide ablations removing sparse memory and diffusion forcing separately? This would help clarify their respective contributions to long-horizon prediction quality. For test-time planning, could the authors compare WEAVER-based action selection with a direct action-value baseline trained using the same reward supervision? This would help isolate the benefit of world-model imagination.

3 comments
AuthorsOpenReview ↗

Response to Reviewer mxzQ (Part 1)

We thank the reviewer for the valuable feedback. The reviewer highlights the generation quality and faster inference than existing world models, and demonstrated performance gains on downstream applications. We address the concerns below:

Q1 How is the claimed 10.3 Hz control frequency achieved?

Thank you for identifying this inconsistency. Our test-time planning uses four candidate action chunks with a 15-step horizon; the reference to 12 steps is a typographical error that we will correct. The reported 10.3 Hz refers to world-model generation throughput, calculated by dividing the 15-step prediction horizon by the 1.4547-second world-model inference time. Including policy inference and reward/critic scoring, a complete planning call takes approximately 1.65 seconds, corresponding to 9.1 action steps per second of planning throughput. Neither figure represents the frequency of separate replanning calls or the robot’s closed-loop control frequency. We will correct the terminology in the manuscript to distinguish computational throughput from execution and replanning frequency.

Q2. Is planning performed asynchronously with robot execution?

No. Our deployment uses synchronous planning and execution. We evaluate the four candidate chunks, execute the selected 15-step chunk, and then replan using updated real observations. Planning and execution do not overlap, so the interval between replanning calls includes both the approximately 1.65-second planning computation and the selected chunk’s execution time. All reported real-robot improvements were obtained with this synchronous implementation. At the same four-candidate, 15-step setting, WEAVER reduces world-model inference time from 29.42 seconds to approximately 1.45 seconds compared with Ctrl-World, achieving a roughly 20-fold speedup.

Q3: Could the authors provide ablations removing sparse memory and diffusion forcing separately?

We have added more discussion on the design choice below:

We thank the reviewer for pointing this out. We provide the ablations below:

Memory and Diffusion Forcing: In the table below and in table 8 of the updated paper, we observe that WEAVER outperforms the variants that do not use (top row) sparse memory and history and (middle row) Diffusion Forcing. Note that for the model trained without Diffusion Forcing, we used the same noise schedule for each future state during inference. Our experiment presents that WEAVER benefits from both memory and Diffusion Forcing.

Table IV Ablation Analysis of WEAVER on Droid & OOD Dataset

DROID

MethodExterior LPIPS ↓Exterior FID ↓Exterior FVD ↓Wrist LPIPS ↓Wrist FID ↓Wrist FVD ↓
WEAVER w/o memory0.14814.1331.110.39328.05121.44
WEAVER w/o Diffusion Forcing0.12611.3629.380.37824.4399.28
WEAVER0.11710.2027.830.37121.5090.72

Task Data (OOD)

MethodExterior LPIPS ↓Exterior FID ↓Exterior FVD ↓Wrist LPIPS ↓Wrist FID ↓Wrist FVD ↓
WEAVER w/o memory0.16826.69103.550.37834.50238.24
WEAVER w/o Diffusion Forcing0.16225.4798.760.37434.08226.11
WEAVER0.15523.9588.270.36430.77184.62

SPRINT. We additionally evaluate WEAVER without SPRINT on BridgeV2 in Table 11 in the updated paper and table below. In this work, we incorporated SPRINT to reduce the computational cost (training FLOPS) of world-model training. Surprisingly, similar to the SPRINT paper on image generation, we observe that SPRINT improves the model performance for world models (table of experiment on BridgeV2 dataset is below).

Table V Ablations of WEAVER with SRPINT on Bridgev2 Dataset

ModelFID ↓FVD ↓LPIPS ↓
No-sprint14.0338.870.2095
no-proprio10.9322.110.1864
SVD Encoder22.2459.240.2059
WEAVER9.5719.080.1845
AuthorsOpenReview ↗

Response to Reviewer mxzQ (Part 2)

Q4. For test-time planning, could the authors compare WEAVER-based action selection with a direct action-value baseline trained using the same reward supervision? This would help isolate the benefit of world-model imagination.

Thank you for this suggestion. We add a direct action verifier trained using the same RoboMeter reward labels as WEAVER’s reward model. The verifier takes the current observation, language input and a 15-step candidate action chunk as inputs and directly predicts a scalar reward score, without generating future observations or performing world-model rollouts.

At test time, we sample four candidate action chunks from the same base policy, score them with the verifier, and execute the highest-scoring candidate. Both methods use the same candidate count, action horizon, and execution/replanning protocol. This comparison tests whether explicit world-model imagination provides additional benefits over direct action-conditioned reward prediction under matched reward supervision and candidate budgets.

Table VI. Real-world task success rates (%) for the base policy, WEAVER-FT planning, and the direct action verifier.

TaskBase policy (π₀.₅)WEAVER-FT planningDirect action verifier
PnP Marker305030
Pour Beans305035
PnP Towel405545
Stack Bowls606565
PnP Bag607060
Average445847

WEAVER-FT planning achieves 58% mean success, outperforming the direct action verifier (47%) by 11 percentage points. Direct verification improves over the base policy by 3 percentage points, while WEAVER-FT planning provides a 14-point improvement, outperforming the verifier on four tasks and matching it on Stack Bowls. These results support the value of explicitly imagining future action consequences for candidate selection, beyond directly predicting reward scores under the same RoboMeter supervision and candidate budget.

We hope our response clarified the concerns and we are happy to answer any further questions.

Best Regards,

The Authors

AuthorsOpenReview ↗

Reminder to Reviewer mxzQ

Dear Reviewer,

Thank you for your positive assessment of our work and your thoughtful feedback. As the discussion period ends today, we would be grateful if you could review our responses and let us know if any questions or concerns remain. We sincerely appreciate your recognition of our contributions and the time you have devoted to reviewing our paper.

Best regards,

The Authors

Reviewer_EKKtOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper combines designs from existing works, including Dreamer-v4, Ctrl-World, and WorldGym, to propose a world model for robotic manipulation called WEAVER. WEAVER incorporates multi-view prediction, memory, and flow matching to predict future visual and robot states conditioned on given actions. It also uses latent reward and critic heads to score trajectories and improves generation efficiency through inference acceleration. The paper evaluates WEAVER on three downstream tasks, including policy evaluation using generated trajectories, policy improvement through selecting high-value synthetic data, and test-time planning using Best-of-N to select candidate actions. The experiments demonstrate improvements over baselines in video generation quality and inference speed, a correlation between world-model-estimated and real-world policy success rates in policy evaluation, and improved real-robot success rates through fine-tuning with synthetic data and test-time planning.

Critical strengths

S1: This paper proposes WEAVER by combining designs from existing world models and evaluates it on three downstream tasks, including Policy Evaluation, Policy Improvement, and Test-time Planning. The experiments demonstrate improvements in video generation quality, inference efficiency, and real-world robot task performance.

Critical weaknesses

W1: Unclear Motivation. In L39–53, the authors argue that Fidelity, Consistency, and Efficiency are necessary requirements shared by the three downstream applications and state that no existing robot WM satisfies all three desiderata in tandem. However, the paper does not clearly explain why existing methods cannot satisfy these three requirements simultaneously. In L45–53, the authors mention that JEPA-style models have latents that may not map back to images, but this does not establish that such models fail to satisfy the requirements of fidelity, consistency, or efficiency.

W2: Unclear Novelty. According to L71–81 and L99–118, the authors adopt the multi-view prediction and memory designs from [1], combine them with flow matching and Diffusion Forcing from [2] and [3], and incorporate the latent reward and value heads from [4], together with an efficient 2D transformer architecture, to construct WEAVER. In particular, Dreamer-v4 [4] has already integrated Flow Matching, Diffusion Forcing, an efficient 2D transformer architecture, and reward, value modeling into a single framework and demonstrated robotic action-conditioned video generation. In addition, WorldGym [5] has already adopted a pretrained SD3 VAE, Diffusion Forcing, and a spatiotemporal Transformer for robotic action-conditioned video generation. However, the paper does not sufficiently explain the substantive differences from these existing designs or why their combination is necessary. It also remains unclear how these designs jointly satisfy the three proposed desiderata. The overall design is entirely based on existing works and does not demonstrate the necessary novelty of WEAVER. In L161–250 further shows that the authors mainly adopt existing world model architectures, training objectives, inference acceleration techniques, and policy optimization designs, without sufficiently explaining why combining these existing designs constitutes a new methodological contribution.

W3: Evaluation and Claim Mismatch. In L35–37, the authors emphasize that WEAVER can reduce real-robot interactions. However, L231–232 explicitly state that real recorded action trajectories are used, and Appendix L752–755 further specifies that policy evaluation directly uses real joint positions. In L392–412, the authors claim to validate offline policy evaluation. However, such open-loop trajectory prediction based on real rollouts of the policy being evaluated does not demonstrate WEAVER's ability to estimate policy performance without executing the target policy, and does not provide evidence that WEAVER can reduce the number of costly rollouts in the real environment.

W4: Unclear Method Definition.

  1. In L189–197, the authors claim to use Diffusion Forcing, but the optimization objective only presents standard Flow Matching without fully defining the actual optimization objective after incorporating Diffusion Forcing.
  2. In L166–172 and Appendix Table 2, the authors explicitly state that the model predicts future proprioceptive states and report a Proprioceptive State loss scale of Lq=0.1L_q = 0.1. However, L189–197 presents the only optimization objective for the World Model, which does not include LqL_q. Therefore, the complete optimization objective remains unclear.

W5: Insufficient Comparison. In L113–118, the authors explicitly identify Ctrl-World [1] and Dreamer-v4 [4] as the closest works to WEAVER. In particular, compared with Ctrl-World, WEAVER is more closely related to Dreamer-v4 in its core designs, including Flow Matching, Diffusion Forcing, an efficient 2D transformer architecture, and latent reward/value modeling. Dreamer-v4 has also already demonstrated robotic action-conditioned video generation. However, in Section 5.1, the authors mainly compare WEAVER with Ctrl-World and Cosmos3-Nano, without providing a direct comparison with Dreamer-v4 or sufficient ablations to distinguish the performance contributions of existing designs from those of the newly introduced WEAVER components. Therefore, the current experiments do not sufficiently demonstrate WEAVER's actual improvements in Fidelity, Consistency, and Efficiency over these closely related baselines.

W6: Unclear Critic Learning. In L215–225, the authors train the Reward Head using reward annotations from RoboMeter and train the Critic using λ\lambda-returns constructed from R(zt,ℓ)R(z_t,\ell) and V(zt+1,ℓ)V(z_{t+1},\ell). However, Appendix L801–809, L834–845 only describes the reward annotation process for real trajectories and does not clearly specify whether the Reward Head and Critic are trained on latent states generated by the World Model. Furthermore, L235–249 directly uses R(z^t+j,ℓ)R(\hat z_{t+j},\ell) and V(z^t+H,ℓ)V(\hat z_{t+H},\ell) to score imagined rollouts in both Policy Improvement and Test-time Planning. During planning, WM-generated latents and latents obtained by encoding real observations may not follow the same distribution, but the authors do not explain how the reliability of these Reward and Value estimates is ensured. Although Appendix Tab. 13 evaluates the ranking accuracy of imagined reward, it does not independently validate the accuracy of the terminal Critic or its actual contribution to action selection.


[1] Guo et al. Ctrl-World: A Controllable Generative World Model for Robot Manipulation. (ICLR 2026)

[2] Lipman et al. Flow Matching for Generative Modeling. (ICLR 2023)

[3] Chen et al. Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion. (NeurIPS 2024)

[4] Hafner et al. Training Agents Inside of Scalable World Models. (arXiv 2025)

[5] Quevedo et al. WorldGym: World Model as an Environment for Policy Evaluation. (ICLR 2026)

Questions for authors

Q1: In Fig. 1, what are the scales and evaluation metrics for the three axes? How are the model positions quantitatively determined? For the three comparisons at different time steps in the middle of the figure, how is consistency measured?

Q2: In L189–197, the authors present a standard Flow Matching objective but do not fully define the actual optimization objective after incorporating Diffusion Forcing. What is the actual optimization objective used in the paper?

Q3: In L166–172, the authors explicitly state that the model predicts future proprioceptive states. Appendix Tab. 2 also reports a Proprioceptive State loss scale of Lq=0.1L_q = 0.1, indicating that an additional loss scale is used for training proprioceptive states. However, L189–197 only presents a Flow Matching optimization objective. Where does LqL_q appear in the optimization objective, and how are the proprioceptive states trained and predicted?

Q4: In L220–250, the authors define and use an advantage estimate. However, for all candidate rollouts starting from the same state ztz_t, V(zt,ℓ)V(z_t,\ell) is constant. Therefore, the so-called advantage-based selection in Test-time Planning is exactly equivalent to selecting the best candidate according to its bootstrapped predicted return. Why is an advantage estimate necessary here? Why not directly use the bootstrapped predicted return?

Q5: According to Appendix Tab. 8, Tab.9, both WorldGym and WEAVER use the SD3 VAE, but WorldGym achieves an FVD of 68.08, whereas WEAVER achieves 19.08. Moreover, simply replacing the SD3 Encoder with the SVD Encoder in WEAVER increases FVD from 19.08 to 59.24. How can the contributions of the Visual Encoder and the World Model Architecture or Training Procedure to the improvement in generation quality be distinguished? In addition, what specific training and evaluation settings were used to obtain the WorldGym results in Tab. 8? Do they use the same BridgeV2 test set, prediction horizon, and FVD computation protocol as WEAVER?

Q6: In L463–475, the authors state that Test-time Planning uses Best-of-N@4 with h = 12 and achieves a control frequency of 10.3 Hz. However, according to Appendix Tab. 7 and A5.4, the full planning latency is approximately 1.45 s for h = 12 and 1.65 s for h = 15. What exactly does the reported 10.3 Hz refer to? Furthermore, according to L313–315, 15 action steps correspond to 1 s. If the real robot uses the same execution frequency, an action chunk with h = 12 would take only 0.8 s to execute, while h = 15 would take only 1 s, both shorter than the full planning latency. During real-robot execution, how many action steps are executed at a time, when is replanning performed, and are planning and action execution asynchronous? How do the authors achieve continuous online control with such planning latency?

10 comments
AuthorsOpenReview ↗

Response to Reviewer EKKt (Part 1)

We are glad that the reviewer appreciated the improvements in video generation quality, inference efficiency, and real-world robot task performance. We present the response below:

W1 Motivation

Our motivation is that a world model should satisfy these properties jointly: (i) fidelity, (ii) consistency, and (iii) efficiency. To provide a detailed comparison with prior methods, we added Table 2 in the revised paper that summarizes their architectural choices and demonstrated downstream applications. Among the reported configurations compared, WEAVER uniquely combines multi-view prediction, robot proprioceptive-state prediction, memory, and expected-return estimation. Together, these capabilities support policy evaluation, policy improvement, and test-time planning within a single world model. We further highlight that the latency reductions needed for practical planning come from carefully selected inference techniques, including KV caching, noise schedules, and rectified-flow post-training, whose effects on generation quality and efficiency we quantify through ablations.

W2: Unclear Novelty

We agree that DreamerV4 and WorldGym adopt several related techniques. Both methods themselves build on established ideas from video generative modeling, including Diffusion Forcing and efficient Transformer architectures. However, their proposed world models operate in single-view settings: WorldGym focuses on policy evaluation, while DreamerV4 demonstrates video generation without evaluating or improving pretrained robotics foundation models on physical hardware. Neither demonstrates test-time planning with pretrained VLAs on real robots. Their world model predicts action-conditioned video without jointly generating synchronized exterior and wrist views and future proprioceptive states. These are the observation inputs required to repeatedly query the multi-view, proprioception-conditioned VLA in our imagined rollouts. Consequently, directly applying their world models to our setting would leave required observation inputs missing; extending them requires jointly modeling these streams and maintaining consistency across camera views and robot states. WEAVER addresses this through joint multi-view and proprioceptive prediction.

AuthorsOpenReview ↗

Response to Reviewer EKKt (Part 2)

We have added Table 2 to the revised paper to distinguish the architectural choices and demonstrated downstream applications of these methods relative to WEAVER. Our contribution lies in integrating and adapting these foundations to jointly satisfy (i) fidelity, (ii) long-horizon consistency, and (iii) efficiency, enabling effective policy evaluation, policy improvement, and test-time planning with a single world model. In addition to ablations on design choice and in contrast to prior methods, we ablate inference-time noise schedules, KV caching, and rectified-flow post-training to quantify their effects on generation quality and latency.

Regarding how the proposed design choices jointly satisfy the three desiderata, our experiments demonstrate:

  • Fidelity and efficiency: With 928M parameters, WEAVER achieves a better generation-quality vs latency tradeoff than Ctrl-World (1.5B parameters) and Cosmos3-Nano (14B parameters) in our evaluations (Table 1). It also outperforms IRASim and WorldGym on BridgeV2, demonstrating effectiveness beyond DROID (Table 10).

  • Long-horizon consistency: Our FID and FVD comparisons across prediction horizons show that WEAVER maintains better generation quality over extended rollouts than the evaluated baselines. Together with our memory and Diffusion Forcing ablations, these results support the effectiveness of its temporal conditioning and training design (Fig. 4 & 12).

  • Test-time planning: Prior work has demonstrated test-time planning using compact, single-task small world models whose latent rollouts are fast to generate. More recent large-scale generative robot world models have improved visual prediction and policy evaluation, but these models are very expensive to generate and therefore have not been practical in test-time planning with pretrained VLAs. WEAVER addresses this gap by combining ReFlow post-training to generate accurate predictions with fewer denoising steps, KV caching to reuse memory and history computations, and latent reward and critic heads to score candidate futures without image decoding or external VLM queries. Together with its efficient spatio-temporal architecture, these choices enable our 928M-parameter world model to retain high generation quality while running efficiently enough for deployment-time action selection, improving pretrained VLA performance on real hardware (Fig. 8).

Our results demonstrate how improvements in prediction quality, temporal consistency, and inference efficiency translate into downstream policy gains. We elaborate on the contributions below:

Firstly, WEAVER contributes a joint generative model of multi-view observations and robot proprioceptive states. Visual and proprioceptive tokens share a spatiotemporal Transformer conditioned on actions, recent history, and sparse memory. Joint supervision makes the robot state an explicitly predicted component of the dynamics. Our BridgeV2 proprioception ablation reports FVD scores of 19.08 and 22.11 across the two variants.

Secondly, we incorporate efficient, task-conditioned reward and value estimation. We distill RoboMeter’s progress supervision into a lightweight latent reward head, while the critic estimates returns beyond the imagined horizon. Together, these heads support advantage-based synthetic-data selection for policy fine-tuning and candidate-action selection for test-time planning, without decoding latents into images or querying an external VLM judge for scoring.

Thirdly, our experiments validate the evaluation module and its downstream benefits. Imagined-rollout scoring achieves 74.7% ranking accuracy versus 62.3% for Ctrl-World plus RoboMeter. On PnP Marker, replacing reward-sum filtering with critic-based advantage filtering improves real-robot success after fine-tuning from 35% to 50%, using 1,000 synthetic segments per condition. Overall, synthetic-data augmentation raises mean policy success from 71% to 82% at a fixed real-data budget, while test-time selection yields a 14% improvement, demonstrating the practical benefits of the integrated framework.

Lastly, we systematically validate both the individual design choices and the complete framework. The original paper includes ablations of proprioceptive state prediction and the visual encoder. During rebuttal, we added ablations of Diffusion Forcing, memory, and SPRINT. In contrast to the prior methods, we also examine inference-time choices like KV caching, noise schedules, and rectified-flow fine-tuning. Beyond these component studies, we jointly demonstrate three downstream applications within a single world model– policy evaluation, policy improvement, and test-time planning (discussed in Table 2 of the revised paper).

AuthorsOpenReview ↗

Response to Reviewer EKKt (Part 3)

W3 Evaluation and Claim mismatch

We would like to clarify the setting used for policy evaluation and improvement, and how WEAVER generates additional training experience to reduce the need for real-world data collection.

Sec. 5.2.1 follows a common protocol for world-model-based policy evaluation: open-loop replay of recorded policy actions. WEAVER receives only the initial joint configuration, initial observations, and a fixed sequence of actions, and autoregressively predicts subsequent observations and states. Holding actions fixed enables a controlled comparison across world models, with corresponding real-world executions providing the ground truth for evaluating predicted policy outcomes. Although the actions are provided, WEAVER must still predict the resulting object interactions and task success or failure; no intermediate ground-truth observations, joint states, or task outcomes are supplied to the model.

This use of recorded actions and paired real-world outcomes is established in prior world-model evaluation. VLAW [1] replays recorded action sequences to assess agreement between predicted and real interaction successes and failures. StressDream [2] evaluates whether imagined rollouts conditioned on initial observations and action sequences identify failures observed in real manipulation trajectories. DreamDojo [3] likewise compares simulated and real rollouts from matching initial frames to assess agreement in policy success rates and rankings. Our evaluation follows this established principle: assessing whether a world model faithfully reproduces the outcomes of policy executions under controlled inputs.

For closed-loop policy interaction in the world model, Sec. 5.2.2 uses closed-loop interaction for synthetic data generation. Starting from the initial observations and joint configuration, the policy proposes an action chunk, WEAVER predicts the resulting observations and states, and the policy uses these predictions to generate the next action chunk. This interaction produces new 36-step synthetic segments without physically executing them or requiring corresponding recorded real trajectories, demonstrating the WEAVER’s capability to rollout with policy in closed-loop. In our experiments, adding these synthetic segments increases mean success from 71% to 82% without increasing the real-data budget for policy fine-tuning, directly demonstrating that experience generated within WEAVER improves physical robot performance by reducing real world interaction data.

W4, Q2, Q3: Unclear Method Definition

We thank the reviewer for highlighting these omissions. We have expanded the training-objective description and explicitly defined the full objective in Eq. (2), L226–229.

**1.**We now define the noising process using an independently sampled flow time for each future timestep, allowing different noise levels across the future sequence. This incorporates Diffusion Forcing within flow matching and the model minimizes the MSE between predicted and target velocities under this per-step noise conditioning.

2. We updated the paper to add separate equations for both the visual and proprioceptive heads predictions~(Eq 2 in revised draft). The proprioceptive loss measures the squared error between the predicted proprioceptive velocity and its ground-truth noise-to-state velocity target. This term is weighted by 0.1, consistent with the appendix, and added to the visual loss in Eq. (2). At inference, future proprioceptive states are obtained by integrating the predicted velocities.

W5: Insufficient Comparison

We agree that Dreamer-v4 is a close baseline and has demonstrated “single-view” action-conditioned video generation. A direct reproduction is challenging because a publicly released official implementation is not available. Moreover, Dreamer-v4 reports training a 2B-parameter model on 256–1024 TPU-v5p devices. Our WEAVER trains on four H100 GPUs which also shows the training efficiency of our pipeline. Lastly, Dreamer-v4 trains its own tokenizer, whereas WEAVER uses a pretrained encoder (similar to WorldGym and Ctrl-World). An encoder trained only on DROID may face distribution shifts on new task OOD datasets.

The paper has ablations of proprioceptive state prediction and the visual encoder. During rebuttal, we added ablations to show the importance of Diffusion Forcing, memory, and SPRINT (check Response of W1 to Reviewer UYtS for details). We also evaluate inference-time choices like KV caching, noise schedules, and rectified-flow fine-tuning. Lastly, we jointly demonstrate three downstream applications within a single world model– policy evaluation, policy improvement, and test-time planning. As discussed in response to W2, these experiments are sufficient to show the improvements of WEAVER.

Reviewer_UYtSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WEAVER is a 928M-parameter latent world model for manipulation. It encodes two camera views and proprioception with the SD3 VAE, predicts future latents with a 2D spatial/temporal transformer trained by flow matching plus diffusion forcing, and conditions on a sparse long-term memory and a two-frame history. It also predicts future proprioception, and carries a reward head and a critic trained on RoboMeter progress labels so that rollouts can be scored in latent space without decoding or calling a VLM. KV caching, a cosine noise schedule, and a ReFlow distillation step bring inference down to a few seconds for 10s rollouts, and to about 1.5s per planning call. The model is pretrained on DROID for 10 days on 4 H100s and fine-tuned on 250 real rollouts from pi0.5 across five tasks. Results: better FID/FVD than Ctrl-World and Cosmos3-Nano at lower inference time (Table 1, Fig. 2); policy-evaluation correlation of 0.517 pretrained and 0.817 after fine-tuning, versus 0.410 and 0.677 for Ctrl-World (Table 10); fine-tuning pi0.5 on WEAVER-generated data raises average success from 44% to 67% (synthetic only) or 82% (real + synthetic); best-of-4 test-time selection raises it to 58%. A BridgeV2 comparison and two small ablations (encoder choice, proprio prediction) are in the appendix.

Critical strengths
  • The same model is used to evaluate policies, to generate training data that improves pi0.5, and to pick actions at test time, on a physical Franka. Most world-model papers only report video quality.

  • Fig. 2 and Table 3 show quality vs. inference time at 8/16/32/50 denoising steps for all three models. WEAVER with 8 steps is better than Ctrl-World with 50.

  • The reward and critic heads read the latent directly, so there is no image decoding and no VLM call. Table 7 shows this step takes under 1 ms.

Critical weaknesses
  1. Almost no ablations. Multi-view, memory + history, diffusion forcing, SPRINT: none are ablated. Only the noise schedule, the encoder and proprio prediction are (and the last two only on BridgeV2). If the contribution is "the right combination," I need to see which parts matter.

  2. The 0.817 in the abstract is the fine-tuned model, on the same tasks and policy family it was fine-tuned on. Zero-shot it's 0.517. Put both numbers up front. Also say which correlation: Table 10 has Spearman 0.817 and Pearson 0.846.

  3. Mixed (2k) vs Real (1k) isn't a fair fight. Twice the data. Need a Real (2k) row. And "synthetic matches real within 4%" is 67 vs 71 with 20 trials per task, which is noise.

  4. 20 trials per task is thin. That's about +/-10 points. Several of the claimed gaps (Syn vs Real, per-task planning gains) are smaller than that.

  5. Success labels on imagined rollouts are humans watching generated video. How many raters, were they blind, what was the agreement? Everything downstream rests on this.

  6. It's an integration paper, which is fine, but the "first" claim needs care. Memory from Ctrl-World, training recipe and reward heads from Dreamer-v4, SD3 VAE, SPRINT, ReFlow. "First to jointly demonstrate evaluation, improvement and planning with a pretrained VLA" should be checked against VLAW (cited) and Genie Envisioner (not cited, 2508.05635, same pitch with a public multi-view simulator). Also uncited and relevant: WMPO (2511.09515), SC3-Eval (2606.18610), OSCAR (2606.04463), PAVXploreRL (2607.16602).

  7. Minor: text says planning gains 15%, Fig. 7 says 14%.

Questions for authors

See Weakness.

7 comments
AuthorsOpenReview ↗

Response to Reviewer UYtS (Part 1)

We are glad that the reviewer highlights WEAVER’s real-world applications in policy evaluation, improvement, and planning, its strong quality– efficiency tradeoff, and its reward and value estimation directly in latent space as strengths of the paper. We address the concerns below:

W1 Component ablations

We have added more discussion on the design choice:

  1. Memory and Diffusion Forcing: In the table below and in table 8 of the updated paper, we observe that WEAVER outperforms the variants that do not use (top row) sparse memory and history and (middle row) Diffusion Forcing. Note that for the model trained without Diffusion Forcing, we used the same noise schedule for each future state during inference. Our experiment presents that WEAVER benefits from both memory and Diffusion Forcing.

Table I Ablation Analysis of WEAVER on Droid & OOD Dataset

DROID

MethodExterior LPIPS ↓Exterior FID ↓Exterior FVD ↓Wrist LPIPS ↓Wrist FID ↓Wrist FVD ↓
WEAVER w/o memory0.14814.1331.110.39328.05121.44
WEAVER w/o Diffusion Forcing0.12611.3629.380.37824.4399.28
WEAVER0.11710.2027.830.37121.5090.72

Task Data (OOD)

MethodExterior LPIPS ↓Exterior FID ↓Exterior FVD ↓Wrist LPIPS ↓Wrist FID ↓Wrist FVD ↓
WEAVER w/o memory0.16826.69103.550.37834.50238.24
WEAVER w/o Diffusion Forcing0.16225.4798.760.37434.08226.11
WEAVER0.15523.9588.270.36430.77184.62
  1. Multi-view prediction. Our multi-view design follows prior work such as Ctrl-World and EnerVerse-AC, which have demonstrated promising results with multi-view world models for robotic manipulation. A controlled single-view comparison would require training an additional model on DROID dataset (substantial computational cost). We therefore skip this ablation.

  2. SPRINT. We additionally evaluate WEAVER without SPRINT on BridgeV2 in Table 11 in the updated paper and table below. In this work, we incorporated SPRINT to reduce the computational cost (training FLOPS) of world-model training. Surprisingly, similar to the SPRINT paper on image generation, we observe that SPRINT improves the model performance for world models (table of experiment on BridgeV2 dataset is below).

Table II Ablations of WEAVER with SRPINT on Bridgev2 Dataset

ModelFID ↓FVD ↓LPIPS ↓
No-sprint14.0338.870.2095
no-proprio10.9322.110.1864
SVD Encoder22.2459.240.2059
WEAVER9.5719.080.1845

W2 Policy Evaluation Results

We thank the reviewer for pointing this out. We have revised the abstract to clearly distinguish the performance of pretrained and fine-tuned WEAVER and added the reported metric as Spearman’s rank correlation ( changes highlighted in red in revised abstract).

To clarify the evaluation settings of WEAVER vs WEAVER-FT, Sec. 5.1 evaluates the pretrained WEAVER world model, while Sec. 5.2.1 (Policy Evaluation) reports results for both the pretrained and fine-tuned models, denoted WEAVER and WEAVER-FT, respectively. Secs. 5.2.2 and 5.2.3 use WEAVER-FT for the two downstream applications because it achieves stronger correlation with ground-truth policy performance.

AuthorsOpenReview ↗

Response to Reviewer UYtS (Part 2)

W3: Clarification on Policy Improvement experiments

Data budgets and synthetic augmentation. Our goal is to achieve strong policy improvements with less real-world data to reduce human collection effort. We follow a similar comparison protocol as VLAW which evaluates policy fine-tuning with real-only, synthetic-only, and combined real and synthetic data to assess the contribution of generated experience. In our experiments, synthetic-only fine-tuning already achieves performance close to real-only fine-tuning: 67% versus 71% mean success, using 1k training segments in both settings. Furthermore, combining the existing 1k real segments with 1k synthetic segments increases success to 82%, an 11 percentage point improvement without increasing the real-data budget for policy fine-tuning. The Mixed (2k) versus Real (1k) comparison therefore directly evaluates the intended benefit of augmentation– obtaining additional policy improvements from generated experience while keeping the available real data fixed.

To further examine real-data efficiency at a fixed total dataset size, we are training Mixed (1k), containing 500 real and 500 synthetic segments per task, for comparison with Real (1k) (which is equivalent to the reviewer’s request of comparing real (2k) vs mixed (2k)). This directly tests whether comparable policy improvements can be achieved with 50% fewer real segments for policy fine-tuning, supporting our objective of reducing dependence on additional physical data collection.

Table III . Policy success rates (%) under different fine-tuning data budgets.

TaskBase policy (π0.5\pi_{0.5})Real (1k)Synthetic (1k)Mixed (1k)Mixed (2k)
PnP Marker3060505570
Pour Beans3060556065
PnP Towel4075757590
Stack Bowls6075758090
PnP Bag6085808095
Average4471677082

Data composition: Real (1k) uses 1k real segments per task; Synthetic (1k) uses 1k synthetic segments; Mixed (1k) uses 500 real + 500 synthetic segments; and Mixed (2k) uses 1k real + 1k synthetic segments.

At a fixed total budget of 1k training segments per task, mean success follows Synthetic (1k) < Mixed (1k) < Real (1k). Mixed (1k) achieves performance close to Real (1k) (70% vs 71%) while using only 500 real segments per task, demonstrating that synthetic augmentation enables comparable policy performance with 50% fewer real segments for policy fine-tuning. These results support our goal of reducing real-world data collection effort while maintaining strong policy performance.

W4: Evaluation trials and interpretation.

Real-world robot evaluation is costly and time-consuming, requiring physical execution and environment resets for each trial. Our budget of 20 real-world trials per task per method is consistent with common practice in closely related studies: PlayWorld uses 20 physical trials per policy, while World2Act and LingBot-VA use 20 trials per task. We evaluate each method across five tasks, totaling 100 physical trials per method. The 67% versus 71% comparison highlights the close observed performance of synthetic-only and real-only fine-tuning. Our central finding is the practical benefit of synthetic augmentation where adding generated experience increases mean success from 71% to 82% while keeping the real-data budget fixed, demonstrating the value of WEAVER-generated data for improving physical robot policies.

AuthorsOpenReview ↗

Response to Reviewer UYtS (Part 3)

W5 Human labels and reward reliability

We thank the reviewer for pointing this out. Each imagined rollout was labeled as success or failure by a single human annotator, who is one of the authors, using explicit, visually observable task-completion criteria:

  • Pour Beans: Pour beans from the cup into the bowl without spilling beans onto the table.

  • PnP Towel: Pick up the towel and place it entirely inside the basket, with no part remaining outside.

  • PnP Marker: Grasp the marker from either end and place it in the cup without tipping the cup over.

  • Stack Bowls: Place the target bowl on top of the other bowl.

  • PnP Bag: Pick up the bag from the table and place it on the plate, with no part touching the table.

A similar pipeline for policy evaluation with annotations from a single judge has been used in prior robotic world-model research, such as VLAW (Appendix C), RLVR-World (Appendix A.5). In our study, inter-rater agreement was not measured because only one annotator was used. We updated the manuscript to explicitly document the annotation setup and task criteria and acknowledge this limitation in Appendix A 6.1.

In this work, the human labels are used for policy evaluation. Policy improvement and test-time planning use the learned reward and critic with RoboMeter-based supervision, and their reported success rates are measured through physical robot execution.

W6 Prior work and contribution

We thank the reviewer for highlighting these related works. We have expanded the related work in Appendix A2 to include discussion on Genie Envisioner’s policy-learning and simulation framework [1], VLAW’s iterative co-improvement [2], WMPO’s VLA policy optimization [3], SC3-Eval and OSCAR’s policy evaluation [4, 5], and PAVXploreRL’s reward-driven world-model refinement [6]. We have also added a comparison of their design choices and demonstrated downstream applications in Table 2 of the revised paper. Our contribution is a unified world-model recipe that improves fidelity, long-horizon consistency, and inference efficiency. This enables the same model to support policy evaluation, policy improvement, and test-time planning with VLAs on real hardware.

VLAW builds on the Ctrl-World backbone and focuses on iterative world-model and policy improvement. Our experiments show that WEAVER achieves a stronger quality-versus-latency tradeoff than Ctrl-World and outperforms fine-tuned Ctrl-World in policy evaluation. WEAVER’s latent reward head achieves 74.7% ranking accuracy, compared with 62.3% for Ctrl-World combined with RoboMeter. Furthermore, at a prediction horizon of 15 and batch size of 4, dynamics inference takes approximately 1.45 s for WEAVER versus 29.4 s for Ctrl-World, a roughly 20× speedup. This latency makes Ctrl-World impractical under our test-time planning budget.

Genie Envisioner pretrains GE-Base on large-scale robotic video data and uses its latent features to learn a policy, GE-Act, from demonstrations. Its action-conditioned simulator, GE-Sim, supports closed-loop policy rollouts for evaluation. Its reported video evaluation includes external VLM-based semantic metrics. The reward prediction can suffer from covariate shift as the generations can be out-of-distribution for the frozen VLM judge. In contrast, WEAVER learns a lightweight reward head that directly scores imagined latents without image decoding or external VLM calls. Moreover, the Genie Envisioner paper does not demonstrate policy improvement using imagined rollouts or test-time planning.

W7 Planning gain

Thank you for pointing out this inconsistency. The average planning gain is 14 percentage points, from 44% to 58%. We have corrected the 15% statement in the text to match Fig. 8 in Sec 5.2.3 and highlight the change in red in the updated manuscript.

We hope our responses and proposed changes have resonated with you, and you consider increasing your score.

Best Regards,

The Authors

References:

[1] Liao, Yue, et al. Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation.arXiv preprint arXiv:2508.05635 (2025).

[2] Guo, Yanjiang, et al. VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model. ICML, 2026.

[3] Zhu, Fangqi, et al. WMPO: World Model-based Policy Optimization for Vision-Language-Action Models.arXiv preprint arXiv:2511.09515 (2025).

[4] Tseng, Wei-Cheng, et al. SC3-Eval: Evaluating Robot Foundation Models via Self-Consistent Video Generation. arXiv preprint arXiv:2606.18610 (2026).

[5] Wu, Zhuoyuan, and Jun Gao. OSCAR: Omni-Embodiment Action-Conditioned World Model for Robotics. arXiv preprint arXiv:2606.04463 (2026).

[6] Wang, Han, et al. PAVXploreRL: Physical-Action-Visual World Model Reinforcement Learning with Action Exploration.] arXiv preprint arXiv:2607.16602 (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.