acceptodds
Under review as a conference paper at ICLR 2027

Grounding Generated Videos in Feasible Plans via World Models

Abstract

Large-scale video generative models have shown emerging capabilities as zero-shot visual planners, yet video-generated plans often violate temporal consistency and physical constraints, leading to failures when mapped to executable actions. To this end, we propose Grounding Video Plans with World Models (GVP-WM), a planning method that grounds video-generated plans into feasible action sequences using a pre-trained action-conditioned world model. At test-time, GVP-WM first generates a video plan from initial and goal observations, then projects the video guidance onto the manifold of dynamically feasible latent trajectories via video-guided latent collocation. In particular, we formulate grounding as a goal-conditioned latent-space trajectory optimization problem that jointly optimizes latent states and actions under world-model dynamics, while preserving semantic alignment with the video-generated plan. Empirically, GVP-WM recovers feasible long-horizon plans from zero-shot video guidance that severely violates physics in out-of-distribution settings and scales to 3D robotic manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Decision making, reinforcement learning, Generalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RbmnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses a key failure mode of large video generative models as zero-shot visual planners: generated video plans often violate temporal consistency and physics (teleportation, morphological drift, spatial bilocation, motion blur), causing failures when inverse-dynamics models map them to actions. The authors propose GVP-WM, a test-time method that (i) generates a video plan from initial and goal observations with a first–last-frame I2V diffusion model, (ii) encodes it into the latent space of a pre-trained action-conditioned world model, and (iii) grounds it via video-guided latent collocation — a constrained latent-space trajectory optimization that jointly optimizes latent states and actions, enforces world-model dynamics as hard constraints via an Augmented Lagrangian Method, and preserves semantic alignment with the video plan through a scale-invariant (cosine) loss, executed with receding-horizon MPC. Experiments span 4 environments, two world-model architectures, and three video-quality regimes. GVP-WM recovers feasible trajectories from severely physics-violating zero-shot videos, is far more robust to motion blur than UniPi, and plans ~3× faster than MPC-CEM. Ablations confirm the necessity of jointly optimizing latent states (success collapses 0.82→0.12 without collocation).

Critical strengths

Clean, well-validated formulation. Grounding as dynamics-constrained latent trajectory optimization — joint state/action optimization, hard dynamics constraints progressively enforced via ALM, scale-invariant alignment for latent magnitude drift — each component is backed by a decisive ablation (0.82→0.12 / 0.64 / 0.60). This is the first systematic test-time grounding path requiring no extra training or environment interaction, complementary to training-time grounding, and agnostic to the world-model architecture.

Rigorous empirics. 4 environments, multiple horizons, seeds and CIs; the motion-blur study cleanly isolates the grounding mechanism (same videos for UniPi; MB-10 T=25: 0.82 vs 0.02). The authors report results that cut against their own narrative (bad guidance < no guidance).

Critical weaknesses

Video guidance hurts in the flagship regime, and the gains are not attributed to the core method. In the OOD settings the paper targets, video-free MPC-CEM with the same world model beats zero-shot guidance throughout (Push-T: 0.75/0.28 vs 0.57/0.13), and the authors' own ablation shows removing guidance (0.68) beats using it (0.56). A key control is missing: adding the same video-alignment term to MPC-CEM's cost. Without it, one cannot tell whether gains come from video-guided collocation or from adding a video term to any planner.

MPC keeps aligning to the remaining frames of a video that execution was forced to deviate from — the more divergence, the more misleading the guidance (consistent with the widening gap at longer horizons). Even with perfect oracle videos, Push-T T=50 reaches only 0.71 and Cube T=50 only 0.49, so long-horizon failures stem from the world model and optimizer, not video quality — contradicting the claim that the method will benefit as video models improve.

Questions for authors

Does Table 2 include the 14B video-generation time? Does the speed advantage survive end-to-end accounting?

Can you add an "MPC-CEM + same video-alignment cost" baseline — the key control for collocation's contribution?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Rbmn (Part 1/2)

We thank Reviewer Rbmn for their thorough review and constructive feedback. We are grateful that the reviewer found our paper to offer a "clean, well-validated formulation" where "each component is backed by a decisive ablation," and recognized GVP-WM as "the first systematic test-time grounding path requiring no extra training or environment interaction, complementary to training-time grounding, and agnostic to the world-model architecture." We address each weakness (W) and question (Q) below. Due to character limits, we have split our response into two separate comments.

W1 & Q2: Video Guidance and MPC-CEM + Video Alignment Baseline

We thank the reviewer for raising both points and agree that video models are most compelling as zero-shot planners when evaluated in settings closer to their real-world pretraining data. In fact, our 3D robotic manipulation environments (Meta-World Reach Wall and OGBench Cube) consist of 3D robot-arm visual scenes much closer to Wan2.1's pretraining distribution, where zero-shot video guidance (WAN-0S) improves planning performance over unguided baselines (Table 3). Conversely, 2D Push-T was intentionally selected as a severe out-of-distribution (OOD) stress test where zero-shot video plans fail catastrophically (0.00 direct execution via UniPi; Appendix B).

Following the reviewer's suggestion (Q2), we also added the exact scale-invariant cosine video-alignment cost (Eq. 2) to MPC-CEM's rollout objective, evaluated on both Push-T (severe OOD) and Cube (3D)

MethodPush-T (T=25)Push-T (T=50)Cube (T=25)Cube (T=50)
MPC-CEM + Vid. Loss (WAN-0S)0.66 [0.59, 0.72]0.18 [0.08, 0.22]0.53 [0.45, 0.61]0.39 [0.31, 0.46]
GVP-WM (WAN-0S)0.57 [0.56, 0.60]0.13 [0.12, 0.16]0.57 [0.49, 0.65]0.45 [0.37, 0.53]
MPC-CEM + Vid. Loss (WAN-FT)0.75 [0.67, 0.83]0.26 [0.19, 0.32]––
GVP-WM (WAN-FT)0.79 [0.76, 0.80]0.29 [0.28, 0.30]––
MPC-CEM + Vid. Loss (Oracle)0.78 [0.71, 0.84]0.36 [0.24, 0.50]0.69 [0.61, 0.76]0.43 [0.35, 0.51]
GVP-WM (Oracle)0.97 [0.96, 0.98]0.71 [0.66, 0.74]0.73 [0.65, 0.79]0.49 [0.41, 0.57]

Our proposed scale-invariant video-alignment loss (Eq. 2) also transfers cleanly to shooting, with MPC-CEM + Vid. Loss improving monotonically across every column as video quality increases. In particular, it improves from 0.66 →\to 0.75 →\to 0.78 at T=25 and 0.18 →\to 0.26 →\to 0.36 at T=50 on Push-T, and from 0.53 →\to 0.69 and 0.39 →\to 0.43 on Cube. MPC-CEM with our proposed video-alignment loss is less sensitive to severely corrupted OOD guidance on Push-T WAN-0S (0.66 / 0.18 vs. 0.57 / 0.13 for GVP-WM), but also substantially less effective—and slower at test time (Table 2), when reasonable video guidance is available: it yields negligible gains over unguided MPC-CEM on zero-shot 3D Cube (0.53 / 0.39 vs. 0.52 / 0.39, compared to 0.57 / 0.45 for GVP-WM) and Push-T WAN-FT (0.75 / 0.26 vs. 0.79 / 0.29), and falls well short of GVP-WM under ORACLE guidance (0.78 vs. 0.97 at T=25 and 0.36 vs. 0.71 at T=50 on Push-T, with non-overlapping CIs; 0.69 / 0.43 vs. 0.73 / 0.49 on Cube).

In addition, to isolate zero-shot video guidance from the collocation optimizer in 3D manipulation, we ran 3-seed GVP-WM (No Vid. Guid.) (λv=0\lambda_v = 0, random Gaussian latent state initialization without the video plan) using the identical evaluation setup across all horizons (Table 2):

MethodReach Wall (25)Reach Wall (50)Reach Wall (80)Cube (25)Cube (50)
MPC-CEM0.75 [0.70, 0.80]0.41 [0.36, 0.48]0.26 [0.22, 0.30]0.52 [0.44, 0.60]0.39 [0.31, 0.47]
GVP-WM (No Vid. Guid.)0.51 [0.43, 0.59]0.25 [0.22, 0.29]0.10 [0.05, 0.15]0.53 [0.45, 0.61]0.39 [0.32, 0.47]
GVP-WM (WAN-0S)0.76 [0.72, 0.80]0.48 [0.44, 0.52]0.35 [0.32, 0.38]0.57 [0.49, 0.65]0.45 [0.37, 0.53]
GVP-WM (Oracle)0.82 [0.80, 0.84]0.54 [0.52, 0.56]0.42 [0.40, 0.44]0.73 [0.65, 0.79]0.49 [0.41, 0.57]

These results separate the contribution of zero-shot video guidance from the change of optimizer. Adding zero-shot guidance (WAN-0S) to the identical collocation planner improves Reach Wall from 0.51 to 0.76 (T=25), 0.25 to 0.48 (T=50), and 0.10 to 0.35 (T=80), with non-overlapping 95% CIs at every horizon. On Cube, the gains are smaller and within confidence intervals (0.53 to 0.57 at T=25; 0.39 to 0.45 at T=50). Oracle guidance improves further in all columns.

Together, these two ablations show that while our video-alignment objective can also guide shooting planners, latent collocation is more effective at exploiting informative video plans, and zero-shot video guidance directly improves collocation in 3D manipulation.

AuthorsOpenReview ↗

Response to Reviewer Rbmn (Part 2/2)

Continuing from our previous response, we address W2 and Q1 below.

W2: Benefit as Video Models Improve and Long-Horizon Failure

We agree with the reviewer that compounding prediction error in the learned world model and aligning to a fixed video plan under execution drift cause performance to degrade with horizon across planners. At the same time, holding that imperfect world model fixed, our results show a clear monotonic trend as video quality improves across every 2D and 3D environment and horizon: progressing from WAN-0S to WAN-FT to Oracle in 2D (e.g., 0.13 to 0.29 to 0.71 on Push-T T=50 vs. 0.28 for MPC-CEM; 0.76 to 0.90 to 1.00 on Wall T=50 vs. 0.74 for MPC-CEM) and from No Vid. Guid. to WAN-0S to Oracle in 3D (e.g., 0.10 to 0.35 to 0.42 on Reach Wall T=80 vs. 0.26 for MPC-CEM). Thus, GVP-WM indeed benefits substantially as video models improve up to the ceiling set by world-model fidelity, while long-horizon world-model degradation explains the remaining gap for Oracle. Consistent with this, in our new ablation above (Table 3, W1 & Q2), GVP-WM (WAN-0S) widens its advantage over both No Vid. Guid. and MPC-CEM as the horizon grows. We have updated Section 6 to qualify our scaling claim.

Q1: End-to-End Inference Time Breakdown

Table 2 originally reported GVP-WM (planning) (encoding + ALM collocation + local refinement on the same GPU hardware as MPC-CEM) and excluded the one-time t=0t=0 Wan2.1-FLF2V-14B synthesis step (Algorithm 1, Line 1). Because video generation runs only once per episode rather than at every MPC step, the speed advantage survives full end-to-end accounting:

MethodPush-T (T=25)Push-T (T=50)Wall (T=25)Wall (T=50)
GVP-WM (planning)861582443
GVP-WM (video)45944492
GVP-WM (total)13125268135
MPC-CEM277596121198

Including 14B video synthesis, GVP-WM (total) remains 2.1x to 2.4x faster end-to-end than MPC-CEM on Push-T (131s vs. 277s at T=25; 252s vs. 596s at T=50) and 1.5x to 1.8x faster on Wall (68s vs. 121s at T=25; 135s vs. 198s at T=50). We have updated Table 2 in Section 4.2 to report planning, video, and total explicitly in the revised version.

Concluding remarks

We thank Reviewer Rbmn again for their constructive review. We added the MPC-CEM + Vid. Loss baseline across Push-T and Cube (Appendix B) and GVP-WM (No Vid. Guid.) baselines in Table 3, revised our framing of 2D OOD stress tests versus 3D zero-shot gains, qualified our scaling discussion in Section 6 to distinguish the compounding world-model ceiling from the monotonic video-prior gains within that ceiling, and updated Table 2 with end-to-end episode runtimes including 14B video generation. We hope these additions resolve the reviewer's concerns, and we look forward to further discussion.

Thanks for your rebuttal! Most of my concerns have been resolved. I will keep my positive score.

AuthorsOpenReview ↗

Response to Reviewer’s Rebuttal Response

Dear Reviewer Rbmn,

We sincerely thank you for your thoughtful engagement throughout the review process. We are grateful that our rebuttal resolved most of your concerns, and we deeply appreciate your continued positive assessment of our work. If any final questions remain, we would be pleased to address them during the remaining discussion period.

Reviewer_ztyQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GVP-WM grounds video plans by optimizing latent states and actions under a frozen world model. It adds video initialization and cosine alignment to latent collocation, then executes actions with MPC. The video generator is pretrained Wan2.1, used directly or adapted on 100 demonstrations for Push-T and Wall. The world models are trained offline for the target environments; “zero-shot” describes the video guidance, not an environment-untrained controller. Evaluation covers four simulated environments.

Critical strengths
  1. The method offers a useful way to use a video plan without forcing the robot to follow impossible frames. The contribution is the video-guided grounding formulation and its evaluation, built on established collocation and world-model components.
  2. The same-planner ablations support this combination when guidance is reliable. On Push-T at T=25, fine-tuned video guidance raises success from 0.68 to 0.82; replacing cosine alignment with MSE reduces it to 0.64 (Table 6). The same-video comparison with inverse dynamics also shows substantially better tolerance to motion blur (Table 4).
Critical weaknesses

The evidence for a benefit from zero-shot guidance is less clear. In the same-planner Push-T ablation, removing it raises success from 0.56 to 0.68 (Table 5). The positive 3D results compare against MPC-CEM, changing both optimization and guidance. The authors acknowledge that poor guidance can hurt; the remaining question is whether the generated videos help the same planner in the 3D settings where the full system improves.

Questions for authors
  1. Do you have no-video collocation results for Reach Wall and Cube, especially at the longer horizons? These would distinguish the value of zero-shot video guidance from the change of optimizer in Table 3.
  2. Equation (1) conditions generation on both start and goal images, whereas Appendix D.2 mentions the first frame. Were both images used for WAN-0S and WAN-FT in every environment? Please clarify how the goal frame enters generation; this determines whether the video prior itself is goal-directed, rather than the goal entering only the planning cost.
  3. Does Table 2 include video generation, encoding and local refinement? What is the end-to-end episode time on the same hardware as MPC-CEM?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ztyQ

We thank Reviewer ztyQ for their thorough review and constructive feedback. We are grateful that the reviewer recognized that our method "offers a useful way to use a video plan" and appreciated our evaluations and ablations. We address each weakness (W) and question (Q) in the order presented below.

W1/Q1: No-Video Collocation for Reach Wall and Cube

We thank the reviewer for this suggestion and have added 3-seed no-video collocation results for both Reach Wall and Cube. To isolate zero-shot video guidance from the optimizer, we ran the exact same No Vid. Guid. ablation (λv=0\lambda_v = 0, random Gaussian latent state initialization without the video plan) using the identical evaluation setup across all horizons:

MethodReach Wall (T=25)Reach Wall (T=50)Reach Wall (T=80)Cube (T=25)Cube (T=50)
MPC-CEM0.75 [0.70, 0.80]0.41 [0.36, 0.48]0.26 [0.22, 0.30]0.52 [0.44, 0.60]0.39 [0.31, 0.47]
GVP-WM (No Vid. Guid.)0.51 [0.43, 0.59]0.25 [0.22, 0.29]0.10 [0.05, 0.15]0.53 [0.45, 0.61]0.39 [0.32, 0.47]
GVP-WM (WAN-0S)0.76 [0.72, 0.80]0.48 [0.44, 0.52]0.35 [0.32, 0.38]0.57 [0.49, 0.65]0.45 [0.37, 0.53]
GVP-WM (Oracle)0.82 [0.80, 0.84]0.54 [0.52, 0.56]0.42 [0.40, 0.44]0.73 [0.65, 0.79]0.49 [0.41, 0.57]

These results separate the contribution of zero-shot video guidance from the change of optimizer. Adding zero-shot guidance (WAN-0S) to the identical collocation planner improves Reach Wall from 0.51 to 0.76 (T=25), 0.25 to 0.48 (T=50), and 0.10 to 0.35 (T=80), with non-overlapping 95% CIs at every horizon. On Cube, the gains are smaller and within confidence intervals (0.53 to 0.57 at T=25; 0.39 to 0.45 at T=50). Oracle guidance improves further in all columns.

Zero-shot performance differs between 2D Push-T and 3D manipulation because Push-T was deliberately selected as a severe OOD stress test for Wan2.1, whereas Reach Wall and Cube feature 3D robot-arm visual scenes closer to Wan2.1's pretraining distribution. Similarly, once video guidance quality on Push-T is improved via fine-tuning (WAN-FT: 0.82) or Oracle trajectories (0.98), performance exceeds the no-video baseline (0.68; Table 5). We have added GVP-WM (No Vid. Guid.) to Table 3 and updated Section 4.3 in the revised version.

Q2: Conditioning on Both Start and Goal Images

We thank the reviewer for raising this phrasing discrepancy in Appendix D.2. We confirm that both the initial observation o0o_0 and the goal observation ogo_g were used to condition generation for both WAN-0S and WAN-FT in every environment, exactly as specified in Section 3.2 (Eq. 1) and Section 4.1. Specifically, we use Wan2.1-FLF2V-14B (First-Last-Frame-to-Video), which employs a first-last frame masking mechanism that fixes both the initial frame o0o_0 and the final goal frame ogo_g to guide the generation of intermediate frames. We have corrected Appendix D.2 (Line 981 in the original submission) to state "conditioned on the first and last (goal) frames" in the revised version. Thank you for helping us improve our work.

Q3: End-to-End Inference Time Breakdown

We agree that an explicit end-to-end runtime breakdown strengthens our paper. In Table 2, the reported numbers for GVP-WM (86s at T=25, 158s at T=50 on Push-T; 24s at T=25, 43s at T=50 on Wall) include latent encoding (Algorithm 1, Line 2), the ALM optimization loop (Lines 6 to 14), and the local refinement step (Line 15), measured on the exact same GPU hardware as MPC-CEM. They did not include the one-time open-loop video generation step at t=0t=0 (Algorithm 1, Line 1). Below is the complete episode runtime breakdown (in seconds) on identical hardware:

MethodPush-T (T=25)Push-T (T=50)Wall (T=25)Wall (T=50)
GVP-WM (planning)861582443
GVP-WM (video)45944492
GVP-WM (total)13125268135
MPC-CEM277596121198

Accounting for full end-to-end episode time including 14B video generation (GVP-WM (total)), our method remains 2.1x to 2.4x faster than MPC-CEM on Push-T (131s vs. 277s at T=25; 252s vs. 596s at T=50) and 1.5x to 1.8x faster on Wall (68s vs. 121s at T=25; 135s vs. 198s at T=50). We have updated Table 2 to report GVP-WM (planning), GVP-WM (video), and GVP-WM (total) explicitly in the revised version.

Concluding remarks

We are grateful to the reviewer for their careful reading and constructive feedback that enabled us to improve our work. In particular, we added 3-seed no-video collocation results across all horizons on Reach Wall and Cube to Table 3, correcting the typo in Appendix D.2, and provided the full end-to-end episode runtime breakdown in Table 2. We hope these clarifications convincingly address all of the reviewer's questions, and we look forward to further discussion.

Reviewer_ztyQOpenReview ↗

Response to authors

Thanks for adding the no-video runs. Seeing success go from 10% to 35% at the longest horizon on Reach Wall with the same planner answers my main question. The clarifications about the goal frame and the full runtime were helpful too.

I’ll keep my score at 3. I’m more convinced that zero-shot video guidance can help, though the Cube and Push-T results still leave me unsure how broadly that benefit carries over.

AuthorsOpenReview ↗

Response to Reviewer’s Rebuttal Response

Dear Reviewer ztyQ,

Thank you for your thoughtful follow-up and engagement. We are pleased that the new results and clarifications addressed your main questions, and we deeply appreciate your continued positive assessment of our work. We hope that the monotonic gains with video quality and the complementary no-video collocation and MPC-CEM + video-loss controls provide encouraging evidence across settings. If any final questions remain, we would be pleased to address them during the remaining discussion period.

Reviewer_dyKhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a test-time method that uses an action-conditioned world model to ground video-model-generated plans into feasible action trajectories. The method first generates a video plan using a pretrained generative video model, then encodes the video into latent states, and uses video-guided latent collocation to jointly optimize latent states and actions in the world model. The method is evaluated on several simulated environments, showing that it can recover feasible plans from imperfect video guidance.

Critical strengths

The proposed concept is interesting, using a video model as a zero-shot planner to provide an initial plan and a more action-grounded world model to refine the plan, avoiding temporal inconsistencies or physical implausibility in the video-generated plan to obtain a feasible action trajectory.

Critical weaknesses

Video models as zero-shot planners are an interesting idea because they have been pretrained on large amounts of video data, especially real-world videos. Therefore, it would be more interesting to see whether the proposed method can be demonstrated in real-world settings. Currently, some of the simulated settings may be out-of-distribution for the video model, so it is unclear whether it makes sense to use Wan2.1 as the initial planner.

The current results, such as Table 5, show that without video guidance, performance is actually better in the zero-shot setting, which makes me wonder whether the current results really support the claim. The planning tasks may already be solved using the world model alone without video planning, and video guidance may even hurt performance in some settings.

Another experiment I am curious about is what would happen if we asked the video planner to generate multiple different videos, or controlled the quality of the video plans, with some having plausible action trajectories and others having temporally inconsistent or physically implausible trajectories. Alternatively, we could use random trajectories that are not generated by the video planner. Would the current world-model collocation system still work, or even perform better without the video planner?

It would be more interesting to see whether this system enables new capabilities that neither the video model nor the world model alone can achieve, i.e., truly leveraging the video planner's zero-shot ability to provide an initial plan in new scenarios, and using the action-conditioned world model to refine it into a physically plausible and executable plan.

Questions for authors

Please see the weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer dyKh (Part 1/2)

We thank the reviewer for their time and constructive feedback. We appreciate that the reviewer found our concept "interesting": a video model serves as a zero-shot planner that provides an initial plan, and an action-grounded world model refines that plan into a feasible action trajectory. Due to character limits, we have split our response into two separate comments. In this first comment (Part 1), we address W1–W2. Please see the subsequent comment (Part 2) for our response to W3–W4.

W1 & W2: Impact of video guidance, OOD vs. 3D regimes, and real-world scaling

We agree that video models are most compelling as zero-shot planners near their real-world pretraining data. To evaluate zero-shot video guidance with the same planner in 3D robotic manipulation, we added 3-seed GVP-WM (No Vid. Guid.) results (λv=0\lambda_v=0, random Gaussian latent initialization without the video plan). These use the identical evaluation setup across all horizons (Table 3):

MethodReach Wall (T=25)Reach Wall (T=50)Reach Wall (T=80)Cube (T=25)Cube (T=50)
MPC-CEM0.75 [0.70, 0.80]0.41 [0.36, 0.48]0.26 [0.22, 0.30]0.52 [0.44, 0.60]0.39 [0.31, 0.47]
GVP-WM (No Vid. Guid.)0.51 [0.43, 0.59]0.25 [0.22, 0.29]0.10 [0.05, 0.15]0.53 [0.45, 0.61]0.39 [0.32, 0.47]
GVP-WM (WAN-0S)0.76 [0.72, 0.80]0.48 [0.44, 0.52]0.35 [0.32, 0.38]0.57 [0.49, 0.65]0.45 [0.37, 0.53]
GVP-WM (Oracle)0.82 [0.80, 0.84]0.54 [0.52, 0.56]0.42 [0.40, 0.44]0.73 [0.65, 0.79]0.49 [0.41, 0.57]

With the identical collocation planner, zero-shot guidance (WAN-0S) improves Reach Wall from 0.51 to 0.76 (T=25), 0.25 to 0.48 (T=50), and 0.10 to 0.35 (T=80), with non-overlapping 95% CIs at every horizon. On Cube, the gains are smaller (0.53 → 0.57; 0.39 → 0.45). Oracle guidance improves further in all columns. In 3D scenes closer to Wan2.1's pretraining distribution, the world model alone therefore does not match its performance with video guidance, and the benefit grows with horizon.

In Table 5 on 2D Push-T (T=25), zero-shot guidance underperforms the no-video baseline (0.56 vs. 0.68). This is because synthetic 2D frames are severely OOD for Wan2.1, and we deliberately chose Push-T as a stress test. Once Push-T video quality improves through fine-tuning (WAN-FT: 0.82) or oracle plans (0.98), performance clearly exceeds the no-video baseline. Our primary direction for future work (Section 6) is real-world robotic manipulation, where zero-shot video plans are closest to the video model's pretraining data. We have added GVP-WM (No Vid. Guid.) to Table 3 and updated Sections 4.2, 4.3, and 6 in the revised version.

AuthorsOpenReview ↗

Continued Response to Reviewer dyKh (Part 2/2)

Continuing from our previous response, we address weaknesses W3–W4 below.

W3: Random Trajectories, Controlled Video Quality, and Multiple Video Plans

  1. Random trajectories without the video planner: Latent Collocation still works when initialized from random trajectories, but consistently underperforms GVP-WM whenever video guidance is informative. In Table 5 of our paper, we evaluate Push-T (T=25) using random Gaussian latent trajectories without the video planner (No Vid. Guid., λv=0\lambda_v = 0) and random initialization with video alignment (No Vid. Init, λv=1\lambda_v = 1), measured by success rate:
Planner Variant (Push-T, T=25, Table 5)WAN-0SWAN-FTOracle
Random Traj. Only (No Vid. Guid., λv=0\lambda_v = 0)0.680.680.68
Random Init + Video Loss (No Vid. Init, λv=1\lambda_v = 1)0.660.600.62
GVP-WM (Ours)0.560.820.98

Random trajectories (0.68) outperform GVP-WM only under severe 2D OOD corruption (WAN-0S: 0.56), falling well below domain-adapted (WAN-FT: 0.82) and Oracle (0.98) plans. Randomizing initialization alone (No Vid. Init) similarly drops WAN-FT from 0.82 to 0.60.

  1. Controlled video quality: GVP-WM remains robust under controlled temporal degradation where direct video-to-action execution collapses, while scaling monotonically as physical plausibility improves (from WAN-0S to WAN-FT to Oracle in Tables 1 and 5). In Table 4, we corrupt Oracle Push-T (T=25) plans with increasing motion-blur windows (MB-3, MB-5, MB-10) and report success rate:
Method (Push-T, T=25, Table 4)OracleMB-3MB-5MB-10
UniPi (Direct Video Execution)0.520.160.040.02
GVP-WM (Ours)0.980.960.940.82

As blur increases, UniPi drops from 0.52 to 0.02, whereas GVP-WM maintains 0.98 to 0.82—outperforming random-trajectory collocation (0.68) across every blur level.

  1. Multiple video plans: Averaging multiple candidate videos degrades planning because GVP-WM optimizes around a single reference trajectory, and averaging multimodal paths around obstacles produces infeasible intermediate targets. Extending GVP-WM to MM candidate videos instead requires grounding each candidate in parallel and selecting the trajectory with the lowest world-model dynamics residual Ldyn\mathcal{L}_{\mathrm{dyn}} (Eq. 5), which we highlighted as future work in Section 6.

W4: Enabling Capabilities Beyond Either the Video Model or World Model Alone

We thank the reviewer for this great point, which aligns closely with the vision of our work. While testing open-world skills outside the offline world model's training distribution is bounded in simulated benchmarks, our results already show that GVP-WM achieves higher task success rates than either standalone component: the video planner alone (UniPi, which maps generated video frames directly to actions via inverse dynamics without a world model) and the world model alone (MPC-CEM and GVP-WM without video guidance, which plan without a video prior). Specifically, GVP-WM outperforms world-model-only planning across all five zero-shot 3D settings in Table 3 (MPC-CEM, GVP-WM without video guidance), while under degraded video guidance on Push-T, GVP-WM (0.82) simultaneously surpasses both the video planner alone (0.02 for UniPi, Table 4) and the world model alone (0.68 for No Vid. Guid., Table 5; 0.75 for MPC-CEM, Table 1). We have expanded this discussion in Section 6 of the revision.

Concluding remarks

We thank the reviewer again for their time and constructive feedback. In the revised version, we added 3-seed GVP-WM (No Vid. Guid.) results across all 3D horizons to Table 3, clarified the distinction between 2D OOD stress tests and 3D zero-shot manipulation, and expanded Section 6 on when to use video guidance and multi-video selection. We hope these additions address the reviewer's concerns, and we look forward to further discussion.

AuthorsOpenReview ↗

Follow-Up on Our Response to Reviewer dyKh

Dear Reviewer dyKh,

We would greatly value hearing whether our response addressed the points you raised as the discussion deadline approaches. In brief, we added three-seed no-video collocation results across all 3D horizons, analyzed random trajectories and controlled video-quality degradation, clarified the distinction between the 2D OOD stress test and 3D zero-shot manipulation, and expanded our discussion of multiple video plans and capabilities beyond either standalone component.

Thank you for your thoughtful feedback and engagement with our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.