- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses a key failure mode of large video generative models as zero-shot visual planners: generated video plans often violate temporal consistency and physics (teleportation, morphological drift, spatial bilocation, motion blur), causing failures when inverse-dynamics models map them to actions. The authors propose GVP-WM, a test-time method that (i) generates a video plan from initial and goal observations with a first–last-frame I2V diffusion model, (ii) encodes it into the latent space of a pre-trained action-conditioned world model, and (iii) grounds it via video-guided latent collocation — a constrained latent-space trajectory optimization that jointly optimizes latent states and actions, enforces world-model dynamics as hard constraints via an Augmented Lagrangian Method, and preserves semantic alignment with the video plan through a scale-invariant (cosine) loss, executed with receding-horizon MPC. Experiments span 4 environments, two world-model architectures, and three video-quality regimes. GVP-WM recovers feasible trajectories from severely physics-violating zero-shot videos, is far more robust to motion blur than UniPi, and plans ~3× faster than MPC-CEM. Ablations confirm the necessity of jointly optimizing latent states (success collapses 0.82→0.12 without collocation).
Critical strengths
Clean, well-validated formulation. Grounding as dynamics-constrained latent trajectory optimization — joint state/action optimization, hard dynamics constraints progressively enforced via ALM, scale-invariant alignment for latent magnitude drift — each component is backed by a decisive ablation (0.82→0.12 / 0.64 / 0.60). This is the first systematic test-time grounding path requiring no extra training or environment interaction, complementary to training-time grounding, and agnostic to the world-model architecture.
Rigorous empirics. 4 environments, multiple horizons, seeds and CIs; the motion-blur study cleanly isolates the grounding mechanism (same videos for UniPi; MB-10 T=25: 0.82 vs 0.02). The authors report results that cut against their own narrative (bad guidance < no guidance).
Critical weaknesses
Video guidance hurts in the flagship regime, and the gains are not attributed to the core method. In the OOD settings the paper targets, video-free MPC-CEM with the same world model beats zero-shot guidance throughout (Push-T: 0.75/0.28 vs 0.57/0.13), and the authors' own ablation shows removing guidance (0.68) beats using it (0.56). A key control is missing: adding the same video-alignment term to MPC-CEM's cost. Without it, one cannot tell whether gains come from video-guided collocation or from adding a video term to any planner.
MPC keeps aligning to the remaining frames of a video that execution was forced to deviate from — the more divergence, the more misleading the guidance (consistent with the widening gap at longer horizons). Even with perfect oracle videos, Push-T T=50 reaches only 0.71 and Cube T=50 only 0.49, so long-horizon failures stem from the world model and optimizer, not video quality — contradicting the claim that the method will benefit as video models improve.
Questions for authors
Does Table 2 include the 14B video-generation time? Does the speed advantage survive end-to-end accounting?
Can you add an "MPC-CEM + same video-alignment cost" baseline — the key control for collocation's contribution?
4 comments
Response to Reviewer Rbmn (Part 2/2)
Continuing from our previous response, we address W2 and Q1 below.
W2: Benefit as Video Models Improve and Long-Horizon Failure
We agree with the reviewer that compounding prediction error in the learned world model and aligning to a fixed video plan under execution drift cause performance to degrade with horizon across planners. At the same time, holding that imperfect world model fixed, our results show a clear monotonic trend as video quality improves across every 2D and 3D environment and horizon: progressing from WAN-0S to WAN-FT to Oracle in 2D (e.g., 0.13 to 0.29 to 0.71 on Push-T T=50 vs. 0.28 for MPC-CEM; 0.76 to 0.90 to 1.00 on Wall T=50 vs. 0.74 for MPC-CEM) and from No Vid. Guid. to WAN-0S to Oracle in 3D (e.g., 0.10 to 0.35 to 0.42 on Reach Wall T=80 vs. 0.26 for MPC-CEM). Thus, GVP-WM indeed benefits substantially as video models improve up to the ceiling set by world-model fidelity, while long-horizon world-model degradation explains the remaining gap for Oracle. Consistent with this, in our new ablation above (Table 3, W1 & Q2), GVP-WM (WAN-0S) widens its advantage over both No Vid. Guid. and MPC-CEM as the horizon grows. We have updated Section 6 to qualify our scaling claim.
Q1: End-to-End Inference Time Breakdown
Table 2 originally reported GVP-WM (planning) (encoding + ALM collocation + local refinement on the same GPU hardware as MPC-CEM) and excluded the one-time Wan2.1-FLF2V-14B synthesis step (Algorithm 1, Line 1). Because video generation runs only once per episode rather than at every MPC step, the speed advantage survives full end-to-end accounting:
| Method | Push-T (T=25) | Push-T (T=50) | Wall (T=25) | Wall (T=50) |
|---|---|---|---|---|
| GVP-WM (planning) | 86 | 158 | 24 | 43 |
| GVP-WM (video) | 45 | 94 | 44 | 92 |
| GVP-WM (total) | 131 | 252 | 68 | 135 |
| MPC-CEM | 277 | 596 | 121 | 198 |
Including 14B video synthesis, GVP-WM (total) remains 2.1x to 2.4x faster end-to-end than MPC-CEM on Push-T (131s vs. 277s at T=25; 252s vs. 596s at T=50) and 1.5x to 1.8x faster on Wall (68s vs. 121s at T=25; 135s vs. 198s at T=50). We have updated Table 2 in Section 4.2 to report planning, video, and total explicitly in the revised version.
Concluding remarks
We thank Reviewer Rbmn again for their constructive review. We added the MPC-CEM + Vid. Loss baseline across Push-T and Cube (Appendix B) and GVP-WM (No Vid. Guid.) baselines in Table 3, revised our framing of 2D OOD stress tests versus 3D zero-shot gains, qualified our scaling discussion in Section 6 to distinguish the compounding world-model ceiling from the monotonic video-prior gains within that ceiling, and updated Table 2 with end-to-end episode runtimes including 14B video generation. We hope these additions resolve the reviewer's concerns, and we look forward to further discussion.
Thanks for your rebuttal! Most of my concerns have been resolved. I will keep my positive score.
Response to Reviewer’s Rebuttal Response
Dear Reviewer Rbmn,
We sincerely thank you for your thoughtful engagement throughout the review process. We are grateful that our rebuttal resolved most of your concerns, and we deeply appreciate your continued positive assessment of our work. If any final questions remain, we would be pleased to address them during the remaining discussion period.
Response to Reviewer Rbmn (Part 1/2)
We thank Reviewer Rbmn for their thorough review and constructive feedback. We are grateful that the reviewer found our paper to offer a "clean, well-validated formulation" where "each component is backed by a decisive ablation," and recognized GVP-WM as "the first systematic test-time grounding path requiring no extra training or environment interaction, complementary to training-time grounding, and agnostic to the world-model architecture." We address each weakness (W) and question (Q) below. Due to character limits, we have split our response into two separate comments.
W1 & Q2: Video Guidance and MPC-CEM + Video Alignment Baseline
We thank the reviewer for raising both points and agree that video models are most compelling as zero-shot planners when evaluated in settings closer to their real-world pretraining data. In fact, our 3D robotic manipulation environments (Meta-World Reach Wall and OGBench Cube) consist of 3D robot-arm visual scenes much closer to Wan2.1's pretraining distribution, where zero-shot video guidance (WAN-0S) improves planning performance over unguided baselines (Table 3). Conversely, 2D Push-T was intentionally selected as a severe out-of-distribution (OOD) stress test where zero-shot video plans fail catastrophically (0.00 direct execution via UniPi; Appendix B).
Following the reviewer's suggestion (Q2), we also added the exact scale-invariant cosine video-alignment cost (Eq. 2) to MPC-CEM's rollout objective, evaluated on both Push-T (severe OOD) and Cube (3D)
Our proposed scale-invariant video-alignment loss (Eq. 2) also transfers cleanly to shooting, with MPC-CEM + Vid. Loss improving monotonically across every column as video quality increases. In particular, it improves from 0.66 → 0.75 → 0.78 at T=25 and 0.18 → 0.26 → 0.36 at T=50 on Push-T, and from 0.53 → 0.69 and 0.39 → 0.43 on Cube. MPC-CEM with our proposed video-alignment loss is less sensitive to severely corrupted OOD guidance on Push-T WAN-0S (0.66 / 0.18 vs. 0.57 / 0.13 for GVP-WM), but also substantially less effective—and slower at test time (Table 2), when reasonable video guidance is available: it yields negligible gains over unguided MPC-CEM on zero-shot 3D Cube (0.53 / 0.39 vs. 0.52 / 0.39, compared to 0.57 / 0.45 for GVP-WM) and Push-T WAN-FT (0.75 / 0.26 vs. 0.79 / 0.29), and falls well short of GVP-WM under ORACLE guidance (0.78 vs. 0.97 at T=25 and 0.36 vs. 0.71 at T=50 on Push-T, with non-overlapping CIs; 0.69 / 0.43 vs. 0.73 / 0.49 on Cube).
In addition, to isolate zero-shot video guidance from the collocation optimizer in 3D manipulation, we ran 3-seed GVP-WM (No Vid. Guid.) (λv=0, random Gaussian latent state initialization without the video plan) using the identical evaluation setup across all horizons (Table 2):
These results separate the contribution of zero-shot video guidance from the change of optimizer. Adding zero-shot guidance (WAN-0S) to the identical collocation planner improves Reach Wall from 0.51 to 0.76 (T=25), 0.25 to 0.48 (T=50), and 0.10 to 0.35 (T=80), with non-overlapping 95% CIs at every horizon. On Cube, the gains are smaller and within confidence intervals (0.53 to 0.57 at T=25; 0.39 to 0.45 at T=50). Oracle guidance improves further in all columns.
Together, these two ablations show that while our video-alignment objective can also guide shooting planners, latent collocation is more effective at exploiting informative video plans, and zero-shot video guidance directly improves collocation in 3D manipulation.