acceptodds
Under review as a conference paper at ICLR 2027

Feasible Actions, Misleading Rollouts: Diagnosing Latent World Model Planning

Abstract

Prediction error, goal-cost disagreement, and actuator imprecision require different evidence in latent world-model planning. We study frozen planners by executing fixed action banks and comparing predicted cost, actual-outcome cost, and physical error. On ten V-JEPA2-AC/RoboCasa sources and ten LeWM/PushT tasks, predicted selection incurs mean range-normalized actual-cost regrets of 19.2% and 6.5%, respectively, while outperforming within-bank uniform selection. Seven LeWM banks select the actual-cost minimum; the distinct banks do not support a model ranking. We then test intermediate known-state feedback in DINO-WM. After a common actuator finish, feedback reduces composite object error relative to native planning, endpoint-only correction, and first-patch consistency control by 0.132, 0.090, and 0.098 on 32 generated tasks. Coarse-success non-inferiority remains unestablished. A finite-history analysis characterizes the learned state-error paths omitted from a shared comparison bound, without guaranteeing smaller realized errors. Separately, a 16-task LeWM terminal-constraint test does not confirm an object benefit. Together, the results show why candidate-level cost regret, physical outcomes, and actuator precision must be evaluated separately.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Visual world models, Model-based planning, Model predictive control, Goal-conditioned control, Planning evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qTKZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper "audits" world models (LeWM, DINO-WM, V-JEPA2-AC) on MPC using collections of "frozen" trajectories (what the authors call "banks" in their abstract, but which are better called probes). They study various error metrics such as next-state/embedding prediction error, and error in prediction future reward/success.

Critical strengths

Generally I think the genre of research is good -- too many papers champion some "novel" tweak to a method, and too few pick apart existing methods and try to understand their painpoints.

Critical weaknesses

The paper reads as an eclectic mix of tests that the authors ran on various JEPA models and various robot control benchmarks. I'm all for benchmark papers, but why only JEPA-style models? Why these benchmarks? There's no stochastic environments, no validation on a real robot, no Dreamer-style or Diamond-style or inverse-dynamics-style world models, and more significantly, no clear-cut indictment or vindication of the JEPA paradigm.

Clarity is also poor; prose appears AI-written and inherits the expected issues, such as obscure ad-hoc terminology ("banks" is but one of many examples).

Questions for authors

What should we really learn from your work? Do you think your findings would transfer to a real robot? To stochastic environments? To long-horizon tasks? To wildly-unobserved decision-making problems? To not just manipulation but also locomotion?

Reviewer_V5s4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies planning failure diagnose with frozen latent world models. It distinguishes between predicted goal cost, computed goal cost from execution, and physical errors. The paper studies the planning failures in V-JEPA2-AC and LeWM and also tests a fix for DINO-WM by using a separate motion model to estimate the pusher’s position and velocity to replace its predictions. On 32 generated PushT tasks, this fix reduces object error compared with the endpoint and spatial controls.

Critical strengths
  • The paper observes and proposes a useful method to distinguish different planning failures. The model may generate a false action outcome prediction, or it could also prefer an outcome with less planning cost but is physically worse. This distinction is important as it tells if improving prediction or objectives is needed.
  • The DINO-WM improvement is well supported. To check if the method only improves the pusher’s final position, the paper uses the same final adjustment procedure to bring the pusher close to its target for all methods and shows that their method still achieves lower mean object error on the 32 generated PushT tasks.
Critical weaknesses
  • The tested actions may not represent what the planners would actually choose. The main audits use ten action banks per system, each containing a zero-action sequence and 15 sequences sampled before CEM optimization. These actions may differ from those selected after search, so it is unclear whether these results explain the failures that arise during actual planning.
  • The paper reports improvements on 32 generated PushT tasks whose goals are created by pushing in a fixed direction. An earlier DINO-WM evaluation on a different set of tasks did not confirm the same improvement in object error, and LeWM tests a different correction. This makes it hard to judge whether the method would help on a new task
Questions for authors
  • Do the main diagnostic findings hold for actions produced by actual planning? In addition to the small DINO-WM cross-scoring test, could you repeat the analysis using more candidates from later CEM iterations, including the final executed plans?
  • The 32 generated PushT tasks show improvements in object error, but the earlier DINO-WM evaluation does not confirm improvements. More analysis is needed about what might account for these results and under what task conditions does the correction help?
Reviewer_Lzs2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper audits frozen latent world-model planners at three levels: predicted cost, actual-outcome cost, and physical error. It has four parts:

  • Fixed-bank audit. On fixed banks of 16 executed, non-optimized candidates, choosing by predicted cost beats choosing at random.

  • Range-normalized regret is 19.2% on V-JEPA2-AC/RoboCasa and 6.5% on LeWM/PushT.

  • Known-state feedback. In DINO-WM, the predicted proprioception in the rollout history is overwritten with the known pusher state, followed by a shared analytic actuator finish. This lowers object error by 0.132, 0.090, and 0.098 relative to the Native, Endpoint, and Spatial controls on 32 generated PushT tasks. Non-inferiority on coarse success is not established.

  • Theory. A finite-history error bound.

Critical strengths

The controls are clean within the cohort.

The Endpoint and Spatial arms separate intermediate state injection from terminal correction and patch consistency. The shared analytic finish removes actuator precision as a confound. Relative to Spatial, the benefit appears before the finish (−0.168) and persists after it (−0.098). → The gain is cleanly attributed, within this cohort.

A reusable audit, reported in full. Executing every candidate separates prediction loss from objective misalignment; in RoboCasa, the physical gap comes mostly from the objective. Failed criteria and earlier cohorts are all reported → a useful template for evaluating world models.

Critical weaknesses

Critical 1: the positive result is the survivor of mixed tests. Earlier attempts went as follows:

The 24-source confirmation failed its coarse-success safeguard. An RGB-calibrated cohort failed its complete rule. A simple analytic-finish baseline beat feedback. The DINO-WM temporal-window cohort pointed the wrong way (+0.058). → One cohort at p ≈ 0.015 can't support a general claim.

Critical 2: the protocol evolved along the way. The object-only endpoint and the shared finish replaced the earlier joint-error criteria, and the Spatial control was added after the cohort had been observed. The pre-declared 0.1 meaningful effect is not met, and coarse success is lower than in the corrected controls → forking paths undercut the headline result.

Secondary: the general lessons are already published and uncited. Objective mismatch (Lambert et al., 2020), value-aware model learning (Farahmand et al., 2017), and value equivalence (Grimm et al., 2020) already make the evaluation point. Grounding look-ahead in known or retrieved real state rather than imagined predictions is the published design of Web ATLAS (Cheng et al., 2025).

Questions for authors

[Clarification] Discrepancies. What explains the gap between the generated-task and temporal-window results? What explains the recurring coarse-success deficit?

[Suggestion] Try confidence-weighted ("soft") state injection, and frame the method as grounded simulation. Split the paper and cut the qualifiers.

  • Scheduled Sampling (Bengio et al., 2015), on errors that build up when a model is fed its own predictions; the overwrite reverses this at inference time;

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.