acceptodds
Withdrawn from ICLR 2027

IMWM: Intuition Models Complement World Models for Latent Planning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Planning with a learned latent world model is a promising route to control from raw pixels, and when such a planner fails, the usual remedy is a better world model. We show experimentally that this is not always enough: even with a perfect world model—obtained by replacing the world model with the simulator itself, which can predict the outcome of any action sequence with zero error—a planner with a finite sampling budget still often fails to reach the goal. This can happen because planning involves two coupled operations: sampling candidate action sequences and evaluating them. A perfect world model removes prediction error from evaluation, but sampling successful action sequences can still be difficult. Motivated by this limitation, we propose IMWM (Intuition Model + World Model), which pairs the world model with an intuition model trained on demonstrations to recognize promising action sequences. Three lightweight components connect them: (i) Retrieval Initialization guides sampling by starting it from a retrieved demonstration; (ii) Hybrid Cost adds the intuition score to evaluation, alongside the world model's goal loss; (iii) Reliability Gate decides, before planning, whether to enable the intuition model. Experiments on six pixel-based goal-reaching tasks show that IMWM improves on or matches the world-model-only planner, with large gains on OGBench-Cube (93.2%, +28.5 percentage points) and Two-Room (99.3%, +10.0 points). On OGBench-Scene and DMC-Cheetah, two tasks added after all planner settings had been fixed, it gains 7.5 and 20.5 points, respectively.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, latent planning, sampling-based planning, learning from demonstrations, contrastive learning, retrieval-augmented planning, goal-conditioned control

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_TVk7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors argue that perfect world model predictions are not sufficient for reaching a desired goal state given a restricted sampling budget. They propose a framework that learns a contrastive intuition model in parallel with the world model and use it to help initialize candidates sampling for planning (e.g., CEM). The main experiments evaluate their method on several simulated environments, such as OGBench-Cube, PushT, and Two-Room.

Critical strengths

None

Critical weaknesses
  1. The evaluation constructs goals in a way that is particularly favorable for demonstration-based retrieval. Although Appendix J evaluates disjoint trajectories, the paper does not test goals requiring novel action sequences or extrapolation beyond the demonstration distribution. It remains unclear whether the proposed approach generalizes to more complex planning settings or situations with limited data support.
  2. The InfoNCE objective learns compatibility between demonstrated actions and start-goal pairs, but does not establish whether the learned intuition scores reliably indicate task success or physical feasibility. It would be helpful for the authors to evaluate how these intuition scores relate to task success. Adding a hybrid cost can also degrade performance, as observed in Push-T (Table 3).
  3. The paper does not discuss prior work on using memory-based warm-starting of model predictive control [1], despite retrieval initialization accounting for a large part of IMWM's performance gains (Table 3).

[1] Mansard, N., Del Prete, A., Geisert, M., Tonneau, S., & Stasse, O. (2018). Using a Memory of Motion to Efficiently Warm-Start a Nonlinear Predictive Controller. IEEE International Conference on Robotics and Automation (ICRA).

Questions for authors
  1. How well does IMWM generalize to goals outside the demonstration distribution or settings with limited data coverage?
  2. Can the authors provide experimental results showing that intuition scores reliability predict task success?
Reviewer_mSKFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

IMWM combines a demonstration-trained intuition model with world-model planning through retrieval initialization, a hybrid cost, and a reliability gate. It evaluates six visual goal-reaching tasks and provides gate stress tests and retraining on disjoint trajectories for three tasks.

Critical strengths

The paper usefully separates prediction quality from finite-budget planning performance. Combining retrieval and candidate scoring is well motivated, and the gains over WM-only are substantial on several tasks.

Critical weaknesses
  1. The most important comparisons for incremental contribution are incomplete. Appendix K lists retrieved-action replay, goal-conditioned behavior cloning, and behavior-cloning-initialized CEM as ongoing experiments whose results are not part of the submission. The evidence therefore establishes improvement over WM-only more clearly than it establishes the value of the proposed combination over simpler uses of the same demonstrations under comparable total computational budgets. This uncertainty matters more than adding unrelated tasks.
  2. The reliability gate has a limited operating scope. In Appendix H, Table 8, Push-T drops from 93.3% for WM-only to 89.8% with the intuition model forced on using a bank of 100 entries, yet the gate enables it on all 12 seeds. Conversely, with 30 entries on Two-Room, forced-on performance is 96.3% versus 89.3% for WM-only, but the gate disables it on every seed. These failures are already disclosed; the issue is what reliability the component actually provides, rather than a lack of testing.
Questions for authors
  1. Are complete results now available for the most decisive comparisons in Appendix K: retrieved-action replay, goal-conditioned behavior cloning, and behavior-cloning-initialized CEM? Please prioritize comparisons on the existing tasks with comparable data access and total inference budgets.
  2. What is the gate's intended objective and operating scope outside the default retrieval-bank setting? Please use the existing stress tests to clarify whether it is a heuristic effective in the development configuration rather than a general guarantee against harmful priors.
Reviewer_NeA6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a limitation of planning with learned latent world models, where improving prediction accuracy does not necessarily resolve planning failures under a finite search budget. The proposed method, IMWM, supplements a fixed world model with a contrastively trained intuition model. It uses demonstration retrieval to initialize CEM, combines an action-compatibility score with the latent goal loss, and introduces a gate that decides whether to enable these components. Experiments on visual goal-reaching tasks show substantial gains on several tasks, supported by component ablations, sampling-budget comparisons, and additional evaluations with disjoint training and evaluation trajectories.

Critical strengths
  • The paper addresses a meaningful distinction between prediction and planning. A model that accurately predicts the consequences of actions does not, by itself, provide an effective procedure for finding those actions. The simulator-based experiment makes this distinction concrete, and the inclusion of multiple sampling-based optimizers helps establish that residual failures are not exclusive to one solver. Although the diagnosis has limitations, the underlying question is relevant to world-model research.

  • The empirical improvements over the tested planner are substantial. IMWM raises success rates on Cube and Two-Room, with improvements on every evaluation seed in both tasks. Increasing the WM-only sampling budget does not close the gap. These results provide convincing evidence that demonstration-guided search can be more effective than simply allocating additional samples to this planner.

  • The ablations and reporting clarify several important aspects of the method. Correctly matched retrieval outperforms random or shuffled alternatives, and the hybrid cost provides benefits even without retrieval on some tasks. Improvements also persist in the three tasks tested with disjoint training and evaluation trajectories. The authors acknowledge negative results, calibration dependencies, and gate failures, which makes the evidence easier to assess.

Critical weaknesses
  • The central comparison with alternative ways of incorporating action knowledge is missing. The experiments establish an advantage over WM-only planning, but do not establish an advantage over a planner equipped with a conventional learned action prior. A goal-conditioned behavior-cloning policy used to initialize CEM is a particularly relevant baseline, since it would also exploit the demonstration data to improve proposals. Retrieved-action replay is necessary to determine how much additional value planning provides.

  • The paper describes these comparisons as “in preparation.” These are central to assessing the contribution, rather than optional extensions. It is inappropriate to defer such essential evidence to promised future results; the submitted work must be evaluated on its completed experiments (This is the main reason of clear rejection). As presented, the paper supports the usefulness of adding behavioral information more strongly than the particular choice of IMWM.

  • The benefit of the proposed intuition-learning framework is not sufficiently isolated. Retrieval initialization alone accounts for most of the gain on the principal tasks. On Cube, it raises success from 64.7% to 90.2%, compared with 93.2% for the full method. This is an important result, but it also raises the question of whether a separately trained intuition model is required. A comparison using the existing world-model encoder for the same retrieval procedure would distinguish the contribution of the learned representation from the benefit of retrieving a suitable initial sequence. The current evidence leaves the incremental value of the full framework less clear than the overall performance improvement suggests.

  • The perfect-world-model diagnosis does not establish which failure mechanism the method resolves. Replacing learned dynamics with the simulator removes prediction error, but retains the latent goal metric and the optimizer. Moreover, diagnostic candidate success requires a successful terminal state, whereas actual execution can succeed upon reaching the goal earlier. Recording only the final CEM iteration also cannot distinguish failure to generate successful candidates from failure to retain them during optimization. The experiment therefore establishes that exact dynamics are insufficient for this planning pipeline, but provides a less decisive diagnosis of the remaining bottleneck.

  • The interpretation of the intuition score and its reliability requires stronger validation. Contrastive training rewards observed start–goal–action matches over actions from other trajectories. Those alternative actions are not verified failures, and the resulting discrimination problem need not align with ranking similar candidates encountered during CEM. The score may still be useful as a behavioral prior, but its relationship to actual success is not directly established.

  • The scope of generalization remains limited. Evaluation goals are obtained from observations 25 steps after demonstrated starting states. The disjoint-split experiments are a useful control for trajectory overlap, but do not establish generalization to unfamiliar start–goal combinations or conditions with poor demonstration coverage. Since retrieval explains much of the improvement, understanding performance when a close demonstrated solution is unavailable is important to judging the method’s broader significance.

Questions for authors
  • Comparison with alternative action priors: Under comparable data and planning budgets, how does IMWM compare with behavior-cloning-initialized CEM and direct replay of retrieved actions? These comparisons would clarify the value of the proposed formulation beyond providing a demonstration-based starting point.

  • Necessity and validity of intuition learning: Does retrieval using the existing world-model encoder provide comparable gains? Within actual CEM candidate sets, does the learned intuition score distinguish successful from unsuccessful sequences beyond what the latent goal loss already captures?

  • Attribution of planning failures: Does the perfect-world-model diagnosis hold when candidate success includes reaching the goal at any intermediate step? Tracking successful candidates across CEM iterations and evaluating the final executed mean would help distinguish candidate-generation failures from evaluation or retention failures.

  • Independent validation of the gate: With its threshold fixed, can the gate correctly disable intuition on a new task where intuition is harmful? The added tasks demonstrate benefits from enabling intuition, but do not establish this complementary capability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.