acceptodds
Withdrawn from ICLR 2027

Representation World Model: Learning States, Transition and Executable Plans in Representation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We propose the Representation World Model (RWM), which learns states, transitions, and executable plans directly in representation space. Unlike existing world models that typically learn latent representations together with explicit dynamics models and perform planning through search, optimization, or policy-based prediction, RWM directly incorporates planning into the learned representation geometry. RWM learns the representation geometry by applying inverse-dynamics supervision locally along latent paths constructed from endpoint representations, requiring these paths to preserve task-relevant state and transition information. At inference, planning is performed by directly constructing a latent path between the current and goal representations, with inverse dynamics used to recover the corresponding actions, without recursive rollouts or action-space search. Experiments on continuous-control benchmarks demonstrate the effectiveness of RWM for direct planning, while results on robotic manipulation further show its potential to extend to more complex embodied control tasks. These results suggest that planning directly in representation space provides a promising alternative to conventional world-model planning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation, World Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_diAeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Summary

This paper introduces the Representation World Model (RWM), a world-modeling approach that jointly learns state representations and actionable transition structure in latent space, without an explicit forward dynamics model. Given start and goal observations, RWM constructs a straight-line path between their latent representations and uses an inverse dynamics model to recover actions along the path. The same construction is used during training and inference, enabling planning without autoregressive rollouts or online action-space search.

The authors evaluate RWM on four continuous-control benchmarks and ten LIBERO-Goal manipulation tasks. The results are strong, with ablations supporting the importance of inverse-dynamics supervision along the interpolated latent path. Additional probes suggest that both encoded and interpolated representations preserve relevant state and action information.

Assessment

I find this paper interesting and technically appealing. RWM offers a simple alternative to conventional latent world models by learning representations in which plans can be constructed directly and translated into actions, avoiding explicit forward-dynamics prediction and online search. The reported results are strong, and the ablations provide useful evidence for the proposed training objective.

My main concerns concern the limitations of straight-line latent planning and dependence on closed-loop execution. These issues limit how broadly the results can be interpreted, but I do not consider them sufficient to negate the contribution demonstrated on the current benchmarks.

Overall, I lean towards acceptance, subject to clarification of the planning horizon, observation modalities, and evaluation assumptions.

Critical strengths

Interesting and elegant formulation. Constructing executable plans directly in representation space is an appealing alternative to conventional world-model planning through forward rollouts and search. Using the same construction during training and inference is particularly nice.

Strong empirical results. RWM achieves a reported macro success rate of 98.25% across the four continuous-control benchmarks and 93.0% on LIBERO-Goal, outperforming the reported latent-world-model baselines.

Useful ablations and representation analysis. The experiments investigate the contribution of each training objective and show that interpolated representations preserve substantial state and action information. The closed-loop interval ablation also provides insight into the importance of feedback during execution.

Critical weaknesses

1. Limitations of straight-line latent planning. RWM constructs plans by linearly interpolating between the start and goal representations. While this works well on the evaluated tasks, it is unclear how well the approach handles environments where multiple qualitatively different trajectories connect the same endpoints or where a valid plan requires navigating around obstacles in a more complex state space. The authors acknowledge the limitation regarding multiple possible plans, but a more thorough discussion of the approach's applicability and failure modes would strengthen the paper.

2. Planning horizon and execution assumptions. The method's dependence on the planning and execution horizon could be clarified. The closed-loop interval ablation shows that performance degrades substantially when more actions are executed between replanning steps. This highlights the importance of feedback, but the paper could better explain the effective planning horizon, the choice of trajectory segment lengths, and how these settings affect performance. It would also be useful to distinguish the length of a constructed latent path from the number of actions executed before replanning.

Questions for authors
  1. What observation modalities are used in each benchmark? In particular, does RWM use proprioceptive information in addition to images, and how does this affect the comparison with baselines?

  2. How are the planning horizon, trajectory segment length, and closed-loop replanning interval chosen? How sensitive are the results to these choices?

  3. Are gradients propagated through both sides of the interpolation-consistency loss, or is one representation detached? If so, what motivates this choice?

  4. How does RWM behave when multiple distinct plans connect the same start and goal, or when the straight-line latent path does not correspond to a feasible physical trajectory?

Reviewer_hT1iOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes the Representation World Model (RWM), which learns an encoder and an inverse dynamics model (IDM) without a forward model or a representation regularizer. During training, it linearly interpolates between the latents of two states on a trajectory and trains the IDM to recover the demonstrated actions from neighboring interpolated points, so that straight lines in latent space become executable. At inference, it interpolates between the current and goal latents, decodes actions along the line with the IDM, and replans after a short chunk. The paper presents this as a third planning paradigm, distinct from search-based planning and goal-conditioned policies, and reports strong search-free results on PushT, Cube, Reacher, TwoRoom and LIBERO-Goal.

Critical strengths
  1. The presentation is clear.

  2. The interpolated latents carry real information. The probing study shows that points on the constructed path keep state and action information close to that of encoded observations, so the path is not an arbitrary line between two codes.

Critical weaknesses

Generally, I appreciate the improvements on benchmarks and explorations in policy learning, but this paper is fundamentally misleading.

  1. A ridiculous point is that this work does not have a transition model, yet it calls itself a "world model". Please keep the writing and claims rigorous, as Eysenbach et al. [1] do. Their work already contains the core idea of this paper, but this paper presents many self-invented terms and does not even cite or discuss it. Eysenbach et al. [1] is in fact closer to a world model than RWM (it yields a closed-form predictive distribution over future representations), yet does not claim to be one; RWM cannot predict any future state from the current one, let alone given an action, yet calls itself a world model.

  2. The novelty is limited:

(1) RWM is essentially a goal-conditioned policy. With linear interpolation, the next point on the path is fully determined by the current latent, the goal latent, and the remaining number of steps. The first decoded action is therefore a policy conditioned on goal and horizon, just parameterized through the IDM. Since the method replans after every short chunk and degrades when executed open-loop, the long path is not really used as a plan. The fair comparison is goal-conditioned IDM, and the paper's gain over it is not clear because the benchmark is not representative.

(2) Planning by interpolating in a learned representation is not new, but none of these works is discussed: Causal InfoGAN [2] plans by latent interpolation, and Eysenbach et al. [1] show that interpolation in contrastive representations supports planning. Work that makes latent trajectories straighter, such as temporal straightening [3] and the straightening observed in LeWM [4], is closely related. Discussions are required.

(3) Dropping SIGReg in favor of an IDM is not new either, and learning action-relevant representations does not mean that it learns informative representations, like SIGReg is doing. That inverse dynamics alone prevents collapse is a known property, already used in prior sensorimotor world-model work [5] and in IDM-based representation learning from ICM [6] to multi-step inverse models [7]. Its known cost is that an inverse model keeps only what is needed to predict the agent's own action and discards the rest of the scene. The latent space is therefore too poor to serve as a "world model" (again, this work is essentially not a world model), only as a policy which requires less on representation informativeness, and the paper's own probing already shows weaker state recovery than LeWM on TwoRoom, even on these simple tasks. RWM ends up much closer to a policy than to a world model.

(4) What remains new is applying the IDM to interpolated points instead of observed neighbors, which adds a straight-line geometric bias to goal-conditioned IDM. This is a neat trick, but it does not amount to a new world-model paradigm. The model also contains no transition model, so calling it a world model is a stretch.

  1. The benchmarks are too toy to test the claims. PushT, Cube, Reacher, and TwoRoom are simple, low-dimensional tasks that strong methods already nearly solve. They cannot show whether straight latent paths remain executable when the shortest path in state space is far from straight, for example with obstacles, contact-rich manipulation, or long multi-stage tasks. These are exactly the cases where the claimed advantage over search or a policy would matter, and they are not evaluated.

Linear interpolation also assumes every straight latent chord is executable. TwoRoom's single door can be straightened by the encoder, but an obstacle that can be passed on either side cannot, and the paper itself concedes that RWM does not support multiple options; this case is never tested.

[1] Eysenbach, Benjamin, et al. "Inference via interpolation: Contrastive representations provably enable planning and inference." Advances in Neural Information Processing Systems 37 (2024).

[2] Kurutach, Thanard, et al. "Learning plannable representations with causal infogan." Advances in Neural Information Processing Systems 31 (2018).

[3] Wang, Ying, et al. "Temporal straightening for latent planning." arXiv preprint arXiv:2603.12231 (2026).

[4] Maes, Lucas, et al. "LeWorldModel: Stable end-to-end joint-embedding predictive architecture from pixels." arXiv preprint arXiv:2603.19312 (2026).

[5] Ivashkov, Petr, Randall Balestriero, and Bernhard Schölkopf. "Sensorimotor World Models: Perception for Action via Inverse Dynamics." arXiv preprint arXiv:2606.20104 (2026).

[6] Pathak, Deepak, et al. "Curiosity-driven exploration by self-supervised prediction." International Conference on Machine Learning. PMLR, 2017.

[7] Lamb, Alex, et al. "Guaranteed discovery of control-endogenous latent states with multi-step inverse models." Transactions on Machine Learning Research (2023).

Questions for authors
  1. How does RWM differ from a goal-conditioned IDM that also takes the remaining number of steps as input? If the two are equivalent under linear interpolation, what does the path add?

  2. LIBERO-Goal is described as a goal-unobserved setting. Where does the goal latent that ends the interpolated path come from, and is the setup still comparable to the baselines?

Reviewer_7xRxOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Representation World Model (RWM), which aims to encode states, transitions, and executable plans directly in a learned representation space. Instead of learning an explicit forward dynamics model and performing rollout-based planning, RWM constructs linear latent paths between endpoint representations and trains an inverse dynamics model to recover the actions along these paths. The resulting representation is intended to support direct, search-free planning through latent interpolation and action decoding. The method is evaluated on four continuous-control benchmarks and ten LIBERO-Goal manipulation tasks.

Critical strengths
  1. An interesting formulation of search-free planning through representation learning. The paper explores a conceptually interesting direction by embedding executable transition structure directly into the learned representation geometry. Using inverse dynamics supervision on constructed latent paths provides a simple way to connect representation learning with goal-directed control, potentially reducing the need for expensive online trajectory search.

  2. Strong empirical performance with useful component analysis. RWM achieves competitive or superior performance on several continuous-control benchmarks and reports strong results on LIBERO-Goal. The ablations show that inverse dynamics supervision along constructed paths is essential for successful control, while the representation probes provide evidence that the learned latent states retain physical-state and action-relevant information.

Critical weaknesses
  1. Limited expressiveness and insufficient validation of interpolation-based planning. The central assumption of RWM is that executable plans can be constructed through straight-line interpolation between endpoint representations. However, this restricts the planning space and may be problematic for multimodal dynamics, obstacle avoidance, and trajectories requiring multiple distinct paths. Although the training objective encourages such paths to be executable, the paper provides neither theoretical conditions for this property nor sufficient empirical analysis of geometric consistency and generalization to unseen start-goal pairs. The representation probing experiments do not fully establish that interpolated paths remain feasible under more complex dynamics.

  2. Unclear contribution of the proposed planning mechanism in complex manipulation tasks. The LIBERO-Goal evaluation introduces an additional task-conditioned latent delta predictor because goal observations are unavailable at inference time. This substantially changes the planning formulation from direct endpoint interpolation to learned latent displacement prediction. However, the paper does not sufficiently isolate the contribution of this additional predictor or compare it with a similarly conditioned direct policy. Consequently, it is difficult to determine how much of the reported improvement on complex manipulation tasks can be attributed to the proposed representation-space planning mechanism.

Questions for authors

Q1. Expressiveness of latent planning. How does RWM handle environments where multiple feasible paths exist between the same start and goal states? Can the authors evaluate the method on tasks requiring distinct alternative routes or strongly nonlinear trajectories, and compare it with a more flexible latent path construction method?

Q2. Geometric consistency and generalization. To what extent do the interpolated latent states correspond to physically reachable intermediate states? Can the authors quantify interpolation consistency and action prediction errors as a function of start-goal distance, trajectory curvature, and whether the endpoint pairs were observed during training?

Q3. LIBERO-Goal implementation. How much of the LIBERO-Goal performance depends on the additional latent delta predictor and task-ID conditioning? Can the authors compare RWM against a direct action-prediction baseline using the same encoder, task information, and training data to isolate the benefit of the proposed latent planning mechanism?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.