acceptodds
Under review as a conference paper at ICLR 2027

world2vec: Stable inference of mechanistic world models

Abstract

World models learned from raw pixels have become a practical basis for planning and control, and recent joint-embedding predictive architectures can be trained stably end-to-end. Yet these models are typically evaluated in environments whose dynamics are largely driven by the agent’s actions, and stable training alone does not ensure that learned latent representations recover the underlying state or dynamics of the environment. Here, we introduce world2vec, a framework for learning mechanistic world models directly from observations. We show that world2vec recovers the ground-truth state under nonlinear observation mixing and learns latent dynamics that faithfully reproduce the true system over long prediction horizons while using one order of magnitude fewer parameters than existing world models. We show state of the art performance in controlling continuous control environments from pixels. world2vec treats the representation objective and dynamics model as hyperparameters, allowing their inductive biases to be automatically matched to the data-generating process. This enables recovery of ground-truth dynamical systems and their governing equations across simple physical environments and a dataset of 71 strange attractors and paves the way towards using world2vec for scientific discovery.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, self-supervised learning, JEPA, dynamical systems, system identification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YuhKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

World models learned from camera images are now a practical basis for planning and control. The authors argue that they are mostly tested on systems driven by the agent's own actions, and that training them stably does not guarantee they recover the system's true state or dynamics. The paper aims for "mechanistic" world models: ones that control well, reproduce the true dynamics over long horizons, and let the governing equations be read out. It builds on representation-learning objectives that are known to recover the true state under idealized conditions.

The proposed framework, world2vec, trains an image encoder together with a small network that models how the encoded state changes over time, as a differential equation; the training objective and the form of the dynamics model are treated as interchangeable choices. It learns offline from random-action data by rolling predictions forward many steps and matching the encodings of later frames. On three simple simulated control tasks, this makes the model's predictions agree with its own future encodings far better than established world models do. The authors report state-of-the-art control: planning towards an encoded goal image matches or beats the strongest baselines on two of the three tasks and trails them on the third. With a sparse polynomial dynamics model, the authors also apply the approach to 71 chaotic systems whose states are passed through a random nonlinear transformation into 50 dimensions. They show one example where near-correct equations are read off after aligning the learned coordinates to the true state.

Critical strengths
  1. Clear ablation of training horizon and dynamics form (Tables 2–3). Two choices both help:

    • training to predict 20–40 steps ahead instead of one;
    • modelling the latent as changing continuously (an ODE) rather than predicting the next state directly.

    The gains also show in a measure the method is not trained on: decoding the true simulator state from the latent. With the contrastive objective, this rises from 93% to 98% R² on Pendulum and from 71% to 94% on Reacher. This is the paper's most trustworthy and transferable result.

Critical weaknesses
  1. The main evidence that world2vec models the world better is a score it is trained to maximize, and the baselines are evaluated under different conditions (Sec. 5, Table 1, App. A, D.1).
    • The headline "equivariance" scores measure how well the model's predictions match its own encoder's encodings of later frames. The training loss (Eq. 7) minimizes exactly this gap over horizons covering the 1 s evaluation, and the baselines' losses do not. World2vec should therefore score higher whether or not its latent captures the true state.
    • The same score picks the best of 90–141 trained models per row. Results are reported on the validation episodes used for that choice, and the test episodes go unused (Sec. 4).
    • The baselines are scored on different episodes than world2vec, and LeWM uses larger images and a modified planner. LeWM's rewards (1 on Pendulum, 48 on Reacher) contradict the 81.5% Reacher success of the authors' own reproduction (App. D.1), which suggests the baseline is broken here.
    • Three of the six world2vec rewards behind "state-of-the-art control" come from the only one of three trained models whose planner worked (Table 1, footnote a). On Cartpole, world2vec gets ~300 against ~900 for DreamerV3 and TD-MPC2.
    • True-state decoding is the one measure that does not depend on the model's own encoder, and it is not reported for the baselines.
    • The result that does hold, that training on multi-step predictions helps (Tables 2–3), is already standard practice: TD-MPC2 (one of the baselines) and SPR train their latents to predict several steps ahead, and the method builds directly on dynamics contrastive learning. Without a comparison at matched data and prediction horizon, it is unclear what world2vec adds.
  2. The experiments do not test the paper's motivating claims: systems that evolve on their own, and equation discovery (Abstract, Secs. 1, 5–7, App. I).
    • The paper motivates itself with systems whose dynamics are not driven by the agent, and claims to be "first ... to control systems with autonomous dynamics" (Sec. 7). Yet control is tested only on the baselines' three standard DeepMind Control tasks.
    • The abstract claims the true state is recovered from nonlinearly distorted observations. Fig. 4 never measures how well the true state can be decoded, only agreement between runs and self-prediction.
    • Equations are shown for only one of the 71 chaotic systems (SprottA), picked by the authors. Obtaining them required:
      • a linear map fitted to the true states;
      • zeroing small coefficients;
      • the known maximum polynomial degree.
    • The true states are unavailable in practice, as the authors note (Sec. 6), so this is not equation discovery from observations alone.
    • In these equation experiments, the encoder is trained with one-step prediction (App. I.1), not the multi-step training the paper calls its key ingredient.
  3. The submission reads as an unfinished draft. A figure caption still contains an editing note ("Figure edit: relabel LRC as Equiv. R²", Fig. 6), two appendices are empty (G.10, H), and the "Baseline" row of Table 1 is never defined in the main text. The main-text table captions are mostly seed and model-selection bookkeeping (e.g. "Only one of three model seeds planned (a: seed 42, b: seed 789)") rather than what the tables show.
Questions for authors
  1. (Weakness 1: evaluation metric and baselines) Please evaluate every method on the same held-out test episodes, at the same image resolution, on measures world2vec is neither trained nor selected on:

    • how well a simple decoder recovers the true simulator state from each method's latent;
    • control reward averaged over all trained models, with failed runs counted.

    If world2vec still recovers the state clearly better than DreamerV3, TD-MPC2 and LeWM and matches their rewards across all models, the "better world model" and "state-of-the-art control" claims would stand. If not, its advantage reflects optimizing its own score.

  2. (Weakness 1: is the gain specific to world2vec?) The paper credits most of its gain to training on 20–40-step predictions, and TD-MPC2 and SPR already train latents to predict several steps ahead. Trained on the same offline random-action data with the same long prediction horizon, does TD-MPC2 or LeWM match world2vec? Does world2vec keep its advantage when trained on the baselines' dataset, where its reward falls to about 179 (Table 5)? If either answer goes against world2vec, the contribution reduces to a known training choice.

  3. (Weakness 2: state and equation recovery without the true state) On how many of the 71 chaotic systems do the recovered equations contain exactly the correct terms, without using the true states to align coordinates (for example, by choosing the alignment that makes the equations sparsest, as in Muratore & Mathis, 2026)? In the distorted-observation experiment (Fig. 4), how much of the true state can a linear map recover from the latent at each distortion depth? Equation recovery on a substantial fraction of systems without access to the true states would support the scientific-discovery claim. Otherwise it remains a single supervised demonstration.

  4. (Weakness 2: framing around systems that evolve on their own) Pendulum, reacher and cart-pole are the same tasks the baselines already solve. What evidence supports the claim to be "first ... to control systems with autonomous dynamics"? Can you show a task with strong dynamics not driven by the agent where world2vec succeeds and the baselines fail? Without it, the paper's stated motivation is not addressed by its experiments.

1 comment

We thank the reviewer for the very thorough review and the concrete suggestions. Changes are incorporated in the revised PDF, with the main changes highlighted in blue (see also the general response).

W1. Evaluation metric, model selection and baselines

Self-referential metrics (Q1). Following the reviewer's suggestion, we added Physical Rollout Consistency (PRC): an MLP probe is fit to decode the true state from encoded latents and then applied unchanged to the predicted latents. PRC is used only for evaluation, never for training or model selection. At 1 s, world2vec-dcl reaches 94.3% PRC, compared with 23–54% for DreamerV3, TD-MPC2 and LeWM, on the same held-out test episodes (Table 1).

Model selection and test episodes (Q1). Hyperparameters are selected on validation EQA, and all metrics (including PRC) are now reported on the held-out test split, which is the same for all models. Planners are selected on 25 validation episodes, and rewards are reported on 25 disjoint test episodes.

Single-seed rewards (Q1). Every cell in Table 1 now reports mean ± bootstrap SE over model seeds, with failed runs included. After fixing a data-collection bug in CartPole (the cart (with the pole) left the camera viewin about 16% of frames), world2vec-dcl solves all three tasks (836 / 897 / 769; CartPole was ~300 before).

LeWM rewards vs. the 81.5% Reacher success. The two numbers measure different things. Our reward is the dm_control return over a full episode (maximum return 1000) toward a target, whereas LeWM's protocol measures success at reaching a recorded goal frame only 25 steps ahead within a fixed budget, which is a much easier task. We now report both (Table 2): under LeWM's protocol, LeWM reaches 81.6% success, and world2vec matches it 25 steps ahead (86.2%) and clearly outperforms it 200 steps ahead (80.2 vs. 52.9%), while planning 3–8x faster.

Is the gain specific to world2vec? (Q2) We do not claim that multi-step prediction is new. Our controlled ablations (Tables 3–4) keep data, architecture and evaluation fixed and change only the training horizon and the dynamics parameterization. Our one-step SIGReg baseline is effectively LeWM, and extending it to a 20–40-step rollout with an Euler-integrated vector field is what makes it solve Pendulum and Reacher; with the DCL regularizer, the same change also solves CartPole. TD-MPC2 also trains multi-step latent consistency, but its objective requires reward and value prediction, so it cannot be trained in the reward-free setting; world2vec shows that the same principle works without any reward signal.


W2. Autonomous dynamics and equation discovery

Autonomous dynamics (Q4). By "autonomous dynamics" we mean environments whose state evolves even without actions, as in Pendulum and CartPole, where gravity drives the system. This is where world2vec's gains in rollout consistency are largest (Fig. 3), and where reward-free baselines fail: LeWM stays low, and our one-step baselines reach at most 199, while world2vec reaches 836 and 769. Under nonlinear mixing of the CartPole state, world2vec still solves the task (~700–850 reward), while the baselines fall to about 200–330 from two mixing layers on (Fig. 6). The claim is restricted to reward- and reconstruction-free models, which we now state explicitly in the conclusion. Reward-based methods like TD-MPC2 also solve these tasks, but use the task reward.

State recovery. Table 1 now reports State R², the R² of an MLP probe that decodes the true simulator state from each method's encoded latents, for all methods on the same held-out test episodes. All methods encode the state well (world2vec-dcl 96.0%, DreamerV3 98.4%, TD-MPC2 99.2%, LeWM 99.7%). The difference lies in the dynamics: decoded from predicted latents after 1 s (PRC), the state is preserved for world2vec-dcl (94.3%) but not for the baselines (23–54%).

Equation discovery (Q3). Equation discovery no longer uses the true states. We now train with multi-step prediction (K ∈ {10, 20, 40}, App. G) and find the affine change of coordinates in which the learned SINDy model is sparsest, following DYSCO (Muratore & Mathis, 2026). The number of terms is chosen from validation LRC alone. Across all 71 systems at α = 0.5, this recovers exactly the correct terms with correct signs for 9 systems (12%) fully unsupervised; 3 of those are in the original dysts axes and the rest are exact affine rewrites. If the number of terms is known, 28% are recovered (Fig. 8c). Ground truth is used only for scoring and for aligning the displayed equations. All 70 systems with a sparse model are listed in STab. 6, and each system is shown as flow fields in SFigs. 4–9. We still assume the polynomial degree is known.


W3. Presentation

Thank you for pointing these out. We have greatly improved the presentation of the paper, and hope the reviewer will find it more clear and informative.

Reviewer_tqXKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes world2vec, a framework for learning mechanistic world models that jointly infer latent representations and dynamics from observations. The goal is to learn representations that not only support downstream control but also faithfully capture the underlying physical states and their temporal evolution. The framework combines multi-step autoregressive latent prediction with a continuous-time dynamics model parameterized as a neural ODE and trained through numerical integration. It supports different representation learning objectives, including LeJEPA with SIGReg and Dynamics Contrastive Learning (DCL), to prevent representation collapse. The authors further integrate Sparse Identification of Nonlinear Dynamics (SINDy) to learn interpretable governing equations in the latent space. The method is evaluated on three DeepMind Control Suite environments for long-horizon prediction, state recovery, and goal-conditioned control, as well as 71 chaotic dynamical systems for mechanistic inference. The results demonstrate improved long-horizon latent prediction consistency over the evaluated baselines.

Critical strengths
  • The paper emphasizes that world models should not only support downstream control but also faithfully capture the underlying states and dynamics of the environment.

  • The proposed framework achieves substantially higher latent rollout consistency than the evaluated baselines. The experimental results also demonstrate the effectiveness of multi-step autoregressive training and continuous-time dynamics parameterization in improving prediction stability.

Critical weaknesses
  • My main concern is that the proposed framework primarily combines existing techniques, including multi-step autoregressive prediction, neural ODE dynamics, established representation learning objectives (SIGReg and DCL), and SINDy for symbolic dynamics identification. These components have been extensively studied in prior work. The paper does not sufficiently clarify what fundamentally new methodological contribution world2vec introduces beyond combining these established approaches.

  • The paper lacks a dedicated related work section and does not systematically position its contributions relative to existing research on multi-step world model learning, neural ODEs, and mechanistic dynamics identification. A more comprehensive analysis of related work is necessary to clarify the novelty and significance of the proposed approach.

  • world2vec does not demonstrate consistently competitive performance on downstream control tasks. These results suggest that improved latent prediction consistency does not necessarily translate into effective planning and control.

  • The main comparisons involve substantially different training setups. World2vec is trained offline on the authors' randomly collected dataset (D1), LeWM is trained on a different dataset (D2), and DreamerV3 and TD-MPC2 are trained online. The methods also differ in model capacity, training objectives, and model selection procedures.

  • The mechanistic inference experiments rely on synthetic dynamical systems, and the recovered SINDy equations require ground-truth state alignment for interpretation. Moreover, the polynomial feature library assumes prior knowledge of each system's maximum polynomial degree. These assumptions weaken the claim of autonomous discovery of unknown physical laws.

  • world2vec estimates the current latent state from a short, fixed window of observations (typically three frames) and subsequently uses Markovian latent dynamics for prediction. The framework does not incorporate an explicit mechanism for maintaining longer-term historical information or belief states.

Questions for authors
  • How are gradients propagated through the autoregressive rollout during training? Are gradients backpropagated through the entire rollout horizon, including intermediate predicted latent states, or are intermediate predictions detached?

  • The paper emphasizes recovering the underlying physical state and faithfully modeling its dynamics. However, a complete state representation may include task-irrelevant variables, potentially consuming model capacity and affecting goal-conditioned planning based on latent Euclidean distance. How does world2vec balance mechanistic state recovery with learning compact, control-relevant representations? Would the method remain effective in environments containing complex task-irrelevant dynamics or visual distractions?

2 comments

We thank the reviewer for the careful review. Changes are incorporated in the revised PDF, with the main changes highlighted in blue (see also the general response).

W1. Novelty beyond combining existing techniques

We revised the abstract, Sec. 1 and Sec. 2 to state the contribution more clearly. World2vec is the first working instance of a mechanistic world model (Posner et al., 2026): a model that jointly achieves (a) good task performance, (b) faithful modeling of the dynamics, and (c) an interpretable parameterization of the dynamics. Individual components (multi-step prediction, neural ODEs, SIGReg/DCL, SINDy) are known. The contribution is showing that their combination, guided by identifiability theory, yields the first reward-free, reconstruction-free world model that solves these control tasks from pixels. Prior models in this class, including LeWM, do not solve any of the three tasks in our setup.


W2. Missing related work section

We thank the reviewer for this suggestion. We added a dedicated related work section (App. A, also requested by yjDX) covering identifiable representation learning, world models, latent neural ODEs and multiple shooting, time series and dynamical systems, and symbolic regression. In each subfield, it states how world2vec relates to prior work. In short, world2vec is a reward-free, decoder-free, action-conditioned world model built on provably identifiable objectives, with the aim of recovering the mechanisms governing the environment's dynamics (Posner et al., 2026).


W3. Downstream control is not consistently competitive

The weak CartPole result was caused by a bug in our data collection (the cart (with the pole) left the camera view in about 16% of frames). After fixing it, world2vec-dcl solves all three environments (836 / 897 / 769 on Pendulum / Reacher / CartPole; CartPole was 298 before). World2vec is now competitive with TD-MPC2 and DreamerV3, and outperforms DreamerV3 on Pendulum and Reacher and TD-MPC2 on Reacher. This is notable given that both are trained online with access to the task reward, while world2vec learns offline from random-policy data and only receives a goal observation. Compared with other reward-free, reconstruction-free world models, world2vec improves over LeWM by more than 800 points on Pendulum and Reacher and by 600 on CartPole.


W4. Different training setups across methods

  • Same evaluation. All methods are now evaluated on the same held-out test episodes, with the same metrics (including the new ground-truth PRC metric).
  • LeWM is now trained on our random-policy dataset and plans with its own CEM planner. We also evaluate under LeWM's own Reacher protocol (Table 2), where world2vec matches LeWM 25 steps ahead (86.2 vs. 81.6%) and outperforms it 200 steps ahead (80.2 vs. 52.9%).
  • TD-MPC2 and DreamerV3 are trained online with rewards, following their published protocols. This setting favors them because world2vec does not use reward.
  • Model selection uses validation EQA; planners are selected on 25 validation episodes and the reward is reported on 25 disjoint test episodes. All candidate planners, with validation and test reward per model, are listed in the new STab. 3 (App. D.1).

W5. Mechanistic inference relies on ground-truth alignment and known polynomial degree

Equation discovery no longer uses the true states. We now train with multi-step prediction (K ∈ {10, 20, 40}, App. G) and find the affine change of coordinates in which the learned SINDy model is sparsest using a new fully unsupervised method inspired by the ideas presented in DYSCO (Muratore & Mathis, 2026). The number of terms is chosen from validation LRC alone. Across all 71 systems at α = 0.5, this recovers exactly the correct terms with correct signs for 11 systems (12%) fully unsupervised; 3 of those are in the original dysts axes and the rest are exact affine reformulations of the reference equations. If the number of terms is known, 28% are recovered (Fig. 8c). Ground truth is used only for scoring and for aligning the displayed equations. All 71 systems with a sparse model are listed in STab. 6, and each system is shown as flow fields in SFigs. 4–9. We still assume the polynomial degree is known.

W6. Fixed short observation window and Markovian dynamics

Markovian latent dynamics are a deliberate design choice, motivated by (a) the goal of a mechanistic world model and (b) identifiability theory.

First, they factorize the world model, so the identified dynamics can be analyzed independently of the encoder, e.g. to find fixed points, attractors, or tipping points, or to study the effect of the control variable on the flow field. Second, world2vec aims to recover the “latent world state” up to some indeterminacy. By definition of the identifiability problem, such a state is Markovian by construction, and inferring it from observations is the encoder’s task. The window length is therefore not fixed by the framework but depends on the environment. In DMC, three frames suffice to infer e.g. angular velocities.

We agree this assumes the state is recoverable from a finite window. More complex, partially observable environments may need longer sequences or encoders that feature more sophistiacted architectures with mechanism to effectively process them. Designing these archiectures and testing them on more complex, highly non-markovian (potentially partially/sparsily observed) environments is important future work.


Questions

Q1. How are gradients propagated through the rollout? Gradients flow through the full autoregressive rollout, including all intermediate predicted latents, all targets h(Xn+j)h(X_{n+j}) and, for DCL, the negatives. No stop-gradient is used (Sec. 2, App. B).

Q2. Task-irrelevant variables and visual distractors. Our goals are images of the target state (e.g. pole upright, at rest, cart centered), so reaching the goal means matching the whole latent state. In DMC, observations depend only on the physical state (fixed background and camera), so there are no nuisance factors to encode.

More generally, identifiability theory tells us world2vec can invert the data-generating process up to a simple indeterminacy. Structured task-irrelevant factors that help predict future states are therefore expected to be represented too. This is a feature, not a bug. World2vec aims to be a generic world model reusable across tasks, and a factor irrelevant for one task may matter for another.

With distractors, full-latent goal matching would no longer be appropriate, so task-specificity should move into the cost, e.g. a learned task-specific cost on top of the frozen world model that focuses on the relevant factors. Results on block-identifiability [Von Kügelgen et al. 2021, Liu et al. 2023] could help separate relevant from irrelevant factors, however we leave this for future work.

  • Von Kügelgen, J., Sharma, Y., Gresele, L., Brendel, W., Schölkopf, B., Besserve, M. and Locatello, F. Self-supervised learning with data augmentations provably isolates content from style. NeurIPS 2021.
  • Liu, Y., Huang, B., Zhu, Z., Tian, H., Gong, M., Yu, Y. and Zhang, K. Learning world models with identifiable factorization. NeurIPS 2023.
Reviewer_yjDXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes world2vec, a decoder-free latent world model. A small ViT encoder maps a window of T frames to a latent. An action-conditioned latent vector field, integrated with explicit Euler steps, is unrolled for K steps. Predicted latents are matched to the encoded future windows, using either a contrastive objective or SIGReg to prevent collapse. The vector field can be an MLP or a sparse polynomial (SINDy). The authors train on random-policy data from three DMC tasks (Pendulum, Cartpole, Reacher). They evaluate with latent-rollout metrics and with reward-free CEM planning toward goal latents. They report far better long-horizon latent prediction than reproduced baselines, competitive reward on two of three tasks, and recovered symbolic equations for one example system.

Critical strengths
  1. The ablation on training horizon is clean and convincing. Varying K from 1 to 40 in Table 2, along with the K = 1 baseline rows in Table 1 that use the same architecture, isolates the effect of the rollout loss well.

  2. The paper is open about its methods and limitations. DCL and SIGReg both work within the same architecture, and the authors openly report that SIGReg is sensitive to lambda. Appendix G also lists how many models were trained and which hyperparameters were selected for each table cell. The retiming demonstration in Fig. 2 also nicely shows what the continuous-time parameterization allows, though it remains qualitative in my eyes.

Critical weaknesses
  1. The claim "state-of-the-art performance in controlling continuous control environments from pixels" is not supported by the empirical evaluation. On Pendulum SIGReg achieves 656±243656\pm 243 vs TD-MPC2 911±4911 \pm 4. For cartpole, the rewards are 231231 and 298298 vs about 895895 for both baselines, so the task is effectively unsolved. There appears to be a favoring in the planner. For the presented methods, the rewards are "from the best planner on the selected models", while baselines use their native controllers with no per-task tuning. It is not stated whether planner selection and reporting use disjoint episodes.

  2. The latent-distance cost is never shown to track task progress (Eqs. 11-12). The planner minimizes squared Euclidean distance to a goal latent z∗=h(X∗)z^* = h(X^*), but I couldn’t find any test of whether that distance actually correlates with task cost. Eq. 8 says the representation is only identified up to an affine map, and Euclidean distance changes under such maps. So the cost depends on an arbitrary choice of gauge. Table 11 applies whitening for some planners but not others, and the planner settings were picked per environment using reward, which makes it hard to say how much of the result comes from the latent geometry and how much from tuning. Every latent dimension counts the same, nuisance dimensions included. SIGReg’s isotropy doesn’t mean distances scale with task cost, and DCL only shapes the geometry locally, relative to whichever negatives were sampled.

  3. The headline representation metrics are partly self-referential and don’t show that latent changes track observation or state changes. The used equivariance R2R^2 and Acc. compare the rollout of an encoder-dynamics pair against its own encoder (Eq. 9), so they measure self-consistency rather than fidelity, and Equivariance Acc. is also the model-selection criterion. world2vec is trained on this very quantity, while DreamerV3 and TD-MPC2 latents are shaped by reward and value prediction and LeWM is evaluated far beyond its training horizon, so a 9595% vs 0−500-50% gap follows from the objectives and doesn’t show a better world model. There is also no decoder, and the main metrics compare predicted latents with encoded real latents, so both sides live in latent space and can’t detect information lost in h. The theory assumes r=h∘gr = h \circ g is invertible, but neither DCL nor SIGReg enforces that. They prevent total collapse and not partial information loss. I’d like to see (i) state decoding from the rolled-out latents, not just from encoded real frames, as a function of horizon, (ii) a post-hoc pixel decoder on frozen latents, or a perturbation analysis, showing that action- or state-induced latent changes map to the expected observation changes, and (iii) the same probes run on the baselines, since at the moment only Fig. 3 gives any qualitative picture of the latent structure. The comparison is further muddied because baseline metrics are computed on a separate set of 1000 episodes from a random policy while world2vec is evaluated on D1 validation episodes, and model steps and frame rates also differ. Finally, the consistency numbers are uninformative without a reference. LeWM gets 99.299.2% consistency with 0.00.0 equivariance, and a random or PCA encoder would probably score high here too, so a reference row is needed to know what the number means.

  4. The pipeline (a window of observations encoded into a latent initial state, learned continuous-time latent dynamics, multi-step rollout training) closely follows the latent-ODE / multiple-shooting literature (e.g., Chen et al., 2018; Rubanova et al., 2019; Yildiz et al., 2019; Norcliffe et al., 2021; Massaroli et al., 2021; Iakovlev et al., 2023). The paper presents as findings that "a sufficiently large training horizon is essential" and that "parameterizing the dynamics as an ODE stabilizes training." Both are well motivated in that literature, where multiple shooting is used precisely to address long-horizon instability. The paper's own K=1K = 1 baseline is essentially LeWM-like, and the DCL variant is close to González Laiz et al.'s dynamics contrastive learning. What remains new is (i) replacing decoder/likelihood training with decoder-free latent matching plus a collapse regularizer, (ii) action conditioning for planning, and (iii) an optional SINDy head. The authors should state this delta explicitly, relate multi-step latent consistency to TD-MPC2 , and compare against at least one latent-ODE / multiple-shooting baseline (e.g., MSVI or Latent ODE) on the 71-system benchmark. Without this, the contribution is largely a rediscovery in a new loss setting.

  5. The theory and method are disconnected. Eqs. (2)-(3) are the standard change-of-variables identity z˙=Jrf\dot{z} = J_r f. alone has a degenerate minimizer (constant encoder, v = 0), and the paper only says "R regularizes the representation." Eq. (7), the loss actually used, is not shown to approximate Eq. (4) or to have minimizers satisfying Eq. (3), and the paper itself says "we hypothesize that modeling state change ... provides a useful inductive bias." Identifiability is imported from one-step contrastive results (González Laiz et al.; Zimmermann et al.) without arguing why their assumptions hold for action-conditioned, deterministic, multi-step, windowed-frame training. The finding that more process noise improves recovery hints that they may not and whether gradients flow through the target h(Xn+j)h(X_{n+j}) in Eq. (7) is not stated.

Chen et al. 2018, Neural Ordinary Differential Equations; Rubanova et al. 2019, Latent ODEs for Irregularly-Sampled Time Series; Yildiz et al. 2019 ODE²VAE: Deep generative second order ODEs with Bayesian neural networks; Norcliffe et al. 2021 Neural ODE Processes; Massaroli et al. 2021 Differentiable Multiple Shooting Layer; Iakolev et al. 2023 Latent Neural ODEs with Sparse Bayesian Multiple Shooting; Gonzalez Laiz et al. 2025 Self-supervised contrastive learning performs non-linear system identification;

Questions for authors
  1. Can you provide 3 seeds with error bars for all Table 1 reward cells, with planner selection done on episodes disjoint from those used for reporting?
  2. What are the control results for Reacher without the oracle elbow choice and without ground-truth inverse kinematics?
  3. Can you report state-decoding R² for the baselines on the same evaluation set as world2vec, plus a random-encoder/PCA reference for consistency and equivariance?
  4. How does latent goal distance correlate with true task cost, and how does planning change under whitened vs raw vs learned metrics and across seeds (i.e., different gauges)?
  5. Does a residual discrete-time model with the same initialization match the Euler model in Table 3?
3 comments

We thank the reviewer for the very detailed and constructive review. Changes are already incorporated in the revised PDF, with the main changes highlighted in blue (see also the general response).

W1. "State-of-the-art control" claim and planner selection

  • CartPole is now solved. The low CartPole reward in the submission was caused by a bug in our data collection: the cart (with the pole) left the camera view in about 16% of all frames. After fixing it, world2vec-dcl reaches 836 / 897 / 769 on Pendulum / Reacher / CartPole (CartPole was 298 before).
  • Toned-down claim. We now say "competitive" with TD-MPC2 and DreamerV3, and claim state-of-the-art only within the class of reward-free, reconstruction-free world models. To our knowledge, no previous model in this class solves any of the three tasks. LeWM, the strongest prior JEPA-based model, stays low on all three in our setup.
  • Disjoint planner selection and reporting. For each environment we evaluate 50 episodes. The planner is now selected on 25 validation episodes and the reward is reported on the remaining 25 test episodes. Model hyperparameters are selected on validation EQA. All metrics are reported on the held-out test split.
  • Seeds. Every cell in Table 1 now reports the mean ± bootstrap SE over model seeds, with no single-seed entries.
  • Fairer LeWM comparison. LeWM is now trained on our random-policy data and plans with its own CEM planner. On Reacher, we also evaluate under LeWM's own goal-reaching protocol (new Table 2). world2vec matches LeWM for goals 25 steps ahead (86.2 vs. 81.6% success, difference not significant), clearly outperforms it for goals 200 steps ahead (80.2 vs. 52.9%), and plans 3–8x faster.
  • Planner tuning. The CEM planner is selected on 25 validation episodes and the reward is reported on 25 separate test episodes. All candidate planners, with validation and test reward for every model, are listed in the new STab. 3 (App. D.1). Our own one-step baselines (SIGReg ≈ LeWM, and DCL) go through exactly the same planner selection and still reach only 0–199 reward. The gains therefore come from the world model, not from the planner. The chosen settings are also nearly the same across environments: the selected planner for world2vec-dcl uses about 1 s of lookahead in all three environments (0.96–1 s). TD-MPC2 and DreamerV3 use their native controllers, which their authors tuned on DMC and which rely on a learned reward/value function.

W2. Does latent goal distance track task cost?

  • Latent goal distance tracks the true distance to the goal (new App. D.2). We compare the latent distance to the goal with the true distance: pole angle from upright (Pendulum), fingertip-to-target distance (Reacher), and distance to the centered upright state (CartPole). Spearman ρ, mean over model seeds:

    PendulumReacherCartPole
    world2vec-dcl0.400.710.68
    world2vec-sigreg0.560.840.04
    Baseline (SIGReg ≈ LeWM)0.05 (n.s.)0.560.02
    Baseline (DCL)0.090.380.19

    For world2vec, the latent distance increases monotonically with the true distance (SFig. 3), in all cases except world2vec-sigreg on CartPole. That case is exactly the one where control fails (221 reward). Here the latent distance is equally small whether the system is close to the goal or far from it, so the planner cannot tell which states bring it closer to the goal. This is consistent with its PRC of 0% on CartPole. The latent cost therefore tracks task progress when the world model is good, and its failure is visible in PRC when it is not.

  • On Affine indeterminacy. Yes, it's true that in the case where the representation is identified up to affine indeterminancy there is a abitrary choice of gauge, which in theory may distort the cost function. However, in practice, we don't observe such distortions to hinder task performance when applying MPC.

W3. Self-referential metrics

  • Physical Rollout Consistency (PRC).

    • (i) We fit an MLP probe that decodes the true state from encoded latents, then apply it unchanged to the predicted latents at horizon j (App. C), and we call this Physical Rollout Consistency (PRC). PRC is used only for evaluation, not for model selection. Across all 21 method–environment pairs, PRC correlates with both latent metrics at 1 s (new SFig. 1): EQA (Pearson r = 0.81, p = 8.6 × 10⁻⁶) more strongly than LRC (r = 0.71, p = 3.5 × 10⁻⁴). The difference comes from collapse, which the reviewer rightly points out. A model that loses state information can predict its own latents well and get a high LRC while its PRC stays low: world2vec-sigreg on CartPole has an LRC of 98% but a PRC of 0%. EQA penalizes this, because a collapsed embedding cannot distinguish the true future from the negatives. This is why we select models on EQA rather than LRC, and the correlation with PRC shows that this criterion tracks the true state.

    • (ii) State R² from encoded frames (= PRC at horizon 0) is now reported for all methods in Table 1. It separates encoder quality from dynamics quality. DreamerV3, TD-MPC2 and LeWM encode the state almost perfectly (State R² 98.4 / 99.2 / 99.7%), but their PRC at 1 s drops to 54 / 29 / 23%. Their latents contain the state, but their dynamics fail at forecasting. For world2vec-dcl, the state is almost as decodable after a 1 s forecast as from the encoded frames (94.3% vs. 96.0%).

    • (iii) PRC and State R² are reported for all methods on the same held-out test episodes in Table 1.

  • Training horizon. The baselines are trained on shorter horizons, but faithful prediction over 1 s is what a world model should provide for planning. Our own one-step variants (K = 1) show the same drop (PRC 50.2% and 29.6% vs. 62.5% and 94.3% for world2vec, Table 1), so the gap reflects the training horizon and dynamics model, not the choice of metric.

  • Consistency needs a reference. A PCA encoder is deterministic and linear, so it is trivially consistent (~100%) across runs without capturing the dynamics. Learned encoders are not consistent by default (DreamerV3: 48.1%). We added this reference (new STab. 2; per environment in STab. 1). For each method, we compute consistency across its seeds at random initialization, before any training. The reviewer is right about LeWM. Its untrained encoders are already 94.4% consistent, close to the 99.7% after training, so most of its consistency comes from the architecture and initialization, not from learning. The untrained world2vec encoders are only 41.9% (sigreg) and 50.2% (dcl) consistent, and training raises this to 97.4% and 98.1%. Training therefore adds 48–56 points for world2vec, compared with 21.0 for TD-MPC2, 15.4 for DreamerV3 and 5.3 for LeWM.


W4. Novelty relative to latent-ODE literature

We thank the reviewer for the pointers. We revised the abstract, Sec. 1 and Sec. 2 to state our goal and contribution more clearly. world2vec is the first working instance of a mechanistic world model in the sense of Posner et al. (2026), meaning a model that jointly achieves (a) good task performance, (b) accurate modeling of the dynamics, and (c) a dynamics model that can be parameterized in a mechanistically interpretable way. We agree with the reviewer's description of what is new: -- (i) decoder-free, reward-free latent dynamics matching with a collapse regularizer, backed by identifiability results, (ii) action conditioning that makes the model usable for planning, and (iii) an optional SINDy parameterization for reading out equations. We do not claim that multi-step training or ODE parameterizations are new in themselves. Rather, we show that these ingredients are what make reward-free, reconstruction-free latent world models work for control, a combination that has not been shown before.

We added a related work section (App. A, also requested by tqXK), including a dedicated paragraph on latent neural ODEs that covers the works the reviewer pointed out. World2vec shares their continuous-time latent dynamics, but these methods are trained with likelihood or reconstruction objectives through a VAE or observation decoder and are mostly not action-conditioned, so they cannot be used for planning in Table 1.

Questions

Q1. Three seeds with error bars; planner selection on disjoint episodes. Every cell in Table 1 reports mean ± bootstrap SE over model seeds. The planner is selected on 25 validation episodes, and the reward is reported on 25 disjoint test episode (see STab. 3, App. D.1).

Q2. Reacher without the oracle elbow choice and inverse kinematics. We do not choose the elbow configuration: both are planned and the average is reported (Sec. 3.1). Inverse kinematics is only used to render a goal observation, since DMC Reacher does not show the arm at the target. It gives no state information to the world model or the planner. For a result without inverse kinematics, Table 2 follows the LeWM protocol, where the goal is the recorded observation 25 or 200 steps ahead in the episode: world2vec-dcl reaches 86.2% / 80.2% success, compared with 81.6% / 52.9% for LeWM.

Q3. State decoding for baselines on the same evaluation set, plus random baseline. Yes. Table 1 now reports State R² (state decoding from encoded frames) and PRC (state decoding from predicted latents at 1 s) for all methods on the same held-out test episodes. Consistency at random initialization is reported as a reference for all methods (STab. 2). For example, LeWM is already 94.4% consistent at initialization, while world2vec gains 48–56 points through training (see W3).

Q4. Correlation of latent goal distance with true task cost For world2vec-dcl, latent goal distance correlates with the true distance to the goal along the MPC test episodes in all three environments (Spearman ρ = 0.40 / 0.71 / 0.68 for Pendulum / Reacher / CartPole, p < 10⁻¹⁰; new App. D.2). The only world2vec failure (world2vec-sigreg on CartPole, ρ = 0.04) coincides with its failed control and its 0% PRC. See W2.

Q5. Residual discrete-time model with the same initialization vs. the Euler model (Table 3). At a fixed step size, an Euler step is equivalent to a residual update, so the gain in Table 4 comes from predicting the state change rather than the next state. The continuous-time view additionally enables retiming at test time without retraining (Fig. 5) and makes SINDy coefficients interpretable as governing equations (Fig. 8, STab. 6).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Paper Revision

Dear reviewers and AC,

First, thank you for taking the time to review our paper in depth. We appreciate the many helpful comments and will post individual, point-by-point replies later this week. We want to take the opportunity to share a thorough revision of our paper early on during the rebuttal process that addresses the most important overall comments.

In particular:

Control performance across all three environments (yjDX, tqXK, YuhK). world2vec-dcl now solves all three environments (rewards 826 / 895 / 774 on Pendulum / Reacher / CartPole; CartPole was 298 in the first submission). This was due to a bug in the data collection process we apologize for (the pole was not visible in about 16% of all frames). We fixed this issue in the data collection process. world2vec-dcl now reaches very competitive performance on all environments; we updated Figure 1 and all other tables/figures showing results on the CartPole environment to convey this finding. Please also see our discussion on Novelty & Contribution below. World2vec achieves SOTA for reward-free, reconstruction-free world models -- as a general claim, we now talk about "competitive" performance vs. models like TD-MPC2 and Dreamer, which we believe is warranted given the new results.

Matched evaluation for all methods (yjDX, tqXK, YuhK). All methods are now evaluated on the same held-out test episodes. Hyperparameters are selected on validation EQA, and planners are selected on validation episodes disjoint from the reported test episodes. LeWM is now trained on our random-policy data and plans with its own CEM planner. On Reacher, we also evaluate under LeWM's own (simplified) goal-reaching protocol (new Table 2), where world2vec matches LeWM 25 steps ahead (86.2 vs. 81.6 % success, difference n.s.) and clearly outperforms it 200 steps ahead (80.2 vs. 52.9 %), while planning 3–8x faster. Every cell in Table 1 now reports the mean ± bootstrap SE over seeds, with no single-seed entries.

Metric naming and Physical Rollout Consistency (yjDX, YuhK). The new Physical Rollout Consistency (PRC) metric decodes the true environment state from the predicted latents with an MLP probe. We report it for all methods, including the baselines. It is used for evaluation only and not for model selection. world2vec-dcl reaches 94.3 % PRC at 1 s, compared with 23–54 % for DreamerV3, TD-MPC2 and LeWM. Note that we renamed the metrics: "Equiv. R²" is now Latent Rollout Consistency (LRC) and "Equiv. Acc." is now Equivariance Accuracy (EQA), see Section 2. LRC, EQA and PRC rank the methods similarly (Table 1), although only PRC uses the true state, justifying that the unsupervised LRC and EQA metrics are well suited for model selection. Thanks for this suggestion!

A tighter link between theory and method (yjDX). Thanks for this comment; we fully agree and acknowledge that the paper had gaps in this aspect. We made a thorough revision of Section 2, and added Appendix A with additional details on the objective functions. We also better explain the relation of the world2vec objective function with prior work. While we believe that the identifiability of the method is sufficiently clear from the literature (and now even more prominently attributed) and rigorously empirically tested in our paper, we would be happy to add a Corollary/additional discussion to the paper and will let you know via an individual reply.

Novelty & Contribution (yjDX, tqXK, YuhK). We noticed a number of concerns regarding the novelty, goals and contribution of our work and revised the abstract, Sec. 1 and 2 to clarify this: World2vec is the first successful instantiation of a mechanistic world model in the sense discussed by Posner et al. (2026). This implies (a) good task performance, but also (b) accurate dynamics modeling and (c) the ability to parameterize the dynamics model in a mechanistically interpretable way.

Regarding (a), to our knowledge, there exists no instance of a reward-free, reconstruction-free world model that is able to solve any of the control tasks considered. The new results on CartPole only reinforce this point. Regarding (b), the LRC and EQA metrics are central, which we now validated with the PRC metric against the ground truth state; for (c) we performed the SINDy experiments on which we will post further updates in due time.

We would very much appreciate a re-evaluation of our work on grounds of these points and will engage in 1:1 discussions in due time. Thanks again for taking the time to help us improve the scope and presentation of our work.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.