acceptodds
Under review as a conference paper at ICLR 2027

DWM: Separating World Effects from Actions in Latent World Models

Abstract

Latent world models provide an effective way to model environment dynamics for control, but existing action-conditioned formulations typically supervise the next-latent transition using a single prediction target, requiring the model to explain all sources of state change through one unified learning signal. In real-world environments, however, state transitions often arise from two distinct sources: changes caused by the agent’s actions and changes induced by the environment’s intrinsic dynamics. The latter can occur even under a null action and include effects such as gravity-driven sliding, inertia, contact rebound, and persistent drift. Modeling these two sources jointly can entangle their effects in the learned latent transition, making it difficult to attribute observed changes to their underlying causes and limiting the transferability of the learned dynamics. In this paper, we introduce DWM (Decomposed World Model), which is a supervision-level framework that explicitly separates these two sources of transition. DWM augments a latent world model with an auxiliary world head that captures action-invariant changes and is regularized by a normalized world-contrastive objective. The original prediction head is coupled with the world head through an orthogonality constraint, encouraging the predicted transition to decompose into an action-invariant component and a complementary action-driven component. This formulation requires no modification to the underlying world-model architecture or inference pipeline. To evaluate DWM in environments with persistent world effects, we construct W-variants of three standard control benchmarks, PushT-W, Reacher-W, and TwoRoom-W, each introducing a distinct form of action-invariant dynamics. Across multiple representative latent world models, DWM consistently improves both prediction quality and downstream control performance on these benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
latent world models, self-supervised representation learning, world/action decomposition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nc32OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes the Decomposed World Model (DWM), which separates the world effect arising from the environment's intrinsic dynamics from the action effect caused by the agent's actions in latent world models. The method uses an auxiliary world head, a world-contrastive objective, and an orthogonality constraint to encourage this decomposition. It is evaluated on PushT-W, Reacher-W, TwoRoom-W, and Ball-in-Cup, and improves prediction and planning performance with both LeWM and DINO-WM backbones.

Critical strengths
  1. The motivation is clear: separating dynamics that persist under a null action, such as gravity, inertia, and drift, from the effects of the agent's actions is a reasonable problem formulation.

  2. The method is simple and can be readily integrated into existing world models. DWM preserves the original model's inference and CEM planning pipeline, and the auxiliary world head is used only during training. The authors observe improvements with both LeWM and the structurally different DINO-WM.

  3. The experimental design is relatively comprehensive. In addition to the main experiments, the paper examines the world head's action invariance, 20-step rollouts, OOD gravity, and flat counterparts without persistent world effects, providing evidence for the method's effectiveness.

Critical weaknesses
  1. The current experiments primarily establish that the world head is insensitive to actions. They do not directly demonstrate that its learned representation corresponds to the counterfactual world effect defined in the paper. Evidence for the identifiability of the central decomposition therefore remains insufficient.

  2. The three core W-variants are purpose-built environments with artificially introduced persistent world effects. The evaluation also lacks a direct comparison with closely related methods that separate controllable and non-controllable dynamics, such as Iso-Dream.

Questions for authors
  1. Could the authors use the simulator to construct counterfactual transitions under a null action from the same state and history, and test whether the learned world component agrees with the corresponding ground-truth zero-action transition?

  2. Could the authors include Iso-Dream or other baselines that separate controllable and non-controllable dynamics, and evaluate the method on additional DMControl or robotic tasks that naturally involve gravity, inertia, or contact dynamics? The current evaluation covers only four environments, which is relatively limited compared with typical evaluations of world-model performance.

I would raise my score if the authors satisfactorily address these two questions, particularly by providing the experimental validation requested in Question 1.

10 comments

We sincerely thank Reviewer nc32 for the careful review and concrete suggestions for strengthening the paper. We address your comments below.


W1. The current experiments primarily establish that the world head is insensitive to actions. They do not directly demonstrate that its learned representation corresponds to the counterfactual world effect defined in the paper. Evidence for the identifiability of the central decomposition therefore remains insufficient.

A1.  Thank you for this important suggestion. To further investigate what the world head actually learns and whether it converges to a trivial solution, we designed an additional experiment. Specifically, we examine whether the world-head predictions correspond to the actual environment evolution under null actions.

We randomly sampled 256 three-frame histories that were not used for training on PushT. For each history, we saved the complete simulator state, including the positions, velocities, angles, and angular velocities of the pusher and block. Starting from the corresponding state and history, the world head predicts the latent representation after a five-step null-action block. Independently, we restored the same simulator state and executed five consecutive null actions, allowing the environment dynamics to evolve. We then encoded the resulting simulator observation using the same encoder to obtain its latent representation. Finally, we measured the correspondence between the world-head predictions and the simulator-derived latent representations using centered linear CKA. We further normalize the CKA using a reference CKA between two independently sampled sets of random features.

RepresentationRelative CKA ↑
World head at initialization1.31
World head after training18.74

The relative CKA increases substantially from 1.31 to 18.74 after training, indicating a strong correspondence between the world-head representations and the actual environment states reached after five null-action steps. This result therefore provides further evidence that the world head learns meaningful representations of action-independent environment dynamics.


W2. The three core W-variants are purpose-built environments with artificially introduced persistent world effects. The evaluation also lacks a direct comparison with closely related methods that separate controllable and non-controllable dynamics, such as Iso-Dream.(part 1/2)

A2. Thank you for raising these concerns. We address them by expanding our evaluation to additional environments and including a comparison with an Iso-Dream baseline.

Indeed, the W-variants are intentionally designed as controlled diagnostic environments. Each preserves the objective and basic structure of its corresponding standard task while introducing one persistent, action-independent source of motion. This design allows us to isolate the effects of external world dynamics without simultaneously changing the task objective.

To examine whether DWM's benefits extend beyond these purpose-built environments, we expanded our evaluation from four to seven environments by adding three unmodified native tasks:

  • MetaWorld Button-Press: A 3D MuJoCo robotic manipulation task requiring continuous control of the end-effector and gripper to establish contact and depress a button.
  • DMControl Finger-Turn-Easy: A two-joint articulated finger manipulation task involving contact, friction, and inertia, where the finger must rotate a spinner toward a target region.
  • DMControl Cartpole Swing-Up: A control task involving gravity, cart-pole inertia, and joint damping.

Importantly, these are unmodified native environments without artificially introduced external forces or visual distractors. They therefore allow us to evaluate DWM in environments with naturally occurring contact and physical dynamics. For each task, LeWM and DWM use the same offline trajectories, initialization, training-update budget, action normalization, CEM planner, goal observations, and fixed evaluation cases.

TaskEvaluation metricLeWMDWM
MetaWorld Button-PressNative task success (%) ↑7.714.0
DMControl Finger-Turn-EasyNative task success (%) ↑12.722.7
DMControl Cartpole Swing-UpEpisodic return ↑20.732.0

DWM consistently outperforms LeWM across all three additional environments, demonstrating that its benefits extend beyond the original 2D navigation, reaching, and pushing tasks to 3D contact-rich manipulation and control under gravity and inertia.

W2. The three core W-variants are purpose-built environments with artificially introduced persistent world effects. The evaluation also lacks a direct comparison with closely related methods that separate controllable and non-controllable dynamics, such as Iso-Dream.(part 2/2)

Second, to directly compare DWM with related approaches that disentangle controllable and non-controllable dynamics, we implemented an Iso-Dream baseline under the same experimental setting as LeWM. Specifically, following Iso-Dream's controllable/non-controllable dynamics factorization, we use separate action-conditioned and action-free latent transition models, together with inverse-action supervision on the difference between their predictions. This allows us to directly compare the dynamics-factorization principle under the same offline data, training protocol, and latent-space CEM planning framework. We also included SD-JEPA, which decomposes the JEPA latent representation into orthogonal progression and content subspaces. Unlike DWM, which separates the causes of the predicted transition into an action-invariant world component and a complementary action-related residual, SD-JEPA separates what the latent state represents (task progression versus content).

All methods use the same offline trajectories, initialization, training-update budget, latent-space CEM planner, and evaluation protocol. Success rates are averaged over three planner seeds.

MethodPushT-WReacher-WTwoRoom-W
LeWM32.060.060.0
Iso-Dream28.758.763.3
SD-JEPA34.762.066.0
DWM44.070.776.7

The Iso-Dream factorization does not consistently improve over LeWM, while SD-JEPA provides modest gains. DWM consistently outperforms both decomposition baselines, with improvements over SD-JEPA of 9.3, 8.7, and 10.7 percentage points on PushT-W, Reacher-W, and TwoRoom-W, respectively. These results demonstrate that DWM's improvements are not simply reproduced by existing controllable/non-controllable dynamics factorization or orthogonal latent-subspace decomposition.

Taken together, these additional experiments demonstrate that DWM's benefits extend beyond artificially constructed W-variants and that its improvements cannot be reproduced by generic controllable/non-controllable dynamics factorization alone. They further support the effectiveness of explicitly learning action-independent world effects for planning under external dynamics.


Q1. Could the authors use the simulator to construct counterfactual transitions under a null action from the same state and history, and test whether the learned world component agrees with the corresponding ground-truth zero-action transition?

A3.  Thank you for this important suggestion. To further investigate what the world head actually learns and whether it converges to a trivial solution, we designed an additional experiment following your suggestion. Specifically, we examine whether the world-head predictions correspond to the actual environment evolution under null actions.

We randomly sampled 256 three-frame histories that were not used for training on PushT. For each history, we saved the complete simulator state, including the positions, velocities, angles, and angular velocities of the pusher and block. Starting from the corresponding state and history, the world head predicts the latent representation after a five-step null-action block. Independently, we restored the same simulator state and executed five consecutive null actions, allowing the environment dynamics to evolve. We then encoded the resulting simulator observation using the same encoder to obtain its latent representation. Finally, we measured the correspondence between the world-head predictions and the simulator-derived latent representations using centered linear CKA. We further normalize the CKA using a reference CKA between two independently sampled sets of random features.

RepresentationRelative CKA ↑
World head at initialization1.31
World head after training18.74

The relative CKA increases substantially from 1.31 to 18.74 after training, indicating a strong correspondence between the world-head representations and the actual environment states reached after five null-action steps. This result therefore provides further evidence that the world head learns meaningful representations of action-independent environment dynamics.

Q2. Could the authors include Iso-Dream or other baselines that separate controllable and non-controllable dynamics, and evaluate the method on additional DMControl or robotic tasks that naturally involve gravity, inertia, or contact dynamics? The current evaluation covers only four environments, which is relatively limited compared with typical evaluations of world-model performance.(1/2)

A4. We expanded our evaluation in two directions: adding closely related dynamics-decomposition baselines and evaluating DWM on additional environments with native physical dynamics.

(1) Comparison with related decomposition baselines. We added two closely related baselines, Iso-Dream and SD-JEPA, and evaluated them on all three W-tasks.

We implemented an Iso-Dream baseline under the same experimental setting as LeWM. Specifically, following Iso-Dream's controllable/non-controllable dynamics factorization, we use separate action-conditioned and action-free latent transition models, together with inverse-action supervision on the difference between their predictions. This allows us to directly compare the dynamics-factorization principle under the same offline data, training protocol, and latent-space CEM planning framework. We also included SD-JEPA, which decomposes the JEPA latent representation into orthogonal progression and content subspaces. Unlike DWM, which separates the causes of the predicted transition into an action-invariant world component and a complementary action-related residual, SD-JEPA separates what the latent state represents (task progression versus content).

All methods use the same offline trajectories, initialization, training-update budget, latent-space CEM planner, and evaluation protocol. Success rates are averaged over three planner seeds.

MethodPushT-WReacher-WTwoRoom-W
LeWM32.060.060.0
Iso-Dream28.758.763.3
SD-JEPA34.762.066.0
DWM44.070.776.7

The Iso-Dream factorization does not consistently improve over LeWM, while SD-JEPA provides modest gains. DWM consistently outperforms both decomposition baselines, with improvements over SD-JEPA of 9.3, 8.7, and 10.7 percentage points on PushT-W, Reacher-W, and TwoRoom-W, respectively. These results demonstrate that DWM's improvements are not simply reproduced by existing controllable/non-controllable dynamics factorization or orthogonal latent-subspace decomposition.

Reviewer_4brDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper notes that state transitions in latent-space world models are affected by two distinct sources: changes induced by the agent’s actions and by the environment (e.g. gravity, inertia). The paper introduces DWM (Decomposed World Model), which disentangles these factors by adding an auxiliary world head and contrastive and orthogonality loss terms at training time, while the inference pipeline is unchanged. The authors evaluate their proposed method in environments with world effects, and show that the proposed modification is applicable to different latent world models, and outperforms the world models without this disentanglement.

Critical strengths
  • The authors apply the idea of learning controllable and non-controllable dynamics to recent latent world models learned from observations, and show that the proposed method, DWM, clearly outperforms the baselines and the original models without explicit separation of world and action effects, while being simple and easy to understand.
  • The experimental results show evidence that the trained model transfers to OOD gravities (Figure 4), the auxiliary world head is action-invariant (Table 1) and the proposed method is applicable to different world models, LeWM and DINO-WM (Figures 3 and 5). The proposed method preserves performance in environments without world effects (Figure 6).
Critical weaknesses
  • It is not obvious why the world and action effects should be orthogonal, when they might, for instance, interact or affect the same physical dimension. Furthermore, the effect of the orthogonality penalty is evaluated only on a single benchmark, where the benefit from orthogonality is 4% (Table 5). The contrastive objective only encourages the auxiliary head to produce action-invariant representations. However, these might not correspond to the null-action counterfactuals, but to, for instance, the expected transition under the behavioral policy or some function of the history. This is arguably a mismatch between the motivation and the presented method.
  • The empirical evaluation could be broader: 4 tasks is a rather limited set, and the task selection is not the most challenging (PushT, TwoRoom, Reacher). Whether the performance transfers to more complex (e.g., 3D) manipulation environments is not shown (both with and without world effects). Finally, it is not clear from the paper whether a single training run was performed or if the uncertainty is only over the planning seeds.
Questions for authors
  • How many training seeds were used, and did you encounter any training instabilities due to using these additional loss components?
  • Could you clarify the argument or present additional evidence that the orthogonality component is necessary, and that the learned z_w actually corresponds to a null-action counterfactual, rather than e.g. the expected transition under the behavioral policy?
4 comments

We sincerely thank Reviewer 4brD for the thoughtful review and constructive feedback.  We address each point below.


Q1. It is not obvious why the world and action effects should be orthogonal, when they might, for instance, interact or affect the same physical dimension?

A1. Thank you for the question. We emphasize that the orthogonality constraint is imposed on learned latent components, not on physical displacements or forces. Therefore, it does not assume that world effects and action effects are orthogonal in the physical space or independent. Gravity and pushing can indeed produce displacements in the same direction, and their nonlinear interactions are not excluded by our formulation.

Instead, the orthogonality constraint serves as a regularizer to encourage the world and action components to capture complementary rather than redundant information. Our motivation is to improve long-horizon generalization, particularly in OOD scenarios where the underlying physical environment changes while the agent's action space remains the same. In such cases, the same action can lead to different state transitions due to changes in environmental dynamics. Explicitly disentangling action-independent world effects from action-dependent effects helps the model distinguish these sources of variation and generalize across different physical environments.


Q2. Furthermore, the effect of the orthogonality penalty is evaluated only on a single benchmark, where the benefit from orthogonality is 4% (Table 5).

A2. Thank you for encouraging a broader evaluation of the orthogonality objective. We extended the evaluation to additional benchmarks to investigate whether orthogonality provides consistent benefits when combined with the world-contrastive objective. Specifically, we compared the contrastive-only variant with the full DWM, which additionally incorporates the orthogonality loss. All methods use the same training and planning protocols.

The additional results show that the orthogonality objective provides further performance gains over contrastive supervision alone across multiple benchmarks. These findings support the complementary role of the two objectives: the world-contrastive objective provides the supervision for learning action-invariant world representations, while the orthogonality loss further encourages the world and action components to capture complementary information. Thus, the benefit of orthogonality is not limited to the single benchmark originally reported in Table 5.

MethodPushT-WReacher-WTwoRoom-WBall in Cup
LeWM32.060.060.036.8
Contrastive only40.066.071.339.6
DWM44.070.776.742.8

Q3. The contrastive objective only encourages the auxiliary head to produce action-invariant representations. However, these might not correspond to the null-action counterfactuals, but to, for instance, the expected transition under the behavioral policy or some function of the history.

A3. Thank you for this important suggestion. To further investigate what the world head actually learns and whether it converges to a trivial solution, we designed an additional experiment. Specifically, we examine whether the world-head predictions correspond to the actual environment evolution under null actions.

We randomly sampled 256 three-frame histories that were not used for training on PushT. For each history, we saved the complete simulator state, including the positions, velocities, angles, and angular velocities of the pusher and block. Starting from the corresponding state and history, the world head predicts the latent representation after a five-step null-action block. Independently, we restored the same simulator state and executed five consecutive null actions, allowing the environment dynamics to evolve. We then encoded the resulting simulator observation using the same encoder to obtain its latent representation. Finally, we measured the correspondence between the world-head predictions and the simulator-derived latent representations using centered linear CKA. We further normalize the CKA using a reference CKA between two independently sampled sets of random features.

RepresentationRelative CKA ↑
World head at initialization1.31
World head after training18.74

The relative CKA increases substantially from 1.31 to 18.74 after training, indicating a strong correspondence between the world-head representations and the actual environment states reached after five null-action steps. This result therefore provides further evidence that the world head learns meaningful representations of action-independent environment dynamics.


Q4. The empirical evaluation could be broader: 4 tasks is a rather limited set, and the task selection is not the most challenging (PushT, TwoRoom, Reacher). Whether the performance transfers to more complex (e.g., 3D) manipulation environments is not shown (both with and without world effects).

A4. We thank the reviewer for this important suggestion.  We have now expanded the evaluation from four to seven environments by adding MetaWorld Button-Press, DMControl Finger-Turn-Easy, and DMControl Cartpole Swing-Up:

  • MetaWorld Button-Press is a 3D MuJoCo robotic manipulation task in which the robot must continuously control its end-effector and gripper, establish physical contact, and depress a passive button.
  • DMControl Finger-Turn-Easy requires a two-joint articulated finger to manipulate a spinner through contact, friction, and inertia until the designated endpoint reaches the target region.
  • DMControl Cartpole Swing-Up evaluates control under gravity, cart–pole inertia, and joint damping.

These are unmodified native environments: we do not introduce artificial external forces or visual distractors. They therefore test whether DWM’s benefit extends beyond the synthetic W-variants to environments containing native contact and physical dynamics.

For each added task, LeWM and DWM use the same offline trajectories, initialization, training-update budget, action normalization, CEM planner, goal observations, and fixed evaluation cases. The only difference is the proposed dynamics-factorization objective.

TaskEvaluation metricLeWMDWM
MetaWorld Button-PressNative task success (%) ↑7.714.0
DMControl Finger-Turn-EasyNative task success (%) ↑12.722.7
DMControl Cartpole Swing-UpEpisodic return ↑20.732.0

DWM improves over LeWM on all three additional environments. The gains extend to 3D contact-rich robotic manipulation in Button-Press,articulated contact dynamics in Finger-Turn-Easy, and gravity- and inertia-dominated control in Cartpole Swing-Up. These additional results show that DWM’s advantage is not restricted to the original 2D navigation, reaching, and pushing benchmarks.


Q5. It is not clear from the paper whether a single training run was performed or if the uncertainty is only over the planning seeds. How many training seeds were used, and did you encounter any training instabilities due to using these additional loss components?

A5. Thank you for the questions. In the original main experiment, each method used one trained world-model checkpoint, and the reported error bars reflected variation across three CEM planner seeds. To further evaluate the robustness of our results to world model training randomness, we conducted additional experiments on PushT-W, Reacher-W and TwoRoom-W independently training both LeWM and DWM with three different random seeds. The results are reported below:

LeWM / DWMTraining Seed 1Training Seed 2Training Seed 3Std
PushT-W32% / 44%32% / 42%30% / 46%1.15 pp / 2.00 pp
Reacher-W60% / 72%56% / 70%62% / 70%3.06 pp / 1.15 pp
TwoRoom-W62% / 78%64% / 78%58% / 74%3.06 pp / 2.31 pp

The results show relatively small variations across training seeds, while DWM consistently outperforms LeWM in all these runs. This further demonstrates that DWM's performance gains are robust to world model training randomness.

Q6. Could you clarify the argument or present additional evidence that the orthogonality component is necessary, and that the learned z_w actually corresponds to a null-action counterfactual, rather than e.g. the expected transition under the behavioral policy?

A6. Thank you for this important question. The world-contrastive objective encourages the world head to be invariant to the current action. However, this objective alone does not prevent the complementary residual from redundantly encoding the same world-effect information. The orthogonality loss addresses this issue by discouraging representational overlap between the world head and the residual, encouraging them to capture complementary aspects of the transition. It therefore plays a complementary role in achieving a cleaner world/action decomposition.

To further evaluate the contribution of orthogonality, we extended our experiments to additional benchmarks, comparing the contrastive-only variant with the full DWM, which additionally incorporates the orthogonality loss. All methods use the same training and planning protocols. The results are reported below:

MethodPushT-WReacher-WTwoRoom-WBall in Cup
LeWM32.060.060.036.8
Contrastive only40.066.071.339.6
DWM44.070.776.742.8

The full DWM consistently outperforms the contrastive-only variant across all four benchmarks, demonstrating that orthogonality provides additional benefits beyond contrastive supervision alone. These results support the complementary roles of the two objectives: contrastive supervision encourages action invariance, while orthogonality further reduces representational overlap between the two components.

Furthermore, to directly investigate whether the learned world representation corresponds to a null-action counterfactual rather than the expected transition under the behavioral policy, we designed an additional simulator-based experiment on PushT. We randomly sampled 256 held-out three-frame histories that were not used for training. For each history, we saved the complete simulator state, including the positions, velocities, angles, and angular velocities of the pusher and block. Starting from the corresponding state and history, the world head predicts the latent representation after a five-step null-action block. Independently, we restored the same simulator state and executed five consecutive null actions, allowing the environment dynamics to evolve. We then encoded the resulting simulator observation using the same encoder to obtain its latent representation. Finally, we measured the correspondence between the world-head predictions and the simulator-derived latent representations using centered linear CKA. To normalize the CKA, we computed a reference CKA between two independently sampled sets of random features.

RepresentationRelative CKA ↑
World head at initialization1.31
World head after training18.74

The relative CKA increases substantially from 1.31 to 18.74 after training, indicating a strong correspondence between the world-head representations and the actual environment states reached after five null-action steps. This provides direct evidence that the learned world representation corresponds to the environment's evolution under null actions, rather than merely capturing the expected transition under the behavioral policy.


We sincerely thank the reviewer again for the thoughtful comments and constructive suggestions. Your feedback helped us clarify the motivation and roles of the contrastive and orthogonality objectives, provide direct counterfactual evidence for the learned world representation, and substantially broaden and strengthen our empirical evaluation. We hope that the additional analyses, experiments, and clarifications above have addressed your concerns, and we greatly appreciate your consideration of our work.

Thank you for the response. I will retain my score

Reviewer_12uJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies action-conditioned latent world models (LeWM, DINO-WM) and argues that, because they use a single next-latent prediction target, they conflate changes caused by actions with changes that occur regardless of the action (gravity-driven sliding, drift, etc., which the authors call world effects), thereby harming prediction and planning. The authors propose DWM: during training, a world head is attached in parallel to the predictor's intermediate state; a contrastive loss requires its output to remain unchanged when the current action is replaced while staying discriminative across samples, and an orthogonality loss requires the world component to be orthogonal to the residual (full prediction minus world component). The world head is removed at inference. On self-constructed PushT-W, Reacher-W, and TwoRoom-W, as well as DMC Ball-in-Cup, the authors report that DWM improves CEM planning success and reduces prediction error for both LeWM and DINO-WM.

Critical strengths

The method is simple to implement: it only adds an auxiliary head and two losses during training, leaves the inference pipeline unchanged, and does not hurt performance on the original tasks without world effects. If the claimed problem truly existed, such a low-cost approach would be of practical value. However, given the weaknesses below, this strength is not sufficient to support acceptance.

Critical weaknesses
  1. The problem the paper sets out to solve is not established. The paper's premise is that "a single next-latent target prevents the model from handling world effects", but this lacks evidence at three levels. First, the only diagnostic experiment is conducted on PushT-W; for Reacher-W, TwoRoom-W, and Ball-in-Cup the paper only reports the baseline's drop and DWM's gain, and a gain by itself does not imply that the problem exists, since a generic regularization effect of the auxiliary losses could produce the same gain. Second, the diagnosis does not hold even on PushT-W: the drop from 94% to 32% is confounded with the task itself becoming harder, and the paper gives no upper bound obtained by planning with the true dynamics; the construction of the zero-action data experiment is not described, and only a single ratio and a single training run are tried; the paper also never directly shows that the baseline mispredicts gravity-driven sliding. The middle row of Figure 1 shows executed trajectories after planning, not a comparison of predicted versus true trajectories under a null action, and the latent errors in Table 2 are not directly comparable because the two models have different encoders. The world effect in TwoRoom-W is just a constant per-step drift that a single-head predictor can represent with a bias term, yet the baseline still drops from 94% to 60%, which suggests a harder planning task rather than a flaw in the supervision target. Third, both base models are small (a ViT-tiny trained from scratch and a frozen DINOv2 ViT-S), the context is only 3 frames, PushT-W is trained for only 5 epochs, and the baseline hyperparameters are not tuned for the new environments, whereas DWM's loss weights were selected on PushT-W by planning success. The paper does not test whether larger, better-trained, or longer-context models still exhibit the problem. This weakness determines my assessment: since the problem itself is not demonstrated, the significance of the method cannot be established.

  2. There is no evidence that the method learns the decomposition it claims, and the experimental results do not support the conclusions. Section 2.2 defines the world effect as the expected change when the current action is replaced by the null action, yet the null action is never used in training; the world head is only required to be insensitive to randomly substituted actions and discriminative across samples, and a trivial solution is to encode the current state itself. Moreover, the definition is stated on the change Δz\Delta z, while the implementation splits the next-latent prediction z^\hat{z}. Table 1 only shows that the world head is insensitive to the action, not that it corresponds to the world effect. The orthogonality constraint has no physical justification, as displacements caused by gravity and by pushing can point in the same direction, and the authors acknowledge that the two may couple nonlinearly. Since the world head is discarded at inference, any gain must come from some regularization of the representation by the auxiliary losses, which the paper neither analyzes nor controls for with an auxiliary head of equal size but an unrelated objective. In the OOD experiment, a model trained under a single gravity achieves exactly 44.0% under two gravities that differ in both direction and magnitude; the authors offer only the cosine similarity of the action component as an explanation and do not report the error of the full prediction under OOD gravity. Error bars in the main experiments reflect only planner seeds, and each world model appears to be trained once; 32% versus 44% corresponds to 16 versus 22 successes out of 50 per seed, and the differences of 40%, 42%, and 44% in the ablation likely lie within noise. This weakness means that, even if the problem exists, the paper does not show that DWM solves it in the way it claims.

Questions for authors

Clarifications:

  • The exact construction of the zero-action data experiment: how the data were collected, whether the ratio is computed per frame or per trajectory, and whether the number of training epochs was kept the same.
  • The fraction of PushT-W training frames in which the pusher actually contacts the block. If most frames already show purely gravity-driven sliding, adding zero-action frames provides no new information.
  • Why the success rates are exactly identical under the three gravities in the OOD experiment, together with the one-step and multi-step prediction errors under OOD gravity and the per-seed success rates.
  • The relationship between the world-head output and the encoding of the current state, i.e., whether the world head merely encodes the state itself.
  • The differences from Iso-Dream, Iso-Dream++ (Model-Based Reinforcement Learning with Isolated Imaginations), and SD-JEPA by Thil et al. SD-JEPA likewise decomposes the LeWM latent space into orthogonal subspaces and is evaluated on Push-T, yet this paper only lists it as a LeWM variant without discussion. Work on separating controllable from exogenous state (the exogenous block MDP of Efroni et al., AC-State of Lamb et al., ACRO of Islam et al.) is also directly relevant to the problem setting.

Experiments that could materially change my assessment:

  • The success-rate upper bound on each W-task obtained by planning with the true simulator dynamics.
  • Whether the baseline still exhibits the problem with longer training, longer context, or a larger model.
  • A control with an auxiliary head of equal size but without the contrastive and orthogonality objectives, and a control using only the orthogonality loss.
10 comments

We sincerely thank Reviewer 12uJ for the detailed review and constructive suggestions. We will first present the results of the three experiments you identified as having the potential to materially change your assessment, followed by point-by-point responses to your remaining concerns.


Q1. The success-rate upper bound on each W-task obtained by planning with the true simulator dynamics.

A1. Thanks for your suggestion. We evaluated planning with the true simulator dynamics on all three W-tasks. The success rates (%) are:

MethodPushT-WReacher-WTwoRoom-W
LeWM32.060.060.0
DWM44.070.776.7
Oracle (true dynamics)70.0100.0100.0

These results show that Reacher-W and TwoRoom-W are perfectly solvable when the true simulator dynamics are available, achieving 100% success on both tasks. Although Oracle achieves a lower success rate of 70% on PushT-W, it still significantly outperforms LeWM baselines. This indicates that the W-tasks are not inherently infeasible due to increased planning difficulty.


Q2. Whether the baseline still exhibits the reported failure with longer training, longer context, or a larger model.

A2. Thank you for suggesting these checks. We examined training duration, context length, and model capacity. All success rates (%) are averaged over three seeds.

(1) Training duration. To examine whether longer training improves baseline performance, we extended training by five additional epochs for all three environments (PushT-W, Reacher-W, and TwoRoom-W), with results averaged over three random seeds. The comparison is shown below:

Training durationPushT-WReacher-WTwoRoom-W
Originally reported32.060.060.0
After 5 additional epochs30.759.360.0

As shown in the table, additional training does not improve planning performance in any of the three environments, supporting the reliability of our originally reported baseline results.

(2) Context length. We doubled the history size from 3 to 6.

History sizePushT-WReacher-WTwoRoom-W
332.0 / 44.060.0 / 70.760.0 / 76.7
630.0 / 44.760.7 / 72.060.0 / 78.0

As shown in the table, longer context does not resolve the LeWM on the W-tasks, and DWM remains consistently better.

(3) Model capacity. We tested three backbone configurations, increasing LeWM’s parameter count from approximately 18.0M to 99.4M.

BackboneLeWM parametersPushT-WReacher-WTwoRoom-W
ViT-tiny~18.0M32.0 / 44.060.0 / 70.760.0 / 76.7
ViT-small~34.5M33.3 / 46.762.0 / 74.062.0 / 82.7
ViT-base99.4M32.7 / 46.060.7 / 74.761.3 / 82.0

As shown in the table, increasing model capacity provides only modest improvements for LeWM, whereas DWM consistently maintains higher success rates.

Taken together, these results suggest that the baseline’s problem in the presence of world effects is not readily resolved by increasing training epochs, model capacity, or context length. DWM outperforms LeWM in every tested configuration. Even the best LeWM results across these settings, i.e., 34.0%, 62.0%, and 62.0%, remain below the original DWM results of 44.0%, 70.7%, and 76.7%, respectively.


Q3. A control with an auxiliary head of equal size but without the contrastive and orthogonality objectives, and a control using only the orthogonality loss.

A3. To distinguish the contribution of additional capacity from that of our objectives, we evaluated two controls:

  • Matched-capacity auxiliary head: We added an auxiliary head of the same size as DWM’s world-effect head and trained it using the planning loss, without the contrastive or orthogonality objectives.
  • Orthogonality-only control: We retained the world-effect head and orthogonality loss but removed the contrastive objective.

The success rates (%) are:

MethodPushT-WReacher-WTwoRoom-WBall in Cup
LeWM32.060.060.036.8
Matched-capacity head32.760.758.736.6
Orthogonality only32.061.362.037.4
DWM44.070.776.742.8

The matched-capacity head provides no consistent improvement, while orthogonality loss alone yields slight gains. Neither control reproduces DWM’s improvements. These results support the importance of contrastive supervision for learning action-independent world effects: DWM’s gains cannot be explained by additional head capacity or orthogonality regularization alone.

We hope these results help address your key experimental concerns. We now turn to your remaining comments and address them one by one.


We first address Weakness 1: “The problem the paper sets out to solve is not established. The paper's premise is that ‘a single next-latent target prevents the model from handling world effects,’ but this lacks evidence at three levels.”


Q4. First, the only diagnostic experiment is conducted on PushT-W; for Reacher-W, TwoRoom-W, and Ball-in-Cup the paper only reports the baseline's drop and DWM's gain, and a gain by itself does not imply that the problem exists, since a generic regularization effect of the auxiliary losses could produce the same gain.

A4. To evaluate whether the performance gains arise from generic regularization effects of auxiliary losses, we conducted additional controlled experiments across PushT-W, Reacher-W, TwoRoom-W, and Ball-in-Cup. Specifically, we retained the world-effect head and orthogonality loss while removing the contrastive objective. This control examines whether an orthogonality regularization alone can account for DWM's improvements. The resulting success rates (%) are reported below:

MethodPushT-WReacher-WTwoRoom-WBall in Cup
LeWM32.060.060.036.8
Orthogonality only32.061.362.037.4
DWM44.070.776.742.8

The results demonstrate that the improvements are substantially smaller than those of the full DWM, indicating that a generic regularization effect of the auxiliary losses is insufficient to explain the observed improvements.


Q5. Second, the diagnosis does not hold even on PushT-W: the drop from 94% to 32% is confounded with the task itself becoming harder, and the paper gives no upper bound obtained by planning with the true dynamics;

A5. We evaluated planning with the true simulator dynamics on all three W-tasks. The success rates (%) are:

MethodPushT-WReacher-WTwoRoom-W
LeWM32.060.060.0
DWM44.070.776.7
Oracle (true dynamics)70.0100.0100.0

These results show that Reacher-W and TwoRoom-W are perfectly solvable when the true simulator dynamics are available, achieving 100% success on both tasks. Although Oracle achieves a lower success rate of 70% on PushT-W, it still significantly outperforms LeWM baselines. This indicates that the W-tasks are not inherently infeasible due to increased planning difficulty.


Q6. The construction of the zero-action data experiment is not described, and only a single ratio and a single training run are tried.

A6. Thank you for pointing out the need for further clarification and evaluation. We provide additional details on the zero-action data construction, and extend the evaluation to multiple mixing ratios with multiple random seeds. Besides, it is worth noting that all results reported in the original paper are already averaged over three independent random seeds.

(1) Zero-action data construction. We define the zero-action ratio per environment decision rather than per trajectory. At each non-initial step, we independently sample a Bernoulli variable. With probability (p), we execute the exact null action ([0,0]); otherwise, we sample an action from the original exploration policy. The simulator then advances normally, allowing gravity, inertia, and contact dynamics to evolve. Thus, the resulting transitions reflect genuine environment dynamics under the executed actions, rather than transitions with retrospectively modified action labels. We exclude the artificial zero-action entry associated with the initial observation of each trajectory when calculating the mixing ratio.

(2) Evaluation across different mixing ratios and random seeds. We evaluate four zero-action mixing ratios: 0%, 10%, 30%, and 50%. For each setting, we independently train the model using three random seeds and report the average success rate. It is worth noting that all results reported in the paper are averaged over three independent runs, and the additional experiments follow the same evaluation protocol. The results are as follows:

Zero-action ratioLeWM success rate (%)
0%32.0
10%30.0
30%30.0
50%26.0

Increasing the proportion of zero-action transitions does not improve the baseline. Its performance remains similar at 10% and 30% and further decreases at 50%. This suggests that insufficient exposure to zero-action transitions alone cannot explain the baseline's performance degradation.


Q7. The paper also never directly shows that the baseline mispredicts gravity-driven sliding. The middle row of Figure 1 shows executed trajectories after planning, not a comparison of predicted versus true trajectories under a null action,

A7. Thank you for pointing this out. We have added a new visualization (https://ibb.co/Dg5TsDCm) that directly compares the predicted trajectory with the ground-truth trajectory under a null action. Specifically, we execute zero actions from the same initial state and compare the trajectory predicted by LeWM with the actual trajectory generated by the environment. Since LeWM does not directly provide decoded observations for visualization, we train an additional decoder to reconstruct the predicted trajectory from its latent representations, following the visualization setup introduced in the LeWM paper. This decoder is used solely for visualization and does not affect the original model or its predictions. The resulting comparison directly illustrates LeWM's difficulty in predicting gravity-driven sliding.


Q8. and the latent errors in Table 2 are not directly comparable because the two models have different encoders.

A8. Thank you for highlighting this concern. Since the two models use different encoders, we revise the prediction error metric to account for differences in their latent scales. Specifically, we normalize the prediction MSE by the average pairwise MSE between randomly sampled latent representations within each model's latent space, enabling a more meaningful comparison. We randomly sample trajectories and use their corresponding 50-step action sequences for evaluation. Starting from the same initial state, each model predicts the latent representation after 50 steps. We then execute the same action sequence in the actual simulator and encode the resulting observation to obtain the ground-truth latent target. The results are reported below:

LeWMDWM
PushT-W0.7550.537
TwoRoom-W0.7040.524
Reacher-W0.6290.447

DWM achieves consistently lower normalized prediction errors across all tasks, demonstrating its advantages in prediction accuracy.


Q9. The world effect in TwoRoom-W is just a constant per-step drift that a single-head predictor can represent with a bias term, yet the baseline still drops from 94% to 60%, which suggests a harder planning task rather than a flaw in the supervision target.

A9. Thank you for raising this concern. As shown in our response to Q1, planning with oracle dynamics achieves 100% success on TwoRoom-W. This demonstrates that the task itself is solvable. Therefore, the performance degradation of LeWM is more likely due to its limitations in modeling the external dynamics, rather than simply the increased difficulty of the planning task. Furthermore, to examine whether DWM's improvements simply result from learning a bias term to capture the constant drift, we evaluate both methods under different OOD drift strengths that are not encountered during training. The success rates (%) are reported below:

Drift settingLeWMDWM
ID (4, 2)60.776.7
OOD: (3.5, 1.75)59.373.3
OOD: (4.9, 2.45)57.3376.0

Here, each pair of values (x, y) represents the per-step drift offsets along the x- and y-axes. If DWM merely learns a fixed bias term to capture the training drift, its performance would be expected to degrade under unseen drift strengths. However, DWM consistently outperforms LeWM in both OOD settings, with improvements of 14.0 and 18.67 percentage points, respectively. These results support the benefit of explicitly modeling world effects for robust planning and generalization.


Q10. Third, both base models are small (a ViT-tiny trained from scratch and a frozen DINOv2 ViT-S), the context is only 3 frames, PushT-W is trained for only 5 epochs, and the baseline hyperparameters are not tuned for the new environments, whereas DWM's loss weights were selected on PushT-W by planning success. The paper does not test whether larger, better-trained, or longer-context models still exhibit the problem.

A10. Thank you for raising this concern. We have systematically examined the effects of training duration, context length, and model capacity in our response to Q1. Specifically, we extended training by five additional epochs, doubled the context length from 3 to 6 frames, and increased LeWM's backbone size from approximately 18.0M to 99.4M parameters. The results in A1 consistently show that none of these changes resolves LeWM's performance degradation on W-tasks, while DWM maintains a clear advantage across all tested configurations.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Summary

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We sincerely appreciate the efforts of the Program Chairs, Senior Area Chairs, and especially the Area Chair in coordinating the review process. We also thank all reviewers for their detailed reviews and constructive suggestions.

We provide this summary to assist the Area Chair in assessing the additional evidence and clarifications presented during the discussion. The main concerns centered on establishing the proposed failure mode, validating the learned world representation, distinguishing the proposed objectives from generic regularization, broadening the evaluation, and clarifying the relationship to existing decomposition methods.

Additional experiments

We conducted the following experiments in response to these concerns:

  • Planning with true simulator dynamics. Following Reviewer 12uJ’s request, we evaluated an oracle planner using the true dynamics on all three W-tasks. It achieves 70.0%, 100.0%, and 100.0% success on PushT-W, Reacher-W, and TwoRoom-W, respectively, compared with 32.0%, 60.0%, and 60.0% for LeWM. These results provide a reference for distinguishing task feasibility from limitations of planning with the learned model.

  • Longer training, longer context, and larger models. We extended baseline training by five epochs, doubled the context from 3 to 6 frames, and increased model capacity from approximately 18.0M to 99.4M parameters. Longer training does not improve the baseline results, and DWM maintains its advantage across all tested context and backbone configurations. These checks directly address Reviewer 12uJ’s concern that the reported behavior might result from insufficient training, context, or capacity.

  • Capacity-matched and objective-ablated controls. We evaluated an equal-sized auxiliary head without the proposed objectives, an orthogonality-only variant, and a contrastive-only variant. Neither additional head capacity nor orthogonality alone reproduces DWM’s gains. Adding orthogonality to contrastive supervision improves results across PushT-W, Reacher-W, TwoRoom-W, and Ball-in-Cup, supporting the complementary contributions of the objectives. These experiments address concerns raised by Reviewers 12uJ and 4brD.

  • Simulator-based null-action validation and CKA ablations. Following all three reviewers’ suggestions, we constructed matched five-step null-action transitions from 256 held-out histories and saved simulator states. The world head’s relative CKA with simulator-derived successor representations increases from 1.31 at initialization to 18.74 after training. In response to Reviewer nc32’s follow-up, we additionally evaluated the ablations: 15.39 for contrastive-only, 9.75 for orthogonality-only, and 18.74 for full DWM. These results provide empirical support for alignment with null-action evolution and for the joint contribution of the proposed objectives.

  • Broader evaluation on native physical environments. Following Reviewers 4brD and nc32, we expanded the native-environment evaluation to 10 DMControl tasks and MetaWorld Button-Press, covering manipulation, balance, locomotion, and swimming. DWM achieves higher task success or episodic return than the matched LeWM baseline on all 11 tasks. For example, success improves from 7.7% to 14.0% on MetaWorld Button-Press and from 12.7% to 22.7% on Finger-Turn-Easy. These unmodified environments broaden the evidence beyond the purpose-built W-variants.

  • Comparisons with closely related decomposition methods. We evaluated an Iso-Dream-inspired offline adaptation and SD-JEPA under matched LeWM training and CEM planning protocols. DWM outperforms both controls on all three W-tasks, exceeding SD-JEPA by 9.3, 8.7, and 10.7 percentage points on PushT-W, Reacher-W, and TwoRoom-W, respectively. These comparisons address the related-work concerns raised by Reviewers 12uJ and nc32.

  • Expanded OOD and full-prediction evaluation. Following Reviewer 12uJ, we enlarged the OOD evaluation set fourfold, evaluated four gravity vectors, and reported scale-normalized one-step and 50-step full-prediction errors, together with per-seed planning results. DWM has lower prediction errors and higher planning success across all four gravity settings. We also evaluated unseen TwoRoom-W drift strengths, under which DWM retains higher success than LeWM.

  • Independent world-model training seeds. Following Reviewers 12uJ and 4brD, we independently trained LeWM and DWM using three training seeds on each W-task and reported standard deviations. DWM outperforms LeWM in every task–training-seed pair. We also clarified that the original main-experiment error bars represented CEM planner-seed variation, distinguishing this from the newly evaluated training randomness.

Additional clarifications and corrections

We also provided the following explanations:

  • Definition and implementation. We clarified the relationship between the state-change definition in Section 2.2 and the next-latent prediction interface in Section 3.1, explaining how the implemented components can be expressed at the increment level without changing the prediction or planning pathway.

  • Role of orthogonality. We clarified that orthogonality regularizes learned latent components to discourage redundant information. It does not assume orthogonal physical forces or exclude nonlinear interactions between world and action effects.

  • Related-work positioning. We explained the distinction between DWM’s transition-effect formulation and the state-factor or progression/content decompositions used in Iso-Dream, Iso-Dream++, and SD-JEPA. We also discussed Exogenous Block MDPs, AC-State, and ACRO, emphasizing that DWM retains environment-driven information that remains relevant to control.

  • Null-action data construction and coverage. We specified that null actions are genuinely executed in the simulator, with mixing ratios defined per decision and matched dataset sizes and training epochs. Additional evaluations at 0%, 10%, 30%, and 50% null-action ratios do not improve the baseline. We also reported pusher–block contact fractions of 41.64% on PushT and 38.92% on PushT-W, documenting that contact interactions remain prevalent.

  • Ablation consistency. We corrected the copy-and-paste error in the contrastive-only results identified by Reviewer nc32 and clarified the corrected comparison. The counterfactual and ablation results are presented as empirical support for the intended decomposition, rather than a proof of unique identifiability.

Discussion outcome and scores

  • Reviewer nc32 acknowledged that our detailed response had “addressed most of my concerns” and increased the rating from 2 to 3.

  • Reviewer 4brD thanked us for the response and retained the rating of 3.

  • Reviewer 12uJ did not provide follow-up feedback throughout the rebuttal period. We provided additional experiments that we believe satisfactorily address the three primary experimental concerns explicitly identified as potentially materially changing Reviewer 12uJ’s assessment: true-dynamics planning, stronger baseline configurations, and capacity-matched/objective-ablated controls. We also provided point-by-point responses to all remaining comments.

We believe these additional experiments and clarifications substantially strengthen the empirical evidence and directly address the major concerns raised in the reviews. Two reviewers currently assign weak-acceptance ratings, while we have not received Reviewer 12uJ’s assessment of the additional evidence. We respectfully ask that the Area Chair consider these results and the discussion feedback alongside the initial reviews.

Thank you once again for your time, careful consideration, and dedication to the community.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.