acceptodds
Withdrawn from ICLR 2027

DasWM: Disentangling Dynamic and Static Factors for Latent World Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce DasWM, a world model representation that softly disentangles dynamic, action-relevant factors from static ones, allowing the allocation of prediction capacity to the dynamic parts. Latent world models make visual planning efficient by encoding observations (e.g. pixels) into compact latent representations. However, not all encoded information needs to be subject to prediction. We show in this work that recently proposed JEPA-like latent world models fail when static visual factors are introduced. We identify the cause of this failure as static-dynamic entanglement in the representation. Our DasWM latent world model encodes each observation as a sequence of tokens, ordered with respect to their dynamic responsiveness. This leads to a representation where dynamic information lies in the early tokens, and static information in the latter ones, minimizing the entanglement. We evaluate DasWM on background-modified versions of several datasets and demonstrate clear improvements in planning, in addition to other benefits such as flexible budget planning at test-time.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, Tokenization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4SYXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that JEPA-style latent world models such as LeWM fail when training data contains static visual variation. The authors attribute the failure to static-dynamic entanglement, meaning the same dynamic transition is encoded differently under different backgrounds. To expose this, they re-render Push-T, Cube and TwoRoom with one of 20 uniform background colors per episode, and show with a variance decomposition of latent changes that LeWM’s representation is dominated by the transition + background interaction. They propose DasWM, which encodes each frame into four causally ordered register tokens. During training, a random prefix of tokens is passed through the predictor and supervised with future-prediction and inverse-dynamics losses, while the suffix is copied and constrained by a persistence loss. The goal is to place action-responsive information in early tokens and static information in later ones. Evaluated with CEM planning on the three background datasets and OGBench-Scene, DASWM outperforms LeWM. The clearest gain is on Cube Background, where Strict success rises from 55% to 74%; gains on the other datasets are smaller. The paper supports this with ablations and a loss-weight sweep on Scene, per-token probes and variance decompositions on Push-T, and planning with only a subset of the tokens. All results come from a single training run per model.

Critical strengths
  1. The background augmentation testbed is cleanly controlled with identical trajectories re-rendered under 20 colors. The paper's strongest element is the latent change decomposition between transition and background, posed as a well-designed diagnostic. However, it characterizes a failure mode whose existence and remedies are already established to some extent.
Critical weaknesses
  1. The problem is known, an existing remedy largely solves it, and the added machinery is not shown to be absolutely necessary. Latent world models encoding predictable but action-irrelevant visual factors is a long-standing problem. Bisimulation, controllable/uncontrollable factorization and inverse-dynamics supervision are existing ways to address it, usually under harder constraints such as natural video. The setting here is especially simple: a uniform color, constant within each episode and therefore identical between start and goal. The paper’s own results show that adding an inverse-dynamics loss from prior work to LeWM recovers most of the failure (Push-T 16.0 to 72.7, TwoRoom 25.3 to 98.7). On top of this, DASWM adds a lot of overhead with ordered register tokens, prefix sampling, a persistence loss, prefix-wise SIGReg and a per-dataset 2-D weight search. Its gain over LeWM+L_inverse is within one standard deviation on three of four datasets, and removing L_inverse drops it below vanilla LeWM (23.3 vs. 24.7). The evidence therefore points to action supervision, not token ordering, as the source of the improvement. The paper also omits comparisons to DINO-Bisim (which it cites), DINO-WM and simple color augmentation, and does not discuss the more likely cause of the failure: SIGReg’s isotropy objective rewards encoding high-variance, easily extracted factors such as episode-level color.

  2. The empirical evidence does not reliably establish the incremental claim. The reported standard deviations reflect only resampling of evaluation pairs. Loss weights were selected on the reported evaluation protocol, with a 36-cell grid for DASWM (Fig. 7) against an apparently 1-D search for the baseline. Core numbers are also inconsistent: for LeWM on Cube with background, Fig. 1b reports 34.7 but Tab. 1 reports 48.0, essentially the original’s 49.3, which undermines the motivating failure on that environment. Tab. 1 also shows Strict > Moderate for LeWM on Push-T Background. Finally, the entanglement analysis (Fig. 2) compares DasWM only to vanilla LeWM, so it cannot attribute the reduced entanglement to token ordering rather than inverse supervision.

Questions for authors
  1. fig. 1b reports 34.7 Moderate for LeWM on Cube with background, but Tab.1 reports 48.0. Which is correct? If it is 48.0, does vanilla LeWM degrade on Cube at all?

  2. How can Strict exceed Moderate for LeWM on Push-T Background (20.0 vs. 16.0)? Are the start–goal pair sets identical across criteria and methods after initial-success filtering?

  3. What grid was searched for the LeWM+L_inverse inverse weight, and was it selected on the same evaluation pairs that are reported?

  4. Why was Tab. 3 run at (λ_persist, λ_inverse) = (6, 0.3) rather than the selected Scene configuration (0.1, 0.1)?

  5. How does the K=4 register encoder with L_inverse, but without the persistence loss or prefix sampling, perform on the three background datasets?

  6. How do DINO-Bisim, DINO-WM and clip-consistent color augmentation perform on these datasets?

Reviewer_PArhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a latent world modeling approach that organizes visual tokens according to their responsiveness to actions. The model predicts an action-sensitive prefix while encouraging the remaining tokens to preserve static information. Experiments on robotic manipulation benchmarks with background variations show improvements over several baselines. However, the current evidence does not fully establish that the proposed representation achieves the claimed disentanglement or that the approach generalizes beyond the evaluated settings.

Critical strengths
  • The paper addresses a relevant problem: world models can be sensitive to irrelevant visual changes, potentially degrading planning performance.

  • The prefix-based representation is conceptually simple, and the experiments include background interventions and comparisons with an inverse-dynamics-enhanced baseline.

Critical weaknesses
  • The central disentanglement claim is insufficiently validated. Improved planning performance and changes in latent variance do not directly demonstrate that the learned tokens separate dynamic and static factors. More direct token-level analyses are needed to establish the proposed mechanism.

  • The experimental scope is limited. The background variations are relatively controlled, and the paper does not establish robustness to broader distribution shifts such as changes in lighting, texture, or scene layout. It is therefore unclear how well the approach addresses the more general problem stated in the paper.

  • The contribution of the token ordering remains unclear. The paper should isolate whether the nested representation itself is responsible for the gains, rather than the auxiliary losses or other training choices. Sensitivity to the number of tokens and loss weights also needs stronger evaluation.

  • The empirical evidence is not yet sufficiently broad to support the generality of the conclusions. The reported improvements are encouraging, but additional controlled comparisons and robustness analyses would make the claims more convincing.

Questions for authors
  1. Can the authors provide quantitative evidence that individual tokens or token groups exhibit the claimed separation between action-responsive and persistent information?

  2. How much of the improvement comes from the nested token structure, compared with using the same auxiliary objectives without the proposed ordering?

  3. Does the method remain effective under unseen background changes and more substantial visual distribution shifts?

Reviewer_1T5UOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work presented a latent world model. The key design is that the latents are ordered into a sequence. During training a prefix of tokens of random length is used to predict scene dynamics. As a result, tokens closer to the start of the sequence are encouraged more to model dynamics while tokens away from the start are then encouraged to model the static part of the scenes. Specifically, inverse dynamics supervision is used to enforce the modelling of action information. Experiments show that the proposed model can generalize more robust across backgrounds of different colors.

Critical strengths
  • The paper identifies that existing world models are sensitive to the change of backgrounds that are commonly irrelevant to the tasks.

  • The proposed solution disentangles dynamics from static regions to greatly improve the robustness.

  • Experiments show that the proposed method is effective and indeed the model learns to distribute dynamic info to tokens up front.

Critical weaknesses
  • It seems that most gains of the proposed method are within evaluation variance of the strongest baseline. The results hints that the inverse dynamic loss is the key element here. Thus, the importance of the token ordering is not clear. Also, given that the testing environments are of limited visual complexity, it is hard to say will the advantages of the proposed method be still noticeable when applied to more complex environments.

  • It seems that the performance is sensitive to weight section, so the tuning and selection procedure should be clarified. The authors should clarify how task-specific loss weights were selected. It seems that different weights are applied to different tasks.

  • The work assume that all dynamic regions are task related. However, it seems that this is an unnecessary assumption as inverse dynamics are invented to rule out action-irrelevant regions.

  • The proposed method borrowed multiple components from previous works like register tokens, inverse dynamics, and SIGReg.

  • The paper presentation needs refinement. I cannot find fig 7b. Cube's training length is omitted from A.1.

Questions for authors
  • Please clarify why the Scene token-count ablation uses different weights from the main Scene experiments?

  • How the tuning procedure compared with that used for the baseline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.