- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that JEPA-style latent world models such as LeWM fail when training data contains static visual variation. The authors attribute the failure to static-dynamic entanglement, meaning the same dynamic transition is encoded differently under different backgrounds. To expose this, they re-render Push-T, Cube and TwoRoom with one of 20 uniform background colors per episode, and show with a variance decomposition of latent changes that LeWM’s representation is dominated by the transition + background interaction. They propose DasWM, which encodes each frame into four causally ordered register tokens. During training, a random prefix of tokens is passed through the predictor and supervised with future-prediction and inverse-dynamics losses, while the suffix is copied and constrained by a persistence loss. The goal is to place action-responsive information in early tokens and static information in later ones. Evaluated with CEM planning on the three background datasets and OGBench-Scene, DASWM outperforms LeWM. The clearest gain is on Cube Background, where Strict success rises from 55% to 74%; gains on the other datasets are smaller. The paper supports this with ablations and a loss-weight sweep on Scene, per-token probes and variance decompositions on Push-T, and planning with only a subset of the tokens. All results come from a single training run per model.
Critical strengths
- The background augmentation testbed is cleanly controlled with identical trajectories re-rendered under 20 colors. The paper's strongest element is the latent change decomposition between transition and background, posed as a well-designed diagnostic. However, it characterizes a failure mode whose existence and remedies are already established to some extent.
Critical weaknesses
-
The problem is known, an existing remedy largely solves it, and the added machinery is not shown to be absolutely necessary. Latent world models encoding predictable but action-irrelevant visual factors is a long-standing problem. Bisimulation, controllable/uncontrollable factorization and inverse-dynamics supervision are existing ways to address it, usually under harder constraints such as natural video. The setting here is especially simple: a uniform color, constant within each episode and therefore identical between start and goal. The paper’s own results show that adding an inverse-dynamics loss from prior work to LeWM recovers most of the failure (Push-T 16.0 to 72.7, TwoRoom 25.3 to 98.7). On top of this, DASWM adds a lot of overhead with ordered register tokens, prefix sampling, a persistence loss, prefix-wise SIGReg and a per-dataset 2-D weight search. Its gain over LeWM+L_inverse is within one standard deviation on three of four datasets, and removing L_inverse drops it below vanilla LeWM (23.3 vs. 24.7). The evidence therefore points to action supervision, not token ordering, as the source of the improvement. The paper also omits comparisons to DINO-Bisim (which it cites), DINO-WM and simple color augmentation, and does not discuss the more likely cause of the failure: SIGReg’s isotropy objective rewards encoding high-variance, easily extracted factors such as episode-level color.
-
The empirical evidence does not reliably establish the incremental claim. The reported standard deviations reflect only resampling of evaluation pairs. Loss weights were selected on the reported evaluation protocol, with a 36-cell grid for DASWM (Fig. 7) against an apparently 1-D search for the baseline. Core numbers are also inconsistent: for LeWM on Cube with background, Fig. 1b reports 34.7 but Tab. 1 reports 48.0, essentially the original’s 49.3, which undermines the motivating failure on that environment. Tab. 1 also shows Strict > Moderate for LeWM on Push-T Background. Finally, the entanglement analysis (Fig. 2) compares DasWM only to vanilla LeWM, so it cannot attribute the reduced entanglement to token ordering rather than inverse supervision.
Questions for authors
-
fig. 1b reports 34.7 Moderate for LeWM on Cube with background, but Tab.1 reports 48.0. Which is correct? If it is 48.0, does vanilla LeWM degrade on Cube at all?
-
How can Strict exceed Moderate for LeWM on Push-T Background (20.0 vs. 16.0)? Are the start–goal pair sets identical across criteria and methods after initial-success filtering?
-
What grid was searched for the LeWM+L_inverse inverse weight, and was it selected on the same evaluation pairs that are reported?
-
Why was Tab. 3 run at (λ_persist, λ_inverse) = (6, 0.3) rather than the selected Scene configuration (0.1, 0.1)?
-
How does the K=4 register encoder with L_inverse, but without the persistence loss or prefix sampling, perform on the three background datasets?
-
How do DINO-Bisim, DINO-WM and clip-consistent color augmentation perform on these datasets?