- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates a failure mode of JEPA-based world models in environments where actions cause only small visual changes. The authors introduce SLIM, a multi-object pushing benchmark with visual and language-conditioned goals, and show that a standard LeWM model loses action-relevant information and struggles with planning. To address this issue, they propose an inverse-dynamics auxiliary loss that improves latent representations and substantially increases planning success without additional inference-time overhead. They further introduce a language-goal head for planning from natural-language instructions, demonstrating promising navigation performance while highlighting remaining challenges in language-conditioned object manipulation.
Critical strengths
-
The paper identifies a meaningful failure mode in JEPA world models, showing that latent representations can lose action-relevant information when only a small portion of the scene changes, even with regularization against representation collapse.
-
The authors provide comprehensive diagnostic experiments and ablations, including action-sensitivity probes, position decodability, encoder-versus-predictor comparisons, and response-to-actions experiments, offering useful insights into why the model fails and how the proposed method improves it.
-
The proposed inverse-dynamics auxiliary loss substantially improves planning performance without adding inference-time overhead, demonstrating a practical way to improve action-relevant representations in latent world models.
Critical weaknesses
W1. Potential Biases and Limited Diversity in Benchmark Construction
SLIM relies on predefined geometric templates and scripted data collection, which may introduce spatial shortcuts and action-distribution biases. In particular, the medium tier admits a 20% success rate from a goal-agnostic policy, while the training and evaluation distributions differ in object counts. These factors raise concerns about whether the benchmark reliably isolates goal-conditioned, object-centric planning ability.
W2. Limited Evaluation Validity for Challenging Planning
The overall performance improvement is dominated by easy navigation (99% success), whereas medium and hard pushing achieve only 24% and 16%, respectively, and language-conditioned pushing performs substantially worse. Furthermore, action sensitivity and latent cost variation do not necessarily reflect accurate action consequences or task-level planning quality, and the strongest cost-alignment evidence comes from navigation rather than object manipulation.
W3. Insufficient Causal Evidence and Limited Methodological Novelty
The proposed explanation attributes representation collapse to limited action-induced visual changes, but the response-to-actions experiments simultaneously modify contact frequency, object dynamics, and task difficulty, making causal attribution difficult. Moreover, inverse-dynamics-based latent shaping has been explored in closely related works, and the additional benefits of the proposed shared encoder–predictor objective remain insufficiently differentiated.
W4. Unclear Motivation and Organization
The paper is somewhat difficult to follow, particularly in the Introduction. The main research question and contributions could be presented more coherently to help readers understand the paper's central focus.
Questions for authors
Regarding W1: Benchmark Construction and Data Bias
Q1. How does planning performance change when the target objects and zones are reassigned within identical scenes? Such counterfactual evaluations would help rule out geometric shortcuts introduced by the predefined templates.
Regarding W2: Evaluation Validity
Q2. Given that the overall improvement is largely driven by easy navigation, what additional evidence supports the effectiveness of the proposed method on challenging goal-directed pushing tasks where goal-agnostic policies fail?
Q3. The simulator-grounded action ranking in Appendix I shows strong correlation for navigation (ρ = 0.84) but weak correlation for pushing (ρ = 0.12). How well do the learned latent costs rank actions in contact-rich pushing scenarios where different actions produce meaningful object displacements?
Regarding W3: Causal Analysis and Novelty
Q4. How can the effects of action-induced visual changes be disentangled from changes in contact frequency, object dynamics, and task difficulty in the response-to-actions experiments?
Q5. In the response-to-actions experiments (Appendix G), the authors simultaneously modify pusher size, object dynamics, and action scale. Could the authors provide controlled ablations that vary these factors independently to better establish what drives the recovery of vanilla LeWM?
Regarding W4: Clarity and Organization
Q6. The Introduction motivates language-conditioned planning but then shifts to representation collapse without a clear transition. It would be helpful to clarify how these two problems are connected and better distinguish the paper's primary contribution from its language-conditioned planning extension.
4 comments
(2/2)
W2/Q2 – Evidence on challenging pushing where goal-agnostic policies fail. (i) Hard tier: 0.157 vs 0 for the goal-agnostic policy and 0.003 for plain. (ii) Given stage frames, the same world model reaches 0.55 on hard. (iii) With stage frames, reassigned push goals are reached in 0.14 / 0.12 of medium / hard trials, where goal-agnostic policies score 0 (above). (iv) Action consequences: on fresh episodes never used in training, LeWM+I's 10-step rollouts are closer to the true latents than copying the current latent in 91/100 sequences (plain: 0/100), so its rollouts follow the executed actions better than a static prediction does.
Q3 – Cost ranking in contact-rich pushing. With the pusher in contact behind the target object, the 64 action sequences from the CEM proposal spread the simulator cost by a median of 29–30 px per scene, and about half of them move the object by more than 5 px. The LeWM+I latent cost ranks them like the simulator: ρ = 0.72 ± 0.01 on medium and 0.73 ± 0.01 on hard, significant in 30/30 scenes each (plain −0.01 and +0.03). With reassigned goals, i.e. directions and distances outside the template: 0.66 and 0.70. The 0.12 in App. I came from the tier start states, where the same actions move the object by a median of 2.6 px.
W3/Q4/Q5 – Causal evidence from the sweep. We agree that lo/hi change several factors at once, and they also make long pushes easier for policies that do not use a learned model. Following Q5 we now train four single-factor plain cells (same data volume, training seed and budget; one seed each) and measured their model-free properties first:
| arm (default 15-px objects) | what changes | changed frame | scripted, very hard | random policy |
|---|---|---|---|---|
| default | – | 0.35% | 0.28 | 0 |
| halo | disc of radius 40 px drawn around the pusher, under zones and objects; physics unchanged | 1.41% | 0.28 | 0 |
| pusher | pusher radius 40 | 1.20% | 0.84 | 0 |
| light | object mass 1, damping 0.95 | 0.36% | 1.00 | 0 |
| speed | action scale 80 | 0.41% | 0.82 | 0 |
| lo / hi (paper) | pusher 28 / 40, object mass 0.5 / 1, damping 0.95, action scale 80 (hi = the three arms above combined) | 1.10% / 1.65% | 1.00 / 1.00 | 0.04 / 0.045 |
The halo arm addresses Q4 directly: it raises the fraction of the frame that responds to actions fourfold while contact frequency, object dynamics and task difficulty stay exactly the default ones. The light and speed arms change object dynamics or control with almost no change in the fraction. Results follow as new replies (halo and pusher on Nov 13–14, light on Nov 15, speed on Nov 16). Until then the revision states the confound explicitly (§5.4, App. G).
W3 – Novelty of the shared objective. Please see "Positioning" in the general response. The shared predictor-side term is a modest complement (+5 pp over encoder-IDM alone across three seeds; +7.8 pp on the canonical seed, McNemar p < 1e-6), not the source of the repair.
W4/Q6 – Introduction. Three concrete changes: (1) it opens with the research question: when and why a JEPA world model stops being plannable, and how to repair it; (2) a new paragraph explains the connection: language goals need scenes in which language discriminates between several objects, several small objects are exactly the regime in which little of the frame responds to actions, and since a language goal enters only through the planner's latent cost, nothing downstream could work until the latent was repaired; (3) the contribution list now puts diagnosis and repair first, SLIM second, and the language head last, as the application. The abstract follows the same order.
Thank you for the additional experiments. I appreciate the authors' efforts, which have addressed most of my original concerns.
My main concern is still the paper's overall positioning and the scientific motivation is still somewhat unclear. The introduction does not clearly explain why the paper poses the broad research question of "when such a model stops being plannable, why, and how to repair it," yet focuses exclusively on one specific failure mode involving limited action-induced visual changes, without sufficiently motivating this particular focus or clarifying its significance within the broader challenges of world-model planning.
Moreover, I appreciate that the paper listed valuable empirical findings, but these observations are not clearly synthesized into a focused scientific question or generalizable insights. I believe the paper would benefit from a more coherent narrative that better connects its motivation, empirical analysis, and main contributions. I keep my original score.
Response to the comment on positioning and motivation
Thank you for the follow-up. We agree, and our final results changed the answer: the explanation the introduction gave for the failure ("little of the frame responds to actions") does not hold, and we withdraw it (top-level note). The introduction of today's revision therefore drops the broad question ("when such a model stops being plannable, why, and how to repair it") and asks only the narrower one the paper answers.
Focused question. The introduction now asks what such a model keeps when the scenes contain much that the actions do not change, as realistic scenes do, and how to keep it plannable. PushT's background and goal region are the same in every episode, so it never poses this question. SLIM's zones and colours differ between episodes, as layouts and backgrounds do in realistic scenes.
How the findings answer it.
- What the model keeps (App. O). The collapsed encoder codes content that differs between episodes but is fixed within one, and drops what the actions move: recolouring a zone moves its latent by 72–84% of the distance between two scenes, moving the pusher by 100 px by less than 5%.
- Removing this content removes the collapse (App. P). With the zones and colours removed from the rendering, vanilla LeWM plans in all four finished runs (89 to 119 of 400, two of them render-only), against 2/400 at the default. Removing only the zones' variation is not reliable (one of three runs plans). Adding per-episode zones to PushT cuts its action spread 30- to 140-fold.
- How to keep the latent plannable (§5.3). One inverse-dynamics loss, which makes the latent identify the action, repairs it (0.348 ± 0.016), and the controls attribute the repair to the encoder-side term.
- How to detect the failure before planning (§5.4). Across 50 trained models, every model below the action-sensitivity gate failed (a necessary, not a sufficient condition).
Generalisable insight. An isotropy regulariser can be met by episode-constant content at no cost to the prediction loss, since copying the current latent already predicts such a latent, and such a latent carries no action information. The interventions suggest that this route opens when episodes differ in much that the actions do not change (SLIM; PushT with added zones); plain escapes in some arms with a larger pusher (pusher 40, lo, hi), for reasons we have not isolated. PushT has almost none of this variation; realistic scenes, if our reading holds, pose the same risk. That SIGReg is met this way is our reading of the interventions, not a tested mechanism.
Narrative. Language remains what made us build SLIM (scenes in which language must discriminate between objects), and the language head remains an application of the repaired latent. The revised abstract, introduction, §5.4 and conclusion now follow the question above.
(1/2) Thank you for the review and the concrete suggestions. We ran the reassignment control you asked for (Q1), started the single-factor sweep cells (Q5; results follow) and restructured the introduction. New results are summarised in the general response.
W1/Q1 – Geometric shortcuts; reassignment in identical scenes. We ran exactly this (new App. M). In each of the 200 evaluation scenes everything stays identical; the first distractor zone becomes the target zone and, on the pushing tiers, the first distractor object becomes the target object (where the scene has one: a new object in 27 / 35 of 50 medium / hard scenes), and the goal frame is rendered for the new pair. LeWM+I (canonical seed), 100 trials per cell:
Navigation follows the reassigned zone, while the goal-agnostic policy, which exploits the template, collapses. On pushing the planner does not fall back to the template goal (at most 1/100): in episodes with a new target object the pusher moves away from the template object (median 95 / 81 px) and ends closer to the new object than it started in 44% / 50% of them, against 7% / 14% for the template object. With subgoal frames it pushes the reassigned object into the reassigned zone (0.14 / 0.12), where the goal-agnostic policy scores 0; this also addresses the medium-tier concern, since on reassigned medium scenes the goal-agnostic policy scores 0.00. Reassigned pushes are harder for every controller: they start beside the object and are 1.7–2× as long (object–zone median 181–185 px vs 94–105 px on medium / hard), which cuts even the privileged scripted controller to 0.60 / 0.62; with a single goal frame our planner does not reach them, as on the very-hard tier. With the pusher in contact behind the reassigned object, the latent cost still ranks pushes like the simulator (ρ = 0.66 / 0.70, significant in 30/30 scenes).
W1 – Object counts and data bias. Training scenes have 3–6 objects, evaluation scenes 1–3. On evaluation scenes with 3–6 objects LeWM+I reaches 0.330 vs 0.353 (p = 0.55; hard 0.11 vs 0.16; App. A). The world-model training data never use the tier generator, and every evaluation scene is mapped through one of eight arena symmetries, so a directional bias of the scripted data cannot line up with goals systematically; the hindsight goal-agnostic row bounds what such a bias could achieve. Plain is trained on exactly the same data and fails.