acceptodds
Under review as a conference paper at ICLR 2027

KEEPING JEPA WORLD MODELS PLANNABLE WHEN LITTLE OF THE FRAME MOVES

Abstract

Specifying a goal in language rather than as a goal frame is a natural interface for planning with a latent world model, but testing it needs scenes in which language must discriminate between several objects. We build SLIM, a pushing benchmark with several small objects and paired visual and language goals on identical scenes. On SLIM a LeWM world model that solves PushT succeeds on under 1% of trials, although a scripted controller with simulator state solves every tier. Probes locate the failure in the encoder: its latent is nearly action-insensitive, neither pusher nor object positions can be decoded from it, and rollouts are no better than copying the current latent forward. One inverse-dynamics auxiliary loss, applied to encoder latents and to predicted latents through a shared head discarded at test time, restores every probe and raises success from 0.003 to 0.35 (0.16 on the hard pushing tier, where a goal-agnostic policy scores zero), and improves PushT at twice the trained horizon. Controls attribute the repair to the gradient into the encoder, and a response sweep shows that the vanilla model plans once enough of the frame responds to actions. A cheap action-sensitivity probe, computable without environment access, acts as an empirical necessary condition: all configurations below its threshold failed to plan. On the repaired latent, a small language-goal head plans from sentences without retraining the world model: it reaches 0.84 on navigation (visual-goal oracle 1.00), follows the named zone when it is swapped with a decoy, and degrades gracefully to unseen nouns, while choosing which object to push from language remains open.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, JEPA, model-based planning, representation collapse, inverse dynamics, language-conditioned planning, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PsLhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates a failure mode of JEPA-based world models in environments where actions cause only small visual changes. The authors introduce SLIM, a multi-object pushing benchmark with visual and language-conditioned goals, and show that a standard LeWM model loses action-relevant information and struggles with planning. To address this issue, they propose an inverse-dynamics auxiliary loss that improves latent representations and substantially increases planning success without additional inference-time overhead. They further introduce a language-goal head for planning from natural-language instructions, demonstrating promising navigation performance while highlighting remaining challenges in language-conditioned object manipulation.

Critical strengths
  1. The paper identifies a meaningful failure mode in JEPA world models, showing that latent representations can lose action-relevant information when only a small portion of the scene changes, even with regularization against representation collapse.

  2. The authors provide comprehensive diagnostic experiments and ablations, including action-sensitivity probes, position decodability, encoder-versus-predictor comparisons, and response-to-actions experiments, offering useful insights into why the model fails and how the proposed method improves it.

  3. The proposed inverse-dynamics auxiliary loss substantially improves planning performance without adding inference-time overhead, demonstrating a practical way to improve action-relevant representations in latent world models.

Critical weaknesses

W1. Potential Biases and Limited Diversity in Benchmark Construction

SLIM relies on predefined geometric templates and scripted data collection, which may introduce spatial shortcuts and action-distribution biases. In particular, the medium tier admits a 20% success rate from a goal-agnostic policy, while the training and evaluation distributions differ in object counts. These factors raise concerns about whether the benchmark reliably isolates goal-conditioned, object-centric planning ability.

W2. Limited Evaluation Validity for Challenging Planning

The overall performance improvement is dominated by easy navigation (99% success), whereas medium and hard pushing achieve only 24% and 16%, respectively, and language-conditioned pushing performs substantially worse. Furthermore, action sensitivity and latent cost variation do not necessarily reflect accurate action consequences or task-level planning quality, and the strongest cost-alignment evidence comes from navigation rather than object manipulation.

W3. Insufficient Causal Evidence and Limited Methodological Novelty

The proposed explanation attributes representation collapse to limited action-induced visual changes, but the response-to-actions experiments simultaneously modify contact frequency, object dynamics, and task difficulty, making causal attribution difficult. Moreover, inverse-dynamics-based latent shaping has been explored in closely related works, and the additional benefits of the proposed shared encoder–predictor objective remain insufficiently differentiated.

W4. Unclear Motivation and Organization

The paper is somewhat difficult to follow, particularly in the Introduction. The main research question and contributions could be presented more coherently to help readers understand the paper's central focus.

Questions for authors

Regarding W1: Benchmark Construction and Data Bias

Q1. How does planning performance change when the target objects and zones are reassigned within identical scenes? Such counterfactual evaluations would help rule out geometric shortcuts introduced by the predefined templates.

Regarding W2: Evaluation Validity

Q2. Given that the overall improvement is largely driven by easy navigation, what additional evidence supports the effectiveness of the proposed method on challenging goal-directed pushing tasks where goal-agnostic policies fail?

Q3. The simulator-grounded action ranking in Appendix I shows strong correlation for navigation (ρ = 0.84) but weak correlation for pushing (ρ = 0.12). How well do the learned latent costs rank actions in contact-rich pushing scenarios where different actions produce meaningful object displacements?

Regarding W3: Causal Analysis and Novelty

Q4. How can the effects of action-induced visual changes be disentangled from changes in contact frequency, object dynamics, and task difficulty in the response-to-actions experiments?

Q5. In the response-to-actions experiments (Appendix G), the authors simultaneously modify pusher size, object dynamics, and action scale. Could the authors provide controlled ablations that vary these factors independently to better establish what drives the recovery of vanilla LeWM?

Regarding W4: Clarity and Organization

Q6. The Introduction motivates language-conditioned planning but then shifts to representation collapse without a clear transition. It would be helpful to clarify how these two problems are connected and better distinguish the paper's primary contribution from its language-conditioned planning extension.

4 comments

(1/2) Thank you for the review and the concrete suggestions. We ran the reassignment control you asked for (Q1), started the single-factor sweep cells (Q5; results follow) and restructured the introduction. New results are summarised in the general response.

W1/Q1 – Geometric shortcuts; reassignment in identical scenes. We ran exactly this (new App. M). In each of the 200 evaluation scenes everything stays identical; the first distractor zone becomes the target zone and, on the pushing tiers, the first distractor object becomes the target object (where the scene has one: a new object in 27 / 35 of 50 medium / hard scenes), and the goal frame is rendered for the new pair. LeWM+I (canonical seed), 100 trials per cell:

easymediumhard
canonical goal0.980.270.16
reassigned goal0.960.000.00
template goal reached instead0.000.000.01
reassigned goal, stage frames–0.140.12
goal-agnostic, canonical / reassigned0.12 / 0.040.20 / 0.000.00 / 0.00
scripted controller, canonical / reassigned1.00 / 1.001.00 / 0.601.00 / 0.62

Navigation follows the reassigned zone, while the goal-agnostic policy, which exploits the template, collapses. On pushing the planner does not fall back to the template goal (at most 1/100): in episodes with a new target object the pusher moves away from the template object (median 95 / 81 px) and ends closer to the new object than it started in 44% / 50% of them, against 7% / 14% for the template object. With subgoal frames it pushes the reassigned object into the reassigned zone (0.14 / 0.12), where the goal-agnostic policy scores 0; this also addresses the medium-tier concern, since on reassigned medium scenes the goal-agnostic policy scores 0.00. Reassigned pushes are harder for every controller: they start beside the object and are 1.7–2× as long (object–zone median 181–185 px vs 94–105 px on medium / hard), which cuts even the privileged scripted controller to 0.60 / 0.62; with a single goal frame our planner does not reach them, as on the very-hard tier. With the pusher in contact behind the reassigned object, the latent cost still ranks pushes like the simulator (ρ = 0.66 / 0.70, significant in 30/30 scenes).

W1 – Object counts and data bias. Training scenes have 3–6 objects, evaluation scenes 1–3. On evaluation scenes with 3–6 objects LeWM+I reaches 0.330 vs 0.353 (p = 0.55; hard 0.11 vs 0.16; App. A). The world-model training data never use the tier generator, and every evaluation scene is mapped through one of eight arena symmetries, so a directional bias of the scripted data cannot line up with goals systematically; the hindsight goal-agnostic row bounds what such a bias could achieve. Plain is trained on exactly the same data and fails.

(2/2)

W2/Q2 – Evidence on challenging pushing where goal-agnostic policies fail. (i) Hard tier: 0.157 vs 0 for the goal-agnostic policy and 0.003 for plain. (ii) Given stage frames, the same world model reaches 0.55 on hard. (iii) With stage frames, reassigned push goals are reached in 0.14 / 0.12 of medium / hard trials, where goal-agnostic policies score 0 (above). (iv) Action consequences: on fresh episodes never used in training, LeWM+I's 10-step rollouts are closer to the true latents than copying the current latent in 91/100 sequences (plain: 0/100), so its rollouts follow the executed actions better than a static prediction does.

Q3 – Cost ranking in contact-rich pushing. With the pusher in contact behind the target object, the 64 action sequences from the CEM proposal spread the simulator cost by a median of 29–30 px per scene, and about half of them move the object by more than 5 px. The LeWM+I latent cost ranks them like the simulator: ρ = 0.72 ± 0.01 on medium and 0.73 ± 0.01 on hard, significant in 30/30 scenes each (plain −0.01 and +0.03). With reassigned goals, i.e. directions and distances outside the template: 0.66 and 0.70. The 0.12 in App. I came from the tier start states, where the same actions move the object by a median of 2.6 px.

W3/Q4/Q5 – Causal evidence from the sweep. We agree that lo/hi change several factors at once, and they also make long pushes easier for policies that do not use a learned model. Following Q5 we now train four single-factor plain cells (same data volume, training seed and budget; one seed each) and measured their model-free properties first:

arm (default 15-px objects)what changeschanged framescripted, very hardrandom policy
default–0.35%0.280
halodisc of radius 40 px drawn around the pusher, under zones and objects; physics unchanged1.41%0.280
pusherpusher radius 401.20%0.840
lightobject mass 1, damping 0.950.36%1.000
speedaction scale 800.41%0.820
lo / hi (paper)pusher 28 / 40, object mass 0.5 / 1, damping 0.95, action scale 80 (hi = the three arms above combined)1.10% / 1.65%1.00 / 1.000.04 / 0.045

The halo arm addresses Q4 directly: it raises the fraction of the frame that responds to actions fourfold while contact frequency, object dynamics and task difficulty stay exactly the default ones. The light and speed arms change object dynamics or control with almost no change in the fraction. Results follow as new replies (halo and pusher on Nov 13–14, light on Nov 15, speed on Nov 16). Until then the revision states the confound explicitly (§5.4, App. G).

W3 – Novelty of the shared objective. Please see "Positioning" in the general response. The shared predictor-side term is a modest complement (+5 pp over encoder-IDM alone across three seeds; +7.8 pp on the canonical seed, McNemar p < 1e-6), not the source of the repair.

W4/Q6 – Introduction. Three concrete changes: (1) it opens with the research question: when and why a JEPA world model stops being plannable, and how to repair it; (2) a new paragraph explains the connection: language goals need scenes in which language discriminates between several objects, several small objects are exactly the regime in which little of the frame responds to actions, and since a language goal enters only through the planner's latent cost, nothing downstream could work until the latent was repaired; (3) the contribution list now puts diagnosis and repair first, SLIM second, and the language head last, as the application. The abstract follows the same order.

Thank you for the additional experiments. I appreciate the authors' efforts, which have addressed most of my original concerns.

My main concern is still the paper's overall positioning and the scientific motivation is still somewhat unclear. The introduction does not clearly explain why the paper poses the broad research question of "when such a model stops being plannable, why, and how to repair it," yet focuses exclusively on one specific failure mode involving limited action-induced visual changes, without sufficiently motivating this particular focus or clarifying its significance within the broader challenges of world-model planning.

Moreover, I appreciate that the paper listed valuable empirical findings, but these observations are not clearly synthesized into a focused scientific question or generalizable insights. I believe the paper would benefit from a more coherent narrative that better connects its motivation, empirical analysis, and main contributions. I keep my original score.

AuthorsOpenReview ↗

Response to the comment on positioning and motivation

Thank you for the follow-up. We agree, and our final results changed the answer: the explanation the introduction gave for the failure ("little of the frame responds to actions") does not hold, and we withdraw it (top-level note). The introduction of today's revision therefore drops the broad question ("when such a model stops being plannable, why, and how to repair it") and asks only the narrower one the paper answers.

Focused question. The introduction now asks what such a model keeps when the scenes contain much that the actions do not change, as realistic scenes do, and how to keep it plannable. PushT's background and goal region are the same in every episode, so it never poses this question. SLIM's zones and colours differ between episodes, as layouts and backgrounds do in realistic scenes.

How the findings answer it.

  1. What the model keeps (App. O). The collapsed encoder codes content that differs between episodes but is fixed within one, and drops what the actions move: recolouring a zone moves its latent by 72–84% of the distance between two scenes, moving the pusher by 100 px by less than 5%.
  2. Removing this content removes the collapse (App. P). With the zones and colours removed from the rendering, vanilla LeWM plans in all four finished runs (89 to 119 of 400, two of them render-only), against 2/400 at the default. Removing only the zones' variation is not reliable (one of three runs plans). Adding per-episode zones to PushT cuts its action spread 30- to 140-fold.
  3. How to keep the latent plannable (§5.3). One inverse-dynamics loss, which makes the latent identify the action, repairs it (0.348 ± 0.016), and the controls attribute the repair to the encoder-side term.
  4. How to detect the failure before planning (§5.4). Across 50 trained models, every model below the action-sensitivity gate failed (a necessary, not a sufficient condition).

Generalisable insight. An isotropy regulariser can be met by episode-constant content at no cost to the prediction loss, since copying the current latent already predicts such a latent, and such a latent carries no action information. The interventions suggest that this route opens when episodes differ in much that the actions do not change (SLIM; PushT with added zones); plain escapes in some arms with a larger pusher (pusher 40, lo, hi), for reasons we have not isolated. PushT has almost none of this variation; realistic scenes, if our reading holds, pose the same risk. That SIGReg is met this way is our reading of the interventions, not a tested mechanism.

Narrative. Language remains what made us build SLIM (scenes in which language must discriminate between objects), and the language head remains an application of the repaired latent. The revised abstract, introduction, §5.4 and conclusion now follow the question above.

Reviewer_mHcCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why a JEPA-style latent world model (LeWM) that performs well on PushT fails on the authors' new SLIM benchmark, which contains multiple small objects and paired visual and language goals. The diagnostics suggest that the encoder retains little information about the pusher and object positions and produces latents that are largely insensitive to actions. The proposed remedy, LeWM+I, adds a shared inverse-dynamics head trained on consecutive encoder latents and on current–predicted latent pairs. The head is discarded at test time.

Across three training runs, mean SLIM success improves from 0.003 to 0.348, while PushT success at twice the training horizon rises from 0.53 to 0.83. The ablations indicate that the encoder-side inverse-dynamics objective accounts for most of the improvement. The authors also train a language-goal head on the repaired world model. It performs well on navigation and passes a zone-swap grounding test, although language-conditioned object pushing remains difficult.

Critical strengths
  1. A well-motivated failure case with substantial experimental improvement. SLIM exposes a failure that is not evident from PushT performance alone. LeWM+I raises mean SLIM success from 0.003 ± 0.001 to 0.348 ± 0.016 across three training seeds. The hard pushing tier improves to 0.157, whereas the goal-agnostic baseline scores zero. Encoder-IDM alone reaches 0.298 ± 0.005, and the shuffled-action control remains at 0.002 ± 0.001. These comparisons provide credible evidence that learning action-relevant features in the encoder, rather than simply adding model capacity, drives much of the gain.

  2. Careful diagnostics and informative controls. On SLIM, pusher-position decoding error decreases from 163 to 8.5 pixels in the simulator coordinate frame, and Appendix I reports an improvement in the correlation between latent planning cost and simulator cost from −0.02 to 0.84 on the easy tier. The zone-swap experiment is also convincing evidence that the language head identifies the named target zone rather than relying only on its typical location. The paper is appropriately candid about the remaining failures of language-conditioned pushing and the fact that a more action-sensitive latent is not always plannable.

Critical weaknesses
  1. The gradient flow of predictor-IDM is unclear. Section 4 states that Equation (3) updates the predictor and the inverse-dynamics head, but not the encoder. However, the stop-gradient is shown only on the head's first input, while the predicted latent in its second input is computed from encoder-generated context latents. Unless this context is separately detached for the predictor-IDM loss, gradients can still reach the encoder through the predictor. Appendix A explicitly describes the stop-gradient as applying to the current latent only. This does not necessarily imply an implementation error, but it makes the claimed separation of encoder- and predictor-side supervision in Table 2 difficult to interpret.

  2. The headline improvement should be distinguished from success on challenging pushing and language-conditioned manipulation. Most of the aggregate SLIM success comes from the easy navigation tier (0.990). The visual-goal planner achieves 0.243 on medium pushing, 0.157 on hard pushing, and 0 on very hard. For language goals, the corresponding medium and hard success rates are only 0.06 and 0.04, compared with 0.30 and 0.17 for the visual-goal oracle in Table 3. The zone-swap test establishes useful zone grounding, but it does not demonstrate reliable language-based object selection and pushing. I would frame the language result primarily as a successful navigation proof of concept.

  3. Generalization beyond the main ViT-Tiny setting remains limited. Appendix C shows that LeWM+I restores action sensitivity with a ViT-Small encoder, yet its mean planning success drops to 0.033. In the walled setting (Appendix B), one of five training runs fails to converge and obtains 0.010 success. The paper reports these limitations transparently, and the ViT-Small comparison uses a matched epoch budget rather than matched convergence. Still, they leave open whether the proposed repair is robust to model scale and more difficult environments.

Questions for authors
  1. Could the authors specify the full detachment and gradient paths for predictor-IDM, ideally with short pseudocode for the forward pass used in Table 2? In particular, is the encoder-produced context detached before being passed to the predictor for Equation (3)?

  2. The abstract says that the plain model's rollouts are no better than copying the current latent forward. Section 5.4 instead compares an observed rollout-error ratio of 12.4 against an approximate copying reference of 10. Could the authors report a directly measured copy-forward baseline on the same held-out sequences and with the same normalization, or qualify this claim?

  3. Can the authors further clarify the failure of ViT-Small despite recovered action sensitivity, using the available training-loss and position-decoding diagnostics? Relatedly, in the response-to-actions sweep, several physics parameters change together. How much of the observed improvement can be attributed specifically to the fraction of the frame responding to actions, rather than to changes in the underlying control difficulty?

3 comments

(1/2) Thank you for the detailed review and for spotting the gradient-path issue. New results are summarised in the general response.

W1/Q1 – Gradient flow of predictor-IDM. You are right, and the paper's sentence was wrong. The stop-gradient is applied to the current latent only; the predictor's context latents are not detached, so Eq. 3 also updates the encoder, indirectly through the predictor. The forward pass used for Table 2 (now in App. A):

z      = f(x[0:4])                        # encoder latents, B x 4 x D
z_hat  = p(z[:, 0:3], g(a[:, 0:3]))       # predictions of z[1:4]; context NOT detached
L_pred = mse(z_hat, z[:, 1:4]);  L_sig = sigreg(z)
L_enc  = mse(h(z[:, 0:3], z[:, 1:4]), a[:, 0:3])    # all consecutive pairs; grad -> f, h
L_pid  = mse(h(sg(z[:, 2]), z_hat[:, 2]), a[:, 2])  # last position; grad -> p, g, h and, via z_hat, f
loss   = L_pred + 0.09 * L_sig + L_enc + L_pid

For Table 2 this means the "predictor-IDM only" row has an indirect encoder path. It still stays at plain's level (4/400, p = 0.13), so the attribution to the encoder-side term holds, and the revision now names that term throughout instead of "the gradient into the encoder". To remove the indirect path we are training LeWM+I with the predictor context detached for Eq. 3 (a second predictor pass on sg(z); we verified that no Eq. 3 gradient then reaches the encoder). It finishes around Nov 15. We will compare it with the LeWM+I retrain on the same data (0.323) and the three-seed range (0.315–0.368); a single run resolves the ~5-pp predictor-side contribution only roughly.

Q2 – Copy-forward baseline. Measured on 100 sequences from 1,000 fresh episodes never used in training, with the same normalisation (error at h = 10 over the true per-step motion): copying the current latent scores 2.29 for plain against a rollout error of 25.3, and 3.42 for LeWM+I against 1.88. Plain's rollouts are worse than the copy in 100/100 sequences, LeWM+I's better in 91/100. (On sequences from the last training episodes, where the paper's 12.4 came from, the copy scores 1.62 and 3.23 with the same orderings.) This confirms and sharpens the abstract's claim; the reference value of about 10 in §5.4 was an estimate and was wrong. Plain's predictor does not copy, it drifts: its error grows by about 0.2 per step while the true latent ends only 0.2 from where it started. Revised in the abstract, introduction, §5.4 and Table 4.

(2/2)

W2 – Headline vs. challenging pushing; language as a navigation proof of concept. Agreed: the trial mean is dominated by navigation, and we use hard (0.16; goal-agnostic 0, plain 0.003) as the evidence tier. New pushing evidence: (i) target object and zone reassigned to distractors in identical scenes: no fallback to the template goal, and with stage frames the reassigned object reaches the reassigned zone in 0.14 / 0.12 of medium / hard trials (goal-agnostic 0; App. M); (ii) with the pusher in contact, the latent cost ranks pushes like the simulator (ρ = 0.72 / 0.73, significant in 30/30 scenes; App. I); (iii) given stage frames, the same world model reaches 0.62 / 0.55 on medium / hard. For language we adopt your framing in §6: with a single goal sentence the head is a navigation proof of concept with zone grounding. Given the stage sentences it was trained on, it reaches 0.25 on medium and on hard (goal sentence 0.04 / 0.04, goal-frame planner 0.27 / 0.16 in the same run), about 40% of a planner given stage frames; with unseen nouns this gain largely disappears.

W3/Q3a – ViT-Small and walls. Diagnostics on the ViT-Small checkpoints (App. C, mostly new):

  • Localisation: pusher-position error 30 px vs 8.5 px at ViT-Tiny (object 72 vs 15 px). Navigation success requires the pusher within 30 px of the target, so the latent barely separates success from a near miss, and easy drops to 0.10 (ViT-Tiny 0.99).
  • Multi-step prediction: ViT-Small+I's 10-step rollouts are no closer to the true latents than copying (1.92 vs 1.86; closer in 45/100 fresh sequences), whereas ViT-Tiny+I nearly halves the copy error (1.88 vs 3.42; 91/100).
  • Cost ranking: Spearman ρ = 0.65 vs 0.84 on navigation and 0.46 vs 0.72 on medium from contact.

Action sensitivity is restored, position precision and multi-step accuracy are not. With the runs at hand we cannot separate a 4× larger model at the same 30 epochs from planner settings tuned at ViT-Tiny; a convergence-matched run does not fit into the discussion period. Walls: 4/5 seeds converge (0.303 ± 0.006); the failed seed is flagged by a training-side criterion fixed before evaluation (validation encoder-IDM loss), so it is detectable without evaluating the planner.

Q3b – Frame fraction vs. control difficulty in the sweep. We agree. The lo/hi arms change pusher size, object dynamics and action scale together, and they also make long pushes easier for policies that do not use a learned model. We now train four single-factor plain cells (same data volume, training seed and budget; one seed each) and measured their model-free properties first:

arm (default 15-px objects)what changeschanged framescripted, very hardrandom policy
default–0.35%0.280
halodisc of radius 40 px drawn around the pusher, under zones and objects; physics unchanged1.41%0.280
pusherpusher radius 401.20%0.840
lightobject mass 1, damping 0.950.36%1.000
speedaction scale 800.41%0.820
lo / hi (paper)pusher 28 / 40, object mass 0.5 / 1, damping 0.95, action scale 80 (hi = the three arms above combined)1.10% / 1.65%1.00 / 1.000.04 / 0.045

The halo arm raises the responding fraction fourfold while physics, the scripted controller and chance levels stay unchanged by construction, so it separates the frame fraction from control difficulty; the light and speed arms change control difficulty with almost no change in the fraction. Results (per-tier success, action sensitivity, position decoding) follow as new replies: halo and pusher on Nov 13–14, light on Nov 15, speed on Nov 16. Until then the revision states the confound explicitly (§5.4, App. G).

AuthorsOpenReview ↗

Follow-up: detached context (W1/Q1), ViT-Small (W3/Q3a), responding fraction (Q3b)

W1/Q1 – Detached predictor context. We trained LeWM+I with the predictor context detached for Eq. 3, so that no Eq. 3 gradient reaches the encoder (two training seeds).

  • Seed 42 trains smoothly and plans: 122/400 (0.305; easy 0.97, medium 0.18, hard 0.07, very hard 0), spread/motion 0.58, pusher decoded to 7.5 px. For comparison, LeWM+I with seed 42 reaches 0.315, encoder-IDM alone 0.298 ± 0.005 over three seeds.
  • Seed 3072 did not train stably: its prediction loss rose after epoch 8 and stayed an order of magnitude higher. Its validation encoder-IDM loss ended at 0.73 (LeWM+I 0.21), so by the criterion of App. B it did not converge. It clears the gate (0.43) but fails (2/400).

Removing the indirect path does not remove the repair. Whether the ~5-pp predictor-side contribution passes through that path, two runs cannot tell: the 1-pp difference at seed 42 is below what retraining with the same seed already changes (0.368 → 0.323 for the canonical cell). Added to §5.3 and App. A.

W3/Q3a – ViT-Small.

  • Planner settings do not explain it: a 3× CEM sample budget with 1.5× the iterations and 3× the elites leaves easy at 0.10 and lowers medium from 0.03 to 0.01; a terminal cost gives 0.04 on easy.
  • Training length does: continuing the paper's checkpoint for 15 further epochs (same data, restarted schedule, fresh IDM head; one run) raises success from 0.033 to 0.373 (easy 1.00, medium 0.32, hard 0.17), against 0.348 ± 0.016 for ViT-Tiny. The pusher is now decoded to 6.9 px (before 30 px), and 10-step rollouts beat copying in 92/100 fresh sequences (before 45/100). By the criterion of App. B the original run had not converged (validation encoder-IDM loss 0.58; continued run 0.24).

The original failure thus points to under-training at this scale rather than a limit of the repair; the restarted schedule and fresh head may have contributed, and the continued run is not convergence-matched. LeWM+I at 112 px navigates at 0.73 with a 67-px pusher probe error, so the localisation argument we gave is a correlation, not a cause. Revised in App. C and the Limitations.

Q3b – Responding fraction vs. control difficulty. The single-factor results arrive later than we announced (Nov 13–16). At unchanged physics and difficulty, quadrupling the responding fraction (render-only halo) gives no improvement: 1/400 and 2/400. Light objects alone and action scale 80 alone leave plain collapsed (1/400, 0/400) although they make long pushes easier for the scripted controller (very hard 1.00 and 0.92 within 20 steps, against 0.42). In all six single- and two-factor runs with a 40-px pusher, plain's training was unstable (5 to 37 of 400); among the response arms, plain plans well only in hi (212/400). Neither the responding fraction nor easier control explains the collapse, and we withdraw the explanation in the paper.

What the evidence points to instead: the collapsed encoder codes what differs between episodes but is fixed within one, above all the zones and colours, and not the pusher. With the zones and colours removed from the rendering plain plans in all four finished runs (89 to 119 of 400, two of them render-only), and adding per-episode zones to PushT cuts its action spread 30- to 140-fold in the first 5–10 epochs (top-level note, App. O and P).

Reviewer_tgyFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why a latent JEPA world model that plans successfully on PushT fails when only a small fraction of the frame responds to actions. It introduces SLIM, a controlled 2D benchmark with several small objects and four difficulty tiers, and shows that the vanilla LeWM encoder becomes nearly action-insensitive: pusher and object positions are poorly decodable, predictor rollouts are close to copying the current latent, and the CEM planning cost becomes flat. A shared inverse-dynamics head applied to consecutive encoder latents and predicted latents during training raises SLIM success from 0.003 to 0.348 across three seeds and improves long-horizon PushT performance. A separate language-goal head maps a frozen sentence embedding and the current latent to a goal latent for planning.

Critical strengths
  1. The paper identifies a clear failure mode of JEPA world models. The action-sensitivity, position-decoding, and cost-landscape analyses support the explanation well.

  2. The ablations are extensive. The encoder-only, predictor-only, shuffled-action, and frozen-encoder results help explain where the improvement comes from.

  3. The improvement is large on SLIM. Success increases from 0.003 to 0.348. PushT success at the longer horizon also improves from 0.53 to 0.83.

  4. SLIM provides a useful controlled setting. It makes it easy to study what happens when only a small part of the image changes.

Critical weaknesses
  1. The method itself is not very novel. Inverse dynamics has long been used to preserve action-relevant information, and DA-LeWM applies a very similar idea to LeWM. Earlier work also used inverse dynamics for planning from visual observations.

  2. The evaluation is limited to one world-model architecture and two synthetic 2D pushing environments. It is unclear whether the same failure mode appears in more realistic visual-control tasks.

  3. The method still performs poorly on harder SLIM tasks. Success is only 0.157 on the hard split and 0 on the very-hard split. The scripted controller reaches 1.0 and 0.28.

  4. The language-conditioned results are weak. The method performs well on simple navigation, but success drops to 0.06 and 0.04 on medium and hard pushing tasks.

  5. The action-sensitivity threshold is only demonstrated in this setting. The paper does not show that it is a reliable diagnostic across different models or environments.

Questions for authors
  1. What is the main technical difference between this method and DA-LeWM or earlier inverse-dynamics planning methods?

  2. Does the same failure mode appear in another JEPA architecture or a more realistic control environment?

  3. How stable is the proposed action-sensitivity threshold across different model architectures, image resolutions, and action spaces?

  4. For language-conditioned pushing, do the failures mainly come from the language-goal representation or from the world model and planner?

3 comments

(1/2) Thank you for the careful review and for recognising the failure mode, the extent of the ablations and SLIM as a controlled setting. New results are summarised in the general response; here we answer point by point.

W1/Q1 – Difference to DA-LeWM and earlier inverse-dynamics methods. We agree that inverse dynamics is not new and do not claim the loss as our contribution (revised Related Work).

  • Method. DA-LeWM (arXiv 2026, concurrent) adds an encoder-side IDM at weight 0.1 with a goal-conditioned action head. We use the encoder-side term at weight 1 plus the same head on (sg(ztz_t), z^t+1\hat z_{t+1}). The predictor-side term is a modest complement (+5 pp over encoder-IDM alone, three seeds), not the source of the repair.
  • Regime. DA-LeWM studies convergence speed and cost geometry on PushT, Reacher, Cube and TwoRoom, where LeWM already plans. We identify a regime in which LeWM does not plan at all (3/1200 trials) and show why: the encoder discards pusher and objects when little of the frame responds to actions. At DA-LeWM's weight 0.1 our recipe reaches 32% of the full result (App. H); the pushing tiers keep improving up to weight 1.
  • Controls. A shuffled-action null at identical architecture and compute (0.002), an encoder-vs-predictor disentangle (canonical seed: encoder-IDM alone 0.290, predictor-IDM alone 0.010, both 0.368) and a frozen-encoder cap (DINOv2-S + I 0.163) attribute the repair to the encoder-side term.
  • Earlier work. Learning to poke plans toward goal images with an inverse model; AC-State and ACRO use inverse models to keep the controllable state. We add the failure regime of JEPA world models, its mechanism, and probes that detect it before planning.

W2/Q2 – Other architectures, more realistic environments. We agree this is the main limitation and now state it in the Limitations paragraph; a new environment family does not fit into the discussion period. Within the paper: (i) the collapse is robust for end-to-end trained encoders: ViT-Tiny and ViT-Small (spread/motion 4e-4 and 2e-4), with walls (plain 2/400), without distractors in training (App. J), and at a third or three times the SIGReg weight (App. H); (ii) a DINO-WM-style model, whose frozen DINOv2 backbone cannot collapse, keeps an action-sensitive latent but plans poorly (0.048), and I still helps (0.163); (iii) on PushT plain does not collapse and I helps at twice the horizon (0.53 → 0.83). (iv) A render-only control is being trained: a disc of radius 40 px drawn around the pusher, under all zones and objects, quadruples the fraction of the frame that responds to actions (0.35% → 1.41%) with physics and task difficulty unchanged; we will post whether vanilla LeWM then plans.

(2/2)

W3 – Hard 0.157, very hard 0. We agree these tiers are far from solved; SLIM has tiers from solved to unsolved by design, and we use hard as evidence of goal-directed pushing (LeWM+I 0.157, plain 0.003, goal-agnostic 0). New results: (i) given intermediate subgoal frames (pusher behind the object, object halfway, object in the zone), the same world model reaches 0.62 / 0.55 on medium / hard, so the latent supports goal-directed pushing and what fails is planning toward the end of a push directly from the start; (ii) with the pusher in contact, the latent cost ranks sampled pushes like the simulator (ρ = 0.72 / 0.73, significant in 30/30 scenes). Very hard stays near 0 even with subgoal frames (0.01): its pushes do not fit 20 steps even for the scripted controller (0.28; 1.00 with 60 steps), so it is headroom. We corrected the abstract, which overstated the scripted controller on very hard.

W4/Q4 – Language pushing: the language representation or world model and planner? Both, and the new staged experiments separate them (same evaluation run, App. N):

goal given to the plannermediumhard
goal sentence0.040.04
goal frame0.270.16
stage sentences0.250.25
stage frames0.620.55

The world model's latent is not the main limit: given nearby subgoal frames, the same world model pushes in 0.62 / 0.55 of trials. One limit is the single-goal planning horizon: decomposing the push into stages helps with frames (0.27 / 0.16 → 0.62 / 0.55) and with sentences (0.04 / 0.04 → 0.25 / 0.25). The other is the language head: at fixed stage structure, sentences instead of frames cost 0.30–0.37 (the head recovers 40–45% of the stage-frame planner). The stage sentences name the same object and zone as the goal sentence and only add how the push proceeds; the head was trained on exactly this kind of stage transition, and from rendered stage frames its output lies nearest to the intended stage in 94–100% of pushing queries (two of three head seeds). New vocabulary remains open: with unseen object and zone nouns the pushing gain largely disappears (medium 0.09, hard 0.06).

W5/Q3 – Stability of the threshold. Across the paper's 28 trained configurations (two benchmarks, ViT-Tiny/Small, trained and frozen encoders, walls, loss and SIGReg weights, sweep arms) spread/motion is bimodal: seven lie at 1e-4–0.011 and all fail, the others at ≥ 0.24, none in between. The data thus place the gate only between 0.011 and 0.24; 0.05 lies in that gap. The ratio is scale-free, but we tested one resolution and one 2-D action space, so the revision words the threshold as observed in these settings and recommends recalibrating it with a few trained models elsewhere.

AuthorsOpenReview ↗

Follow-up: render-only control (W2/Q2 iv), retraction (W1), second resolution (Q3), threshold (W5/Q3)

W2/Q2 (iv) – Render-only control. A disc of radius 40 px drawn around the pusher, under zones and objects and at unchanged physics, quadruples the fraction of the frame that responds to actions (0.35% → 1.41%; PushT 0.57%). Vanilla LeWM still does not plan with either of two training seeds: 1/400 and 2/400 successes, spread/motion 2.5e-4 and 1.3e-3, pusher decoded to 162 and 151 px (a constant predictor scores ≈ 165 px).

W1 – Retraction and new diagnosis. Our first reply said that the encoder discards pusher and objects "when little of the frame responds to actions". The halo contradicts this, and we withdraw it. What the revision shows instead (top-level note, App. O and P):

  • The collapsed encoder codes what differs between episodes but is fixed within one (zones, their number, colours) and not the pusher.
  • Removing the zones and colours from the rendering makes plain plannable in all four finished runs (89 to 119 of 400, two of them render-only), against 2/400 at the default.
  • Removing only the zones' variation is not reliable (one of three runs plans), and adding per-episode zones to PushT cuts plain's action spread 30- to 140-fold.
  • That SIGReg is met by this episode-constant content is our reading, not a tested mechanism.

The difference to DA-LeWM stated in our first reply is unaffected by this: a regime in which LeWM does not plan at all, now tied to episode-constant scene content rather than to the responding fraction, and the controls that attribute the repair.

Q3 – Second resolution. Frames are area-downsampled to 112 px before the same ViT-Tiny (64 instead of 256 patches); one training seed each, default physics.

  • Plain collapses as at 224 px: spread/motion 1.2e-3, pusher undecodable (167 px), 0/400.
  • LeWM+I stays action-sensitive (0.43) and plans on navigation (easy 0.73) but hardly pushes (medium 0.11, hard 0.01): trial mean 0.213 against 0.323 for the 224-px cell retrained with the same data and seed (the paper's checkpoint: 0.368). It localises the pusher to 67 px and the objects hardly at all (158 px); one patch spans 64 px of the simulator frame, and its training loss stayed high from the sixth epoch on, so we do not attribute the drop to one cause.

At both resolutions the two models fall on the same sides of the gate. Added to App. C.

W5/Q3 – Threshold. Our first reply said that spread/motion is bimodal, with no configuration between 0.011 and 0.24. This no longer holds. Of the 50 trained models in the threshold figure (App. A), three lie in that interval, and all fail:

  • pusher 40, seed 42 (0.0345, 7/400; training diverged);
  • pusher 40, seed 7 (0.14, 21/400; loss diverged after 13 smooth epochs);
  • pusher 40 + action scale 80 (0.22, 6/400; pusher at chance).

What holds: every model below the gate fails, now 15 of 15, and no model below 0.24 exceeds a trial mean of 0.053. Above the gate success ranges from 0.005 to 0.83. The new above-gate failures include two runs whose action sensitivity returned only after unstable training (zones hidden and zones fixed, App. P; trial means 0.015 and 0.023). The data therefore bound the threshold only from above, by 0.24; 0.05 is our choice below that bound, and the revision says so (§5.4, App. A). In a new setting the threshold should be recalibrated with a few trained models.

Comments on the paper

3 comments
AuthorsOpenReview ↗

New evidence and revision

We thank the reviewers for careful and constructive reviews, and for recognising the failure mode and its diagnostics, the extent of the ablations and controls, and SLIM as a controlled setting. We uploaded a revised PDF (new or changed text in blue; main text within the 10-page limit). New planning evaluations use the paper's protocol (the same 50 seeds × 4 tiers × 2 horizons), and each new result either tests a claim already in the paper or answers a reviewer's question.

In brief. (1) The planner follows the goal, not the tier template: with target object and zone reassigned in identical scenes, navigation stays at 0.96 and pushing does not fall back to the template goal. (2) The repaired latent ranks contact-rich pushes like the simulator (ρ ≈ 0.7, significant in 30/30 scenes). (3) Given nearby subgoals, the same world model pushes far better (0.62 / 0.55 on medium / hard), and stage sentences lift language pushing sixfold (0.04 → 0.25). (4) We corrected three statements, most importantly the predictor-IDM gradient path; none of the corrections changes a conclusion.

New evidence

Concern (reviewers)New resultWhere
Template shortcuts (PsLh)Target object and zone reassigned to distractors in identical scenes: navigation 0.96 (canonical 0.98) while a goal-agnostic policy drops from 0.12 to 0.04. On pushing the planner does not fall back to the template goal (≤ 1/100) and, with stage frames, reaches the reassigned goal in 0.14 / 0.12 of medium / hard trials (goal-agnostic 0); reassigned pushes are 1.7–2× as long and start beside the object, so a single goal frame does not reach them, as on very hardApp. M
Cost ranking in contact-rich pushing (PsLh)Pusher in contact behind the object: Spearman ρ = 0.72 / 0.73 on medium / hard, significant in 30/30 scenes; 0.66 / 0.70 with reassigned goals; plain ≈ 0App. I
Pushing vs. navigation (all)Given stage frames, the same world model reaches 0.62 / 0.55 on medium / hard (goal frame 0.27 / 0.16); stage sentences lift language pushing from 0.04 / 0.04 to 0.25 / 0.25§6, App. N
Copy-forward baseline (mHcC)Measured on fresh episodes never used in training: plain's rollouts are worse than copying the current latent in 100/100 sequences, LeWM+I's better in 91/100§5.4, Table 4
ViT-Small (mHcC)Action sensitivity recovers, but 10-step rollouts are no better than copying and the pusher is localised only to 30 px, the success thresholdApp. C
Train/eval object counts (PsLh)3–6 objects per evaluation scene, as in training: 0.330 vs 0.353 (p = 0.55)App. A
Sweep confound (mHcC, PsLh)Four single-factor plain cells† (render-only halo around the pusher at unchanged physics, pusher size, object dynamics, action scale) are being trained; their model-free properties are already measured (see the responses to mHcC and PsLh). Results follow on Nov 13–16App. G
Predictor-IDM gradient path (mHcC)LeWM+I with the predictor context detached for Eq. 3† (verified: no Eq. 3 gradient reaches the encoder); result around Nov 15§4, §5.3

† newly trained models; all other rows evaluate the paper's checkpoints.

Corrections. (i) Section 4 stated that Eq. 3 does not update the encoder. The stop-gradient is on ztz_t only; the predicted latent depends on the encoder's context latents, so Eq. 3 also reaches the encoder through the predictor. We corrected §4 and App. A (with pseudocode), and the paper now attributes the repair to the encoder-side term; predictor-IDM alone stays at plain's level despite the indirect path. (ii) The abstract said that the scripted controller "solves every tier"; it solves easy, medium and hard in every trial and very hard in 0.28 of trials within 20 steps, as Table 1 states. (iii) Section 5.4 used the rollout length as a stand-in for the copy baseline; it is now measured (above).

Positioning (tgyF, PsLh). We do not claim inverse dynamics as new. Our contribution is (1) the regime in which an end-to-end trained JEPA world model stops being plannable, and its mechanism (the encoder discards the bodies the actions move when little of the frame responds to actions), supported by one model-free and four model-internal probes; (2) controls that attribute the repair: a shuffled-action null at identical compute, an encoder-vs-predictor disentangle and a frozen-encoder cap; (3) SLIM, which makes this regime and language goals measurable. Concurrent arXiv work such as DA-LeWM studies settings in which LeWM already plans.

Presentation (PsLh). The introduction and the abstract now open with the research question and present language-goal planning as the application that motivated the benchmark.

Dear Reviewers,

The authors have now submitted their rebuttal and responded to the concerns raised in the reviews. We would greatly appreciate it if you could take some time to review their responses and participate in the discussion, particularly regarding whether your concerns have been adequately addressed.

Thank you very much for your time and valuable contributions!

AuthorsOpenReview ↗

Final update: a retraction, a new diagnosis of the collapse, and the revised PDF

We uploaded a revised PDF (changes in blue) and post all remaining results; details are under each review.

1. Retraction. We withdraw the explanation that LeWM collapses on SLIM because little of the frame responds to actions. A disc of radius 40 px drawn around the pusher at unchanged physics quadruples that fraction (0.35% → 1.41%; PushT 0.57%), and vanilla LeWM stays collapsed with both training seeds. Lighter objects and faster actions leave it collapsed too, as do PushT-style dynamics. The revision removes the explanation throughout, and the title now reads "Keeping JEPA World Models Plannable in Multi-Object Pushing".

2. What the collapse depends on instead (new App. O and P).

  • What the collapsed encoder keeps: content that differs between episodes but is fixed within one (zones, their number, colours). At the default rendering at most 0.5% of its latent variance changes within an episode; recolouring a zone moves the latent by 72–84% of the distance between two scenes (74–75% at the default), moving the pusher by 100 px by less than 5%, and the pusher is not decodable.
  • Removing it removes the collapse. With the zones not drawn and all objects grey, plain trains smoothly, codes the pusher (8–9 px) and plans in all four finished runs: 106 and 89 of 400 with the objects drawn as discs, 116 and 119 of 400 render-only (states and actions unchanged), against 2 of 400 at the default.
  • Partial removal is not enough. Removing only the colours leaves plain collapsed. Removing only the zones' variation (zones hidden, or the same two zones in every episode) makes it action-sensitive in all four runs, but two of the three finished runs train unstably and end with a latent dominated by the objects' colours that does not plan (6 and 9 of 400); the third plans (107 of 400).
  • On PushT the converse points the same way. Adding one to three such zones per episode to PushT's training data cuts the spread of plain's predictions across actions 30- to 140-fold, although the actions move a large block in every episode. Its spread/motion ratio stays above the gate (0.07–0.24) because the predicted motion shrinks too; App. P reports both.
  • Not the whole story. With a larger pusher plain escapes the collapse in some runs although zones and colours are present (pusher 40, lo, hi; App. G; only hi plans well); we have not isolated why.
  • Reading. SIGReg asks the latents of a batch to be isotropic Gaussian at every time step. Content that differs between episodes but not within them meets this at no cost to the prediction loss, since copying the current latent already predicts such a latent; the inverse-dynamics loss closes this route by requiring the latent to identify the action. This is a reading of the interventions, not a tested mechanism. If it holds, the failure matters beyond SLIM: PushT has almost no episode-constant content, while realistic scenes typically have far more (backgrounds, layouts, object identities).

3. What stands. On SLIM vanilla LeWM fails (3/1200 trials), and four model-internal probes locate the failure in the encoder. One inverse-dynamics loss repairs it (0.348 ± 0.016), and the controls attribute the repair to the encoder-side term; it survives detaching the predictor context for Eq. 3 (the converged one of two runs reaches 0.305). LeWM+I also plans at 112 px (0.213, mainly navigation) and, after 15 further epochs, at ViT-Small (0.373).

4. Further corrections.

  • Threshold: across 50 trained models every model below the action-sensitivity gate fails (15 of 15), but three failing models lie in the former gap between 0.011 and 0.24 (0.0345, 0.14 and 0.22). Spread/motion is therefore not bimodal, and the data bound the threshold only from above (reply to tgyF).
  • ViT-Small: by the training-side criterion of App. B the 30-epoch run had not converged; continued for 15 epochs it plans at ViT-Tiny's level (reply to mHcC).
  • Reference baselines: the scripted controller, the random policy and the goal-agnostic policy now reset each scene with the evaluation seed of the planner (before, object shapes could differ from the evaluated scene). Scripted very hard rises from 0.28 to 0.42 within 20 steps (1.00 with 60 steps) and the goal-agnostic medium value from 0.20 to 0.22; at the default physics, policies that ignore the goal still score 0 on hard. The per-arm values in our earlier tables change likewise (scripted very hard: halo 0.28 → 0.42, pusher 40 0.84 → 0.88, scale 80 0.82 → 0.92, hi 1.00 → 0.98; random policy lo/hi 0.04/0.045 → 0.035/0.050). No comparison with a learned model changes direction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.