- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether the action distinctions a latent world model makes correspond to the distinctions that matter once actions are executed, using fixed LeWM checkpoints on Reacher, TwoRoom, Cube, and PushT. Rather than proposing a new model, it proposes a diagnostic protocol that executes the planner's own candidate actions from a common initial state and rescores them at the model's prediction endpoint using realized rather than predicted representations, then applies controlled interventions to locate where the mismatch arises. Realized scoring ranks candidates better than predicted scoring in all four tasks, and the mismatch runs in both directions, since some action changes that alter object motion are nearly invisible to the encoder while other command differences that execution discards still change predictions. The behavioral tests are largely negative, as interventions that improve local rankings rarely change task success.
Critical strengths
-
The paper shows that encoder invariance and physical equivalence come apart in both directions, with controls rather than assertion. Cube's rank deficient action layer ignores components that still move the block several centimeters, while TwoRoom's clipping leaves the executed path identical yet changes predictions. This turns a vague concern about objective mismatch into two concrete interface failures a designer can act on.
-
The negative behavioral result is reported honestly and is arguably the most useful finding. Interventions that improve fixed pool rankings change the selected commands in nearly every episode yet almost never change success. Since fixed pool ranking is becoming a common evaluation shortcut, direct evidence that it can move without affecting task performance is valuable.
Critical weaknesses
- All findings come from one model family and one planner, with two extra TwoRoom weight variants as the only generality check and independent seeds and architectures left explicitly unverified.
- The practical payload is also thin. No deployable intervention improves closed loop success, and the two clear behavioral effects come from changing physics in Reacher or from an oracle that requires ground truth execution in PushT. The proposed remedies in Section 6 are untested, and the one mechanism the authors did test at inference time showed no stable gain.
Questions for authors
- The main concordance comparison uses the first planning call's pool, while the closed loop tests that span replanning show no effect. Does the realized versus predicted gap persist if measured on pools from later planning calls, or is it specific to the step zero pool?
5 comments
Response to Reviewer fPPt (Part 2/3): Other models, seeds, architectures, and planner
2. Other world models. We tested two other world models on PushT and on two of their benchmark tasks, Wall (2-D navigation) and Metaworld reach-wall (robot-arm reaching). DINO-WM (Zhou et al., 2025) conditions on actions by concatenating them with the visual features; we use the weights released by Terver et al. JEPA-WM (Terver et al., 2026) conditions on actions through adaptive layer normalization. As in Table 1, each query contributes the candidate pool of its first planning call, and every candidate is executed over the model's native 30-step horizon. Realized scoring again ranks candidates better, with adjusted intervals above zero for JEPA-WM on PushT and for both models on Wall.
| Task | Model | Predicted C ↑ | Realized C ↑ | Mixed/total pools |
|---|---|---|---|---|
| PushT | DINO-WM | 0.798 | 0.868 | 29/150 |
| PushT | JEPA-WM | 0.679 | 0.951 | 13/150 |
| Wall | DINO-WM | 0.693 | 0.998 | 46/96 |
| Wall | JEPA-WM | 0.686 | 0.991 | 57/96 |
C: concordance. Mixed pools contain both successful and failed candidates. Bold realized scores mark gaps with adjusted intervals above zero.
The same scoring mechanism we identified in LeWM appears in both models. On Wall, the ranking gap comes from the visual predictions. Both models predict visual features and proprioception; replacing only the visual predictions with executed endpoints recovers the full gain. Splitting the visual error as in Eq. 4 (Section 4.3), removing the directional error term raises concordance to 0.977 (DINO-WM) and 0.973 (JEPA-WM), whereas removing the squared error lowers it to 0.191 and 0.255. As in LeWM, prediction accuracy alone does not determine preferences: the error's direction relative to the goal strongly shapes which candidate is preferred.
Better scoring also changes behavior beyond LeWM. On Metaworld reach-wall, where candidates rarely succeed within 30 steps, repeating our Table 7 diagnostic (realized scores choose the first call's output, then native planning resumes) raises JEPA-WM's success from 9/48 to 20/48 (+22.92 percentage points; adjusted interval [2.08, 43.75]); DINO-WM moves from 13/48 to 14/48.
3. Independent seeds, architectures, and planner.
- Training seeds. We trained two new TwoRoom models from scratch with seeds 42 and 43, which change weight initialization and data order. TwoRoom clips each command component to [−1, 1] at execution, yet the model sees the unclipped command, so commands that execute identically receive different predictions (Section 4.2). Clipping commands before scoring improves fixed-pool ranking in both new models, as it did for the paper's two weight variants: concordance rises from 0.738 to 0.812 and from 0.726 to 0.814, with adjusted intervals above zero.
- Planner seeds. Every new closed-loop comparison is repeated with five planner seeds, which change the candidates CEM samples while keeping the model fixed. The three deployable gains in Part 3 hold under every seed: per-seed changes range from +4.0 to +8.0 percentage points for Cube, +7.3 to +18.0 for Reacher, and +1.3 to +4.0 for PushT.
- Architectures. Beyond the two other model families above, we also changed the architecture within LeWM. Cube's original action encoder maps 25 action dimensions to 10, so the model cannot see action changes along the remaining directions; on 750 such action pairs, 105 still lead to different task outcomes. We trained Cube models from scratch at two seeds with three action encoders: the original 25→10 design, a full 25→25 design, and a 25→25 design whose input is restricted to what the original encoder sees. Both full 25→25 models distinguish all 750 pairs, whereas the restricted models, like the original model, distinguish none. The blind spot is therefore set by the action interface, consistently across both training seeds.
- Planner. All evaluated systems plan with CEM; we will state this scope in Section 6.
We will add these results to Sections 4–5, with full results and intervals in the appendix.
Response to Reviewer fPPt (Part 3/3): Deployable improvements and the Section 6 remedies
4. Deployable improvements. Training-free changes to the planner's input or output improve 50-step closed-loop success in three tasks, using only information available at decision time:
- Cube: project the planner's output onto the action components that the action encoder retains, so the executed command contains only directions the model evaluated.
- Reacher: give the model past observations and executed actions at each replanning step, instead of a single frame.
- PushT: correct each candidate's prediction with the prediction error observed after the previous executed action sequence.
| Task / intervention | Native success (%) ↑ | Intervention success (%) ↑ | Change (pp) [adjusted interval] |
|---|---|---|---|
| Cube / output projection | 50.67 | 56.80 | +6.13 [2.80, 9.60] |
| Reacher / context input | 82.40 | 94.27 | +11.87 [8.40, 15.33] |
| PushT / online calibration | 84.27 | 86.80 | +2.53 [0.13, 5.60] |
| TwoRoom / reranking with clipped commands | 83.52 | 82.73 | −0.78 [−2.27, 0.55] |
Success means completing the task within 50 environment steps, as in Table 7. Rates average five planner seeds over 150 initial states per task (256 for TwoRoom). Brackets give 95% bootstrap confidence intervals, Bonferroni-adjusted over each experiment's planned comparisons; bold marks gains with adjusted intervals above zero. We chose how to apply the PushT correction, and how strongly, on development episodes separate from the test set, then evaluated it unchanged on all test episodes. In Reacher, a control that repeats the current frame with the same past actions reaches 93.47%, so the gain comes from the added context rather than specifically from visual history.
5. Section 6 remedies. We tested the remedies for each task:
- Reacher, image–action history: adding past observations and actions improves success; the repeated-frame control shows that the gain comes from the added context rather than the visual history itself.
- Cube, a wider action encoder: restores the response to the ignored action differences (Part 2). However, the change it predicts between paired actions is no closer to the realized change than a prediction of no change, and 50-step success shows no reliable gain over the original architecture. Output projection is the effective Cube change.
- TwoRoom, training on executed controls or searching within legal bounds: neither improves success. Fine-tuning the model with added commands that execute identically to legal ones gives −3.52 pp [−7.34, −0.08] and −2.19 pp [−5.59, 0.94] against controls fine-tuned the same way without the added commands, at two seeds; projecting candidates onto legal bounds inside CEM gives −12.50 pp [−17.50, −7.81]. The inference-time clipping you mention, now applied as final reranking, still shows no reliable gain (−0.78 pp, table above).
- PushT: instead of the suggested goal-relative score, we targeted the score term B (Section 4.4). Fine-tuning the predictor with an added loss on B, and a small head trained to predict term B from the model's features and remove it from the score, did not outperform the original model (and, for the loss variant, a control fine-tuned without the B loss) on development episodes, so neither was evaluated on the test set; online calibration improves success (above).
In these tests, direct changes to the planner's input or output were more effective than retraining: they deliver all three gains, whereas the architecture and training remedies restore or target the relevant distinctions but have not yet improved closed-loop success. We will revise Section 5 to include the deployable gains and Section 6 to report these tests; protocols and controls, including the full comparison of predicted and realized changes in Cube, will be in the appendix.
Response to Reviewer fPPt (Part 1/3): Later planning calls
Thank you for the careful review. Part 1 answers your question on later planning calls; Part 2 addresses generality across models, seeds, architectures, and planners; Part 3 addresses deployable improvements and the Section 6 remedies.
1. Later planning calls. The gap persists beyond the first call. We extended native planning from 50 to 100 steps, with replanning every 25 steps, on 700 initial queries across the four tasks, and audited later pools exactly as the first-call pool: every candidate is executed and scored at the same 25-step endpoint. The table gives the realized-minus-predicted concordance gap; n counts pools that contain both successful and failed candidates.
Brackets give 95% bootstrap confidence intervals, Bonferroni-adjusted across all 12 later-call comparisons (second to fourth calls); bold gaps have adjusted intervals above zero. All later-call estimates are positive, with adjusted intervals above zero in Reacher and PushT at both later calls and in TwoRoom at the third. Fourth-call estimates, from at most five such pools per task, are also positive and are reported in the appendix. Most later Cube pools contain no successful candidate, so there is nothing to rank. Later pools come from episodes that have not yet succeeded, so the gap persists in harder, still-unsolved states.
We will add these results to Section 4, with full results and coverage in the appendix.