acceptodds
Under review as a conference paper at ICLR 2027

How World Models Misjudge Actions: A Case Study of LeWM

Abstract

World models guide planning by comparing predicted action consequences, but the distinctions they make need not match those that matter for execution. We study this mismatch in LeWM across Reacher, TwoRoom, Cube, and PushT through matched candidate executions and interventions on physical states, commands, scoring, and output formation. Beyond an average ranking advantage of realized over predicted representations at matched endpoints in all four tasks, these interventions reveal mismatches in both directions. Cube action encoding nearly ignores differences that change object motion, whereas TwoRoom predictions respond to command differences discarded during execution. In Reacher, single-frame inputs conceal velocities that change future motion. In PushT pools admitting valid orientation-edited references for every candidate, scoring interactions involving the edit and prediction error can reinforce or offset incorrect action preferences. Averaging can carry nearly ignored action differences into execution or break equivalence between individually interchangeable commands, changing success without changing selected members. In two additional TwoRoom weight variants, removing execution-irrelevant command differences improves fixed-pool rankings, while separate closed-loop tests yield no observed gains in success. These findings motivate testing both sensitivity to consequential differences and respect for execution equivalence, together with how scoring and output formation turn model distinctions into decisions.

Primary area:
applications to robotics, autonomy, planning
Keywords:
latent world models, model-based planning, world model evaluation, action selection, action-conditioned prediction, model predictive control

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fPPtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether the action distinctions a latent world model makes correspond to the distinctions that matter once actions are executed, using fixed LeWM checkpoints on Reacher, TwoRoom, Cube, and PushT. Rather than proposing a new model, it proposes a diagnostic protocol that executes the planner's own candidate actions from a common initial state and rescores them at the model's prediction endpoint using realized rather than predicted representations, then applies controlled interventions to locate where the mismatch arises. Realized scoring ranks candidates better than predicted scoring in all four tasks, and the mismatch runs in both directions, since some action changes that alter object motion are nearly invisible to the encoder while other command differences that execution discards still change predictions. The behavioral tests are largely negative, as interventions that improve local rankings rarely change task success.

Critical strengths
  • The paper shows that encoder invariance and physical equivalence come apart in both directions, with controls rather than assertion. Cube's rank deficient action layer ignores components that still move the block several centimeters, while TwoRoom's clipping leaves the executed path identical yet changes predictions. This turns a vague concern about objective mismatch into two concrete interface failures a designer can act on.

  • The negative behavioral result is reported honestly and is arguably the most useful finding. Interventions that improve fixed pool rankings change the selected commands in nearly every episode yet almost never change success. Since fixed pool ranking is becoming a common evaluation shortcut, direct evidence that it can move without affecting task performance is valuable.

Critical weaknesses
  • All findings come from one model family and one planner, with two extra TwoRoom weight variants as the only generality check and independent seeds and architectures left explicitly unverified.
  • The practical payload is also thin. No deployable intervention improves closed loop success, and the two clear behavioral effects come from changing physics in Reacher or from an oracle that requires ground truth execution in PushT. The proposed remedies in Section 6 are untested, and the one mechanism the authors did test at inference time showed no stable gain.
Questions for authors
  • The main concordance comparison uses the first planning call's pool, while the closed loop tests that span replanning show no effect. Does the realized versus predicted gap persist if measured on pools from later planning calls, or is it specific to the step zero pool?
5 comments
AuthorsOpenReview ↗

Response to Reviewer fPPt (Part 1/3): Later planning calls

Thank you for the careful review. Part 1 answers your question on later planning calls; Part 2 addresses generality across models, seeds, architectures, and planners; Part 3 addresses deployable improvements and the Section 6 remedies.

1. Later planning calls. The gap persists beyond the first call. We extended native planning from 50 to 100 steps, with replanning every 25 steps, on 700 initial queries across the four tasks, and audited later pools exactly as the first-call pool: every candidate is executed and scored at the same 25-step endpoint. The table gives the realized-minus-predicted concordance gap; n counts pools that contain both successful and failed candidates.

TaskSecond planning callThird planning call
Reacher0.217 [0.159, 0.283]; n=940.202 [0.105, 0.331]; n=23
TwoRoom0.118 [−0.018, 0.277]; n=230.291 [0.036, 0.813]; n=5
Cube0.199 [−0.535, 0.568]; n=5n=1
PushT0.248 [0.097, 0.403]; n=350.383 [0.064, 0.716]; n=13

Brackets give 95% bootstrap confidence intervals, Bonferroni-adjusted across all 12 later-call comparisons (second to fourth calls); bold gaps have adjusted intervals above zero. All later-call estimates are positive, with adjusted intervals above zero in Reacher and PushT at both later calls and in TwoRoom at the third. Fourth-call estimates, from at most five such pools per task, are also positive and are reported in the appendix. Most later Cube pools contain no successful candidate, so there is nothing to rank. Later pools come from episodes that have not yet succeeded, so the gap persists in harder, still-unsolved states.

We will add these results to Section 4, with full results and coverage in the appendix.

AuthorsOpenReview ↗

Response to Reviewer fPPt (Part 2/3): Other models, seeds, architectures, and planner

2. Other world models. We tested two other world models on PushT and on two of their benchmark tasks, Wall (2-D navigation) and Metaworld reach-wall (robot-arm reaching). DINO-WM (Zhou et al., 2025) conditions on actions by concatenating them with the visual features; we use the weights released by Terver et al. JEPA-WM (Terver et al., 2026) conditions on actions through adaptive layer normalization. As in Table 1, each query contributes the candidate pool of its first planning call, and every candidate is executed over the model's native 30-step horizon. Realized scoring again ranks candidates better, with adjusted intervals above zero for JEPA-WM on PushT and for both models on Wall.

TaskModelPredicted C ↑Realized C ↑Mixed/total pools
PushTDINO-WM0.7980.86829/150
PushTJEPA-WM0.6790.95113/150
WallDINO-WM0.6930.99846/96
WallJEPA-WM0.6860.99157/96

C: concordance. Mixed pools contain both successful and failed candidates. Bold realized scores mark gaps with adjusted intervals above zero.

The same scoring mechanism we identified in LeWM appears in both models. On Wall, the ranking gap comes from the visual predictions. Both models predict visual features and proprioception; replacing only the visual predictions with executed endpoints recovers the full gain. Splitting the visual error as in Eq. 4 (Section 4.3), removing the directional error term raises concordance to 0.977 (DINO-WM) and 0.973 (JEPA-WM), whereas removing the squared error lowers it to 0.191 and 0.255. As in LeWM, prediction accuracy alone does not determine preferences: the error's direction relative to the goal strongly shapes which candidate is preferred.

Better scoring also changes behavior beyond LeWM. On Metaworld reach-wall, where candidates rarely succeed within 30 steps, repeating our Table 7 diagnostic (realized scores choose the first call's output, then native planning resumes) raises JEPA-WM's success from 9/48 to 20/48 (+22.92 percentage points; adjusted interval [2.08, 43.75]); DINO-WM moves from 13/48 to 14/48.

3. Independent seeds, architectures, and planner.

  • Training seeds. We trained two new TwoRoom models from scratch with seeds 42 and 43, which change weight initialization and data order. TwoRoom clips each command component to [−1, 1] at execution, yet the model sees the unclipped command, so commands that execute identically receive different predictions (Section 4.2). Clipping commands before scoring improves fixed-pool ranking in both new models, as it did for the paper's two weight variants: concordance rises from 0.738 to 0.812 and from 0.726 to 0.814, with adjusted intervals above zero.
  • Planner seeds. Every new closed-loop comparison is repeated with five planner seeds, which change the candidates CEM samples while keeping the model fixed. The three deployable gains in Part 3 hold under every seed: per-seed changes range from +4.0 to +8.0 percentage points for Cube, +7.3 to +18.0 for Reacher, and +1.3 to +4.0 for PushT.
  • Architectures. Beyond the two other model families above, we also changed the architecture within LeWM. Cube's original action encoder maps 25 action dimensions to 10, so the model cannot see action changes along the remaining directions; on 750 such action pairs, 105 still lead to different task outcomes. We trained Cube models from scratch at two seeds with three action encoders: the original 25→10 design, a full 25→25 design, and a 25→25 design whose input is restricted to what the original encoder sees. Both full 25→25 models distinguish all 750 pairs, whereas the restricted models, like the original model, distinguish none. The blind spot is therefore set by the action interface, consistently across both training seeds.
  • Planner. All evaluated systems plan with CEM; we will state this scope in Section 6.

We will add these results to Sections 4–5, with full results and intervals in the appendix.

AuthorsOpenReview ↗

Response to Reviewer fPPt (Part 3/3): Deployable improvements and the Section 6 remedies

4. Deployable improvements. Training-free changes to the planner's input or output improve 50-step closed-loop success in three tasks, using only information available at decision time:

  • Cube: project the planner's output onto the action components that the action encoder retains, so the executed command contains only directions the model evaluated.
  • Reacher: give the model past observations and executed actions at each replanning step, instead of a single frame.
  • PushT: correct each candidate's prediction with the prediction error observed after the previous executed action sequence.
Task / interventionNative success (%) ↑Intervention success (%) ↑Change (pp) [adjusted interval]
Cube / output projection50.6756.80+6.13 [2.80, 9.60]
Reacher / context input82.4094.27+11.87 [8.40, 15.33]
PushT / online calibration84.2786.80+2.53 [0.13, 5.60]
TwoRoom / reranking with clipped commands83.5282.73−0.78 [−2.27, 0.55]

Success means completing the task within 50 environment steps, as in Table 7. Rates average five planner seeds over 150 initial states per task (256 for TwoRoom). Brackets give 95% bootstrap confidence intervals, Bonferroni-adjusted over each experiment's planned comparisons; bold marks gains with adjusted intervals above zero. We chose how to apply the PushT correction, and how strongly, on development episodes separate from the test set, then evaluated it unchanged on all test episodes. In Reacher, a control that repeats the current frame with the same past actions reaches 93.47%, so the gain comes from the added context rather than specifically from visual history.

5. Section 6 remedies. We tested the remedies for each task:

  • Reacher, image–action history: adding past observations and actions improves success; the repeated-frame control shows that the gain comes from the added context rather than the visual history itself.
  • Cube, a wider action encoder: restores the response to the ignored action differences (Part 2). However, the change it predicts between paired actions is no closer to the realized change than a prediction of no change, and 50-step success shows no reliable gain over the original architecture. Output projection is the effective Cube change.
  • TwoRoom, training on executed controls or searching within legal bounds: neither improves success. Fine-tuning the model with added commands that execute identically to legal ones gives −3.52 pp [−7.34, −0.08] and −2.19 pp [−5.59, 0.94] against controls fine-tuned the same way without the added commands, at two seeds; projecting candidates onto legal bounds inside CEM gives −12.50 pp [−17.50, −7.81]. The inference-time clipping you mention, now applied as final reranking, still shows no reliable gain (−0.78 pp, table above).
  • PushT: instead of the suggested goal-relative score, we targeted the score term B (Section 4.4). Fine-tuning the predictor with an added loss on B, and a small head trained to predict term B from the model's features and remove it from the score, did not outperform the original model (and, for the loss variant, a control fine-tuned without the B loss) on development episodes, so neither was evaluated on the test set; online calibration improves success (above).

In these tests, direct changes to the planner's input or output were more effective than retraining: they deliver all three gains, whereas the architecture and training remedies restore or target the relevant distinctions but have not yet improved closed-loop success. We will revise Section 5 to include the deployable gains and Section 6 to report these tests; protocols and controls, including the full comparison of predicted and realized changes in Cube, will be in the appendix.

Reviewer_wg41OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether latent-distance planning objectives rank candidate action sequences in the way their executed outcomes do. The authors show (i) that most models do no better than chance when it comes to manipulation tasks, and that on Push-T the released model's predictor causes this; (ii) that wider search and more frequent replanning contribute to planning success; and (iii) that planners can fail, even when action sequences are present among their candidates.

Critical strengths

Overall, this paper was clear and easy to follow---even for someone who works in different areas, like me. The question the authors address is clear and well-posed, the methodology is very careful, and the authors are upfront about limitations.

Critical weaknesses

My one concern is that the paper's results don't quite seem to match its narrative. For example, in section 6, the authors say that the objective ranks the planner's own candidates well, and better fixed-candidate ranking do not improve its success. To me, the contribution would be clearer here if it were framed around the gap between fixed-candidate audits and planning, rather than around models failing to select the right actions. Aside from that, however, I can't find much wrong with the paper.

Questions for authors

Could the authors mildly reframe the paper, so that it better matches their key results?

3 comments
AuthorsOpenReview ↗

Response to Reviewer wg41: Reframing around the gap between fixed-candidate audits and planning

Thank you for the constructive review. Yes, we will mildly reframe the paper around the gap between fixed-candidate audits and planning, as you suggest.

On how well the objective ranks the planner's own candidates: Table 1 compares two scores that use the same squared latent distance to the goal. Predicted scoring uses the model's predicted endpoint representations; realized scoring uses representations of the endpoints reached by actually executing each candidate, and serves as a diagnostic reference. Predicted scoring has concordance 0.54 to 0.72, measured as the rate at which successful candidates rank above failed ones, and ranks candidates less reliably than realized scoring in all four tasks.

Better fixed-pool ranking does not always reach behavior. In an additional TwoRoom checkpoint, treating commands that execute identically as identical raises fixed-pool concordance from 0.7385 to 0.8438, yet closed-loop success does not improve (from 217 to 208 of 256 episodes at 50 steps; Table 13). Conversely, using realized scores to select the first planning call's output, with normal replanning afterwards, improves 50-step success in Reacher and PushT (Table 7, Appendix C.2). This requires executing every candidate beforehand and is therefore diagnostic. Better fixed-pool ranking can help, but does not guarantee closed-loop gains.

We will revise the contribution statement and the transition from Section 5 to Section 6 accordingly, adding the following paragraph:

We study the gap between how LeWM judges candidate actions on a fixed pool and how well it plans in closed loop. Fixed-pool audits show that the model can nearly ignore action differences that matter physically, or respond to command differences that do not change execution. Behavioral tests show when these judgments affect the executed actions, and that better fixed-pool ranking does not guarantee higher planning success.

Note to the authors and AC: Sincere apologies. My original review was posted to the wrong submission (one with a similar title to this one---hence my confusion), and I have now replaced it with my review of this paper. The request to reframe the paper around the gap between fixed-candidate audits and planning came from my review of another submission, so please don't feel bound by the changes made in response to it. How embarrassing!

Paper Summary

This paper starts by noting that world models guide planning by comparing predicted action consequences. However, the authors also note that the distinctions needed here do not always match those that matter for execution. They thus study this mismatch in LeWM across Reacher, TwoRoom, etc., through matched candidate executions and interventions on physical states, commands, scoring, and output formation. The authors claim that their key finding motivate testing both sensitivity to consequential differences and respect for execution equivalence. Together, they argue, this would show how scoring and output formation turn model distinctions into decisions.

Critical Strengths

The question the authors address is interesting and well-posed. The Cube finding is interesting and actionable. The paper is upfront about limitations, corrections, etc. For instance, null-results are stated plainly.

Critical Weaknesses

The scope of this paper is arguably a bit narrower than its title suggests. For instance, all results come from a single model family. The constructive evidence is also arguably a bit thin. For example, the interventions that seek to improve success either use realized endpoints, or change the physical state itself, rather than the model. Finally, the presentation in this paper is a bit tough-going in places. Page limits notwithstanding, I found the writing here a bit dense, and tricky to follow.

Questions For Authors

Have you tested any of the mechanisms on another world model? Additionally, the PushT analysis is a bit hard to interpret. Is there a more transparent way to communicate its point?

Final Verdict

Having reviewed the rebuttals and the other reviewers' replies, I'm inclined to maintain my original, positive score.

AuthorsOpenReview ↗

Response to Reviewer wg41's updated review

Thank you for the note, for taking the time to review our paper, and for finding the Cube result interesting and actionable. Below we briefly answer your questions and comments.

Other world models. We tested how prediction errors enter the score with DINO-WM and JEPA-WM. On Wall, as in LeWM, the direction of the prediction error relative to the goal drives the ranking gap in both models (Section 4.3; Appendix F.3). The Cube blind spot comes from LeWM's action encoder, which maps each 25-dimensional action block to 10 dimensions; both other models encode actions at full rank (Sections 4.2 and 4.3).

Deployable improvements. Changes to the planner's input or output that use only information available at decision time improve 50-step success in Cube, Reacher, and PushT (Section 5.2; Appendix F.4). These changes leave the model itself unchanged; retraining the model has not yet improved success (Section 6).

PushT. The point is that a prediction can be wrong in a direction that makes a candidate look closer to the goal than it is, or farther. Such errors can misorder candidates even when they are small. In PushT, success also depends on the block's orientation, so for each candidate we render its executed endpoint with the block turned to the goal orientation. This separates the effect of orientation from that of the prediction error. Correcting only the part that comes from the prediction error's direction closes 69% of the gap between predicted and realized ranking, in pools where every candidate has such a rendering (Section 4.4).

Reviewer_uuZsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how action judgments in LeWorldModel (LeWM) can diverge from actual physical consequences across Reacher, TwoRoom, Cube, and PushT. Using matched candidate executions and controlled interventions, the authors identify several failure mechanisms: physically consequential action variations suppressed by the Cube action encoder, execution-equivalent commands distinguished by the TwoRoom model, missing velocity information in Reacher, and scoring interactions that distort action preferences in PushT. A central finding is that improved prediction accuracy or candidate ranking does not necessarily translate into better closed-loop planning performance.

Critical strengths
  1. Careful mechanistic analysis. The controlled interventions provide concrete evidence of where model predictions diverge from physical execution. The Cube null-space and TwoRoom execution-equivalence experiments are particularly well designed.

  2. Useful distinction between prediction, ranking, and execution. The paper goes beyond evaluating prediction error and demonstrates that better action rankings do not necessarily improve downstream planning success. The analysis of elite averaging is also informative.

  3. Thorough empirical evaluation. The authors provide extensive experiments, paired comparisons, uncertainty estimates, and additional checkpoint analyses. Negative results and counterexamples are appropriately acknowledged.

Critical weaknesses
  1. Limited generalizability. The identified mechanisms are largely specific to individual LeWM architectures and environments. For instance, Cube's null space arises from a rank-10 projection of 25-dimensional actions. While the physical consequences of the suppressed directions are interesting, the existence of this null space is mathematically guaranteed. It remains unclear whether similar failures occur in world models with different action-conditioning architectures.

  2. Insufficient connection to overall planning degradation. The experiments establish that these failure mechanisms exist, but do not quantify how much they contribute to the observed ranking and planning failures. In Cube, for example, it is unclear what fraction of the substantial predicted-versus-realized ranking gap can be explained by null-space insensitivity rather than other prediction errors.

  3. Limited actionable improvements. The paper diagnoses several problems but does not demonstrate that correcting them through model architecture or training improves planning. The TwoRoom results further show that local ranking improvements do not reliably translate into closed-loop gains. The strongest PushT interventions additionally rely on oracle information and restricted subsets.

Questions for authors

To what extent is the identified action-nullspace failure an artifact of LeWM's particular temporal action-compression architecture, rather than a general limitation of learned world models? More importantly, how much of the observed planning degradation can actually be attributed to this mechanism?

5 comments
AuthorsOpenReview ↗

Response to Reviewer uuZs (Part 1/3): Specific to LeWM or a general limitation

Thank you for the careful review. Part 1 addresses whether the mechanisms are specific to LeWM, including your question on the action null space; Part 2 quantifies how much the mechanisms explain; Part 3 addresses actionable improvements.

1. Specific to LeWM, or a general limitation? We agree that the mechanisms we located are largely tied to LeWM's design and tasks. Cube's null space follows from its rank-10 action projection; TwoRoom's response to excess commands comes from encoding commands before clipping; and Reacher's hidden velocity comes from single-frame input. Other world models make different design choices and need not share these mechanisms, and we did not trace model-specific mechanisms in DINO-WM or JEPA-WM. What carries over is the scoring-level failure of Sections 4.3–4.4: scores built on predicted endpoints misjudge executed outcomes, and the direction of prediction error relative to the goal shapes preferences (Eq. 4). Both other world models show this failure.

Beyond LeWM, scores still misjudge executed outcomes without action compression. DINO-WM (Zhou et al., 2025) maps actions at full rank (10→10) and concatenates them with the visual features; we use the weights released by Terver et al. JEPA-WM (Terver et al., 2026) maps actions at full rank (10→400) and conditions on them through adaptive layer normalization. As in Table 1, we score the candidate pool of each query's first planning call, executing every candidate over the model's native 30-step horizon. Realized scoring still ranks candidates better on PushT and Wall (2-D navigation), with adjusted intervals above zero for both models on Wall and for JEPA-WM on PushT:

TaskModelPredicted C ↑Realized C ↑Mixed/total pools
PushTDINO-WM0.7980.86829/150
PushTJEPA-WM0.6790.95113/150
WallDINO-WM0.6930.99846/96
WallJEPA-WM0.6860.99157/96

C: concordance. Mixed pools contain both successful and failed candidates. Bold realized scores mark gaps with adjusted intervals above zero.

The error enters through the visual predictions: both models predict visual features and proprioception, and replacing only the visual predictions with executed endpoints recovers the full Wall gain. Splitting the visual error as in Eq. 4 (Section 4.3), removing the directional error term raises Wall concordance to 0.977 (DINO-WM) and 0.973 (JEPA-WM), whereas removing the squared error lowers it to 0.191 and 0.255.

On Cube specifically, the null space is an artifact of LeWM's temporal action-compression architecture, and changing the action encoder removes it. As you note, its existence is mathematically guaranteed; whether changes along its directions matter physically is an empirical question, and they do: when we replace the null-space component of each of the planner's candidates with those of eight other candidates (360,000 executions), 13.3% change whether the candidate succeeds at step 25.

Within LeWM, the blind spot follows the action interface. We trained Cube models from scratch at two seeds with three action encoders: the original 25→10 design, a full 25→25 design, and a 25→25 design whose input is restricted to what the original encoder sees. On 750 action pairs that the original encoder cannot tell apart, 105 lead to different task outcomes. Both full 25→25 models distinguish all 750 pairs, whereas the restricted models, like the original model, distinguish none.

We will add these results to Sections 4–5, with full results and intervals in the appendix.

AuthorsOpenReview ↗

Response to Reviewer uuZs (Part 2/3): How much the mechanisms explain

2. How much the mechanisms explain. For ranking, each mechanism alone closes most of its task's predicted-versus-realized gap (67% to 82%); in Cube, null-space insensitivity closes 82%, while all other prediction errors together close 49%. For planning, correcting a mechanism raises success within the planning call it corrects, significantly in Reacher and PushT, and in Cube the null-space component of executed outputs contributes to failure: removing it raises success by 6.13 percentage points.

Ranking. We compare four scores on the same candidate pools: predicted, predicted with only the mechanism corrected, predicted with all other prediction errors corrected, and realized. For each task, we first isolate the part of the score error due to the mechanism: the effect of the null-space component (Cube), of the initial velocity (Reacher, measured by re-executing candidates with zero initial velocity), of excess command magnitude (TwoRoom), and term B (PushT). "Mechanism corrected" removes only this part from the predicted score; "other errors corrected" removes everything else. In Cube, the predicted score ignores each candidate's null-space component, so we add to it the difference between the candidate's realized cost and the mean realized cost after swapping in the null-space components of eight other candidates.

Task (mechanism)PredictedMechanism correctedOther errors correctedRealizedGap closed by mechanism
Reacher (velocity)0.6600.9230.7280.99878% [71, 84]
Cube (null space)0.5420.8410.7210.90482% [77, 87]
TwoRoom (excess command)0.7410.7780.7560.79667% [20, 96]
PushT (term B)0.6940.7540.5820.78069% [48, 87]

Concordance on pools with both successful and failed candidates. Gap closed: the mechanism-only gain divided by the realized-minus-predicted gap; brackets give 95% bootstrap confidence intervals, Bonferroni-adjusted. TwoRoom uses the 512 additional episodes of Section 4.3, and PushT the complete-reference pools of Section 4.4.

All four intervals lie above zero; TwoRoom's is wide because its gap is small (0.056). In Cube, the two corrections overlap: once other errors are corrected, the null-space correction closes the remaining 51% of Cube's gap, so the shares do not add up to a single split. In PushT, correcting other errors alone lowers concordance, because those errors had been offsetting term B (Section 4.4).

Planning. The mechanisms do reach planning, most clearly within the planning call they correct; how much survives replanning depends on the task and on the remaining errors. The table corrects the mechanism in the first planning call only, because the correction uses executed outcomes of every candidate; later calls use the native planner.

TaskWithin 25 stepsBy 50 stepsBy 50 steps, other errors also corrected
Reacher+34.0 [15.5, 49.5]+9.3 [−2.3, 20.0]+18.0 [5.6, 28.8]
Cube+10.0 [−1.9, 20.9]+8.7 [−4.1, 20.6]−2.0 [−10.1, 6.4]
TwoRoom+3.1 [−0.9, 7.0]−0.6 [−5.0, 3.9]−0.4 [−4.6, 3.8]
PushT+5.6 [0.7, 10.1]+1.5 [−2.7, 5.7]+7.1 [1.8, 12.0]

Change in success (percentage points) against the native planner; brackets give 95% intervals, Bonferroni-adjusted at each horizon over eight comparisons (four tasks, with and without other errors corrected), so they differ from Table 3; bold marks intervals above zero.

In Reacher and PushT, the correction raises success within the corrected call, and the gain shrinks once native replanning resumes; in TwoRoom, 13 of the 20 episodes rescued in the first 25 steps are also solved by the native planner by step 50. The deployable changes in Part 3 act at every planning call and keep their gains at 50 steps. In Cube, the mechanism acts through the executed output. Removing the null-space component that the model never evaluates from every output, a deployable change, raises success from 50.67% to 56.80% (+6.13 pp [2.80, 9.60]; Part 3), and changing only this component of the first output turns 17 of the 78 native failures into successes in at least one of eight random draws, although average success stays the same (47.9% vs. 48.0% on the first planner seed, as in Table 7).

We will add these analyses to Sections 4–5, with full results and intervals in the appendix.

AuthorsOpenReview ↗

Response to Reviewer uuZs (Part 3/3): Actionable improvements

3. Actionable improvements. Correcting the identified problems at inference time improves planning in three tasks; corrections through architecture or training (point 4 below) have not yet done so. The training-free changes use only information available at decision time:

  • Cube: remove from the planner's output the null-space component that the model never evaluates.
  • Reacher: give the model past observations and executed actions at each replanning step, instead of a single frame.
  • PushT: correct each candidate's prediction with the prediction error observed after the previous executed action sequence.
Task / interventionNative success (%) ↑Intervention success (%) ↑Change (pp) [adjusted interval]
Cube / output projection50.6756.80+6.13 [2.80, 9.60]
Reacher / context input82.4094.27+11.87 [8.40, 15.33]
PushT / online calibration84.2786.80+2.53 [0.13, 5.60]
TwoRoom / reranking with clipped commands83.5282.73−0.78 [−2.27, 0.55]

Success means completing the task within 50 environment steps, as in Table 7. Rates average five planner seeds over 150 initial states per task (256 for TwoRoom). Brackets give 95% bootstrap confidence intervals, Bonferroni-adjusted over each experiment's planned comparisons; bold marks gains with adjusted intervals above zero. Each gain is positive under every planner seed. In Reacher, a control that repeats the current frame with the same past actions reaches 93.47%, so the gain comes from the added context rather than the visual history itself.

The PushT calibration does not rely on oracle information or restricted subsets: it uses only prediction errors already observed after executed actions, needs no edited references, and is evaluated on all test episodes. We chose how to apply the correction, and how strongly, on development episodes separate from the test set. For TwoRoom, consistent with your observation, reranking with clipped commands still gives no reliable closed-loop gain.

4. Architecture and training corrections. We also tested corrections through architecture and training.

  • Cube: the full 25→25 encoders restore the response to null-space differences (Part 1). However, the change they predict between paired actions is no closer to the realized change than a prediction of no change, and 50-step success shows no reliable gain over the original design at either seed (4,500 closed-loop runs).
  • TwoRoom: fine-tuning with added commands that execute identically to legal ones gives −3.52 pp [−7.34, −0.08] and −2.19 pp [−5.59, 0.94] against controls fine-tuned the same way without the added commands, at two seeds.
  • PushT: fine-tuning the predictor with an added loss on term B, and a small head trained to predict term B from the model's features and remove it from the score, did not outperform the original model (and, for the loss variant, a control fine-tuned without the B loss) on development episodes, so neither was evaluated on the test set.

Architecture and training changes can make the model register the relevant differences, as in Cube, but have not yet taught it their consequences or improved planning. So far, the gains come from direct changes to the planner's input and output.

We will revise Sections 5–6 to report the deployable gains and these tests, with protocols, controls, and the full comparison of predicted and realized changes in Cube in the appendix.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank all reviewers for their careful reviews. We have uploaded a revised PDF, with changes shown in blue. Original numbering is unchanged, so the references in our individual responses still apply. New results are in Appendix F (Tables 14–21).

Main findings added.

  • The realized-versus-predicted ranking gap persists in later planning calls, clearly in Reacher and PushT.
  • Each mechanism we located alone closes most of its task's ranking gap (67% to 82%). In a diagnostic that corrects the first planning call using executed outcomes, 25-step success rises, and the gain shrinks after normal replanning.
  • The scoring-level mismatch also appears in two other world models, DINO-WM and JEPA-WM, whose internal mechanisms we did not trace. The specific mechanisms are largely tied to LeWM's design: in Cube, retrained models whose action encoder keeps all 25 action dimensions register every tested null-space difference, whereas those whose input is restricted to the public model's 10-dimensional retained component register none.
  • Training-free changes to the planner's input or output improve 50-step success in Cube (+6.13 percentage points), Reacher (+11.87), and PushT (+2.53), positive under every planner seed; architecture and training corrections have not yet improved planning.

Changes and locations.

ChangeLocationRaised by
Contribution framed around the gap between fixed-candidate audits and planningSection 1; transition to Section 6wg41
Ranking gap in later planning callsSection 4.1; Appendix F.1, Table 14fPPt
How much each mechanism explains, for ranking and planningSections 4.4 and 5.2; Appendix F.2, Tables 15–16uuZs
DINO-WM and JEPA-WM on PushT, Wall, and Metaworld reach-wallSection 4.3; Appendix F.3, Tables 17–18fPPt, uuZs
Deployable changes, with five planner seeds and controlsSection 5.2; Appendix F.4, Table 19fPPt, uuZs
Independently trained TwoRoom modelsSection 4.3; Appendix F.5, Table 20fPPt
Cube action-encoder architectures and the comparison of predicted and realized changesSection 4.2; Appendix F.6, Table 21fPPt, uuZs
Training-based corrections for TwoRoom and PushTSection 6; Appendix F.7fPPt, uuZs
Section 6 rewritten: tested suggestions, CEM scope, mechanism specificitySection 6fPPt, uuZs
Provenance of the two TwoRoom weight variantsTable 13 caption; Appendix D.7Clarification

We also added one sentence to the abstract summarizing the deployable improvements and the cross-model result, and a pointer to Appendix F in the reproducibility statement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.