acceptodds
Under review as a conference paper at ICLR 2027

DO LATENT WORLD MODELS REALLY SELECT THE RIGHT ACTIONS?

Abstract

Latent world models have recently shown strong results in planning from images, particularly on navigation and manipulation tasks. They score candidate action sequences by the distance between their predicted embeddings and the embedding of a goal image. We find that on fixed candidate sets this score usually misses the best action sequence, and that on the manipulation tasks most models do no better than a random choice. On Push-T the released model's predictor causes this. If these scores misrank fixed candidates, why does planning still work so well? Part of the answer is that the same scores rank the planner's own candidates far above a random choice, with the best one on top in 18.1% of Push-T searches against 1.6%, and we show that wider search and more frequent replanning also contribute to this success. PointMaze success falls from 91% to 76% without replanning. On Push-T, a wider search raises success without replanning from 13% to 67%. Further analysis on PointMaze finds a successful action sequence among the planner's candidates in over 90% of episodes, yet the candidate the model scores highest succeeds in 63%. Our findings suggest that strong results in planning do not show that a latent world model selects the right actions.

Primary area:
applications to robotics, autonomy, planning
Keywords:
latent world model, planning, JEPA, model predictive control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vRhhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits whether latent world-model scores correctly rank action sequences by their executed physical outcomes. It evaluates fixed candidate sets, compares image- and video-based representations, bypasses predictors with observed outcomes, and studies the contributions of search budget and replanning. The paper finds severe ranking failures for particular released models on scripted manipulation candidates, while also showing that the same scores rank planner-generated candidates substantially above random. Oracle selection and matched-budget experiments reveal different bottlenecks across PointMaze and Push-T. The main recommendation is to report ranking quality and open-loop performance alongside closed-loop success.

Critical strengths
  1. The distinction between model scoring and overall planner success is important. The study directly measures candidate outcomes rather than inferring model quality from episode success. Hit@1, regret, and rank correlation provide complementary evidence.

  2. The paper includes strong diagnostic interventions. On Push-T, replacing predicted endpoints with observed ones raises Hit@1 from 2.4% to 20.7%, implicating the predictor in the below-random result. On PointMaze, a successful candidate exists in 132 of 144 planner pools, while the top-scored candidate succeeds in only 90. This directly identifies recoverable selection failures.

  3. Counterevidence is reported rather than suppressed. The planner's own Push-T candidates yield 18.1% Hit@1 versus 1.6% at random. At matched total search, one-shot Push-T planning exceeds the replanning condition. End-to-end models also rank expert continuations well. These observations prevent an overly simple conclusion that all latent scores are uninformative.

  4. The appendices provide unusually useful audit detail. They document action preprocessing corrections, simulator replay checks, alternative cost definitions, duplicate-scenario sensitivity, paired statistical tests, and limitations of the attempted learned scorers. The distinction between a diagnostic oracle and a deployable planner is explicit.

Critical weaknesses
  1. Candidate construction limits the breadth of the main negative finding. Navigation uses trajectory-derived candidates, whereas manipulation uses scripted families. On MetaWorld, the largest +y action is best on 619 of 750 anchors; on Push-T, the planner-generated pools reverse the below-random conclusion. These limitations are acknowledged, but a common proposal-family sweep and within-family rankings are needed before interpreting the result as a general manipulation-versus-navigation distinction.

  2. Ranking targets and success criteria are not uniform across experiments. Push-T's main audit uses best-step dense reward, the default score predicts the endpoint, and closed-loop success also checks pusher position and block angle. The terminal-reward sensitivity and success-aligned oracle are useful controls, but the paper's own full-state oracle failure shows how strongly conclusions depend on the target. A consolidated comparison using one success-aligned cost would make the headline claim easier to interpret.

  3. Some audit findings remain entangled with evaluation-interface issues. Appendix A documents disagreement between Wall frames and recorded states, and the Push-T audit and closed-loop tests use different environment implementations. Alternative Wall dynamics are evaluated, so this is not an omitted check. Nevertheless, the cleanest main result should use fully reconciled image/state/action interfaces. Similarly, the encoder-swap models use one training seed and a fixed training recipe, limiting conclusions about encoder families.

  4. The incremental contribution and practical recommendation need sharper boundaries. Prior trajectory reachability work already studies latent-distance misranking and shared-candidate audits. This paper's stronger contribution is its broader diagnosis of predictors, candidate distributions, search, and selection. Open-loop success is useful supplementary evidence, but feedback can legitimately compensate for partial observability and disturbances. A deployable scoring improvement is not necessary for an analysis paper, yet the failed scorer experiments should not be treated as exhausting possible remedies.

Questions for authors
  1. How do fixed scripted, random in-distribution, and CEM-generated candidates compare under the same horizon, candidate count, and physical cost?
  2. Can the main tables include success-aligned terminal regret and top-candidate success alongside exact-best Hit@1, using the same definition throughout each task?
  3. Which findings remain after fully reconciling Wall's visual/state transition mismatch and using one Push-T environment implementation?
  4. Can you separate the consequences of score misranking, executing the CEM elite mean, and insufficient candidate coverage in one summary table?
2 comments
AuthorsOpenReview ↗

Official Response to Reviewer vRhh [1/2]

We thank the reviewer for a review that stated precisely which experiment was missing. We designed the new protocol from your four questions: Tables 1 and 2 answer Q1 and Q2, Table 3 answers Q4, and Q3 determined how both are run.

1. "A common proposal-family sweep and within-family rankings" (W1, Q1)

Table 1 is this sweep. At the planner's first decision in every closed-loop episode (3 seeds x 48 episodes per task and model) we collect four candidate families: the planner's last CEM round, its first CEM round, the recorded action windows from the dataset that start nearest to the current state (Data), and goal-blind sequences such as constant, random and sinusoidal actions (Wide). Each family has 64 candidates of 30 environment steps, all executed from the same start state and judged by the same cost, and each cell ranks within one family.

Normalised regret (NR) of the planning score; 1 is a random pick and 0 is perfect.

TaskModelCEM lastCEM firstDataWide
PointMazeJEPA-WM0.610.390.310.31
PointMazeDINO-WM0.590.350.350.27
Push-TJEPA-WM0.580.310.270.32
Push-TDINO-WM0.570.340.290.47
WallJEPA-WM0.380.370.050.21
WallDINO-WM0.500.490.090.18
MetaWorldJEPA-WM0.940.341.990.25

Your doubt about a general manipulation-versus-navigation distinction was justified. Once the horizon, candidate count and cost are held equal, Push-T behaves like PointMaze and the score beats a random pick in every family, so we have withdrawn that claim. What remains is narrower: MetaWorld is the one task where the score is close to random on the planner's own candidates (0.94, 95% interval 0.87 to 1.02) and worse than random on recorded windows (1.99).

The MetaWorld row has two limitations, both stated in the paper. It covers JEPA-WM only, because the released DINO-WM MetaWorld checkpoint is for a different task. It is also planned with 5 CEM rounds of 64 samples instead of the released 15 x 300, so that the CEM families are the planner's own rounds of 64.

On PointMaze and Push-T the score ranks the last CEM round less well than the first (on Wall the two are nearly equal). We expect this, because the last round has already been selected by the same score.

2. "One success-aligned cost", "top-candidate success, using the same definition throughout" (W2, Q2)

There is now one cost per task, used in all main experiments: the evaluation's own success test written as a margin J, with J < 1 if and only if the episode succeeds. For Push-T, J is the larger of the position error divided by its threshold and the angle error divided by its threshold, at the final step. We no longer use the best-step dense reward, and NR is the terminal regret under J. You also asked for exact-best Hit@1: on the last CEM round it is 4.9% to 30.6% on these three tasks, against 1.6% for a random pick (Appendix C).

Top-candidate success under the same definition, on the last CEM round, among episodes in which some candidate succeeds:

TaskModelEpisodes with a success in the poolTop-scored succeedsRandom candidate succeeds
PointMazeJEPA-WM130 / 14462%38%
PointMazeDINO-WM128 / 14464%38%
Push-TJEPA-WM38 / 14468%7%
Push-TDINO-WM34 / 14471%9%
WallJEPA-WM52 / 14483%11%
WallDINO-WM37 / 14489%11%

Two findings of the submission changed under the single cost.

The first is the below-random result on Push-T. It depended on the dense-reward target and on the scripted candidates, and it does not appear under J.

The second is the Push-T predictor bypass (Hit@1 from 2.4% to 20.7%), which you listed as a strength. Under the single protocol, scoring the observation that each candidate actually reaches changes NR on Push-T only from 0.58 to 0.53 for JEPA-WM, compared with 0.61 to 0.06 on PointMaze and 0.38 to 0.25 on Wall. The predictor is therefore the main source of error on PointMaze, but on Push-T the error lies mostly in the latent distance. The revised paper reports the result in this form.

As a control, scoring the same candidates against the previous episode's goal gives NR between 1.01 and 1.33 on these three tasks, so on the planner's own candidates the ranking depends on the goal. On the three broader families the swapped goal still gives NR below 1 (0.38 to 0.87, Table 5), so part of the ranking there does not depend on the particular goal.

(Continued in 2/2.)

AuthorsOpenReview ↗

Official Response to Reviewer vRhh [2/2]

3. "Which findings remain after fully reconciling Wall's visual/state mismatch and using one Push-T environment?" (W3, Q3)

Every candidate in Tables 1 and 2 is executed in the environment that produces the closed-loop success rate, which is the evaluator's own and is the only one we use for each task. Every observation is rendered by that environment during execution, so the stored Wall frames are not used. Costs are computed from the simulator state and not from frames, and Push-T uses the single implementation that ships with the model release. As a check, we re-executed the logged plan of every plan-once episode from its start state, and it reproduced the logged outcome in all of them.

The findings that remain are those in the two tables. The score ranks better than chance on three tasks for both models, it still misses successes in the pool at the small budget, and at the released budget its top pick succeeds about as often as a random member of the final candidates.

We also disclose one issue with the released Wall environment. A purely vertical move into the border wall produces an undefined position because of a division by zero. This affects two of the 64 Wide candidates in each Wall episode, and we exclude them.

On the encoder-swap models, we agree that one training seed and one recipe cannot support conclusions about encoder families. We did not have the compute to retrain them during the discussion period, so the revision moves this experiment to the appendix and presents it as an observation about those particular checkpoints.

4. "Separate the consequences of score misranking, executing the CEM elite mean, and insufficient candidate coverage in one summary table." (Q4)

Table 3 is this table. For every plan-once episode we execute every member of the final CEM population. "Some member succeeds" measures coverage, "Top-scored member" is the choice the score would make, "Plan once" is the executed elite mean, and "Random member" is the baseline for the score.

Success (%), 144 episodes per row. Small is 5 CEM rounds x 64 samples and released is 30 x 300.

TaskModelBudgetReplanPlan onceSome member succeedsTop-scored memberRandom member
PointMazeJEPA-WMsmall89.661.890.356.234.0
PointMazeJEPA-WMreleased91.077.191.775.776.9
PointMazeDINO-WMsmall82.656.988.956.933.9
PointMazeDINO-WMreleased85.475.086.872.973.6
Push-TJEPA-WMsmall36.112.526.418.11.8
Push-TJEPA-WMreleased68.167.468.866.766.7
Push-TDINO-WMsmall41.016.723.616.72.0
Push-TDINO-WMreleased68.861.863.961.860.4
WallJEPA-WMsmall54.927.136.129.93.9
WallJEPA-WMreleased87.573.684.074.371.7
WallDINO-WMsmall31.916.025.722.92.8
WallDINO-WMreleased72.964.672.264.662.6
  • Misranking is the gap between coverage and the top-scored member. It is largest on PointMaze at the small budget (90.3 against 56.2).
  • The effect of the elite mean is the gap between the top-scored member and the executed plan. It is small and has no consistent sign (+5.6 on PointMaze and -5.6 on Push-T for JEPA-WM at the small budget).
  • Coverage is what limits Push-T and Wall at the small budget (24% to 36%).
  • At the released budget the top-scored member and the elite mean are within 3 points of a random member in every row. On PointMaze coverage is still 14 to 16 points higher, so some successes remain in the population that the score does not pick.

5. Prior trajectory reachability work (W4)

We agree that Li et al. (arXiv:2605.22164) already show that latent distance misranks candidates on a shared candidate set, for LeWM and PLDM on TwoRoom and TopoNav, and that they propose a learned reachability cost. The related-work part of Section 2 now opens with this. Our contribution relative to theirs is to locate the error inside a released planner and measure what it costs: by candidate family (Table 1), by predictor versus latent distance (the bypass control), and by coverage, selection and the executed mean (Table 3), for two released models in their own closed loops.

6. "Feedback can legitimately compensate" (W4)

We agree, and the revision no longer presents planning once as the fair setting. We use it as a diagnostic that shows how much a single decision of the score is worth before feedback corrects it, and we recommend reporting it alongside the closed-loop number and not in its place.

7. "The failed scorer experiments should not be treated as exhausting possible remedies." (W4)

We agree. The text now says that the two learned scorers we tried ranked fixed candidates better without raising planning success, and that other scorers may do better.

We are glad to clarify any of these points further.

Reviewer_C2aMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper starts by noting that world models guide planning by comparing predicted action consequences. However, the authors also note that the distinctions needed here do not always match those that matter for execution. They thus study this mismatch in LeWM across Reacher, TwoRoom, etc., through matched candidate executions and interventions on physical states, commands, scoring, and output formation. The authors claim that their key finding motivate testing both sensitivity to consequential differences and respect for execution equivalence. Together, they argue, this would show how scoring and output formation turn model distinctions into decisions.

Critical strengths

The question the authors address is interesting and well-posed. The Cube finding is interesting and actionable. The paper is upfront about limitations, corrections, etc. For instance, null-results are stated plainly.

Critical weaknesses

The scope of this paper is arguably a bit narrower than its title suggests. For instance, all results come from a single model family. The constructive evidence is also arguably a bit thin. For example, the interventions that seek to improve success either use realized endpoints, or change the physical state itself, rather than the model. Finally, the presentation in this paper is a bit tough-going in places. Page limits notwithstanding, I found the writing here a bit dense, and tricky to follow.

Questions for authors

Have you tested any of the mechanisms on another world model? Additionally, the PushT analysis is a bit hard to interpret. Is there a more transparent way to communicate its point?

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer C2aM

We thank the reviewer for the encouraging comments on the question and on how the null results are reported. We answer your two questions first and then the remaining points.

1. "Have you tested any of the mechanisms on another world model?"

We have now. We added the released DINO-WM checkpoints and ran the whole protocol on them, with the same episodes, candidates and cost as for JEPA-WM. The two models behave very similarly.

Normalised regret of the planning score on the planner's final candidates (1 is a random pick, 0 is perfect):

TaskJEPA-WMDINO-WM
PointMaze0.610.59
Push-T0.580.57
Wall0.380.50

Success (%) at the released search budget, 144 episodes per row:

TaskModelReplanPlan onceTop-scored memberRandom member
PointMazeJEPA-WM91.077.175.776.9
PointMazeDINO-WM85.475.072.973.6
Push-TJEPA-WM68.167.466.766.7
Push-TDINO-WM68.861.861.860.4
WallJEPA-WM87.573.674.371.7
WallDINO-WM72.964.664.662.6

For both models the score ranks better than chance, and for both the top-scored member of the final population succeeds within 3 points of a random member.

2. "The PushT analysis is a bit hard to interpret. Is there a more transparent way to communicate its point?"

The old analysis was hard to read because it used three different notions of a good action: the best dense reward during the rollout, the predicted end state, and the closed-loop success test. The main experiments now use only the last one, the success test of the evaluation itself, and a single Push-T environment.

With this change the Push-T result can be read directly from one row of Table 3. For JEPA-WM at the small search budget, planning once, some candidate in the final pool succeeds in 26.4% of episodes. The top-scored candidate succeeds in 18.1% and a random candidate in 1.8%, so the score is about ten times better than chance. The executed plan succeeds in 12.5%, and replanning raises this to 36.1%. What limits Push-T at this budget is mainly that the pool seldom contains a success. At the released budget the pool contains one in 68.8% of episodes, and the top-scored and random candidates both succeed in 66.7%.

We also need to correct one statement from the submission. If we score the observation that each candidate actually reaches instead of the predicted latent, the regret on Push-T changes only from 0.58 to 0.53, whereas on PointMaze it falls from 0.61 to 0.06. On Push-T most of the regret therefore remains when the reached observations are scored, so a better predictor alone would not fix the ranking there. The submission attributed it to the predictor, based on scripted candidates, and this does not hold under the single protocol.

3. "All results come from a single model family", "narrower than its title suggests."

The study now covers two released families, and the introduction states the scope: two released latent world models on four simulated tasks.

4. "The constructive evidence is a bit thin."

We agree. The paper is a diagnosis and does not propose a better scorer, and we have revised the text so that it no longer suggests otherwise. What the diagnosis gives is an indication of which component to improve. Using the true end state in place of the predicted one removes nearly all of the regret on PointMaze (0.61 to 0.06), which points to the predictor there, while on Push-T it does not, which points to the distance. The two learned scorers we tried are kept in the appendix as negative results, and we do not claim that they exhaust the options.

5. "The writing is a bit dense, and tricky to follow."

We revised the abstract, the introduction and Sections 3 to 6 for this. The setup section now introduces each quantity by what it is needed for, each result section rests on one main table, and the protocol details are in a single appendix section.

Note to the authors and AC: Apologies. My original review was posted to the wrong submission (one with a similar title to this one---hence my confusion), and I have now replaced it with my review of this paper. Apologies also to the authors for the delay this has caused.

Paper Summary

This paper asks whether latent-distance planning objectives rank candidate action sequences in the way their executed outcomes do. The authors show (i) that most models do no better than chance when it comes to manipulation tasks, and that on Push-T the released model's predictor causes this; (ii) that wider search and more frequent replanning contribute to planning success; and (iii) that planners can fail, even when action sequences are present among their candidates.

Critical Strengths

Overall, this paper was clear and easy to follow---even for someone who works in different areas, like me. The question the authors address is clear and well-posed, the methodology is very careful, and the authors are upfront about limitations.

Critical Weaknesses

My one concern is that the paper's results don't quite seem to match its narrative. For example, in section 6, the authors say that the objective ranks the planner's own candidates well, and better fixed-candidate ranking do not improve its success. To me, the contribution would be clearer here if it were framed around the gap between fixed-candidate audits and planning, rather than around models failing to select the right actions. Aside from that, however, I can't find much wrong with the paper.

Questions For Authors

Could the authors mildly reframe the paper, so that it better matches their key results?

Final Verdict

I thought more highly of this paper than the other reviewers, evidently. So I will maintain my original score.

Thank you for the careful updated review, for the kind words about clarity, and for your support of the paper. We agree with your suggestion and have followed it.

"Could the authors mildly reframe the paper, so that it better matches their key results?"

Yes. We reframed the paper around the gap between ranking and planning, as it appears in the revised experiments. In the revised PDF, we made the following changes:

  1. Abstract. The opening sentence now states the key result: "This paper shows that the planning success of a latent world model does not reveal how well its score selects actions."

  2. Two results in the summary were revised when we reran the ranking experiments under one protocol (the same evaluation episodes, and one cost, the evaluation's own success test, for every candidate set). The score beats a random pick on Push-T, as on PointMaze and Wall, so we withdrew the split between navigation and manipulation. On Push-T, bypassing the predictor lowers regret only a little (0.58 to 0.53), so we corrected the statement that the predictor causes the misranking.

  3. The gap between fixed candidates and the planner's own candidates also changed. Under one protocol, the planner's final candidates are ranked least well (normalised regret 0.38 to 0.61, against 0.05 to 0.47 for recorded and hand-built candidates), though still better than chance. The gap that remains, and that the paper is now framed around, is between how well the score ranks and how well the planner succeeds. With a small search on PointMaze, the final candidates contain a success in 90% of episodes, yet the top-scored one succeeds in 56%, and replanning brings success to 90%. With the released search, success is 91%, while the top-scored final candidate succeeds about as often as a random one.

  4. The introduction poses this question and states three observations, and Section 6 answers it directly ("Does planning success tell us? No.").

Thank you again for your constructive comments.

Reviewer_XPXaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how latent world models choose actions when they plan. It uses JEPA-WM (Terver et al, 2025), which has two parts: a frozen DINOv2 encoder and a learned predictor. To plan, the model tries many candidate action sequences. For each one, the predictor estimates where the latent state will end up, and the sequence is scored by its distance to the latent of a goal image. The planner then picks the sequences with the best scores.

The paper asks whether this score really prefers the action sequences that get closest to the goal. To test this, the authors build fixed sets of candidate action sequences. They run each candidate in the simulator to get its true outcome, then check whether the model ranks the candidates in the right order.

Author find that 1) the score ranks better than random on the navigation tasks, but not on the manipulation tasks 2) on Push-T, the authors score the latent of the observation each candidate actually reaches, instead of the predicted latent. Ranking then becomes better than random, 3) on the candidates the planner itself generates, the score ranks well above random.

Critical strengths

Authors asks an important and useful question that success rates alone can’t answer. The recommendation of reporting success without replanning, plus a direct measure of ranking, is also pratical.

Critical weaknesses
  1. The presentation is poor and paper is very hard to follow, and its main message gets lost. Concrete examples:
  • Basic context is missing. JEPA-WM is never introduced. Section 3 opens with “JEPA-WM’s release covers four simulated tasks…” without saying what JEPA-WM is, what the release contains, or that this one release sets the scope of the whole study.
  • The abstract mixes setups. “91% to 76% without replanning” uses the large search budget. Section 5.1 reports 89.6% to 61.8% at the small budget. Readers can’t tell which numbers belong together.
  • The same metric means two things. Hit@3 has two different definitions, and the paper itself warns readers not to compare the values.
  • The paper describes how its experiments evolved: an “earlier version” of the PointMaze audit, a re-run after a Push-T bug, and runs done in “two campaigns”, but this make the experimental setup configusion
  • Sentences are long and convoluted. Many need several readings, for example the explanation of the selection-rule experiment in Section 5.2.
  • Results scattered and hard to compare

In my opinion, the paper needs a major rewrite around one clear message, with the protocol details moved to one appendix section.

  1. The scope of the paper is narrow. It covers one model release and four simple simulated tasks, with no real robot.
Questions for authors

see weaknesses

1 comment
AuthorsOpenReview ↗

Official Response to Reviewer XPXa

We thank the reviewer for the detailed comments on presentation, which we found fair. Since the confusion in the text came from the experiments themselves (several protocols reported side by side), we reran them under one protocol and revised the paper around the results. The General Response describes the protocol; below, we go through your points.

1. "The paper needs a major rewrite around one clear message, with the protocol details moved to one appendix section."

We revised the paper around one message: a latent world model can plan successfully without its score selecting the right actions. The abstract now opens with this sentence, the introduction poses it as a question and answers it with three observations, Sections 4 and 5 each rest on one main table, and the protocol details are collected in Appendix A.

2. "JEPA-WM is never introduced."

Section 2 now gives this background. It explains that a latent world-model planner combines a frozen DINOv2 encoder, a learned predictor, and a CEM search, and that the search scores each action sequence by the latent distance between its predicted end state and the goal image. It then describes what the JEPA-WM and DINO-WM releases contain (checkpoints, tasks, and evaluation configurations) and states that our study is limited to these releases.

3. "The abstract mixes setups."

Thank you for catching this. The 91% to 76% figures were from the released search budget and the 89.6% to 61.8% figures from the small one. The abstract now reports a single number, at the released budget, and each budget has its own labelled row in Table 3.

4. "Hit@3 has two different definitions."

We removed Hit@3. The paper now uses two metrics, each defined once in Section 3: normalised regret (the regret of the top-scored candidate divided by that of a random pick, so 1 is random and 0 is perfect) and the success rate of the top-scored candidate.

5. "An earlier version, a re-run after a Push-T bug, runs done in two campaigns."

We removed this history from the paper. All numbers in the main text now come from a single run of a single protocol (one matched-search comparison in Section 5.1 is a separate, labelled run), with two models, 3 seeds x 48 episodes per task, and one cost per task (the evaluation's own success test, written as a margin).

6. "Sentences are long and convoluted", "results scattered and hard to compare."

The selection-rule experiment in Section 5.2, which you used as the example, is now summarised in Table 3. For the planner's final candidates, each row reports how often some member succeeds, how often the top-scored member succeeds, how often the executed plan succeeds, and how often a random member succeeds. The rows for the released budget are below (success in %, 144 episodes per row).

TaskModelReplanPlan onceSome member succeedsTop-scored memberRandom member
PointMazeJEPA-WM91.077.191.775.776.9
PointMazeDINO-WM85.475.086.872.973.6
Push-TJEPA-WM68.167.468.866.766.7
Push-TDINO-WM68.861.863.961.860.4
WallJEPA-WM87.573.684.074.371.7
WallDINO-WM72.964.672.264.662.6

In every row, the top-scored member and a random member succeed within 3 points of each other, so at this budget the final choice of the score changes the success rate by at most 2.6 points.

7. "The scope is narrow: one model release and four simple simulated tasks, with no real robot."

The study now covers two released model families, JEPA-WM and DINO-WM, and the table above shows the same pattern for both. The ranking results in the General Response cover four tasks.

We could not add a real robot. The measurement requires executing all 64 candidate action sequences from exactly the same start state in every episode, which needs an environment that can be reset exactly, and at present this means a simulator. We state this in the discussion. The abstract names the two models, and the introduction states the scope.

We would be grateful to hear which passages, if any, remain unclear.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to Reviewers

We thank the reviewers for their careful reading. The reviews point to the same problem: the submission mixed several candidate sets, costs, and search budgets, so the results were hard to compare, and the message was hard to find. We agree, so we reran the main experiments under a single protocol and revised the paper around the two tables below (changes are blue). Every new number in our responses comes from this one run.

1. Protocol

We use the released JEPA-WM and DINO-WM checkpoints without retraining on the closed-loop evaluation episodes themselves (3 seeds x 48 per task and model). At the planner's first decision in each episode, we collect 64 action sequences of 30 environment steps from each of four sources: the last CEM round, the first CEM round, recorded action windows from the dataset (Data), and goal-blind sequences such as constant or sinusoidal actions (Wide). Each candidate is executed from the episode's start state in the evaluator's own environment and judged by a single cost J, which is the evaluation's success test written as a margin (J < 1 if and only if the episode succeeds). We report normalised regret (NR), the regret of the top-scored candidate divided by that of a random pick, so 1 is random and 0 is perfect.

2. Does the score rank actions?

Table 1. NR of the planning score (lower is better).

TaskModelCEM lastCEM firstDataWide
PointMazeJEPA-WM0.610.390.310.31
PointMazeDINO-WM0.590.350.350.27
Push-TJEPA-WM0.580.310.270.32
Push-TDINO-WM0.570.340.290.47
WallJEPA-WM0.380.370.050.21
WallDINO-WM0.500.490.090.18
MetaWorldJEPA-WM0.940.341.990.25

On PointMaze, Push-T and Wall the score does better than a random pick for all four sources and both models. The split between navigation and manipulation in the submission came from comparing different costs and candidate sets, and we have withdrawn it. MetaWorld is the exception, with NR 0.94 on the planner's own candidates and 1.99 on recorded windows.

We ran two controls on the last CEM round (Table 2 in the paper). When the same candidates are scored against the previous episode's goal, NR rises to between 1.01 and 1.33 on the three tasks, so this ranking uses the goal. When we score the observation that each candidate actually reaches instead of the predicted latent, NR for JEPA-WM falls from 0.61 to 0.06 on PointMaze and from 0.38 to 0.25 on Wall, but only from 0.58 to 0.53 on Push-T. Prediction error therefore accounts for most of the regret on PointMaze, while on Push-T most of it remains even when the prediction is exact.

3. Where does closed-loop success come from?

Table 3. Success (%), 144 episodes per row. The last three columns come from the final CEM population of the plan-once episodes, where we execute every member. Small is 5 CEM rounds of 64 samples and released is 30 rounds of 300.

TaskModelBudgetReplanPlan onceSome member succeedsTop-scored memberRandom member
PointMazeJEPA-WMsmall89.661.890.356.234.0
PointMazeJEPA-WMreleased91.077.191.775.776.9
PointMazeDINO-WMsmall82.656.988.956.933.9
PointMazeDINO-WMreleased85.475.086.872.973.6
Push-TJEPA-WMsmall36.112.526.418.11.8
Push-TJEPA-WMreleased68.167.468.866.766.7
Push-TDINO-WMsmall41.016.723.616.72.0
Push-TDINO-WMreleased68.861.863.961.860.4
WallJEPA-WMsmall54.927.136.129.93.9
WallJEPA-WMreleased87.573.684.074.371.7
WallDINO-WMsmall31.916.025.722.92.8
WallDINO-WMreleased72.964.672.264.662.6

At the small budget the score selects much better than chance but still misses successes in the pool. On PointMaze with JEPA-WM, 90.3% of episodes have a successful member, yet the top-scored member succeeds in only 56.2%. With replanning, success is 89.6%.

At the released budget the top-scored member, the executed plan and a random member succeed within 3 points of one another in every row. Even in the episodes where some members succeed and others fail, the top-scored member succeeds about as often as a random one (72% against 74% on PointMaze, JEPA-WM). A high closed-loop success rate at this budget therefore tells us little about how well the score selects actions, and this is the main message of the revised paper.

Dear reviewers,

As the discussion deadline approaches, we wanted to follow up on our rebuttal, in which we addressed the concerns raised in your reviews. We would appreciate it if you could take a look at our responses and reconsider your scores in light of them. If any concerns remain, we would be happy to clarify.

Thank you for your time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.