- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper audits whether latent world-model scores correctly rank action sequences by their executed physical outcomes. It evaluates fixed candidate sets, compares image- and video-based representations, bypasses predictors with observed outcomes, and studies the contributions of search budget and replanning. The paper finds severe ranking failures for particular released models on scripted manipulation candidates, while also showing that the same scores rank planner-generated candidates substantially above random. Oracle selection and matched-budget experiments reveal different bottlenecks across PointMaze and Push-T. The main recommendation is to report ranking quality and open-loop performance alongside closed-loop success.
Critical strengths
-
The distinction between model scoring and overall planner success is important. The study directly measures candidate outcomes rather than inferring model quality from episode success. Hit@1, regret, and rank correlation provide complementary evidence.
-
The paper includes strong diagnostic interventions. On Push-T, replacing predicted endpoints with observed ones raises Hit@1 from 2.4% to 20.7%, implicating the predictor in the below-random result. On PointMaze, a successful candidate exists in 132 of 144 planner pools, while the top-scored candidate succeeds in only 90. This directly identifies recoverable selection failures.
-
Counterevidence is reported rather than suppressed. The planner's own Push-T candidates yield 18.1% Hit@1 versus 1.6% at random. At matched total search, one-shot Push-T planning exceeds the replanning condition. End-to-end models also rank expert continuations well. These observations prevent an overly simple conclusion that all latent scores are uninformative.
-
The appendices provide unusually useful audit detail. They document action preprocessing corrections, simulator replay checks, alternative cost definitions, duplicate-scenario sensitivity, paired statistical tests, and limitations of the attempted learned scorers. The distinction between a diagnostic oracle and a deployable planner is explicit.
Critical weaknesses
-
Candidate construction limits the breadth of the main negative finding. Navigation uses trajectory-derived candidates, whereas manipulation uses scripted families. On MetaWorld, the largest +y action is best on 619 of 750 anchors; on Push-T, the planner-generated pools reverse the below-random conclusion. These limitations are acknowledged, but a common proposal-family sweep and within-family rankings are needed before interpreting the result as a general manipulation-versus-navigation distinction.
-
Ranking targets and success criteria are not uniform across experiments. Push-T's main audit uses best-step dense reward, the default score predicts the endpoint, and closed-loop success also checks pusher position and block angle. The terminal-reward sensitivity and success-aligned oracle are useful controls, but the paper's own full-state oracle failure shows how strongly conclusions depend on the target. A consolidated comparison using one success-aligned cost would make the headline claim easier to interpret.
-
Some audit findings remain entangled with evaluation-interface issues. Appendix A documents disagreement between Wall frames and recorded states, and the Push-T audit and closed-loop tests use different environment implementations. Alternative Wall dynamics are evaluated, so this is not an omitted check. Nevertheless, the cleanest main result should use fully reconciled image/state/action interfaces. Similarly, the encoder-swap models use one training seed and a fixed training recipe, limiting conclusions about encoder families.
-
The incremental contribution and practical recommendation need sharper boundaries. Prior trajectory reachability work already studies latent-distance misranking and shared-candidate audits. This paper's stronger contribution is its broader diagnosis of predictors, candidate distributions, search, and selection. Open-loop success is useful supplementary evidence, but feedback can legitimately compensate for partial observability and disturbances. A deployable scoring improvement is not necessary for an analysis paper, yet the failed scorer experiments should not be treated as exhausting possible remedies.
Questions for authors
- How do fixed scripted, random in-distribution, and CEM-generated candidates compare under the same horizon, candidate count, and physical cost?
- Can the main tables include success-aligned terminal regret and top-candidate success alongside exact-best Hit@1, using the same definition throughout each task?
- Which findings remain after fully reconciling Wall's visual/state transition mismatch and using one Push-T environment implementation?
- Can you separate the consequences of score misranking, executing the CEM elite mean, and insufficient candidate coverage in one summary table?
2 comments
Official Response to Reviewer vRhh [2/2]
3. "Which findings remain after fully reconciling Wall's visual/state mismatch and using one Push-T environment?" (W3, Q3)
Every candidate in Tables 1 and 2 is executed in the environment that produces the closed-loop success rate, which is the evaluator's own and is the only one we use for each task. Every observation is rendered by that environment during execution, so the stored Wall frames are not used. Costs are computed from the simulator state and not from frames, and Push-T uses the single implementation that ships with the model release. As a check, we re-executed the logged plan of every plan-once episode from its start state, and it reproduced the logged outcome in all of them.
The findings that remain are those in the two tables. The score ranks better than chance on three tasks for both models, it still misses successes in the pool at the small budget, and at the released budget its top pick succeeds about as often as a random member of the final candidates.
We also disclose one issue with the released Wall environment. A purely vertical move into the border wall produces an undefined position because of a division by zero. This affects two of the 64 Wide candidates in each Wall episode, and we exclude them.
On the encoder-swap models, we agree that one training seed and one recipe cannot support conclusions about encoder families. We did not have the compute to retrain them during the discussion period, so the revision moves this experiment to the appendix and presents it as an observation about those particular checkpoints.
4. "Separate the consequences of score misranking, executing the CEM elite mean, and insufficient candidate coverage in one summary table." (Q4)
Table 3 is this table. For every plan-once episode we execute every member of the final CEM population. "Some member succeeds" measures coverage, "Top-scored member" is the choice the score would make, "Plan once" is the executed elite mean, and "Random member" is the baseline for the score.
Success (%), 144 episodes per row. Small is 5 CEM rounds x 64 samples and released is 30 x 300.
| Task | Model | Budget | Replan | Plan once | Some member succeeds | Top-scored member | Random member |
|---|---|---|---|---|---|---|---|
| PointMaze | JEPA-WM | small | 89.6 | 61.8 | 90.3 | 56.2 | 34.0 |
| PointMaze | JEPA-WM | released | 91.0 | 77.1 | 91.7 | 75.7 | 76.9 |
| PointMaze | DINO-WM | small | 82.6 | 56.9 | 88.9 | 56.9 | 33.9 |
| PointMaze | DINO-WM | released | 85.4 | 75.0 | 86.8 | 72.9 | 73.6 |
| Push-T | JEPA-WM | small | 36.1 | 12.5 | 26.4 | 18.1 | 1.8 |
| Push-T | JEPA-WM | released | 68.1 | 67.4 | 68.8 | 66.7 | 66.7 |
| Push-T | DINO-WM | small | 41.0 | 16.7 | 23.6 | 16.7 | 2.0 |
| Push-T | DINO-WM | released | 68.8 | 61.8 | 63.9 | 61.8 | 60.4 |
| Wall | JEPA-WM | small | 54.9 | 27.1 | 36.1 | 29.9 | 3.9 |
| Wall | JEPA-WM | released | 87.5 | 73.6 | 84.0 | 74.3 | 71.7 |
| Wall | DINO-WM | small | 31.9 | 16.0 | 25.7 | 22.9 | 2.8 |
| Wall | DINO-WM | released | 72.9 | 64.6 | 72.2 | 64.6 | 62.6 |
- Misranking is the gap between coverage and the top-scored member. It is largest on PointMaze at the small budget (90.3 against 56.2).
- The effect of the elite mean is the gap between the top-scored member and the executed plan. It is small and has no consistent sign (+5.6 on PointMaze and -5.6 on Push-T for JEPA-WM at the small budget).
- Coverage is what limits Push-T and Wall at the small budget (24% to 36%).
- At the released budget the top-scored member and the elite mean are within 3 points of a random member in every row. On PointMaze coverage is still 14 to 16 points higher, so some successes remain in the population that the score does not pick.
5. Prior trajectory reachability work (W4)
We agree that Li et al. (arXiv:2605.22164) already show that latent distance misranks candidates on a shared candidate set, for LeWM and PLDM on TwoRoom and TopoNav, and that they propose a learned reachability cost. The related-work part of Section 2 now opens with this. Our contribution relative to theirs is to locate the error inside a released planner and measure what it costs: by candidate family (Table 1), by predictor versus latent distance (the bypass control), and by coverage, selection and the executed mean (Table 3), for two released models in their own closed loops.
6. "Feedback can legitimately compensate" (W4)
We agree, and the revision no longer presents planning once as the fair setting. We use it as a diagnostic that shows how much a single decision of the score is worth before feedback corrects it, and we recommend reporting it alongside the closed-loop number and not in its place.
7. "The failed scorer experiments should not be treated as exhausting possible remedies." (W4)
We agree. The text now says that the two learned scorers we tried ranked fixed candidates better without raising planning success, and that other scorers may do better.
We are glad to clarify any of these points further.
Official Response to Reviewer vRhh [1/2]
We thank the reviewer for a review that stated precisely which experiment was missing. We designed the new protocol from your four questions: Tables 1 and 2 answer Q1 and Q2, Table 3 answers Q4, and Q3 determined how both are run.
1. "A common proposal-family sweep and within-family rankings" (W1, Q1)
Table 1 is this sweep. At the planner's first decision in every closed-loop episode (3 seeds x 48 episodes per task and model) we collect four candidate families: the planner's last CEM round, its first CEM round, the recorded action windows from the dataset that start nearest to the current state (Data), and goal-blind sequences such as constant, random and sinusoidal actions (Wide). Each family has 64 candidates of 30 environment steps, all executed from the same start state and judged by the same cost, and each cell ranks within one family.
Normalised regret (NR) of the planning score; 1 is a random pick and 0 is perfect.
Your doubt about a general manipulation-versus-navigation distinction was justified. Once the horizon, candidate count and cost are held equal, Push-T behaves like PointMaze and the score beats a random pick in every family, so we have withdrawn that claim. What remains is narrower: MetaWorld is the one task where the score is close to random on the planner's own candidates (0.94, 95% interval 0.87 to 1.02) and worse than random on recorded windows (1.99).
The MetaWorld row has two limitations, both stated in the paper. It covers JEPA-WM only, because the released DINO-WM MetaWorld checkpoint is for a different task. It is also planned with 5 CEM rounds of 64 samples instead of the released 15 x 300, so that the CEM families are the planner's own rounds of 64.
On PointMaze and Push-T the score ranks the last CEM round less well than the first (on Wall the two are nearly equal). We expect this, because the last round has already been selected by the same score.
2. "One success-aligned cost", "top-candidate success, using the same definition throughout" (W2, Q2)
There is now one cost per task, used in all main experiments: the evaluation's own success test written as a margin J, with J < 1 if and only if the episode succeeds. For Push-T, J is the larger of the position error divided by its threshold and the angle error divided by its threshold, at the final step. We no longer use the best-step dense reward, and NR is the terminal regret under J. You also asked for exact-best Hit@1: on the last CEM round it is 4.9% to 30.6% on these three tasks, against 1.6% for a random pick (Appendix C).
Top-candidate success under the same definition, on the last CEM round, among episodes in which some candidate succeeds:
Two findings of the submission changed under the single cost.
The first is the below-random result on Push-T. It depended on the dense-reward target and on the scripted candidates, and it does not appear under J.
The second is the Push-T predictor bypass (Hit@1 from 2.4% to 20.7%), which you listed as a strength. Under the single protocol, scoring the observation that each candidate actually reaches changes NR on Push-T only from 0.58 to 0.53 for JEPA-WM, compared with 0.61 to 0.06 on PointMaze and 0.38 to 0.25 on Wall. The predictor is therefore the main source of error on PointMaze, but on Push-T the error lies mostly in the latent distance. The revised paper reports the result in this form.
As a control, scoring the same candidates against the previous episode's goal gives NR between 1.01 and 1.33 on these three tasks, so on the planner's own candidates the ranking depends on the goal. On the three broader families the swapped goal still gives NR below 1 (0.38 to 0.87, Table 5), so part of the ranking there does not depend on the particular goal.
(Continued in 2/2.)