acceptodds
Under review as a conference paper at ICLR 2027

Sanity Checks for Planning with Video World Models

Abstract

Video world models support planning by predicting the outcomes of candidate plans. We examine how strongly those predictions depend on the plan and how consistently the evaluator selects it. We introduce three checks that compare plan changes with sampler noise, compare predictions with simulator futures under alternative plans, and repeat the evaluator’s decision on identical inputs. In World-in-World image-goal navigation with Wan2.2, we analyze 33 episodes and 536 replanning steps. Changing one action affects the goal-similarity score less than changing the sampler seed: the ratio is 0.81 (95% CI [0.69, 0.95]) for the VLM planner and 0.57 ([0.54, 0.60]) for the heuristic planner. Predictions favor their own plan’s true future over perturbed plans’ futures by a margin equal to 1.5% of prediction error. At 43% of evaluated steps, none of five repeated VLM queries reproduces the executed plan. Four training-free confidence signals have episode-success AUCs of 0.47–0.66. Prediction fidelity has the opposite association (AUC = 0.12), with more accurate predictions in failed episodes. These measurements distinguish action sensitivity, decision consistency, and task success, and provide a practical protocol for evaluating confidence signals in video-based planners.

Primary area:
applications to robotics, autonomy, planning
Keywords:
video world models, embodied planning, action conditioning, uncertainty quantification, confidence estimation, decision consistency, image-goal navigation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7TFjOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper conducted three sanity checks for world model planning, including: (1) whether changing the plan affects the imagined score more than changing the sampling seed; (2) whether the imagination is closer to the true future of its own plan than to those of perturbed plans; and (3) whether the VLM evaluator reproduces its own decision under repeated queries with fixed inputs.

Critical strengths

The three checks are important and timely.

Critical weaknesses
  1. The writing of this paper is very unclear. The overall writing and structure are more like an experiment report than an academic paper.
  2. The innovation is limited. Even if the results are interesting, the contribution does not seem to meet the quality of top venues like ICLR.
  3. The quality of the results is doubtful since only one world model and two planners are used.
Questions for authors
  1. Please consider clarifying why the three sanity checks could support a technical paper and the robustness of the results.
  2. Why did the authors choose to ask the three questions for the sanity check? What is their relationship to and meaning for future research?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 7TFj

Thank you for finding the three checks important and timely, and for the direct assessment of the presentation and the contribution. The revision restructures the paper around one question, whether the imagined outcome depends on the plan being scored, and adds theory and experiments that address the contribution and robustness concerns.

Q2 (why these three questions, and what they mean for future work). Planning with a world model chains three components: the generator must produce futures that depend on the plan, those futures must match what the plan would actually do, and the evaluator that compares them must return a stable choice. A failure at any link makes the planner rank something other than its plans, and none of them is visible in success rates: on World-in-World's 144 navigation episodes, an image-to-video model that receives only the initial heading of a plan (40.3%) matches a trajectory-conditioned navigation model (40.3%). Each check tests one link against a reference scale that the system itself supplies (sampler seeds, the simulator's futures of alternative plans, re-queries of the evaluator), which is why they apply to any generator without training. Section 1 now states this argument. Section 6 gives the consequences for future work: report the ratio per conditioning channel, budget rollouts with Proposition 1, calibrate fidelity against the true-future effect, and test confidence signals at the step level with scene and episode length fixed.

Q1 and W2 (what supports a technical paper). Four results go beyond measurement. (i) Proposition 1 shows that the plan-to-seed ratio bounds the expected-score gap between neighboring plans, ∣μ(a′)−μ(a)∣≤2 σR|\mu(a')-\mu(a)|\le\sqrt2\ \sigma R for any score distribution, and, under Gaussian noise, the probability that a planner comparing KK rollouts ranks them correctly, Φ(2K/π R)\Phi(\sqrt{2K/\pi}\ R); at the measured ratios a single rollout is correct at most 74% (VLM planner) and 68% (heuristic planner) of the time. (ii) The channel decomposition locates the failure: in the same model, run and seeds, perturbations that only rotate the generator's input move the score 1.92 times as much as reseeding, while perturbations conveyed only by the text prompt move it 0.32 times as much. This is also a positive control showing that Check 1 discriminates. (iii) The calibrated Check 2 shows that every perturbation turns the true future's final view by 22.5 or 45 degrees and that the re-rendered true futures end 0.62 LPIPS apart, while the imagination recovers 1.6% of that distance. (iv) The within-episode analysis shows that the imagined score predicts the progress of each step (partial correlation 0.19, 95% CI [0.05, 0.32]) while the confidence signals proposed for world models do not, and that the oracle's episode-level inversion comes from differences between episodes.

W3 (one world model and two planners). The revision adds a second generator. The benchmark also provides LTX-Video 0.9.1, a separately trained image-to-video model with its own autoencoder and text encoder. With LTX in place of Wan2.2 and the same view rotation, prompts, seeds and scorer, we regenerated all eight rollouts of all 326 VLM-planner steps. The channel split reappears: text-only perturbations move the LTX score 0.27 times as much as reseeding (95% CI [0.22, 0.33]) and perturbations that change the input view move it 2.42 times as much ([2.05, 2.83]; 2.60 when only the view changes), against 0.32 and 1.48 (1.92) for Wan2.2 on the same steps. On the final frame the LTX ratio is 0.75 ([0.64, 0.87]). On Check 2 the LTX imagination recovers 2.8% of the true-future distance (95% CI [0.5, 5.2]), Wan2.2 1.6% on the same steps. On 32 conditioning frames offline (Figure 2 shows one), writing three turns instead of three forward steps moves LTX's final frame 0.54 times as much as reseeding ([0.39, 0.72]), while the same turns applied as a rotation of the input move it 1.58 times as much; for Wan2.2 the two ratios are 0.61 and 1.05 (Section 5.1, Appendix E). Inside each system the same generator passes Check 1 through one channel and fails it through the other, and Check 2 is calibrated against the simulator's exact futures, re-rendered for all 1,304 compared plans. Robustness is reported with episode-clustered intervals throughout; the main ratios exclude 1 and the plan effect is smaller than the seed effect in 20 of 23 and 10 of 10 episodes.

W1 (writing and structure). The introduction now opens with the problem and the evidence that success rates cannot reveal it, then motivates one check per link of the planning chain and states the proposition and the findings. Section 4 defines each check by a question, a reference scale and its decision consequence, and Section 5 leads each subsection with its finding. Descriptive statistics that did not support a conclusion were moved out of the main text. The title is unchanged; the abstract was revised to state the results directly.

Thank you for your response. I increased my score to 2.

Thank you for reading our response and updating your assessment. In case it is useful alongside your original questions on technical scope and practical use, the latest PDF adds two brief results prompted by our discussion with the other reviewers: Corollary 1 and Appendix J (Table 8), which show that Check 1 (RR) tracks single-rollout agreement with true simulator rankings across both models and channels (from 49% to 68%, rank correlation 0.94), and "Using the checks" in Section 6, which turns Checks 1-3 into a step-by-step pre-deployment decision rule. Thank you again for your time and feedback.

Reviewer_3Bw7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes three sanity checks for planning with video world models, focusing on action sensitivity, prediction fidelity, and VLM evaluator consistency. The authors evaluate these checks on the World-in-World image-goal navigation task using Wan2.2-TI2V-5B and two planners, covering 33 episodes and 536 replanning steps. The results show that sampler variation generally has a larger effect on prediction scores than single-action perturbations, imagined futures exhibit weak preference for their corresponding ground-truth outcomes, and the VLM evaluator shows substantial decision variability across repeated queries. The paper further examines several training-free confidence signals and their relationship with task success.

Critical strengths

S1. The paper addresses an important but often overlooked issue in video world model planning: whether generated predictions meaningfully respond to action conditions and whether planning decisions are consistent. S2. The three proposed sanity checks are intuitive and relatively easy to implement, requiring no additional model training. They offer potentially useful diagnostic tools for analyzing world-model-based planning systems. S3. The experiments are conducted in a closed-loop planning environment, with simulator-rendered futures used for comparison. The statistical analysis also accounts for dependencies between steps within the same episode.

Critical weaknesses

W1. Limited experimental scale and coverage. The evaluation is restricted to a single video world model (Wan2.2-TI2V-5B), one navigation task, and 33 episodes, with the confidence estimation analysis based on only 23 episodes. Furthermore, each configuration in Table 2 is evaluated using only two conditioning images. For a paper whose primary contribution is an evaluation protocol, the current experimental scope is insufficient to establish the robustness and generalizability of the proposed checks. W2. Insufficient direct validation of the proposed sanity checks. The paper mainly reports measurements of the three checks on Wan2.2, without positive or negative control experiments or a clear connection to actual planning quality. For example, it remains unclear whether the plan-to-seed ratio in Check 1 can reliably distinguish world models with different levels of action-conditioning capability. As a result, the experiments primarily characterize the behavior of a particular system rather than establish the diagnostic validity of the proposed evaluation protocol.

Questions for authors

1.Could the authors evaluate the proposed sanity checks on additional video world models or tasks to better demonstrate their robustness and applicability? 2.Could the authors include control models with different action-conditioning capabilities and investigate how the proposed metrics relate to actual planning performance?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3Bw7

Thank you for the clear summary and for the two questions. They point to the two directions in which the revision extends the evidence, its scale and the validity of the checks.

W1 and Q1 (scale and coverage). The closed-loop evidence covers 536 replanning steps with both of the benchmark's planners, 4,288 instrumented rollouts and 2,608 replayed LTX-Video rollouts, 1,620 evaluator re-queries and, new in the revision, 1,304 replayed true futures: the executed plan and the three perturbed plans of every VLM-planner step, re-rendered in the simulator from the decision pose and stepped on the scene's navigation mesh with the simulator's motion model, which reproduces the pose reached after each of the 326 executed plans to within 5e-7 m. Every interval resamples episodes, so the 23 and 10 episodes are the units of inference, and the main ratios exclude 1 at that level (0.81, 95% CI [0.69, 0.95]; 0.57, [0.54, 0.60]; plan effect smaller in 20 of 23 and 10 of 10 episodes). New in the revision, a second generator the benchmark provides, LTX-Video, was replayed on all 326 steps (2,608 rollouts) and run offline on 32 conditioning frames, on which the offline check of Wan2.2 now runs as well (Section 5.1, Appendix E). The checks themselves need only a sampler seed, a way to perturb a plan and, for Check 2, simulator branching or the navigation mesh, so they apply unchanged to every generator and task in the benchmark.

W2 and Q2 (validity of the checks, relation to planning performance). The revision validates the checks in three ways.

Controls with different action-conditioning capability. The benchmark passes the leading turns of a plan to the generator by rotating its input view and the remaining primitives as text. These two channels are conditions with very different action-conditioning capability, inside one model and in the same run under the same seeds, which holds resolution, training data and prompt construction fixed. Splitting the 978 perturbations by channel gives R=0.32R=0.32 (95% CI [0.27, 0.37]) when only the prompt changes and R=1.92R=1.92 ([1.40, 2.41]) when only the input view changes. Check 1 thus separates a conditioning channel the generator follows from one it largely ignores, which is the discrimination you ask about. LTX-Video shows the same split, 0.27 ([0.22, 0.33]) through the prompt and 2.42 ([2.05, 2.83]) through the view (2.60 through the view alone, Figure 1b), and on Check 2 it recovers 2.8% (95% CI [0.5, 5.2]). For Check 2 the simulator is the positive control (normalized margin 100%) and a plan-blind predictor the negative one (0); the imagination scores 1.6% (95% CI [0.3, 2.8]), although every perturbation turns the true future's final view by 22.5 or 45 degrees and the re-rendered true futures of the compared plans end 0.62 LPIPS apart.

A theoretical link between the ratio and planning. Proposition 1 shows that the ratio bounds the difference in expected score between neighboring plans in units of the seed noise, ∣μ(a′)−μ(a)∣≤2 σR|\mu(a')-\mu(a)|\le\sqrt2\ \sigma R for any score distribution, and that under Gaussian noise a planner comparing KK rollouts per candidate ranks two neighboring plans correctly with probability at most Φ(2K/π R)\Phi(\sqrt{2K/\pi}\ R). At the heuristic planner's R=0.57R=0.57 a single-rollout comparison is correct at most 68% of the time, and at least 8 rollouts per candidate are needed for 90%; that planner draws one and succeeds in none of its 10 episodes, against 15 of 23 for the VLM planner (Figure 1c, Section 5.5).

Planning outcomes at the step level. Within episodes, holding scene, goal and episode length fixed, the imagined score predicts the step's progress toward the goal (partial ρw=0.19\rho_w=0.19, 95% CI [0.05, 0.32], geodesic progress), while seed spread, evaluator agreement and the imagination-accuracy oracle do not (Section 5.4, Table 2). World-in-World's own table shows why such checks are needed: on its 144 navigation episodes an image-to-video model that sees only the initial heading (40.3%) matches a trajectory-conditioned navigation model (40.3%), and the default Wan2.2 performs the same with and without action post-training (38.9%), so success rates alone cannot tell whether a generator follows the plan (Section 1).

Thank you for the detailed answers and the additional experiments. I appreciate the effort to address the concerns, particularly the evaluation with LTX-Video and the additional analysis of step-level planning progress. These additions make the experimental evidence more convincing. However, I still have two concerns:

  1. Diagnostic validity: While the results demonstrate that the proposed checks can identify certain failure modes in the evaluated systems, the relationship between these metrics and actual planning performance remains relatively weak. For example, the correlation between the imagined score and step-level progress is only 0.19, while the proposed confidence signals show limited predictive performance at the episode level.

  2. Practical implications: I wonder whether the authors could further clarify how these checks should be interpreted and used in practice. Are they primarily intended as diagnostic tools for identifying potential issues in world models, or can they also provide reliable evidence of a model's actual planning capability? In particular, how should practitioners translate the results of these checks into decisions about whether a world model is suitable for planning?

Clarifying these points would help me better understand the practical significance and scope of the proposed evaluation framework.

Thank you for the follow-up. Both points are now answered in the revised PDF with two new analyses of the existing runs (Sections 5.1 and 5.4, Appendix J) and a procedure for using the checks (Section 6).

1. Diagnostic validity. The 0.19 and the episode-level AUCs are measurements of the benchmark's score and of four training-free confidence signals, which are what the checks test, not values of the checks, and the checks predict them. By Check 1 the score barely depends on the plan, so its association with progress must come from the state. Section 5.4 now tests this directly: the mean score of the three perturbed plans, which were never executed, predicts the executed step's progress as well as the executed plan's own score does (partial ρw=0.20\rho_w = 0.20, 95% CI [0.04, 0.34], against 0.19), whereas the executed plan's advantage over them under the same seed, the comparison a score-ranking planner makes, has ρw=−0.01\rho_w = -0.01 ([-0.11, 0.10]). The weak link between score and planning is the failure that Check 1 detects. The same holds for the confidence signals: computed from rollouts that do not depend on the plan, or from an evaluator that does not reproduce its decision, they have little information about the plan's outcome to carry, and within episodes their conditional rank correlations with progress are near zero up to 13 primitives (∣ρw∣≤0.09|\rho_w| \le 0.09, Appendix F). At the episode level each signal is confounded with scene and episode length (no scene has both a success and a failure), which is why the analysis works within episodes.

The checks themselves track the decision they are meant to predict. Appendix J (Table 8) compares Check 1 with how often a single-rollout comparison orders two plans as their re-rendered simulator futures do. Across the two generators and two channels, R=0.27R = 0.27, 0.32, 1.48 and 2.42 give 49%, 55%, 63% and 68% (chance 50%), every value is below its bound from Corollary 1, and over the six disjoint generator-channel groups the rank correlation between RR and accuracy is 0.94. For LTX-Video, perturbations of the view are ranked correctly 19 points more often than prompt-only ones (95% CI [13, 26]). Check 2 is calibrated in the same way: a plan-blind crop of the input scores -2.7%, a crop rotated by the plan 7.7% and the simulator 100%. In closed loop, the heuristic planner, which ranks only by the score (R=0.57R = 0.57, bound 0.68), succeeds in none of 10 episodes, against 15 of 23 for the VLM planner, which also sees the current view and the goal. At the benchmark level, World-in-World's finding that a generator given only the initial heading (40.3%) matches trajectory conditioning (40.3%) is consistent with Check 1's channel split: the default generator responds to a plan through its initial heading (R=1.92R = 1.92) and hardly through its text (0.32).

2. Practical implications. The checks are diagnostic necessary conditions with quantitative consequences, and the new paragraph "Using the checks" (Section 6) states how to act on them. Each runs on stored states, without closed-loop episodes. Check 1, computed per channel, bounds the average accuracy of the planner's comparisons by Φ(R2K/π)\Phi(R\sqrt{2K/\pi}) and gives the rollout budget K≥(π/2)(z1−α/R)2K \ge (\pi/2)(z_{1-\alpha}/R)^2. If the bound at the affordable KK is below the accuracy the planner needs, ranking by the score is not justified, and the channel or KK must change. Here a single rollout is correct at most 0.74 of the time on average, and plans that differ only in their prompt need at least 25 rollouts for 90%; the benchmark draws one. Check 2 should exceed the crop of the input view rotated by the plan, which needs no generator; Wan2.2 (1.6%) and LTX-Video (2.8%) fall below it (paired 95% CIs [2.7, 10.3] and [2.9, 7.4] points), so their imaginations identify the plan's future less well than rotating the input does. Check 3, at the deployed decoding settings, tells whether the evaluator's choice can be reproduced; at q=0.24q = 0.24, with no match at 43% of steps, decoding should be deterministic or queries aggregated before the choice is trusted or gated by a confidence signal.

A failed check is reliable evidence against the corresponding capability: Proposition 1 bounds every planner that compares the means of KK rollouts per plan, and Table 8 checks the bound against the simulator with no distributional assumption. Passing all three is necessary for the world model to inform the decision through the plan; whether this raises task success also depends on the candidate plans and the scoring rule, which closed-loop evaluation measures. For the default configuration studied here, the checks show that ranking by the score at K=1K = 1 is not justified for plans that differ in their prompt, and that its imagination identifies a plan's future less well than rotating the input does. They point to conveying the whole plan geometrically as the change to test, and Check 1 tests it without closed-loop runs.

Reviewer_FVhFOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces three tests for planning with video world models. The first shows how changing one action in a plan compares with changing the world model sampling seed. The second compares a putative future with simulated (rendered) futures under the executed plan and under perturbations. The third queries repeatedly the VLM evaluator on same inputs, measuring how often it reproduces the executed plan. Further, the paper evaluates four training free confidence signals. The obtained results are interpreted as evidence that action sensitivity, evaluator consistency, and task success are distinct properties that should be evaluated separately.

Critical strengths
  • The main contribution lies in the isolation of three questions that, to my understanding, are often conflated in the literature of world model planning. Sensitivity of imagined outcomes to actions, evaluator reproducibility, and prediction of downstream task success are, indeed, genuinely different properties. The proposed separation is simple and potentially useful.

  • The paper recognizes dependence among steps, constructs cluster-bootstrap confidence intervals using episodes as the resampling units, and uses episode-level sign-flip tests for the main inferential claims. It also explicitly reports important confounds and negative results. I appreciated the transparency and willingness to report inconclusive confidence signals.

Critical weaknesses
  • The central metric does not seem to be implemented as the paper defines it. Eq. 1 and the surrounding text define the score using LPIPS between the goal and the final frame of the imagined video. In the implementation, it is explicitly computed as a mean over all timestamps. Thus the headline ratios 0.81 and 0.57, Figure 1a, and the SC-k/APS confidence features are based on a temporally averaged LPIPS score, not the final-frame score described in the paper. For this reason the intended final-frame results cannot be reconstructed and it is unknown whether the main conclusion survives under the stated metric.

  • The empirical evidence is not yet broad or controlled enough to support several of the paper's interpretations. The main evaluation covers a single world model and task, only 33 episodes, and no positive-control model known to exhibit strong action conditioning. The offline study is even more limited, relying on two conditioning images and fixed random seeds. More importantly, Check 2 does not measure how far apart the simulator futures of the original and perturbed plans actually are. A cross/self ratio close to one could therefore indicate that the model ignores actions, but it could equally arise because the selected perturbations lead to visually similar true outcomes. The relationship between the proposed signals and episode success is likewise difficult to disentangle. The analysis contains only 15 successful and 8 failed episodes; all failures exhaust the 21-step budget; and success and failure never occur within the same scene. The oracle AUC of 0.12 is consequently confounded with episode length, agent behavior, and scene identity. Although the authors discuss these limitations, the current experiments do not control for them. The confidence-signal estimates are also too uncertain to demonstrate either predictive value or a meaningful absence of it. Thus, the paper currently presents a promising diagnostic case study rather than a well-validated, general evaluation methodology.

Questions for authors
  • Please clarify whether the intended use was for the final-frame LPIPS or the benchmark's temporally averaged per_video LPIPS. Please trace the exact tensor shape and returned field used for the reported scores. If final-frame LPIPS was intended, can the authors recompute all Check-1 ratios, Figure 1a, SC-k, and APS using only the last frame and report whether the conclusions change? Providing per-frame distances or the final generated frames would materially address this concern.

  • How different are the simulator-rendered true futures under the executed and perturbed plans? Please report a direct true-future effect such as the LPIPS distance between o*(a) and each o*(a'). Without this, what evidence rules out the interpretation that Check 2 is close to one because many interventions have little visual consequence?

  • Is it possible to add a positive control (for example, a trajectory-conditioned navigation world model expected to respond strongly to actions) and at least one additional model or task? Demonstrating that the checks distinguish a known action-sensitive system from the present system would substantially strengthen the claim that these are validated sanity checks rather than merely plausible descriptive statistics.

5 comments
AuthorsOpenReview ↗

Response to Reviewer FVhF (1/2)

(1/2) Thank you for reading the code closely and for catching the mismatch between Eq. (1) and the scorer. Your three requests led to the largest additions of the revision; each is answered below with the new measurements and their place in the revised PDF.

Q1 (which LPIPS score). The benchmark's scorer, evaluation/FVD/calculate_lpips.py, receives the goal image repeated over time and the imagined clip, both of shape [B, 14, 3, 384, 384], computes LPIPS frame by frame, and returns per_video, the mean over the 14 kept frames, whatever the value of only_final. World-in-World's own heuristic planner (solver_IGNav.fetch_action_by_LPIPS) ranks its candidates by this per_video value, and our hook called the same function on the same tensors. The ratios 0.81 and 0.57, Figure 1a, SC-k and APS are therefore measured on the frame-averaged score, which is exactly the quantity the planner optimizes. Check 2 passes only the last frame (a[-1:]), so its distances are final-frame distances. You are right that the submitted Eq. (1) described the final frame. The revised Eq. (1) defines both the frame-averaged score SS, which the planner uses, and the final-frame score SfinS_{\mathrm{fin}}, and Appendix C traces the call, the tensors and the returned field.

Three final-frame measurements point the same way. The closed-loop run stored the frame-averaged scores rather than the generated videos, but seeds and perturbed plans are deterministic functions of episode and step, and the observation, plan and goal image of every step are saved, so the replay script in the supplement regenerates all eight rollouts of any VLM-planner step with the benchmark's own preprocessing and records the LPIPS of every frame. Run with LTX-Video, the second generator the benchmark provides, on all 326 steps, it gives a final-frame plan-to-seed ratio of 0.75 (95% CI [0.64, 0.87]; Section 5.1). Check 2 compares final frames and finds the imagination almost blind to the plan (next paragraph). The offline check compares final frames of whole three-turn plans written as text: with LTX on 32 conditioning frames the plan moves the final frame 0.54 times as much as the seed ([0.39, 0.72]), and with Wan2.2 on the same frames 0.61 times as much ([0.51, 0.71]), less than the seed in all four generator configurations (Appendix E).

Q2 (how different are the true futures). We now measure the true-future effect exactly. The run logged the distances between imagined and true frames but not the true frames themselves, so for every VLM-planner step we re-rendered the simulator futures of the executed plan and of its three perturbed plans from the decision pose; the re-rendered start view equals the saved observation pixel for pixel at all 326 steps. Replaying the same plans on the navigation mesh with the simulator's motion model gives their exact final poses (the executed primitives land on the recorded next pose to within 5e-7 m). The interventions have large consequences. Because the 13-primitive futures differ in their net number of turns, every perturbation turns the final view by 22.5 degrees (81% of perturbations) or 45 degrees (19%), a quarter or half of the 90 degree field of view, and the final positions are 0.73 m apart on average. With the benchmark's LPIPS, the final frames of the compared true futures are δ⋆=0.62\delta^\star=0.62 apart (95% CI [0.60, 0.66]), 63 times the imagination's margin of 0.010. Divided by δ⋆\delta^\star, the margin recovers 1.6% of the difference (95% CI [0.3, 2.8]), where an imagination equal to the true future scores 100% and a plan-blind one 0, and it does not grow with δ⋆\delta^\star but shrinks slightly (within-episode rank correlation −0.14, p = 0.01). Check 2 is near zero because the imagination does not separate futures that are far apart, not because the futures are alike (Section 5.2, Figure 3a, Appendix D). LTX-Video, replayed on all 326 of them, recovers 2.8% (95% CI [0.5, 5.2]; Section 5.1).

Continued in part 2/2 below.

AuthorsOpenReview ↗

Response to Reviewer FVhF (2/2)

(2/2) Q3 (positive control). The deployed system contains a channel that the generator follows, which gives a positive control in the same model, the same run and under the same seeds. The evaluation pipeline applies the leading run of same-direction turns by rotating the panoramic observation before the 90 degree input view is cropped, and passes the remaining primitives to the generator as text. Because the perturbed plans are deterministic, we reconstructed all 978 of them and split them by channel. The 577 perturbations that change only the prompt give R=0.32R=0.32 (95% CI [0.27, 0.37]); the 68 that change only the input view, with an identical prompt, give R=1.92R=1.92 ([1.40, 2.41]); all 401 that change the view give R=1.48R=1.48 ([1.25, 1.73]). Check 1 therefore reports a large ratio when the action reaches the generator geometrically and a small one when it arrives as text. Geometric conditioning is how trajectory-conditioned navigation models such as NWM receive actions, so the view channel is the in-system counterpart of the positive control you suggest, with resolution, training data and prompt construction held fixed. LTX-Video, a second generator the benchmark provides, shows the same split on all 326 steps: 0.27 ([0.22, 0.33]) through the prompt and 2.42 ([2.05, 2.83]) through the view (2.60 through the view alone, Figure 1b). For Check 2 the positive control is the simulator itself, which scores 100% on the normalized margin, against 1.6% for the imagination.

Link to planning quality. Proposition 1 (new, proof in Appendix B) turns the ratio into a decision quantity. For any score distribution ∣μ(a′)−μ(a)∣≤2 σR|\mu(a')-\mu(a)|\le\sqrt2\ \sigma R, and under Gaussian noise a planner that compares the means of KK rollouts per candidate orders two neighboring plans correctly with probability at most Φ(2K/π R)\Phi(\sqrt{2K/\pi}\ R). At R=0.57R=0.57 this caps single-rollout comparisons at 0.68, and the heuristic planner, which ranks single rollouts by this score, succeeds in 0 of 10 episodes; plans that differ only in their prompt (R=0.32R=0.32) would need at least 25 rollouts each for 90% accuracy (Figure 1c). Measured against the re-rendered simulator futures, scored the same way, single-rollout comparisons of neighboring plans have the sign of the true score difference 58% of the time (95% CI [55, 62]), where chance is 50%.

Confounds in the success analysis. We added a within-episode analysis that holds scene, goal and episode length fixed. The outcome is the progress of each step, the reduction in geodesic distance to the goal achieved by the executed primitives, computed on the navigation mesh; plan composition, remaining distance and step index are partialled out. The imagined score predicts progress (partial ρw=0.19\rho_w=0.19, episode-clustered 95% CI [0.05, 0.32]) information that its episode-level AUC of 0.50 hides. SC-k, VLM-C and the imagination-accuracy oracle carry no step-level information (−0.03, 0.01 and −0.04, every interval within ±0.18), which bounds what they could contribute, and LL and APS are weaker (0.15 and 0.11). The oracle's episode-level inversion is a between-episode effect: 40% of its variance lies between episodes, and within episodes its correlation with progress is −0.04 ([−0.17, 0.08]). Section 5.4, Figure 3c and Table 2 report these results; Euclidean progress gives the same picture (score 0.24, [0.08, 0.40]).

Scope of the evidence. The conclusions rest on 536 closed-loop replanning steps, 4,288 instrumented rollouts, 2,608 replayed LTX-Video rollouts, 1,620 evaluator re-queries and 1,304 re-rendered true futures, with every interval clustered by episode. The revision adds the score definition, the channel decomposition as a positive control, its replication with a second generator, Proposition 1, the calibration of Check 2 against re-rendered true futures and the within-episode outcome analysis. The title is unchanged; the abstract and Sections 1, 4 and 5 were revised to report these results.

Thank you for the additional analysis. Several of my original concerns are addressed. In particular, the revised definition of SS resolves the mismatch, the re-rendered simulator futures address my concern that Check 2 could be uninformative, and LTX replication and expansion of the offline study also strengthen the evidence.

I have three remaining questions about the interpretation of the revised results.

  • Proposition 1 fixes a decision point and a pair of plans, whereas the reported experimental RR pools effects over heterogeneous steps and perturbations. Appendix B calls the numerical accuracies and rollout counts “plug-in values,” but Sections 5.1 and 5.5 describe bounds on each comparison. What justifies this aggregation? In particular, the pooled ratio weights local ratios by their seed effects, so it need not bound individual or ordinary average comparison accuracy. Please either establish the required aggregation result or qualify the numerical caps and rollout counts throughout as conditional illustrations. Please also distinguish ranking by expected imagined score from ranking by true simulator outcomes.

  • I appreciate the view-channel control as evidence that Check 1 detects a strong input effect within this pipeline. However, rotating an already observed panorama changes the pixels supplied to the generator; a model that preserves its conditioning image could respond strongly without accurately predicting subsequent action effects. Could the paper distinguish this narrower validation from validation against a model independently known to predict action-conditioned trajectories? Similarly, the simulator’s 100% normalized margin is a useful reference endpoint obtained by construction.

  • The within-episode analysis is useful and reduces the between-episode confounding I identified, but its endpoint is immediate progress rather than eventual episode success. Moreover, small partial rank correlations do not establish that the signals “carry no information” or bound their general predictive contribution. Please scope these conclusions to the measured conditional associations and acknowledge the different prediction horizons of the imagination-accuracy oracle and executed-step progress.

For completeness, the new final-frame experiments provide useful supporting evidence, although they do not reconstruct the original Wan2.2 final-frame Check-1 and confidence-feature analyses.

A smaller clarification: Tables 3 and 5 describe bootstrap intervals over frames, but the rebuttal states that every interval is clustered by episode. Please clarify the episode/scene coverage of the 32 frames and the resampling procedure.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank the three reviewers for their careful reading, which extended to the code. The reviews raised four requests: the exact score behind Check 1, the size of the true-future effect behind Check 2, controls that validate the checks, and analyses of confidence signals that are not confounded by scene and episode length. The revised PDF adds each of them, keeps every submitted number, and is organized around a single question: does the imagined outcome depend on the plan being scored?

Score definition (FVhF). The benchmark's LPIPS scorer returns the mean over the 14 kept frames, and World-in-World's heuristic planner ranks candidates by this mean, so the submitted ratios are measured on the planner's own decision variable. Eq. (1) now defines both this score and the final-frame score, and Appendix C traces the call. Check 2, the offline check and the LTX replay, whose final-frame ratio is 0.75 (95% CI [0.64, 0.87]), compare final frames and agree with Check 1.

Where the action is lost, as a positive control (all reviewers). The benchmark passes the leading turns of a plan by rotating the generator's input view and the rest as text. Perturbations that change only the prompt give a plan-to-seed ratio of 0.32 (95% CI [0.27, 0.37]); perturbations that change only the input view give 1.92 ([1.40, 2.41]), in the same model, run and seeds. Check 1 separates a channel the generator follows from one it ignores (Section 5.1, Figure 1b).

A second generator (3Bw7, 7TFj). The benchmark also provides LTX-Video 0.9.1, a separately trained image-to-video model with its own autoencoder and text encoder. With LTX in place of Wan2.2 and the same view rotation, prompts, seeds and scorer, we regenerated all eight rollouts of all 326 VLM-planner steps. The channel split reappears: text-only perturbations move the LTX score 0.27 times as much as reseeding (95% CI [0.22, 0.33]) and perturbations that change the input view move it 2.42 times as much ([2.05, 2.83]; 2.60 when only the view changes), against 0.32 and 1.48 (1.92) for Wan2.2 on the same steps. On 32 conditioning frames offline, rewording the prompt with the same actions moves LTX's final frame 82% as much as changing three actions (Wan2.2: 92%), and the same turns applied as a rotation of the input view move it 1.58 times as much as reseeding (Section 5.1, Figure 2, Appendix E). Check 2 replicates as well: against the re-rendered true futures, the LTX imagination recovers 2.8% of the distance between the compared futures (95% CI [0.5, 5.2]) against 1.6% for Wan2.2 on the same steps, where the simulator scores 100%.

Calibrated Check 2 (FVhF). We re-rendered the simulator futures of all 1,304 compared plans from the decision pose (the re-rendered start view equals the saved observation pixel for pixel at every step) and replayed them on the navigation mesh, which reproduces every executed pose to within 5e-7 m. Each perturbation turns the true future's final view by 22.5 or 45 degrees and moves it by 0.73 m on average, and the final frames of the compared futures are 0.62 LPIPS apart, 63 times the imagination's margin. The imagination recovers 1.6% (95% CI [0.3, 2.8]) of the difference between the futures it should tell apart (Section 5.2, Figure 3a). Scored the same way, single-rollout comparisons of neighboring plans have the sign of the true score difference 58% of the time, where chance is 50% (Section 5.1).

Theory (3Bw7, 7TFj). Proposition 1 shows that the plan-to-seed ratio bounds the expected-score gap between neighboring plans in units of seed noise for any score distribution and, under Gaussian noise, the probability that a planner comparing K rollouts ranks them correctly. At the measured ratios a single rollout is correct at most 74% (VLM planner) and 68% (heuristic planner) of the time, and plans that differ only in their prompt need at least 25 rollouts each for 90% (Figure 1c).

Step-level confidence analysis (FVhF, 3Bw7). Within episodes, with scene, goal and length fixed, the imagined score predicts each step's geodesic progress toward the goal (partial correlation 0.19, 95% CI [0.05, 0.32]), while seed spread, evaluator agreement and the imagination-accuracy oracle do not; the oracle's episode-level inversion is a between-episode effect (Section 5.4, Table 2).

The title is unchanged. The abstract and Sections 1, 4 and 5 were revised to report these results, the references were checked against their sources, and the related work now also covers video language planning, visual foresight, WorldSimBench, action-adapted and camera-controlled video diffusion, the physical fidelity of video generators and training-free uncertainty for video models. Code and per-step outputs for every new number are in the supplementary material.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.