acceptodds
Under review as a conference paper at ICLR 2027

Separating Command Error, Chaining Cost and One-Call Error in the Rollout Evaluation of a Video World Model

Abstract

Action-conditioned video world models are usually evaluated by rolling them out under logged actions and plotting the error against horizon, and the growth of that error is attributed to errors compounding through the model's own outputs. The rollout score alone cannot support that attribution, because it has no reference: it does not show whether the evaluation gave the model the right commands, how much of the growth a single call to the same moment would already show, or whether the actions mattered. For models that take a time shift as input, three additional generations on the same items provide these references without retraining: a rollout with the commands recomputed as in training, one call to each scored horizon, and a rollout that repeats the first command. We applied them to the public evaluation of a released family of navigation world models (NWM; 49M to 1.01B parameters), on fresh test cohorts and with decision rules registered in advance. Correcting the released rollout code, which gives every call after the first a wrong command, lowers the 1B model's 4 FPS rollout LPIPS by 18.8% [16.9, 20.6]; in a post-hoc analysis the growth of the command error accounts for a third of the released curve's rise from 2 to 8 s. The release's authors attribute a crossover between their 1 FPS and 4 FPS rollouts to accumulated errors and loss of context. On the release's own 20-trajectory protocol, the released 1B checkpoint shows this crossover under the released commands only in the point estimates, and it disappears once the commands are corrected (post hoc; we cannot tell whether their figure used this code). With correct commands, the growth of the error that chaining adds over a single call accounts for between a sixth and two fifths of the rise from 2 to 8 s, and the two cohorts' estimates of that growth disagree beyond their intervals (post hoc). The larger share's post-hoc interval reaches about one half, so removing this chaining cost while leaving the one-call error unchanged would leave at least about half of the rise. Scale lowers the one-call error, although the released 1B checkpoint is worse than the 700M one on single calls (post hoc). The effect of scale on the chaining cost is unresolved on LPIPS; on DreamSim the cost decreases. We recommend reporting rollout scores together with these references.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, action-conditioned video prediction, camera control, navigation, planning, homography, evaluation, scaling

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_BNL6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper re-examines the public rollout evaluation of Navigation World Models (NWM). It decomposes the rollout error into the one-call error JJ, the chaining cost CC, and the command error KK, using additional generations on the same items. It finds defects in the released rollout code, and correcting them lowers the 1B model's rollout LPIPS by 18.8%. All adjudicated claims were pre-registered.

Critical strengths

S1. Verified defects in a public evaluation. The paper identifies concrete defects in the released rollout code, derives their effect (App. B), and confirms their impact with pre-registered tests (H1, H8).

Why it is consequential: The findings are directly useful to users of this evaluation, and show that a rollout score alone cannot reveal such errors.

S2. Rigorous and transparent methodology. The study uses pre-registration, disjoint cohorts, paired comparisons, and trajectory-level bootstrap intervals, and reports voided claims and failed expectations.

Why it is consequential: It is clear which conclusions are confirmed and which are exploratory.

Critical weaknesses

W1. Beyond the code correction, the decomposition provides limited insight into the causes of error growth. The defects identified in Sec. 4 are a useful correction, but the empirical validation is limited to one model family and one dataset. The broader contribution therefore rests on the decomposition into JJ, CC, and KK, which combines existing reference comparisons, as the authors note. Its conditional arithmetic holds, and the qualitative point that rollout error growth should not be attributed entirely to accumulated errors is useful for interpreting rollout curves. However, as the authors acknowledge in Sec. 3.1, JJ includes both the uncertainty of the future and the model's weakness at long shifts, and CC includes sampling noise and the autoencoder round trip in addition to accumulated errors. Both JJ and CC thus mix several sources, limiting what can be inferred about the underlying causes. In addition, the conditional conclusion that removing the chaining cost would leave at least about half of the rise assumes that JJ stays unchanged, and it is unclear which actual changes to a model would satisfy this condition.

Why it is consequential: Beyond the code correction, the broader value of the paper depends on what the decomposition adds to our understanding of rollout error growth. The authors note that the decomposition does not identify a mechanism, and beyond showing that not all growth comes from accumulated errors, it is unclear what further understanding it provides. (See Q1–Q2.)

W2. The quantitative conclusions about chaining remain uncertain. For the 1B model, the two cohorts give different chaining costs at 8 s (0.0911 vs. 0.0560) and different shares of the rise (about two fifths vs. one sixth). The authors could not identify the cause, as the cohort and the GPU type changed together. In addition, each item is generated once, so it is unclear how much of these estimates reflects random variation in sampling. The analysis of the release's crossover between 1 FPS and 4 FPS is also based on 20 trajectories, and the authors note that its intervals may be too narrow. It is therefore unclear whether this result would hold on a larger set of trajectories.

Why it is consequential: The share of the error growth attributed to chaining is the main quantitative result of the decomposition, and it is not yet stable across cohorts. (See Q3–Q4.)

Questions for authors

I appreciate the rigor of the statistical procedure, and the defects identified in the released rollout code are useful for users of this evaluation. However, the empirical validation is limited to one model family and one dataset, and the broader contribution rests on the decomposition of the rollout error. It is not yet clear what this decomposition reveals about the sources of error growth, and its main quantitative result changes across cohorts. The questions below focus on these points.

[R] = additional result requested, [C] = clarification.

  • Q1 [C/R] (W1). Methods that reduce accumulated errors, such as training on the model's own outputs, may change both JJ and CC. Could the authors clarify what can and cannot be inferred from the decomposition about the effect of such methods? If feasible, applying one such method and reporting how JJ and CC change would make this concrete, although this is not essential.
  • Q2 [C] (W1). CC includes sampling noise and the autoencoder round trip, which do occur when calls are chained. Are there controls that separate these effects from accumulated prediction errors? If not, how should conclusions about accumulated prediction errors be limited?
  • Q3 [R] (W2). The disagreement between the two cohorts could come from the trajectories or from the GPU type, which changed together. Could the authors re-run a subset of trajectories from both cohorts on the same GPU type, with the same seeds and inference settings, to separate the two?
  • Q4 [R] (W2). Could the authors repeat the main analyses with several seeds on a subset of trajectories, and report whether the chaining share and the crossover result remain consistent?

Suggestions (not affecting the assessment)

  • Sec. 4.1 describes the defects in a few dense sentences with the arithmetic deferred to App. B. A short worked example of how one command is computed in the training convention and in the released code would make the defects much easier to follow.
  • A summary table of the main results and an early table defining the claims and cohorts would make the paper easier to follow.
Reviewer_ipTnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines rollout evaluation of action-conditioned video world models, arguing that error growth with prediction horizon alone cannot establish error accumulation through recursive generation. For models accepting a time-shift input, it adds matched references: a rollout with corrected commands, a single call to the target horizon, and a rollout repeating the initial command. These probe command construction, chaining cost, and dependence on action conditioning. An audit of NWM's public evaluation finds that correcting command construction reduces prediction error and changes the interpretation of some observed patterns. Chaining explains only part of the error growth, with one-call error also contributing substantially. Larger models generally improve one-call prediction, but changes in chaining cost depend on the evaluation metric and remain partly unresolved.

Critical strengths

S1. Exceptionally rigorous and transparent methodology. Every adjudicated claim was pre-registered with an estimand, threshold, control and kill condition (Sec. 3.3; App. A, pp. 13–14); the five fresh cohorts are mutually disjoint and disjoint from the release's index files (App. D, pp. 14–15); intervals come from resampling whole trajectories rather than frames (20,000 draws), and arms are paired through shared first-call noise (Sec. 3.2, p. 4); every post-hoc analysis is labelled, and a Bonferroni check is provided (App. G, p. 16). The paper also honestly reports failed expectations and voided claims (Apps. C, E) as well as compute and cost (≈60 GPU-hours, ≈USD 45; App. F).

S2. A concrete, verifiable and consequential finding. App. B (p. 14) derives step by step the command-construction error in the released rollout code: normalizing before differencing cancels the intercept, so the model decodes each command as ΔX+1.25\Delta X + 1.25; commands also stay in the first frame's axes, and the heading wraps. The registered test shows that correcting it lowers the 1B model's 4 FPS rollout LPIPS by 18.8% [16.9, 20.6] (Sec. 4.2, p. 5), and Sec. 7 provides command-composition unit tests that need no frame generation.

Critical weaknesses

W1. Narrow scope: whether the conclusions extend beyond one release is not demonstrated

The authors state that "None of the three is new as an idea" (p. 2). The empirical part covers one model family, one dataset (RECON), one checkpoint per size and one sample per item (Sec. 7, Limitations, p. 9). The command-construction error is specific to NWM's released code, and the paper cannot determine whether the NWM paper's figures came from this code path (pp. 5, 9). The general recommendation in Sec. 7 ("Generate the three references alongside every rollout") therefore rests on a single case.

W2. The quantitative answer to the core question is post hoc, inconsistent across cohorts, and affected by a single-future metric

The motivating question is whether rollout error growth comes from compounding. Yet the chaining growth over the 2→8 s rise is 0.0564 [0.0382, 0.0745] on one cohort and 0.0217 [0.0059, 0.0377] on the other, with non-overlapping intervals (shares 0.3953 and 0.1725), and the 1B model's 32-call cost at 8 s is 0.0911 vs. 0.0560, again with non-overlapping intervals (Sec. 5.1, p. 7). The two runs used different GPU types (A100 vs. A30; App. D, Table 1, p. 15), and the authors acknowledge that cohort variation, hardware numerics and chance "cannot be separated after the fact" (p. 7). The registered scale test H9 was "foreseeably underpowered" (p. 9). Moreover, JJ includes the irreducible uncertainty of a single recorded future and there is one sample per item (pp. 4, 9), so "the one-call error accounts for the larger part of the rise" may largely reflect intrinsic uncertainty about the future rather than model error.

Questions for authors

See the Weaknesses section for details.

Reviewer_Uo3rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that a rollout score alone cannot justify attributing the error growth of action-conditioned video world models to compounding, because it lacks a reference. For time-shift-conditioned models, the authors add three references without retraining: a rollout with commands recomputed in the training convention, a single call to each scored horizon, and a rollout that repeats the first command. Applying them to the public evaluation of the released navigation world models with fresh cohorts and pre-registered decision rules, they find that the released rollout code gives every call after the first a wrong command, and correcting it substantially lowers the rollout error. With correct commands, chaining accounts for a minority of the error growth on the point estimates. Scale generally lowers the one-call error, but the chaining cost decreases detectably with scale on DreamSim.

Critical strengths
  1. The paper identifies a concrete and verifiable defect in the released rollout evaluation code. Because the actions are normalized before being differenced, the intercept cancels, and every call after the first is commanded to move 1.25 waypoint units too far forward. These issues raise concerns about the correctness of the rollout evaluation.

  2. The three references are simple and inexpensive. They require no retraining. Each command reference costs one additional rollout per item, and the one-call reference costs one call per item and scored horizon. Because the one-call reference conditions on the same real context frames as the rollout, its difference from the rollout reflects the prediction procedure rather than the visual information available to the model.

  3. Section 7 proposes checks of the command composition that can be run before any frame is generated. Examples include requiring that a constant-velocity straight path yield the same command at every call, and that every command decodes back to the logged relative pose. The authors note that the latter check alone catches all three defects.

Critical weaknesses
  1. The decomposition E_released = J + C + K presented in the introduction is never computed on a single cohort: K is estimated on the confirmation cohort, whereas J and C are estimated on the probe and scale cohorts. Comparisons between the command error and the chaining cost are therefore unpaired. Running the one-call arm on the confirmation cohort, on the same GPU type as its 1B arms, would provide all three terms with paired intervals.

  2. The two cohorts disagree on the chaining cost. For the 1B model, the 8 s cost is 0.091 on the probe cohort and 0.056 on the scale cohort, with non-overlapping intervals. The corresponding shares of the 2–8 s rise attributable to chaining are 0.40 and 0.17. The authors state that cohort variation, hardware numerics, and chance cannot be separated after the fact. However, the two 1B runs used different GPU types (A100 vs. A30), whereas the 50M model, generated on the A30 for both cohorts, agrees across them. I suggest rerunning 1B probe arms on an A30, which would directly test the hardware explanation.

  3. The share of the rise attributed to chaining requires qualification. Each item is sampled once and scored against a single recorded future, so part of the growth of the one-call error may reflect irreducible uncertainty rather than a model deficiency. In that case, chaining would account for a larger fraction of the reducible error, and several samples per item would help separate the two.

  4. All results come from a single model family on a single dataset, and the one-call reference requires time-shift conditioning. The evaluation is confined to frame-level comparisons, with no downstream task such as planning. I suggest three additions: (i) repeating the main analyses on at least one further dataset that the released models are trained and evaluated on; (ii) applying the two command references, which require no time shift, to at least one fixed-step world model; (iii) testing with the released planner whether the chaining cost affects planning performance.

  5. The writing should be improved. The text reports a large number of estimates, and much of it describes results without explaining them. For example, Section 6.2 reports eight per-size estimates, two slopes, and two between-size differences for the chaining cost. These show no trend with scale on LPIPS but a decrease on DreamSim. Except for noting that the LPIPS test was underpowered, the paper does not interpret this difference. The main text should convey a clear storyline and explanation without interrupting the reader with too many numbers.

Questions for authors
  1. Could you run the one-call arm on the confirmation cohort on the same GPU type?

  2. Could you rerun the 1B probe arms on an A30 to test whether the GPU type explains the cross-cohort disagreement on the chaining cost?

  3. How much of the growth of the one-call error do you attribute to irreducible uncertainty about the future rather than to a deficiency of the model?

  4. Have you applied the analysis to another dataset that the release trains and evaluates on? Does chaining cost affect the released planner's performance?

Reviewer_VR8ZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper audits the evaluation process of the Navigation World Model (NWM) family. It analyzes the sources of error in rollout predictions via three references: corrected-command rollouts, direct predictions of the same target frames, and rollouts that repeat the first command. This paper also points out command errors in publicly released code. After errors are corrected, the 1B model achieves an 18.8% reduction in LPIPS. This indicates that the growth of error cannot be simply attributed to accumulation.

Critical strengths
  • Positive empirical results on NWM. This paper specifies command errors in the evaluation code for NWM and quantifies the impact through experiments. The 1B model’s 4FPS rollout LPIPS reduces by 18.8% after correction, demonstrating the value.

  • Honest experiment reporting. This paper distinguishes tests from post-hoc analyses explicitly. Inconclusive results, failed expectations and limitations are reported as well. This transparency is greatly helpful for readers to assess the strength of evidence.

Critical weaknesses
  • The writing and presentation need substantial improvement. The structure of this paper is unconventional, and the paper lacks clear statements for its contributions and background information. Also, the motivation is not carefully articulated in this paper, leading to difficulty in understanding the experiments. In addition, terms in the table in Appendix A are not clearly defined, and mathematical notation is not used.

  • Concern for the generalizability of the methodology. The improvement of the NWM family relies on the fix of a command error. It is unclear whether this command error is widespread across world models or is specific to NWM. Therefore, I’m concerned about whether the methodology is feasible in different models.

  • Lack of a focused conclusion. This paper presents findings on multiple aspects, but does not synthesize them into a clear central conclusion. It would be better to explicitly elaborate a focused conclusion.

Questions for authors

1.Could you evaluate the proposed methodology of eliminating command error on other world models outside the NWM family? This can demonstrate its general applicability. 2.Could you clarify the motivation and articulate the main contributions of this paper relative to prior work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.