- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper re-examines the public rollout evaluation of Navigation World Models (NWM). It decomposes the rollout error into the one-call error , the chaining cost , and the command error , using additional generations on the same items. It finds defects in the released rollout code, and correcting them lowers the 1B model's rollout LPIPS by 18.8%. All adjudicated claims were pre-registered.
Critical strengths
S1. Verified defects in a public evaluation. The paper identifies concrete defects in the released rollout code, derives their effect (App. B), and confirms their impact with pre-registered tests (H1, H8).
Why it is consequential: The findings are directly useful to users of this evaluation, and show that a rollout score alone cannot reveal such errors.
S2. Rigorous and transparent methodology. The study uses pre-registration, disjoint cohorts, paired comparisons, and trajectory-level bootstrap intervals, and reports voided claims and failed expectations.
Why it is consequential: It is clear which conclusions are confirmed and which are exploratory.
Critical weaknesses
W1. Beyond the code correction, the decomposition provides limited insight into the causes of error growth. The defects identified in Sec. 4 are a useful correction, but the empirical validation is limited to one model family and one dataset. The broader contribution therefore rests on the decomposition into , , and , which combines existing reference comparisons, as the authors note. Its conditional arithmetic holds, and the qualitative point that rollout error growth should not be attributed entirely to accumulated errors is useful for interpreting rollout curves. However, as the authors acknowledge in Sec. 3.1, includes both the uncertainty of the future and the model's weakness at long shifts, and includes sampling noise and the autoencoder round trip in addition to accumulated errors. Both and thus mix several sources, limiting what can be inferred about the underlying causes. In addition, the conditional conclusion that removing the chaining cost would leave at least about half of the rise assumes that stays unchanged, and it is unclear which actual changes to a model would satisfy this condition.
Why it is consequential: Beyond the code correction, the broader value of the paper depends on what the decomposition adds to our understanding of rollout error growth. The authors note that the decomposition does not identify a mechanism, and beyond showing that not all growth comes from accumulated errors, it is unclear what further understanding it provides. (See Q1–Q2.)
W2. The quantitative conclusions about chaining remain uncertain. For the 1B model, the two cohorts give different chaining costs at 8 s (0.0911 vs. 0.0560) and different shares of the rise (about two fifths vs. one sixth). The authors could not identify the cause, as the cohort and the GPU type changed together. In addition, each item is generated once, so it is unclear how much of these estimates reflects random variation in sampling. The analysis of the release's crossover between 1 FPS and 4 FPS is also based on 20 trajectories, and the authors note that its intervals may be too narrow. It is therefore unclear whether this result would hold on a larger set of trajectories.
Why it is consequential: The share of the error growth attributed to chaining is the main quantitative result of the decomposition, and it is not yet stable across cohorts. (See Q3–Q4.)
Questions for authors
I appreciate the rigor of the statistical procedure, and the defects identified in the released rollout code are useful for users of this evaluation. However, the empirical validation is limited to one model family and one dataset, and the broader contribution rests on the decomposition of the rollout error. It is not yet clear what this decomposition reveals about the sources of error growth, and its main quantitative result changes across cohorts. The questions below focus on these points.
[R] = additional result requested, [C] = clarification.
- Q1 [C/R] (W1). Methods that reduce accumulated errors, such as training on the model's own outputs, may change both and . Could the authors clarify what can and cannot be inferred from the decomposition about the effect of such methods? If feasible, applying one such method and reporting how and change would make this concrete, although this is not essential.
- Q2 [C] (W1). includes sampling noise and the autoencoder round trip, which do occur when calls are chained. Are there controls that separate these effects from accumulated prediction errors? If not, how should conclusions about accumulated prediction errors be limited?
- Q3 [R] (W2). The disagreement between the two cohorts could come from the trajectories or from the GPU type, which changed together. Could the authors re-run a subset of trajectories from both cohorts on the same GPU type, with the same seeds and inference settings, to separate the two?
- Q4 [R] (W2). Could the authors repeat the main analyses with several seeds on a subset of trajectories, and report whether the chaining share and the crossover result remain consistent?
Suggestions (not affecting the assessment)
- Sec. 4.1 describes the defects in a few dense sentences with the arithmetic deferred to App. B. A short worked example of how one command is computed in the training convention and in the released code would make the defects much easier to follow.
- A summary table of the main results and an early table defining the claims and cohorts would make the paper easier to follow.