acceptodds
Under review as a conference paper at ICLR 2027

SkipWorld: Latent Skip Planning with Distilled World Models for Efficient Visual Navigation

Abstract

World-model-based visual navigation enables action planning by predicting future states under candidate trajectories. However, long-horizon rollout remains challenging: models are often trained with ground-truth visual histories but deployed in closed-loop prediction, and evaluating many candidates with step-by-step rollout is computationally expensive. We propose SkipWorld, a latent skip planning framework for efficient visual navigation. SkipWorld introduces three components: (i) Latent AutoRoll, which trains a world model to recursively roll out future states in latent space, reducing the mismatch between training and closed-loop inference; (ii) SkipDistill, which distills the latent autoregressive model into a skip world model that directly predicts long-stride future latent states; and (iii) Skip-then-Refine Planning, a dual-model coarse-to-fine planner that first filters candidate trajectories with the skip model and then reranks selected candidates with the stronger autoregressive model. This design preserves accurate trajectory evaluation while reducing redundant rollout computation. Experiments on RECON, HuRoN, SCAND, and TartanDrive show that, compared with NWM, SkipWorld reduces the average ATE by 4.2% and RPE by 2.7%, while decreasing planning time by 62.4% with a 2.7 speedup.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Visual Navigation, World Models, Efficient Planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fztgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SkipWorld, a latent planning framework for visual navigation that combines closed-loop latent rollout training, skip-model distillation, and coarse-to-fine planning.

Critical strengths
  1. The authors tend to address an important efficiency bottleneck in world model-based navigation.

  2. The overall framework is intuitive and clearly presented.

Critical weaknesses
  1. Accuracy gains are relatively modest. The main improvements are in runtime, while the improvements in ATE/RPE over the retrained NWM baseline are fairly small (e.g., ATE 3.33 to 3.19, RPE 0.73 to 0.71).

  2. The effect of SkipDistill is not fully isolated. It remains unclear whether the gains come from the learned skip model itself or simply from evaluating fewer candidates with the stronger teacher model.

  3. The method mainly combines existing ideas, including closed-loop rollout training, distillation, latent prediction, and coarse-to-fine planning. It is difficult to identify a clearly novel technical contribution beyond their combination.

  4. No variance estimates are reported.

Questions for authors
  1. Have the authors evaluated multiple random seeds or measured variance statistics?

  2. Can the authors compare against simpler candidate-filtering baselines?

  3. Can the authors provide a direct quantitative evaluation of skip-model prediction quality?

6 comments
AuthorsOpenReview ↗

Answers for Q1&W4

We thank the reviewer for the thoughtful comments on our method and evaluation.

Q1&W4: Have the authors evaluated multiple random seeds or measured variance statistics?

A1: To assess whether the reported gains persist under planning randomness, we evaluated both SkipWorld and the retrained NWM baseline over three runs, each covering the same 400 queries (100 per dataset). The test queries, model checkpoints, and planning hyperparameters are held fixed across runs, while the evaluation seed varies. The comparison retains the same candidate count, prediction horizon, and dense repetition budget for both methods; SkipWorld uses its distilled student and AutoRoll teacher for screening and refinement. These experiments measure evaluation repeatability with fixed trained models, rather than variation across independently trained checkpoints.

The first two tables show ATE / RPE for each run, followed by an aggregate comparison.

SkipWorld: three evaluation runs

DatasetRun 1Run 2Run 3
RECON1.1908 / 0.34441.2272 / 0.34361.2476 / 0.3509
HuRoN/SACSoN4.8587 / 1.03944.8263 / 1.03444.8692 / 1.0419
TartanDrive5.7362 / 1.19605.7206 / 1.19315.6824 / 1.1841
SCAND0.9803 / 0.27671.0101 / 0.28081.0085 / 0.2795
Four-dataset average3.1915 / 0.71413.1960 / 0.71303.2019 / 0.7141

NWM: three evaluation runs

DatasetRun 1Run 2Run 3
RECON1.3810 / 0.35101.3345 / 0.34711.4545 / 0.3647
HuRoN/SACSoN5.0760 / 1.08305.0600 / 1.08525.0057 / 1.0699
TartanDrive5.7910 / 1.20905.8285 / 1.21615.7748 / 1.2038
SCAND1.0810 / 0.29001.1161 / 0.29561.0938 / 0.2899
Four-dataset average3.3323 / 0.73323.3348 / 0.73603.3322 / 0.7321

Overall comparison: mean ± standard deviation across three runs

DatasetNWM ATESkipWorld ATENWM RPESkipWorld RPE
RECON1.3900 ± 0.06051.2219 ± 0.02870.3543 ± 0.00920.3463 ± 0.0040
HuRoN/SACSoN5.0472 ± 0.03684.8514 ± 0.02241.0794 ± 0.00831.0385 ± 0.0038
TartanDrive5.7981 ± 0.02765.7130 ± 0.02761.2096 ± 0.00621.1911 ± 0.0062
SCAND1.0970 ± 0.01770.9996 ± 0.01680.2919 ± 0.00330.2790 ± 0.0021
Four-dataset average3.3331 ± 0.00153.1965 ± 0.00520.7338 ± 0.00200.7137 ± 0.0007

SkipWorld achieves lower ATE and RPE on every dataset in all three runs. Thus, the improvement is consistent across both datasets and evaluation seeds, rather than being driven by a single favorable run or dataset. The four-dataset mean ATE decreases from 3.3331 to 3.1965 (4.10%), and mean RPE from 0.7338 to 0.7137 (2.73%). Together, these results support the repeatability of the observed planning advantage under changes in evaluation randomness.

AuthorsOpenReview ↗

Answers for Q2&W2

Q2&W2: Can the authors compare against simpler candidate-filtering baselines?

A2: We compared random selection, undistilled teacher skipping, and distilled student screening on 80 fixed queries, 20 per dataset. Each filter selects 20 of the same 120 candidates, followed by evaluation with the same frozen AutoRoll teacher and refinement budget. This separates the value of candidate selection from the benefit of reducing candidate count.

1. Overall planning quality and cost.

MethodAverage ATE ↓Average RPE ↓Seconds/query ↓
Dense teacher3.54150.79083606.7
Random + teacher3.78220.8439654.6
Teacher skip + teacher3.64690.8073804.0
Student + teacher3.48390.7832803.4

At nearly identical runtime, distilled screening lowers observed ATE from 3.6469 to 3.4839 relative to undistilled skipping. Random selection is cheaper but has worse ATE/RPE. Compared with full dense evaluation, the student-screened planner achieves 4.49× acceleration.

2. Dataset-wise planning results.

DatasetDense teacher ATE / RPERandom ATE / RPETeacher skip ATE / RPEStudent ATE / RPE
RECON1.1033 / 0.27701.1585 / 0.30371.0565 / 0.27461.0335 / 0.2774
HuRoN/SACSoN4.2841 / 0.92744.6337 / 0.99724.5652 / 0.97614.2326 / 0.9156
TartanDrive7.4394 / 1.61717.9142 / 1.71937.6286 / 1.65387.3678 / 1.6075
SCAND1.3393 / 0.34171.4223 / 0.35551.3372 / 0.32451.3018 / 0.3321

The student achieves the lowest observed ATE on all four datasets in this diagnostic, showing that the aggregate improvement is not driven by a single dataset.

3. Candidate-ranking quality.

DatasetRandom top-5 Recall@20 ↑Teacher-skip top-5 Recall@20 ↑Student top-5 Recall@20 ↑
RECON16.35%61.00%77.00%
HuRoN/SACSoN16.33%9.00%58.00%
TartanDrive16.79%39.00%48.00%
SCAND17.17%18.00%50.00%
Average16.66%31.75%58.25%

Recall measures retention of the dense teacher's top-five candidates within the 20-candidate shortlist. Random recall averages 100 shortlist draws; its planning results use one fixed shortlist.

The student improves recall over both simpler filters on every dataset. Relative to teacher skipping, average recall increases by 26.5 percentage points, and best-candidate retention rises from 31.25% to 63.75%. This explains the planning benefit: the learned predictor uses the same shortlist budget to retain more teacher-preferred candidates. Together, these experiments support both the distillation target and the learned screening mechanism.

AuthorsOpenReview ↗

Answers for Q3

Q3: Can the authors provide a direct quantitative evaluation of skip-model prediction quality?

A3: We evaluated 400 queries across four datasets, with three stochastic repetitions per query. The table compares predicted endpoints against ground truth at stride eight:

DatasetTeacher LPIPS ↓Student LPIPS ↓Teacher latent RMSE ↓Student latent RMSE ↓
RECON0.31270.59880.74920.5970
HuRoN/SACSoN0.35130.59560.82990.6582
TartanDrive0.38120.77300.78060.6468
SCAND0.45430.72400.94130.7196
Average0.37490.67290.82520.6554

The student has lower latent RMSE but higher decoded-image LPIPS. Thus, it does not reproduce the teacher's visual fidelity; lower latent error should not be interpreted as better image quality.

For screening, the decisive question is whether predictions preserve the relative ordering of candidate trajectories. Ground-truth endpoint error measures reconstruction fidelity, whereas screening compares predicted endpoints with a goal to select promising actions. A coarse predictor can therefore be useful without matching every visual detail, provided its ranking retains candidates that merit teacher refinement.

Our separate 80-query filtering experiment in A2 tests this directly. The student retains 58.25% of the dense teacher's top-five candidates, versus 16.66% for random filtering and 31.75% for undistilled teacher skipping. After refinement, it achieves observed ATE 3.4839, compared with 3.5415 for full dense evaluation, at 4.49× lower planning time.

DatasetRandom top-5 Recall@20 ↑Teacher-skip top-5 Recall@20 ↑Student top-5 Recall@20 ↑
RECON16.35%61.00%77.00%
HuRoN/SACSoN16.33%9.00%58.00%
TartanDrive16.79%39.00%48.00%
SCAND17.17%18.00%50.00%
Average16.66%31.75%58.25%
MethodAverage ATE ↓Average RPE ↓Seconds/query ↓
Dense teacher3.54150.79083606.7
Random + teacher3.78220.8439654.6
Teacher skip + teacher3.64690.8073804.0
Student + teacher3.48390.7832803.4

These results support the intended division of labor: the student supplies inexpensive, useful candidate rankings, while the teacher provides detailed evaluation of the shortlist. The benefit is demonstrated through selection and planning outcomes, rather than inferred from reconstruction quality alone.

Reviewer_1r1QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

World-model-based visual navigation plans actions by predicting the outcomes of candidate trajectories, but long-horizon planning faces two major challenges: (1) accumulated prediction errors from autoregressive rollouts and (2) high computational costs from sequentially evaluating many candidates. SkipWorld addresses these challenges through three components. (1) Latent AutoRoll trains an autoregressive world model using its own predicted latent states as context, (2) SkipDistill trains a student model to predict long-stride future states by matching the endpoints of dense teacher rollouts, and (3) Skip-then-Refine Planning uses the student to screen candidate trajectories before the teacher reranks a shortlist. The authors report substantial improvements in planning efficiency by reducing expensive autoregressive evaluations through this coarse-to-fine strategy.

Critical strengths
  • The method addresses two concrete challenges in world-model-based planning through complementary components. Latent AutoRoll targets the mismatch between training with ground-truth histories and inference with model-generated predictions, which can lead to accumulated errors during autoregressive rollout. SkipDistill addresses the computational cost of sequential prediction by distilling multiple teacher steps into a long-stride student prediction. These components fit together naturally that the teacher is trained to produce more reliable closed-loop predictions, while the student approximates its behavior at a lower computational cost.

  • The coarse-to-fine planning strategy provides a reasonable efficiency–accuracy trade-off. Skip-then-Refine Planning uses inexpensive student predictions to evaluate the full candidate set and reserves expensive autoregressive teacher rollouts for a smaller shortlist. This avoids spending dense rollout computation on every candidate while retaining a refinement stage to improve the reliability of final action selection. The paper also provides component ablations and analyzes key planning choices, including skip stride and shortlist size, helping clarify how these choices affect computational efficiency and planning accuracy.

Critical weaknesses
  • The methodological connection between SkipWorld and visual navigation is insufficiently justified.

    • The authors introduce SkipWorld as "a latent skip planning framework for efficient visual navigation". However, it remains unclear which aspects of the proposed methodology are specifically motivated by, or particularly suited to, visual navigation.
    • The three main component, such as Latent AutoRoll, SkipDistill, and Skip-then-Refine Planning, primarily address general challenges in world-model-based planning, including autoregressive error accumulation and expensive trajectory evaluation. These challenges are not unique to visual navigation, and thus the proposed techniques appear applicable to other world-model planning tasks as well.
    • Although the implementation includes navigation-related choices, such as action aggregation over skip intervals and goal-image-based endpoint scoring, the paper does not sufficiently explain how these choices exploit properties specific to visual navigation or why the proposed framework is particularly effective for this task.
    • Consequently, the relationship between the target task and the methodological contribution remains unclear. If SkipWorld is specifically designed for visual navigation, the authors should better justify how its components are grounded in navigation-specific characteristics. Alternatively, if the method is intended as a general framework for efficient world-model planning, demonstrating its effectiveness on other world-model or robotic planning tasks would help establish the broader applicability of the proposed approach.
  • The reliability of the reported accuracy improvements is difficult to assess.

    • The authors claim that SkipWorld "reduces the average ATE by 4.2%". However, the experiments do not report variation across independent training seeds or provide standard deviations or confidence intervals for the reported accuracy metrics.
    • Furthermore, Table 1 reports an average ATE of 3.19 for SkipWorld, compared with 3.33 for the authors' retrained NWM baseline and 3.04 for NWM (original). Although Section 4.2 acknowledges differences in training settings between the NWM baselines, this distinction makes it important to clarify the conditions under which the reported improvements hold.
    • Without uncertainty estimates and clearer baseline comparability, it is difficult to determine how robust the accuracy improvements are, particularly when the reported gains are relatively modest.
Questions for authors
  • Regarding the connection to visual navigation, which specific characteristics of visual navigation motivate the design of Latent AutoRoll, SkipDistill, and Skip-then-Refine Planning? How do these components exploit properties of visual navigation beyond action aggregation and goal-image-based endpoint scoring? If the framework is intended to be more general, could the authors provide evidence of its applicability to other world-model or robotic planning tasks?

  • Regarding statistical reliability, are the reported accuracy improvements consistent across independent training seeds and repeated evaluations? Could the authors provide standard deviations or confidence intervals for ATE and RPE to clarify the robustness of the reported gains?

  • Regarding baseline comparability, could the authors further clarify how differences in training and evaluation settings between NWM (original), NWM (pretrained), and SkipWorld affect the interpretation of the reported accuracy improvements? In particular, would the improvements remain consistent under fully matched experimental conditions?

5 comments
AuthorsOpenReview ↗

Answers for Q1&W1

We thank the reviewer for the careful assessment and helpful suggestions.

Q1&W1: Which specific characteristics of visual navigation motivate the design of Latent AutoRoll, SkipDistill, and Skip-then-Refine Planning? How do these components exploit properties of visual navigation beyond action aggregation and goal-image-based endpoint scoring? If they address general world-model planning challenges, can their applicability to other models or robotic tasks be demonstrated?

A1: Visual navigation motivates SkipWorld through a specific computational imbalance: searching over low-dimensional robot motions requires expensive predictions of high-dimensional, changing camera views. Candidate paths have not been executed, so their intermediate observations must be imagined rather than measured. Our components address this structure as follows:

Navigation characteristicDesign response
Translation and rotation change the egocentric view; later imagined views depend on earlier predictionsLatent AutoRoll trains on predicted latent histories to match the context encountered during planning.
Reaching a target viewpoint requires multiple motion steps, but coarse scoring uses the endpointSkipDistill learns the multi-step teacher endpoint directly, avoiding intermediate visual predictions during screening.
Many candidate paths branch from the same observed context, although most will be rejectedSkip-then-Refine screens all candidates cheaply and preserves dense, ordered-action evaluation for the shortlist.

These choices connect the method to navigation beyond its action encoding and scoring function. They also apply to other tasks with similar planning structure; the additional cross-model and PushT experiments below test that broader applicability.

Cross-model and cross-task evaluation

We transfer endpoint distillation and candidate screening to RAE-NWM, Vid2World, and JEPA-WM. For each backbone, a student predicts the endpoint of a frozen pretrained teacher's rollout, screens candidate actions, and passes a shortlist to the teacher for refinement. Dense and screened planning are evaluated on the same tasks within each backbone.

Backbone / taskEvaluationDense → screened qualityDense → screened timeSpeedup
RAE-NWM / navigation80 queries; four datasetsATE: 3.8799 → 3.6806; RPE: 0.8989 → 0.8264158.81 → 61.51 s/query2.58×
Vid2World / RECON10 fixed queriesATE: 2.1990 → 2.2045; RPE: 0.4959 → 0.49661916.28 → 597.44 s/query3.21×
JEPA-WM / PushT96 tasks × 3 planning seedsSuccess: 68.06% → 64.24%105.79 → 31.58 s/task3.35×

The additional experiments span navigation with different visual world-model implementations and manipulation with a JEPA-based model. On RAE-NWM, the four-dataset average ATE decreases by 5.14% while planning is 2.58× faster. On Vid2World, the observed ATE changes by only 0.25% while runtime falls by 3.21×. On PushT, JEPA-WM achieves 3.35× acceleration, with success changing from 196/288 to 185/288 evaluations. The latter is a useful speed–quality operating point, with a 3.82-percentage-point success-rate cost.

The RAE-NWM results can also be examined dataset by dataset:

DatasetDense ATE / RPEScreened ATE / RPEDense → screened time (s)Speedup
RECON1.6423 / 0.41661.7562 / 0.4230158.77 → 61.462.58×
HuRoN/SACSoN3.1423 / 0.83922.9023 / 0.6709158.87 → 61.632.58×
TartanDrive (held out from distillation)8.0383 / 1.75867.8284 / 1.7094158.75 → 61.532.58×
SCAND2.6966 / 0.58112.2352 / 0.5022158.87 → 61.412.59×
The student is distilled on RECON, HuRoN/SACSoN, and SCAND. TartanDrive is held out from this distillation stage.

Taken together, these results demonstrate the cross-model transferability of our distill-and-screen strategy. Its benefits therefore extend beyond the original NWM implementation: endpoint distillation and selective teacher refinement provide a transferable approach to reducing planning computation across different world-model representations and action spaces.

AuthorsOpenReview ↗

Answers for Q2&W2

Q2&W2: Are the accuracy results consistent across independent training seeds and repeated evaluations? Can standard deviations or variance estimates be provided?

A2: To assess whether the reported gains persist under planning randomness, we evaluated both SkipWorld and the retrained NWM baseline over three runs, each covering the same 400 queries (100 per dataset). The test queries, model checkpoints, and planning hyperparameters are held fixed across runs, while the evaluation seed varies. The comparison retains the same candidate count, prediction horizon, and dense repetition budget for both methods; SkipWorld uses its distilled student and AutoRoll teacher for screening and refinement. These experiments measure evaluation repeatability with fixed trained models, rather than variation across independently trained checkpoints.

The first two tables show ATE / RPE for each run, followed by an aggregate comparison.

SkipWorld: three evaluation runs

DatasetRun 1Run 2Run 3
RECON1.1908 / 0.34441.2272 / 0.34361.2476 / 0.3509
HuRoN/SACSoN4.8587 / 1.03944.8263 / 1.03444.8692 / 1.0419
TartanDrive5.7362 / 1.19605.7206 / 1.19315.6824 / 1.1841
SCAND0.9803 / 0.27671.0101 / 0.28081.0085 / 0.2795
Four-dataset average3.1915 / 0.71413.1960 / 0.71303.2019 / 0.7141

NWM: three evaluation runs

DatasetRun 1Run 2Run 3
RECON1.3810 / 0.35101.3345 / 0.34711.4545 / 0.3647
HuRoN/SACSoN5.0760 / 1.08305.0600 / 1.08525.0057 / 1.0699
TartanDrive5.7910 / 1.20905.8285 / 1.21615.7748 / 1.2038
SCAND1.0810 / 0.29001.1161 / 0.29561.0938 / 0.2899
Four-dataset average3.3323 / 0.73323.3348 / 0.73603.3322 / 0.7321

Overall comparison: mean ± standard deviation across three runs

DatasetNWM ATESkipWorld ATENWM RPESkipWorld RPE
RECON1.3900 ± 0.06051.2219 ± 0.02870.3543 ± 0.00920.3463 ± 0.0040
HuRoN/SACSoN5.0472 ± 0.03684.8514 ± 0.02241.0794 ± 0.00831.0385 ± 0.0038
TartanDrive5.7981 ± 0.02765.7130 ± 0.02761.2096 ± 0.00621.1911 ± 0.0062
SCAND1.0970 ± 0.01770.9996 ± 0.01680.2919 ± 0.00330.2790 ± 0.0021
Four-dataset average3.3331 ± 0.00153.1965 ± 0.00520.7338 ± 0.00200.7137 ± 0.0007

SkipWorld achieves lower ATE and RPE on every dataset in all three runs. Thus, the improvement is consistent across both datasets and evaluation seeds, rather than being driven by a single favorable run or dataset. The four-dataset mean ATE decreases from 3.3331 to 3.1965 (4.10%), and mean RPE from 0.7338 to 0.7137 (2.73%). Together, these results support the repeatability of the observed planning advantage under changes in evaluation randomness.

AuthorsOpenReview ↗

Answers for Q3&W2

Q3&W2: Could the authors further clarify how differences in training and evaluation settings between NWM (original), NWM (pretrained), and SkipWorld affect the interpretation of the reported accuracy improvements? In particular, would the improvements remain consistent under fully matched experimental conditions?

A3: We retrained NWM specifically to enable a fair comparison. As explained in Table 1 in the manuscript, privacy policy restrictions limit access to HuRoN to a low-resolution subset. Reproducing the original NWM data setting was therefore infeasible; instead, we aligned the baseline and SkipWorld with the same available data.

MethodData and training setting
NWM (original)Published results from the original data setting and 100K-step model.
NWM (pretrained)Initialized from the 100K checkpoint and trained for another 50K steps on our four datasets, including the available low-resolution HuRoN subset.
SkipWorldAutoRoll teacher initialized from the same 100K checkpoint and trained for 50K additional steps on the same datasets; student separately distilled for 50K steps.

As specified in Section 4.1 and Appendix A.1 in the manuscript, the local comparison also uses the same test splits, candidate count, CEM settings, prediction horizon, and dense repetition count. The rollout training and screening mechanisms are the method differences being evaluated.

Under these common data and evaluation conditions, SkipWorld achieves lower reported ATE and RPE on all four datasets, reducing average ATE from 3.33 to 3.19 (4.2%). This gain is measured against the retrained baseline; NWM (original), with ATE 3.04, is included as a published reference. The retraining addresses the data-access mismatch and makes the local comparison fairer, rather than introducing an arbitrary difference in settings.

Reviewer_FYtBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a framework to improve the efficiency of world model-based visual navigation planning. The key idea is to distill a student model that can directly predict longer future states, rather than relying on step-by-step rollouts of the teacher world model for all candidate trajectories. The student model is first used to filter candidate trajectories and identify the most promising ones. These selected candidates are then evaluated using the teacher world model to determine the final trajectories. Experiments on four visual navigation datasets demonstrate significant improvements in planning efficiency while maintaining comparable planning performance.

Critical strengths
  • The proposed framework is well-motivated and provides an effective approach to reducing the computational cost of world model rollouts. Experimental results demonstrate significant speedups in planning tasks.
  • The ablation studies demonstrate the effectiveness of the proposed approach and validate the contributions of its key components.
Critical weaknesses
  • My major concern is that the proposed framework is evaluated only as an extension of the Navigation World Model (NWM). Its generalizability to other world models, particularly more recent SOTA approaches, has not been evaluated or discussed. Evaluating the framework across different world models would help establish its broader applicability and effectiveness.
  • In terms of efficiency, the paper only compares against the vanilla NWM, for which the observed speedup is expected. To further strengthen the paper's contributions, it would be valuable to compare the proposed method against other efficiency focused world model approaches.
  • In the student model, step-by-step actions are directly aggregated into a final position. The rationale behind this design choice requires further clarification and evaluation, particularly regarding its effectiveness in handling complex maneuvers.
Questions for authors
  • I would like to see the proposed framework evaluated on multiple world models rather than focusing solely on NWM. Additionally, comparisons with other efficiency focused world model approaches would help better establish the effectiveness of the proposed method.
  • Please provide further justification for the choice of simply aggregating step-by-step actions into a final position in the student model. In particular, it would be valuable to investigate whether this design remains effective for complex maneuvers.
2 comments
AuthorsOpenReview ↗

Answers for Q1&W1&W2

We thank the reviewer for the constructive feedback and recognition of our motivation.

Q1&W1&W2: Can the framework generalize to other world models, including recent approaches? How does it compare with other efficiency-focused world-model methods?

A1: We now provide evidence on three additional backbones and a comparison with three recent efficiency-oriented approaches.

1. Cross-model and cross-task evaluation

We transfer endpoint distillation and candidate screening to RAE-NWM, Vid2World, and JEPA-WM. For each backbone, a student predicts the endpoint of a frozen pretrained teacher's rollout, screens candidate actions, and passes a shortlist to the teacher for refinement. Dense and screened planning are evaluated on the same tasks within each backbone.

Backbone / taskEvaluationDense → screened qualityDense → screened timeSpeedup
RAE-NWM / navigation80 queries; four datasetsATE: 3.8799 → 3.6806; RPE: 0.8989 → 0.8264158.81 → 61.51 s/query2.58×
Vid2World / RECON10 fixed queriesATE: 2.1990 → 2.2045; RPE: 0.4959 → 0.49661916.28 → 597.44 s/query3.21×
JEPA-WM / PushT96 tasks × 3 planning seedsSuccess: 68.06% → 64.24%105.79 → 31.58 s/task3.35×

The additional experiments span navigation with different visual world-model implementations and manipulation with a JEPA-based model. On RAE-NWM, the four-dataset average ATE decreases by 5.14% while planning is 2.58× faster. On Vid2World, the observed ATE changes by only 0.25% while runtime falls by 3.21×. On PushT, JEPA-WM achieves 3.35× acceleration, with success changing from 196/288 to 185/288 evaluations. The latter is a useful speed–quality operating point, with a 3.82-percentage-point success-rate cost.

The RAE-NWM results can also be examined dataset by dataset:

DatasetDense ATE / RPEScreened ATE / RPEDense → screened time (s)Speedup
RECON1.6423 / 0.41661.7562 / 0.4230158.77 → 61.462.58×
HuRoN/SACSoN3.1423 / 0.83922.9023 / 0.6709158.87 → 61.632.58×
TartanDrive (held out from distillation)8.0383 / 1.75867.8284 / 1.7094158.75 → 61.532.58×
SCAND2.6966 / 0.58112.2352 / 0.5022158.87 → 61.412.59×
The student is distilled on RECON, HuRoN/SACSoN, and SCAND. TartanDrive is held out from this distillation stage.

Taken together, these results demonstrate the cross-model transferability of our distill-and-screen strategy. Its benefits therefore extend beyond the original NWM implementation: endpoint distillation and selective teacher refinement provide a transferable approach to reducing planning computation across different world-model representations and action spaces.

2. Relationship to other efficiency-oriented methods

SkipWorld offers a distinct mechanism for accelerating world-model planning. Relevant published approaches reduce visual token counts or integrate action generation with visual prediction. SkipWorld instead reduces the temporal rollout cost of candidate screening and reserves dense teacher evaluation for a small shortlist.

MethodApproach to efficiencyRuntime reported in the original paper
CompACTCompact visual tokens5.78 / 4.83 s per trajectory optimization with 16 / 8 tokens; RECON, RTX 6000 Ada
SWAMJoint visual-future and action generation16.91 s/sample; serial inference, RECON/SCAND/TartanDrive evaluation
UniWMUnified action prediction and visual foresight82 s/trajectory averaged over Go Stanford/RECON/SCAND/HuRoN

A fully fair direct comparison is difficult because these methods use different model sizes and architectures, target different sources of computational cost, and adopt different hardware and planning settings. We therefore list only the efficiency results reported in their original papers, rather than present them as a matched comparison with SkipWorld.

Our innovation is the coupling of multi-step endpoint distillation with selective teacher refinement, which accelerates planning without requiring a new visual tokenizer or a unified action-generation architecture. Beyond the 2.66× speedup in our main navigation experiment, the cross-model adaptations above achieve 2.58× on RAE-NWM, 3.21× on Vid2World, and 3.35× on JEPA-WM. These results support the effectiveness and transferability of our efficiency mechanism across different world-model architectures and both navigation and manipulation tasks.

AuthorsOpenReview ↗

Answers for Q2&W3

Q2&W3: Why aggregate actions, and does this remain effective for complex maneuvers?

A2: Aggregation is appropriate for the student's endpoint-screening role: it provides compact action conditioning for predicting progress toward a goal image without rolling out every intermediate state. The student also receives scene context and temporal displacement, and learns from endpoints generated by the teacher using the full action sequence. Aggregation is confined to coarse scoring; dense refinement retains the original ordered actions. Its usefulness therefore rests on shortlist quality, rather than lossless trajectory representation.

Turning trajectories. We evaluated 400 queries across four datasets, with three stochastic repetitions, and grouped eight-step segments by cumulative absolute yaw change:

Dataset≤15°: count / student LPIPS15–45°: count / student LPIPS>45°: count / student LPIPS
RECON—25 / 0.578275 / 0.6057
HuRoN/SACSoN3 / 0.528629 / 0.576468 / 0.6067
TartanDrive77 / 0.767117 / 0.79126 / 0.7980
SCAND63 / 0.688629 / 0.77708 / 0.8117

The evaluation includes substantial turning: 75% of RECON and 68% of HuRoN/SACSoN queries exceed 45°. Prediction error increases with turning magnitude, so we also examine whether the coarse predictions remain useful for planning; this stratification alone does not isolate aggregation's effect.

Screening utility. On 80 fixed queries (20 per dataset), all filters select 20 of the same 120 candidates for evaluation by the same frozen teacher:

MethodATE ↓RPE ↓Seconds/query ↓Teacher top-5 Recall@20 ↑Best-candidate retention ↑
Dense AutoRoll teacher3.54150.79083606.7——
Random shortlist + teacher3.78220.8439654.616.66%17.01%
Undistilled teacher skip + teacher3.64690.8073804.031.75%31.25%
Distilled student + teacher3.48390.7832803.458.25%63.75%
The dense baseline evaluates all 120 candidates without screening. Shortlist-retention metrics therefore do not apply.

At nearly identical runtime, distillation raises top-five retention from 31.75% to 58.25% and lowers observed ATE from 3.6469 to 3.4839 relative to undistilled skipping. The full screened planner is 4.49× faster than dense evaluation. These results show that the student remains useful for candidate screening with aggregated action conditioning, while dense refinement evaluates the original ordered actions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback and for recognizing the importance of reducing the computational cost of world-model planning.

The revision includes new results in Sections 4.3–4.4 and Appendices A.7–A.10; changes are marked in blue. The table below summarizes our responses, followed by the supporting evidence.

Concern (raised by)Response and evidence
Accuracy and variability — 1r1Q Q2/W2; fztg W1/W4/Q1Lower ATE and RPE on all four datasets in all three evaluations, alongside 62.4% lower planning time.
Cross-model and cross-task applicability — FYtB Q1/W1; 1r1Q Q1/W12.58–3.35× acceleration on RAE-NWM, Vid2World, and JEPA-WM, including PushT manipulation.
Contribution of learned screening — fztg Q2/W2At nearly identical runtime, distillation improves top-five retention from 31.75% to 58.25% and lowers ATE.
Prediction quality and complex maneuvers — FYtB Q2/W3; fztg Q3Endpoint and turning-stratified evaluations on 400 queries; ordered actions remain available during dense refinement.
Baseline comparability — 1r1Q Q3/W2Local NWM and AutoRoll use the same initialization, available training data, and 50K training steps.
Novelty and efficiency mechanism — FYtB Q1/W2; fztg W3Learned temporal shortcuts reduce sequential prediction during screening; dense simulation is reserved for a shortlist.

1. Joint efficiency and accuracy gains (Table 1; Appendix A.7).

Three fixed-checkpoint evaluations give the following mean ± sample SD:

DatasetNWM ATE ↓SkipWorld ATE ↓NWM RPE ↓SkipWorld RPE ↓
RECON1.390 ± 0.0611.222 ± 0.0290.354 ± 0.0090.346 ± 0.004
HuRoN/SACSoN5.047 ± 0.0374.851 ± 0.0221.079 ± 0.0081.039 ± 0.004
TartanDrive5.798 ± 0.0285.713 ± 0.0281.210 ± 0.0061.191 ± 0.006
SCAND1.097 ± 0.0181.000 ± 0.0170.292 ± 0.0030.279 ± 0.002
Average3.333 ± 0.0013.196 ± 0.0050.734 ± 0.0020.714 ± 0.001

Both metrics improve on every dataset in every run. Together with 2.66× acceleration in the main timing experiment, this demonstrates an improvement in both efficiency and accuracy.

2. Cross-model and cross-task transfer (Section 4.4; Appendix A.10).

Backbone / taskEvaluationDense → screened qualitySpeedup
RAE-NWM / navigation80 queries, four datasetsATE: 3.880 → 3.6812.58×
Vid2World / RECON10 queriesATE: 2.199 → 2.2053.21×
JEPA-WM / PushT96 tasks × 3 seedsSuccess: 68.06% → 64.24%3.35×

The distill-and-screen strategy transfers across representations and action spaces, improving or largely preserving navigation accuracy and accelerating manipulation with a measured success-rate trade-off.

3. Controlled screening comparisons (Section 4.3; Appendix A.8).

On 80 queries, all filters retain 20 of the same 120 candidates for the same frozen teacher.

MethodATE ↓RPE ↓Seconds/query ↓Top-five Recall@20 ↑
Dense teacher3.5420.7913606.7—
Random + teacher3.7820.844654.616.66%
Teacher skip + teacher3.6470.807804.031.75%
Distilled student + teacher3.4840.783803.458.25%

Distillation improves retention and planning accuracy at nearly identical runtime, establishing the benefit of learning which candidates to evaluate.

4. Additional analysis and clarifications.

  • Endpoint quality and turning (Appendix A.9): 400 queries, three repetitions, and strides 2/4/8; 75% of RECON and 68% of HuRoN queries exceed 45° cumulative turning.
  • Fair comparison (Table 1; Section 4.1): the local comparison uses shared available data and evaluation queries, accounting for restricted access to high-resolution HuRoN data.
  • Methodological distinction (Sections 2–3): temporal skipping and selective refinement complement approaches that reduce per-step prediction cost.

Our individual responses provide the remaining reviewer-specific details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.