- Rating
- 2
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SkipWorld, a latent planning framework for visual navigation that combines closed-loop latent rollout training, skip-model distillation, and coarse-to-fine planning.
Critical strengths
-
The authors tend to address an important efficiency bottleneck in world model-based navigation.
-
The overall framework is intuitive and clearly presented.
Critical weaknesses
-
Accuracy gains are relatively modest. The main improvements are in runtime, while the improvements in ATE/RPE over the retrained NWM baseline are fairly small (e.g., ATE 3.33 to 3.19, RPE 0.73 to 0.71).
-
The effect of SkipDistill is not fully isolated. It remains unclear whether the gains come from the learned skip model itself or simply from evaluating fewer candidates with the stronger teacher model.
-
The method mainly combines existing ideas, including closed-loop rollout training, distillation, latent prediction, and coarse-to-fine planning. It is difficult to identify a clearly novel technical contribution beyond their combination.
-
No variance estimates are reported.
Questions for authors
-
Have the authors evaluated multiple random seeds or measured variance statistics?
-
Can the authors compare against simpler candidate-filtering baselines?
-
Can the authors provide a direct quantitative evaluation of skip-model prediction quality?
6 comments
Answers for Q2&W2
Q2&W2: Can the authors compare against simpler candidate-filtering baselines?
A2: We compared random selection, undistilled teacher skipping, and distilled student screening on 80 fixed queries, 20 per dataset. Each filter selects 20 of the same 120 candidates, followed by evaluation with the same frozen AutoRoll teacher and refinement budget. This separates the value of candidate selection from the benefit of reducing candidate count.
1. Overall planning quality and cost.
| Method | Average ATE ↓ | Average RPE ↓ | Seconds/query ↓ |
|---|---|---|---|
| Dense teacher | 3.5415 | 0.7908 | 3606.7 |
| Random + teacher | 3.7822 | 0.8439 | 654.6 |
| Teacher skip + teacher | 3.6469 | 0.8073 | 804.0 |
| Student + teacher | 3.4839 | 0.7832 | 803.4 |
At nearly identical runtime, distilled screening lowers observed ATE from 3.6469 to 3.4839 relative to undistilled skipping. Random selection is cheaper but has worse ATE/RPE. Compared with full dense evaluation, the student-screened planner achieves 4.49× acceleration.
2. Dataset-wise planning results.
| Dataset | Dense teacher ATE / RPE | Random ATE / RPE | Teacher skip ATE / RPE | Student ATE / RPE |
|---|---|---|---|---|
| RECON | 1.1033 / 0.2770 | 1.1585 / 0.3037 | 1.0565 / 0.2746 | 1.0335 / 0.2774 |
| HuRoN/SACSoN | 4.2841 / 0.9274 | 4.6337 / 0.9972 | 4.5652 / 0.9761 | 4.2326 / 0.9156 |
| TartanDrive | 7.4394 / 1.6171 | 7.9142 / 1.7193 | 7.6286 / 1.6538 | 7.3678 / 1.6075 |
| SCAND | 1.3393 / 0.3417 | 1.4223 / 0.3555 | 1.3372 / 0.3245 | 1.3018 / 0.3321 |
The student achieves the lowest observed ATE on all four datasets in this diagnostic, showing that the aggregate improvement is not driven by a single dataset.
3. Candidate-ranking quality.
| Dataset | Random top-5 Recall@20 ↑ | Teacher-skip top-5 Recall@20 ↑ | Student top-5 Recall@20 ↑ |
|---|---|---|---|
| RECON | 16.35% | 61.00% | 77.00% |
| HuRoN/SACSoN | 16.33% | 9.00% | 58.00% |
| TartanDrive | 16.79% | 39.00% | 48.00% |
| SCAND | 17.17% | 18.00% | 50.00% |
| Average | 16.66% | 31.75% | 58.25% |
Recall measures retention of the dense teacher's top-five candidates within the 20-candidate shortlist. Random recall averages 100 shortlist draws; its planning results use one fixed shortlist.
The student improves recall over both simpler filters on every dataset. Relative to teacher skipping, average recall increases by 26.5 percentage points, and best-candidate retention rises from 31.25% to 63.75%. This explains the planning benefit: the learned predictor uses the same shortlist budget to retain more teacher-preferred candidates. Together, these experiments support both the distillation target and the learned screening mechanism.
Answers for Q3
Q3: Can the authors provide a direct quantitative evaluation of skip-model prediction quality?
A3: We evaluated 400 queries across four datasets, with three stochastic repetitions per query. The table compares predicted endpoints against ground truth at stride eight:
| Dataset | Teacher LPIPS ↓ | Student LPIPS ↓ | Teacher latent RMSE ↓ | Student latent RMSE ↓ |
|---|---|---|---|---|
| RECON | 0.3127 | 0.5988 | 0.7492 | 0.5970 |
| HuRoN/SACSoN | 0.3513 | 0.5956 | 0.8299 | 0.6582 |
| TartanDrive | 0.3812 | 0.7730 | 0.7806 | 0.6468 |
| SCAND | 0.4543 | 0.7240 | 0.9413 | 0.7196 |
| Average | 0.3749 | 0.6729 | 0.8252 | 0.6554 |
The student has lower latent RMSE but higher decoded-image LPIPS. Thus, it does not reproduce the teacher's visual fidelity; lower latent error should not be interpreted as better image quality.
For screening, the decisive question is whether predictions preserve the relative ordering of candidate trajectories. Ground-truth endpoint error measures reconstruction fidelity, whereas screening compares predicted endpoints with a goal to select promising actions. A coarse predictor can therefore be useful without matching every visual detail, provided its ranking retains candidates that merit teacher refinement.
Our separate 80-query filtering experiment in A2 tests this directly. The student retains 58.25% of the dense teacher's top-five candidates, versus 16.66% for random filtering and 31.75% for undistilled teacher skipping. After refinement, it achieves observed ATE 3.4839, compared with 3.5415 for full dense evaluation, at 4.49× lower planning time.
| Dataset | Random top-5 Recall@20 ↑ | Teacher-skip top-5 Recall@20 ↑ | Student top-5 Recall@20 ↑ |
|---|---|---|---|
| RECON | 16.35% | 61.00% | 77.00% |
| HuRoN/SACSoN | 16.33% | 9.00% | 58.00% |
| TartanDrive | 16.79% | 39.00% | 48.00% |
| SCAND | 17.17% | 18.00% | 50.00% |
| Average | 16.66% | 31.75% | 58.25% |
| Method | Average ATE ↓ | Average RPE ↓ | Seconds/query ↓ |
|---|---|---|---|
| Dense teacher | 3.5415 | 0.7908 | 3606.7 |
| Random + teacher | 3.7822 | 0.8439 | 654.6 |
| Teacher skip + teacher | 3.6469 | 0.8073 | 804.0 |
| Student + teacher | 3.4839 | 0.7832 | 803.4 |
These results support the intended division of labor: the student supplies inexpensive, useful candidate rankings, while the teacher provides detailed evaluation of the shortlist. The benefit is demonstrated through selection and planning outcomes, rather than inferred from reconstruction quality alone.
Answers for Q1&W4
We thank the reviewer for the thoughtful comments on our method and evaluation.
Q1&W4: Have the authors evaluated multiple random seeds or measured variance statistics?
A1: To assess whether the reported gains persist under planning randomness, we evaluated both SkipWorld and the retrained NWM baseline over three runs, each covering the same 400 queries (100 per dataset). The test queries, model checkpoints, and planning hyperparameters are held fixed across runs, while the evaluation seed varies. The comparison retains the same candidate count, prediction horizon, and dense repetition budget for both methods; SkipWorld uses its distilled student and AutoRoll teacher for screening and refinement. These experiments measure evaluation repeatability with fixed trained models, rather than variation across independently trained checkpoints.
The first two tables show ATE / RPE for each run, followed by an aggregate comparison.
SkipWorld: three evaluation runs
NWM: three evaluation runs
Overall comparison: mean ± standard deviation across three runs
SkipWorld achieves lower ATE and RPE on every dataset in all three runs. Thus, the improvement is consistent across both datasets and evaluation seeds, rather than being driven by a single favorable run or dataset. The four-dataset mean ATE decreases from 3.3331 to 3.1965 (4.10%), and mean RPE from 0.7338 to 0.7137 (2.73%). Together, these results support the repeatability of the observed planning advantage under changes in evaluation randomness.