- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AnchorWorld, an anchor-guided latent world model for long-horizon Vision-and-Language Navigation. The method introduces a Prior Anchor Extractor (PAE), which predicts sparse future anchors from navigation history and language instructions under shortest-path supervision, and an Anchor-based Latent Planner (ALP), which generates multi-step latent predictions for candidate navigation actions conditioned on these anchors. The predicted future representations are incorporated into a DUET-based navigation policy through attention and gated feature fusion. Experiments on R2R and R4R demonstrate competitive navigation performance. Additional experiments on REVERIE, an alternative backbone, and computational cost are provided in the appendix.
Critical strengths
-
The proposed framework addresses a relevant challenge in long-horizon VLN by combining sparse global trajectory priors with candidate-conditioned latent rollouts. This design provides a potentially efficient alternative to computationally expensive explicit trajectory search.
-
AnchorWorld achieves competitive performance on R2R and R4R, with improvements particularly evident in longer navigation trajectories. The component ablations indicate that PAE and ALP provide complementary benefits.
Critical weaknesses
-
Insufficiently specified candidate-conditioned future supervision and probabilistic world-model objective. A central claim is that ALP generates distinct future latent trajectories for different candidate actions. However, Appendix A.3 defines the future-state supervision using shortest-path states without clearly specifying candidate-dependent targets. It is therefore unclear how counterfactual candidate actions are supervised, particularly when their consequences are not observed in the expert trajectory. This ambiguity raises concerns about whether the learned model truly captures action-dependent future dynamics. Moreover, although ALP is described as a CVAE, the reported objective contains only an MSE prediction loss, without an explicit variational posterior or KL regularization. The probabilistic formulation therefore requires further clarification.
-
Potential mismatch between shortest-path supervision and long-horizon instruction following. PAE is supervised using future anchors uniformly sampled from the shortest path between the current position and the goal. However, long-horizon VLN instructions, particularly in R4R, may specify routes that differ from the shortest path. Consequently, the proposed anchors may favor goal-directed shortcuts rather than faithful execution of intermediate language instructions. The current R4R evaluation emphasizes SR, SPL, and NE but does not directly measure whether the agent follows the instructed route. This limits the evidence supporting the claimed improvements in long-horizon instruction-grounded planning.
Questions for authors
-
How are the ground-truth future states constructed for different candidate actions in ALP? Does each candidate receive a distinct target trajectory? If not, how does the model avoid predicting similar futures for different actions? Quantitative analysis of action-conditioned prediction accuracy would materially strengthen this claim.
-
Could the authors clarify the conditional prior, posterior distribution, and complete optimization objective of ALP? Why is no KL divergence term included in the reported loss? How is meaningful stochastic variation maintained in the learned latent predictions?
-
Given that R4R reference trajectories are not necessarily shortest paths, could the authors compare shortest-path anchor supervision with instruction-trajectory-based supervision? Could they also report route-fidelity metrics such as nDTW, SDTW, or CLS to determine whether improvements in goal-reaching efficiency preserve instruction adherence?
7 comments
Response to Reviewer rxMs (Part 2/2)
3. Shortest-path supervision and instruction adherence
PAE learns anchor targets aligned with the supervision it receives. We compare the two supervision strategies below:
(i) With supervision from uniformly sampled points along the shortest path, our method achieves better goal-reaching performance and navigation efficiency (NE, SR, and SPL in B1). These results suggest that PAE can filter out textual information that is redundant for goal reaching when fusing instruction and visual features (Fig. 2).
(ii) With instruction-aligned trajectory supervision, our method achieves higher instruction fidelity, as measured by CLS and nDTW (B2). This supervision explicitly encourages PAE to generate anchors that follow the instructed route.
| Idx | Setting | NE ↓ | SR ↑ | SPL ↑ | CLS ↑ | nDTW ↑ | SDTW ↑ |
|---|---|---|---|---|---|---|---|
| B1 | Ours (shortest-path supervision) | 5.53 | 51.35 | 47.62 | 46.43 | 40.62 | 27.43 |
| B2 | Ours (Instruction-aligned trajectory supervision) | 5.65 | 46.95 | 40.37 | 47.90 | 43.74 | 26.86 |
Together, these results suggest that PAE can prioritize features relevant to the chosen learning objective, allowing the same framework to accommodate different route requirements.
Response to Question1
Thanks for your thoughtful comment. our responses are mainly three-fold:
(i) During latent rollout, our future world model predicts based on the position at each step and key anchor conditions along the shortest path from that position to the navigation goal, rather than rolling out based on candidates’ future actions. The advantage is that our learning objective does not require multiple counterfactual expert trajectories (the dataset contains only one expert trajectory). It requires only the shortest path from each navigation step to the goal. Accordingly, PAE is first supervised by key points uniformly sampled along the shortest path. The anchors predicted by PAE are then provided to ALP for prediction, and ALP's outputs are aligned with the shortest path.
(ii) Regarding the discriminability of counterfactual predictions raised by the reviewer, predictions of different futures in our method are conditioned on the same anchors along the future shortest path. Our predictions are therefore more deterministic, focusing on shortest-path navigation. We introduce Gaussian sampling [1, 2] to allow the predictions to reflect real-world uncertainty. Nevertheless, discriminability is a valuable research question that we plan to investigate further. Thank you again for this insightful question.
[1] DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents. CVPR, 2017.
[2] Hierarchical Latent Structure for Multi-modal Vehicle Trajectory Forecasting. ECCV, 2022.
(iii) There is currently no unified standard for evaluating counterfactual reasoning in world-model research. This is mainly because expert ground-truth trajectories for different future actions from the same initial state are unknown in the test set and difficult to collect. We therefore use an alternative evaluation to quantify our method. Specifically, in our setting, “future-state prediction” could refer either to PAE’s anchor predictions or to ALP’s predicted latent states. To address both interpretations, we report two prediction-accuracy analyses on R2R Val Unseen. We measure the cosine similarity and mean squared error (MSE) between the states predicted by PAE or ALP and their ground-truth targets. The table reports the mean ± standard deviation for each step and across all three steps.
| Module | Step | Cosine Similarity ↑ | MSE ↓ |
|---|---|---|---|
| PAE | 1 | 0.805 ± 0.061 | 0.182 ± 0.054 |
| PAE | 2 | 0.761 ± 0.076 | 0.221 ± 0.069 |
| PAE | 3 | 0.730 ± 0.089 | 0.247 ± 0.081 |
| PAE | Mean | 0.765 ± 0.083 | 0.216 ± 0.074 |
| ALP | 1 | 0.742 ± 0.083 | 0.241 ± 0.072 |
| ALP | 2 | 0.697 ± 0.090 | 0.281 ± 0.081 |
| ALP | 3 | 0.673 ± 0.093 | 0.303 ± 0.085 |
| ALP | Mean | 0.710 ± 0.093 | 0.270 ± 0.083 |
Response Reminder
Dear Reviewer,
As we approach the end of the discussion period, we invite you to review the authors’ response to your comments. Please update us on whether your concerns have been sufficiently addressed and whether you wish to pose further questions to the authors.
We greatly appreciate your contribution.
Response to Reviewer rxMs (Part 1/2)
Response to Weakness1 and Question2
1. Clarifying Candidate-Specific Future Supervision
Thank you for the thoughtful comment. During latent rollout, our world model uses the state at each step and key anchors along the shortest path from that state to the navigation goal as conditions, rather than rolling out a sequence of future candidate actions. This design does not require multiple counterfactual expert trajectories; the dataset provides only one expert trajectory per instruction. Instead, supervision requires only shortest paths from navigation states to the goal. Specifically, PAE is supervised by key points sampled approximately uniformly along a shortest path. Its predicted anchors then guide ALP’s rollout, whose outputs are aligned with shortest-path states.
We have elaborated on the candidate-specific supervision in Appendix A.3.1.
2. Clarifying the loss of ALP
Thank you for your careful check. In describing ALP, we focused on the future-state prediction objective without fully presenting the associated CVAE training formulation. Consequently, the KL regularization term was not explicitly shown in the reported loss. We have expanded Appendix A.3.1 to describe ALP’s training procedure in detail.
For each candidate j, we construct a shortest-path continuation from its resulting viewpoint to the goal and encode the consecutive viewpoint states as targets yt,jk⋆. Targets for unselected candidates are derived from the training graph; they provide counterfactual supervision rather than representing observed executions. The candidate embedding ct,j, previous predicted state, and shared PAE anchor define the rollout context ut,jk. ALP specifies a conditional prior pψ(zt,jk∣ut,jk) and a training-time recognition distribution qφ(zt,jk∣ut,jk,yt,jk⋆). Its complete objective combines state reconstruction with a posterior-to-prior KL term, LALP=Lrec+βLKL; the original MSE-only description omitted the latter. Training uses reparameterized samples from the recognition distribution for reconstruction, whereas inference and policy scoring use separate prior-sampled rollouts without access to target states. The learned candidate-conditioned Gaussian permits stochastic predictions, and the KL term aligns the training-time distribution with the inference-time prior.
Response to Weakness2 and Question3
1. Clarifying our objective and evaluation focus.
Our work does not claim superior instruction-following capability; it targets improved goal reaching and navigation efficiency in long-horizon navigation, as stated in Lines 107 and 199.
Our reported metrics emphasize goal-reaching performance (NE/SR) and navigation efficiency (SPL), whereas nDTW/SDTW/CLS assess instruction fidelity through adherence to reference routes. These metrics evaluate different aspects of navigation that can involve a trade-off. In R4R, reference routes are constructed by concatenating adjacent trajectories and are not always the shortest paths. Consequently, closer adherence to the reference route does not necessarily yield the most efficient path to the destination.
2. Additional evaluation on R4R.
As suggested, we additionally report CLS, nDTW, and SDTW to assess instruction fidelity alongside goal-reaching performance and navigation efficiency on R4R Validation Unseen. Here, we use α to tradeoff of instruction fidelity and navigation efficiency: α=0 disables the shortest-path enhancement in our approach; α=0.5 partially incorporates our method, and α=1 applies it fully.
Overall, our method substantially improves navigation efficiency, while gains in route fidelity are modest. One possible explanation is that R4R’s longer instructions may describe detours rather than the most efficient shortest paths.