acceptodds
Under review as a conference paper at ICLR 2027

EgoExo-VLN: Egocentric-Exocentric Vision-and-Language Navigation in Dynamic Environments

Abstract

Existing Vision-Language Navigation (VLN) methods primarily rely on egocentric observations, limiting their ability to perceive important environmental changes beyond the robot's current view. To address this limitation, we propose a novel VLN task, named Egocentric-Exocentric Vision-Language Navigation (EgoExo-VLN), which enables agents to follow language instructions in dynamic indoor environments by combining egocentric observations with information from fixed exocentric cameras. EgoExo-VLN goes beyond simply providing additional views by requiring agents to identify task-relevant information across viewpoints, determine whether observed evidence remains valid over time, and decide whether it should influence the current VLN decision. To systematically evaluate these capabilities in the EgoExo-VLN setting, we construct the first VLN benchmark that explicitly incorporates exocentric cameras, consisting of 8,096 episodes across 90 indoor scenes with scene-disjoint training, validation, and test splits. The benchmark focuses on dynamic target relocation and introduces controlled absent, conflict, and non-conflict pedestrian conditions, while also including static-scene tasks to cover common VLN scenarios. Building on this benchmark, we propose a VLN framework that selects task-relevant camera information, updates target representations over time to revise VLN goals, and reasons about pedestrian conflicts to determine when the agent should wait or continue moving. EgoExo-VLN establishes a new benchmark for studying VLN with egocentric observations and exocentric camera information in dynamic environments.

Primary area:
datasets and benchmarks
Keywords:
Vision-and-Language Navigation, embodied AI, multimodal learning, multi-view perception, robot navigation, safe navigation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dCncOpenReview ↗
Rating
1(−1)
Confidence
4(+2)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EgoExo-VLN studies language-guided navigation with onboard observations and fixed external cameras in environments with target relocation and moving pedestrians. The benchmark contains 8,096 episodes across 90 scenes with scene-disjoint splits. The proposed system combines camera relevance, temporal target memory, geometric navigation, and predictive waiting, achieving 55.31% test SR and 53.33% SafeSR on pedestrian-present episodes.

Critical strengths
  1. The task captures a meaningful limitation of onboard sensing: relevant changes can occur outside the robot’s view. Matched absent, conflict, and non-conflict pedestrian conditions provide a useful basis for studying when external evidence should change an action.
  2. The ablations provide evidence beyond simply adding cameras. Learned camera relevance raises SR from 41.93% to 54.60%, while target-evidence memory improves SR from 52.00% to 55.31% over current-only observations. These results support selective fusion and temporal memory within the proposed system.
Critical weaknesses
  1. The comparisons failed to isolate the source of the gains. The full system utilizes external depth, calibrated camera geometry, robot pose, a NavMesh-based follower, and a separate arrival predictor, but Section 5.2 does not specify which inputs and controller components each baseline received. Furthermore, removing external observations also disabled the camera-guided geometric execution in Appendix D, therefore the increase in SR cannot be entirely attributed to visual fusion.
  2. Appendix B evaluates every STOP against the final location B, including stops before or during transport, without requiring the target to have arrived. Therefore, the episode may be considered successful even if the target is still elsewhere. While this aligns with the final location objective set, it makes it unclear whether the success rate reflects timely target tracking and navigation target updates. The paper does not report the frequency of such early successes, nor does it analyze whether they affect the main conclusions.
Questions for authors

See Weakness

18 comments
AuthorsOpenReview ↗

Response to Reviewer dCnc

W1: The comparisons failed to isolate the source of the gains. The full system utilizes external depth, calibrated camera geometry, robot pose, a NavMesh-based follower, and a separate arrival predictor, but Section 5.2 does not specify which inputs and controller components each baseline received. Furthermore, removing external observations also disabled the camera-guided geometric execution in Appendix D, therefore the increase in SR cannot be entirely attributed to visual fusion.

We thank the reviewer for this suggestion. To make the contributions of external visual information and auxiliary execution mechanisms clearer, we have added the following hierarchical ablation study and will clarify the input and controller configurations of each baseline in Section 5.2.

All seven baselines retain their original egocentric observations and core navigation mechanisms, without incorporating the exocentric RGB or other external data used in our method. We will further clarify the specific mechanisms used by each method in Section 5.2.

The ablation results are presented below:

Camera MetadataExo RGBNavMesh FollowerArrival PredictorNE ↓ (m)SR ↑SPL ↑Reloc. SR ↑SafeSR ↑CF-SPL ↑
––––6.68817.1516.0616.9418.3315.12
✓–––6.79317.2414.0616.5818.7514.98
✓✓––4.13237.6836.7638.7834.3932.20
✓✓✓–3.44749.2047.1048.6051.2847.90
✓✓✓✓3.30955.3154.4856.7453.3350.20

The additional comparisons demonstrate that external RGB still provides clear benefits when the additional geometric execution and Arrival Predictor are disabled. Adding only Camera Metadata changes SR from 17.15% to 17.24%, without showing a consistent performance improvement. With metadata available and both geometric execution and the Arrival Predictor disabled, further introducing external RGB increases SR from 17.24% to 37.68% (+20.44 percentage points). Reloc. SR and SafeSR also increase from 16.58% and 18.75% to 38.78% and 34.39%, respectively. This comparison supports the navigation value of external visual evidence independently of the aforementioned auxiliary execution and termination modules.

On this basis, enabling the NavMesh Follower increases SR from 37.68% to 49.20% (+11.52 percentage points). Further introducing the Arrival Predictor increases SR to 55.31% (+6.11 percentage points). These comparisons also quantify the contributions of geometric execution and arrival prediction to the full system. Regarding depth information, Table 4 of the paper shows that, with egocentric depth retained, SR/SPL are 52.27%/51.46% without exocentric depth, and increase to 55.31%/54.48% when exocentric depth is used. Based on these results, we will revise the relevant descriptions.

W2: Appendix B evaluates every STOP against the final location B, including stops before or during transport, without requiring the target to have arrived. Therefore, the episode may be considered successful even if the target is still elsewhere. While this aligns with the final location objective set, it makes it unclear whether the success rate reflects timely target tracking and navigation target updates. The paper does not report the frequency of such early successes, nor does it analyze whether they affect the main conclusions.

We thank the reviewer for the suggestion to examine the timing of STOP decisions. To verify the evaluation results, we individually examined all target-relocation trajectories classified as successful under the original protocol in the Test set, comparing the time when the robot executed STOP with the times when the target started moving and completed its relocation. Among all successful target-relocation trajectories, there were 0 premature successes before the target started moving and 0 during relocation, resulting in an overall occurrence rate of 0%. We further introduced a temporal constraint requiring that the target had completed its relocation and reached location B, and re-evaluated the same trajectories offline. The overall SR and Reloc. SR of our method remained 55.31% and 56.74%, respectively, unchanged from the original results. This check confirms that premature STOP decisions did not inflate the reported success rate of our method.

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 1 day, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

I thank for the authors' detailed rebuttal. However, I'm still concerned about the baseline comparison and the usage of other camera views. I decide to maintain my score.

Reviewer_iyEjOpenReview ↗
Rating
4(+2)
Confidence
4(+1)
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper proposes EgoExo-VLN setting, builds a novel large-scale benchmark containing exocentric cameras, and proposes a novel VLN framework that actively chooses task-relevant camera perspective when navigating in a dynamic environment, leading to success and safety performance gains.

Critical strengths
  1. The motivation is clear and well illustrated. Most existing VLN benchmarks assume static environments. Introducing dynamic changes into the environment can break some basic assumptions of conventional VLN.

  2. Introducing WAIT and learning when to wait or stop under the EgoExo-VLN framework is important for improving the safety of VLN models.

  3. Built a large dataset and benchmark for EgoExo-VLN setting.

Critical weaknesses

Weaknesses:

  • It is generally hard to isolate the effectiveness of exocentric information. Is the module processing exocentric images actually contributing to the improvement, or is the performance gain mainly due to the additional prior knowledge and geometric information provided to the model? For example, Appendix D mentions that removing exocentric observations also disables image-based geometric execution. I think more controlled ablations are needed to make this distinction clear.
  • The real-world experiments are not sufficient, with the method evaluated in only one scene. Since the method is fully trained in simulation and adopts a modular approach with relatively few trainable parameters, real-world generalizability should be a major concern. I would like to see quantitative results across more real-world scenes.

Suggestions:

  • It would be useful to report the number of parameters in each module, especially the trainable modules.

writing:

  1. For Tables 3 and 4, consider adding vertical lines to separate the settings from the metric results.
  2. The terms "target-relocation" and "pedestrian-present" episodes should be defined before they appear in Section 3.2.
  3. Is CF-SPL introduced in this work or adapted from prior work? It seems related to existing safety-aware navigation metrics, such as the Social Navigation Score (SNS) [1].

References:

  • [1] Deitke et al, Retrospectives on the embodied ai workshop. arXiv preprint arXiv:2210.06849
Questions for authors
  1. Why is a relevance gate used to control information from external views? An external view may be completely irrelevant to the current egocentric observation or route, yet still capture important information, such as the relocation of the target object. How does the relevance gate handle such cases?

  2. Regarding the benchmark setting, doesn't target relocation sometimes make the original language instruction outdated? The authors should clarify when this happens, whether it could be misleading to ego-only methods, and how much this affects their performance.

11 comments
AuthorsOpenReview ↗

Response to Reviewer iyEj - 1 / 3

W1: It is generally hard to isolate the effectiveness of exocentric information. Is the module processing exocentric images actually contributing to the improvement, or is the performance gain mainly due to the additional prior knowledge and geometric information provided to the model? For example, Appendix D mentions that removing exocentric observations also disables image-based geometric execution. I think more controlled ablations are needed to make this distinction clear.

We thank the reviewer for this constructive suggestion. To make the contributions of exocentric visual information and geometric execution clearer, we have added controlled component ablations, as shown below.

The additional comparisons demonstrate that exocentric RGB still provides clear benefits when the additional geometric execution and Arrival Predictor are disabled. Adding only Camera Metadata changes SR from 17.15% to 17.24%, without showing a noticeable performance improvement. With metadata available and both geometric execution and the Arrival Predictor disabled, further introducing exocentric RGB increases SR from 17.24% to 37.68% (+20.44 percentage points). Reloc. SR and SafeSR also increase from 16.58% and 18.75% to 38.78% and 34.39%, respectively. On this basis, enabling the NavMesh Follower increases SR from 37.68% to 49.20% (+11.52 ). Further introducing the Arrival Predictor increases SR to 55.31% (+6.11 ). These results demonstrate that external visual information provides independent performance gains, while geometric execution and arrival prediction offer further improvements.

Camera MetadataExo RGBNavMesh FollowerArrival PredictorNE ↓ (m)SR ↑SPL ↑Reloc. SR ↑SafeSR ↑CF-SPL ↑
––––6.68817.1516.0616.9418.3315.12
✓–––6.79317.2414.0616.5818.7514.98
✓✓––4.13237.6836.7638.7834.3932.20
✓✓✓–3.44749.2047.1048.6051.2847.90
✓✓✓✓3.30955.3154.4856.7453.3350.20

W2: The real-world experiments are not sufficient, with the method evaluated in only one scene. Since the method is fully trained in simulation and adopts a modular approach with relatively few trainable parameters, real-world generalizability should be a major concern. I would like to see quantitative results across more real-world scenes.

We thank the reviewer for the suggestion. Navigation may have a relatively smaller sim-to-real gap than manipulation-intensive embodied tasks because it does not require complex physical interactions; moreover, our simulation uses HM3D scenes reconstructed from real-world scans. Accordingly, our policies were trained entirely in simulation, without real-world training data or fine-tuning. Following the reviewer's suggestion, we have added quantitative experiments in three real-world scenarios: Office, Corridor, and Public Area. Each scenario includes 20 navigation tasks, with Ego-only and EgoExo-VLN evaluated under the same conditions, resulting in a total of 120 trials.

As shown in the table below, EgoExo-VLN achieves better performance across all three scenarios. Overall SR, Reloc. SR, and SafeSR increase from 45.00%, 20.00%, and 26.67% to 66.67%, 53.33%, and 66.67%, respectively. These results provide preliminary evidence of the generalization capability of our method across different real-world dynamic environments. We will include the corresponding experimental settings and result analysis in the revised manuscript.

Real-world SceneMethodNumber of TrialsSR ↑Reloc. SR ↑SafeSR ↑
OfficeEgo-only2050.00%20.00%40.00%
OfficeEgoExo-VLN2070.00%60.00%60.00%
CorridorEgo-only2045.00%20.00%20.00%
CorridorEgoExo-VLN2070.00%60.00%80.00%
Public AreaEgo-only2040.00%20.00%20.00%
Public AreaEgoExo-VLN2060.00%40.00%60.00%
OverallEgo-only6045.00%20.00%26.67%
OverallEgoExo-VLN6066.67%53.33%66.67%
AuthorsOpenReview ↗

Response to Reviewer iyEj - 2 / 3

S1: It would be useful to report the number of parameters in each module, especially the trainable modules.

We thank the reviewer for this suggestion. We have provided the parameter counts and training status of each module, as shown in the table below. The model contains approximately 236.73M parameters in total. During exocentric viewpoint adaptation, the pretrained feature encoders and navigation backbone remain frozen, while the newly introduced exocentric-related modules are primarily trained. We will explicitly report the total and trainable parameter counts of each module in the revised manuscript.

ModuleParameters
RoBERTa Text Encoder124.06M
CLIP ViT-B/16 Visual Encoder86.19M
DD-PPO Depth Encoder7.07M
Ego-Language Transformer12.61M
Camera Metadata Encoder0.27M
Camera Spatial Attention1.05M
Cross-view Attention1.05M
Relevance Gate0.79M
Camera Relevance and Null Scoring Head0.099M
Camera-specific Temporal Memory1.38M
Navigation GRU1.63M
Arrival Predictor0.002M

S2: writing:

  1. For Tables 3 and 4, consider adding vertical lines to separate the settings from the metric results.

  2. The terms "target-relocation" and "pedestrian-present" episodes should be defined before they appear in Section 3.2.

  3. Is CF-SPL introduced in this work or adapted from prior work? It seems related to existing safety-aware navigation metrics, such as the Social Navigation Score (SNS) [1].

  1. We thank the reviewer for this suggestion. We will add vertical lines to Tables 3 and 4 in the revised manuscript to clearly distinguish experimental settings from evaluation metrics.

  2. We thank the reviewer for this reminder. We will explicitly define Target-relocation episodes and Pedestrian-present episodes when these terms first appear in Section 3.2.

  3. CF-SPL was not first introduced in this work but is an existing collision-free path efficiency metric based on SPL. In this work, we apply it to dynamic navigation scenarios involving pedestrians. We will add relevant references for CF-SPL, SPL, and SNS in the revised manuscript and clarify their definitions and differences.

Q1: Why is a relevance gate used to control information from external views? An external view may be completely irrelevant to the current egocentric observation or route, yet still capture important information, such as the relocation of the target object. How does the relevance gate handle such cases?

  1. Why is a relevance gate used to control information from external views?

Different exocentric cameras provide cues of varying value for the current task. The gate allows the model to prioritize views relevant to the instruction target, reducing interference from irrelevant cues during navigation.

  1. How does the relevance gate handle such cases?

The model uses the instruction to query each camera view and combines recent observations to assign scores to each viewpoint and the Null option (no reliable external evidence). A camera capturing the target's new location can receive a higher weight based on target-matching evidence, helping the robot adjust its heading. When evidence from all viewpoints is insufficient, the model can assign a higher weight to Null.

AuthorsOpenReview ↗

Response to Reviewer iyEj - 3 / 3

Q2: Regarding the benchmark setting, doesn't target relocation sometimes make the original language instruction outdated? The authors should clarify when this happens, whether it could be misleading to ego-only methods, and how much this affects their performance.

We thank the reviewer for this question. In dynamic target-relocation scenarios, some spatial information in the original language instructions can become outdated as the environment changes. In EgoExo-VLN, the instructions are generated before the task begins, primarily describing the navigation route and target category, and remain unchanged throughout task execution.

Specifically, before the target moves, the route information in the instruction is consistent with the target's original location. When the target object is relocated from its initial position A to a new position B, the route and endpoint cues associated with the original target location may no longer be accurate. However, the target category and object identity specified in the instruction remain unchanged. Therefore, the navigation agent needs to update its estimate of the target location based on new observations obtained during execution and adjust its navigation decisions accordingly. This is also one of the core dynamic navigation challenges that EgoExo-VLN aims to investigate.

Regarding the performance impact, Table 2 of the original paper reports the results on target-relocation tasks in the Test set. The Reloc. SR values of the conventional Ego-only methods CMA, ETPNav, and BEVBert are 10.21%, 40.85%, and 48.65%, respectively, while our EgoExo-VLN method achieves 56.74%, outperforming the best original Ego-only baseline by 8.09 percentage points.

Furthermore, in the newly added controlled ablation study, with Camera Metadata available and both camera-guided geometric execution and the additional Arrival Predictor disabled, introducing exocentric RGB increases Reloc. SR from 16.58% to 38.78% (+22.20 ). This result further demonstrates that external visual information can effectively help navigation models handle target-relocation tasks.

We will further clarify in the revised manuscript which spatial information in the original instructions may become invalid due to target movement and distinguish the performance under stationary-target and target-relocation conditions. The above results reflect the performance of different methods on relocation tasks and the value of external visual information.

Reviewer_MagkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, a VLN benchmark, EgoExo-VLN, is introduced that includes both egocentric and exocentric observations, built around a new VLN task that combines egocentric and exocentric navigation. The benchmark specifically focuses on tasks that are challenging without exocentric views, such as occluded dynamic obstacles, changes in the scene that are not observed, and so on. Additionally, a method for leveraging exocentric camera views is introduced and compared against state-of-the-art methods in vision-language navigation.

Critical strengths

The work proposes a benchmark, provides a dataset, and introduces a method that can leverage both egocentric and exocentric viewpoints, therefore allowing for validation of their full system and giving confidence to readers about the downstream use of the work.

  • They provide dense annotations for the data, which allows future works more freedom in using the benchmark and dataset
  • The ablations are clear and demonstrate the motivation for exocentric views, depth information, and a dedicated system for waiting behaviors in the presence of conflicts
Critical weaknesses

This work does not seem to acknowledge the main reason why exocentric cameras are often used for visual navigation tasks, which is that they require prior access and installation in a scene, which introduces additional infrastructural challenges for deploying a robot. I would ask the authors to include a more thorough discussion of the limitations of exocentric viewpoints and why they are generally not common. However, I also recognize that this work could be valuable for settings where this infrastructure is possible to introduce.

  • In the baselines provided, it is unclear whether they have also been modified to be able to leverage exocentric observations - the inclusion of these observations fundamentally changes the observation spaces between the methods. This makes it challenging to argue whether these baselines are fair comparisons for the proposed model
    • Can you be clear about what information these policies have access to? Which of these methods have history, and which only have the current egocentric view?
    • The authors state “we adapt these methods to our benchmark while preserving their core navigation mechanisms” - can you clarify if this means these methods were trained on the data from the EgoExo VLN benchmark?
    • Working under the assumption that others may use these numbers in the future when using the benchmark, this information should be extremely clear
    • A possible extension would be to adapt variations of these methods to accept exocentric views, although I recognize this is an additional challenge
  • The EgoExo Model description has some missing details:
    • Can you further describe what the ego-language transformer is? I assume this is trained from scratch?
    • How does the relevance gate work?
    • How does the camera-specific temporal memory work? Is this something that is learned, or how is new evidence incorporated?
    • This system seems somewhat complicated, and no ablations on the architecture itself are performed; it would be helpful to provide ablations or reference prior work to justify this system
Questions for authors
  • Is additional data used to train the real-world policy? Or is the policy transferred zero-shot from simulated data training?
  • Given the above weaknesses, can you clarify how the baselines were adapted to the benchmark?
12 comments
AuthorsOpenReview ↗

Response to Reviewer Magk - 1 / 3

W1: This work does not seem to acknowledge the main reason why exocentric cameras are often used for visual navigation tasks, which is that they require prior access and installation in a scene, which introduces additional infrastructural challenges for deploying a robot. I would ask the authors to include a more thorough discussion of the limitations of exocentric viewpoints and why they are generally not common. However, I also recognize that this work could be valuable for settings where this infrastructure is possible to introduce.

Previous embodied navigation research has primarily focused on relatively static scenarios, where onboard observations have generally been sufficient and external cameras were not typically needed. As embodied agents become more capable, navigation in dynamic real-world environments is becoming increasingly important to study. This motivates EgoExo-VLN, which investigates how exocentric observations can support navigation decisions when relevant changes occur outside the robot's field of view. We will clarify this motivation and the scope of our setting in the revised manuscript.

We would like to clarify that EgoExo-VLN is not intended to replace traditional egocentric VLN operating in arbitrary unknown environments, but rather to serve as a complement to existing navigation tasks. Instead, we investigate an infrastructure-assisted navigation setting. In such settings, fixed cameras are already available, such as in restaurants, hospitals, and hair salons. In these dynamic environments, where people and objects frequently move, exocentric cameras can help robots perceive environmental changes in advance, reduce navigation uncertainty caused by blind spots, and thereby support more timely and safer navigation decisions.

W2: In the baselines provided, it is unclear whether they have also been modified to be able to leverage exocentric observations - the inclusion of these observations fundamentally changes the observation spaces between the methods. This makes it challenging to argue whether these baselines are fair comparisons for the proposed model

  • Can you be clear about what information these policies have access to? Which of these methods have history, and which only have the current egocentric view?

  • The authors state “we adapt these methods to our benchmark while preserving their core navigation mechanisms” - can you clarify if this means these methods were trained on the data from the EgoExo VLN benchmark?

  • Working under the assumption that others may use these numbers in the future when using the benchmark, this information should be extremely clear

  • A possible extension would be to adapt variations of these methods to accept exocentric views, although I recognize this is an additional challenge

  1. Regarding the observations and historical information accessible to each baseline.*All seven conventional VLN baselines in the original manuscript use only egocentric observations, receive no exocentric camera input, and retain their respective original history-modeling mechanisms. We will clarify their observation modalities and history settings in the revised manuscript.

  2. All seven conventional VLN baselines were trained on the EgoExo-VLN training set, rather than directly transferring the original model weights in a zero-shot manner. All methods were evaluated on the same EgoExo-VLN validation and test sets.

  3. We thank the reviewer for the suggestion. We will provide complete baseline configurations and evaluation details in the revised manuscript to ensure the transparency and reproducibility of the results.

  4. Regarding the extension of baselines to exocentric viewpoints. We thank the reviewer for this suggestion. We have additionally adapted HPN and Recurrent to receive exocentric observations, and the before-and-after comparisons are presented below. Both methods achieve higher SR and SPL with exocentric input, further demonstrating the value of third-person visual evidence. We will include these settings and results in the revised manuscript.

MethodNE ↓ (m)SR ↑SPL ↑Reloc. SR ↑SafeSR ↑CF-SPL ↑
HPN (Ego-only)7.9398.368.098.238.177.88
HPN + Exo7.75918.1417.1717.8616.1715.30
Recurrent (Ego-only)7.52110.068.4211.4910.008.36
Recurrent + Exo3.54046.2343.7144.6445.3342.08
AuthorsOpenReview ↗

Response to Reviewer Magk - 2 / 3

W3: The EgoExo Model description has some missing details:

  • Can you further describe what the ego-language transformer is? I assume this is trained from scratch?

  • How does the relevance gate work?

  • How does the camera-specific temporal memory work? Is this something that is learned, or how is new evidence incorporated?

  • This system seems somewhat complicated, and no ablations on the architecture itself are performed; it would be helpful to provide ablations or reference prior work to justify this system

(1) Ego–Language Transformer.

The Ego–Language Transformer is a multimodal fusion module consisting of 4 Transformer Encoder layers, 8 attention heads, and a 512-dimensional hidden space. It fuses language instructions with the robot's egocentric RGB-D observation features to generate a navigation state representation.

This module is trained from scratch and jointly trained with the navigation policy during the supervised learning stage.

(2) Relevance Gate.

The model first uses Softmax to assign relevance weights to each available camera and the Null option. When the Null weight is high, the contribution of external evidence is reduced. Subsequently, the main navigation branch extracts external context through cross-view attention, scales it using a learnable Sigmoid gate, and then performs residual fusion with the egocentric–language representation.

(3) Camera-specific Temporal Memory.

The model employs a trainable GRU with shared parameters across cameras, while each camera maintains its own independent hidden state. The current evidence combines instruction-conditioned image features, robot state, relative camera geometry, and predicted waypoint direction:

ht,j=GRUcam(et,j,ht−1,j)h_{t,j}=GRU_{cam}(e_{t,j},h_{t-1,j})

e~t,j=φ([et,j;ht,j])ẽ_{t,j}=φ([e_{t,j};h_{t,j}])

The updated evidence is used for camera relevance estimation, target localization, and waypoint refinement.

(4) We thank the reviewer for this suggestion. We have added component ablation experiments to separately evaluate the contributions of external RGB, camera-guided geometric execution, and the Arrival Predictor. The results are presented in the following table.

Camera MetadataExo RGBNavMesh FollowerArrival PredictorNE (m)SRSPLReloc. SRSafeSRCF-SPL
––––6.68817.1516.0616.9418.3315.12
✓–––6.79317.2414.0616.5818.7514.98
✓✓––4.13237.6836.7638.7834.3932.20
✓✓✓–3.44749.2047.1048.6051.2847.90
✓✓✓✓3.30955.3154.4856.7453.3350.20

The additional comparisons demonstrate that external RGB still provides clear benefits when the additional geometric execution and Arrival Predictor are disabled. Adding only Camera Metadata changes SR from 17.15% to 17.24%, without showing a consistent performance improvement. With metadata available and both geometric execution and the Arrival Predictor disabled, further introducing external RGB increases SR from 17.24% to 37.68% (+20.44 percentage points). Reloc. SR and SafeSR also increase from 16.58% and 18.75% to 38.78% and 34.39%, respectively. This comparison supports the navigation value of external visual evidence independently of the aforementioned auxiliary execution and termination modules.

On this basis, enabling the NavMesh Follower increases SR from 37.68% to 49.20% (+11.52 percentage points). Further introducing the Arrival Predictor increases SR to 55.31% (+6.11 percentage points). These comparisons also quantify the contributions of geometric execution and arrival prediction to the full system. In addition, the original Table 3 and Table 9 in Appendix D validate the effectiveness of the camera relevance mechanism and temporal memory, respectively.

Regarding the architectural design, gated fusion, cross-view attention, and navigation history modeling are supported by prior studies (Arevalo et al., 2017; Li et al., 2022; Hong et al., 2021). We will add these references in the revised manuscript and further explain the design rationale of each module.

AuthorsOpenReview ↗

Response to Reviewer Magk - 3 / 3

Q1: Is additional data used to train the real-world policy? Or is the policy transferred zero-shot from simulated data training?

We thank the reviewer for raising this question. Our navigation policy was trained entirely in simulation and then directly transferred to a real robot, without using any additional real-world data for policy training or fine-tuning. We have also added quantitative real-world experiments, as shown in the following table.

Real-world SceneMethodNumber of TrialsSR ↑Reloc. SR ↑SafeSR ↑
OfficeEgo-only2050.00%20.00%40.00%
OfficeEgoExo-VLN2070.00%60.00%60.00%
CorridorEgo-only2045.00%20.00%20.00%
CorridorEgoExo-VLN2070.00%60.00%80.00%
Public AreaEgo-only2040.00%20.00%20.00%
Public AreaEgoExo-VLN2060.00%40.00%60.00%
OverallEgo-only6045.00%20.00%26.67%
OverallEgoExo-VLN6066.67%53.33%66.67%

Q2: Given the above weaknesses, can you clarify how the baselines were adapted to the benchmark?

We preserved the original network architectures and core navigation mechanisms of all seven conventional VLN baselines. All baselines were trained on the EgoExo-VLN training set and evaluated on the same validation and test sets. The adapted conventional baselines still use only egocentric observations and do not receive information from exocentric cameras.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.