- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ErrandVLN, a continuous VLN benchmark built on InteriorGS scenes in Isaac Sim (913 scenes, 65k trajectories, 502k instructions). Each episode is an ordered sequence of up to four landmark goals, and an LLM rewrites the same skeleton into instructions framed by different household roles and activities. The paper also proposes VLN-JEPA: a flow-based proposer samples candidate trajectories on a depth-derived occupancy map, a latent world model rolls out future latent states for each candidate, and these are fed as tokens to a 2B VLM that outputs the next action. It reports 12.6 SR / 20.5 GCR on Test Scene-Unseen, and 62.5 SR on R2R-CE when the proposer and world model are attached to JanusVLN.
Critical strengths
- Ordered multi-goal navigation in continuous, physically executable 3DGS scenes is a reasonable target, and the dataset is large.
- Separate held-out splits for scene, path and instruction style are useful for diagnosing generalization.
- The proposer and world model are trained only on ErrandVLN and still improve JanusVLN on R2R-CE, which is a non-trivial cross-simulator transfer result.
Critical weaknesses
-
The intent / role-conditioned aspect is what separates ErrandVLN from LH-VLN, GOAT-Bench and IVLN, but no experiment supports it. The underlying task is ordered multi-landmark navigation, and the roles only change the wording. There is no comparison against plain landmark-list instructions on the same trajectories, and Seen-Style-Unseen results are not in the main paper, so we cannot tell whether role framing adds any difficulty or requires different reasoning. Both the area graph and the instructions are produced by Qwen-3.5-Plus, with no human verification and no human performance reference. It is unclear how many episodes are ambiguous, e.g. when a landmark category has several instances but success is measured against one specific target position.
-
Table 2 does not support the claims. All VLN baselines are run zero-shot from released checkpoints in a new simulator and rendering domain, while VLN-JEPA is trained on ErrandVLN, so the comparison mostly measures domain shift. Some baseline numbers look like interface failures rather than policy behavior (NaVILA: PL 446.5 m). Even in this setting, zero-shot DualVLN has higher SR than the trained method (15.1 vs 12.6). At least a few baselines (e.g. StreamVLN, NaVid) should be fine-tuned on the ErrandVLN train split. With SR around 12%, it is also impossible to tell whether the difficulty comes from the multi-stage structure or from the sim/visual gap. Single-stage results on the same scenes would separate the two.
-
Sensor mismatch. VLN-JEPA uses depth to build the occupancy grid and to score candidates by traversability, while the compared VLMs are RGB-only. The 40% collision reduction highlighted in the abstract is largely what depth-based traversability filtering would give anyway. The same applies on R2R-CE: JanusVLN is RGB-only, and the augmented version adds depth (Table 3). JanusVLN + TP-Flow without NaviLat is needed to attribute the gain to latent prediction rather than to depth.
-
The evidence that the world model matters is weak. Removing NaviLat lowers SR from 12.6 to 9.8, and zeroing the future tokens lowers it to 10.2. Zeroed or shuffled tokens at test time are out-of-distribution inputs for a controller trained with real ones, so these drops do not show that the predicted content is informative; shuffling hurting more than zeroing suggests reliance on slot position. The right control is a controller trained with uninformative futures. No seeds or confidence intervals are given and the test split size is not stated in the main text, so 1-3 point differences are hard to judge.
-
The R2R-CE claim of "leading performance across all published continuous-navigation baselines" is not accurate. AwareVLN (CVPR 2026, arXiv 2605.22816) reports 65.4 SR / 73.5 OSR / 4.02 NE on R2R-CE Val-Unseen with monocular RGB only, above the 62.5 / 69.2 / 4.59 reported here with RGB + depth. The SPL here is higher (59.7 vs 55.1), which is worth discussing, but the claim should be corrected and the work compared. It is also relevant in substance, since it models task-progress awareness over multi-part instructions, which is exactly the failure mode of multi-stage errands.
-
Minor: SR and OSR use 3.0 m while GCR and GC-SPL use 1.5 m. Please justify. 3.0 m is loose for indoor landmarks, and with different thresholds SR can exceed GCR for a model that stops early (DualVLN: 15.1 vs 12.9).
Questions for authors
- How does performance change on the same trajectories with role-free, landmark-list instructions? What are the results on Val Seen-Path-Unseen and Seen-Style-Unseen?
- How are landmark instances disambiguated when a category appears more than once? Was any subset human-checked, and what is human SR?
- Results for baselines fine-tuned on ErrandVLN train?
- On R2R-CE, what does JanusVLN + TP-Flow alone (no NaviLat) achieve? Results over multiple seeds?
- Test split size, and a breakdown by number of stages (M = 1..4) in the main text.
12 comments
Question for the Authors Q2 (original):
How are landmark instances disambiguated when a category appears more than once? Was any subset human-checked, and what is human SR?
Response:
We thank the reviewer for asking how target instances are distinguished and how the data are verified.
Target descriptions and evaluation correspondence.
When a category has multiple instances, we describe the target in the generated instruction using its functional region, spatial relationships to surrounding objects, and visual attributes such as color and state. For example, “the chair beside the table in the dining room” provides language cues for distinguishing instances of the same category.
For annotation and evaluation, each stage uses an instance ID to link the reference trajectory and the corresponding reachable target position, keeping the instance used for instruction generation consistent with the evaluation target. Instance IDs are not supplied to the agent. Success is measured relative to that target position, using a 3 m threshold for SR and a 1.5 m threshold for each stage target in GCR. Evaluation thus measures arrival at the intended target, rather than proximity to any object of the same category.
Human verification and human performance.
The revision provides a human-verified subset of 1,000 trajectories, with trajectory–instruction pairs individually checked for physical reachability and multi-goal consistency. A dedicated audit of target-instance uniqueness across all instructions has not been conducted, and the proportion of ambiguous tasks has not been quantified. Given the dataset scale and the time and human effort required for navigation evaluation, human navigation SR/GCR results are not yet available in this revision.
W2 / Q3 (combined summary):
Please compare ErrandVLN-fine-tuned baselines to distinguish domain shift from multi-stage difficulty, and explain possible interface failures and DualVLN's higher SR.
Response:
We thank the reviewer for requesting matched-domain comparisons and stage-wise analysis.
NaViLA's 446.5 denotes mean collisions; its mean path length is 10.1 m.
We fine-tune NaViLA and StreamVLN on the official ErrandVLN training split with optimization schedules and supervision objectives matched to VLN-JEPA, evaluating all three on the same 1,000 human-verified trajectories.
Table 2. Overall comparison after matched in-domain fine-tuning. Δ denotes VLN-JEPA minus StreamVLN.
| Method | SR (%) ↑ | OSR (%) ↑ | GCR (%) ↑ | GC-SPL (%) ↑ | PL (m) | Col ↓ |
|---|---|---|---|---|---|---|
| NaViLA | 9.60 | 24.30 | 11.51 | 9.93 | 3.55 | 25.67 |
| StreamVLN | 19.40 | 37.00 | 20.88 | 18.63 | 5.88 | 59.57 |
| VLN-JEPA | 23.00 | 46.30 | 30.21 | 22.17 | 9.14 | 21.76 |
| Δ (vs. StreamVLN) | +3.60 | +9.30 | +9.33 | +3.54 | +3.26 | −37.81 |
After in-domain fine-tuning, VLN-JEPA leads both baselines on SR, OSR, GCR, and GC-SPL, with fewer collisions. Over StreamVLN, GCR and SR improve by 9.33 and 3.60 percentage points. Our advantage thus persists after adapting the external baselines to ErrandVLN.
Table 3. Stage-wise comparison after matched in-domain fine-tuning. Values are means [95% bootstrap CIs].
| Method | M | GCR (%) ↑ | GC-SPL (%) ↑ | SR (%) ↑ |
|---|---|---|---|---|
| NaViLA | 1 | 10.94 [7.8, 14.6] | 8.08 [5.4, 11.2] | 16.72 [12.8, 21.3] |
| — | 2 | 11.04 [8.2, 14.4] | 9.72 [7.1, 12.9] | 5.71 [3.7, 8.4] |
| — | 3 | 12.98 [8.1, 18.6] | 12.76 [8.0, 18.4] | 5.23 [2.3, 9.3] |
| — | 4 | 12.76 [6.8, 20.1] | 12.01 [6.3, 19.3] | 9.38 [4.2, 16.7] |
| StreamVLN | 1 | 24.01 [19.6, 28.8] | 19.18 [15.1, 23.7] | 27.66 [22.8, 32.8] |
| — | 2 | 20.47 [16.6, 24.6] | 19.16 [15.4, 23.3] | 18.11 [14.4, 22.3] |
| — | 3 | 18.60 [13.2, 24.8] | 17.90 [12.6, 24.0] | 10.47 [6.4, 15.7] |
| — | 4 | 15.89 [9.1, 23.7] | 15.89 [9.1, 23.7] | 12.50 [6.3, 20.8] |
| VLN-JEPA | 1 | 38.60 [33.4, 44.1] | 22.59 [18.3, 27.3] | 32.83 [28.0, 38.3] |
| — | 2 | 28.29 [24.1, 32.8] | 23.61 [19.6, 28.0] | 19.60 [15.9, 23.8] |
| — | 3 | 24.61 [18.4, 31.4] | 20.82 [14.9, 27.3] | 18.02 [12.8, 24.4] |
| — | 4 | 19.53 [12.2, 27.9] | 17.15 [10.2, 25.3] | 12.50 [6.3, 20.8] |
At M=3, VLN-JEPA exceeds StreamVLN in SR by 7.55 points. At M=4, SR ties at 12.50%, but our GCR and GC-SPL are 3.64 and 1.26 points higher. GCR and GC-SPL gains across all groups show consistently better ordered and path-efficiency-weighted completion. Lower multi-stage SR after adaptation also indicates remaining task difficulty.
Table 4. Stage-wise comparison with DualVLN on the main Test Scene-Unseen set. DualVLN uses public weights. OSR includes 95% bootstrap CIs; Δ denotes VLN-JEPA minus DualVLN.
| Method | M | GCR (%) ↑ | SR (%) ↑ | OSR (%) ↑ |
|---|---|---|---|---|
| DualVLN | 1 | 26.87 | 44.78 | 56.72 [44.8, 68.7] |
| — | 2 | 16.02 | 20.18 | 29.08 [24.3, 34.1] |
| — | 3 | 10.41 | 9.47 | 18.60 [14.0, 23.2] |
| — | 4 | 9.08 | 8.36 | 13.50 [10.0, 17.4] |
| — | Overall | 12.99 | 15.10 | 23.10 [20.5, 25.7] |
| VLN-JEPA | 1 | 37.31 | 32.84 | 62.69 [50.7, 74.6] |
| — | 2 | 24.78 | 17.21 | 36.80 [31.8, 41.8] |
| — | 3 | 16.73 | 9.47 | 25.26 [20.4, 30.5] |
| — | 4 | 15.84 | 6.11 | 17.36 [13.2, 21.5] |
| — | Overall | 20.54 | 12.60 | 29.20 [26.4, 32.0] |
| Δ | 1 | +10.44 | −11.94 | +5.97 |
| — | 2 | +8.76 | −2.97 | +7.72 |
| — | 3 | +6.32 | 0.00 | +6.66 |
| — | 4 | +6.76 | −2.25 | +3.86 |
| — | Overall | +7.55 | −2.50 | +6.10 |
DualVLN's 44.78% single-stage SR demonstrates effective navigation in this simulator; its four-stage SR falls to 8.36%, further illustrating the challenge of multi-stage execution.
VLN-JEPA achieves 20.54% overall GCR and 29.20% OSR, exceeding DualVLN by 7.55 and 6.10 percentage points, respectively, with gains in every stage-count group. These results demonstrate higher ordered subgoal completion and all-target reachability. TP-Flow supplies traversability-informed candidate actions; NaviLat forecasts their consequences, providing future-state information for VLM action generation. Module ablations further support these gains (Table 6 in the revised manuscript).
DualVLN has higher overall SR (15.10% versus 12.60%); its training covers ScaleVLN and more extensive DAgger data, providing broader scene and correction experience. We will expand training-data coverage and interactive correction training to further improve whole-task success.
Question for the Authors Q1 (original):
Response:
We thank the reviewer for requesting clarification of the task setting and results across evaluation splits.
Task definition and contribution.
ErrandVLN evaluates whether an agent can fulfill the multi-stage navigation requirements of a semantically coherent household errand from natural-language instructions and RGB observations. Stages serve a common activity purpose: the agent must infer destinations and their order from a unified request and navigate in a continuous environment. Roles express the speaker's perspective and motivation; the task contribution is the organization of navigation requirements around a common activity purpose.
Landmarks serve different roles in dataset construction and agent execution. We use landmarks to identify reachable target positions, connect consecutive goals using A* to construct reference trajectories, and generate natural-language instructions in an activity context. Target positions support evaluation of stage completion and visiting order. The agent receives only instructions and RGB observations, without landmark lists, instance identifiers, goal coordinates, or structured visiting sequences. Using landmarks internally to construct trajectories therefore does not make the agent's task landmark-list navigation.
This follows the standard VLN definition: R2R [1] requires agents to interpret and follow natural-language navigation instructions using visual observations, and VLN-CE [2] extends this paradigm to continuous 3D environments. ErrandVLN extends it to multi-stage requests with a shared activity purpose. LH-VLN [3] sequentially combines subtasks without requiring a common overarching activity; ErrandVLN connects stages semantically. GOAT-Bench [4] supplies successive goals through category names, descriptions, or instance images; ErrandVLN expresses interconnected requirements in one natural-language request. IVLN [5] emphasizes memory across episodes in a persistent environment; ErrandVLN emphasizes multi-stage execution within an episode.
Landmark-list comparison and cross-split results.
We have not conducted a comparison with plain landmark-list instructions on the same trajectories. Our setting evaluates understanding and executing navigation requirements expressed in a high-level request, whereas a landmark list explicitly enumerates destinations and their order. The table below reports cross-split results under the current instruction setting.
Table 1. VLN-JEPA navigation performance across ErrandVLN evaluation splits. Values are means [95% bootstrap confidence intervals].
On Val Seen-Style-Unseen, GCR is 18.15%, SR is 9.10%, and GC-SPL is 15.46%, reporting navigation performance under unseen persona-conditioned expressions in familiar scenes.
References:
[1] Anderson et al. Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments. CVPR, 2018. [2] Krantz et al. Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments. ECCV, 2020. [3] Song et al. Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Method. CVPR, 2025. [4] Khanna et al. GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation. CVPR, 2024. [5] Krantz et al. Iterative Vision-and-Language Navigation. CVPR, 2023.