- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces VCN-Bench, where an agent uses a prior video to identify a target and navigate to it. It also proposes MV-DualVLN as a baseline.
Critical strengths
Separating target identification from subsequent navigation is useful for analyzing failures.
Critical weaknesses
1.Only a few navigation methods are evaluated. Testing many MLLMs on goal identification does not make up for the limited coverage of navigation baselines, especially for a benchmark paper.
2.I am not convinced that the paper clearly establishes the value of this new task. What useful insight does adding a prior video provide beyond existing VLN-CE and memory-based navigation tasks? The paper needs stronger evidence that this is a meaningful evaluation setting.
Questions for authors
What concrete gap in existing navigation evaluation does this task address, and what evidence shows that the observed difficulty persists across stronger, properly adapted navigation methods?
3 comments
Response to Reviewer E9vo (2/3)
W2: The value of this new task.
The contribution of VCN-Bench is to evaluate whether an agent can resolve a destination from supplied visual experience and use that experience to guide subsequent closed-loop navigation. Its value lies in the new navigation paradigm and the capability being examined, rather than the additional input modality alone.
1. Existing navigation performance does not directly establish this capability.
As stated in L57-63, conventional route-following VLN benchmarks provide instructions describing paths, landmarks, and actions. In many goal-oriented navigation settings, the destination is explicitly specified, while scene knowledge is acquired through exploration. Lifelong navigation further evaluates the reuse of accumulated experience, but the available experience depends on the agent’s previous trajectories and coverage. These settings certainly involve spatial reasoning; however, their aggregate scores do not directly reveal how well an agent can reason over a supplied prior visual experience and translate it into navigation decisions.
2. VCN-Bench makes the role of prior experience explicit.
Each episode supplies a prior video covering the initial location and destination, reducing variation in experience acquisition. In particular, Room-to-Object instructions define the destination through relations within that experience—for example, the second bedroom visited or the farthest bedroom among those shown in the video. The video therefore participates in determining which destination is intended, rather than merely helping the agent search for an already specified target. Moreover, its trajectory is not a prescribed route to replay: the agent must navigate from its actual initial state using changing online observations. The benchmark thus examines both destination resolution and the subsequent use of experience for action.
3. The experiments provide insights beyond showing that the task is difficult.
-
Prior scene context is useful beyond the destination frame. As shown in Table 3 in the main paper, the full-video MV-DualVLN achieves 27.6% SR, compared with 19.7% for the variant trained and evaluated with only the ground-truth target frame. Thus, a correct target reference does not substitute for the broader scene context available in the prior video.
-
Correct destination information does not eliminate navigation failures. As shown in Appendix D.2, replacing predicted goal frames with ground-truth frames throughout navigation in the same trained MV-DualVLN improves SR from 27.6% to 33.1%. With model parameters and the remaining evaluation setup unchanged, substantial failures persist, showing that inaccurate destination resolution is not the sole bottleneck.
Taken together, we believe VCN-Bench provides a meaningful evaluation setting beyond conventional route-following or goal-oriented navigation, by explicitly examining whether spatial knowledge inferred from prior visual experience can be carried forward to guide navigation.
Response to Reviewer E9vo (3/3)
Q1: What concrete gap in existing navigation evaluation does this task address, and what evidence shows that the observed difficulty persists across stronger, properly adapted navigation methods?
The gap between VCN-Bench and existing navigation benchmarks is discussed in response to W2. The key distinction is that strength on existing navigation benchmarks does not automatically imply strength in reasoning over prior visual experience. Existing tasks certainly involve spatial reasoning, but evaluate it jointly with route following, exploration, and memory acquisition. Their aggregate scores do not directly establish whether an agent can resolve an experience-dependent destination from a supplied video and use that experience to guide subsequent closed-loop navigation. This is the complementary capability that VCN-Bench is designed to evaluate.
We would like to clarify that our evaluation already includes strong, representative navigation methods—not merely generic MLLMs applied to navigation without adaptation. We select MTU3D, Uni-NaVid, and 3D-Mem, and build MV-DualVLN, covering complementary approaches to goal-oriented navigation, video-based action prediction, explicit memory, and waypoint planning. These methods are explicitly adapted to VCN-Bench, as detailed in Appendix C.3. Beyond zero-shot evaluation, we fine-tune 3D-Mem on VCN-Bench and extend DualVLN into MV-DualVLN to jointly process prior video, navigation history, and multi-view observations. The latter two achieve 18.1% and 27.6% SR, respectively. Thus, substantial difficulty remains even with task-specific training, rather than only under unadapted cross-task transfer.
Importantly, our argument does not rely on low success rates alone. In MV-DualVLN, providing ground-truth goal frames throughout navigation increases SR from 27.6% to 33.1%, showing that incorrect destination identification is not the sole bottleneck. Meanwhile, MV-DualVLN outperforms the target-frame-only variant (27.6% vs. 19.7% SR), demonstrating the utility of prior scene context beyond the target frame itself. Together, these results show that the benchmark probes a meaningful capability not established by performance on conventional navigation tasks, and that the challenge persists across the evaluated navigation paradigms, including task-fine-tuned methods.
Response to Reviewer E9vo (1/3)
W1: Only a few navigation methods are evaluated
We would like to clarify that we have tried our best to evaluate as many navigation models as possible that can be adapted to VCN-Bench, covering both hierarchical and end-to-end paradigms. We also fine-tuned 3D-Mem and MV-DualVLN to better adapt them to VCN-Bench. We welcome the reviewer to provide specific models suitable for evaluation on VCN-Bench; we would be happy to include their evaluation results.
VCN-Bench differs from conventional route-following VLN tasks in several aspects. First, the destination is not explicitly specified by a route description or an object category. Instead, it must be resolved from a prior video according to spatial or temporal relations. Second, the prior video serves as persistent scene context that should be leveraged throughout navigation, rather than a sequence of observations to be directly followed. Third, the agent must make sequential navigation decisions based on this context and its evolving egocentric observations. In contrast, many publicly available VLN models are primarily designed for R2R/RxR-style route-following instructions. Although some goal-oriented navigation models support object-centric tasks, they typically assume an explicitly specified object category, instance description, or target representation, rather than a destination that must first be inferred from a prior video.
Therefore, we select methods for which a meaningful adaptation to our evaluation protocol is feasible, rather than reporting results from incompatible settings. The table below reports the performance of NaVILA on VCN-Bench. As NaVILA is designed exclusively for instruction-following navigation, its results on VCN-Bench are inherently limited. For meaningful comparison, we excluded models that support instruction following only, which accounts for the relatively small number of methods in Table 1.
Our current evaluation covers complementary navigation paradigms: (1) a hierarchical approach that combines MLLM-based target-frame selection with MTU3D, which is a SoTA model for goal-oriented navigation; (2) end-to-end methods, including zero-shot 3D-Mem and Uni-NaVid, which are inherently capable of both video understanding and navigation; and (3) task-adapted end-to-end methods, including 3D-Mem and our MV-DualVLN, fine-tuned on VCN-Bench training episodes. In particular, the fine-tuned 3D-Mem and MV-DualVLN achieve 18.1% and 27.6% SR, respectively. These comparisons provide evidence across different navigation paradigms.