acceptodds
Under review as a conference paper at ICLR 2027

InteractiveVLN: Proactive Interaction via Uncertainty-Aware Decision for Zero-Shot Vision Language Navigation

Abstract

Most existing Vision-Language Navigation (VLN) methods are evaluated in static environments with single-turn instructions, limiting their ability to generalize to tasks in dynamic, interactive, realistic settings. Moreover, pre-trained navigation policies often struggle to understand complex natural language instructions, tending to overfit to narrow, task-specific directives. To address these challenges, we propose a zero-shot, interaction-enabled VLN agent framework, complementary to task-specific fine-tuning, in which the agent can actively query for task-relevant information during navigation. Unlike traditional model-based approaches, our framework centers on a multimodal long-short memory and an uncertainty-triggered Ask action. This integration enables the agent to anticipate and navigate unseen scenarios when its uncertainty is high. We evaluate our method on the FreeAskWorld and R2R-CE public benchmarks. Without any task-specific fine-tuning, our method achieves the first non-zero success among the zero-shot VLN methods we compare on FreeAskWorld. Notably, the agent autonomously determines when to request guidance: when memory is removed, it seeks assistance more often because uncertainty rises, rather than navigating blindly. These results highlight the potential of our approach for embodied AI and suggest a promising direction for deploying agents in dynamic, collaborative, realistic environments.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision Language Navigation, Proactive Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qeapOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the challenges of limited generalization to dynamic and interactive environments and difficulties in understanding complex navigation instructions in existing vision-and-language navigation (VLN) methods. To this end, the authors propose InteractiveVLN, a zero-shot proactive interaction framework that requires no task-specific fine-tuning. Specifically, the framework employs a multimodal long-short memory module to preserve historical visual information, spatial exploration experience, and task execution progress. It further integrates visual uncertainty, policy uncertainty, and memory uncertainty to determine when the agent should proactively request external navigation guidance. Experiments conducted on the FreeAskWorld and R2R-CE benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The problem addressed in this paper is of practical significance, as feedback, whether visual or human-provided, is essential for effective navigation in real-world dynamic environments.

  2. The proposed method integrates multiple modules to address these challenges, resulting in a relatively comprehensive framework that demonstrates a certain degree of extensibility and potential for practical applications.

Critical weaknesses
  1. The overall technical novelty of the proposed method is relatively limited. Proactive interaction, uncertainty-driven feedback requests [1], and fast-slow reasoning [2] have already been explored in existing vision-and-language navigation research. The proposed method primarily integrates existing techniques, including long-short memory, task planning, execution verification, and proactive querying, into a unified framework, but lacks substantial technical innovation.

  2. The validity of the uncertainty estimation method requires further investigation. The paper determines whether to execute the Ask action by combining visual uncertainty, policy uncertainty, and memory uncertainty through a weighted sum. However, these uncertainty measures primarily rely on relatively simple heuristic methods and may be affected by the complexity of real-world environments. The current experiments provide insufficient evidence that these uncertainty measures can accurately reflect the reliability of navigation decisions.

[1] Heeju Ko, Sung June Kim, Gyeongrok Oh, Jeongyoon Yoon, Honglak Lee, Sujin Jang, Seungryong Kim, and Sangpil Kim. Active Test-time Vision-Language Navigation. Advances in Neural Information Processing Systems (NeurIPS), 2025.

[2] Yang Li, Aming Wu, Zihao Zhang, and Yahong Han. Towards Open Environments and Instructions: General Vision-Language Navigation via Fast-Slow Interactive Reasoning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026.

Questions for authors
  1. How can the authors demonstrate that the proactive querying mechanism effectively improves navigation performance? The paper presents uncertainty-driven proactive interaction as a core contribution. However, the configurations with and without the Ask action in Table 4 show no difference in navigation success rates.

  2. Can the current uncertainty estimation method accurately reflect the actual uncertainty of navigation decisions? The paper employs grayscale image entropy, action sampling entropy, and visual feature distance to estimate uncertainty. However, these measures may be influenced by visual complexity, viewpoint changes, and environmental dynamics. Could the authors provide direct quantitative experiments to investigate the relationships between these uncertainty measures and the probability of navigation errors or the risk of task failure?

Reviewer_ftEDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work introduces InteractiveVLN, a training-free agent designed for vision-language navigation in dynamic and interactive environments, uniquely equipped to ask for directions during transit. The authors formalize the task as a Partially Observable Markov Decision Process (POMDP), expanding the traditional action space to include an "Ask" action alongside standard navigation primitives. The model's objective function is designed to balance task success against uncertainty and query cost. In practice, the "Ask" action is triggered when a fixed weighted sum of three hand-designed uncertainty metrics exceeds a threshold of 0.65. These metrics consist of grayscale-histogram entropy, action disagreement across eight sampled planner actions, and the embedding distance between the current view and the most recent image stored in memory. Upon triggering, the agent receives an answer via a templated query directed to the simulator's ground-truth oracle. Architecturally, the agent is built on AgentFlow and integrates an LLM-based planner, a verifier, a long-short multimodal memory module, RTAB-Map for value mapping, and Grounded-SAM2 tools. Evaluated on the FreeAskWorld benchmark, the GPT-4o-powered version achieves a 1.09% Success Rate (SR) and a 4.35% Oracle Success Rate (OSR), results the authors highlight as the first non-zero success among the evaluated zero-shot methods. Furthermore, it achieves a 26.8% SR and an 8.26 Success weighted by Path Length (SPL) on the R2R-CE val-unsee split. The paper rounds out its evaluation with an ablation study, qualitative examples, and a single real-world demonstration using a Unitree Go2 robot.

Critical strengths
  1. The paper addresses an important and underexplored paradigm: closed-loop navigation in dynamic environments where the agent can actively ask for language assistance. The human baselines in Table 1 clearly validate this approach, demonstrating that asking for help roughly doubles the success rate (from 40.2% to 82.6%).

  2. The comparative analysis is comprehensive, spanning pretrained VLN models (with and without fine-tuning), navigation foundation models, zero-shot LLM agents, and four distinct VLM backbones. By reporting results across fine runs with standard deviations (Table 2), the evaluation achieves a level of rigor often missing in typical agent-based research.

  3. The uncertainty trigger is explicitly defined (Equations 4, 5, 6, and 7), and its weights and threshold remain fixed across all episodes and backbones. Leveraging the disagreement among K=8K = 8 sampled planner actions is a highly practical and black-box uncertainty signal for evaluating closed-source LLMs.

  4. The proposed framework is entirely training-free and highly modular. It transfers effectively to the R2R-CE benchmark, achieving a competitive Success Rate (26.8%, compared to 25.5% for AO-Planner and 25.3% for CA-Nav). Also, its deployability is validated on a physical Unitree Go2 robot.

  5. The authors acknowledge key limitations, noting that the metric weights and threshold were not systematically calibrated, the SPL lags behind specialized planners, and simply toggling the "Ssk" feature on or off does not independently change the SR or OSR. This transparency significantly aids the reader in objectively assessing the validity of the claims.

Critical weaknesses
  1. With a test set of only 46 episodes, a 1.09±0.6%1.09 \pm 0.6\% SR translates to roughly one successful episode every two evaluation runs. Moreover, since ViNT achieves a higher SR (2.17%) and SPL (2.13%), the claim of delivering the "first non-zero success" relies on arbitrary method categorization rather than a meaningful performance advantage.

  2. The central "Ask" mechanism is severely underutilized, triggering only 0.05 times per episode compared to 0.78 for humans. The paper's own ablation study shows that enabling or disabling this feature leaves both SR and OSR completely unchanged. Consequently, the primary benefit of interaction is never empirically substantiated.

  3. The abstract asserts that removing memory increases the frequency of queries because "uncertainty rises". However, removing the verifier increases queries even more (NDI 0.36 vs. 0.28 without memory). Since UmemoryU_{memory} is calculated relative to a stored image, removing memory likely inflates the score mechanically, not reflecting genuine reasoning uncertainty. Furthermore, the ablation study fails to evaluate the full system with the asking feature disabled.

  4. The employed uncertainty score is an arbitrary heuristic that is neither derived from nor optimized by the POMDP objective (Eq. 2), as the parameters λ\lambda and β\beta never appear in the implemented method. Additionally, grayscale-histogram entropy measures image texture instead of navigational uncertainty. The selected weights (0.1/0.7/0.2) and the 0.65 threshold lack any tuning, justification, or sensitivity analysis.

  5. A robust failure analysis is noticeably absent. The grant travels an average of only 13.6 m (vs. 59.9 m for humans), strongly suggesting that failures predominantly stem from the 150 s time limit (given the 5-6 s per decision latency), rather than the claimed "limited distance perception". On the R2R-CE benchmark, the SPL (8.26) is the lowest among all compared methods; the marginal +1.3 SR gain lacks variance reporting, and the paper fails to clarify whether the "Ask" action is even available in this setting.

  6. The agent receives answers exclusively from a simulator oracle equipped with ground-truth knowledge of the environment, restricted to just two fixed query templates. This highly idealized setup fails to evaluate the agent's ability to interact with actual pdestrans, which is the realistic scenario the paper uses to motivate the research.

  7. The comparative setup introduces several confounding variables that undermine fair benchmarking. Zero-shot methods are allocated 150 seconds per episode, while other baselines are restricted to 100 seconds. Also, navigation foundation models are evaluated using goal images, and on the R2R-CE benchmark, InteractiveVLN leverages all four sensor modalities while competing methods, such as AO-Planner, are restricted to fewer inputs.

Questions for authors
  1. How many episodes are in the FreeAskWorld test set, and what are the exact raw success counts per evaluation run? More importantly, does the full system's performance change if the "Ask" action is explicitly disabled?

  2. How is UmemoryU_{memory} mathematically computed during the ablation where the memory module is removed? Also, how do the query frequency and SR fluctuate across different heuristic weights and uncertainty thresholds?

  3. What proportion of the failures on the FreeAskWorld benchmark are strictly attributable to timeouts? Moreover, regarding the R2R-CE evaluation: is the "Ask" action actively available to the agent, and was the model evaluated on the complete val-unseen split?

  4. How does the agent's performance degrade when oracle responses are noisy, partial, or occasionally incorrect, mirroring real-world interactions with pedestrians? Finally, what are the per-episode API costs and wall-clock execution times in comparison to the evaluated baselines?

Reviewer_BwjzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces InteractiveVLN, a zero-shot vision-language navigation framework designed for dynamic and interactive environments. The main idea is to augment a general-purpose vision-language model with multimodal long-short memory, a verifier, mapping, structured planning, and a proactive Ask action. The Ask action is triggered when a hand-designed uncertainty score exceeds a fixed threshold. The method is evaluated mainly on FreeAskWorld, where the agent can request directions from a simulator oracle, and additionally on R2R-CE. On FreeAskWorld, InteractiveVLN with GPT-4o reaches 1.09% success rate and 4.35% oracle success rate, while several compared zero-shot methods obtain zero success. On R2R-CE, the method obtains 26.8% SR but a relatively low SPL of 8.26%. The paper also reports ablations on memory, verification, and proactive interaction, and includes a qualitative real-world rollout on a Unitree Go2.

Critical strengths

S1. The paper addresses an important limitation of standard VLN evaluation. Allowing an agent to request information during navigation is a natural extension of the standard single-turn VLN setting. I think the emphasis on interaction is relevant for embodied agents operating in realistic environments.

S2. The system is evaluated in more than one setting and includes a physical deployment. The FreeAskWorld experiments, R2R-CE evaluation, and Unitree Go2 rollout provide a broader validation than a purely simulator-specific demonstration. I also appreciate that the authors openly report that low-level navigation remains a weakness on R2R-CE.

Critical weaknesses

W1. The main claim about proactive interaction is not convincingly demonstrated by the results. This is the most important concern for me. The paper is centered on the Ask action, but the ablation does not show a success improvement caused by asking. When proactive action is switched off in the planner-only configuration, SR and OSR remain at zero. The reported benefit is mainly a shorter trajectory and a small reduction in navigation error. Conversely, the largest degradation comes from removing memory, not from removing Ask. This makes the title and central framing somewhat stronger than the experimental evidence. It seems to me that the current results demonstrate an interesting interactive mechanism, but not yet that proactive interaction materially improves navigation success.

W2. The uncertainty mechanism is largely heuristic and insufficiently validated. The Ask decision is central to the paper, but the uncertainty function combines image histogram entropy, disagreement across eight planner samples, and embedding distance from the latest memory image using fixed manually chosen coefficients. The authors explicitly state that these weights and the threshold were not calibrated and that alternative thresholds were not tested. I find the visual uncertainty term particularly difficult to justify. Grayscale histogram entropy measures low-level image complexity, but it is not clear why this should correspond to uncertainty relevant to navigation. Without sensitivity analysis or comparison to simpler uncertainty baselines, it is difficult to know whether the proposed decision mechanism is robust or whether the reported behavior depends on these arbitrary choices.

Questions for authors

Q1. Can the authors provide a stronger experiment isolating the causal benefit of the Ask action? In particular, what happens if the same agent is evaluated with Ask enabled and disabled while keeping memory, verifier, planner, and all other components identical? The current ablation does not seem to demonstrate an improvement in SR or OSR due to asking.

Q2. How sensitive are the results to the uncertainty weights and the threshold of 0.65? A sweep over the threshold, or at least a comparison with simpler criteria such as policy disagreement alone, would materially affect my confidence in the proposed uncertainty mechanism.

Q3. How many FreeAskWorld evaluation episodes are used in total? The reported success rates appear to correspond to very few successful trajectories. For example, 1.09% seems compatible with a single successful episode. If so, I would like to understand how statistically meaningful the comparison between methods is.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.