acceptodds
Withdrawn from ICLR 2027

AvatarVLN: Agentic World Modeling for Vision-and-language Navigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World models provide a promising way for vision-and-language navigation (VLN) agents to anticipate the consequences of candidate actions before physically executing them. However, imagined futures are not inherently actionable. A synthetic trajectory may span multiple instruction-relevant events, omit events that would occur, fabricate events that would not, and provide increasingly unreliable geometry over long rollouts. We present AvatarVLN, an agentic world modeling framework that turns imperfect imagined futures into actionable evidence for VLN. Rather than relying on a task-specific evaluator, AvatarVLN leverages the reasoning capability of the navigation agent itself. The agent decomposes an instruction into ordered events and tracks its progress through them. Specifically, we introduce Anticipated Progress Grounding (APG), which models navigation progress as a latent state. Rather than committing to a single progress estimate, APG weights possible progress states according to real history and imagined rollouts to estimate the reduction in remaining navigation cost. In addition, this formulation uses instruction order to accommodate omitted intermediate events while discounting unsupported progress suggested by fabricated content. Beyond semantic grounding, we introduce Uncertainty-aware Geometric Grounding (UGG), which estimates the reliability of imagined depth from accumulated motion along each rollout and selectively projects trustworthy geometry into candidate maps for spatial reasoning. The world model is trained on a navigation corpus derived from VLNVerse and transfers zero-shot to unseen VLN benchmarks. Experiments on three VLN benchmarks demonstrate the effectiveness of AvatarVLN, with ablations showing consistent gains on success rate over the base agents on R2R-CE.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-and-language navigation, agentic navigation, depth calibration, world model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tA4UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AvatarVLN, which equips a frozen navigation agent with candidate future trajectories generated by a world model. APG evaluates progress through ordered instruction events, while UGG calibrates and filters predicted depth to construct temporary candidate maps; the world model is trained on VLNVerse and transferred to R2R-CE and RxR-CE without fine-tuning the navigation agent. The complete framework improves SR by 10–12 percentage points in matched-backbone R2R-CE ablations, with additional full-split evaluations and detailed success and failure analyses.

Critical strengths
  1. It distinguishes instruction-progress assessment from geometric reliability and keeps predicted maps temporary, grounding decisions in real observations after each move. This design directly addresses omitted events, fabricated scenes, and geometric drift that can mislead navigation.

  2. Table 3 compares three backbones on the same 100 episodes, with paired confidence intervals excluding zero for the full framework’s SR improvement over each base agent. The appendix also documents misinterpreted rollouts, premature stopping based on predictions, and deliberation-induced timeouts, clearly illustrating important limitations.

Critical weaknesses
  1. The main text defines APG through progress distributions, expected remaining costs, and discounted scores, but Appendix A.6 explicitly states that adaptive APG does not require numerical posteriors or all scoring terms, instead using qualitative event-comparison prompts. Consequently, the strongest results support the prompting protocol’s effectiveness without establishing the necessity of the proposed latent-progress inference mechanism.

  2. Without a control retaining these reasoning instructions but removing world-model imagination, the contributions of imagined evidence and the specific APG mechanism remain difficult to separate.

  3. Although calibration, motion-dependent decay, and cross-frame filtering are described in detail, the paper does not report their effects on depth error, falsely traversable regions, or the relationship between confidence and actual geometric error. Table 3 only evaluates adding UGG as a whole, leaving unclear whether the gains arise from providing an additional BEV or from the proposed uncertainty estimation and filtering.

  4. Each configuration in Table 3 is run once, and all confidence intervals compare against the base agent rather than adjacent variants; forced APG improves SR over naive imagination by only one percentage point for each backbone. Evidence that the complete framework works therefore does not establish independent, stable benefits from each central component.

  5. The appendix reports over 30 seconds per decision round for the 27B agent and an additional 8–15 seconds per imagination request, including a case where extra deliberation turns an otherwise successful episode into a timeout. Without complete runtime distributions and compute-matched repeated-reasoning or candidate-reranking baselines, it remains unclear whether world-model imagination is an efficient way to obtain these gains.

Questions for authors
  1. Which APG computations are actually executed in the best-performing configuration? Please distinguish programmatically computed quantities from qualitative model judgments and compare numerical belief inference, deterministic event tracking, and simplified prompting under matched budgets.

  2. How does the agent perform with the same event-reasoning prompts but without the world model? Could this control explain part of the improvement over the base agent?

  3. Does UGG measurably improve geometric accuracy? Please report depth and traversability errors, separately ablate calibration, motion-dependent decay, and cross-frame filtering, and compare against BEV construction from directly fused predicted depth.

  4. Are the individual component gains stable? Can you provide paired confidence intervals between adjacent variants and results across repeated runs, rather than only comparisons against the base agent?

  5. Does the advantage persist under matched time or inference budgets? Please report episode runtimes and timeout rates for the complete method and compare against alternatives that spend the additional budget on direct reasoning or candidate reranking.

Reviewer_E8jYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

To address issues such as event illusion and geometric distortion in the future trajectories generated by the world model in visual language navigation, AvatarVLN proposes two modules, APG and UGG, to evaluate the extent to which the imagined trajectory advances ordered language instructions and the reliability of the predicted geometric information, respectively, and to use them to assist the frozen navigation agent in selecting actions.

Critical strengths
  1. The method design is meaningful. The paper proposes two modules, APG and UGG, to evaluate the extent to which imagined trajectories advance ordered language instructions, and to calibrate and filter unreliable predicted geometric information, respectively. The two modules have a clear functional division and can provide additional decision-making support to the basic navigation agent without fine-tuning it.
  2. Experiments cover multiple benchmarks and base models of varying sizes. The paper conducts experiments on R2R-CE, RxR-CE, and VLNVerse, and evaluates the method using visual language models of different sizes. This experimental design allows for some verification of whether the method depends on a specific model size or a single navigation benchmark.
  3. A relatively systematic ablation experiment and statistical analysis are provided. The paper progressively incorporates components such as imaginary ablation, APG, adaptive APG, and UGG, and reports the paired bootstrap confidence intervals between some configurations and the base model. These analyses help to understand the performance variations of the complete framework and provide some statistical support for the experimental conclusions.
Critical weaknesses
  1. The core methodology lacks innovation. The paradigm of using detailed futures generated by world models for navigation, supplemented by external modules for filtering, has existed for a long time.
  2. The authors identify the current challenge as "However, imagined futures are not inherently actionable," but instead of directly addressing this issue by improving the world model itself. Since the paper argues that the future generated by the world model is unreliable, why not study how to improve the reliability of the world model, instead of relying mainly on additional screening mechanisms? Compared to directly improving the world model, does this post-processing method have sufficient innovation and practical advantages? 3.Using the final navigation success rate in experiments cannot prove that the module solves the problems claimed by the authors. For example, APG claims to solve the problems of missing and fabricated command events, but the navigation success rate only reflects whether the final task is completed and cannot directly indicate whether APG has actually identified command progress more accurately. 4.The practical efficiency of the method remains an issue. World model imagination and additional visual-language model inference introduce significant computational overhead. The paper reports a high frequency of imagination calls and long single-step decision times in some settings; therefore, a more thorough analysis of the trade-off between improved success rate and end-to-end inference latency is needed to demonstrate the method's practical application value.
Questions for authors
  1. Please provide more direct mechanisms for verification. For example, for the accuracy of APG progress assessment, and for the depth error and geometric consistency of UGG measurements before and after filtering.
  2. If all future trajectories generated by the world model contain obvious errors, can APG and UGG identify these errors and prevent the navigation agent from making decisions based on erroneous predictions?
  3. How sensitive is AvatarVLN to changes in the underlying world model? Would APG and UGG remain effective when applied to world models with different prediction quality?
Reviewer_j8YEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AvatarVLN, an agentic world-modeling framework for vision-and-language navigation. The framework uses an RGB-D video world model to imagine candidate navigation trajectories and introduces two complementary mechanisms for interpreting these predictions. Anticipated Progress Grounding (APG) evaluates candidate trajectories against ordered instruction events using a latent progress formulation, while Uncertainty-aware Geometric Grounding (UGG) calibrates and filters predicted geometry based on real observations and motion-dependent confidence. The framework operates with frozen navigation agents and requires no additional task-specific learned decision module. Experiments on R2R-CE, RxR-CE, and VLNVerse demonstrate improvements over zero-shot navigation baselines.

Critical strengths
  1. Well-motivated problem formulation. The paper identifies an important limitation of world-model-based navigation: plausible imagined trajectories do not necessarily provide reliable evidence for action selection. Addressing both instruction progress and geometric uncertainty is a meaningful direction.
  2. Detailed experimental analysis. The paper provides component ablations, paired confidence intervals, computational-cost measurements, and extensive case studies, including failure examples. These analyses help characterize both the benefits and limitations of imagination-based navigation.
Critical weaknesses
  1. Discrepancy between the mathematical formulation and implementation. APG is formulated using progress-state beliefs, expected remaining costs, and guarded progress scores. However, Appendix A.6 clarifies that these quantities are not explicitly computed through a numerical inference procedure. Instead, the implementation relies primarily on LLM prompting and qualitative judgments. This weakens the technical contribution of the proposed probabilistic formulation and makes it difficult to distinguish the benefits of the mathematical design from those of structured prompting.
  2. Insufficient isolation of the proposed components. The adaptive APG protocol changes both when structured reasoning is invoked and how extensively the agent reasons. Consequently, its improvement over forced APG cannot be attributed solely to progress-aware scoring. Similarly, UGG provides relatively modest additional gains, without sufficiently isolating the contributions of geometric calibration, confidence weighting, and consistency filtering to navigation performance.
  3. Limited reliability of imagined-progress reasoning. The qualitative analysis reveals cases where agents hallucinate events absent from generated rollouts, stop based on predicted rather than observed goal completion, or provide incorrect explanations for otherwise successful decisions. These failures raise concerns about whether the proposed safeguards reliably prevent the errors they are designed to address.
  4. Substantial inference overhead. Each imagination request adds approximately 8–15 seconds, while a single 27B agent decision can take over 30 seconds. The resulting computational cost substantially limits practical deployment, particularly for embodied agents requiring frequent closed-loop decisions. The accuracy-efficiency trade-off remains insufficiently characterized.
Questions for authors
  1. Are the progress beliefs, remaining costs, and guarded scores in Eqs. (3)–(6) explicitly estimated in the deployed system, or do they primarily serve as conceptual guidance for LLM prompting? Can the authors provide a controlled comparison between the full APG formulation and a simpler ordered-event prompting baseline?
  2. Can the authors disentangle the effects of adaptive tool invocation, reasoning verbosity, and progress-aware scoring? In particular, how does adaptive APG compare with an equally concise event-based reasoning protocol under a matched imagination budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.