acceptodds
Withdrawn from ICLR 2027

GuidedNav: Shaping Vision-Language Navigation Representations through Pre-Action Attention and Spatial Guidance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-and-language navigation (VLN) requires an agent to track progress through an extended instruction, identify landmarks relevant to its current objective, and understand the surrounding spatial structure to act. End-to-end navigation policies typically rely on action supervision to implicitly learn these capabilities. Without explicit guidance on the evidence underlying each decision, however, policies can overfit to correlations predictive of training actions rather than learn the taskrelevant grounding and geometry needed for generalization. Our insight is to guide the representations that connect instructions to actions: identifying what remains to be done determines which visual evidence matters, while understanding the geometry around that evidence informs how to move. We introduce GuidedNav, a unified framework that explicitly shapes these representations through trainingtime guidance. Subtask and landmark guidance steer native pre-action attention toward the active instruction segment and relevant visual regions, grounding action prediction in the current objective. Spatial guidance aligns the policy’s internal representations with geometric features that capture long-horizon context and local spatial structure. Together, these objectives strengthen navigation capabilities while avoiding the inference overhead of intermediate textual reasoning and online geometric encoding. We also introduce Guided-R2R and Guided-RxR, two annotation-enriched navigation datasets that support more interpretable VLN research. GuidedNav establishes state-of-the-art performance among the compared monocular RGB policies, improving success rates by 8.1% and 13.1% relative to the strongest prior results on R2R-CE and RxR-CE val-unseen, respectively, while achieving faster inference. Extensive experiments further demonstrate the effectiveness and real-world transferability of our approach.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-and-Language Navigation, Representation Learning, Attention Guidance, Landmark Grounding, Spatial Perception

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QBXUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces GuidedNav, a training-time guidance framework for vision-and-language navigation. It combines subtask attention supervision, landmark attention supervision, and VGGT-based spatial feature distillation to improve direct action prediction without intermediate textual reasoning or an online geometric teacher at inference. Guided-R2R and Guided-RxR augment existing navigation data with active subtask spans and landmark masks. Evaluations include R2R-CE, RxR-CE, component ablations, representation analyses, and real-robot experiments.

Critical strengths
  1. The framework coherently addresses instruction progress, landmark grounding, and spatial understanding through training-time supervision. The annotation-enriched datasets are useful resources, and removing the teacher and auxiliary projectors at inference is practically relevant.
  2. The evaluation is extensive. Matched component comparisons support the guidance modules, measured inference efficiency supports deployment benefits, and attention-localization and spatial-probing analyses provide additional evidence of better task grounding and spatial representations.
Critical weaknesses
  1. The novelty appears to lie mainly in the particular supervision position and temporal spatial objectives rather than the general use of attention guidance and geometric alignment. The paper acknowledges related approaches, but the additional value of supervising native pre-action attention, compared with a different supervision position under the same policy and targets, remains less clearly isolated.
  2. The individual attention-guidance variants reveal a trade-off that needs explanation. In Table 3, subtask and landmark guidance increase SR from 59.3 to 62.9 and 62.6, but reduce SPL from 56.1 to 55.2 and 55.8. The OSR-SR gap increases from 4.5 to 9.4 and 9.5 percentage points. The full model improves both SR and SPL, so this concern is specific to the individual attention modules.

Overall, the integrated framework, data resources, and broad empirical evidence constitute a worthwhile contribution. The remaining attribution and behavioral questions temper my enthusiasm, but I lean toward a weak accept.

Questions for authors
  1. Which specific design choices account for the main advance beyond existing attention supervision and geometric alignment? If available, a comparison varying the attention-supervision position while keeping the policy, targets, and training budget fixed would help clarify the contribution.
  2. Do the lower SPL and larger OSR-SR gap for the individual attention variants reflect less efficient trajectories, inaccurate stopping, or both? Trajectory-length statistics and analysis of episodes that enter the success region but fail at termination would help. How does spatial guidance mitigate these effects in the full model?
3 comments

W1 & Q1: Contribution beyond existing supervision methods and the role of the supervision position.

A1:

Thank you for your suggestion, which have been invaluable in helping us refine our work. We conducted three controlled experiments to examine the supervision position, temporal context, and spatial-guidance depth beyond the existing component ablations.

  1. Where should attention guidance be applied?

We compared supervision positions using Subtask (S) + Landmark (L) guidance without Spatial guidance, keeping the policy architecture, targets, training data, initialization, and training budget fixed. First, we jointly moved S and L among current-image end, input-end marker, and native pre-action. All three positions access the complete navigation input, but only pre-action directly supplies the hidden state predicting the first action token. Second, we moved only S to instruction end or before the current image, keeping L at pre-action so that its landmark targets remained accessible under causal attention.

Table A. Attention-supervision positions on R2R-CE val-unseen.

S positionL positionNE (m) ↓OSR (%) ↑SR (%) ↑SPL (%) ↑
Current-image endCurrent-image end4.5468.060.653.7
Input-end markerInput-end marker4.2570.561.157.8
Native pre-actionNative pre-action4.1873.163.756.4
Instruction endNative pre-action4.9266.958.549.8
Before current imageNative pre-action4.8967.358.451.5

Overall, native pre-action supervision achieves the strongest goal-reaching performance among the tested positions, with an SR of 63.7%. It improves SR by 3.1 and 2.6 percentage points over current-image-end and input-end supervision, respectively. Since these positions share complete input access, the results support the benefit of supervising the action-prediction interface. With L fixed at pre-action, placing S at pre-action also outperforms instruction-end and before-current-image supervision in both SR and SPL, reaching 63.7% and 56.4%. Together, these comparisons support native pre-action as an effective supervision position, combining current-image access with direct guidance of the action readout.

  1. Does geometric supervision benefit from temporal context beyond a single frame?

We additionally examined the local temporal window and the historical context used to construct long-horizon teacher targets. For the window comparison, we jointly varied the neighboring frames used for fusion and Local supervision while keeping the long-horizon branch fixed. For the teacher-context comparison, we retained the four-frame local window and the student’s historical observations, removing history only from the long-horizon VGGT teacher.

Table B. Local temporal window and long-horizon VGGT teacher context.

Local window (frames)Long-horizon teacher inputNE ↓OSR ↑SR ↑SPL ↑
4Cur. + his.3.8371.565.459.1
2Cur. + his.4.1668.163.857.4
6Cur. + his.4.0470.664.559.3
8Cur. + his.4.5866.061.953.6
4Cur.3.9169.563.757.8

The four-frame window achieves the best performance. Expanding the window to eight frames reduces SR and SPL by 3.5 and 5.5 points relative to four frames, showing that a larger local window does not necessarily improve navigation. Removing history from the long-horizon teacher reduces SR by 1.7 points, despite preserving the student’s historical inputs. This supports the value of history-conditioned geometric targets beyond single-frame distillation.

  1. At which depth should spatial-feature alignment be applied?

We compared alignment at 25%, 50%, 75%, and 100% of LLM depth, keeping attention supervision, the geometric teacher, and the remaining spatial components fixed.

Table C. Depth of the student LLM representation receiving spatial alignment.

LLM depthDecoder blockNE ↓OSR ↑SR ↑SPL ↑
25%94.3868.861.954.8
50%184.0870.364.257.6
75%273.8371.565.459.1
100%364.2570.763.156.8

Block 27 performs best among the tested layers across all four metrics, exceeding final-block alignment by 2.3 points in both SR and SPL. This supports a relatively deep, non-final representation as the spatial-alignment site in our policy.

We have added the supervision-position and temporal-context comparisons to Tables 4 and 5, and the spatial-alignment-depth comparison to Table 14. We have also added comparisons of attention-supervision depth and the number and grouping of supervised attention heads to Tables 12 and 13, respectively.

W2 & Q2: Path-efficiency and termination trade-offs of individual attention guidance.

A2:

Thank you for this insightful observation. It motivated us to examine the behavioral effects of attention guidance more closely and provides an opportunity to clarify our original motivation for combining semantic and spatial guidance, as well as their complementary roles in navigation.

  1. Trade-offs of individual attention guidance.

    Subtask and landmark guidance improve SR from 59.3% to 62.9% and 62.6%, respectively. These gains are accompanied by trade-offs in path efficiency and termination behavior. First, mean normalized path efficiency among successful episodes, computed as SPL/SR\mathrm{SPL}/\mathrm{SR}, decreases from 94.6% for Base to 87.8% and 89.1%—reductions of approximately 6.8 and 5.5 percentage points. These averages describe each model’s successful-episode set. Second, the proportion of reached episodes that ultimately fail, computed as (OSR−SR)/OSR(\mathrm{OSR}-\mathrm{SR})/\mathrm{OSR}, increases from 7.1% to 13.0% and 13.2%. Episode-level records further clarify the termination behavior. For landmark guidance, 175 of 1,326 episodes that entered the success region ultimately failed. Of these, 173 ended outside the 3 m region, while two remained inside but exhausted the action budget. The dominant observed pattern is therefore reaching the goal vicinity and subsequently leaving it without completing successfully.

    These findings highlight the different behavioral effects of semantic guidance: it improves overall task success, while path efficiency and termination behavior exhibit accompanying trade-offs. This motivates combining semantic and spatial guidance to support both recognition of relevant instructions and landmarks and the movement and stopping decisions needed to complete navigation efficiently.

  2. Complementary gains from spatial guidance.

    The existing component ablation shows that Spatial guidance alone increases SPL from 56.1% to 57.2%. The additional Subtask +Landmark guidance configuration without Spatial guidance achieves 63.7% SR and 56.4% SPL, whereas the full model reaches 65.4% SR and 59.1% SPL, improving these metrics by 1.7 and 2.7 percentage points, respectively. The behavioral diagnostics also show improved efficiency and completion. Among the 1,155 episodes in which both Landmark and the full model reached the success region, unsuccessful completion decreases from 142/1,155 (12.29%) to 92/1,155 (7.97%). On the 980 episodes successfully completed by both models, mean normalized path efficiency increases from 90.71% to 92.12%.

Together, these results support the complementary roles of the three guidance objectives: subtask and landmark supervision improve semantic grounding, while spatial supervision contributes geometric and temporal information for navigation. Their combination achieves a better balance between task completion and path efficiency than individual attention guidance.

AuthorsOpenReview ↗

Follow-up on Additional Ablations and Behavioral Analysis

Dear Reviewer QBXU,

Thank you again for your thoughtful feedback. May we ask whether our additional experiments and analyses have addressed your concerns? Briefly:

  • Design contributions: Controlled ablations support the benefits of native pre-action supervision, history-conditioned geometric targets, and spatial alignment at a deep, non-final layer.
  • Efficiency and stopping: Episode-level analysis identifies both reduced path efficiency and failures after entering the goal region with individual attention guidance. Adding spatial guidance improves path efficiency and successful completion, supporting their complementary roles. These findings have been incorporated into the revised manuscript. We would greatly appreciate any further suggestions or clarification of remaining concerns.

Best regards,

The Authors

Reviewer_RHrDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GuidedNav, a framework for monocular RGB vision-and-language navigation based on training-time representation guidance. It supervises the model’s native attention at the position immediately preceding the first action token, encouraging it to focus on the active instruction span and relevant landmark regions. Spatial guidance aligns current-frame policy representations with history-conditioned VGGT features and supervises temporal fusion residuals using differences between teacher features computed with and without nearby observations. The paper also constructs annotation-enriched versions of R2R and English RxR.

At inference, the geometric teacher and auxiliary projectors are removed, while temporal fusion is retained. Experiments report improved navigation performance on the R2R-CE and RxR-CE val-unseen splits, supported by component ablations, representation analyses, inference measurements, and qualitative real-robot demonstrations.

Critical strengths
  1. The overall improvement is supported by a reasonably controlled internal comparison. GuidedNav improves R2R-CE SR from 59.3 to 65.4 and SPL from 56.1 to 59.1 relative to the Base Model. Appendix A.1 specifies a shared Stage-2 initialization, the same underlying expert data, and matched Stage-3 training schedules. Tables 3, 7, and 8 further examine the contributions of individual components and compare alternative supervision strategies.

  2. The framework combines useful navigation gains with a relatively simple deployment setup. Subtask and landmark supervision target the existing action-predicting readout, while geometric supervision is transferred into the policy without retaining the teacher at inference. Temporal fusion remains active, but the auxiliary projectors and intermediate textual reasoning are not required. The reported results on both navigation benchmarks, together with the annotation pipeline, make this a potentially useful training framework for navigation.

Critical weaknesses
  1. It is unclear which retained policy parameters receive gradients from the local spatial loss. Equations (6)–(7) apply this loss to a residual constructed from visual tokens before the language model and before the learned gate. Appendix A.2 explicitly states that gradients pass through the selected features and that downstream objectives train the gate. However, the Stage-3 description does not specify the LoRA target modules or whether the visual encoder and multimodal projector are updated. If the modules producing these visual tokens are frozen and LoRA is confined to the language model, the local loss would directly update only its auxiliary projector, which is discarded at inference. This leaves an important implementation detail unresolved and makes the reported contribution of local spatial supervision difficult to interpret.

  2. The route-length analysis in Figure 3 needs to be reconciled with Table 3. The four group-wise SR gains read from Figure 3 are approximately 1.5, 0.2, 3.9, and 4.8 percentage points, whereas Table 3 reports an overall gain of 6.1 points. Appendix A.1 states that the same Base Model is used for component ablations and route-length analysis. If both results use the same checkpoints and evaluation episodes, with a common partition and episode-level aggregation, the overall gain must be a weighted average of the group-wise gains and therefore cannot exceed the largest one. If grouping instead depends on each model’s executed path, group membership can differ across models, which would complicate the comparison. The paper does not clearly define the grouping, leaving the relationship between these results unresolved.

Questions for authors
  1. Clarification of the local guidance objective: Which modules are trainable during Stage 3, and which parameters retained at inference receive nonzero gradients specifically from the local spatial loss? Please specify the LoRA target modules and the update settings for the visual encoder and multimodal projector that produce the tokens used in Equation (6).

  2. Clarification of the route-length analysis: How is path length defined in Figure 3, and are the same episodes assigned to each group for both models? Please provide the group sizes and numerical SR values, confirm whether Figure 3 and Table 3 use the same checkpoints and evaluation episodes, and explain how the group-wise results relate to the overall results.

5 comments

W1 & Q1: Trainable modules and gradient flow of local spatial supervision.

A1:

Thank you for identifying this missing implementation detail. Stage-3 LoRA adaptation includes the visual encoder and multimodal mergers, in addition to the LLM. Consequently, the local spatial loss updates visual parameters retained at inference.

  1. We freeze the pretrained weights and train LoRA adapters with rank r=32r=32, α=16\alpha=16, and zero dropout in the following modules:
  • Visual encoder: attention projections (qkv, attn.proj) and MLP layers (linear_fc1, linear_fc2).
  • Multimodal mergers: linear_fc1 and linear_fc2 in the primary merger and separate DeepStack mergers.
  • LLM: attention projections (q_proj, k_proj, v_proj, o_proj) and MLP projections (gate_proj, up_proj, down_proj).

The fusion gate and both auxiliary alignment projectors are also trainable; VGGT remains frozen.

  1. Thank you for correctly observing that the local loss operates before the gate and LLM. Its gradient path is:

    Local loss → local auxiliary projector → pre-gate residual → selected visual tokens → primary-merger LoRA → visual-encoder LoRA.

    Although matching indices are computed without gradients, the selected feature values are gathered from undetached tensors. The local loss therefore updates the visual-encoder and primary-merger adapters, whose updates are retained at inference. The auxiliary local projector is discarded. The gate, LLM adapters, separate DeepStack-merger adapters, and long-horizon projector receive no gradients directly from this loss; they are trained through other objectives.

  2. To verify this gradient path, we used the final checkpoint and three real four-frame training windows. We cleared all gradients and backpropagated only 0.1Llocal0.1\mathcal{L}_{\mathrm{local}}, without an optimizer update. The module-wise gradient L2L_2 norms were:

    ModuleWindow 1Window 2Window 3Retained at inference
    Visual-encoder LoRA0.035390.029820.01755Yes
    Primary multimodal-merger LoRA0.014860.010880.00635Yes
    Local auxiliary projector0.056380.044610.02970No

    All 216 visual-encoder LoRA tensors and 4 primary-merger LoRA tensors received finite, nonzero gradients in every window. Detaching the residual eliminated gradients to both adapter groups while leaving the loss value and auxiliary-projector gradient norm unchanged. This verifies that local supervision reaches inference-retained policy parameters through the residual feature path.

We have clarified the Stage-3 trainable module groups in Section 4.1 and the local-loss gradient path in Appendix A.3.

W2 & Q2: Reconciling the route-length analysis with the overall results.

A2:

Thank you for identifying this ambiguity. We agree that the grouping protocol and evaluation coverage should have been stated explicitly.

  1. Route-length definition and evaluation protocol.

    R2R-CE val-unseen contains 613 reference routes, each associated with three instruction episodes. We define route length as the shortest navigable distance from the reference route’s start position to its goal and divide the routes into four distance-quartile groups, with boundaries of 6.93, 8.43, and 10.43 m. Group assignments are fixed and independent of either model’s executed path. For the original Figure 3, we randomly sampled one instruction episode per route, yielding 613 episodes, distributed as 154/153/153/153 across the groups. Both models used identical sampled episode IDs and group assignments, with the model configurations reported in the manuscript.

  2. Why the aggregate gains differ. Figure 3 evaluates the one-instruction-per-route subset, whereas Table 3 evaluates all 1,839 instruction episodes. Yours weighted-average argument is correct for a common evaluation set: the original group-wise gains aggregate to the gain on the 613-episode subset, rather than the 6.1-point SR gain on the complete benchmark. The original manuscript did not clearly distinguish these evaluation sets.

  3. Revised episode-level analysis.

    We have additionally evaluated both models on the same 1,839-episode set used in Table 3, retaining the fixed route-length boundaries. The results are provided below.

    Table. Route-length analysis on R2R-CE val-unseen. SR/SPL are percentages; Δ denotes percentage-point differences.

    Reference-route lengthRoutesEpisodesBase SR / SPLGuidedNav SR / SPLΔSR / ΔSPL
    ≤6.93 m15446266.23 / 61.7969.26 / 61.49+3.03 / −0.30
    (6.93, 8.43] m15345963.40 / 61.4367.54 / 60.73+4.14 / −0.70
    (8.43, 10.43] m15345961.22 / 60.3069.72 / 63.30+8.50 / +3.00
    >10.43 m15345946.41 / 40.8555.12 / 50.73+8.71 / +9.88
    Overall6131,83959.33 / 56.1065.42 / 59.07+6.09 / +2.96

    The episode-weighted results round to 59.3 → 65.4 SR and 56.1 → 59.1 SPL, matching Table 3. GuidedNav retains larger SR gains in the two longer-route groups, preserving the qualitative observation from the original analysis.

We have updated Figure 3 with SR results on the complete 1,839-episode evaluation set.

AuthorsOpenReview ↗

Follow-up on Our Clarifications and Revised Analysis

Dear Reviewer RHrD,

Thank you again for your feedback. May we ask whether our responses and revisions have addressed your concerns? Briefly:

  • Local spatial supervision: We clarified the trainable modules and verified that the local loss produces nonzero gradients in visual-encoder and primary-merger LoRA adapters retained at inference.
  • Route-length analysis: The original discrepancy arose from different evaluation sets. We updated Figure 3 using all 1,839 episodes with fixed group assignments; the weighted results now match Table 3. We would greatly appreciate any further suggestions or clarification of remaining concerns.

Best regards,

The Authors

Reviewer_Hg5gOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for vision and language navigation (VLN). Their main modeling contribution is through representation learning during training via guiding sub-task attention, landmark grounding and spatial supervision. They use automatic fine-grained annotations as a source of supervision for grounding the instruction into such visual details. They use Qwen3-VL-8B as the base model and fine tune it with these objectives. They improve results after tuning on training and testing on test dataset of each benchmark compared to a selected set of baselines.

Critical strengths

Strengths: 1-The paper proposes a sound approach for VLN in continuous environments following a fine-tuning approach for a backbone vision and language model (VLM). 2- The approach for guiding the attention based on automatic fine-grained annotations and focusing on grounding the original instruction to the visual and spatial details in indeed interesting. 3- The results are compared against both customized older neural models and the VLM based models and show improvements compared to both categories of models.

Critical weaknesses

Weakness:

I have a few major concerns:

  1. the VLM-based models need to use the same backbone for a fair comparison. The proposed model uses Qwen3-VL-8B, is this the same for other VLM-based baselines?
  2. The evaluation should be done cross datasets, for example training on one dataset and testing on another to make sure that the model does not simply overfit on the environment that it is trained on. This can help evaluate the generalization of the fine-tuned model.
  3. It would have been useful to evaluate the automatic annotation quality. That could help to see if the model really learned useful information from training on that or just learned representation shortcuts from noisy annotations. For example, grounding instruction to the landmarks actually can be counter productive (some older research result shows this), since sometimes the landmarks mentioned in the instruction are not visible before the agent moves forward and conduct actions.

While I mentioned the weakness points might be fixable, I doubt if the model will be generalizable with cross dataset evaluation and also I doubt that this is the best model if the same VLM backbone is used for other baselines. So depending on these additional experiments, we can see if the weakness points are really fixable.

Questions for authors

Please see my above weakness points and related questions.

4 comments

W1: Comparison with VLM-based baselines using the same backbone.

A1:

Thank you for highlighting the importance of backbone consistency. We have completed additional experiments using LLaVA-Video-7B-Qwen2 and Qwen2.5-VL-7B, enabling comparisons with StreamVLN and JanusVLN using the same VLM backbone and nominal parameter scale within each group.

Table A. Comparisons on R2R-CE and RxR-CE val-unseen. Δ (%) reports relative changes against the corresponding baseline; positive NE values indicate error reduction. External data quantities follow the reported counting conventions.

MethodVLM backboneR2R NE ↓R2R OSR ↑R2R SR ↑R2R SPL ↑RxR NE ↓RxR SR ↑RxR SPL ↑RxR nDTW ↑External data
StreamVLNLLaVA-Video-7B-Qwen24.9864.256.951.96.2252.94661.9∼26,330K
GuidedNavLLaVA-Video-7B-Qwen24.5966.458.252.35.8554.945.762.74,917K
Δ (%) vs. StreamVLN—7.83.42.30.85.93.8−0.71.3≈−81.3
JanusVLNQwen2.5-VL-7B4.7865.260.556.86.0656.247.562.110,692K
GuidedNavQwen2.5-VL-7B4.1567.962.157.84.8757.850.266.64,720K
Δ (%) vs. JanusVLN—13.24.12.61.819.62.85.77.2−55.9
GuidedNavQwen3-VL-8B3.8371.565.459.14.464.954.369.23,765K

With Qwen2.5-VL-7B, GuidedNav improves SR over JanusVLN by 1.6/1.6 percentage points on R2R/RxR and SPL by 1.0/2.7 points, using 55.9% fewer reported external samples. With LLaVA-Video-7B-Qwen2, GuidedNav improves SR over StreamVLN by 1.3/2.0 percentage points on R2R/RxR, while reducing NE by 0.39/0.37 m. R2R SPL increases by 0.4 points, whereas RxR SPL decreases slightly by 0.3 points. RxR nDTW improves by 0.8 points, indicating closer adherence to the instructed route. GuidedNav uses approximately 81.3% fewer reported external samples in this comparison.

Higher SR on both datasets in both backbone-matched comparisons supports GuidedNav’s effectiveness beyond the choice of a newer or larger VLM. GuidedNav improves all eight navigation metrics over JanusVLN and seven of eight over StreamVLN, with a small trade-off in RxR SPL.

We have added the backbone-matched comparisons to Table 1, with the VLM backbones and nominal parameter scales specified in the table.

W2: Generalization to additional benchmarks and real-world environments.

A2:

Thank you for suggesting broader evaluation. We have extended our evaluation to NavRAG-CE and conducted a quantitative experiment on real-world navigation tasks. These experiments examine transfer to demand-oriented instructions, held-out environments, and physical robot execution without target-dataset or real-world fine-tuning.

  1. Zero-shot generalization to NavRAG-CE

    We directly evaluate GuidedNav on NavRAG-CE in unseen environments, without using NavRAG training data. NavRAG instructions express user demands and require the agent to infer an appropriate destination. This evaluates transfer to a different instruction distribution in environments held out from navigation training.

    Table B. NavRAG-CE val-unseen results. ✓ indicates use of NavRAG training data, including fine-tuning.

    MethodNavRAG training dataNE ↓OSR ↑SR ↑SPL ↑
    InstructNav✗9.8324.117.410.9
    g3D-LF✓8.8531.821.413.5
    NaVid✓9.3529.619.413.9
    Dynam3D✓8.1238.424.718.8
    GA-VLN✓7.8846.422.218.2
    GuidedNav✗7.4753.823.817.8

    Without target-dataset training, GuidedNav achieves 23.8% SR and 17.8% SPL, together with the lowest NE and highest OSR among the listed methods. Compared with GA-VLN, it improves SR by 1.6 percentage points and OSR by 7.4 points, while its SPL is lower by 0.4 points. These results provide evidence of zero-shot transfer ability on unseen instructions and environments.

  2. Quantitative real-world generalization

    We deploy GuidedNav on an AgiBot X2 humanoid robot with a forward-facing RGB camera, with inference performed on an RTX 4090 server, without real-world fine-tuning. We evaluate four indoor tasks: approaching stairs, navigating around lockers to a vending machine, passing sofas and turning toward a blue sofa, and navigating past orange chairs to a coffee machine. For each task and each model, we conduct 50 trials with randomized initial positions and orientations, covering daytime and nighttime conditions within these 50 trials.

    Table C. Real-world navigation success. Each task has 50 trials per model; ΔSR denotes the absolute improvement in percentage points.

    TaskBase ModelGuidedNavΔSR
    1. Stairs64% (32/50)70% (35/50)6
    2. Vending machine46% (23/50)56% (28/50)10
    3. Blue sofa36% (18/50)50% (25/50)14
    4. Coffee machine38% (19/50)48% (24/50)10
    Overall46% (92/200)56% (112/200)10

    GuidedNav improves success on all four tasks, increasing overall SR from 46% to 56%. The largest improvement occurs on the blue-sofa task, which requires passing intermediate landmarks, executing a turn, and stopping at a specifically described target. This pattern is consistent with the intended role of guidance in connecting instruction progress, visual landmarks, and spatial context. Together, these results support the transfer of the guidance-trained policy to real camera observations and physical robot execution.

We have added the NavRAG-CE zero-shot results to Table 6, and the quantitative real-world evaluation to Appendix E.

W3: Automatic annotation quality.

A3:

Thank you for highlighting annotation reliability and landmark visibility. We have conducted a manual quality assessment and a controlled experiment to examine whether the navigation gains depend on meaningful grounding.

  1. Human assessment of annotation quality.

    We randomly sampled 600 navigation steps, with 300 from Guided-R2R and 300 from Guided-RxR, for manual inspection. We checked whether the active subtask matched the agent’s progress and whether the landmark annotation agreed with the current observation. Samples with incorrect subtask assignments, incorrect landmark regions, or spatial supervision assigned to invisible landmarks were considered unsatisfactory. 552/600 samples passed inspection, yielding a qualification rate of 92%. We have added the Human assessment details into the appendix B.

  2. Landmark visibility.

    We agree that instructions may mention landmarks before they become visible. GuidedNav applies landmark supervision only when an accepted current-frame mask is available. Otherwise, the landmark loss is omitted, while the full instruction and action-prediction objective remain available. Thus, the agent can learn to follow instructions before observing the referenced landmark.

  3. Meaningful grounding versus annotation shortcuts.

    We compared the original landmark targets with targets shuffled across unrelated episodes, preserving the supervised steps and matching the mask-area distribution.

    Table A. Landmark-target comparison on R2R-CE val-unseen. SR and SPL are percentages.

    ConfigurationSR ↑SPL ↑
    Base Model59.356.1
    Base + shuffled landmark targets50.845.7
    Base + original landmark targets62.655.8

    Original targets outperform shuffled targets by 11.8 percentage points in SR and 10.1 points in SPL. Shuffled supervision reduces SR and SPL by 8.5 and 10.4 points, respectively, relative to the Base Model. In contrast, original supervision improves SR by 3.3 points, with a 0.3-point decrease in SPL. These results show that incorrect grounding can substantially impair navigation and support the importance of meaningful instruction–image correspondence, rather than auxiliary supervision alone.

We have added the manual annotation-quality assessment to Appendix B.

AuthorsOpenReview ↗

Follow-up on Our Responses and Additional Experiments

Dear Reviewer Hg5g,

Thank you again for your feedback. May we ask whether our rebuttal and additional experiments have addressed your concerns? Briefly:

  • Fair comparison: Backbone-matched experiments show consistent SR improvements on both benchmarks.
  • Generalization: We added zero-shot NavRAG-CE evaluation and real-world experiments without target-domain fine-tuning.
  • Annotation reliability: Manual inspection found a 92% qualification rate; visibility-aware supervision and shuffled-target experiments further support meaningful grounding.

We would greatly appreciate any further suggestions or clarification of remaining concerns.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response by Authors

We sincerely thank all reviewers for their careful reading and constructive feedback. We are grateful for the many valuable suggestions, which have helped us improve the rigor, completeness, and presentation of this work. We appreciate the recognition of our grounding approach (Reviewer Hg5g), controlled comparisons and practical deployment design (Reviewer RHrD), and integrated framework, data resources, and representation analyses (Reviewer QBXU).

Our original motivation is to make end-to-end VLM-based navigation more interpretable: clarifying which instruction segment to follow, which visual regions to attend to, and how to learn 3D spatial information from 2D observations. GuidedNav addresses these questions through training-time subtask, landmark, and spatial guidance while preserving direct action prediction at inference. Our anonymous project page (https://anonymousauthor-s.github.io/GuidedNav/) provides real-world demonstrations, dataset annotation visualizations, model weights, and training and inference code.

In the following responses, we address each concern through additional experiments, analyses, and implementation clarifications, and describe the corresponding manuscript revisions. All revisions are highlighted in light blue in the revised manuscript. We deeply appreciate the reviewers’ time and the opportunity to strengthen our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.