acceptodds
Under review as a conference paper at ICLR 2027

Do Vision-and-Language Navigation Agents Follow Instructions—or Just the Wording?

Abstract

Vision-and-language navigation (VLN) requires agents to follow natural-language route instructions using visual observations. Users can express the same navigation requirements in different words, but aggregate success rates alone do not reveal whether agents consistently solve the same tasks across these descriptions. Our episode-level analysis of seven agents on R2R-CE and RxR-CE reveals a substantial gap beneath nearly unchanged aggregate scores. On each benchmark, results pooled across agents show that more than half of the episodes solved under the original instructions fail under at least one valid rephrasing. Gains on other episodes mask these losses, and for some agents, even capitalization or punctuation changes can reverse the outcome. To examine whether greater instruction diversity can reduce this wording dependence, we fine-tune three agents using only rephrasings of their own training instructions while retaining the existing navigation supervision. The intervention increases reliable success, reduces outcome reversals, and improves performance on the original benchmark instructions. Reversals also decrease on average for rephrasing types held out from fine-tuning. Physical-robot experiments with human rephrasings of ten indoor route instructions reproduce both wording dependence and the intervention's benefits. Without real-world-specific adaptation, the fine-tuned checkpoints increase the fraction of routes completed successfully under all five descriptions from 24.4% to 34.4%, averaged across models. Together, these findings distinguish successful navigation under a particular phrasing from reliable instruction following. VLN evaluation should assess whether success persists across semantically equivalent descriptions, and training should treat instruction diversity as a core consideration. Our code is available https://anonymous.4open.science/r/VLN_follow_instruction-D4D8-F1015here.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-and-language navigation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5dDqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether vision-and-language navigation agents consistently complete the same tasks under semantically equivalent instructions. It evaluates seven agents on two benchmarks using fourteen controlled rephrasing types and reports outcome flip rate and reliable success alongside conventional success rate. It also investigates rephrasing-only fine-tuning of three agents and evaluates the resulting checkpoints on a physical robot with human-written rephrasings.

Critical strengths
  1. A useful diagnostic finding beyond aggregate performance. The paper shows that nearly unchanged success rates can conceal substantial changes in which episodes succeed. The resulting differences in reliable success and model rankings make this a practically relevant evaluation issue.
  2. A well-controlled and reasonably broad evaluation. The agents share fixed, verified rephrasing sets, and the simulation comparisons control the initial state, checkpoint, and inference settings. The combination of rule-based checks, LLM verification, and manual review is an important safeguard for semantic preservation.
  3. Constructive follow-up experiments. Rephrasing-only fine-tuning improves the reported checkpoints without additional navigation supervision, and the human-rephrasing robot study provides useful evidence beyond synthetic simulation inputs. The paper also acknowledges heterogeneous gains and appropriately limits claims about internal semantic understanding.
Critical weaknesses
  1. The training intervention does not isolate the effect of instruction diversity. The manuscript compares released checkpoints with checkpoints receiving additional rephrasing-based fine-tuning, but does not report a matched original-instruction continued-fine-tuning control. Holding navigation supervision fixed is valuable, yet the gains may partly reflect additional optimization or changes in sampling rather than rephrasing diversity itself.
  2. The physical replication does not separate wording effects from execution variability. Repeating each route–phrasing–checkpoint combination three times is useful, but cross-phrasing FR can still include ordinary run-to-run variation. Within-phrasing disagreement and the execution-order/reset protocol are needed to assess how much of the observed variability is specifically attributable to wording.
  3. The repeatability of smaller improvements is unclear. The fine-tuning results are reported without uncertainty estimates or variability across independent training runs. This is particularly relevant for small changes such as NaVILA’s 0.08-point FR reduction on RxR-CE and some held-out-type results.
Questions for authors
  1. Could you compare rephrasing-only fine-tuning with continued fine-tuning on the original instructions, starting from the same checkpoints and matching the underlying navigation examples, sampling frequency, optimizer settings, and number of updates?
  2. Using the existing repeated robot trials, could you report within-phrasing outcome variability and compare it with cross-phrasing FR? Please also clarify execution ordering and resets, and provide uncertainty estimates that preserve route-level clustering.
  3. Do the observed outcome reversals arise because the agents fail to interpret semantically equivalent instructions consistently? For representative cases where the original instruction succeeds but a rephrasing fails, could the authors analyze whether the agent misinterprets the required action, landmark, spatial relation, or stopping condition, or instead fails despite correctly identifying these requirements? Such an analysis, particularly for rephrasing types held out from fine-tuning, would help clarify what the observed reversals reveal about the agents’ semantic generalization.
4 comments

We thank the reviewer for recognizing the diagnostic value of our evaluation and the benefits of the training intervention. Your suggestions have helped us strengthen the comparisons and clarify both the reliability of the gains and the behavior behind outcome reversals.

Q1. Does the improvement come from instruction diversity or simply additional training?

We conducted continued fine-tuning on the original instructions, starting from the same released checkpoints and keeping all training settings identical to rephrasing-only fine-tuning. The results on RxR-CE are shown below; all values are percentages.

AgentCheckpointSR ↑RSR ↑FR ↓
AwareVLNReleased65.3328.2919.34
AwareVLNOriginal-instruction fine-tuning64.8428.7819.49
AwareVLNRephrasing-only fine-tuning68.6831.5617.40
Uni-NaVidReleased58.1427.8816.30
Uni-NaVidOriginal-instruction fine-tuning57.8127.4716.18
Uni-NaVidRephrasing-only fine-tuning59.9330.8315.47
NaVILAReleased52.6623.2215.64
NaVILAOriginal-instruction fine-tuning52.8223.6315.86
NaVILARephrasing-only fine-tuning55.3624.0415.56

As shown in the table, continued training leaves performance essentially unchanged: SR decreases slightly for AwareVLN and Uni-NaVid and increases by only 0.16 percentage points for NaVILA. Rephrasing-only fine-tuning improves SR, RSR, and FR over this control for all three agents.

This comparison also helps explain our design. We start from the well-trained checkpoints selected by the original authors under their training recipes, which typically combine multiple datasets. Our intervention introduces expression diversity while keeping the environments, instruction semantics, and navigation supervision unchanged. Your suggested control strengthens this design by showing that additional training on the original instructions does not reproduce the gains from expression diversity.

Q2. How does within-phrasing variability compare with cross-phrasing FR, and how were execution and uncertainty handled?

Thank you for suggesting this comparison. We calculated within-phrasing disagreement by comparing the three repeated outcomes pairwise for each fixed route, instruction, and checkpoint. The table compares this measure with cross-phrasing FR. We estimated BCa 95% confidence intervals using 50,000 route-level bootstrap resamples, keeping all models, checkpoints, instructions, and repetitions together within each sampled route.

AgentCheckpointWithin-phrasing disagreement (%)BCa 95% CICross-phrasing FR (%)BCa 95% CI
NaVILAReleased10.67[5.33, 16.00]28.33[16.67, 35.83]
NaVILARephrasing-trained8.00[2.67, 16.00]19.17[10.83, 25.00]
AwareVLNReleased9.33[4.00, 17.33]18.33[9.17, 25.00]
AwareVLNRephrasing-trained6.67[2.67, 13.33]11.67[4.17, 20.00]
Uni-NaVidReleased8.00[4.00, 14.67]25.83[20.00, 29.17]
Uni-NaVidRephrasing-trained5.33[1.33, 9.33]21.67[14.17, 26.67]

As shown in the table, within-phrasing disagreement is low and falls below cross-phrasing FR for every agent and checkpoint. Relative to the released checkpoints, fine-tuning reduces both measures across all three agents, indicating improved consistency across repeated executions and alternative expressions.

Each execution begins after a full reset, at the same starting position and orientation. For each route and checkpoint, we execute the reference instruction followed by its four rephrasings as one group, keeping the wording comparisons close in time. We repeat this five-instruction group three times; some repetitions took place on different days. We will include these protocol details and the accompanying statistics in the revision.

Q3. What do representative outcome reversals reveal about instruction understanding and execution, particularly for held-out rephrasing types?

Thank you for highlighting generalization to held-out expressions. We examined a representative episode in which the agent succeeds under the original instruction but fails under two rephrasings from held-out types. The original instruction is: “Walk behind the couch and turn right. Walk forward and wait near the dining table.” With this instruction, the agent passes behind the couch, turns right, and reaches the dining table.

Numbered steps: action-sequence omission. This variant simply prefixes the two sentences with “1.” and “2.”, preserving all action and landmark wording. The agent reaches the correct couch but continues straight after passing behind it, omitting the required right turn and failing to reach the dining table. The behavior diverges at a required action despite unchanged action wording.

Combined action and landmark: incorrect landmark selection. “Walk behind the couch and turn right” becomes “Go behind the sofa and turn right.” The agent instead moves behind a nearby armchair and turns right from that location, taking a different path and failing to reach the dining table. Here, the first divergence is the selected landmark, although “couch” and “sofa” refer to the same object in the task.

These held-out cases highlight two requirements for following semantically equivalent instructions: grounding equivalent landmark expressions in the same object and maintaining the required action sequence across instruction formats. They motivate including semantically equivalent descriptions of the same navigation tasks in training mixtures, so that expression diversity is learned together with consistent grounding and execution.

Q4. How reliable are the fine-tuning gains, particularly the small FR changes and held-out results?

We used 50,000 paired episode-level bootstrap resamples to estimate BCa 95% confidence intervals for performance changes between the evaluated checkpoints, keeping each episode's original instruction and all valid rephrasings together. The table reports intervals for key metrics in percentage points, with Δ defined as fine-tuned minus released.

BenchmarkModelΔSR ↑ΔRSR ↑ΔFR ↓Held-out ΔFR ↓
R2R-CENaVILA[1.31, 2.56][8.86, 11.64][-2.28, -1.63][0.35, 0.60]
AwareVLN[1.47, 2.77][15.99, 19.52][-15.40, -12.60][-15.40, -12.57]
Uni-NaVid[3.37, 5.27][5.44, 7.67][-0.82, -0.56][-1.31, -0.81]
RxR-CENaVILA[1.88, 3.76][0.41, 1.47][-0.31, 0.08][-1.17, -0.31]
AwareVLN[2.45, 4.50][2.37, 4.33][-2.60, -1.40][-2.52, -1.32]
Uni-NaVid[1.14, 2.70][2.13, 4.01][-1.16, -0.58][-1.89, -0.93]

For NaVILA, the interval for the 0.08-point FR decrease on RxR-CE includes zero, and held-out FR on R2R-CE increases slightly. Across all three agents and both benchmarks, the intervals support SR and RSR improvements, showing improved task completion and reliable success alongside variation in FR gains across agents and rephrasing types. We will include the full uncertainty estimates and estimation procedure in the revised appendix.

Thank you for your thoughtful feedback on the training and robot evaluations. Our added matched training control shows that continued training on original instructions does not reproduce the rephrasing gains. We also added robot variability comparisons and uncertainty estimates, providing further support for improved consistency. We would be grateful for your thoughts on these additions when time permits.

Dear Reviewer 5dDq,

As the discussion deadline approaches, we wanted to kindly check whether you have had a chance to look at our response to your comments. For convenience, we summarize the main additions below:

  1. Matched training control (Q1): We continued training on original instructions from the same checkpoints under matched settings. On RxR-CE, this control left performance largely unchanged, whereas rephrasing training improved task success, reliable success across phrasings, and outcome consistency for every evaluated agent.

  2. Robot variability (Q2): Repeated robot trials show less disagreement when the instruction is held fixed than when its wording changes. Fine-tuning reduces both forms of variability. We also provided route-level uncertainty estimates and clarified the full resets, shared starting poses, and execution ordering.

  3. Held-out cases (Q3): We analyzed an episode under held-out rephrasing types. Adding numbered steps led the agent to omit a required turn despite unchanged action wording; an equivalent action-and-landmark rephrasing led it to select an armchair instead of the couch. These cases illustrate failures in maintaining action sequences and grounding equivalent expressions.

  4. Uncertainty estimates (Q4): We added paired confidence intervals that keep each episode and its rephrasings together. These support improved task success and reliable success across agents and benchmarks, while clarifying the uncertainty surrounding the smaller flip-rate changes.

If time permits, we would be grateful for any further thoughts on our response.

Reviewer_22PWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether VLN agents follow route instructions or merely match familiar wording. The authors construct 14 types of meaning-preserving rephrasings (text formatting, navigation expressions, instruction organization, language style) for R2R-CE and RxR-CE, evaluate 7 released agents at episode level, and propose two metrics—flip rate (FR) and reliable success rate (RSR). They find that fewer than half of original successes survive all rephrasings, and even capitalization/punctuation edits can flip outcomes for some agents. They then fine-tune 3 agents with rephrasing-only data (keeping trajectories and supervision fixed), reporting higher RSR, lower FR, and gains on the original benchmark instructions, with partial transfer to held-out rephrasing types.

Critical strengths
  1. Systematic diagnostic setup.

Fourteen rephrasing types across two benchmarks, seven agents, fixed shared instruction sets, and deterministic inference constitute a well-organized evaluation effort.

  1. The paper is clearly stated and easy to follow.
Critical weaknesses
  1. Limited novelty. The observation that VLN agents do not truly understand instructions has been established: [1] showed top models rely almost exclusively on noun tokens and underuse spatial/directional language; [2] diagnosed agent reliance on instruction parts; [3] already used instruction perturbation for attack and adversarial training. The Related Work also omits directly relevant prior work: [4] proposed instruction-fidelity metrics for R2R precisely because goal-completion metrics conflate success with instruction following—arguably the same thesis as this paper's central claim.

  2. Real-robot evidence is not verifiable from the manuscript.

Appendix E provides only a photo of the robot and camera rig (Fig. 5); all ten route layouts are schematic ("layouts are schematic and not to scale"), and no photograph of the real environment, no first-person camera views, and no video/supplementary link are provided. Given the claim of 450 physical runs (3 agents × 2 checkpoints × 10 routes × 5 phrasings × 3 repetitions), the lack of any visual or video evidence makes the "real-world replication" contribution difficult to assess and verify.

[1] Which way is' right'?: Uncovering limitations of Vision-and-Language Navigation Models. (AAMAS 2023)

[2] Diagnosing vision-and-language navigation: What really matters (NAACL 2022)

[3] Adversarial reinforced instruction attacker for robust vision-language navigation (TPAMI 2021)

[4] Stay on the path: Instruction fidelity in vision-and-language navigation (ACL 2019)

Questions for authors
  1. What was the inter-annotator agreement on rephrasing review? What fraction of generated candidates were rejected or manually repaired, and does repair affect naturalness?
  2. Have you measured human performance under the same rephrased instructions as an upper bound?
3 comments

We thank the reviewer for recognizing the systematic evaluation and clear presentation. Your comments on prior VLN research, experimental evidence, and annotation quality help us make the paper's position and its practical implications more explicit. We address each concern below.

Q1. How does the paper advance prior work on language use and instruction fidelity in VLN?

Thank you for highlighting these connections. We will revise the Related Work to explicitly cite all four studies and clarify their relationship to ours, as summarized below.

WorkAgentsSettingAnalysis or interventionQuestion addressed
Hahn et al. (2023)BERT-based VLN agentsNavigation graphsMasking word groups and swapping directionsWhich linguistic cues guide navigation?
Zhu et al. (2022)Recurrent and pretrained Transformer agentsNavigation graphsAblating or replacing linguistic and visual cuesHow do language and vision affect navigation decisions?
Lin et al. (2021)Encoder–decoder navigation agentsNavigation graphsDynamically corrupting key instruction information; adversarial trainingHow can agents navigate despite corrupted instruction information?
Jain et al. (2019)LSTM-based navigation agentsR2R/R4R navigation graphsReference-path fidelity (CLS); extended routes (R4R)Does reaching the goal entail following the described route?
Our workNavigation policies built on general-purpose generative LLMsContinuous environments and a physical robotMeaning-preserving rephrasings; episode-level comparisons and rephrasing-only fine-tuningDoes success on the same task persist across equivalent descriptions, and how can training improve it?

Building on these studies, we test a further requirement for instruction following: whether success on the same task persists across different valid descriptions when all navigation requirements are held fixed. We find that semantically equivalent rephrasings flip individual episodes from success to failure and from failure to success, even for navigation agents built on general-purpose generative LLMs. These gains and losses largely cancel out, leaving aggregate SR nearly unchanged. On R2R-CE, only 46.43% of episodes solved under the original instruction succeed under every valid rephrasing.

This changes how agents compare: a model with higher SR on the original instructions can be less reliable across equivalent rephrasings. We also show that training on equivalent rephrasings improves original-instruction performance and reliable success, using the same navigation supervision. We therefore argue that semantic understanding should be trained and evaluated explicitly: training mixtures should include equivalent descriptions of the same task, and evaluation should check whether the agent succeeds across them.

Q2. Can the real-robot experiments be inspected through videos?

We now provide two anonymized videos showing the scene and instructions used in the paper's case study, from first-person and third-person views: anonymized videos. The original submission did not include the recordings because they contain institutional logos and other details that could reveal the authors' identities. We used schematic layouts to present the routes and their correspondence with the instructions clearly and in a consistent visual style. We will release the complete experimental video collection once author anonymity is no longer required.

Q3. How were rephrasings reviewed, how often were they revised, and how was naturalness checked?

During generation, automated verification rejected 17.10% of candidates. Five PhD-level annotators reviewed the 33,160 evaluation rephrasings across the eleven LLM-generated types, with a total effort of approximately 120 person-hours. Each rephrasing was independently assessed by two annotators, with an observed agreement of 99.81% before adjudication. Any case flagged by either annotator was discussed by all five annotators to resolve the concern and agree on any necessary revisions. In total, 236 rephrasings (0.71%) underwent manual revision. Each revised instruction then received an additional LLM review for semantic consistency with the original instruction and linguistic naturalness, including grammaticality, coherence, and preservation of the intended rephrasing style.

Q4. Have you measured human performance under the same rephrased instructions as an upper bound?

Yes. Three human participants independently followed the same rephrased instructions in navigation trials, and each achieved a 100% success rate.

Your feedback helped us clarify the paper’s positioning and make the experimental evidence more accessible. Our response now includes comparisons with the prior studies you highlighted, annotation-review details, and human navigation results, alongside anonymized videos of the robot case study from first- and third-person views. We hope these additions are helpful and would appreciate hearing your thoughts when you have a chance.

Dear Reviewer 22PW,

With the discussion period drawing to a close, we would like to gently follow up and ask whether you have had a chance to read our response to your questions. We include a brief summary below for ease of reference:

  1. Relationship to prior work (Q1): We clarified how our study differs from the prior work you highlighted on linguistic cues, instruction corruption, and path fidelity. Our focus is whether success on the same task persists across equivalent descriptions: improvements and failures on different episodes can cancel out, concealing wording dependence in aggregate success rates.

  2. Real-robot evidence (Q2): Our response provides anonymized first- and third-person videos showing the physical scene and instructions used in the paper’s case study. These make the illustrated robot behavior directly viewable. We also committed to releasing the complete video collection once author anonymity is no longer required.

  3. Annotation quality (Q3): We detailed independent double review, team adjudication of flagged cases, and rejection and revision statistics. Annotator agreement was high, and manual revisions were infrequent. Revised instructions received an additional review for semantic preservation and naturalness, including grammaticality, coherence, and adherence to the intended style.

  4. Human performance (Q4): We added human navigation trials using the same rephrased instructions. All participants completed their trials successfully, providing the human-performance reference you requested.

We hope these additions are helpful and would appreciate hearing your thoughts when you have a chance.

Reviewer_A3vCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper diagnoses VLN agents' sensitivity to 14 types of meaning-preserving instruction variations at the episode level. Across 7 public checkpoints, average success rate (SR) remains nearly unchanged, but in R2R-CE, only 46.43% of originally successful episodes succeed under all phrasings. The paper reports that fine-tuning with rephrasing improves consistency.

Critical strengths
  • The paper formulates an important problem at the right unit of analysis. Different people describing the same route differently is a real condition faced by language-controlled agents, and the paper measures this using episode-level RSR (success under all phrasings) and FR (outcome flips), rather than average SR. This reveals instability concealed by average SR. In R2R-CE, only 46.43% of originally successful episodes succeed under all phrasings, and AwareVLN has an SR of 63.02% but an RSR of 21.21% (p.5).
  • The diagnosis is controlled and large in scale. The paper filters 14 types of phrasing variations through automated checks, LLM verification, and human review to retain only meaning-preserving variants (p.4), and evaluates 7 recent agents on R2R-CE and RxR-CE across 3,062 episodes, changing only the phrasing under identical initial states and deterministic inference. Thus, differences in outcomes do not arise from initial states or sampling randomness.
Critical weaknesses
  1. It has not been validated whether the diagnostic metrics measure dependence on phrasing. The average FR for meaning-preserving formatting changes (capitalization, punctuation, and whitespace) is at least 77% of the average FR for rephrasings of action, landmark, and spatial-relation expressions across all agents in both datasets. In R2R-CE, for all 7 agents it is at least 93%, and higher for 4 of them (Fig. 2, Appendix Tables 3 and 4). This may be general closed-loop instability, where even slight input changes cause trajectories to diverge and flip outcomes for episodes near the success boundary, but there is no FR baseline under nonlinguistic perturbations to distinguish the two.
  2. The cause and magnitude of the fine-tuning improvements are unclear. Without a control that continues training on the original instructions using the same data and steps, it is unclear whether the RSR gain comes from phrasing diversity or additional training. The real-robot improvement over 10 routes is RSR 24.4%→34.4%, or roughly 1 route per model, with no confidence intervals (p.2, p.8).
Questions for authors
  1. What are the SR, RSR, and FR of a control that continues training on the original instructions with the same budget?
  2. What is the FR under nonlinguistic perturbations such as slight changes in starting pose or image noise? Are the final distances to the goal (navigation error, NE) of episodes with flipped outcomes concentrated near the success threshold (3m)?
4 comments

We thank the reviewer for recognizing the importance of episode-level evaluation and the value of revealing wording dependence hidden by aggregate scores. Your suggestions have helped strengthen the experimental comparisons and clarify the findings. We address each concern below.

Q1. Does the improvement come from instruction diversity or simply additional training?

We conducted continued fine-tuning on the original instructions, keeping all training settings identical to rephrasing-only fine-tuning. The results on RxR-CE are shown below; all values are percentages.

AgentCheckpointSR ↑RSR ↑FR ↓
AwareVLNReleased65.3328.2919.34
AwareVLNOriginal-instruction fine-tuning64.8428.7819.49
AwareVLNRephrasing-only fine-tuning68.6831.5617.40
Uni-NaVidReleased58.1427.8816.30
Uni-NaVidOriginal-instruction fine-tuning57.8127.4716.18
Uni-NaVidRephrasing-only fine-tuning59.9330.8315.47
NaVILAReleased52.6623.2215.64
NaVILAOriginal-instruction fine-tuning52.8223.6315.86
NaVILARephrasing-only fine-tuning55.3624.0415.56

As shown in the table, continued training leaves performance essentially unchanged: SR decreases slightly for AwareVLN and Uni-NaVid and increases by only 0.16 percentage points for NaVILA. Rephrasing-only fine-tuning improves SR, RSR, and FR over this control for all three agents.

This comparison also helps explain our design. We start from the well-trained checkpoints selected by the original authors under their training recipes, which typically combine multiple datasets. Our intervention introduces expression diversity while keeping the environments, instruction semantics, and navigation supervision unchanged. Your suggested control strengthens this design by showing that additional training on the original instructions does not reproduce the gains from expression diversity.

Q2. How does FR under instruction rephrasing compare with nonlinguistic perturbations?

We evaluated all three agents on R2R-CE with perturbations to the starting pose or input images, retaining the original instructions. Starting-pose perturbations use a horizontal radius of 10 cm and yaw changes of ±5°. Image perturbations add Gaussian noise with σ = 0.02 to RGB values in [0,1], followed by clipping.

AgentStarting-pose FR (%)Image-noise FR (%)Rephrasing FR (%)
NaVILA6.588.8614.79
AwareVLN8.5410.5527.05
Uni-NaVid5.606.4712.35

As shown in the table, both tested nonlinguistic perturbations produce substantially lower FR than instruction rephrasing for every agent. Your suggested comparison provides a useful reference for the magnitude of wording sensitivity. Pose and image perturbations alter the physical or visual inputs, while our instruction comparisons keep the task and execution setup fixed and vary only how the same requirements are expressed. Even formatting changes leave these requirements intact, but can change whether an agent completes the task. This gap motivates our focus on instruction understanding: following the same navigation requirements consistently across valid expressions should be an explicit consideration in training and evaluation.

Q3. Are flipped outcomes concentrated near the 3 m success threshold?

We examined the full NE distributions of flipped original–rephrasing pairs on R2R-CE. Across all seven agents and both instruction conditions, 73.73%–82.74% of endpoints are below 2 m or at least 4 m from the goal, whereas only 9.23%–15.79% lie within [2.5, 3.5) m. These distributions show that flipped outcomes are not concentrated near the 3 m success threshold.

The table reports endpoint distributions (%), separately for each instruction condition.

AgentInstruction<2 m2–<2.5 m2.5–<3.5 m3.5–<4 m≥4 m
NaVILAOriginal44.177.0912.592.2333.92
NaVILARephrased34.134.4412.083.4845.87
AwareVLNOriginal40.983.6510.782.8341.76
AwareVLNRephrased40.944.9510.753.5639.80
Uni-NaVidOriginal39.284.7412.094.4239.47
Uni-NaVidRephrased40.014.3912.914.0138.68
StreamVLNOriginal39.244.299.235.3441.91
StreamVLNRephrased42.045.3111.153.1438.36
DualVLNOriginal40.344.8015.794.4734.59
DualVLNRephrased37.466.1814.134.8337.39
JanusVLNOriginal36.696.6813.662.6940.28
JanusVLNRephrased35.238.2014.163.9138.50
NaVidOriginal37.346.3412.765.0538.51
NaVidRephrased35.095.9113.133.5442.34

Q4. How reliable is the real-robot RSR improvement?

The mean RSR gain across the three agents is 10.0 percentage points (95% BCa CI: [5.56, 14.44]), estimated using 50,000 paired bootstrap resamples at the route level. The interval lies entirely above zero, supporting a positive mean improvement after accounting for route-level sampling variability. All three agents show higher RSR, which requires successful completion of a route under all five descriptions. The gain therefore reflects more reliable task completion across human expressions of the same navigation requirements. We will include the confidence interval and estimation procedure in the revised manuscript.

We appreciate your suggestions for strengthening the diagnostic checks. Our added analyses show that rephrasing produces more outcome reversals than the tested nonlinguistic perturbations, and that reversals are not concentrated near the success threshold. We also added the requested training control and real-robot confidence intervals. If you have a chance to revisit our response, we would welcome any further comments.

Thank you for the thorough response. The matched training control, the nonlinguistic-perturbation baseline, and the NE distribution are exactly the checks I asked for, and they address my concerns. I have raised my soundness score to 3 and keep my positive rating.

Comments on the paper

1 comment

Dear Area Chair,

Thank you for your time and effort in coordinating the reviews during this busy review period. To facilitate your assessment, we summarize the reviewers’ feedback and the main outcomes of our rebuttal below.

A3vC and 5dDq highlighted the value of revealing task-level instability hidden by average success rates, while all reviewers recognized the systematic, controlled evaluation. Following their suggestions, we strengthened the evidence in the following areas:

  1. Training gains and reliability (A3vC Q1; 5dDq Q1, Q4). Under identical training settings, using different wordings of the same instructions improved navigation success and consistency more than continuing to train on the original wording. Added confidence intervals also support improvements in completing tasks successfully across the tested phrasings.

  2. Effects of changing instruction wording (A3vC Q2–Q3; 5dDq Q3). Changing an instruction’s wording altered task success more often than the tested small changes in starting pose or image noise. Most final positions in these cases were well away from the threshold used to judge success. Examples also showed agents missing required turns or choosing the wrong landmark after rewording.

  3. Robot reliability and videos (5dDq Q2; A3vC Q4; 22PW Q2). Robot outcomes were more consistent when repeating the same instruction than when changing its wording. Training with varied wording increased the number of routes completed successfully under every tested description, with confidence intervals supporting this improvement. We also clarified execution and reset procedures and provided videos of the case study.

  4. Distinction from prior work (22PW Q1). The cited studies examine which language cues agents use, how they handle incorrect instructions, or whether they follow the described route. We test whether the same task still succeeds when its instructions are expressed differently without changing their meaning.

  5. Instruction quality and human performance (22PW Q3–Q4). Independent annotators checked whether each rephrasing preserved the intended task, with high agreement and infrequent revisions. Flagged cases were jointly reviewed, and revised instructions were checked again for meaning and naturalness. All human participants successfully followed the same rephrased instructions in navigation trials.

Following these additions, A3vC explicitly confirmed that the requested checks addressed their concerns, raised their soundness score, and maintained their positive rating.

We hope the strengthened evidence supports a positive assessment of our submission. Thank you again for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.