Do Vision-and-Language Navigation Agents Follow Instructions—or Just the Wording?

Vision-and-language navigation (VLN) requires agents to follow natural-language route instructions using visual observations. Users can express the same navigation requirements in different words, but aggregate success rates alone do not reveal whether agents consistently solve the same tasks across these descriptions. Our episode-level analysis of seven agents on R2R-CE and RxR-CE reveals a substantial gap beneath nearly unchanged aggregate scores. On each benchmark, results pooled across agents show that more than half of the episodes solved under the original instructions fail under at least one valid rephrasing. Gains on other episodes mask these losses, and for some agents, even capitalization or punctuation changes can reverse the outcome. To examine whether greater instruction diversity can reduce this wording dependence, we fine-tune three agents using only rephrasings of their own training instructions while retaining the existing navigation supervision. The intervention increases reliable success, reduces outcome reversals, and improves performance on the original benchmark instructions. Reversals also decrease on average for rephrasing types held out from fine-tuning. Physical-robot experiments with human rephrasings of ten indoor route instructions reproduce both wording dependence and the intervention's benefits. Without real-world-specific adaptation, the fine-tuned checkpoints increase the fraction of routes completed successfully under all five descriptions from 24.4% to 34.4%, averaged across models. Together, these findings distinguish successful navigation under a particular phrasing from reliable instruction following. VLN evaluation should assess whether success persists across semantically equivalent descriptions, and training should treat instruction diversity as a core consideration. Our code is available https://anonymous.4open.science/r/VLN_follow_instruction-D4D8-F1015here.