- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies whether vision-and-language navigation agents consistently complete the same tasks under semantically equivalent instructions. It evaluates seven agents on two benchmarks using fourteen controlled rephrasing types and reports outcome flip rate and reliable success alongside conventional success rate. It also investigates rephrasing-only fine-tuning of three agents and evaluates the resulting checkpoints on a physical robot with human-written rephrasings.
Critical strengths
- A useful diagnostic finding beyond aggregate performance. The paper shows that nearly unchanged success rates can conceal substantial changes in which episodes succeed. The resulting differences in reliable success and model rankings make this a practically relevant evaluation issue.
- A well-controlled and reasonably broad evaluation. The agents share fixed, verified rephrasing sets, and the simulation comparisons control the initial state, checkpoint, and inference settings. The combination of rule-based checks, LLM verification, and manual review is an important safeguard for semantic preservation.
- Constructive follow-up experiments. Rephrasing-only fine-tuning improves the reported checkpoints without additional navigation supervision, and the human-rephrasing robot study provides useful evidence beyond synthetic simulation inputs. The paper also acknowledges heterogeneous gains and appropriately limits claims about internal semantic understanding.
Critical weaknesses
- The training intervention does not isolate the effect of instruction diversity. The manuscript compares released checkpoints with checkpoints receiving additional rephrasing-based fine-tuning, but does not report a matched original-instruction continued-fine-tuning control. Holding navigation supervision fixed is valuable, yet the gains may partly reflect additional optimization or changes in sampling rather than rephrasing diversity itself.
- The physical replication does not separate wording effects from execution variability. Repeating each route–phrasing–checkpoint combination three times is useful, but cross-phrasing FR can still include ordinary run-to-run variation. Within-phrasing disagreement and the execution-order/reset protocol are needed to assess how much of the observed variability is specifically attributable to wording.
- The repeatability of smaller improvements is unclear. The fine-tuning results are reported without uncertainty estimates or variability across independent training runs. This is particularly relevant for small changes such as NaVILA’s 0.08-point FR reduction on RxR-CE and some held-out-type results.
Questions for authors
- Could you compare rephrasing-only fine-tuning with continued fine-tuning on the original instructions, starting from the same checkpoints and matching the underlying navigation examples, sampling frequency, optimizer settings, and number of updates?
- Using the existing repeated robot trials, could you report within-phrasing outcome variability and compare it with cross-phrasing FR? Please also clarify execution ordering and resets, and provide uncertainty estimates that preserve route-level clustering.
- Do the observed outcome reversals arise because the agents fail to interpret semantically equivalent instructions consistently? For representative cases where the original instruction succeeds but a rephrasing fails, could the authors analyze whether the agent misinterprets the required action, landmark, spatial relation, or stopping condition, or instead fails despite correctly identifying these requirements? Such an analysis, particularly for rephrasing types held out from fine-tuning, would help clarify what the observed reversals reveal about the agents’ semantic generalization.
4 comments
Q3. What do representative outcome reversals reveal about instruction understanding and execution, particularly for held-out rephrasing types?
Thank you for highlighting generalization to held-out expressions. We examined a representative episode in which the agent succeeds under the original instruction but fails under two rephrasings from held-out types. The original instruction is: “Walk behind the couch and turn right. Walk forward and wait near the dining table.” With this instruction, the agent passes behind the couch, turns right, and reaches the dining table.
Numbered steps: action-sequence omission. This variant simply prefixes the two sentences with “1.” and “2.”, preserving all action and landmark wording. The agent reaches the correct couch but continues straight after passing behind it, omitting the required right turn and failing to reach the dining table. The behavior diverges at a required action despite unchanged action wording.
Combined action and landmark: incorrect landmark selection. “Walk behind the couch and turn right” becomes “Go behind the sofa and turn right.” The agent instead moves behind a nearby armchair and turns right from that location, taking a different path and failing to reach the dining table. Here, the first divergence is the selected landmark, although “couch” and “sofa” refer to the same object in the task.
These held-out cases highlight two requirements for following semantically equivalent instructions: grounding equivalent landmark expressions in the same object and maintaining the required action sequence across instruction formats. They motivate including semantically equivalent descriptions of the same navigation tasks in training mixtures, so that expression diversity is learned together with consistent grounding and execution.
Q4. How reliable are the fine-tuning gains, particularly the small FR changes and held-out results?
We used 50,000 paired episode-level bootstrap resamples to estimate BCa 95% confidence intervals for performance changes between the evaluated checkpoints, keeping each episode's original instruction and all valid rephrasings together. The table reports intervals for key metrics in percentage points, with Δ defined as fine-tuned minus released.
| Benchmark | Model | ΔSR ↑ | ΔRSR ↑ | ΔFR ↓ | Held-out ΔFR ↓ |
|---|---|---|---|---|---|
| R2R-CE | NaVILA | [1.31, 2.56] | [8.86, 11.64] | [-2.28, -1.63] | [0.35, 0.60] |
| AwareVLN | [1.47, 2.77] | [15.99, 19.52] | [-15.40, -12.60] | [-15.40, -12.57] | |
| Uni-NaVid | [3.37, 5.27] | [5.44, 7.67] | [-0.82, -0.56] | [-1.31, -0.81] | |
| RxR-CE | NaVILA | [1.88, 3.76] | [0.41, 1.47] | [-0.31, 0.08] | [-1.17, -0.31] |
| AwareVLN | [2.45, 4.50] | [2.37, 4.33] | [-2.60, -1.40] | [-2.52, -1.32] | |
| Uni-NaVid | [1.14, 2.70] | [2.13, 4.01] | [-1.16, -0.58] | [-1.89, -0.93] |
For NaVILA, the interval for the 0.08-point FR decrease on RxR-CE includes zero, and held-out FR on R2R-CE increases slightly. Across all three agents and both benchmarks, the intervals support SR and RSR improvements, showing improved task completion and reliable success alongside variation in FR gains across agents and rephrasing types. We will include the full uncertainty estimates and estimation procedure in the revised appendix.
Thank you for your thoughtful feedback on the training and robot evaluations. Our added matched training control shows that continued training on original instructions does not reproduce the rephrasing gains. We also added robot variability comparisons and uncertainty estimates, providing further support for improved consistency. We would be grateful for your thoughts on these additions when time permits.
Dear Reviewer 5dDq,
As the discussion deadline approaches, we wanted to kindly check whether you have had a chance to look at our response to your comments. For convenience, we summarize the main additions below:
-
Matched training control (Q1): We continued training on original instructions from the same checkpoints under matched settings. On RxR-CE, this control left performance largely unchanged, whereas rephrasing training improved task success, reliable success across phrasings, and outcome consistency for every evaluated agent.
-
Robot variability (Q2): Repeated robot trials show less disagreement when the instruction is held fixed than when its wording changes. Fine-tuning reduces both forms of variability. We also provided route-level uncertainty estimates and clarified the full resets, shared starting poses, and execution ordering.
-
Held-out cases (Q3): We analyzed an episode under held-out rephrasing types. Adding numbered steps led the agent to omit a required turn despite unchanged action wording; an equivalent action-and-landmark rephrasing led it to select an armchair instead of the couch. These cases illustrate failures in maintaining action sequences and grounding equivalent expressions.
-
Uncertainty estimates (Q4): We added paired confidence intervals that keep each episode and its rephrasings together. These support improved task success and reliable success across agents and benchmarks, while clarifying the uncertainty surrounding the smaller flip-rate changes.
If time permits, we would be grateful for any further thoughts on our response.
We thank the reviewer for recognizing the diagnostic value of our evaluation and the benefits of the training intervention. Your suggestions have helped us strengthen the comparisons and clarify both the reliability of the gains and the behavior behind outcome reversals.
Q1. Does the improvement come from instruction diversity or simply additional training?
We conducted continued fine-tuning on the original instructions, starting from the same released checkpoints and keeping all training settings identical to rephrasing-only fine-tuning. The results on RxR-CE are shown below; all values are percentages.
As shown in the table, continued training leaves performance essentially unchanged: SR decreases slightly for AwareVLN and Uni-NaVid and increases by only 0.16 percentage points for NaVILA. Rephrasing-only fine-tuning improves SR, RSR, and FR over this control for all three agents.
This comparison also helps explain our design. We start from the well-trained checkpoints selected by the original authors under their training recipes, which typically combine multiple datasets. Our intervention introduces expression diversity while keeping the environments, instruction semantics, and navigation supervision unchanged. Your suggested control strengthens this design by showing that additional training on the original instructions does not reproduce the gains from expression diversity.
Q2. How does within-phrasing variability compare with cross-phrasing FR, and how were execution and uncertainty handled?
Thank you for suggesting this comparison. We calculated within-phrasing disagreement by comparing the three repeated outcomes pairwise for each fixed route, instruction, and checkpoint. The table compares this measure with cross-phrasing FR. We estimated BCa 95% confidence intervals using 50,000 route-level bootstrap resamples, keeping all models, checkpoints, instructions, and repetitions together within each sampled route.
As shown in the table, within-phrasing disagreement is low and falls below cross-phrasing FR for every agent and checkpoint. Relative to the released checkpoints, fine-tuning reduces both measures across all three agents, indicating improved consistency across repeated executions and alternative expressions.
Each execution begins after a full reset, at the same starting position and orientation. For each route and checkpoint, we execute the reference instruction followed by its four rephrasings as one group, keeping the wording comparisons close in time. We repeat this five-instruction group three times; some repetitions took place on different days. We will include these protocol details and the accompanying statistics in the revision.