- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper questions whether proximity to a demonstrated endpoint adequately measures satisfaction of the instruction’s terminal spatial requirements in vision-language navigation. It constructs instruction-conditioned goal regions for R2R and RxR and introduces Goal-Set Success Rate (GSR) and Goal-Set Oracle Success Rate (GOSR). Re-evaluating 17 navigation methods changes success labels and rankings. The paper also proposes NavJev, a vision-language-model-based stopping controller for frozen navigators, reporting an increase in mean GSR from 41.05% to 42.97% across six R2R policies.
Critical strengths
The paper identifies a valid problem with endpoint-based evaluation: a nearby position can violate the instruction, while a more distant position can satisfy it. This observation exposes a limitation that cannot generally be resolved by adjusting the distance threshold and motivates examining the semantic requirements of termination.
Critical weaknesses
- The construction of the evaluation target is insufficiently established to support the proposed metrics. GSR and GOSR are straightforward membership statistics; their validity rests on how the goal sets are constructed. Although Appendix A describes geometric operations, important semantic decisions remain difficult to reproduce, including the parsing prompts, resolution of ambiguous referents, and treatment of multiple valid interpretations. The nominal definition 𝐺(𝐼,𝐸) also conceals dependencies on the demonstrated endpoint and reference route, which determine the search area, entry opening, and approach direction. The paper does not establish whether these choices exclude valid alternatives. Fixed geometric interpretations, such as treating both "near" and "beside" through a 3,m threshold, are not accompanied by quantitative robustness analysis. The independent audit covers only R2R, leaving the multilingual RxR annotations without comparable verification. Consequently, the ranking changes do not sufficiently establish that the proposed criterion is a generally more faithful evaluation standard. 2, NavJev is neither specified sufficiently for reproduction nor evaluated in a way that establishes its claimed contribution. The model and main optimization settings are given, but "terminal stage" is not operationally defined, and the construction of hard-positive and hard-negative states and their corresponding visual histories is unclear. The complete completion prompt and backbone-specific STOP-masking interfaces are also missing. More fundamentally, the experiments introduce a 4B vision-language model, goal-set supervision, hard-example sampling, and forced continuation together, then compare the complete system only with native stopping. Without an otherwise identical controller trained on endpoint-based labels, the gains cannot be attributed to the proposed supervision. Changes in aggregate GSR and GOSR also cannot establish whether improvements arise from correcting termination decisions after the trajectory has already reached a valid goal region, or from the effect of suppressing the navigator’s native STOP action and allowing the frozen policy to continue executing additional actions before termination. These omissions leave both the method's reproducibility and the explanation for its improvement unresolved.
Questions for authors
- How are ambiguous referents and multiple valid interpretations converted into a final goal set? When does reference-route context merely resolve ambiguity, and when does it exclude alternative stopping regions?
- Do the principal ranking reversals persist under reasonable alternative spatial thresholds? What independent evidence supports the semantic accuracy of the multilingual RxR goal sets?
- What precisely defines a terminal-stage training state? How are hard-positive and hard-negative positions sampled, and how are their visual histories constructed from the demonstration data?
- How does NavJev compare with an otherwise identical controller trained on endpoint-based labels? Among its recovered successes, how many had already reached the goal set in the original run, and how many reached it only after the original STOP was suppressed?
4 comments
Part II. Additional results on alternative spatial thresholds and multilingual semantic accuracy
Alternative spatial threshold? (W1, Q2a)
We added a 1m comparison: NaVIDA and StreamVLN remain the leaders under both settings, while some intermediate ranks change.** We rescore the same six policies and all 1,839 R2R val_unseen episodes, holding trajectories, referents, and other spatial constraints fixed. GSR measures explicit STOP inside the goal set; GOSR measures whether the trajectory ever enters it.
| Method | GSR@3 m (%) | GSR@1 m (%) | GOSR@3 m (%) | GOSR@1 m (%) |
|---|---|---|---|---|
| NaVIDA | 50.35 | 36.70 | 57.15 | 49.92 |
| StreamVLN | 46.82 | 32.57 | 52.47 | 43.83 |
| ETP-R1 | 44.05 | 28.38 | 49.92 | 38.88 |
| Uni-NaVid | 43.01 | 29.58 | 47.09 | 38.66 |
| AwareVLN | 42.52 | 29.74 | 50.79 | 41.11 |
| NaVILA | 39.37 | 27.62 | 45.51 | 36.32 |
NaVIDA and StreamVLN remain ahead of ETP-R1 and AwareVLN, and Uni-NaVid remains ahead of NaVILA under both GSR settings. ETP-R1 and AwareVLN exchange third and fifth place. GSR falls by 11.75–15.67 percentage points and GOSR by 7.23–11.04 points, quantifying sensitivity in absolute scores and some relative orderings. These results and region comparisons are included in Appendix B.4.
multilingual semantic accuracy? (W1, Q2b)
We added a blinded audit of 500 RxR positions; goal sets reduce errors by 15–20 percentage points relative to the 3 m endpoint rule. The sample comprises 100 actual policy stops per language and 200 diagnostic positions across region interiors, boundaries, and exteriors. Two annotators judge independently, with a third resolving disagreements; masks, policy names, automatic decisions, and highlighted reference endpoints are hidden. Instructions are read in their original language, with bilingual adjudication for Hindi and Telugu.
| RxR sample | Positions | Endpoint errors | Goal-set errors | Goal-set precision / recall (%) | Error reduction (pp) |
|---|---|---|---|---|---|
| English policy stops | 100 | 24 | 8 | 95.6 / 92.9 | 16.0 |
| Hindi policy stops | 100 | 25 | 10 | 93.9 / 91.2 | 15.0 |
| Telugu policy stops | 100 | 24 | 9 | 94.1 / 92.8 | 15.0 |
| Diagnostic positions | 200 | 62 | 22 | 89.0 / 89.0 | 20.0 |
Within each language, both criteria are compared against human labels at the same positions. Policy-stop precision of 93.9–95.6% and recall of 91.2–92.9% provide direct evidence of semantic accuracy. The protocol and results are included in Appendix A.4 and Table 5.
Part III. Additional details and ablations on NavJev
How are training states, histories, and STOP interfaces specified? (W2, Q3)
We added the sampling rules, complete completion prompt, and shared control procedure in Appendix.
- States and hard examples. The terminal stage starts at the demonstration's first goal-set entry, or comprises its final steps if it never enters; the sampler records and the trigger step. Hard positives lie inside the goal set and outside the 3 m endpoint neighborhood; hard negatives satisfy the reverse membership. Both are sampled with ordinary examples from the same suffix state pool.
- Visual history. Each state receives the instruction and 16 uniformly sampled frames from the trajectory start through that state, repeating frames for short prefixes. Training uses the corresponding demonstration prefix; deployment uses the executed prefix.
- Score and control. Qwen3-VL-4B-Instruct predicts
stopornot stop, with score from first-token probabilities. Queries begin after 20 low-level actions; two consecutive scores of at least 0.5 trigger STOP. Full NavJev masks native STOP throughout the episode and requests a movement at the same step whenever continuation is required. All six backbones share the checkpoint and control rule.
Training uses 168,726 R2R decisions: 32,290 stop and 136,436 not stop. The checkpoint after one training epoch serves every policy. These details, the full prompt, and the deployment procedure appear in Section 5.1 and Appendices C.1–C.3.
Can the gains be attributed to goal-set supervision? (W2, Q4)
We added a crossed comparison of supervision and native STOP handling: goal-set labels outperform endpoint labels under both STOP rules. Controllers share the 4B model, training states including hard examples, instructions, and 16-frame histories; only labels differ. All configurations use the same 1,839 episodes and action budget. “Honored” lets either navigator or controller stop; “Masked” gives the controller stopping authority. GSR (%) follows.
| Supervision | Native STOP | ETP-R1 | AwareVLN | StreamVLN |
|---|---|---|---|---|
| Native baseline | Honored | 44.05 | 42.52 | 46.82 |
| Endpoint | Honored | 44.10 | 42.63 | 46.87 |
| Goal set | Honored | 45.13 | 43.88 | 47.15 |
| Endpoint | Masked | 44.21 | 42.74 | 46.98 |
| Goal set | Masked | 46.17 | 45.19 | 47.42 |
With masking fixed, goal-set labels add 1.96, 2.45, and 0.44 points, respectively. With goal-set labels fixed, masking adds 1.04, 1.31, and 0.27 points. Every endpoint-supervised configuration stays within 0.25 points of the native baseline. This positively shows that the endpoint region is an inappropriate stop-supervision target: under matched DAgger-style training (same model, visited states, and demonstration prefixes, differing only in the label), DAgger expands the visited-state distribution, but if the STOP label is still defined by the reference endpoint, more states do not redefine which positions satisfy the instruction's terminal condition, so endpoint supervision barely changes the stop decision; goal-set supervision instead materially improves the actual stop decision, confirming the need for stop supervision. The six-policy mean of 41.05%→42.97% measures the full system and this additional experiment appears in Section 5.2 and Table 3.
How many recoveries follow prior arrival or continued execution? (W2, Q4) We added an episode-paired analysis: most new successes for each policy had already visited the goal set in the native run, and continuation produces additional arrivals. For first arrivals after STOP suppression, we verify that both runs share the action history through the native stopping state.
| Method | New successes | Goal set visited in native run | First reached after STOP masking | Successes lost |
|---|---|---|---|---|
| ETP-R1 | 51 | 33 | 18 | 12 |
| AwareVLN | 63 | 53 | 10 | 14 |
| StreamVLN | 24 | 22 | 2 | 13 |
After subtracting successes lost, the three policies net 39, 49, and 11 additional successful episodes. Most new successes are cases where the native trajectory had already reached the goal set but stopped at the wrong position, so the gain mainly reflects more accurate stopping; continued execution after STOP masking adds further arrivals. This is consistent with Q5: goal-set stop supervision improves the actual stop decision, and continuation expands arrival. Results are included in Appendix C.4 and Table 10.
Sincerely looking forward to your more feedbacks
Dear Reviewer kyez,
We sincerely appreciate your efforts put into the reviewing process, which has significantly contributed to the refinement of our manuscript. We've carefully examined all the constructive questions and have accordingly revised our work. To make the best use of the discussion period and to improve our work, we are eager to know whether our answers adequately address your concerns, as it is crucial for us to have a candid and thorough discussion to continuously strengthen our method. Please share your thoughts on viewing our reply. We hope to resolve your doubts with our best efforts.
We are ready to respond to any further issues raised. Please let us know.
Sincerely,
The Authors
Part I. Details on goal-set construction
Thank you for recognizing the limitation of endpoint-based evaluation and for asking how we turn terminal semantics into a reproducible target.
We specify parsing, grounding, and geometric construction, explicitly including the demonstrated endpoint and route in G(I,E,Dref). The endpoint localizes candidate entities; the route supplies entry and approach context (Section 3.2, Eq. 1).
(a) Parsing inputs, outputs, and validation (Appendix A.1). Qwen3.8-Flash receives a fixed prompt and JSON containing only the case identifier and full instruction. Each case in the output
resultscontainsentities(IDs, object/room categories, primary/reference roles, and attributes),constraints(operators, entity IDs,stop/identityscope, and supportingsource_text), and the verbatimterminal_textandorientation_text. For example, “enter the bedroom and stop beside the chair” yields room-membership and chair-proximity conditions. Validation checks JSON parsing, a nonempty terminal clause, and complete entity references. Decoding uses temperature 0, extended thinking disabled, and an 8,192-token cap with prompt shown in the appendix A.1.(b) Resolving ambiguity and selecting interpretations (Appendices A.1 and A.3). Candidates are searched on the endpoint's floor, in its source room and rooms connected through adjacent openings: the Matterport3D inventory supplies candidate objects/rooms, overhead views fix connectivity and extent, and RGB-D panoramas and object crops resolve appearance and size, checking categories and attributes, inter-entity relations, spatial extent, and floor.
Selection follows “identity first, then region, effective only on confirmation.” A descriptive relation in the instruction first fixes entity identity (“the chair beside a fireplace” identifies the target through that relation, which serves identity only); the stopping region is then compiled around the confirmed entity, with the side/orientation frame set by the terminal predicate type (later in d). Competing readings are kept as separate hypotheses and compiled into separate candidate regions, never merged prematurely. A reading becomes the effective region only when the instruction and scene evidence jointly confirm it (confirm-only): the reference endpoint/route supplies search context and frame only. Within a confirmed reading all terminal predicates are intersected, and an empty region triggers inspection; any entity, relation, or orientation that cannot be uniquely resolved is disclosed together with all competing candidates and sent to human review. All 12,845 validation cases are additionally reviewed individually against the instruction, route, and scene views.
(c) Coordinate alignment and reachable regions (Appendix A.2). Matterport3D house coordinates are transformed into Habitat coordinates:
(xH,yH,zH)=(xM,zM,−yM).
Here, yH is vertical and xH–zH is horizontal; objects, routes, regions, and stops share this convention. A 0.1 m grid represents reachable surfaces, retaining height to distinguish overlapping floors. Object proximity propagates from cells with an unobstructed connection to the surface, using the surface gap plus the shortest traversable grid path and retaining positions within 3 m that satisfy the other terminal constraints.
(d) Directional projection and route dependence (Section 3.2; Appendix A.2). Taking the object center as the local origin, we project the candidate stop's horizontal offset onto a forward axis and its perpendicular leftward axis. The projection signs select the front/back and left/right half-planes. The source of the forward axis determines route dependence:
Among side predicates with recorded frames, 60.8% on R2R and 47.9% on RxR use approach directions; the remainder use object axes. Intersecting these conditions with reachability, room membership, and proximity gives the goal set for that reference context.