acceptodds
Withdrawn from ICLR 2027

Beyond Success: Assessing Human-Plausible Execution in Vision-and-Language Navigation with HumanPathScore

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-and-language navigation (VLN) benchmarks measure whether an agent reaches its goal, how efficiently, and how closely its route follows a reference. These essential measures do not fully characterize execution. A successful trajectory may still intersect occupied space, turn abruptly or oscillate, undo progress, or lack visual evidence for an upcoming action. We introduce HumanPathScore (HPS), a reference-trajectory-free, scene-evidence-driven metric that assesses a single trajectory along four observable diagnostic dimensions of human-plausible execution: Physical Walkability, Motion Naturalness, Behavioral Coherence, and View for Action. Together, these constructs operationalize human-plausible execution by measuring whether a human-sized body could traverse the route, movement remains controlled, progress is preserved, and scene evidence supports action. Each construct and its evidence remain visible beside an aggregate that excludes task success. A common trajectory-evidence formulation handles graph-based and continuous trajectories and supports calibrated observations from both mesh-based and 3D Gaussian Splatting reconstructions. We validate HPS in discrete and continuous navigation using blinded human video judgments and matched metric comparisons. HPS shows the strongest overall alignment with human preference among the compared metrics. A separate multi-benchmark analysis shows that HPS produces method rankings different from those produced by SR, SPL, and nDTW. These results support HPS as a diagnostic complement to task-performance and reference-fidelity metrics: existing metrics assess whether an agent succeeds, whereas HPS directly assesses how the execution unfolds.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Visual Language Navigation, Metrics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gEcUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a reference-free metric for assessing the execution quality of VLN trajectories beyond task success. It scores each trajectory along four hand-designed dimensions (body-scene intrusion and clearance, turn regularity, backtracking, and visibility of the upcoming route and instruction cues) computed from reconstructed scene geometry, and aggregates them with a geometric mean. The metric is validated against human ratings on R2R and R2R-CE, used to re-rank existing methods across five VLN settings, and tested as an auxiliary reward.

Critical strengths
  1. Coverage. The metric applies to both discrete and continuous VLN settings and to both mesh-based and 3DGS scenes.

  2. Diagnostics. Reporting per-dimension scores with localized evidence makes low scores interpretable rather than hiding them in a single number.

Critical weaknesses
  1. Motivation. The failures highlighted in Figure 1 are not blind spots of existing evaluation: collisions are already measured in VLN-CE by works the paper cites (Safe-VLN, SAGE-Bench). Motion smoothness, jerk, and clearance are standard metrics in navigation benchmarks such as SocNavBench [1] and Arena-Rosnav [2], neither of which is cited.

  2. Cause. Abrupt turns, backtracking, and view mismatch largely stem from how training trajectories are generated, namely shortest-path or oracle supervision with discrete action spaces, rather than from inadequate metrics. These behaviors are more naturally addressed at the data level, as prior ObjectNav work on human demonstrations has done [3, 4], than by introducing a metric for a problem that existing evaluation does not actually have.

  3. Confound. In discrete R2R, agents move along nav-graph edges between panoramas, so body intrusion and abrupt turns largely reflect the graph itself rather than the agent, and the human-sized body ignores the agent dimensions each benchmark defines. Scores for the ground-truth reference paths are needed to show that the metric measures agent behavior rather than environment discretization.

  4. Contribution. The metric is a collection of hand-crafted heuristics combined by a geometric mean, and its key parameters, such as body size, thresholds, and preview horizons, are deferred to the supplement. The technical contribution is too thin.

  5. Validation. Annotators rate trajectories along the metric's own four dimensions, which makes the human alignment partly circular. Moreover, the advantage over nDTW in the continuous setting is marginal.

  6. Utility. The paper provides no evidence that a higher score corresponds to better downstream behavior: its rankings diverge from task metrics without showing which ordering is more meaningful, and the reward-shaping gains are negligible.

[1] Biswas, Abhijat, et al. "SocNavBench: A grounded simulation testing framework for evaluating social navigation." ACM Transactions on Human-Robot Interaction (2022).
[2] Kästner, Linh, et al. "Arena-Rosnav 2.0: A development and benchmarking platform for robot navigation in highly dynamic environments." IROS. 2023.
[3] Ramrakhya, Ram, et al. "Habitat-Web: Learning embodied object-search strategies from human demonstrations at scale." CVPR. 2022.
[4] Ramrakhya, Ram, et al. "PIRLNav: Pretraining with imitation and RL finetuning for ObjectNav." CVPR. 2023.
Questions for authors
  1. How does the metric score degenerate trajectories, such as an agent that stops immediately or barely moves, which produce no penalized events?

  2. Who were the eight annotators, and how were they recruited and instructed?

Reviewer_jYjcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HumanPathScore (HPS), a reference-trajectory-free evaluation metric for assessing human-plausible execution in Vision-and-Language Navigation. Unlike conventional metrics that primarily focus on task success, path efficiency, or reference trajectory fidelity, HPS evaluates navigation behavior along four dimensions: Physical Walkability, Motion Naturalness, Behavioral Coherence, and View for Action. The proposed metric combines nine interpretable diagnostic components using a geometric mean and relies on calibrated scene evidence rather than human reference trajectories. The authors validate HPS through human evaluation involving 771 trajectories, eight annotators, and 257 paired trajectory comparisons. Experiments on discrete and continuous navigation benchmarks show stronger overall alignment with human judgments than the compared evaluation metrics. Additional experiments investigate component contributions, method rankings, and reward shaping.

Critical strengths
  1. The paper addresses an important limitation of current VLN evaluation protocols. Standard metrics such as SR, SPL, and nDTW do not directly characterize physical feasibility, motion regularity, or other properties of the executed navigation process. HPS provides an interpretable assessment of these properties and complements existing task-performance metrics rather than replacing them.

  2. The authors conduct relatively large-scale human evaluation involving both scalar ratings and independent paired preferences. The component ablations and cross-benchmark analyses further support its diagnostic usefulness.

Critical weaknesses
  1. HPS evaluates visual evidence through a standardized observer aligned with the local direction of travel. However, this may differ from the actual camera orientation or observations available to the navigation agent during execution. Consequently, the metric may assess whether relevant evidence is theoretically observable rather than whether it was actually visible during navigation. Furthermore, human annotators are not provided with the complete navigation instructions, which may limit their ability to assess instruction-conditioned visual relevance.

  2. Figure 5 evaluates several VLN approaches, including StreamVLN, Uni-NaVid, and ETP-R1. However, the results are presented only as rankings, without reporting the actual HPS values or the four constituent scores (P, M, B, and V). Furthermore, some of the most recent methods published in 2026 are missing.

Questions for authors
  1. How does the View for Action score differ when using the standardized trajectory observer versus the agent's actual recorded camera orientation? Could the authors also explain how instruction-conditioned visual relevance is validated when human annotators do not receive the complete navigation instructions?

  2. How sensitive are Motion Naturalness and the overall HPS to trajectory sampling frequency, waypoint spacing, and interpolation methods? Would the same physical trajectory receive comparable scores when represented as a discrete graph path and as a continuous trajectory?

  3. How do GroundingDINO false negatives and reconstruction inaccuracies affect the individual scores and final HPS? In particular, what happens when an otherwise plausible trajectory receives a zero View for Action score due to a missed landmark detection? Could the authors provide sensitivity results for the aggregation strategy and relevant scoring thresholds?

  4. Could the authors provide detailed per-method scores and statistical uncertainty estimates? In particular, why do some recent methods, such as ETP-R1, exhibit substantial ranking drops under HPS? Are these differences attributable to genuine execution-quality deficiencies or to variations in action representation, control frequency, and trajectory sampling across different navigation paradigms? It is recommended that some recent studies be included for evaluation to enhance the overall contribution.

I would currently lean toward a borderline reject, but I would be open to increasing the rating if the rebuttal provides clarifications on the questions and concerns.

Reviewer_1KagOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes HumanPathScore, a metric for assessing how plausible a navigation trajectory looks for a human, separately from whether it reaches the goal. It combines physical walkability, motion naturalness, behavioral coherence, and viewing opportunities using a geometric mean. The score uses trajectory information, reconstructed scene geometry, and instruction cues, without requiring a reference trajectory. The authors evaluate it against human ratings on 771 trajectories and separate A/B preferences on R2R and R2R-CE.

Critical strengths
  1. The paper addresses a useful gap in navigation evaluation. Reaching the correct endpoint clearly does not tell us whether the route was physically plausible or whether the movement looked reasonable. Keeping these questions separate makes sense.
  2. Reporting the four components alongside the aggregate makes the metric more useful for diagnosis. Knowing whether a low score comes from clearance, turning, retracing, or visibility is more actionable than receiving a single unexplained number.
Critical weaknesses
  1. I am not very confident, but I think there are some AI hallucinations: There are reference errors that resemble hallucinated citation metadata. The Taioli et al. entry combines the real Mind the Error! title with incorrect coauthors, venue, and pages; the actual paper appeared at IROS 2024, pp. 12993–13000. The ObjectNav Revisited entry also lists incorrect coauthors despite giving the correct arXiv identifier. These are verifiable errors, although I cannot establish whether AI caused them.

[1] Mind the Error! Detection and Localization of Instruction Errors in Vision-and-Language Navigation Francesco Taioli, Stefano Rosa, Alberto Castellini, Lorenzo Natale, Alessio Del Bue, Alessandro Farinelli, Marco Cristani, Yiming Wang

[2] ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, Erik Wijmans

  1. Some important evaluated methods are missing from the bibliography. DualVLN and JanusVLN appear in Figure 5, and DualVLN is central to the discussion of the continuous-navigation results, but neither has a corresponding reference. Please provide the citations and clearly identify the evaluated versions and trajectory sources. This matters for interpreting the method rankings, especially when the comparison depends on the details of the executed trajectories.

  2. My main concern is that the experiments do not establish how much the proposed design adds over a simpler combination of existing signals. Table 2 compares HPS mainly against task metrics, reference-path metrics, and a smoothness metric. Table 3 includes useful geometric proxies, but evaluates them individually. Table 4 removes branches from HPS, which shows that those branches help this particular construction; it does not show that the nine-leaf hierarchy and aggregation are needed. A simple reference-free combination of clearance, turning, and revisitation, using the same scene evidence and development protocol, would be a much more informative baseline. Without this comparison, I find it hard to tell whether the contribution is a substantially better evaluation metric or a useful collection of familiar heuristics. Different method rankings in Figure 5 do not resolve that question.

Questions for authors
  1. How does HPS compare with a simple combination of clearance, turning, and revisitation signals computed from the same evidence? Were such alternatives evaluated using the same separate development annotations?
  2. Among trajectories receiving V2=0V_2=0, what fraction have genuinely absent visual cues according to human annotation? How much do overall alignment and method rankings change if detector misses are corrected or the zero-score rule is relaxed?
  3. On R2R-CE, A/B agreement improves from 75.6% for nDTW to 78.9% for HPS. What is the paired confidence interval for this improvement, accounting for the four judgments per pair and any shared tasks or scenes? Also, does the scalar-rating advantage persist within methods, rather than mainly across methods with different overall execution quality?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.