- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a reference-free metric for assessing the execution quality of VLN trajectories beyond task success. It scores each trajectory along four hand-designed dimensions (body-scene intrusion and clearance, turn regularity, backtracking, and visibility of the upcoming route and instruction cues) computed from reconstructed scene geometry, and aggregates them with a geometric mean. The metric is validated against human ratings on R2R and R2R-CE, used to re-rank existing methods across five VLN settings, and tested as an auxiliary reward.
Critical strengths
-
Coverage. The metric applies to both discrete and continuous VLN settings and to both mesh-based and 3DGS scenes.
-
Diagnostics. Reporting per-dimension scores with localized evidence makes low scores interpretable rather than hiding them in a single number.
Critical weaknesses
-
Motivation. The failures highlighted in Figure 1 are not blind spots of existing evaluation: collisions are already measured in VLN-CE by works the paper cites (Safe-VLN, SAGE-Bench). Motion smoothness, jerk, and clearance are standard metrics in navigation benchmarks such as SocNavBench [1] and Arena-Rosnav [2], neither of which is cited.
-
Cause. Abrupt turns, backtracking, and view mismatch largely stem from how training trajectories are generated, namely shortest-path or oracle supervision with discrete action spaces, rather than from inadequate metrics. These behaviors are more naturally addressed at the data level, as prior ObjectNav work on human demonstrations has done [3, 4], than by introducing a metric for a problem that existing evaluation does not actually have.
-
Confound. In discrete R2R, agents move along nav-graph edges between panoramas, so body intrusion and abrupt turns largely reflect the graph itself rather than the agent, and the human-sized body ignores the agent dimensions each benchmark defines. Scores for the ground-truth reference paths are needed to show that the metric measures agent behavior rather than environment discretization.
-
Contribution. The metric is a collection of hand-crafted heuristics combined by a geometric mean, and its key parameters, such as body size, thresholds, and preview horizons, are deferred to the supplement. The technical contribution is too thin.
-
Validation. Annotators rate trajectories along the metric's own four dimensions, which makes the human alignment partly circular. Moreover, the advantage over nDTW in the continuous setting is marginal.
-
Utility. The paper provides no evidence that a higher score corresponds to better downstream behavior: its rankings diverge from task metrics without showing which ordering is more meaningful, and the reward-shaping gains are negligible.
[1] Biswas, Abhijat, et al. "SocNavBench: A grounded simulation testing framework for evaluating social navigation." ACM Transactions on Human-Robot Interaction (2022).
[2] Kästner, Linh, et al. "Arena-Rosnav 2.0: A development and benchmarking platform for robot navigation in highly dynamic environments." IROS. 2023.
[3] Ramrakhya, Ram, et al. "Habitat-Web: Learning embodied object-search strategies from human demonstrations at scale." CVPR. 2022.
[4] Ramrakhya, Ram, et al. "PIRLNav: Pretraining with imitation and RL finetuning for ObjectNav." CVPR. 2023.
Questions for authors
-
How does the metric score degenerate trajectories, such as an agent that stops immediately or barely moves, which produce no penalized events?
-
Who were the eight annotators, and how were they recruited and instructed?