acceptodds
Withdrawn from ICLR 2027

EgoPathBench: Evaluating Zero-Shot Egocentric Waypoint Decision-Making in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Zero-shot waypoint navigation requires vision-language models to select, from the current first-person observation, a sequence of spatial actions that is feasible for the agent and reaches the goal, placing joint demands on the integrated spatial intelligence of today's foundation VLMs. Existing spatial-intelligence benchmarks primarily evaluate isolated judgments of relations, directions, or targets and therefore do not directly measure the integrated navigation ability required to combine target recognition, action-consequence assessment, distance estimation, and path planning. To fill this evaluation gap, we introduce EgoPathBench, a dataset and five-task benchmark for first-person waypoint decision-making. Each question presents an egocentric RGB image, a natural-language goal, and numbered visible waypoints; a model returns traversable candidates or an ordered route. Predictions are evaluated for candidate feasibility, adjacent-edge legality, and goal arrival under point-agent or embodied geometry. EgoPathBench contains 31,852 training, 1,345 validation, and 1,111 benchmark questions and retains at least one geometrically verified reference route for every route question. Across nine VLMs, the highest EgoPath Score is only 28.3. The top-ranked model reaches 35.9% success on Point Path, but only 2.9% and 4.0% on Embodied Path and Intent Path, respectively, showing that current models remain limited in forming complete, goal-consistent routes under embodiment constraints. Beyond the evaluation data, we release the corresponding training resource. Fine-tuning Qwen 3.5 4B on the released training split raises its EgoPath Score from 3.9 to 38.9 and improves all four reported evaluations across three external spatial benchmarks, with gains of 1.4–9.6 points.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Spatial Intelligence, Waypoint Decision-Making

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6oc5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EgoPathBench, a dataset focus on first-person waypoint decision-making tasks. It includes 31K training data, 1.3K validation data and 1.1K testing data. From the results, this benchmark is quite hard that the best evaluated model can only get 28.3 and by fine-tuning Qwen3.5-4.5 on it, it boosts the performance from 3.9 to 38.9.

Critical strengths

The use of point traversability and embodied traversability are interesting. The samples inside the EgoPathBench is large. Based on the Tab. 1, the EgoPathBench has strong potential to be used for fine-tuning as it helps the Qwen3.5 to improve the performance from 3.9 to 38.9.

Critical weaknesses
  1. Although the author has mentioned [1] inside related work, it would be better if the author can more explicitly distinguish the difference between this paper and [1] as I found the general idea is very similar.
  2. I'm wondering the single image setting is insufficient for the task of ego-view navigation. In general robot navigation task, it mainly includes exploration, observation updates, memory and replanning. Using a single ego-view image, due to the viewpoint, will distort the object position in further area, which might make the planned path cannot fully reveal the capability of VLMs.
  3. The designed benchmark provide limited new understanding of VLM limitations. Although EgoPathBench is so hard that even the best evaluated VLM gets only 28.3, its success jointly depends on visual grounding, spatial reasoning, and route construction, which the results offer limited insight.

[1] Tim Windecker, Manthan Patel, Moritz Reuss, Richard Schwarzkopf, Cesar Cadena, Rudolf Lioutikov, Marco Hutter and Jonas Frey. "NaviTrace: Evaluating Embodied Navigation of Vision-Language Models"

Questions for authors

Similar as Weaknesses.

Reviewer_6Df1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EgoPathBench, a benchmark for evaluating vision-language models' waypoint decisions from a single first-person RGB image. Models receive numbered candidate waypoints and a natural-language task, then return either a set of traversable candidates or an ordered route. Across five tasks, the benchmark uses scene geometry to evaluate traversability, route-edge legality, and goal arrival under point-agent and embodied constraints.

The experiments show that current VLMs struggle to generate complete, feasible routes under embodiment constraints, even when their initial actions are legal. The authors support the benchmark with human judgments, geometric validation, and diagnostic analyses. They also release training data with Spatial CoT supervision and show that fine-tuning improves performance on EgoPathBench and external spatial benchmarks.

Critical strengths
  1. The waypoint evaluation grounded in scene geometry has clear value. Each displayed waypoint corresponds to a physical location in the scene, and the evaluator checks direct traversability between consecutive waypoints and final goal arrival. Compared with NaviTrace, which also studies path generation from a single image, the main contribution is the explicit treatment of geometric collision constraints for specified robot dimensions, together with success criteria based on edge legality and goal conditions. This allows different routes satisfying the constraints to be accepted and provides explicit geometric criteria for evaluation.
  2. The validation is broad, and the data demonstrates practical training value. Beyond reporting model rankings, the authors check reference-route visibility and legality and provide human judgments, geometric sensitivity analyses, and visual-input dependence controls. Fine-tuned Qwen3.5-4B improves on all five tasks, supporting the usefulness of the resource for learning the evaluated abilities.
Critical weaknesses
  1. The prompts do not clearly specify the endpoint rules or stopping criteria. The prompts in Table S3 specify the robot’s diameter and navigation goal but omit the exact conditions for reaching the goal. Some failures may therefore reflect differences in interpreting the task requirements, rather than deficiencies in path planning alone. In addition, Point Path and Embodied Path differ in how their goal-distance thresholds are determined, which should be considered when comparing their performance.
  2. The handling of stairs and slopes requires clarification. The paper uses a fixed height interval to identify obstacle geometry for its 2D map. However, stairs and slopes can be traversable depending on the robot's mobility capabilities. A height-based approximation may classify some traversable terrain as obstacles if changes in ground elevation are not handled appropriately. Do the authors apply additional processing for these cases?
  3. Geometric feasibility needs validation through physical-robot execution. Geometric feasibility is a central aspect of this work, and the paper explicitly models robot dimensions and obstacle collisions. However, it does not report experiments on physical robots to verify whether these constraints accurately reflect actual traversability. Such tests would help establish the correspondence between the geometric criteria and physical execution.
Questions for authors
  1. Does the benchmark include stairs and slopes, and how does the height-based geometric model determine their traversability?
  2. The dataset focuses on local navigation planning from a single frame, so improvements in local planning after fine-tuning do not necessarily translate into better performance in actual closed-loop navigation. It would be helpful to provide closed-loop experimental results to assess this transfer.
  3. Could you provide physical-robot execution results to test whether geometrically feasible routes are actually traversable?
Reviewer_yTBqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a benchmark that tests whether VLMs can plan a path from a single first-person image. The model sees numbered waypoints on the image and must either pick the walkable ones or give an ordered route to a target. Routes are scored against the 3D scene geometry. Across nine VLMs, the best route success is 35.9% for the point agent and only about 3% for the robot, far below humans. Fine-tuning Qwen3.5-4B base on the released training data improves it a lot on the benchmark and slightly on OOD spatial benchmarks.

Critical strengths
  1. Paper is well written and easy to follow.
  2. The generated 32K training set results in OOD improvement on VSI-Bench. Thus I believe the training set is useful for the research community.
  3. Careful benchmarking: authors test whether results change when the robot size, map resolution or goal distance change slightly; the model ranking stays the same. They also show blind vlm tests (text only) and run small human studies. Many benchmark papers skip these checks.
Critical weaknesses
  1. The paper only compares itself to indoor navigation and general spatial benchmarks. But the autonomous driving community has worked on very similar problems for a few years, and none of it is discusses. Several driving papers already use VLMs to plan paths from camera images, like DriveLM, GPT-Driver, DriveVLM and OmniDrive. There are also driving benchmarks for VLM spatial reasoning (SURDS, NuScenes-SpatialQA, ScenePilot-Bench, DriveSpatial, Ego3D-Bench). They find the same thing this paper finds: models understand the scene OK but are bad at planning, and there is a big gap to humans.

  2. The main claim, scoring a path by whether it is actually feasible instead of how close it is to a reference path, is also not new. NAVSIM (neurips 2024) does exactly this for driving. A trajectory gets zero if it collides or leaves the drivable area, and it checks the full vehicle footprint, much like the 0.6 m robot here.

  3. The 8k token completion budget for frontier models such as claude-opus-4-8, gemini-3.1-propreview, gpt-5.5, grok-4.3-fast, kimi-k2.6, MiniMax-M3, mistralai/mistral-large-3-675b-instruct-2512 is very small. These models require multi-turn reasoning with longer max response length to reach to the correct answer. Therefore, it is not clear of the reported numbers for these models are fair.

  4. Only one open-source model is evaluated with the proposed benchmark.

  5. We don't observe a baseline method along with the benchmark.

Questions for authors

See the weakness section

Note: I'm willing to change my score base on the rebuttal

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.