- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents SuperNav, a system that lets a general AI model (GPT) control a navigating robot without training the model for navigation. The model makes the high-level decisions which is, it reads the request, looks at camera images, decides where to go, tracks progress and decides when the task is done. Movement is handled by separate tools. The model picks a point in one of four camera images, and a motion system drives the robot there. System has 4 main parts, Agent Loop, Navigation Skills, Tools and Two motion systems.
Critical strengths
Author has done a good job in separating “deciding” from “moving” through image points, it's simple and practical. The same decision system works with different motion systems. Improvement over baselines are very significant.
Critical weaknesses
The harness’s own contribution is not proven. SuperNav differs from the baselines in many ways at once: a stronger AI model, four cameras instead of one, a perfect map, and a longer time budget. It is unclear how much of the gain comes from the proposed harness itself. There are some evaluation concern as well. The agent had full file access and was only instructed not to read the answer files. SuperNav gets 1 hour per episode, while other methods get 500 steps, and SuperNav’s step count is not reported. The HM3Dv2 setup for SuperNav (number of episodes, stopping rule) is not described, yet its result is marked as the best. The ablation tests use 120 episodes, so most of the differences are within noise (about ±8 points). The title says “any task in any scene,” but the tests are indoor navigation only. The real-robot results are demos with no success rates.
Questions for authors
-
Can you compare SuperNav with a simple fixed-rule system that uses the same GPT model, the same four cameras, the same motion planner and the same budget? This would show whether the harness itself causes the gains.
-
How many steps does SuperNav use per episode, and what was the HM3Dv2 setup?
3 comments
Thanks to the authors for addressing the concerns and new experiments.
The new comparison with the InstructNav baseline is helpful and addresses my main concern.
The additional details on HM3D evaluation and step counts also clear up my second question.
I still have some concerns about the small evaluation set, lack of quantitative results on real robots and the broader claims made in the paper. But overall, the response addresses my main concerns and I will keep my original rating of 3 (Weak Acceptance) with same confidence.
Thank you for taking the time to read our response and the new results. We’re glad the InstructNav comparison addressed your main concern and that the HM3D evaluation details clarified your second question.
Response: Controlled Baseline Comparison and HM3Dv2 Evaluation Details
Thank you for these suggestions!
For question1: We added an InstructNav-adapted fixed-sequence baseline using the same GPT-5.6 Terra, four cameras, same motion planner, and time budget on the same 120 OVON episodes. At 0.25 m/1 m, SuperNav achieves 68.33%/73.33% SR and 0.3837/0.4105 SPL, versus 41.67%/54.17% SR and 0.1865/0.2306 SPL for the InstructNav-adapted method, supporting the harness’s contribution beyond these shared components. Details are provided in Table 2 (Section 4.3) and Appendix B.7 of the revised paper.
For question2: SuperNav averages 239.62 steps per HM3D v0.2 episode. We evaluate ObjectNav on the HM3D v0.2 validation split (36 scenes, 1,000 tasks). Success requires the final position’s navmesh geodesic distance to the nearest goal viewpoint to be strictly <0.2 m; we additionally report results at 1 m. The evaluation protocol is detailed in Section 4.3 and Appendix B.5 of the revised paper. We would be happy to provide a more detailed statistics table if needed.