- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies navigation in which an agent must infer the target from a human intent that does not name the object, and then locate it. IntentionNav has 500 base tasks and 2,000 episodes. Each task comes with four English expressions (formal, natural, casual, and emotional) that share the same target object, scene, and starting pose. The paper runs one reference navigation system with three VLM backends from the GPT, Qwen, and Gemini families and analyzes performance across expressions and consistency in task completion. It also compares the reference system with MTU3D on a revised set of episodes, under shared predicted categories and ground-truth category controls, to examine search and completion performance.
Critical strengths
- The benchmark pairs four implicit expressions with a fixed target object, scene, and starting pose, so differences in performance across expressions can be measured under matched physical task conditions.
- Reporting success rates for each expression next to All-four, the fraction of tasks completed successfully under all four expressions, exposes differences in task-level consistency that average success rates alone can hide.
Critical weaknesses
-
The evidence on consistency across expressions comes from a single navigation architecture. RQ1 compares three VLM backends inside the same reference system. RQ2 includes MTU3D, but it gives MTU3D category inputs and only formal requests, so it does not test the four expressions on a second architecture. The paper also does not evaluate DDN and CogDDN, which combine demands with visual observations during navigation. As a result, there is little evidence on whether the sensitivity to expressions observed here also occurs in existing demand-driven navigation methods.
-
The main metrics do not adequately capture whether the stated task was completed. Section 3.1 says the agent should issue STOP near the target, but RQ1 computes SR and All-four from the final recorded position and never requires an explicit STOP. The fallback in Appendix B can also pick that position from previously visited locations once the search horizon runs out. So an episode can count toward both the success and the consistency scores without a successful stop within that horizon. The metrics cannot separate a successful stop from proximity reached through the fallback, which limits how well they characterize task completion and its consistency across expressions. The STOP analysis in RQ2 does not resolve this mismatch in RQ1’s main evaluation.
-
The paper says little about whether the final annotations were checked for semantic consistency and remaining ambiguity. Appendix F reports inconsistencies across expressions and ambiguity before refinement, and says that some tasks were revised afterwards. It does not say how far these problems were resolved in the final evaluation set. That makes it hard to judge how much annotation inconsistency or ambiguity contributes to the performance differences across expressions.
Questions for authors
-
Do the authors have results for DDN or CogDDN on all four expressions, including success rates for each expression and All-four?
-
How do the success rates for each expression and All-four in RQ1 change when an explicit STOP is required under the same distance criterion? Can the authors report how often fallback termination occurs and how many successes it accounts for, per expression and backend?
-
Was the final evaluation set rechecked for semantic consistency across the four expressions and for remaining ambiguity? What fraction of tasks admit more than one valid target, and how are those cases handled?