acceptodds
Under review as a conference paper at ICLR 2027

IntentionNav: A Benchmark for Intent-Driven Object Navigation from Implicit Human Instruction

Abstract

Goal-specified navigation matches a known target description against scene objects. An implicit request can suggest several plausible targets, requiring goal interpretation alongside object search. Average success alone obscures consistency across expressions and execution of inferred goals. We introduce IntentionNav, a benchmark that evaluates expression consistency and navigation under shared goal inputs. It pairs four English expressions of each of 500 intents with the same scene, designated target, and starting pose across 176 indoor scenes. A complementary study uses a revised episode manifest to compare two navigation systems under shared category predictions and correct-category controls. Across three reference backends, 36.6–45.4% of tasks exhibit both successful and unsuccessful outcomes across their expressions, despite similar average success rates. Repeated executions provide a reference for within-expression variability. With identical category predictions, the reference system and MTU3D reach the designated target neighborhood in 60.8% and 34.6% of episodes, and finish there in 18.4% and 15.0%, respectively. Task-level consistency and separate measures of search coverage and completion reveal differences that average success obscures. Code and data are available at https://anonymous.4open.science/r/IntentionNav-D7D1 and https://huggingface.co/datasets/Anonymous260726/IntentionNav, respectively.

Primary area:
datasets and benchmarks
Keywords:
Embodied AI, Object Navigation, Implicit Human Intent, Vision-Language Models, Diagnostic Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_agqwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies navigation in which an agent must infer the target from a human intent that does not name the object, and then locate it. IntentionNav has 500 base tasks and 2,000 episodes. Each task comes with four English expressions (formal, natural, casual, and emotional) that share the same target object, scene, and starting pose. The paper runs one reference navigation system with three VLM backends from the GPT, Qwen, and Gemini families and analyzes performance across expressions and consistency in task completion. It also compares the reference system with MTU3D on a revised set of episodes, under shared predicted categories and ground-truth category controls, to examine search and completion performance.

Critical strengths
  1. The benchmark pairs four implicit expressions with a fixed target object, scene, and starting pose, so differences in performance across expressions can be measured under matched physical task conditions.
  2. Reporting success rates for each expression next to All-four, the fraction of tasks completed successfully under all four expressions, exposes differences in task-level consistency that average success rates alone can hide.
Critical weaknesses
  1. The evidence on consistency across expressions comes from a single navigation architecture. RQ1 compares three VLM backends inside the same reference system. RQ2 includes MTU3D, but it gives MTU3D category inputs and only formal requests, so it does not test the four expressions on a second architecture. The paper also does not evaluate DDN and CogDDN, which combine demands with visual observations during navigation. As a result, there is little evidence on whether the sensitivity to expressions observed here also occurs in existing demand-driven navigation methods.

  2. The main metrics do not adequately capture whether the stated task was completed. Section 3.1 says the agent should issue STOP near the target, but RQ1 computes SR and All-four from the final recorded position and never requires an explicit STOP. The fallback in Appendix B can also pick that position from previously visited locations once the search horizon runs out. So an episode can count toward both the success and the consistency scores without a successful stop within that horizon. The metrics cannot separate a successful stop from proximity reached through the fallback, which limits how well they characterize task completion and its consistency across expressions. The STOP analysis in RQ2 does not resolve this mismatch in RQ1’s main evaluation.

  3. The paper says little about whether the final annotations were checked for semantic consistency and remaining ambiguity. Appendix F reports inconsistencies across expressions and ambiguity before refinement, and says that some tasks were revised afterwards. It does not say how far these problems were resolved in the final evaluation set. That makes it hard to judge how much annotation inconsistency or ambiguity contributes to the performance differences across expressions.

Questions for authors
  1. Do the authors have results for DDN or CogDDN on all four expressions, including success rates for each expression and All-four?

  2. How do the success rates for each expression and All-four in RQ1 change when an explicit STOP is required under the same distance criterion? Can the authors report how often fallback termination occurs and how many successes it accounts for, per expression and backend?

  3. Was the final evaluation set rechecked for semantic consistency across the four expressions and for remaining ambiguity? What fraction of tasks admit more than one valid target, and how are those cases handled?

Reviewer_ZgJmOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

IntentionNav is a diagnostic benchmark for object navigation from implicit human requests. Its first study pairs four expressions of each of 500 intents across 176 scenes, holding the scene, designated target and starting pose fixed, and evaluates three VLM backends within a common navigation stack. A second study uses a revised task manifest to compare two navigators under shared predicted categories and correct-category controls. The evaluation measures expression consistency, category agreement, search coverage, endpoint success, explicit stopping and visual grounding. The main findings are that similar average success can conceal different cross-expression consistency, and that the benefit of correct-category input differs between the two navigation configurations.

Critical strengths
  1. The paper builds a relatively comprehensive evaluation framework for demand-driven navigation. It pairs four expressions of each of 500 intents across 176 scenes, holding the scene, designated target and starting pose fixed within each task. The separate shared-category study compares navigators under matched predicted and correct-category inputs. Together with measurements of expression consistency, category agreement, search coverage, endpoint success, explicit stopping and visual grounding, these protocols examine multiple stages of task execution. The released task specifications, records and evaluation code provide a useful basis for subsequent comparisons and diagnostic work.

  2. The experiments provide useful evidence about instruction robustness and navigation failures under different settings. Gemini and GPT have similar mean SR but differ by 4.0 percentage points in All-four success, with a task-bootstrap 95% interval of [1.60, 6.40] (Appendix C). Repeated execution on 40 tasks gives disagreement rates of 9.17% within an expression and 22.78% across expressions, helping distinguish expression sensitivity from execution variability (Appendix D). Shared-category controls also reveal an 8.0-point difference in the navigators' responses to category correction, with an interval of [-13.0, -2.8] for MTU3D minus reference (Table 15). Separate coverage and STOP analyses identify different failure patterns. These results provide information about task reliability and execution behavior that average success alone misses.

Critical weaknesses
  1. The task follows DDN, and the added evaluation protocols yield limited new understanding. DDN [1] already asks an agent to infer suitable objects from a request that does not name them and navigate to an object satisfying that demand. Section 2 acknowledges this task family and defines intent as the practical goal communicated by a request, but does not establish an operational distinction from demand. The task specification in Section 3.1 retains this core problem, and Intent Matching measures target-category agreement (Eq. 1). The additions are the paired-expression and shared-category evaluation protocols. Their significance therefore rests on the knowledge they produce beyond existing demand-navigation evaluations [1, 2] and related language diagnostics [3]. The paper does not sufficiently develop that case beyond more detailed measurements of robustness and failure patterns.

  2. The failure analysis remains largely descriptive, limiting the depth and practical value of the findings. The paper measures sensitivity to instruction phrasing and differences in search, endpoint success and stopping, but provides limited follow-up analysis of why the tested methods exhibit these failures. The 8.0-point difference in responses to category correction establishes that the navigators use goal information differently, without identifying which aspects of their perception, search or completion behavior account for the difference. Likewise, reaching a target region without STOP identifies a failure pattern but does not distinguish insufficient visual evidence, incorrect target grounding and an inadequate stopping decision. The analysis also offers limited guidance on how to correct these failures. Correct-category inputs and the first-entry oracle expose performance gaps under idealized conditions; they do not establish which feasible change to a tested system would reduce those gaps. The paper reports where outcomes differ but does not develop these observations into sufficiently specific conclusions about what to improve and why. This limits the explanatory depth and practical value of the empirical findings.

Questions for authors
  1. Task definition and substantive insights beyond DDN: What operational distinction, if any, separates intent as defined in Section 2 from demand in DDN? Please clarify whether IntentionNav defines a different task or provides new evaluation protocols for the existing task. Which substantive insight enabled by the paired-expression or shared-category protocols goes beyond prior demand-navigation evaluations? Please support the answer with the task specification and reported results, and explain how this insight changes our understanding or assessment of the tested systems.

  2. Interpretation and practical implications of the failure analysis: What do the reported results establish about why the two navigators respond differently to category correction, or why episodes reach a target region without completing successfully? Which concrete corrective decision is supported by the existing evidence, and which explanations remain hypotheses? Please distinguish idealized correct-category or oracle outcomes from improvements that a system could implement.

Reviewer_3NbSOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces IntentionNav, a benchmark for object navigation from implicit human requests. It contains 500 intents across 176 indoor scenes, each expressed in four language styles with the same target and starting pose. The authors evaluate three VLM backends within a shared navigation pipeline combining goal interpretation, object detection, spatial memory, and waypoint planning. A separate study compares a reference system with MTU3D under shared category predictions and correct-category controls. The evaluation distinguishes target-neighborhood coverage, endpoint success, and explicit stopping, with repeated runs assessing execution variability. The results reveal that similar average success rates can conceal inconsistent outcomes across expressions, while reaching the target neighborhood does not necessarily imply successful completion.

Critical strengths

1、What I find particularly useful in this work is the paired evaluation design. Keeping the scene, target, and starting pose fixed makes consistency across expressions observable at the task level. Different All-four scores despite similar average success rates reveal information hidden by aggregate performance, while repeated executions help distinguish expression-related variation from execution noise.

2、The shared-category comparison is another strong point. Identical category predictions and correct-category controls clarify the relationship between goal specification and execution. Separating coverage, endpoint success, and explicit stopping reveals why reaching a target neighborhood may not lead to completion. The sensitivity and uncertainty analyses further strengthen these diagnostics.

Critical weaknesses

1、IM scores one designated category, while SR/OSR score one designated instance, despite potentially valid alternative targets. Same-category stopping analysis does not address cross-category ambiguity, and Appendix F reports only pre-refinement ambiguity rates. Excluding two ambiguous tasks in RQ2 provides limited reassurance but does not validate RQ1’s cross-expression findings. Reasonable interpretations may therefore be counted as failures, weakening the interpretation of SR and All-four as successful assistance.

2、Main SR evaluates terminal proximity without requiring STOP. At the horizon, the hosted policy selects a previously visited position (Appendix B), but whether it executes a budgeted return or retrospectively scores that historical pose is unclear. Table 16 shows criterion-dependent ordering in RQ2’s predicted-category condition, motivating clearer terminal-pose accounting and an explicit-STOP sensitivity analysis for RQ1.

Questions for authors
  1. What are the post-refinement ambiguity rates and human-review coverage for the final RQ1 release, distinguished from RQ2’s reviewed-v3? Do SR and All-four conclusions persist on a human-validated unambiguous subset or under acceptable-goal sets allowing cross-category alternatives? Please report subset sizes and uncertainty.

  2. Does horizon-based selection trigger an executed, budgeted return to the selected pose, or retrospectively score a historical pose? Do RQ1’s per-style SR, mixed-outcome, and All-four conclusions persist when explicit STOP is required at an actually executed terminal pose, keeping targets and center-distance thresholds unchanged?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.