- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DemandNav, a navigation task that requires agents to infer functional requirements and their prerequisite relations from high-level human demands, then ground these requirements to objects in an unfamiliar environment. The proposed DemandAgent combines MCP-based navigation tools, an Adaptive Demand Ledger, spatial memory, and procedural skills within a closed-loop agent framework. The authors construct Demand-Bench, containing 200 episodes across distinct ProcTHOR scenes, with an evaluator supporting alternative object assignments and dependency-consistent completion orders. DemandAgent achieves 52.5% success on Demand-Bench, while an adapted version achieves 65.7% on EB-Navigation. Component ablations further examine the contributions of task-state management, memory, and skill guidance.
Critical strengths
-
The task formulation addresses a meaningful aspect of embodied assistance. Everyday requests often leave the required objects, intermediate steps, and execution order implicit. Explicitly modeling functional requirements and their prerequisites provides a useful way to study the connection between demand understanding and embodied navigation. This motivation is relevant to developing agents that can act on less prescriptive human instructions.
-
The benchmark incorporates useful flexibility and validation. Demand-Bench supports multiple admissible object assignments and valid execution orders within each reference dependency graph. Runtime feasibility checks and manual annotation review strengthen its construction. The distinction between requirement coverage and dependency-consistent progress also provides more informative evaluation than reporting full-task success alone.
-
The experiments provide encouraging evidence for the overall framework. DemandAgent improves success from 31.5% for Vanilla Codex to 52.5% under a shared backbone and navigation interface. Removing the ledger, spatial memory, or skill guidance reduces performance, supporting the utility of these components. The improvement from 59.3% to 65.7% on EB-Navigation provides additional evidence that the framework can be adapted beyond the proposed benchmark.
Critical weaknesses
-
The framework is evaluated with only one backbone model. All DemandAgent evaluations, including its component ablations and EB-Navigation adaptation, use GPT-5.6-Luna with maximum reasoning effort. No corresponding framework evaluations are reported with alternative backbones such as Qwen3.8-Max, Gemini-3.5-Flash, or DeepSeek-V4-Pro. Consequently, it remains unclear whether the benefits of the ledger, memory, and skills generalize across models with different reasoning and tool-use capabilities.
-
The evaluation captures a restricted approximation of everyday demand fulfillment. Tasks involving cooking, cleaning, or storage are evaluated through navigation and visual verification, without requiring the corresponding physical manipulation or object-state changes. Moreover, the point-navigation executor uses privileged simulator geometry to construct collision-free motion. Although this scope is explicitly acknowledged and reasonable as an initial step, the experiments provide limited evidence that the inferred dependencies and execution strategy remain effective when actual manipulation and navigation uncertainty are introduced.
-
The specific benefit of adaptive replanning is not consistently demonstrated. Disabling replanning reduces overall success only from 52.5% to 51.5%. More notably, Table 11 shows that the variant without replanning achieves higher success on the 108 episodes with prerequisite relations: 38.9%, compared with 27.8% for the full system. These results support maintaining explicit task state, but leave the additional value of online plan revision less clear. Repeated evaluations and a targeted analysis of beneficial versus harmful revisions would strengthen this part of the contribution.
Questions for authors
1, Do the improvements persist when DemandAgent uses another backbone? Can the authors compare the full framework against the corresponding base agent for each model under matched observation, tool, and execution-budget settings?
2, How would the framework and its completion criteria extend to demands requiring actual object-state changes? Can the authors clarify which conclusions depend on the navigation-only abstraction and simulator-assisted motion execution?
3, Why does disabling replanning improve performance on tasks with prerequisite relations? Can the authors quantify how often replanning removes an unnecessary requirement versus incorrectly retires a necessary one, and assess whether the observed differences are stable across repeated runs?