acceptodds
Under review as a conference paper at ICLR 2027

DemandAgent: Towards an Embodied Agent Harness for Everyday Human Demands

Abstract

Real-world embodied assistance often begins with a high-level human demand rather than a fully specified task. Existing embodied navigation settings typically ask agents either to reach a specified object goal or to navigate by following detailed route instructions. We argue that satisfying real-world human demands goes beyond localizing relevant targets and instead requires transforming high-level demands into executable embodied behaviors while considering inherent task dependencies. We introduce DEMANDNAV, a dependency-aware task formulation that represents a human demand as functional requirements and their prerequisite relations. To address this task, we present DEMANDAGENT, an embodied agent harness built around dedicated MCP tools for embodied interaction and persistent task-state and spatial-state management, guided by specialized embodied agent skills. We further introduce a top-down data construction pipeline and instantiate DEMAND-BENCH with 200 episodes from distinct ProcTHOR scenes. On DEMAND-BENCH, DEMANDAGENT achieves a 52.5% success rate, substantially outperforming adapted navigation baselines with static LLM-based demand decomposition and goal translation. Further evaluation on the public EB-Navigation benchmark shows that an adapted DEMANDAGENT achieves the best performance among the evaluated baselines, with a 65.7% average success rate, demonstrating the cross-benchmark adaptability of the proposed embodied agent harness. Code and data will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Embodied Agents, Agent Harness, Agentic Systems, Human Demand Realization, Embodied Navigation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zVN6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DemandNav, a navigation task that requires agents to infer functional requirements and their prerequisite relations from high-level human demands, then ground these requirements to objects in an unfamiliar environment. The proposed DemandAgent combines MCP-based navigation tools, an Adaptive Demand Ledger, spatial memory, and procedural skills within a closed-loop agent framework. The authors construct Demand-Bench, containing 200 episodes across distinct ProcTHOR scenes, with an evaluator supporting alternative object assignments and dependency-consistent completion orders. DemandAgent achieves 52.5% success on Demand-Bench, while an adapted version achieves 65.7% on EB-Navigation. Component ablations further examine the contributions of task-state management, memory, and skill guidance.

Critical strengths
  • The task formulation addresses a meaningful aspect of embodied assistance. Everyday requests often leave the required objects, intermediate steps, and execution order implicit. Explicitly modeling functional requirements and their prerequisites provides a useful way to study the connection between demand understanding and embodied navigation. This motivation is relevant to developing agents that can act on less prescriptive human instructions.

  • The benchmark incorporates useful flexibility and validation. Demand-Bench supports multiple admissible object assignments and valid execution orders within each reference dependency graph. Runtime feasibility checks and manual annotation review strengthen its construction. The distinction between requirement coverage and dependency-consistent progress also provides more informative evaluation than reporting full-task success alone.

  • The experiments provide encouraging evidence for the overall framework. DemandAgent improves success from 31.5% for Vanilla Codex to 52.5% under a shared backbone and navigation interface. Removing the ledger, spatial memory, or skill guidance reduces performance, supporting the utility of these components. The improvement from 59.3% to 65.7% on EB-Navigation provides additional evidence that the framework can be adapted beyond the proposed benchmark.

Critical weaknesses
  • The framework is evaluated with only one backbone model. All DemandAgent evaluations, including its component ablations and EB-Navigation adaptation, use GPT-5.6-Luna with maximum reasoning effort. No corresponding framework evaluations are reported with alternative backbones such as Qwen3.8-Max, Gemini-3.5-Flash, or DeepSeek-V4-Pro. Consequently, it remains unclear whether the benefits of the ledger, memory, and skills generalize across models with different reasoning and tool-use capabilities.

  • The evaluation captures a restricted approximation of everyday demand fulfillment. Tasks involving cooking, cleaning, or storage are evaluated through navigation and visual verification, without requiring the corresponding physical manipulation or object-state changes. Moreover, the point-navigation executor uses privileged simulator geometry to construct collision-free motion. Although this scope is explicitly acknowledged and reasonable as an initial step, the experiments provide limited evidence that the inferred dependencies and execution strategy remain effective when actual manipulation and navigation uncertainty are introduced.

  • The specific benefit of adaptive replanning is not consistently demonstrated. Disabling replanning reduces overall success only from 52.5% to 51.5%. More notably, Table 11 shows that the variant without replanning achieves higher success on the 108 episodes with prerequisite relations: 38.9%, compared with 27.8% for the full system. These results support maintaining explicit task state, but leave the additional value of online plan revision less clear. Repeated evaluations and a targeted analysis of beneficial versus harmful revisions would strengthen this part of the contribution.

Questions for authors

1, Do the improvements persist when DemandAgent uses another backbone? Can the authors compare the full framework against the corresponding base agent for each model under matched observation, tool, and execution-budget settings?

2, How would the framework and its completion criteria extend to demands requiring actual object-state changes? Can the authors clarify which conclusions depend on the navigation-only abstraction and simulator-assisted motion execution?

3, Why does disabling replanning improve performance on tasks with prerequisite relations? Can the authors quantify how often replanning removes an unnecessary requirement versus incorrectly retires a necessary one, and assess whether the observed differences are stable across repeated runs?

Reviewer_4EbrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper transforms high-level everyday demands into dependency-aware functional-requirement DAGs, builds a closed-loop embodied agent with ADL, Spatial Memory, MCP tools, and skill guidance, and releases DEMAND-BENCH to evaluate dependency-aware demand navigation. The direction is valuable and the system is well engineered, but complex demand inference, real-world generalization, and physical manipulation remain clear weaknesses.

Critical strengths
  1. It combines new agent-related elements such as LLMs, agent harnesses, and skills, but transplanting them into VLN feels mechanical and does not address the core pain points of VLN. 2. The comparison methods are relatively numerous, the experiments are fairly comprehensive, and the models are relatively new.
Critical weaknesses

I think the user’s vague demands and the underlying logical relations are not the core focus of VLN; they should be handled by understanding models or LLMs, not by VLN. In other words, as LLM capabilities improve, the authors’ method will likely become obsolete. The paper’s formalization of demands as functional-requirement DAGs, allowing different object bindings and topological orders, is reasonable. But wouldn’t training a full-thinking LLM with maximum reasoning effort directly solve this? Then what is the significance of the paper?

Questions for authors

I understand that non-VLN tasks, such as travel planning, also have this kind of demand ambiguity. What is the significance of studying it separately in the context of VLN?

How do you judge the effectiveness of your method?

Do you think the problem you care about will be solved within a few months?

Reviewer_BmpfOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DemandNav, a navigation task in which an agent must infer functional requirements and their prerequisite relations from a high-level human demand, then navigate to suitable objects in a dependency-consistent order. It presents Demand-Bench, comprising 200 episodes in distinct ProcTHOR scenes, with alternative object assignments and valid completion orders under each episode’s reference DAG. The proposed DemandAgent combines a Codex-based reasoning agent with point-based navigation tools, an Adaptive Demand Ledger, area-level spatial memory, and procedural skills. It achieves 52.5% success on Demand-Bench, compared with 31.5% for Vanilla Codex using the same underlying model and navigation interface. An adaptation to EB-Navigation improves average success from 59.3% to 65.7%.

Critical strengths

The harness improves performance under a meaningful shared-interface comparison. Vanilla Codex retains the same backbone, visual input, point-based executor, budgets, and evaluator. Increasing SR from 31.5% to 52.5% therefore provides evidence beyond comparisons against heterogeneous navigation baselines. Removing spatial memory reduces SR to 37.5%, and removing ADL reduces it to 42.0%, supporting the practical value of persistent state.

The evaluation carefully separates internal claims from externally verified completion. The evaluator checks object compatibility, visibility, distance, instance reuse, and predecessor credit independently of the agent’s ledger. Accepting alternative admissible bindings and topological orders also avoids requiring one reference trajectory. The EB-Navigation evaluation provides additional evidence that the harness is useful beyond the newly constructed benchmark.

Critical weaknesses

I really do not think this work is innovative enough due to simple, due to simple simulation can't simulate the real world.

The evidence for adaptive replanning is weak, and aggregate performance obscures a consequential regression. Removing REPLAN changes overall SR only from 52.5% to 51.5%. More importantly, on the 108 dependency-bearing episodes, the full system achieves 27.8% SR, compared with 38.9% without REPLAN; PSR similarly decreases from 44.9% to 37.9% (Table 11). The aggregate advantage comes from better performance on single-requirement tasks, rather than improved success on the task category most central to the paper. No uncertainty estimates or repeated-run results establish the reliability of these differences. Figure 7 also shows substantially greater cumulative input-token usage than Vanilla Codex, leaving the performance–computation tradeoff unresolved. These results support structured state management more clearly than they support adaptive replanning.

Questions for authors

Could you clarify how the point-based navigation executor converts an image-space target into a reachable 3D endpoint and generates the collision-free path? Does it use A, the simulator’s navigation mesh, or another planning algorithm? What simulator information is available to the executor, and how might this privileged geometric assistance affect comparisons with baselines using their own navigation controllers?*

If you use the gt of simulator, maybe that is insufficient when transfer to real world application, there is a long way to go.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.