acceptodds
Withdrawn from ICLR 2027

Seek, Ground, Recover: An Agentic Framework for Autonomous Humanoid Loco-Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autonomous long-horizon humanoid loco-manipulation in open scenes requires task decisions to adapt as locomotion and posture changes alter observations, body-relative target geometry, and action feasibility. Reliable execution requires acquiring sufficient perceptual evidence, converting semantic targets into accurate metric goals, and restoring physical preconditions after failures. We present an agentic framework that couples active evidence acquisition, explicit geometric perception, and execution feedback in a closed decision loop. A vision-language model (VLM) selects observation actions, identifies and verifies task targets, and provides initial target localization. Explicit geometric perception refines these predictions into metric 3D goals for pretrained whole-body policies, separating semantic verification from metric accuracy. During and after execution, the VLM interprets temporal feedback to assess progress and identify failures; the runtime combines these assessments with controller checks to coordinate further observation, recovery, and continuation. Persistent task state retains objectives, perceptual evidence, and execution progress, supporting target re-localization and resumption of interrupted tasks after recovery. A shared goal-and-outcome interface separates semantic coordination from physical control and supports extension of the motor-skill library. Experiments demonstrate the benefits of embodied evidence acquisition and explicit geometric perception for target localization, and characterize recovery under controlled physical failures.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Humanoid Loco-Manipulation, Agentic Systems, Character Animation, Robotics, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aEzpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The framework combines VLM-guided active observation, geometric object grounding, pretrained humanoid skills, and execution monitoring with recovery. Persistent task state supports continuation after failures.

Critical strengths

The implementation addresses meaningful integration challenges. Controlled experiments show that active approach improves localization, and the paper reports physical recovery outcomes and complete-task failures rather than only successful demonstrations.

Critical weaknesses

The contribution is primarily engineering integration, with limited methodological novelty. Closely related combinations of hierarchical skill execution, geometric verification or recovery have already been explored. The new coordination mechanism is insufficiently distinguished from these precedents [1–3].

Framework-level evaluation are not established by matched end-to-end comparisons. Complete-task success is 11/30, without no-recovery or rule-based recovery baselines using the same skills. Tests also share one known simulation layout and rely on known object geometry, limiting the generality.

The related-work discussion overlooks relevant work on VLM-driven humanoid execution and tool-based 3D spatial reasoning [4–6].

[1] Cybo-Waiter — Ren et al.

[2] AEROS — Qin et al.

[3] Hierarchical Vision-Language Planning for Multi-Step Humanoid Manipulation — Schakkal et al.

[4] BiBo: Endowing GPT-4 with a Humanoid Body — Jian et al.

[5] Skill-3D: Evolving Scene-Aware Skills for Agentic 3D Spatial Reasoning — Li et al.

[6] Think3D: Thinking with Space for Spatial Reasoning — Zhang et al.

Questions for authors
  1. With the same perception and motor skills, how does complete-task success compare against a conventional state machine with rule-based recovery and against a system without recovery?

  2. Does the method generalize to new layouts and objects whose exact meshes are unavailable? How much does localization accuracy depend on the known object geometry?

  3. Which failure diagnoses or recovery decisions require VLM reasoning? Does it improve recovery success over rules using the same contact and posture measurements?

Reviewer_vqHiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an agentic framework for long-horizon humanoid loco-manipulation, structured around three stages: Seek, Ground, and Recover. The core idea is to integrate a VLM-based high-level agent with explicit geometric perception and pretrained whole-body humanoid control policies. In the Seek stage, the VLM actively selects observation actions, such as sweeping, gazing, approaching, or shifting viewpoint, to obtain better visual evidence of the source object and destination. In the Ground stage, semantically verified visual regions are converted into robot-relative metric goals through depth estimation, detector refinement, point-cloud processing, and mesh/ICP registration. In the Recover stage, the system monitors execution using RGB-D, proprioception, and contact summaries, detects failures such as drops or falls, and invokes recovery behaviors, including regrasping, floor search, and get-up policies. The main contribution is a relatively comprehensive closed-loop system architecture for humanoid manipulation that explicitly separates semantic reasoning, metric grounding, physical execution, and recovery. The evaluation is conducted primarily in simulation using ReplicaCAD and Isaac Gym. The experimental results suggest that active embodied observation substantially improves source localization, explicit mesh-based geometry improves localization accuracy over depth-only estimates, and the recovery module can handle certain controlled failure cases. The paper also reports adapting a local Qwen-based VLM using Luna-generated demonstrations. Overall, this work presents an ambitious and well-engineered integration of VLM reasoning, active perception, explicit geometry, and humanoid control. However, I have concerns about the strength of the scientific contribution and the generality of the evaluation. In particular, the system is evaluated in a single simulated layout, relies on known object shape models and numerous engineered components, and achieves relatively limited end-to-end task success. These limitations raise questions about the robustness, scalability, and practical applicability of the proposed framework.

Critical strengths

Strength 1: I think this paper presents a coherent closed-loop architecture integrating VLM reasoning, active perception, geometric localization, whole-body control, monitoring, and recovery. Unlike many VLM-robotics papers that focus on planning or short-horizon manipulation, this work addresses the full loop required for long-horizon humanoid loco-manipulation. The persistent task state and the separation of target identity, metric goal validity, and physical readiness are also sensible design choices.

Strength 2: This paper provides useful ablations supporting the roles of active perception and explicit geometry. The Qwen-FT action ablation shows substantial improvements in Loc@2cm from sweep-only to in-place gaze and full approach. The offline geometry analysis demonstrates that mesh registration substantially improves metric localization over depth median and center-pixel depth estimates. The grasp-bias experiment further highlights the practical importance of centimeter-level localization. Together, these results support the argument that semantic recognition alone is insufficient for humanoid manipulation.

Critical weaknesses

Weakness 1: The experimental evaluation is restricted to simulation and primarily uses a single furnished layout. Although the paper considers different object placements and instruction instances, the diversity of the evaluation environments remains limited. This is a significant concern given that the proposed framework is intended for open-scene long-horizon humanoid loco-manipulation. It remains unclear to what extent the reported performance depends on the specific layout, object set, available meshes, calibrated simulator geometry, and prompt/tool design. This limitation is particularly important because the main contribution is a system architecture rather than a single algorithm. For a system-oriented paper, a broader and more convincing evaluation is needed to establish the generality of the proposed approach.

Weakness 2: The reported end-to-end performance remains modest, with only 11/30 sampled cases successfully completing placement. Although the recovery experiments provide useful evidence of the system's ability to handle failures, they are conducted on small, controlled event sets, and the reported results measure physical correction rather than successful task resumption or final task completion. For example, floor drops are detected reliably, but recovery succeeds in only 15/30 cases within four attempts. This is an important limitation given that the paper aims to achieve robust long-horizon execution under changing physical conditions. The proposed recovery loop is promising, but the current experimental evidence does not yet establish reliable autonomous task completion.

Weakness 3: The Ground stage relies on canonical object meshes and mesh/ICP registration. While effective in the reported setup, this dependence limits the applicability of the approach to open-world settings where accurate object models may not be available. The system also relies on numerous hand-designed checks, prompt rules, thresholds, action budgets, and recovery-specific procedures. Although these engineering choices make the system practical, they also raise questions about how much of the reported performance stems from a general framework versus a carefully engineered pipeline tailored to the chosen simulator and task family.

Questions for authors

I have two parts questions wants to ask the author.

The first part, some clarification questions:

1.The fisrt question is about scene diversity and generalization. How many distinct scene layouts are used in the main grounding and end-to-end experiments? The paper states that the dev/test splits use “one furnished layout” with different placements. Have the authors evaluated the method on unseen layouts or substantially different room configurations?

2.How many object categories require canonical mesh models for ICP registration? How does the method handle objects without known mesh models or with significant shape variation within a category?

3.In the end-to-end evaluation, localization succeeds in 27/30 cases, whereas placement succeeds in only 11/30 cases. What are the dominant failure modes after grounding? For example, do failures primarily arise from grasp instability, navigation, carry dynamics, placement geometry, VLM monitoring, or recovery failure?

4.The recovery table reports correction rather than final task completion. Among the cases classified as corrected, how many successfully resume and complete the original task?

The second part is some potentially assessment-changing questions

5.Could the authors provide an additional evaluation on unseen scenes/layouts? If this is not feasible, a more detailed discussion of the limitations of the current single-layout evaluation and its implications for the claimed contribution would be helpful.

6.Could the authors provide component ablations in the full end-to-end setting, such as removing recovery, approach, ICP/mesh registration, or persistent task state? These experiments would help quantify the contribution of each component to final task success.

Reviewer_hqJ8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents an agentic framework for long-horizon humanoid object relocation. A VLM selects observation actions (gaze, viewpoint shifts, approach), verifies source and destination targets, and monitors execution. A fine-tuned YOLO detector, depth back-projection and mesh ICP turn verified regions into metric goals for pretrained whole-body controllers built on OmniGrasp and PULSE-X. Contact and pelvis-height summaries drive recovery from drops and falls, and a persistent task state allows resumption. Experiments in one ReplicaCAD layout report about 84–91% source localization within 2 cm, recovery counts on 30 controlled failures per category, 11/30 end-to-end completions, and a QLoRA-adapted Qwen3.8-27B that improves grounding over its base model.

Critical strengths
  1. The ablations show that active observation and explicit geometric refinement each improve target localization.
Critical weaknesses
  1. All 200 tasks take place in a single ReplicaCAD layout, and the detector is fine-tuned on 7,260 frames rendered in that same layout with the same 51 GRAB objects. Development and test tasks differ only in object placements and instructions. The claims about open scenes and open-ended instructions are therefore untested, and the Qwen-FT gain from 63% to 91% amounts to adaptation to the test scene.
  2. Metric grounding relies on simulator assets. ICP requires the canonical mesh of every target, and approach planning uses an occupancy grid built from the scene meshes. The claim that execution runs "without simulator-provided object-state truth" is misleading when object shape and the scene map both come from the simulator.
  3. The paper compares against no existing system. The only external references are one-shot VLM predictions on fixed images, which reach 0.7% and 3.0% localization with one call against about 34 calls for the full agent. HumanCLAW and the other agentic humanoid systems discussed in Section 2 are never evaluated.
  4. The end-to-end evidence is minimal for a paper about long-horizon loco-manipulation. The main tables score localization only, and grasping and placement are not scored there. The full task is evaluated on 30 cases, 11 of which complete placement, with no baseline, so the paper does not show that the framework improves task completion.
  5. The contribution is system integration of existing components: a commercial VLM, YOLOv8, ICP, A* and the pretrained OmniGrasp and PULSE-X controllers on a simulated SMPL-X humanoid. Equations 1–3, 5 and 8 formalize bookkeeping (state records, validity flags, retry counters) rather than an algorithm, and the paper offers no new learning method or insight that would transfer beyond this setup.
Questions for authors
  1. Are any of the 51 objects, or their meshes, held out from detector training and ICP at test time?
  2. How many test tasks reuse the source object and destination of a development task that contributed demonstrations to Qwen-FT?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.