- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work aims to analyze action intelligence beyond low-level motor control, which can guide vision-language models (VLMs) to become better suited to robotic applications. To this end, the work proposes HumanClaw-Bench, a navigation benchmark accompanied by the HumanClaw harness and a half-physics simulation. The analysis indicates that body self-awareness remains a challenging issue for VLMs.
Critical strengths
- To overcome the gap between semantic intelligence in VLMs and robotics control, this work proposes a skill-conditioned motion generator that is extensible with plug-and-play skill modules.
- The work proposes a benchmark consisting of 1,218 episodes and 41 types of layouts, spanning easy, medium, and hard tasks. A half-physics simulation is constructed to preserve the necessary interactions between objects and the human agent.
- The analysis metrics include both navigation success rate and body awareness. Through comprehensive failure analysis, the work finds that reasoning ability (without harness guidance) and body awareness remain challenging for VLMs before GPT-6 or Gemini-4.
Critical weaknesses
- The motivation is valuable, but the selected task is limited to support it and remains far from real-world applications. For navigation tasks, introducing a humanoid agent seems unnecessary. To emphasize the use of the humanoid, the work adds a "sit" action, which cannot reflect real-world applications. For manipulation tasks, the humanoid would be a good choice, but the work has not considered this yet.
- The body-awareness problem is influenced by the experimental setting. Since an ego-view-only setup cannot provide sufficient signals for the model to recognize its own body, even when it collides with the environment. A suggestion is to add a wrist view or camera control while retaining the vision-only setting, allowing the model to collect enough visual information to improve body awareness.
- The ablation study on history images seems somewhat counterintuitive, since history images help construct the navigation map, which is valuable for navigation tasks; further analysis is needed to explain the results.
Questions for authors
The ablation study on history images seems somewhat counterintuitive, since history images help construct the navigation map, which is valuable for navigation tasks; further analysis is needed to explain the results.
3 comments
[2/3] W2: The body-awareness problem is influenced by the experimental setting.**
Our intended meaning of embodied self-awareness is the ability to estimate one's body state (especially from ego view), its relation to the scene, and the outcomes of recent actions during a closed-loop task, not direct access to every joint or contact event.
As requested, we conduct additional ablations on the same 100-episode mini-val split with Gemma-4-31B, providing: (1) explicit body states given as text: SMPL-X joint rotations, plus the root turn and body-frame displacement actually produced by the previous action; (2) an additional rear RGB camera; (3) privileged geometry, including ground-truth distances to surrounding objects; and (4) privileged contact information for each body region.
| Model | FindSR ↑ | NavSR@20cm ↑ | InteractSR ↑ | Coll. ↓ | #Dtb/ep ↓ | dDtb(m) ↓ | Jerk ↓ | Avg. steps | Input tok/step | Output tok/step |
|---|---|---|---|---|---|---|---|---|---|---|
| Baseline | 58.0% | 27.0% | 18.9% | 41.4% | 1.35 | 9.38 | 4.9 | 78.5 | 4676 | 322 |
| + Body state (SMPLX joint) | 63.0% | 20.0% | 1.9% | 48.9% | 1.25 | 6.97 | 4.57 | 84.2 | 6077 | 342 |
| + Rear camera | 70.0% | 31.0% | 20.8% | 38.6% | 1.02 | 2.27 | 4.84 | 75.0 | 5327 | 328 |
| + Geometry (previlaged) | 64.0% | 38.0% | 26.4% | 36.2% | 0.89 | 1.74 | 5.05 | 76.0 | 5354 | 326 |
| + Contact (previlaged) | 68.0% | 36.0% | 22.6% | 41.3% | 1.15 | 0.97 | 5.00 | 73.1 | 5941 | 331 |
(1) Explicit body states do not improve performance and even substantially reduce InteractSR (18.9% → 1.9%), suggesting that the evaluated model struggles to use body states in this raw, high-dimensional form.
(2) A rear camera improves FindSR (58% → 70%), but provides only a small improvement in interaction (18.9% → 20.8%).
(3) Privileged geometry and contact feedback improve navigation by 11 and 9 percentage points, respectively, but InteractSR remains low (26.4% and 22.6%). Explicit geometry also reduces collisions (41.4% → 36.2%).
Importantly, some of these additional inputs are privileged information that partially bypasses the very capability we aim to evaluate: whether VLMs can estimate their embodied state and action outcomes, rather than being directly given such information.
Our original finding remains supported. Privileged geometry and contact feedback improve navigation but do not eliminate the observed failures: InteractSR remains low (26.4% and 22.6%, respectively). This suggests that limited observability is not the only contributing factor in this setting. Moreover, on the same mini-val split, GPT-6 Astra reaches 50.9% InteractSR using only egocentric observations, showing that the egocentric interface itself does not prevent substantially higher performance. The key challenge remains whether VLMs can estimate and use their embodied state to guide subsequent actions.
[3/3] W3, Q1: The ablation study on history images seems somewhat counterintuitive.**
We agree. On the 100-episode mini-val, increasing visual history from 1 to 10 frames reduces NavSR from 27.0% to 13.0% and InteractSR from 18.9% to 3.8% (Table 2).
An explanation is that a longer visual context introduces competing or stale spatial evidence and strains the model's ability to track the current state makes it more hallucination. This is also consistent what we observes when we develop short-context verifier
[1/3] W1: The motivation is valuable, but the selected task is limited to support it and remains far from real-world applications.
We respectfully disagree that the selected tasks are far from real-world applications. Finding, navigating are basic real-world full-body behaviors; sitting is relevant to applications such as VR and embodied virtual agents, even if it seems less central to robotics.
These tasks provide meaningful tests of VLM action intelligence and embodied self-awareness. We do not claim that HumanCLAW-Bench covers all aspects of action intelligence.
To further address task diversity, we introduce a preliminary Reach–Pick–Carry–Place sub-benchmark. It contains 100 tasks across 10 indoor layouts, 5 object categories (boxes, teddy bears, balls, vases, and baskets), and 2 object sizes. Initial hand–object distances range from 0.12 to 6.18 m. We add IK-based upper-body end-effector control alongside the existing locomotion skills, allowing both to execute concurrently.
We evaluate four progressive milestones: ReachSR (hand contact or within 10 cm), PickSR (lifting the object ≥1 cm without environmental support), CarrySR (transporting the object reaching within 0.5 m of the destination), and PlaceSR (releasing it with stable support). We additionally measure object fall rate. Each episode allows up to 150 action steps.
The results reveal a substantial gap between reaching and manipulating objects. Even GPT-6 Astra achieves only 8% PickSR, 1% CarrySR, and 0% PlaceSR. Six of the eight successful pickups occur in the layout where the object starts within arm's reach (0.12–0.32 m from the palms). Qualitative failures include poor coordination between body approach and reaching, as well as difficulty maintaining a grasp during locomotion. In the sole successful carry episode, GPT-Astra brings a teddy bear near the destination table but drop it before reaching table.
This extension broadens HumanCLAW from navigation and sitting to whole-body loco-manipulation, involving object interaction, object transport, and placement. These preliminary results suggest related challenges in coordinating whole-body actions and predicting their outcomes.