acceptodds
Withdrawn from ICLR 2027

HumanCLAW: Can Vision-Language Models Act Through a Body?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evaluating whether a vision-language model (VLM) can act through a physical body is challenging. The outcome of an action couples the VLM's decision with motor control. When a task fails, it is hard to tell whether the VLM made a bad choice or the motor controller simply failed to execute it, e.g., losing balance and falling. In this work, we introduce HumanCLAW, an evaluation framework that decouples action decision-making from low-level execution. At every step, a harnessed, off-the-shelf VLM issues an atomic skill command, and the command is translated into a sub-second chunk of continuous full-body motion with real physical consequences, including gravity and collisions. The body can therefore act freely in the physical world, while execution-side disturbances, balance and motor errors, are factored out. What remains measurable is the model's action intelligence: its moment-to-moment choice of what the body should execute next. Based on this framework, we build HumanCLAW-Bench: 1,218 long-horizon, egocentric find-navigate-interact episodes across 41 indoor scenes. We test nine state-of-the-art VLMs and find that none solves the benchmark; the best model reaches only a 16.8% success rate. Recognizing the target is not the bottleneck. What current VLMs lack is embodied self-awareness: they lose track of their own body, failing to tell where it is, whether it has reached the goal, or whether it has hit an obstacle. Video demos, the leaderboard, and analysis are available on the anonymous supplementary webpage.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, multimodal large language models, VLM evaluation, benchmark, embodied reasoning, spatial reasoning, egocentric vision, embodied self-awareness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_j7dQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work aims to analyze action intelligence beyond low-level motor control, which can guide vision-language models (VLMs) to become better suited to robotic applications. To this end, the work proposes HumanClaw-Bench, a navigation benchmark accompanied by the HumanClaw harness and a half-physics simulation. The analysis indicates that body self-awareness remains a challenging issue for VLMs.

Critical strengths
  1. To overcome the gap between semantic intelligence in VLMs and robotics control, this work proposes a skill-conditioned motion generator that is extensible with plug-and-play skill modules.
  2. The work proposes a benchmark consisting of 1,218 episodes and 41 types of layouts, spanning easy, medium, and hard tasks. A half-physics simulation is constructed to preserve the necessary interactions between objects and the human agent.
  3. The analysis metrics include both navigation success rate and body awareness. Through comprehensive failure analysis, the work finds that reasoning ability (without harness guidance) and body awareness remain challenging for VLMs before GPT-6 or Gemini-4.
Critical weaknesses
  1. The motivation is valuable, but the selected task is limited to support it and remains far from real-world applications. For navigation tasks, introducing a humanoid agent seems unnecessary. To emphasize the use of the humanoid, the work adds a "sit" action, which cannot reflect real-world applications. For manipulation tasks, the humanoid would be a good choice, but the work has not considered this yet.
  2. The body-awareness problem is influenced by the experimental setting. Since an ego-view-only setup cannot provide sufficient signals for the model to recognize its own body, even when it collides with the environment. A suggestion is to add a wrist view or camera control while retaining the vision-only setting, allowing the model to collect enough visual information to improve body awareness.
  3. The ablation study on history images seems somewhat counterintuitive, since history images help construct the navigation map, which is valuable for navigation tasks; further analysis is needed to explain the results.
Questions for authors

The ablation study on history images seems somewhat counterintuitive, since history images help construct the navigation map, which is valuable for navigation tasks; further analysis is needed to explain the results.

3 comments
AuthorsOpenReview ↗

[1/3] W1: The motivation is valuable, but the selected task is limited to support it and remains far from real-world applications.

We respectfully disagree that the selected tasks are far from real-world applications. Finding, navigating are basic real-world full-body behaviors; sitting is relevant to applications such as VR and embodied virtual agents, even if it seems less central to robotics.

These tasks provide meaningful tests of VLM action intelligence and embodied self-awareness. We do not claim that HumanCLAW-Bench covers all aspects of action intelligence.

To further address task diversity, we introduce a preliminary Reach–Pick–Carry–Place sub-benchmark. It contains 100 tasks across 10 indoor layouts, 5 object categories (boxes, teddy bears, balls, vases, and baskets), and 2 object sizes. Initial hand–object distances range from 0.12 to 6.18 m. We add IK-based upper-body end-effector control alongside the existing locomotion skills, allowing both to execute concurrently.

We evaluate four progressive milestones: ReachSR (hand contact or within 10 cm), PickSR (lifting the object ≥1 cm without environmental support), CarrySR (transporting the object reaching within 0.5 m of the destination), and PlaceSR (releasing it with stable support). We additionally measure object fall rate. Each episode allows up to 150 action steps.

ModelReachSR ↑PickSR ↑CarrySR ↑PlaceSR ↑Fall rate ↓
Astra66.0%8.0%1.0%0.0%16.0%
Gemma-4-31B42.0%6.0%0.0%0.0%17.0%
InternVL-3.5-38B15.0%0.0%0.0%0.0%7.0%

The results reveal a substantial gap between reaching and manipulating objects. Even GPT-6 Astra achieves only 8% PickSR, 1% CarrySR, and 0% PlaceSR. Six of the eight successful pickups occur in the layout where the object starts within arm's reach (0.12–0.32 m from the palms). Qualitative failures include poor coordination between body approach and reaching, as well as difficulty maintaining a grasp during locomotion. In the sole successful carry episode, GPT-Astra brings a teddy bear near the destination table but drop it before reaching table.

This extension broadens HumanCLAW from navigation and sitting to whole-body loco-manipulation, involving object interaction, object transport, and placement. These preliminary results suggest related challenges in coordinating whole-body actions and predicting their outcomes.

AuthorsOpenReview ↗

[2/3] W2: The body-awareness problem is influenced by the experimental setting.**

Our intended meaning of embodied self-awareness is the ability to estimate one's body state (especially from ego view), its relation to the scene, and the outcomes of recent actions during a closed-loop task, not direct access to every joint or contact event.

As requested, we conduct additional ablations on the same 100-episode mini-val split with Gemma-4-31B, providing: (1) explicit body states given as text: SMPL-X joint rotations, plus the root turn and body-frame displacement actually produced by the previous action; (2) an additional rear RGB camera; (3) privileged geometry, including ground-truth distances to surrounding objects; and (4) privileged contact information for each body region.

ModelFindSR ↑NavSR@20cm ↑InteractSR ↑Coll. ↓#Dtb/ep ↓dDtb(m) ↓Jerk ↓Avg. stepsInput tok/stepOutput tok/step
Baseline58.0%27.0%18.9%41.4%1.359.384.978.54676322
+ Body state (SMPLX joint)63.0%20.0%1.9%48.9%1.256.974.5784.26077342
+ Rear camera70.0%31.0%20.8%38.6%1.022.274.8475.05327328
+ Geometry (previlaged)64.0%38.0%26.4%36.2%0.891.745.0576.05354326
+ Contact (previlaged)68.0%36.0%22.6%41.3%1.150.975.0073.15941331

(1) Explicit body states do not improve performance and even substantially reduce InteractSR (18.9% → 1.9%), suggesting that the evaluated model struggles to use body states in this raw, high-dimensional form.

(2) A rear camera improves FindSR (58% → 70%), but provides only a small improvement in interaction (18.9% → 20.8%).

(3) Privileged geometry and contact feedback improve navigation by 11 and 9 percentage points, respectively, but InteractSR remains low (26.4% and 22.6%). Explicit geometry also reduces collisions (41.4% → 36.2%).

Importantly, some of these additional inputs are privileged information that partially bypasses the very capability we aim to evaluate: whether VLMs can estimate their embodied state and action outcomes, rather than being directly given such information.

Our original finding remains supported. Privileged geometry and contact feedback improve navigation but do not eliminate the observed failures: InteractSR remains low (26.4% and 22.6%, respectively). This suggests that limited observability is not the only contributing factor in this setting. Moreover, on the same mini-val split, GPT-6 Astra reaches 50.9% InteractSR using only egocentric observations, showing that the egocentric interface itself does not prevent substantially higher performance. The key challenge remains whether VLMs can estimate and use their embodied state to guide subsequent actions.

AuthorsOpenReview ↗

[3/3] W3, Q1: The ablation study on history images seems somewhat counterintuitive.**

We agree. On the 100-episode mini-val, increasing visual history from 1 to 10 frames reduces NavSR from 27.0% to 13.0% and InteractSR from 18.9% to 3.8% (Table 2).

An explanation is that a longer visual context introduces competing or stale spatial evidence and strains the model's ability to track the current state makes it more hallucination. This is also consistent what we observes when we develop short-context verifier

Reviewer_8SjjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HumanCLAW evaluates frozen VLMs in a closed-loop humanoid simulation. The VLM selects parameterized skills, a motion generator produces continuous full-body motion, and a half-physics simulator executes it with physical interactions while abstracting away balance control. The benchmark contains 1,218 episodes across 41 indoor scenes, including 597 episodes requiring sitting after navigation. The authors attribute many failures to limitations in embodied self-awareness.

Critical strengths
  1. A useful evaluation objective. Studying embodied decision-making while reducing the influence of balance and motor-tracking failures is worthwhile. The combination of parameterized skills, continuous full-body motion, and physical contact provides a concrete setting for investigating how visual reasoning translates into actions with spatial consequences.

  2. Informative diagnostic measurements. Separating finding, navigation, and interaction, and distinguishing geometric achievement from the model’s acknowledgment of completion, provides more information than a single task-success metric. The multi-model evaluation and harness ablations also reveal substantial differences in how models maintain objectives and decide when to stop.

Critical weaknesses
  1. Limited task coverage relative to the conclusions. The benchmark primarily evaluates object-goal navigation and sitting. Although scene layouts vary, the evaluated behaviors largely concern locomotion and seat alignment. It remains unclear whether the findings extend to interactions involving different body coordination and contact patterns, such as reaching, manipulating objects, or adjusting posture around obstacles. The broad conclusions about action intelligence and embodied self-awareness should therefore be more explicitly qualified.

  2. Model limitations are not sufficiently distinguished from interface constraints. The VLM receives egocentric RGB and textual history without explicit proprioceptive or contact feedback. Whole-body turning is available, but the described skill interface lacks independent head or camera control, potentially making it difficult to inspect a seat after turning away from it. Some failures may therefore reflect restricted information access rather than an intrinsic lack of embodied self-awareness. Recent work such as RoboDawn and RoboICL further motivates examining interface sensitivity. Although their settings are not directly comparable, they motivate testing interface sensitivity before attributing failures primarily to an intrinsic lack of embodied self-awareness.

Questions for authors

How sensitive are task performance and failure attributions to the motion generator? Table 4 compares command-following fidelity with MoMask, but not end-to-end task performance. Could an alternative generator with comparable execution fidelity be evaluated on a representative subset, keeping the VLM, harness, skill interface, and simulator fixed?

3 comments
AuthorsOpenReview ↗

[1/3] W1: Limited task coverage relative to the conclusions

We thank the reviewer for this suggestion. While recent work has explored VLM-based robotic arm control, HumanCLAW takes an initial step toward evaluating sub-second atomic action decision-making through a full human body. We introduce action intelligence as a research concept, but do not claim that our benchmark comprehensively measures general action intelligence.

To further address task diversity, we introduce a preliminary Reach–Pick–Carry–Place sub-benchmark. It contains 100 tasks across 10 indoor layouts, 5 object categories (boxes, teddy bears, balls, vases, and baskets), and 2 object sizes. Initial hand–object distances range from 0.12 to 6.18 m. We add IK-based upper-body end-effector control alongside the existing locomotion skills, allowing both to execute concurrently.

We evaluate four progressive milestones: ReachSR (hand contact or within 10 cm), PickSR (lifting the object ≥1 cm without environmental support), CarrySR (transporting the object reaching within 0.5 m of the destination), and PlaceSR (releasing it with stable support). We additionally measure object fall rate. Each episode allows up to 150 action steps.

ModelReachSR ↑PickSR ↑CarrySR ↑PlaceSR ↑Fall rate ↓
Astra66.0%8.0%1.0%0.0%16.0%
Gemma-4-31B42.0%6.0%0.0%0.0%17.0%
InternVL-3.5-38B15.0%0.0%0.0%0.0%7.0%

The results reveal a substantial gap between reaching and manipulating objects. Even GPT-6 Astra achieves only 8% PickSR, 1% CarrySR, and 0% PlaceSR. Six of the eight successful pickups occur in the layout where the object starts within arm's reach (0.12–0.32 m from the palms). Qualitative failures include poor coordination between body approach and reaching, as well as difficulty maintaining a grasp during locomotion. In the sole successful carry episode, GPT-Astra brings a teddy bear near the destination table but drop it before reaching table.

This extension broadens HumanCLAW from navigation and sitting to whole-body loco-manipulation, involving object interaction, object transport, and placement. These preliminary results suggest related challenges in coordinating whole-body actions and predicting their outcomes.

AuthorsOpenReview ↗

[2/3] W2: Model limitations are not sufficiently distinguished from interface constraints.

Our intended meaning of embodied self-awareness is the ability to estimate one's body state (especially from ego view), its relation to the scene, and the outcomes of recent actions during a closed-loop task, not direct access to every joint or contact event.

As requested, we conduct additional ablations on the same 100-episode mini-val split with Gemma-4-31B, providing: (1) explicit body states given as text: SMPL-X joint rotations, plus the root turn and body-frame displacement actually produced by the previous action; (2) an additional rear RGB camera; (3) privileged geometry, including ground-truth distances to surrounding objects; and (4) privileged contact information for each body region.

ModelFindSR ↑NavSR@20cm ↑InteractSR ↑Coll. ↓#Dtb/ep ↓dDtb(m) ↓Jerk ↓Avg. stepsInput tok/stepOutput tok/step
Baseline58.0%27.0%18.9%41.4%1.359.384.978.54676322
+ Body state (SMPLX joint)63.0%20.0%1.9%48.9%1.256.974.5784.26077342
+ Rear camera70.0%31.0%20.8%38.6%1.022.274.8475.05327328
+ Geometry (previlaged)64.0%38.0%26.4%36.2%0.891.745.0576.05354326
+ Contact (previlaged)68.0%36.0%22.6%41.3%1.150.975.0073.15941331

(1) Explicit body states do not improve performance and even substantially reduce InteractSR (18.9% → 1.9%), suggesting that the evaluated model struggles to use body states in this raw, high-dimensional form.

(2) A rear camera improves FindSR (58% → 70%), but provides only a small improvement in interaction (18.9% → 20.8%).

(3) Privileged geometry and contact feedback improve navigation by 11 and 9 percentage points, respectively, but InteractSR remains low (26.4% and 22.6%). Explicit geometry also reduces collisions (41.4% → 36.2%).

Importantly, some of these additional inputs are privileged information that partially bypasses the very capability we aim to evaluate: whether VLMs can estimate their embodied state and action outcomes, rather than being directly given such information.

Our original finding remains supported. Privileged geometry and contact feedback improve navigation but do not eliminate the observed failures: InteractSR remains low (26.4% and 22.6%, respectively). This suggests that limited observability is not the only contributing factor in this setting. Moreover, on the same mini-val split, GPT-6 Astra reaches 50.9% InteractSR using only egocentric observations, showing that the egocentric interface itself does not prevent substantially higher performance. The key challenge remains whether VLMs can estimate and use their embodied state to guide subsequent actions.

AuthorsOpenReview ↗

[3/3] Q1: Could an alternative generator with comparable execution fidelity be evaluated on a representative subset, keeping the VLM, harness, skill interface, and simulator fixed?

To our knowledge, there is no readily available alternative generator that supports the same parameterized atomic skills with comparable execution fidelity. Nevertheless, we replace our motion generator with MoMask and evaluate it under the same VLM, harness, skill interface, and simulator.

SettingFindSR ↑NavSR ↑InteractSR ↑
Baseline58.0%27.0%18.9%
MoMask (lower execution fidelity)66.0%17.0%0.0%

Despite a higher FindSR, MoMask substantially reduces navigation success and completely fails at interaction. This is consistent with its lower parameter-following fidelity (Table 4), highlighting the importance of reliable motion execution.

More importantly, we validate long-horizon task solvability through a privileged-state oracle. It uses offline A* planning and online VLM-free rule-based skill selection, while executing actions through exactly the same atomic skills, motion generator, and half-physics simulator as the evaluated VLMs.

ModelFindSR ↑NavSR ↑InteractSR ↑Coll. ↓#Dtb/ep ↓dDtb (m) ↓Motion Jerk ↓Avg. Steps ↓
Gemini-3.164.90%42.40%16.80%39.48%1.501.225.7059.10
GPT-6 Astra75.50%57.10%46.60%41.47%1.700.655.3060.50
Oracle99.01%98.28%98.99%26.39%0.730.445.2134.49

The oracle achieves 98.28% navigation and 98.99% interaction success through long-horizon multi-skill execution. Although this does not establish independence from the choice of motion generator, it demonstrates that our execution pipeline can reliably solve nearly all benchmark tasks. Therefore, the substantially lower VLM success rates cannot be explained simply by limitations of the execution pipeline.

Reviewer_uxv8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HumanCLAW, an embodied evaluation framework that decouples high-level VLM decision-making from low-level motor execution through atomic skills, motion generation, and half-physics simulation. The authors further develop HumanCLAW-Bench, comprising 1,218 tasks across 41 indoor scenes, to evaluate VLMs on object finding, navigation, and interaction, and analyze their failures in embodied state awareness and action decision-making.

Critical strengths
  1. The paper addresses an important and well-motivated problem of evaluating embodied decision-making in VLMs while reducing the confounding effects of low-level motor control.
  2. The proposed framework provides a meaningful evaluation setting by integrating fine-grained atomic action decisions, continuous whole-body motion, and physically grounded interactions.
  3. The paper provides detailed failure analyses and useful ablation studies, offering insights into the challenges of embodied state estimation, action termination, and physical interaction.
Critical weaknesses
  1. The validation of the motion layer (Section 4.1 and Table 4) mainly focuses on isolated, single-skill parameter tracking accuracy (where even stair skills show a noticeable 0.74–0.79 tracking gain), without quantitatively demonstrating that execution drift or transition dynamics have negligible effects during continuous multi-skill transitions and close-proximity interactions. Therefore, attributing failures entirely to VLM decision-making is not fully justified.
  2. Although the authors acknowledge in Section 5 that the interface lacks proprioceptive or contact feedback, this sensory deprivation directly undermines their core claim in Finding 6 ('no current VLM perceives its own body'). Because limb collisions (e.g., arms or legs hitting low furniture) often occur outside the egocentric camera's field of view, the experiments cannot disentangle intrinsic limitations in VLM embodied reasoning from partial observability without an ablation that provides basic contact or body-state signals in the prompt.
  3. Although the paper emphasizes general action intelligence, the tasks primarily focus on object finding, navigation, and sitting. The absence of oracle or conventional planning baselines makes it difficult to assess task solvability and the actual contribution of VLM-based decision-making.
Questions for authors
  1. Section 2 mentions a textual history of 'past decisions and feedback,' while Figure 3 shows the history containing visual state, mid-level plan & reason, low-level skill & reason, and verifier record. Could the authors clarify whether all items in this history are purely self-generated by the VLM/verifier from earlier RGB frames, or if any explicit execution feedback (e.g., actual displacement or collision flags) from the simulator is injected?
  2. The skill-specific verifier plays a crucial role in the framework, as removing it reduces NavSR from 27.0% to 2.0%. Could the authors clarify whether the verifier uses the same VLM backbone as the planner, whether it requires an additional inference call, and how corrected actions are generated when a proposed skill is rejected?
  3. To sit on a target, the agent must first turn around so the seat lies behind its body (Table 9), which moves the target completely out of the current egocentric view. Since the baseline only receives a single current RGB frame (img 1) and lacks head/camera pan-tilt control, how is the VLM expected to accurately estimate the precise alignment and sitting height h once its back is turned? To what extent is the dominant "sit on air" failure mode (58%) an artifact of this fixed camera field-of-view limitation rather than a failure of embodied reasoning?
6 comments
AuthorsOpenReview ↗

[1/6] W1: quantitatively demonstrating that execution drift or transition dynamics have negligible effects

We thank the reviewer for this suggestion. Table 4 quantifies individual skill fidelity, while supplementary videos demonstrate smooth transitions. We further evaluated five representative skill transitions: (a) walk→turn (0.942+-0.067), (b) turn→walk (0.908+-0.103), (c ) walk→side-step (0.901 +-0.110), (d) forward→backward (0.246+-0.137), and (e) turn −45°→right +45° (0.672+-0.262).

Transitions without opposing momentum (a-c) maintain relatively high execution fidelity. Opposing motions (d-e) introduce larger short-term deviations due to the need for deceleration. Importantly, HumanCLAW operates in a closed loop every 0.5 seconds, allowing the agent to adapt subsequent actions based on actual execution outcomes rather than assuming perfect motion tracking.

More importantly, we validate long-horizon execution through a privileged-state oracle. It uses offline A* planning and online VLM-free rule-based skill selection, while executing actions through exactly the same atomic skills, motion generator, and half-physics simulator as the evaluated VLMs.

ModelFindSR ↑NavSR ↑InteractSR ↑Coll. ↓#Dtb/ep ↓dDtb (m) ↓Motion Jerk ↓Avg. Steps ↓
Gemini-3.164.90%42.40%16.80%39.48%1.501.225.7059.10
GPT-6 Astra75.50%57.10%46.60%41.47%1.700.655.3060.50
Oracle99.01%98.28%98.99%26.39%0.730.445.2134.49

The oracle achieves 98.28% navigation and 98.99% interaction success, despite using the same atomic skills, motion generator, and simulator. This demonstrates that the execution pipeline can support reliable long-horizon task completion when guided by effective action decisions, including the skill transitions that occur in these episodes. While execution errors are not entirely eliminated, they do not constitute a fundamental bottleneck for solving the benchmark. The substantially lower VLM success rates therefore cannot be primarily attributed to limitations of the execution pipeline.

AuthorsOpenReview ↗

[2/6] W2: the experiments cannot disentangle intrinsic limitations in VLM embodied reasoning from partial observability without an ablation that provides basic contact or body-state signals in the prompt.

Our intended meaning of embodied self-awareness is the ability to estimate one's body state (especially from ego view), its relation to the scene, and the outcomes of recent actions during a closed-loop task, not direct access to every joint or contact event.

As requested, we conduct additional ablations on the same 100-episode mini-val split with Gemma-4-31B, providing: (1) explicit body states given as text: SMPL-X joint rotations, plus the root turn and body-frame displacement actually produced by the previous action; (2) an additional rear RGB camera; (3) privileged geometry, including ground-truth distances to surrounding objects; and (4) privileged contact information for each body region.

ModelFindSR ↑NavSR@20cm ↑InteractSR ↑Coll. ↓#Dtb/ep ↓dDtb(m) ↓Jerk ↓Avg. stepsInput tok/stepOutput tok/step
Baseline58.0%27.0%18.9%41.4%1.359.384.978.54676322
+ Body state (SMPLX joint)63.0%20.0%1.9%48.9%1.256.974.5784.26077342
+ Rear camera70.0%31.0%20.8%38.6%1.022.274.8475.05327328
+ Geometry (previlaged)64.0%38.0%26.4%36.2%0.891.745.0576.05354326
+ Contact (previlaged)68.0%36.0%22.6%41.3%1.150.975.0073.15941331

(1) Explicit body states do not improve performance and even substantially reduce InteractSR (18.9% → 1.9%), suggesting that the evaluated model struggles to use body states in this raw, high-dimensional form.

(2) A rear camera improves FindSR (58% → 70%), but provides only a small improvement in interaction (18.9% → 20.8%).

(3) Privileged geometry and contact feedback improve navigation by 11 and 9 percentage points, respectively, but InteractSR remains low (26.4% and 22.6%). Explicit geometry also reduces collisions (41.4% → 36.2%).

Importantly, some of these additional inputs are privileged information that partially bypasses the very capability we aim to evaluate: whether VLMs can estimate their embodied state and action outcomes, rather than being directly given such information.

Our original finding remains supported. Privileged geometry and contact feedback improve navigation but do not eliminate the observed failures: InteractSR remains low (26.4% and 22.6%, respectively). This suggests that limited observability is not the only contributing factor in this setting. Moreover, on the same mini-val split, GPT-6 Astra reaches 50.9% InteractSR using only egocentric observations, showing that the egocentric interface itself does not prevent substantially higher performance. The key challenge remains whether VLMs can estimate and use their embodied state to guide subsequent actions.

AuthorsOpenReview ↗

[3/6] W3: Although the paper emphasizes general action intelligence, the tasks primarily focus on object finding, navigation, and sitting. The absence of oracle or conventional planning baselines makes it difficult to assess task solvability and the actual contribution of VLM-based decision-making.**

(1) Oracle. See response to W1

(2) Benchmark scope. We study a general decision-making capability through a focused, diagnostic task, without claiming exhaustive task coverage.

To further address task diversity, we introduce a preliminary Reach–Pick–Carry–Place sub-benchmark. It contains 100 tasks across 10 indoor layouts, 5 object categories (boxes, teddy bears, balls, vases, and baskets), and 2 object sizes. Initial hand–object distances range from 0.12 to 6.18 m. We add IK-based upper-body end-effector control alongside the existing locomotion skills, allowing both to execute concurrently.

We evaluate four progressive milestones: ReachSR (hand contact or within 10 cm), PickSR (lifting the object ≥1 cm without environmental support), CarrySR (transporting the object reaching within 0.5 m of the destination), and PlaceSR (releasing it with stable support). We additionally measure object fall rate. Each episode allows up to 150 action steps.

ModelReachSR ↑PickSR ↑CarrySR ↑PlaceSR ↑Fall rate ↓
Astra66.0%8.0%1.0%0.0%16.0%
Gemma-4-31B42.0%6.0%0.0%0.0%17.0%
InternVL-3.5-38B15.0%0.0%0.0%0.0%7.0%

The results reveal a substantial gap between reaching and manipulating objects. Even GPT-6 Astra achieves only 8% PickSR, 1% CarrySR, and 0% PlaceSR. Six of the eight successful pickups occur in the layout where the object starts within arm's reach (0.12–0.32 m from the palms). Qualitative failures include poor coordination between body approach and reaching, as well as difficulty maintaining a grasp during locomotion. In the sole successful carry episode, GPT-Astra brings a teddy bear near the destination table but drop it before reaching table.

This extension broadens HumanCLAW from navigation and sitting to whole-body loco-manipulation, involving object interaction, object transport, and placement. These preliminary results suggest related challenges in coordinating whole-body actions and predicting their outcomes.

Reviewer_PqMCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HumanCLAW, an evaluation framework designed to measure the action intelligence of vision-language models (VLMs) in embodied settings. The central motivation is to disentangle high-level action decision-making from low-level motor execution, which are often coupled in conventional humanoid evaluations. HumanCLAW addresses this by having an off-the-shelf VLM select parameterized atomic skills, which are then translated into continuous full-body motion by a skill-conditioned motion generator. The resulting motion is executed in a half-physics simulator that models gravity, collisions, friction, and object displacement while abstracting away balance and motor-tracking failures.

Based on this framework, the authors construct HumanCLAW-Bench, consisting of 1,218 egocentric find-navigate-interact episodes across 41 indoor scenes. The benchmark evaluates nine VLMs on tasks that require finding a target object, navigating to it, and sitting on it when applicable. The strongest model achieves a 16.8% interaction success rate. Through extensive failure analysis, the authors argue that current VLMs are limited less by object recognition than by embodied self-awareness, particularly their ability to estimate body position, recognize arrival, and understand the physical consequences of their actions.

The paper addresses an important problem and presents a reasonably comprehensive evaluation framework. However, I have concerns about the validity and scope of the proposed evaluation abstraction, the extent to which the benchmark measures general action intelligence, and the strength of the conclusions drawn from the selected tasks and observations.

Critical strengths

S1. Important and well-motivated research question.

The paper identifies a meaningful gap between conventional vision-language agent benchmarks and end-to-end embodied control. Existing agent benchmarks often abstract away physical execution, while end-to-end robot policies entangle decision-making with low-level control. Separating these factors is a useful research direction, particularly for understanding the capabilities and limitations of general-purpose VLMs.

S2. Interesting evaluation framework.

The combination of atomic skill selection, continuous motion generation, and physics-based scene interaction is a compelling design. In particular, the half-physics simulator provides an intermediate setting between purely kinematic playback and fully dynamic humanoid simulation. It retains important physical consequences, such as collisions and object displacement, without making balance failures the dominant source of task failure.

S3. Comprehensive benchmark construction and empirical evaluation.

The benchmark includes 1,218 episodes across 41 indoor scenes and evaluates nine VLMs. The authors report multiple metrics beyond task success, including navigation success, interaction success, collision frequency, object displacement, motion jerk, and inference cost. The additional difficulty analysis and harness ablations provide useful information about the evaluated systems.

Critical weaknesses

W1. The proposed half-physics abstraction may not sufficiently capture embodied action intelligence.

The central contribution of HumanCLAW is the claim that it measures action decision-making while retaining physically grounded consequences. However, the humanoid is driven by kinematic velocities derived from generated motion rather than by joint-level dynamics and torque-based control. This design intentionally removes balance and motor-tracking failures, but it also changes the nature of the interaction between the body and the environment.

For example, the simulator does not require the agent to maintain dynamic balance, recover from disturbances, or account for the feasibility of a motion under contact constraints in the same way as a real humanoid. These factors are not merely low-level execution details; they can influence which action should be selected next. A decision that is appropriate for a kinematically driven body may not be appropriate for a dynamically controlled humanoid.

The paper therefore measures a particular intermediate form of embodied decision-making rather than action intelligence in the broader sense implied by the framing. The authors acknowledge that transfer to physical humanoids remains future work, but this limitation directly affects the interpretation of the central contribution. More validation is needed to establish which decision-level conclusions remain consistent when the same tasks are evaluated with a fully dynamic controller or a physical robot.

W2. The embodied self-awareness conclusion is stronger than the available evidence.

The paper concludes that current VLMs lack embodied self-awareness, particularly the ability to estimate their own body state and understand the physical consequences of their actions. However, the evaluated VLMs receive only egocentric RGB observations and textual action history, without proprioceptive body state, explicit contact signals, or other structured state information.

This creates an important ambiguity. The observed failures may reflect a limitation in the models' internal representations, but they may also arise because the input interface does not provide sufficient information to infer body configuration and contact state reliably. The authors briefly acknowledge this possibility in the discussion, but the experiments do not distinguish between these explanations.

For instance, the failure to recognize that the body is jammed against an obstacle could be caused by insufficient visual observability rather than an intrinsic inability to reason about body state. Similarly, sitting in the wrong location may reflect a lack of explicit geometric or proprioceptive feedback.

A more convincing analysis would compare the RGB-only setting with variants that provide body-state estimates, contact information, or explicit geometric feedback. Such experiments would help determine whether the bottleneck is primarily visual reasoning, state observability, or embodied representation.

W3. The benchmark task coverage is too narrow to support broad conclusions about action intelligence.

HumanCLAW-Bench primarily evaluates indoor navigation followed by sitting on a target object. Although the episodes involve long-horizon decision-making and physical interactions, the interaction component is limited to a relatively simple action. The benchmark does not substantially test object manipulation, articulated interaction, pushing, pulling, carrying, tool use, or complex contact-rich behaviors.

This is important because the paper positions action intelligence as a general capability that converts spatial understanding into closed-loop embodied decisions. The current benchmark mainly evaluates spatial navigation, body positioning, and sitting. These are useful capabilities, but they do not cover the broader range of embodied action decisions implied by the paper.

The authors should either narrow their claims to egocentric navigation and simple whole-body interaction or provide additional task families that demonstrate the generality of the framework.

Questions for authors

Validity of the half-physics abstraction: How well do the decision-level failure patterns observed in HumanCLAW correlate with those observed under fully dynamic humanoid control? Can the authors provide evidence that the main conclusions remain valid when balance and contact-dependent motion feasibility are no longer abstracted away?

Dependence on the atomic skill vocabulary: How sensitive are the results to the number and granularity of available skills? Would the conclusions change if the VLM were given a richer action space or additional interaction primitives?

Embodied self-awareness versus missing observations: Have the authors tested whether providing proprioceptive state, contact information, or explicit body geometry improves navigation and interaction success? Such experiments would help distinguish limitations in model reasoning from limitations in the observation interface.

4 comments
AuthorsOpenReview ↗

[1/4] W1,Q1: Validity of the half-physics abstraction.

We agree that balance and dynamics can influence action selection in fully dynamic humanoids. However, our goal is not to evaluate "Can VLMs control physical humanoids?" but rather "Can VLMs translate spatial understanding and reasoning into sub-second atomic action decisions in a physically grounded loop?"

In fully dynamic settings, the same action decision may succeed with controller A but fail with controller B due to balance or tracking errors. Consequently, task failure becomes difficult to attribute: did the VLM make a poor decision, or did the controller fail to execute it? HumanCLAW addresses this ambiguity by abstracting away motor-control failures while preserving physical consequences, including collisions, contact, friction, gravity, and object displacement.

Following the reviewer's suggestion, we additionally evaluate two full-physics alternatives on val-100: (a) direct PD tracking and (b) SONIC, a learned whole-body controller.

Execution MethodFindSR ↑NavSR ↑InteractSR ↑Fall Rate ↓
Full physics (PD)17.0%0.0%0.0%100%
Full physics (SONIC)46.0%16.0%3.8%8%
Half-physics (Ours)58.0%27.0%18.9%0%

Direct PD tracking typically results in falls within approximately three action steps, while SONIC also experiences balance failures under relatively minor collisions. These failures can terminate episodes before the VLM's long-horizon decision-making can be meaningfully assessed, illustrating how full-physics evaluation becomes confounded by controller performance.

We emphasize that half-physics is not intended to replace fully dynamic humanoid evaluation. Rather, it provides a controlled setting that reduces confounding effects from motor execution when studying atomic action decision-making, while retaining the physical consequences needed to evaluate embodied decisions.

AuthorsOpenReview ↗

[2/4] W2, Q3: A more convincing analysis would compare the RGB-only setting with variants that provide body-state estimates, contact information, or explicit geometric feedback.

We agree that direct sensing and inference from vision must be distinguished.

Our intended meaning of embodied self-awareness is the ability to estimate one's body state (especially from ego view), its relation to the scene, and the outcomes of recent actions during a closed-loop task, not direct access to every joint or contact event.

As requested, we conduct additional ablations on the same 100-episode mini-val split with Gemma-4-31B, providing: (1) explicit body states given as text: SMPL-X joint rotations, plus the root turn and body-frame displacement actually produced by the previous action; (2) an additional rear RGB camera; (3) privileged geometry, including ground-truth distances to surrounding objects; and (4) privileged contact information for each body region.

ModelFindSR ↑NavSR@20cm ↑InteractSR ↑Coll. ↓#Dtb/ep ↓dDtb(m) ↓Jerk ↓Avg. stepsInput tok/stepOutput tok/step
Baseline58.0%27.0%18.9%41.4%1.359.384.978.54676322
+ Body state (SMPLX joint)63.0%20.0%1.9%48.9%1.256.974.5784.26077342
+ Rear camera70.0%31.0%20.8%38.6%1.022.274.8475.05327328
+ Geometry (previlaged)64.0%38.0%26.4%36.2%0.891.745.0576.05354326
+ Contact (previlaged)68.0%36.0%22.6%41.3%1.150.975.0073.15941331

(1) Explicit body states do not improve performance and even substantially reduce InteractSR (18.9% → 1.9%), suggesting that the evaluated model struggles to use body states in this raw, high-dimensional form.

(2) A rear camera improves FindSR (58% → 70%), but provides only a small improvement in interaction (18.9% → 20.8%).

(3) Privileged geometry and contact feedback improve navigation by 11 and 9 percentage points, respectively, but InteractSR remains low (26.4% and 22.6%). Explicit geometry also reduces collisions (41.4% → 36.2%).

Importantly, some of these additional inputs are privileged information that partially bypasses the very capability we aim to evaluate: whether VLMs can estimate their embodied state and action outcomes, rather than being directly given such information.

Our original finding remains supported. Privileged geometry and contact feedback improve navigation but do not eliminate the observed failures: InteractSR remains low (26.4% and 22.6%, respectively). This suggests that limited observability is not the only contributing factor in this setting. Moreover, on the same mini-val split, GPT-6 Astra reaches 50.9% InteractSR using only egocentric observations, showing that the egocentric interface itself does not prevent substantially higher performance. The key challenge remains whether VLMs can estimate and use their embodied state to guide subsequent actions.

AuthorsOpenReview ↗

[3/4] W3: Limited task coverage relative to action intelligence

We thank the reviewer for this suggestion. While recent work has explored VLM-based robotic arm control, HumanCLAW takes an initial step toward evaluating sub-second atomic action decision-making through a full human body. We introduce action intelligence as a research concept, but do not claim that our benchmark comprehensively measures general action intelligence.

To further address task diversity, we introduce a preliminary Reach–Pick–Carry–Place sub-benchmark. It contains 100 tasks across 10 indoor layouts, 5 object categories (boxes, teddy bears, balls, vases, and baskets), and 2 object sizes. Initial hand–object distances range from 0.12 to 6.18 m. We add IK-based upper-body end-effector control alongside the existing locomotion skills, allowing both to execute concurrently.

We evaluate four progressive milestones: ReachSR (hand contact or within 10 cm), PickSR (lifting the object ≥1 cm without environmental support), CarrySR (transporting the object reaching within 0.5 m of the destination), and PlaceSR (releasing it with stable support). We additionally measure object fall rate. Each episode allows up to 150 action steps.

ModelReachSR ↑PickSR ↑CarrySR ↑PlaceSR ↑Fall rate ↓
Astra66.0%8.0%1.0%0.0%16.0%
Gemma-4-31B42.0%6.0%0.0%0.0%17.0%
InternVL-3.5-38B15.0%0.0%0.0%0.0%7.0%

The results reveal a substantial gap between reaching and manipulating objects. Even GPT-6 Astra achieves only 8% PickSR, 1% CarrySR, and 0% PlaceSR. Six of the eight successful pickups occur in the layout where the object starts within arm's reach (0.12–0.32 m from the palms). Qualitative failures include poor coordination between body approach and reaching, as well as difficulty maintaining a grasp during locomotion. In the sole successful carry episode, GPT-Astra brings a teddy bear near the destination table but drop it before reaching table.

This extension broadens HumanCLAW from navigation and sitting to whole-body loco-manipulation, involving object interaction, object transport, and placement. These preliminary results suggest related challenges in coordinating whole-body actions and predicting their outcomes.

AuthorsOpenReview ↗

[4/4] Q2: How sensitive are the results to the number and granularity of available skills?

SettingFindSR ↑NavSR ↑InteractSR ↑
Baseline58.0%27.0%18.9%
−3 Skills (upstairs, downstairs, side-step)61.0%28.0%17.0%
MoMask (lower execution fidelity)66.0%17.0%0.0%

(1) Number of skills. We remove three skills (upstairs, downstairs, and side-step). Performance remains comparable to the baseline: 61.0% / 28.0% / 17.0% vs. 58.0% / 27.0% / 18.9% (Find/Nav/Interact). Removing these three skills has little effect on performance in our mini-val setting.

(2) Skill granularity. We replace our motion generator with MoMask, which has much lower parameter-following accuracy (Table 4). Interestingly, FindSR increases to 66.0%, while NavSR drops to 17.0% and InteractSR collapses to 0.0%.

Comments on the paper

4 comments

We thank all reviewers for their constructive comments. HumanCLAW is an evaluation framework that separates a VLM's action decisions from low-level motor execution, and HumanCLAW-Bench uses it to test whether current VLMs can act through a body in 1,218 long-horizon egocentric episodes.

We are encouraged that all four reviewers consider the problem important and well motivated. The main concerns ask for additional experiments, in particular (i) richer input interfaces for the VLM, (ii) a broader set of tasks, and (iii) further validation of the execution pipeline. We have addressed each of them:

  1. 🌟Additional inputs (PqMC W2, uxv8 W2, 8Sjj W2, j7dQ W2). We add four input variants to the harness ablation: explicit body state, a rear camera, privileged geometry, and privileged contact (Table 2 and Section 4.3). Privileged geometry and contact improve NavSR by 11 and 9 points, but InteractSR remains low (26.4% and 22.6%, against 18.9% for the baseline), which suggests that limited observability is not the only factor.

  2. 🌟Broader tasks (PqMC W3, uxv8 W3, 8Sjj W1, j7dQ W1). We introduce a preliminary Reach–Pick–Carry–Place sub-benchmark for whole-body loco-manipulation with 100 tasks (Appendix F). GPT-6 Astra reaches the object in 66.0% of the tasks but achieves only 8.0% PickSR and 0.0% PlaceSR.

  3. 🌟Task solvability and execution pipeline (uxv8 W1, W3; 8Sjj Q1). We add a privileged Oracle that uses the same atomic skills, motion generator, and simulator without any VLM call (Table 1 and Section 4). It achieves 98.3% NavSR and 99.0% InteractSR, so the execution pipeline is not the main bottleneck.

  4. 🌟Further analyses reported in the individual responses. These include a comparison with two full-physics controllers (PqMC W1), an ablation of the skill set and of the motion generator (PqMC Q2, 8Sjj Q1), and the fidelity of skill transitions (uxv8 W1).

We have also revised the wording of our claims about embodied self-awareness and action intelligence so that they match the evidence more closely. All changes in the revised paper are marked in cyan.

We respond to each reviewer in detail below. We appreciate your further response and discussions.

Dear Area Chair,

We would like to raise a concern about the reviews of our submission.

The four reviews are highly consistent: they raise the same weaknesses and ask for the same two remedies, more ablation studies and more benchmark tasks, and three of them give identical scores on every item. When we asked ChatGPT to review our paper, it produced the same score and the same points; apart from their order, the content is the same.

Requests for an ablation on each prominent statement and for further benchmark tasks are, in our experience, characteristic of LLM-generated reviews. We have provided all of them in the rebuttal and the revision. Our concern is that reviews of this kind do not weigh the novelty and insight of the work.

We kindly ask you to consider this when weighing the reviews, and to assess the contribution of the paper independently.

Sincerely, ICLR 2027 Conference Submission5166 Authors

Dear Reviewers,

Thank you again for your time and thoughtful feedback. As the discussion period is drawing to a close, we would like to kindly ask whether you have any further comments on our rebuttal.

If our responses have addressed your concerns, we would greatly appreciate it if you could let us know and consider updating your score. If any concerns remain, we would be glad to discuss them further and provide additional clarification.

Best regards, The Authors

Dear Reviewers,

Thank you again for your time and thoughtful comments. We greatly value your feedback. As the discussion period is coming to an end, we would appreciate hearing whether our responses have addressed your concerns.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.