acceptodds
Withdrawn from ICLR 2027

HuGe: Scaling Humanoid Generalist Vision-Language-Action Models with Egocentric Experience

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

Extending Vision-Language-Action models (VLAs) to humanoid robots requires coordinated whole-body control and diverse demonstrations that are costly to collect in real-world scenarios. Although egocentric human demonstrations provide a complementary source of task experience, differences in visual embodiment and missing robot proprioception complicate transfer to humanoid policies. To address these challenges, we introduce HuGe, a family of Humanoid Generalist VLAs pre-trained on over 2,000 hours of egocentric human and robot data. Specifically, HuGe employs three existing representative VLA architectures to predict latent motion tokens for a pretrained whole-body controller, together with hand-control signals. Based on this, we introduce a three-stage training pipeline that combines egocentric pre-training, humanoid adaptation, and downstream fine-tuning. To facilitate transfer across embodiments, we construct pseudo-proprioceptive states through simulation playback during pre-training and modify arm appearance in egocentric observations through re-rendering during downstream fine-tuning. With this pipeline, HuGe achieves the state-of-the-art on simulation benchmark HumanoidArena. Furthermore, real-world experiments on the Unitree G1 indicate that augmenting robot demonstrations with visually adapted egocentric demonstrations significantly increases success on a pick-and-drop task.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Humanoid Vision-Language-Action Models, Whole-Body Control, Pre-Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YB3sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HuGe, a training framework for humanoid VLA policies built on three existing VLA architectures. The main contribution is a three-stage training recipe consisting of large-scale egocentric pretraining, humanoid adaptation, and downstream task fine-tuning. The paper additionally introduces simulation-derived pseudo-proprioceptive states for egocentric data and visual adaptation that replaces human arms and hands with rendered robot counterparts. The training corpus exceeds 2,000 hours. HuGe obtains strong results on HumanoidArena and shows that egocentric demonstrations can improve a real-world Unitree G1 pick-and-drop task.

Critical strengths

The paper provides a useful scaling study across three VLA families rather than relying on a single architecture. This gives the central result broader relevance and suggests that the benefit of egocentric pretraining is not tied to one particular model design. The empirical analysis is also fairly comprehensive at the training-recipe level. The paper studies full fine-tuning, pseudo-proprioception, pretraining scale, data composition, and visual adaptation, and includes both simulation and physical-robot experiments.

Critical weaknesses

The main concern is that the reported benchmark improvements are not cleanly separated from data and compute scale. HuGe benefits from a very large additional training corpus and a dedicated humanoid adaptation stage, while several baseline results are taken from the original HumanoidArena evaluation. The current results establish strong performance, but do not fully identify how much of the gain is due to the proposed transfer recipe rather than substantially greater pretraining. The real-world validation is narrow relative to the generalist framing. The physical evaluation consists of a single pick-and-drop task with a largely fixed task structure. Although it requires locomotion and manipulation, it does not demonstrate broad task composition, semantic variation, or transfer across substantially different humanoid skills. In addition, the improvement attributed specifically to visual adaptation is from 18/30 to 20/30 successes, which is too small to interpret confidently without uncertainty estimates.

Questions for authors
  1. Can the authors provide a data- and compute-matched control using the same backbone and approximately the same optimization budget, but without the proposed egocentric pretraining recipe? This would help separate the effect of the method from the effect of scale.
  2. For the real-world experiment, can the authors report confidence intervals or an appropriate statistical comparison for the 13/30, 18/30, and 20/30 success rates? In particular, the evidence for the additional benefit of visual adaptation is currently limited.
  3. In the pretraining scaling experiment, are optimization steps and total sample exposure controlled when moving from 10% to 100% of the egocentric dataset? If not, the result may conflate data scaling with additional optimization.
  4. Pseudo-proprioception improves the GR00T and Pi variants but degrades the OFT variant. Can the authors test whether this is primarily caused by textual state serialization and truncation rather than by pseudo-proprioception itself?
Reviewer_g7gXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HuGe, a family of humanoid VLAs pre-trained on over 2,000 hours of egocentric human and robot data. It instantiates three existing VLA architectures (OFT-, GR00T-, and π-style) to predict 64-D latent motion tokens for SONIC whole-body controller, together with two binary hand-control signals. The model is trained with a three-stage pipeline: egocentric pre-training (~1,670 h, including 322 h of simulated BEHAVIOR-1K robot data), humanoid adaptation on real G1 teleoperation data (~418 h), and downstream fine-tuning. To make better use of human data, it constructs pseudo-proprioceptive states through simulated G1 replay during pre-training, and replaces human arms with rendered robot hands using Phantom during downstream fine-tuning. HuGe-GR00T and HuGe-Pi reach good performance on HumanoidArena as well as one real-world G1 pick-and-drop task.

Critical strengths

Scaling heterogeneous pre-training data for humanoid VLAs is a promising direction. The paper converts ~2,087 h of human, simulated, and real-G1 data into a shared SONIC action and state interface, with enough detail (App. A–B) to be reproducible. The recipe is evaluated across three action-head designs without architectural changes, and a real-world G1 experiment shows a long-horizon loco-manipulation task (approach, grasp, turn, walk, crouch to place).

Critical weaknesses

The experiments do not fully isolate the contribution of egocentric pre-training, which is the central claim. All components (StarVLA heads, SONIC, Phantom, datasets) already exist (L96, L132), so the paper should be judged on whether the recipe is validated, and it is not yet. There is no stage-wise ablation (Stage 1 only, Stage 2 only, Stage 1+2), so the pre-training gain in Fig. 2 also includes 418 h of teleoperated G1 data from Stage 2, and the contribution of stage 1 pre-training is never measured on its own. The gain in simulation environment is also modest: +8.34 (GR00T), +4.76 (Pi), and −0.95 pp (OFT) (Fig. 2). Without pre-training, HuGe-Pi (60.24) and HuGe-GR00T (58.33) already exceed π0.5 (53.81), so it remains to be justified that much of the SOTA margin in Table 4 comes from egocentric data. In real world experiments, the authors even attribute the 0–15% DoubleDesk success, below ACT, DP, and π0.5 (20–35%), to pre-training (L367–368). This may suggest that the pre-training data can also limit fine-grained manipulation. The central scaling-egocentric-data claim is unsupported. The real-world evidence is limited and overclaimed. There is only one task, one architecture (Pi), and 30 trials per configuration, yet the paper overclaims it as "comprehensive real-world experiments" (L70, L77). All three configurations mentioned in real world experiments (L462) use the fully pre-trained model and vary only the Stage-3 data, so there is no real-world ablation of pre-training or pseudo-proprioception. Since pseudo-proprioceptive states are produced by simulated replay and only validated in simulation (Table 3), it is unclear whether they help on hardware. In the meantime, although the title claims a "Generalist" policy pre-trained on diverse data, generalization is only evaluated by bottle placement on one task and one qualitative perturbation (Fig. 3e), without testing on unseen objects, scenes, instructions, or tasks. Paper needs further clarification. The method section is overly detailed on per-dataset conversion (Sec. 3.2), but does not motivate the three-stage design, the stage order, or comparison to co-training on all data.

Questions for authors

What does "From Scratch" (Fig. 2) and "without pre-training" (Table 2) include? Are both Stage 1 and Stage 2 skipped? Could you report Stage 1 only, Stage 2 only, and Stage 1+2 results on HumanoidArena with variance over multiple training seeds? Could you show the generalization of the fine-tuned model, like its capability on other tasks? Could you report the performance of the trained models on more real world tasks? Is the OFT state truncation problem (L324) fixable? If so, could you provide the corresponding results after fixing the issue?

Reviewer_rZoYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a VLA that takes images, instructions, and robot states as input to predict WBC actions using a frozen SONIC. It combines over 2k hours of mixed human and robot data, using simulation-generated body states and generated arms to transfer human demonstrations to robots. The best model achieves 66.67% SR on 7 Humanoid Arena tasks, while adding visually adapted human demonstrations improves real-world tasks' success from 43.3% to 66.7%.

Critical strengths
  • Practical use of human data. The paper combines human and robot demonstrations in a shared whole-body action space. Its pseudo-state ablation shows improved average success for two VLA variants.

  • Evidence from a real robot. In the G1 pick-and-drop task, adding human demonstrations improves success from 13/30 to 18/30 while keeping the robot demonstrations fixed. This provides evidence that human data can help with downstream training.

Critical weaknesses

I have the following concerns:

  • The main concern is the novelty: SONIC already supports VLA-based WBC, EgoScale and Psi0 already use human egocentric pretraining followed by robot adaptation, and GRAIL has shown visual policies that predict SONIC latents for humanoid control. HuGe’s main addition is a training recipe using real egocentric demonstrations and pseudo-states. All components seem to exist.

  • The training recipe is not consistently effective across architectures. Pseudo-states improve the GR00T- and Pi-style models, but reduce the OFT-style model’s success rate. The authors attribute this to input truncation, but do not report results after fixing it.

  • Generalization to new tasks, objects, and environments is unclear: all seven simulation tasks are used for fine-tuning, and the real-world evaluation covers only one task with varying bottle positions.

Questions for authors
  1. Could you provide the OFT results after fixing the truncation issue?
  2. Does the “From Scratch” baseline in Figure 2 include stage 2 humanoid adaptation before downstream fine-tuning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.