acceptodds
Under review as a conference paper at ICLR 2027

CrossBFM: Distilling a Shared Latent Behavior Space Across Humanoid Embodiments

Abstract

Behavior Foundation Models (BFMs) give humanoids a promptable policy over a latent behavior space, enabling one single vector to represent a motion to imitate, a pose to reach, or a reward to maximize. Forward-Backward representations successfully produce such spaces, but at the cost of hundreds of GPU-hours for a single robot. Moreover, when the training process is repeated for a second robot, it produces a second space unrelated to the first, resulting in embodiment-specific latents that do not unify or transfer. We address these problems with CrossBFM, treating the latent space as the transferable asset for various embodiments. As retargeting provides frame-level cross-embodiment correspondence, we propose a unified encoder architecture with no robot-specific parameters for distilling the behavior space to address all training embodiments simultaneously in less than a GPU-hour. Following this encoder, latent-conditioned trackers turn the distilled latent into whole-body control in a conventional PPO training manner in just 10 more GPU-hours. On three distilled humanoids, all three prompting modes transfer: motion tracking with latent-conditioned policy losing only 0.025 rad to its joint-conditioned counterpart, smooth goal reaching between poses with no falls, and reward optimization for all 41 reward prompts. Our experiments further reveal that 1) regressing the encoder on a quarter of the motion corpus costs only 5% of tracking performance and 2) training the encoder on a subset of robots and evaluating on an unseen one recovers up to 89% of the tracking performance of seen robots, demonstrating cross-embodiment generalization to morphologically similar robots. We also verify the pipeline on real robots across all three prompting modes and with flow-based generated latents. Project website: https://crossbfm.github.io/

Primary area:
applications to robotics, autonomy, planning
Keywords:
Behavior Foundation Model, Cross-Embodiment Transfer, Humanoid Whole-body Control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9YBoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work presents a method for creating a shared behavior latent space that can be shared across humanoid robots with different morphologies. Starting with a pre-trained BFM on a single robot, such as the G1, the method distills the encoder into a general encoder that can be shared by robots with different morphologies. First a retargeted dataset is created for the target robots that provide a one for one correspondence between motions from the source robot to the target robots. Given the retargeted motions, an encoder is then trained to map the motions from different robots to the same latents as the source robot, using the source encoder as supervision. Once the latent space is learned, it can be used as a reference motion encoder to train tracking controllers on different robots, without the need to directly condition the tracking controllers on the target motion.

The general input representation is proposed for general encoder that can handle inputs from robots with different morphologies and degrees of freedom.

Experiments are conducted on 4 robots. Tracking, goal reaching, and optimization performance are shown for the different robots.

Critical strengths

The method itself is fairly simple and straightforward and the results look promising. It is encouraging to see that the latent representation learned for a G1 can be effective for a variety of different robots.

Distilling from a pre-trained G1 latent space can significantly reduce the training cost required for new robots. Cutting training down from over 100 hours to about 10 hours for each new robot.

Critical weaknesses

Beyond faster training time, it’s not clear what the benefit of reusing a G1 latent space is compared to training robot specific models. After distilling the G1 latent space into a shared encoder, separate motion trackers still need to be trained for each new robot. While training the motion tracker, a new motion encoder could also be trained from scratch alongside the tracker. So from the current experiments, it’s not clear what the benefits are from using the frozen encoder distilled from the G1 encoder.

The distilled model is only trained on three different robots. So it is unlikely that it will generalize well to new robot morphologies. Table 4 shows that performance can degrade significantly when testing on unseen robots. Therefore, if new robots are added, then it seems that a new encoder will still need to be trained for finetuned on the new robot. This doesn’t show much benefit in having the shared encoder.

Very little discussion on flow-based latent generator in experiments. It’s not clear how the flow model was trained, or how the experiments were set up. If this is an important component of the paper, then more clarification is needed on these experiments.

Questions for authors

Table 1 compares the proposed method with a tracker that directly receives the reference motion as input. It appears that performance is evaluated on the training motions. How well does the joint and latent model generalize to unseen motions that are different from the training set (not simply different variations of similar motions in the training set)?

Table 3 compares performance on training and validation motions using the general encoder vs the oracle latents. But it’s not clear how the oracle latents are determined. Are they inferred using the original G1 source encoder?

For experiments such those in Table 2 and Table 3, an important baseline is missing, which is training robot specific BFM models for each robot. Comparisons are mainly to models using the original G1 model, which may not provide a good estimate of the performance of models trained specifically for each robot. Without this baseline, it’s difficult to assess how much performance the proposed method is sacrificing compared to training robot specific models.

If the goal is to construct a general behavior latent space that can be shared for different robots, is it really optimal to use a latent space that is trained specifically for the G1? It seems that a more suitable latent space can be obtained by optimizing the latent space directly for a wide variety of robots as opposed to only the G1. If that is the case, it’s not clear why the proposed method using a G1 latent space would be preferred over a latent space that is optimized for a wider variety of robots.

While there have not been much prior work on training backward and forward models that can be applied to different morphologies, there have been previous works on training general tracking controllers that can be applied to different morphologies: Learning body shape variation in physics-based characters These and similar method should be used as baselines for the tracking experiments. The current experiments only compare to variations of the proposed method, and little to no comparisons are provided to previous work that tackle similar problems.

How does performance compare if instead of distilling from the G1 encoder, the tracking controller for each robot was trained with a frozen randomly initialized encoder? Is using the G1 encoder offering useful features compared to a random encoder, especially for the tracking experiments? Note that for this to be a meaningful experiment, some care is needed in how the random encoder is initialized. For example, the random encoder should be initialized in a way that does not map all target motions to the same point in the latent space.

Minor questions:

Section 5.2: how is the validation set constructed?

Line 310: “…medium robot remarked by 1.0m hand height would…”, what does “remarked” mean in this context?

Line 398: what does “maintaining good balance for the whole trajectories” mean? What is “good balance” referring to?

Reviewer_EPQMOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

CrossBFM transfers the latent behavior space of a pretrained G1 behavior foundation model to other humanoids. The same motion trajectories are retargeted to each robot, providing frame-level correspondence across embodiments. A shared encoder is then trained by supervised cosine regression to predict the frozen G1 model’s latent at each corresponding frame from the target robot’s motion window. Each target robot learns a separate latent-conditioned tracking policy using PPO. For reward prompting, the method aggregates encoded states from a target-side motion cache, with reward thresholds adjusted for body proportions. Experiments on three target humanoids evaluate tracking, goal reaching, and reward prompting, alongside studies of encoder data efficiency and generalization to held-out motions and embodiments. The paper also reports hardware results and demonstrates a flow-based latent generator.

Critical strengths
  1. A practical formulation of cross-embodiment behavior transfer. The paper exploits frame-level correspondence between retargeted trajectories to distill a frozen BFM’s latent space through supervised regression. This is a simple but effective way to reuse an existing behavior representation without repeating expensive FB pretraining for every robot. The shared encoder accommodates different joint configurations, and Table 16 shows improved cross-robot latent consistency compared with separate encoders. Together, these provide a useful foundation for a common behavior interface across embodiments.
  2. Convincing tracking results supported by hardware validation and ablations. Under matched training settings, the latent-conditioned policies remain within 0.025 rad of the joint-conditioned baselines across three target humanoids, with a similar accuracy gap on M3 hardware. This is a reasonable trade-off for the shared interface. The studies of encoder data efficiency, causal attention, and retargeting errors further help explain when the approach works and where it degrades, strengthening the practical evidence beyond representation-alignment metrics alone.
Critical weaknesses
  1. Limited task-level evaluation of reward-prompting transfer. Table 2 reports average returns across 41 reward tasks. Although the aggregate improvement is encouraging, it remains unclear whether the gains extend across task categories and which capabilities remain difficult to transfer. Results by task category and an analysis of failures would better establish how consistently reward-prompting capabilities are preserved.
  2. Limited explanation of transfer gains across embodiments. In Table 2, target-side prompting improves returns over source-side prompting by approximately 3.50× on M3 and 2.95× on N1, but only 1.14× on T1. The paper provides limited analysis of why these gains differ so substantially, leaving the factors governing transfer effectiveness and the conditions under which the method is most beneficial insufficiently understood.
Questions for authors
  1. Could you provide a breakdown of the Table 2 returns by task category for both source-side and target-side prompting? Which categories benefit consistently, and are there tasks where target-side prompting performs worse? This would help assess how broadly reward-prompting capabilities transfer.
  2. What explains the smaller improvement on T1 compared with M3 and N1? Does the task-level breakdown reveal particular behaviors responsible for this difference, and is there evidence linking it to motion-data coverage, encoder alignment, or target-policy performance?
  3. For clarification, I assume both prompting methods in Table 2 are evaluated using the same target-side reward function, thresholds, and rollout protocol. Could you confirm this?
Reviewer_3wSKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CrossBFM, which reuses the latent space of a pretrained Forward–Backward behavior foundation model for other humanoids. The same motion clips are retargeted to every robot, so each target frame can be labeled with the G1 latent of the matching frame. A single encoder shared by all robots is trained to predict these labels. Each robot then gets its own policy that is driven by the latent instead of joint targets. In experiments on three humanoids, this policy tracks motions almost as well as a standard joint-target tracker. The paper also shows goal reaching, reward prompting, transfer to a held-out robot and real-robot runs.

Critical strengths
  • Using retargeted clips as correspondence turns transfer into plain regression. The encoder stage is simple, fast and easy to reproduce.
  • The latent tracker and the joint-conditioned tracker are compared fairly. They share rewards, terminations, randomization and the PPO budget, differ only in the actor input, and are reported over three seeds
  • The writing is clear, and the split between encoder distillation and policy training makes the pipeline easy to follow.
Critical weaknesses

W1. Distilling only BB does not carry over the BFM prompting modes.

  • The paper says that matching the source latent space on a new robot gives all three prompting modes (Sec. 2, end of Sec. 3.3). But BB only turns a prompt into a latent. Acting on that latent is the job of πz\pi_z, which maximizes F⊤zF^\top z (Eq. 3). The target robots have neither FF nor πz\pi_z. They only have a PPO tracker trained to follow references, using per-frame G1 latents (Sec. 4.4, App. A.5).
  • So tracking and goal reaching work because the tracker learns to reach ss from B(s)B(s), not because BFM semantics transfer. The label is also BB of one frame (Eq. 5), not the look-ahead sum of Sec. 3.3.
  • Reward prompting is where the closed form matters most, and it uses a new procedure (Eq. 8) that averages dataset latents. The tracker was not trained on such averaged latents. It works to some extent (Table 2), but the paper does not explain why.

W2. The benefit of the FB latent is not isolated.

  • There is no non-FB latent baseline. The same tracker could be trained on another latent, for example an autoencoder latent learned on the same LAFAN data. That would show whether goal transitions, reward prompting and the flow generator depend on the FB space. This is the key experiment for why a BFM should be distilled at all.
  • Table 2 has a single weak baseline. It samples from the G1 RL buffer, which can contain poses the target cannot reach, and it may not use the transported thresholds. The 2.35× gain mostly shows that sampling from the target's own data helps. There is also no reference point, such as the original G1 BFM on the same tasks, to judge the absolute returns.

W3. It is unclear what the encoder is needed for.

  • The trackers are trained on the oracle latent z⋆z^\star from the G1 retarget (App. A.5, Sec. 5.4). This includes the held-out robot in Table 4.
  • All motions come from LAFAN, so a G1 retarget always exists. Deployment already needs a retargeted reference anyway (Sec. 4.5). Retargeting to G1 and applying BSB_S would therefore give z⋆z^\star directly, as in the oracle condition of Table 3.
  • The encoder only matters for data that exists solely on the target robot, and this case is not tested.
  • The unified encoder's advantage over per-robot encoders is higher agreement between robots, while alignment is similar or slightly lower (Table 16). Agreement would matter for running one robot's motion on another robot, which is not evaluated.

W4. It is unclear which latent produced the main results.

  • The setup is described inconsistently in two places.
    • App. A.5 says the trackers use the frozen G1 latent, but App. D says the corpus latents come from the Stage-1 encoder.
    • Table 8 lists bidirectional attention as the default, while Sec. 4.3 and 4.5 describe a causal encoder.
  • The tracking errors are hard to reconcile.
    • At full data: 0.2043 / 0.2068 in Table 15 (robot not stated) vs. 0.2132 in Fig. 4.
    • At 25% data: 0.2045 / 0.2129 in Table 15 vs. 0.2241 in Fig. 4.

W5. The data-efficiency result is narrower than stated.

  • The reduced sets in Sec. 5.3 take segments from all 40 motions, which likely include the 10 validation clips (Table 9). So neither the 25% setting nor the validation-clip result in Fig. 3b tests unseen motions.
  • Only the encoder uses reduced data; the tracker uses all of it. The abstract states this correctly, but the conclusion (Sec. 6) reads as a claim about the whole pipeline. The result is also shown only on M3.

W6. The cost comparison and related work are overstated.

  • "Under one GPU-hour vs. about 10210^2 GPU-hours" (Sec. 4.3) compares fitting existing labels with creating them.
    • A new robot still needs retargeting, the encoder, and about 10 hours of PPO. That PPO budget is the same as the joint-conditioned tracker's (App. A.5).
    • A pretrained source BFM is also required.
    • Contribution 2's claim that the method removes the simulator and RL training holds only for the backward map.
  • The introduction describes prior cross-embodiment work as single-task. Multi-humanoid motion trackers that follow arbitrary motions, such as EAGLE and VENOM, are not discussed. Since tracking is the main evaluated mode, they are relevant comparisons.

References not cited in the paper:

  • Q. Peng, Y. Lin, Y. Xue, J. Pang, and W. Zhang. Embodiment-Aware Generalist Specialist Distillation for Unified Humanoid Whole-Body Control (EAGLE). arXiv:2602.02960, 2026.
  • S. Padmanabhan, K. Miyazawa, and T. Horii. VENOM: Versatile Embodied Network for Omni-bodied Motion tracking. arXiv:2606.16696, 2026.
Questions for authors
  1. How does the same tracker perform with a non-FB latent, for example an autoencoder latent learned on the same LAFAN data? Please report tracking, goal reaching, and reward prompting with Eq. 8.

  2. How does Eq. 8 compare with simply passing the latent of the highest-reward frame, or the top-k frames? What does the original G1 BFM score on the same 41 tasks? Was threshold transport applied to the source baseline in Table 2?

  3. Could you report Table 2 by task group (Table 13)? The latent is fixed for 300 steps (App. B.2). Do the locomotion and rotation tasks produce sustained motion?

  4. When is the encoder required, given that z⋆z^\star can be computed from a G1 retarget? Could you show results on data recorded only on the target robot, or run one robot's motion on another robot's tracker?

  5. App. A.5 and Sec. 5.4 say the trackers were trained on the frozen G1 latent z⋆z^\star, but App. D says the corpus latents come from the Stage-1 encoder. Which is correct? For Table 1, is the policy driven by encoder outputs or by z⋆z^\star, which encoder setting (causal or bidirectional) is used, and which clips are evaluated?

  6. Sec. 5.3 builds the reduced training sets from segments of all 40 motions. Does this include the 10 validation clips from Table 9? If so, the encoder has already seen parts of those clips, and Fig. 3b no longer shows generalization to unseen motions. Which clips are used for evaluation in Fig. 4 and Fig. 3b?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.