- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work presents a method for creating a shared behavior latent space that can be shared across humanoid robots with different morphologies. Starting with a pre-trained BFM on a single robot, such as the G1, the method distills the encoder into a general encoder that can be shared by robots with different morphologies. First a retargeted dataset is created for the target robots that provide a one for one correspondence between motions from the source robot to the target robots. Given the retargeted motions, an encoder is then trained to map the motions from different robots to the same latents as the source robot, using the source encoder as supervision. Once the latent space is learned, it can be used as a reference motion encoder to train tracking controllers on different robots, without the need to directly condition the tracking controllers on the target motion.
The general input representation is proposed for general encoder that can handle inputs from robots with different morphologies and degrees of freedom.
Experiments are conducted on 4 robots. Tracking, goal reaching, and optimization performance are shown for the different robots.
Critical strengths
The method itself is fairly simple and straightforward and the results look promising. It is encouraging to see that the latent representation learned for a G1 can be effective for a variety of different robots.
Distilling from a pre-trained G1 latent space can significantly reduce the training cost required for new robots. Cutting training down from over 100 hours to about 10 hours for each new robot.
Critical weaknesses
Beyond faster training time, it’s not clear what the benefit of reusing a G1 latent space is compared to training robot specific models. After distilling the G1 latent space into a shared encoder, separate motion trackers still need to be trained for each new robot. While training the motion tracker, a new motion encoder could also be trained from scratch alongside the tracker. So from the current experiments, it’s not clear what the benefits are from using the frozen encoder distilled from the G1 encoder.
The distilled model is only trained on three different robots. So it is unlikely that it will generalize well to new robot morphologies. Table 4 shows that performance can degrade significantly when testing on unseen robots. Therefore, if new robots are added, then it seems that a new encoder will still need to be trained for finetuned on the new robot. This doesn’t show much benefit in having the shared encoder.
Very little discussion on flow-based latent generator in experiments. It’s not clear how the flow model was trained, or how the experiments were set up. If this is an important component of the paper, then more clarification is needed on these experiments.
Questions for authors
Table 1 compares the proposed method with a tracker that directly receives the reference motion as input. It appears that performance is evaluated on the training motions. How well does the joint and latent model generalize to unseen motions that are different from the training set (not simply different variations of similar motions in the training set)?
Table 3 compares performance on training and validation motions using the general encoder vs the oracle latents. But it’s not clear how the oracle latents are determined. Are they inferred using the original G1 source encoder?
For experiments such those in Table 2 and Table 3, an important baseline is missing, which is training robot specific BFM models for each robot. Comparisons are mainly to models using the original G1 model, which may not provide a good estimate of the performance of models trained specifically for each robot. Without this baseline, it’s difficult to assess how much performance the proposed method is sacrificing compared to training robot specific models.
If the goal is to construct a general behavior latent space that can be shared for different robots, is it really optimal to use a latent space that is trained specifically for the G1? It seems that a more suitable latent space can be obtained by optimizing the latent space directly for a wide variety of robots as opposed to only the G1. If that is the case, it’s not clear why the proposed method using a G1 latent space would be preferred over a latent space that is optimized for a wider variety of robots.
While there have not been much prior work on training backward and forward models that can be applied to different morphologies, there have been previous works on training general tracking controllers that can be applied to different morphologies: Learning body shape variation in physics-based characters These and similar method should be used as baselines for the tracking experiments. The current experiments only compare to variations of the proposed method, and little to no comparisons are provided to previous work that tackle similar problems.
How does performance compare if instead of distilling from the G1 encoder, the tracking controller for each robot was trained with a frozen randomly initialized encoder? Is using the G1 encoder offering useful features compared to a random encoder, especially for the tracking experiments? Note that for this to be a meaningful experiment, some care is needed in how the random encoder is initialized. For example, the random encoder should be initialized in a way that does not map all target motions to the same point in the latent space.
Minor questions:
Section 5.2: how is the validation set constructed?
Line 310: “…medium robot remarked by 1.0m hand height would…”, what does “remarked” mean in this context?
Line 398: what does “maintaining good balance for the whole trajectories” mean? What is “good balance” referring to?