- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents SMR, a real-time streaming retargeting pipeline for human-to-humanoid retargeting. The authors propose to first learn groups of expert motion trackers that follow landmarks from human data. Next, they gather a dataset of expert rollouts that demonstrate high-quality retargeting. Those rollouts are then used to learn a causal transformer that consumes human motion and predicts robot motions. The authors show that this pipeline improves physical validity and supports real-time teleoperation. The authors show that SMR follows human motion semantics better and leads, in general, to more robust tracking behavior.
In summary, the authors show two main contributions
- a semantic extraction method to define targets for the expert policies
- a causal transformer trained on human-to-humanoid trajectories
The general idea aligns with NMR; however, the authors skip the initial IK retargeting and instead use pre-computed landmarks as goals for the expert policies, similar to the objective used in methods such as ReActor. Using a causal neural model is also similar to other retargeting work with paired data.
Nevertheless, in its configuration and as a whole, the method is clean and well put together. To improve my score, I'd like a more honest comparison with other work in the field, clearly stating the method as a whole as the key contribution and acknowledging that parts are conceptually and practically very similar to prior work. Furthermore, I would like to see some challenging motions that push the method to its limits.
Critical strengths
-
The qualitative and quantitative results are clear and convincing.
-
The method is simple and intuitive.
Critical weaknesses
-
The novelty is limited and incremental. It follows ideas from NMR and ReActor. The key differences need to be stated more precisely.
-
Challenging results are missing and should be added, e.g., backflips, cartwheels, expressive dancing, etc. I would be interested to see the transformer output for those cases.
-
Table 1. Once trained, both IKMR and ReActor can be applied to real-time applications.
Questions for authors
-
I find one big limitation of this paper to be the transformer model. Starting from a dataset that respects all physical constraints, the neural method adds back artifacts such as ground penetration or limit violations. Instead of distilling states, could you distill expert actions, like in BeyondMimic, and get a final retargeting policy similar to ReActor? This would also work in real-time and map directly from states to actions.
-
I'm missing information about the NMR evaluation. Did you retrain the method on all the data, or did you simply pick a pre-trained checkpoint? While Figure 4 shows saturation at 50% data, data diversity might be a key reason for SMR's better quality.
-
For challenging motions like backflips and cartwheels, expert rollouts might be below the required retargeting quality and therefore not part of the training for the transformer. Yet if you ask the transformer for such a motion, what would happen? I would like to see some limitations or challenging motions.
-
How tedious is the "semantic extraction" step? What if defining correspondences is difficult because the morphology is very different?
-
Figure 5 shows a sudden jump in reward at the last 600 iterations for all methods and a very unexpected fall in reward after; why is this the case?