- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work presents a reinforcement learning framework for training dexterous manipulation controllers using a “hybrid imitation learning” (HIL) method. The controller is trained in simulation using two different types of environments simultaneously. First, in the tracking environments, the controller is trained to manipulate objects while following a dense reference motion from human data. This encourages the controller to manipulate objects while following human-like strategies. Then, in the generalization environments, the controller is trained without reference trajectories from a human, and instead is trained only through a sparse object goal reward. By simultaneously training controllers on these two environments, this hybrid approach allows training to be guided by human demonstrations, while generalizing to scenarios that are not covered by the human data.
Experiments are conducted in simulation with a variety of different dexterous robotic hands. A small number of sim-to-real experiments are also demonstrated with a real robotic hand.
Critical strengths
The overall approach is fairly simple and elegant. The combination of the tracking and generalization environments is reasonable, and the results are quite compelling.
The experiments do a good job of ablating the importance of the hybrid environments.
The paper is well written and easy to follow.
Critical weaknesses
The methodological innovation is a bit modest, since the core method was mainly adopted from the original HIL paper.
Robotic manipulation was a primary motivation for this work, but the real robot experiments are fairly limited. No quantitative results are reported for the real world experiments so it is unclear how well the controller transfers to the real robot. Furthermore, the controller was applied in an open-loop manner in the real world, which would seem to lead to fairly brittle behaviors.
Questions for authors
Can the authors provide more info on the real world experiments? For example, what was the success rate and tracking errors when deployed on the real robot?
Why was the policy executed open-loop on the real robot instead of closed-loop? The text briefly motivated this as being due to real-time constraints. But this could lead to very brittle behaviors on the real robot.
In the original HIL paper, the generalization environments incorporated an adversarial reward that encourages the controller to maintain human-like behaviors. However, in this work, the GenEnvs applied only an object reward. Could this lead the controller to develop non-human-like behaviors in the generalization environments? Why was the adversarial reward not used in this work?
3 comments
Response to reviewer ss6X [2/2]
3. Relationships to HIL and GfR
Our work does adapt the idea of training one goal-conditioned policy on tracking and generalization environments comes from hybrid imitation for humanoid control (HIL, GfR) and we will revise §3.2 to say so directly. We would, however, distinguish the principle from what it takes to realize it in a new domain. For instance, physics-based motion tracking has been established since DeepMimic (Peng et al., 2018), yet the methods that later made tracking work on new embodiments and tasks are rightly regarded as contributions (e.g., HumanPlus, Fu et al., 2024; BeyondMimic, Liao et al., 2025). Similarly, MAE (He et al., 2022) brought BERT’s masked prediction to images, and its value lay in the vision-specific redesign that made the idea work. AdaDex is a contribution of this kind. Moving hybrid imitation from humanoid bodies to dexterous hand-object interaction is highly untrivial and requires redesigning nearly every component of the system.
The central difficulty is where the goal lives. In humanoid control, the goal is defined on the agent’s own body. In dexterous manipulation, it is defined on an object that the hand can move only through fine-grained, intermittent contact, and that may leave the hand entirely, as in throw-and-catch. Therefore, making hybrid imitation work for hand–object interaction required new design choices in both environment types: In GenEnv, goals are drawn as SE(3) perturbations of look-ahead reference frames, which keeps them reachable with the skills that TrackEnv teaches; The GenEnv reward is object-centric, with dense wrist guidance and deliberately no finger terms, so the finger strategy is guided by a human prior but changes flexibly based on RL exploration at each specific scenario; TrackEnv adds a contact reward derived from reference fingertip-object distances, strict termination on MANO link error, and variance-based bin sampling over reference phases, so that the prior captures contact-rich behaviors such as finger gaiting and regrasping. These choices respond to properties of hand–object interaction that do not arise when the goal lies on the agent’s own body (i.e., humanoid control). Together, they turn the shared hybrid imitation principle into a recipe that works for dexterous manipulation.
Humanoid hybrid imitation also builds on a mature tracking ecosystem of large motion-capture corpora, standard retargeting tools, and tracking policies refined over years since DeepMimic. Dexterous motion tracking is way harder. For methods like ManipTrans, it only reaches 35.20% progress on the very clips it was trained on, while AdaDex reaches 97.47%. On noisy references reconstructed from casual video, AdaDex reaches 80.53% mean tracking progress versus 31.70% for ManipTrans (Fig. 4), enabling learning dexterous skills from in-the-wild videos. Humanoid hybrid imitation could take its tracker and references as given, whereas we had to build both.
Rebuttal follow-up
Dear reviewer ss6X,
Thank you again for your time and valuable feedback. As the author–reviewer discussion period ends on November 18, we wanted to kindly follow up on our responses. We hope they have addressed your concerns, and if so, we would be grateful if you would consider increasing your score. If there are any further questions, we would also be happy to clarify them before the deadline.
Best regards,
The Authors
Response to reviewer ss6X [1/2]
Dear reviewer ss6X,
Thank you for your constructive feedback. Regarding you main concerns, we address as follows:
1. Closed-loop real-robot experiments
We thank the reviewers for raising this point. Open-loop replay is a standard protocol in prior work (Liu et al., 2025a; Li et al., 2025; Pan et al., 2025; Liang et al., 2026; Paliwal et al., 2026). It checks whether rollouts generated in simulation are physically realizable on hardware, so that replayed rollouts can be collected as real-robot demonstrations for downstream policy training, making simulation a scalable data generator. We followed this protocol because the core contribution of our paper is the learning recipe that turns seconds of human motion into an adaptable skill. Sim-to-real engineering is complementary to this contribution and largely orthogonal to it. We agree, however, that open-loop replay does not demonstrate adaptation. We have therefore also deployed the policy fully closed-loop on the xArm6 + XHand system.
To close the sim-to-real gap, we combine system identification of the arm and hand, domain randomization, action latency addon, and action masking during training. At deployment, the object pose is estimated in real time with FoundationPose, using the object mesh, RGB images, and Fast-FoundationStereo depth. The policy runs at 30 Hz and is conditioned on proprioception, an RMA-encoded proprioceptive history (Kumar et al., 2021), the object BPS code, and current and goal object and wrist poses.
We evaluate the hammer and cup tasks over 10 trials each, with randomized initial object poses and randomized goals. A trial counts as a success when all goals are accomplished. The closed-loop policy evaluation results are shown below, and we also show closed-loop deployment videos on the anonymous rebuttal page https://adadexanonymous.github.io/rebuttal.
These results show that the policy operates on real perception and adapts to new initial states and goals on hardware. Failures were mainly caused by failing to grasp the object or losing pose tracking of the object from FoundationPose. Please see closed-loop videos available on the anonymous project page, and we will include the results and implementation details in the revision.
2. Adversarial rewards
We thank the reviewer for this question. We removed the adversarial reward deliberately, because TrackEnv already supplies the human prior: half of the environments remain TrackEnvs, so the shared actor is continually trained to reproduce the human grasp, finger gaiting, and release. GenEnvs, meanwhile, add only a sparse object reward and dense wrist guidance. Under this design, an additional discriminator brings little benefit. Removing it keeps the pipeline simple and minimal, and it isolates the core idea: the hybrid split of environments alone is enough to transfer the human prior to off-reference goals.
Adversarial rewards are also notoriously unstable and difficult to tune, a problem made worse here because the discriminator would be trained on a single few-second clip. Such a discriminator can easily overfit and penalize any state that deviates from the reference, which conflicts with the purpose of GenEnvs. GfR similarly dispenses with the discriminator.
On whether GenEnvs could lead to non-human-like behavior, our results indicate that they do not. In the user study, participants preferred AdaDex over SimToolReal in 94.9% of comparisons, including unseen motions, where the policy must deviate from the reference. Together, these results show that TrackEnv, not an adversarial reward, is what preserves human-likeness under generalization.