acceptodds
Under review as a conference paper at ICLR 2027

Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking

11 views

Abstract

Dexterous multi-finger hands hold promise to unlock diverse skills like functional grasps, in-hand reorientation, and dynamic handover that the human-designed world assumes, yet existing learning recipes force a trade-off. Goal-reaching reinforcement learning generalizes to arbitrary object poses but, lacking any human prior, converges to unnatural and unreliable finger strategies; motion tracking reproduces human-like behavior from a single clip but cannot deviate from it. We present AdaDex, a goal-conditioned RL framework that turns a single hand-object trajectory of a few seconds—from mocap data or HOI reconstruction from in-the-wild RGB(D) video—into an adaptable dexterous skill. At the core of AdaDex is Hybrid Dexterous Mimicking: a shared policy is trained on two populations of massively parallel environments: motion tracking environments trained with dense motion tracking objective, which instill the human prior from human motion; and generalization environments with noised initial states and goals and a sparse object-success reward, which force the skill to work far off the reference. From seconds of human motion per skill, the resulting policies can learn diverse adaptable dexterous skills, such as functional grasping, object reposing, in-hand reorientation with finger gaiting, and dynamic behaviors like throw-and-catch. Our method outperforms baselines in terms of generalization and functionality, and we demonstrate transfer across simulators and to the real world.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dexterous Manipulation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ss6XOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work presents a reinforcement learning framework for training dexterous manipulation controllers using a “hybrid imitation learning” (HIL) method. The controller is trained in simulation using two different types of environments simultaneously. First, in the tracking environments, the controller is trained to manipulate objects while following a dense reference motion from human data. This encourages the controller to manipulate objects while following human-like strategies. Then, in the generalization environments, the controller is trained without reference trajectories from a human, and instead is trained only through a sparse object goal reward. By simultaneously training controllers on these two environments, this hybrid approach allows training to be guided by human demonstrations, while generalizing to scenarios that are not covered by the human data.

Experiments are conducted in simulation with a variety of different dexterous robotic hands. A small number of sim-to-real experiments are also demonstrated with a real robotic hand.

Critical strengths

The overall approach is fairly simple and elegant. The combination of the tracking and generalization environments is reasonable, and the results are quite compelling.

The experiments do a good job of ablating the importance of the hybrid environments.

The paper is well written and easy to follow.

Critical weaknesses

The methodological innovation is a bit modest, since the core method was mainly adopted from the original HIL paper.

Robotic manipulation was a primary motivation for this work, but the real robot experiments are fairly limited. No quantitative results are reported for the real world experiments so it is unclear how well the controller transfers to the real robot. Furthermore, the controller was applied in an open-loop manner in the real world, which would seem to lead to fairly brittle behaviors.

Questions for authors

Can the authors provide more info on the real world experiments? For example, what was the success rate and tracking errors when deployed on the real robot?

Why was the policy executed open-loop on the real robot instead of closed-loop? The text briefly motivated this as being due to real-time constraints. But this could lead to very brittle behaviors on the real robot.

In the original HIL paper, the generalization environments incorporated an adversarial reward that encourages the controller to maintain human-like behaviors. However, in this work, the GenEnvs applied only an object reward. Could this lead the controller to develop non-human-like behaviors in the generalization environments? Why was the adversarial reward not used in this work?

3 comments
AuthorsOpenReview ↗

Response to reviewer ss6X [1/2]

Dear reviewer ss6X,

Thank you for your constructive feedback. Regarding you main concerns, we address as follows:

1. Closed-loop real-robot experiments

We thank the reviewers for raising this point. Open-loop replay is a standard protocol in prior work (Liu et al., 2025a; Li et al., 2025; Pan et al., 2025; Liang et al., 2026; Paliwal et al., 2026). It checks whether rollouts generated in simulation are physically realizable on hardware, so that replayed rollouts can be collected as real-robot demonstrations for downstream policy training, making simulation a scalable data generator. We followed this protocol because the core contribution of our paper is the learning recipe that turns seconds of human motion into an adaptable skill. Sim-to-real engineering is complementary to this contribution and largely orthogonal to it. We agree, however, that open-loop replay does not demonstrate adaptation. We have therefore also deployed the policy fully closed-loop on the xArm6 + XHand system.

To close the sim-to-real gap, we combine system identification of the arm and hand, domain randomization, action latency addon, and action masking during training. At deployment, the object pose is estimated in real time with FoundationPose, using the object mesh, RGB images, and Fast-FoundationStereo depth. The policy runs at 30 Hz and is conditioned on proprioception, an RMA-encoded proprioceptive history (Kumar et al., 2021), the object BPS code, and current and goal object and wrist poses.

We evaluate the hammer and cup tasks over 10 trials each, with randomized initial object poses and randomized goals. A trial counts as a success when all goals are accomplished. The closed-loop policy evaluation results are shown below, and we also show closed-loop deployment videos on the anonymous rebuttal page https://adadexanonymous.github.io/rebuttal.

MetricHammerCupTotal
Success / Trial6/108/1014/20
Success Rate60%80%70%

These results show that the policy operates on real perception and adapts to new initial states and goals on hardware. Failures were mainly caused by failing to grasp the object or losing pose tracking of the object from FoundationPose. Please see closed-loop videos available on the anonymous project page, and we will include the results and implementation details in the revision.

2. Adversarial rewards

We thank the reviewer for this question. We removed the adversarial reward deliberately, because TrackEnv already supplies the human prior: half of the environments remain TrackEnvs, so the shared actor is continually trained to reproduce the human grasp, finger gaiting, and release. GenEnvs, meanwhile, add only a sparse object reward and dense wrist guidance. Under this design, an additional discriminator brings little benefit. Removing it keeps the pipeline simple and minimal, and it isolates the core idea: the hybrid split of environments alone is enough to transfer the human prior to off-reference goals.

Adversarial rewards are also notoriously unstable and difficult to tune, a problem made worse here because the discriminator would be trained on a single few-second clip. Such a discriminator can easily overfit and penalize any state that deviates from the reference, which conflicts with the purpose of GenEnvs. GfR similarly dispenses with the discriminator.

On whether GenEnvs could lead to non-human-like behavior, our results indicate that they do not. In the user study, participants preferred AdaDex over SimToolReal in 94.9% of comparisons, including unseen motions, where the policy must deviate from the reference. Together, these results show that TrackEnv, not an adversarial reward, is what preserves human-likeness under generalization.

AuthorsOpenReview ↗

Response to reviewer ss6X [2/2]

3. Relationships to HIL and GfR

Our work does adapt the idea of training one goal-conditioned policy on tracking and generalization environments comes from hybrid imitation for humanoid control (HIL, GfR) and we will revise §3.2 to say so directly. We would, however, distinguish the principle from what it takes to realize it in a new domain. For instance, physics-based motion tracking has been established since DeepMimic (Peng et al., 2018), yet the methods that later made tracking work on new embodiments and tasks are rightly regarded as contributions (e.g., HumanPlus, Fu et al., 2024; BeyondMimic, Liao et al., 2025). Similarly, MAE (He et al., 2022) brought BERT’s masked prediction to images, and its value lay in the vision-specific redesign that made the idea work. AdaDex is a contribution of this kind. Moving hybrid imitation from humanoid bodies to dexterous hand-object interaction is highly untrivial and requires redesigning nearly every component of the system.

The central difficulty is where the goal lives. In humanoid control, the goal is defined on the agent’s own body. In dexterous manipulation, it is defined on an object that the hand can move only through fine-grained, intermittent contact, and that may leave the hand entirely, as in throw-and-catch. Therefore, making hybrid imitation work for hand–object interaction required new design choices in both environment types: In GenEnv, goals are drawn as SE(3) perturbations of look-ahead reference frames, which keeps them reachable with the skills that TrackEnv teaches; The GenEnv reward is object-centric, with dense wrist guidance and deliberately no finger terms, so the finger strategy is guided by a human prior but changes flexibly based on RL exploration at each specific scenario; TrackEnv adds a contact reward derived from reference fingertip-object distances, strict termination on MANO link error, and variance-based bin sampling over reference phases, so that the prior captures contact-rich behaviors such as finger gaiting and regrasping. These choices respond to properties of hand–object interaction that do not arise when the goal lies on the agent’s own body (i.e., humanoid control). Together, they turn the shared hybrid imitation principle into a recipe that works for dexterous manipulation.

Humanoid hybrid imitation also builds on a mature tracking ecosystem of large motion-capture corpora, standard retargeting tools, and tracking policies refined over years since DeepMimic. Dexterous motion tracking is way harder. For methods like ManipTrans, it only reaches 35.20% progress on the very clips it was trained on, while AdaDex reaches 97.47%. On noisy references reconstructed from casual video, AdaDex reaches 80.53% mean tracking progress versus 31.70% for ManipTrans (Fig. 4), enabling learning dexterous skills from in-the-wild videos. Humanoid hybrid imitation could take its tracker and references as given, whereas we had to build both.

AuthorsOpenReview ↗

Rebuttal follow-up

Dear reviewer ss6X,

Thank you again for your time and valuable feedback. As the author–reviewer discussion period ends on November 18, we wanted to kindly follow up on our responses. We hope they have addressed your concerns, and if so, we would be grateful if you would consider increasing your score. If there are any further questions, we would also be happy to clarify them before the deadline.

Best regards,

The Authors

Reviewer_JY3bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AdaDex turns a single few-second human hand-object clip into a goal-conditioned dexterous policy by training one PPO policy on two kinds of environments at once. In the tracking environments, a dense reward makes the policy imitate the human, which teaches it the human's grasps, finger gaiting and release timing. In the generalization environments, episodes start from noised states and the policy gets only a sparse success reward for reaching goals far from the clip, which pushes it beyond the demonstration. The core idea is not new: it is essentially GfR/HIL-style hybrid imitation from humanoid control moved to dexterous hands, and its clearest result is much better tracking of unseen but similar motions (80% vs. 32% without the generalization environments and 7% for ManipTrans). The other contributions are a fast RGB-D hand-object reconstruction pipeline built mostly from existing tools and qualitative demos across several hands, but each policy covers only one object and one clip, the real-robot runs are open-loop replays rather than closed-loop adaptation, and "human-likeness" is never measured.

Critical strengths

AdaDex has one policy that is both human-like and adaptable, and its tracking of unseen motions reaches 80%, vs 32% without the generalization environments and 7% for ManipTrans. Dexterous learning has been stuck choosing between natural-looking but rigid tracking and flexible but unnatural goal-reaching. The ablations show the gain comes from combining the two and not from tuning.

Each skill is learned from a few seconds of human motion, including clips reconstructed from casual RGB-D video, and the policy needs only goal poses at test time. Dexterous data is expensive to collect, and a policy any goal source can drive (such as a VLM planner) can be reused for new tasks instead of only replaying its demo. The results in simulation also seem promising from the reference motions.

The recipe produces hard skills like finger gaiting and throw-and-catch on four robot hands, transfers to MuJoCo, and comes with a reconstruction pipeline about 30–80× faster and more accurate than the baselines. This shows the method is general rather than tuned to one hand or one simulator, and the faster reconstruction makes it practical to turn everyday human videos into robot training data.

Critical weaknesses
  1. The core formulation closely follows GfR (Wang et al., 2026c). One goal-conditioned policy trained on dense-reward imitation and sparse-reward goal reaching, with reference state initialization, an asymmetric critic, a mixing curriculum ending at 0.5/0.5, and no reference at test time. The paper cites it only as "inspiration" and claims the formulation as its main contribution. The actual differences (goals perturbed from reference frames, no finger rewards in GenEnv, a reference-masked critic instead of a task flag, adaptive bin sampling) are never ablated individually.
  2. On hardware, the robot only replays joint commands recorded in simulation so the policy never sees real state. Any error in object placement, slip or friction mismatch cannot be corrected, and the robustness to noise learned in GenEnv is never tested on the real robot. GfR, by contrast, deploys closed-loop with mocap feedback. With only 5 demos and no success rates or perturbation tests, the results show the motions are physically feasible, not that the controller adapts, which contradicts the "adaptable" claim.
  3. "Unseen" motions are other GRAB clips of the same object from different subjects, and goal offsets are small (6–10 cm, 25–45°). New objects and large workspace shifts are never tested.
  4. Each object and clip needs its own policy, at two training stages of 1–2 days each on an RTX 4090, which limits scalability.
  5. Human-likeness is never measured: Table 1 reports only object and wrist errors, with no finger, contact or grasp-type metrics and no user study. The naturalness claims over SimToolReal rely on figures, while SimToolReal has the higher full success rate (31.7% vs 27.1%).
  6. The baselines are not compared on equal terms: SimToolReal runs on its original Kuka + Sharpa embodiment with object-only goals, and ManipTrans gets hand references selected by a contact heuristic (Appendix F).
  7. The VLM tasks and multi-hand demos report no success rates or trial counts, so they remain anecdotal.
  8. Statistical significance is not established. The ± is never defined (across rollouts or across objects?), there is no mention of multiple training seeds, and the paper reports no confidence intervals or tests. Key comparisons fall within one standard deviation of each other: in goal reaching, 51.8 ± 21.8% vs 44.6 ± 31.6% progress and 27.1 ± 17.4% vs 31.7 ± 32.6% full success against SimToolReal; on seen motions, 96.7 ± 6.2% vs 97.5 ± 8.4% for the no-GenEnv ablation. With only 12 objects, only the gap on unseen motions (80.0% vs 31.8%) looks robust.
Questions for authors
  1. Can you evaluate the policy closed-loop on hardware, with live object-pose feedback, and report success rates over repeated trials under perturbed starting poses, new goals and pushes mid-task, compared with open-loop replay?
  2. How does AdaDex compare to a direct GfR reimplementation on the same benchmark? Can you ablate each dexterous-specific change individually: goals perturbed from reference frames vs. independent goals, removing finger rewards in GenEnv, the masked-reference critic vs. a task flag, and adaptive vs. uniform sampling?
  3. How many training seeds were used, and what does the ± in Tables 1–2 represent? Can you report confidence intervals and significance tests, especially for the goal-reaching comparison with SimToolReal, where the two methods are within one standard deviation?
  4. Can you quantify human-likeness under random goals, for example finger-joint or contact-pattern error against the reference and grasp-type agreement, to show the human prior holds far from the reference?
  5. How does success change as goals move further from the reference (larger offsets, different intents, other objects of the same category), and does the advantage over SimToolReal hold when it runs on the same hand with the same wrist goals?
5 comments
AuthorsOpenReview ↗

Response to reviewer JY3b [1/3]

Dear reviewer JY3b,

Thank you for your constructive feedback. Regarding you main concerns, we address as follows:

1. Relationships to HIL and GfR

Our work does adapt the idea of training one goal-conditioned policy on tracking and generalization environments comes from hybrid imitation for humanoid control (HIL, GfR) and we will revise §3.2 to say so directly. We would, however, distinguish the principle from what it takes to realize it in a new domain. For instance, physics-based motion tracking has been established since DeepMimic (Peng et al., 2018), yet the methods that later made tracking work on new embodiments and tasks are rightly regarded as contributions (e.g., HumanPlus, Fu et al., 2024; BeyondMimic, Liao et al., 2025). Similarly, MAE (He et al., 2022) brought BERT’s masked prediction to images, and its value lay in the vision-specific redesign that made the idea work. AdaDex is a contribution of this kind. Moving hybrid imitation from humanoid bodies to dexterous hand-object interaction is highly untrivial and requires redesigning nearly every component of the system.

The central difficulty is where the goal lives. In humanoid control, the goal is defined on the agent’s own body. In dexterous manipulation, it is defined on an object that the hand can move only through fine-grained, intermittent contact, and that may leave the hand entirely, as in throw-and-catch. Therefore, making hybrid imitation work for hand–object interaction required new design choices in both environment types: In GenEnv, goals are drawn as SE(3) perturbations of look-ahead reference frames, which keeps them reachable with the skills that TrackEnv teaches; The GenEnv reward is object-centric, with dense wrist guidance and deliberately no finger terms, so the finger strategy is guided by a human prior but changes flexibly based on RL exploration at each specific scenario; TrackEnv adds a contact reward derived from reference fingertip-object distances, strict termination on MANO link error, and variance-based bin sampling over reference phases, so that the prior captures contact-rich behaviors such as finger gaiting and regrasping. These choices respond to properties of hand–object interaction that do not arise when the goal lies on the agent’s own body (i.e., humanoid control). Together, they turn the shared hybrid imitation principle into a recipe that works for dexterous manipulation.

Humanoid hybrid imitation also builds on a mature tracking ecosystem of large motion-capture corpora, standard retargeting tools, and tracking policies refined over years since DeepMimic. Dexterous motion tracking is way harder. For methods like ManipTrans, it only reaches 35.20% progress on the very clips it was trained on, while AdaDex reaches 97.47%. On noisy references reconstructed from casual video, AdaDex reaches 80.53% mean tracking progress versus 31.70% for ManipTrans (Fig. 4), enabling learning dexterous skills from in-the-wild videos. Humanoid hybrid imitation could take its tracker and references as given, whereas we had to build both.

2. Design choices and ablations on reference-perturbed versus independently sampled GenEnv goals

Since GfR doesn’t involve any object interaction at all, we cannot directly compare against it. As to some key design choices, we didn’t involve finger rewards in GenEnv because the finger strategies in the reference might not be always suitable for perturbed goals so we let RL explore its own finger strategies instead. For masked-reference critic observation, it’s a simple mechanism to let the critic know which environment it's in. And adaptive sampling is a commonly used technique to help exploration.

We also trained a baseline with hybrid environments but replaces reference-perturbed GenEnv goals with independently sampled random object goals. We provide side-by-side videos for AprilCube, Marker, and Pen. Within each matched-command comparison, the two policies receive the same test commands and use the same Stage2 training iteration checkpoint. The AprilCube example tests two consecutive rotations; Marker and Pen follow noisy-reference goal sequences. We show the qualitative results on our anonymous rebuttal page: https://adadexanonymous.github.io/rebuttal.

From the videos, we clearly see that training with perturbed goals from the reference can preserve the motion intent, while using completely random goals breaks the intent and reduces the policy to wrist movement only. In these representative rollouts, reference-plus-noise training better preserves the demonstrated manipulation behavior: in our perturbed setting, the in-hand reorientation of the cube, marker, and pen is effectively preserved, while the ablation loses such dexterity. We also provide each policy under its own native training goal sampler. These qualitative examples support reference-perturbed goals for preserving demonstration-specific skills, rather than universal superiority on arbitrary targets.

AuthorsOpenReview ↗

Response to reviewer JY3b [2/3]

3. Closed-loop real-robot experiments

We thank the reviewers for raising this point. Open-loop replay is a standard protocol in prior work (Liu et al., 2025a; Li et al., 2025; Pan et al., 2025; Liang et al., 2026; Paliwal et al., 2026). It checks whether rollouts generated in simulation are physically realizable on hardware, so that replayed rollouts can be collected as real-robot demonstrations for downstream policy training, making simulation a scalable data generator. We followed this protocol because the core contribution of our paper is the learning recipe that turns seconds of human motion into an adaptable skill. Sim-to-real engineering is complementary to this contribution and largely orthogonal to it. We agree, however, that open-loop replay does not demonstrate adaptation. We have therefore also deployed the policy fully closed-loop on the xArm6 + XHand system.

To close the sim-to-real gap, we combine system identification of the arm and hand, domain randomization, action latency addon, and action masking during training. At deployment, the object pose is estimated in real time with FoundationPose, using the object mesh, RGB images, and Fast-FoundationStereo depth. The policy runs at 30 Hz and is conditioned on proprioception, an RMA-encoded proprioceptive history (Kumar et al., 2021), the object BPS code, and current and goal object and wrist poses.

We evaluate the hammer and cup tasks over 10 trials each, with randomized initial object poses and randomized goals. A trial counts as a success when all goals are accomplished. The closed-loop policy evaluation results are shown below, and we also show closed-loop deployment videos on the anonymous rebuttal page https://adadexanonymous.github.io/rebuttal.

MetricHammerCupTotal
Success / Trial6/108/1014/20
Success Rate60%80%70%

These results show that the policy operates on real perception and adapts to new initial states and goals on hardware. Failures were mainly caused by failing to grasp the object or losing pose tracking of the object from FoundationPose. Please see closed-loop videos available on the anonymous project page, and we will include the results and implementation details in the revision.

4. Separate policy for each motion

Regarding training one policy per object and reference motion: per-clip or per-object training is the standard setting in this line of work. ManipTrans learns a residual policy for each trajectory, and DexMachina trains one policy per demonstration. We follow this setting deliberately so that our experiments isolate the effect of the training recipe from confounds such as data scale, motion diversity, and multi-task interference, and so that comparisons with these baselines remain controlled. Within this setting, our claims of generalization concern initial states, goals, and motions for a given object, and we will make this scope explicit in the revision. The setting also highlights data efficiency: each skill is learned from a single human clip of 3–9 seconds (Table 6), including clips reconstructed from casual RGB-D video, and trains on a single consumer GPU. Nothing in the formulation is tied to a single clip, however. The policy is conditioned only on proprioception, object pose, a BPS shape code, and goal poses, so training across many objects and motions requires no change to the method. Large-scale motion tracking for humanoids, as in SONIC (Luo et al., 2026), shows that tracking-based recipes benefit substantially from scale. Extending Hybrid Dexterous Mimicking to a single multi-object, multi-skill policy is therefore a natural next step, as noted in Appendix H.

5. Evaluating human-likeness

To measure human-likeness directly, we ran a user study with 12 participants on motion naturalness. Each participant viewed 36 trajectory pairs (12 GRAB objects × 3 motions), each comparing AdaDex and SimToolReal side by side in randomized order, without method labels. For each pair, they chose which motion looked more natural, with a tie/unsure option. Across the 432 judgments, AdaDex was preferred in 410 (94.9%), SimToolReal in 11 (2.5%), and 11 (2.5%) were ties. Excluding ties, AdaDex was preferred in 97.4% of comparisons.

The preference holds for every participant. Because SimToolReal received only 11 preferences in total, each of the 12 participants preferred AdaDex more often than SimToolReal. These results support our qualitative observation (Fig. 3, and comparison videos from the supplementary website) that goal reaching without a human prior produces unnatural grasps and finger strategies, while AdaDex preserves human-like motion. We will include the study protocol and results in the revision.

AuthorsOpenReview ↗

Response to reviewer JY3b [3/3]

6. Clarification on the evaluation condition

We thank the reviewer for asking for this clarification. ManipTrans uses the same embodiment, observation space, and action space as AdaDex (Appendix F). As a motion tracking method, it is conditioned on the dense hand–object reference. This is strictly more information than AdaDex receives, since AdaDex never observes the finger reference, so ManipTrans’s lower performance does not come from an information disadvantage.

For SimToolReal, we finetune its released pretrained checkpoint on each individual object, which gives it object-specific specialization comparable to our per-object policies. Retraining SimToolReal from scratch on Arti-MANO would discard the large-scale pretraining that underlies its performance. We therefore keep its original Kuka + Sharpa embodiment, whose Sharpa hand has the same number of finger DoF as Arti-MANO. Following its original interface, SimToolReal receives object goal poses only. We will add this information to the revision.

7. Clarification on statistical significance

Meaning of ±. For the progress and success results in Tables 1–2, ± denotes the population standard deviation across task-level means: 12 seen motions, 24 unseen motions, or 12 objects for random-goal reaching, with 128 evaluation rollouts averaged per motion/object. The ± therefore measures how much performance varies across objects and rollouts with different geometries and motions, not rollout-to-rollout noise. About training seeds, since prior works ManipTrans and SimToolReal also only reports one seed, we follow their settings for evaluation.

Confidence intervals and significance tests. In the table below, we analyzed the evaluations using 12 paired object clusters, keeping the two unseen motions for each object together. We computed pointwise 95% percentile-bootstrap confidence intervals using 100,000 object resamples and exact two-sided paired permutation tests over all 4,096 object-level method-label assignments. Confidence intervals describe AdaDex minus each baseline in percentage points (pp). P-values are Holm-adjusted jointly across all 16 comparisons (four baselines × four metrics).

MethodMetricMean ± SD (%)95% CI for difference (pp)Holm-adjusted p
AdaDex (AD)Seen tracking progress (SP)96.74 ± 6.21——
ADUnseen tracking progress (UP)80.02 ± 26.01——
ADRandom-goal progress (RP)51.80 ± 21.80——
ADAll-ten success (AS)27.08 ± 17.41——
ManipTrans (MT)SP35.20 ± 29.71[44.65, 77.08]0.00781
MTUP7.28 ± 6.57[57.84, 85.92]0.00781
MTRP1.72 ± 2.27[37.30, 61.86]0.00781
MTAS0.00 ± 0.00[17.19, 36.78]0.00781
SimToolReal (STR)SP9.72 ± 5.44[81.21, 92.19]0.00781
STRUP13.68 ± 17.88[46.39, 83.46]0.00781
STRRP44.60 ± 31.59[-18.03, 32.39]1.00000
STRAS31.71 ± 32.56[-29.69, 19.53]1.00000
w/o TrackEnv (-T)SP8.45 ± 4.98[83.04, 92.83]0.00781
-TUP9.62 ± 17.71[52.24, 85.87]0.00781
-TRP4.34 ± 7.38[34.27, 60.35]0.00781
-TAS1.43 ± 3.58[15.95, 35.35]0.00781
w/o GenEnv (-G)SP97.47 ± 8.40[-4.94, 3.32]1.00000
-GUP31.75 ± 31.23[28.75, 65.82]0.00781
-GRP14.53 ± 9.22[27.32, 46.85]0.00781
-GAS0.52 ± 1.03[16.80, 36.00]0.00781

We will include the statistical significance results in the revised manuscript.

8. “Small offsets” and far from reference

We would like to clarify the random goal reaching protocol. The ranges of 6–10 cm and 25–45° describe the change between two consecutive goals, not the total offset from the reference. Each trial chains 10 consecutive goals, so within a single trial the object can traverse up to 100 cm of cumulative translation and 450° of cumulative rotation. Later goals therefore lie progressively farther from the reference trajectory. Because progress is measured as the fraction of consecutive goals reached, it directly reflects how well the policy keeps succeeding as goals move away from the reference. On this metric, AdaDex reaches 51.80%, compared with 14.53% for its tracking-only counterpart (w/o GenEnv) and 4.34% without TrackEnvs (Table 2), even surpassing SimToolReal which explicitly trains on this objective.

The downstream tasks provide further evidence. In the Hammer task, the coding agent directs the policy to strike two nails at distinct locations, so the hammer is carried to positions that the reference never visits. The policy also strikes a nail vertically, although the reference contains only horizontal hammering. These goals come from the agent rather than from the reference, and the policy still reaches them while keeping the human functional grasp (Fig. 5; videos on the anonymous project page). Regarding new objects, please see our response on per-object training above.

Reviewer_ndGmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the limitations in the generalization capabilities of robot policies learned from human hand movements. To address this, the authors propose AdaDex—a method that utilizes two types of environments to train a unified, goal-conditioned policy. A tracking environment enables the policy to learn from human demonstrations, while a generalization environment enhances the policy's adaptability by introducing noisy goals and sparse rewards. The authors evaluate this approach using objects from the GRAB dataset, actions reconstructed from videos, and a variety of robotic hand morphologies. Furthermore, they conduct real-world experiments to validate the effectiveness of the method.

Critical strengths

The motivation is clear. Combining human motion priors with goal-conditioned RL is a reasonable approach to improve adaptability, and the ablation results support this design choice. The experimental results are promising, particularly for unseen motion tracking. It outperforms the reported baselines while using only a small amount of human motion data. The experiments cover different robotic hands, simulation environments, and real-world deployment. These results suggest that the method can generalize across different settings.

Critical weaknesses

The novelty is weak for ICLR. the method is closely related to GfR and does not explicitly state the differences. It also didn't explain whether joint training offers any advantages over a two-stage approach—consisting of initial pre-training for tracking followed by fine-tuning for target reaching. The comparisons are not entirely fair, In the paper author separately compared against two goal-reaching baselines and the motion-tracking baseline, which could be beneficial for an ablation study but is certainly unfair. Especially if ManipTrans got SR during the experiment, that means this won't fit as a clear, suitable baseline. Although the paper claims to preserve human-like grasping and contact strategies, the experiments mainly measure tracking errors and task success without directly evaluating these behaviors. Relying on commercial VLM models, the performance was not ablated, and since GPT -6 is powerful, the strength of the proposed method is weakened. The generalization claims are not fully supported because policies are trained separately for each object and reference motion, real-world experiments use open-loop replay.

Questions for authors

How to differentiate between yous and GfR, and novelty over Gfr? Add an ablation study to 3 of the open-source VLMs rather than only using GPT -6 Astra for generating the waypoints. Could the authors clarify whether all methods in Tables 1–2 use the same robot, action space, and goal information? Also, please find more related baselines for Table 2; a direct comparison with SR on ManipTrans is not informative. The performance gap in Table 1 is too large and isn't explained clearly. This suggests that the baseline is not suited for this problem setting rather than just claiming the method is substantially better. Could the authors explain what causes the baselines to perform poorly under this setting and justify why they are appropriate comparisons? If not, please compare with other methods that tackle the same problem.

4 comments
AuthorsOpenReview ↗

Response to reviewer ndGm [1/3]

Dear reviewer ndGm,

Thank you for your constructive feedback. Regarding you main concerns, we address as follows:

1. Relationships to HIL and GfR

Our work does adapt the idea of training one goal-conditioned policy on tracking and generalization environments comes from hybrid imitation for humanoid control (HIL, GfR) and we will revise §3.2 to say so directly. We would, however, distinguish the principle from what it takes to realize it in a new domain. For instance, physics-based motion tracking has been established since DeepMimic (Peng et al., 2018), yet the methods that later made tracking work on new embodiments and tasks are rightly regarded as contributions (e.g., HumanPlus, Fu et al., 2024; BeyondMimic, Liao et al., 2025). Similarly, MAE (He et al., 2022) brought BERT’s masked prediction to images, and its value lay in the vision-specific redesign that made the idea work. AdaDex is a contribution of this kind. Moving hybrid imitation from humanoid bodies to dexterous hand-object interaction is highly untrivial and requires redesigning nearly every component of the system.

The central difficulty is where the goal lives. In humanoid control, the goal is defined on the agent’s own body. In dexterous manipulation, it is defined on an object that the hand can move only through fine-grained, intermittent contact, and that may leave the hand entirely, as in throw-and-catch. Therefore, making hybrid imitation work for hand–object interaction required new design choices in both environment types: In GenEnv, goals are drawn as SE(3) perturbations of look-ahead reference frames, which keeps them reachable with the skills that TrackEnv teaches; The GenEnv reward is object-centric, with dense wrist guidance and deliberately no finger terms, so the finger strategy is guided by a human prior but changes flexibly based on RL exploration at each specific scenario; TrackEnv adds a contact reward derived from reference fingertip-object distances, strict termination on MANO link error, and variance-based bin sampling over reference phases, so that the prior captures contact-rich behaviors such as finger gaiting and regrasping. These choices respond to properties of hand–object interaction that do not arise when the goal lies on the agent’s own body (i.e., humanoid control). Together, they turn the shared hybrid imitation principle into a recipe that works for dexterous manipulation.

Humanoid hybrid imitation also builds on a mature tracking ecosystem of large motion-capture corpora, standard retargeting tools, and tracking policies refined over years since DeepMimic. Dexterous motion tracking is way harder. For methods like ManipTrans, it only reaches 35.20% progress on the very clips it was trained on, while AdaDex reaches 97.47%. On noisy references reconstructed from casual video, AdaDex reaches 80.53% mean tracking progress versus 31.70% for ManipTrans (Fig. 4), enabling learning dexterous skills from in-the-wild videos. Humanoid hybrid imitation could take its tracker and references as given, whereas we had to build both.

2. Joint training vs. tracking-then-reaching

We thank the reviewer for suggesting this comparison. AdaDex is already trained in two stages (Appendix C.2): Stage I uses only TrackEnvs, and Stage II continues from the Stage I checkpoint with a 50/50 mixture of TrackEnvs and GenEnvs. The proposed tracking-then-reaching baseline therefore corresponds to running Stage II with GenEnvs only. We implement this variant from the same Stage I checkpoint, with the same Stage II training budget, and report results below.

MethodSeen progress (%)Unseen progress (%)Random-goal progress (%)All-ten success (%)
AdaDex96.74 ± 6.2180.02 ± 26.0151.80 ± 21.8027.08 ± 17.41
w/o GenEnv97.47 ± 8.4031.75 ± 31.2314.53 ± 9.220.52 ± 1.03
w/o TrackEnv8.45 ± 4.989.62 ± 17.714.34 ± 7.381.43 ± 3.58
100% TrackEnv -> 100% GenEnv11.75 ± 6.5412.48 ± 24.8322.85 ± 29.0713.54 ± 24.55

The tracking-then-reaching variant performs significantly worse on seen motions, unseen motions, and goal reaching. Without TrackEnvs in Stage II, GenEnvs pull the policy far from the reference, and the motion prior learned in Stage I is gradually lost under prolonged pure goal-reaching RL. The policy then behaves like SimToolReal with unnatural movements. This result shows that TrackEnvs serve as a continual anchor that keeps the shared actor on the manifold of human-like behavior throughout training, not merely as an initialization. Joint training is what lets GenEnvs extend the skill to new states and goals without discarding the prior. We will include this comparison in the revision.

AuthorsOpenReview ↗

Response to reviewer ndGm [2/3]

3. Baseline selection and performance analysis

We thank the reviewer for this question. ManipTrans and SimToolReal were chosen as representative state-of-the-art methods of the two paradigms that AdaDex unifies: motion tracking and goal reaching. The large gap to ManipTrans on seen motions reflects the absence of a mature tracking recipe for dexterous manipulation, as discussed in our response on HIL and GfR. ManipTrans lacks design choices that we found critical, such as adaptive sampling for reference state initialization and object-frame termination design, etc. The clearest evidence is our own w/o GenEnv ablation. It uses only our tracking environments, with the same observation and action space as ManipTrans, and reaches 97.47% tracking progress versus 35.20% (Table 1). The same gap appears on noisy references reconstructed from casual video, where AdaDex reaches 80.53% mean tracking progress versus 31.70% for ManipTrans (Fig. 4). SimToolReal, by contrast, is trained for sparse goal reaching from object goals alone and is not designed to follow a dense trajectory, which explains its low tracking progress. On its own task, random goal reaching (Table 2), AdaDex performs on par with it.

The further drop of ManipTrans on unseen motions and random goals has a different cause. A motion tracking policy is fit to one specific trajectory and has no mechanism to handle states and goals away from it. ManipTrans falls from 35.20% to 7.28% progress on unseen motions and reaches 1.72% goal-reaching progress with 0% success. This limitation is not specific to ManipTrans: our own tracker without GenEnv also drops from 97.47% to 31.75% on unseen motions and reaches only 14.53% goal-reaching progress. With GenEnv, AdaDex retains 80.02% on unseen motions. The gap in these settings therefore measures the inherent limitation of tracking-only training, which is precisely what GenEnv is designed to address.

We appreciate the reviewer’s suggestion to include more baselines. To our knowledge, existing reinforcement learning-based dexterous methods follow one of these two paradigms, and we selected a representative method from each. We would be grateful if the reviewer could point us to specific methods beyond these two paradigms that address the same problem, and we would be happy to compare against them.

4. Evaluating human-likeness

To measure human-likeness directly, we ran a user study with 12 participants on motion naturalness. Each participant viewed 36 trajectory pairs (12 GRAB objects × 3 motions), each comparing AdaDex and SimToolReal side by side in randomized order, without method labels. For each pair, they chose which motion looked more natural, with a tie/unsure option. Across the 432 judgments, AdaDex was preferred in 410 (94.9%), SimToolReal in 11 (2.5%), and 11 (2.5%) were ties. Excluding ties, AdaDex was preferred in 97.4% of comparisons.

The preference holds for every participant. Because SimToolReal received only 11 preferences in total, each of the 12 participants preferred AdaDex more often than SimToolReal. These results support our qualitative observation (Fig. 3, and comparison videos from the supplementary website) that goal reaching without a human prior produces unnatural grasps and finger strategies, while AdaDex preserves human-like motion. We will include the study protocol and results in the revision.

5. On the use of GPT-6 Astra

We would like to clarify the role of GPT-6 Astra. It is used only in the downstream task demonstrations (§4.3), where a coding agent (Codex with GPT-6 Astra) generates high-level wrist and object waypoints for the learned skill to execute. These demonstrations are not part of our benchmark. All quantitative and qualitative comparisons (Tables 1–2, Figs. 3–4) use goals taken from reference motions or sampled procedurally, and no method, including ours, has access to GPT-6 Astra or any other VLM. The downstream tasks serve a different purpose: they show that the learned policy can be called as a \skill by a high-level coding agent within an agentic robotics pipeline. Because the policy observes only the state and a stream of goal poses, it does not depend on any particular planner, and the same interface accepts goals from the reference itself, a video planner, or any other agent. Therefore, GPT-6 Astra is not giving AdaDex any advantages over the baselines.

AuthorsOpenReview ↗

Response to reviewer ndGm [3/3]

6. Separate policy for each motion

Regarding training one policy per object and reference motion: per-clip or per-object training is the standard setting in this line of work. ManipTrans learns a residual policy for each trajectory, and DexMachina trains one policy per demonstration. We follow this setting deliberately so that our experiments isolate the effect of the training recipe from confounds such as data scale, motion diversity, and multi-task interference, and so that comparisons with these baselines remain controlled. Within this setting, our claims of generalization concern initial states, goals, and motions for a given object, and we will make this scope explicit in the revision. The setting also highlights data efficiency: each skill is learned from a single human clip of 3–9 seconds (Table 6), including clips reconstructed from casual RGB-D video, and trains on a single consumer GPU. Nothing in the formulation is tied to a single clip, however. The policy is conditioned only on proprioception, object pose, a BPS shape code, and goal poses, so training across many objects and motions requires no change to the method. Large-scale motion tracking for humanoids, as in SONIC (Luo et al., 2026), shows that tracking-based recipes benefit substantially from scale. Extending Hybrid Dexterous Mimicking to a single multi-object, multi-skill policy is therefore a natural next step, as noted in Appendix H.

7. Closed-loop real-robot experiments

We thank the reviewers for raising this point. Open-loop replay is a standard protocol in prior work (Liu et al., 2025a; Li et al., 2025; Pan et al., 2025; Liang et al., 2026; Paliwal et al., 2026). It checks whether rollouts generated in simulation are physically realizable on hardware, so that replayed rollouts can be collected as real-robot demonstrations for downstream policy training, making simulation a scalable data generator. We followed this protocol because the core contribution of our paper is the learning recipe that turns seconds of human motion into an adaptable skill. Sim-to-real engineering is complementary to this contribution and largely orthogonal to it. We agree, however, that open-loop replay does not demonstrate adaptation. We have therefore also deployed the policy fully closed-loop on the xArm6 + XHand system.

To close the sim-to-real gap, we combine system identification of the arm and hand, domain randomization, action latency addon, and action masking during training. At deployment, the object pose is estimated in real time with FoundationPose, using the object mesh, RGB images, and Fast-FoundationStereo depth. The policy runs at 30 Hz and is conditioned on proprioception, an RMA-encoded proprioceptive history (Kumar et al., 2021), the object BPS code, and current and goal object and wrist poses.

We evaluate the hammer and cup tasks over 10 trials each, with randomized initial object poses and randomized goals. A trial counts as a success when all goals are accomplished. The closed-loop policy evaluation results are shown below, and we also show closed-loop deployment videos on the anonymous rebuttal page https://adadexanonymous.github.io/rebuttal.

MetricHammerCupTotal
Success / Trial6/108/1014/20
Success Rate60%80%70%

These results show that the policy operates on real perception and adapts to new initial states and goals on hardware. Failures were mainly caused by failing to grasp the object or losing pose tracking of the object from FoundationPose. Please see closed-loop videos available on the anonymous project page, and we will include the results and implementation details in the revision.

8. Clarification on the evaluation condition

We thank the reviewer for asking for this clarification. ManipTrans uses the same embodiment, observation space, and action space as AdaDex (Appendix F). As a motion tracking method, it is conditioned on the dense hand–object reference. This is strictly more information than AdaDex receives, since AdaDex never observes the finger reference, so ManipTrans’s lower performance does not come from an information disadvantage.

For SimToolReal, we finetune its released pretrained checkpoint on each individual object, which gives it object-specific specialization comparable to our per-object policies. Retraining SimToolReal from scratch on Arti-MANO would discard the large-scale pretraining that underlies its performance. We therefore keep its original Kuka + Sharpa embodiment, whose Sharpa hand has the same number of finger DoF as Arti-MANO. Following its original interface, SimToolReal receives object goal poses only. We will add this information to the revision.

AuthorsOpenReview ↗

Rebuttal follow-up

Dear reviewer ndGm,

Thank you again for your time and valuable feedback. As the author–reviewer discussion period ends on November 18, we wanted to kindly follow up on our responses. We hope they have addressed your concerns, and if so, we would be grateful if you would consider increasing your score. If there are any further questions, we would also be happy to clarify them before the deadline.

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Thank you and rebuttal follow-up

Dear Area Chair and Reviewers,

Thank you again for your time and constructive feedback on our submission. As the author-reviewer discussion period ends tomorrow, November 18, we would like to kindly remind you of our responses. In them, we address the shared concerns on the relationship to HIL and GfR, the real-robot closed-loop evaluation, evaluating human-likeness via user study, and the quantitative support for our claims, and we answer each reviewer’s individual questions. We hope these responses resolve your concerns, and if so, we would be grateful if you would consider updating your assessment. If any questions remain, we would be glad to clarify them before the deadline.

Best regards,

The Authors

AuthorsOpenReview ↗

We look forward to your feedback

As the discussion period is coming to an end, we would like to kindly remind you of our responses, as well as our rebuttal page: https://adadexanonymous.github.io/rebuttal. We would greatly appreciate any further feedback you may have and are happy to answer any further questions about our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.