acceptodds
Under review as a conference paper at ICLR 2027

DexLink: Synthesizing Dexterous Demonstrations from chained Hand-object Poses

Abstract

Synthesizing diverse dexterous manipulation policies is challenging: existing simulation-based approaches rely on human demonstrations which are difficult to scale, or use generic reward functions, which provide limited control over the hand-object interactions. We propose \framework, which addresses these limitations by synthesizing diverse dexterous behaviors from sparse, reusable hand-object interaction templates. Given key task waypoints, DexLink retargets static interaction templates to target objects using an optimization that preserves spatial hand-object relationships. It then links the retargeted poses into an approximate reference trajectory and uses residual reinforcement learning to refine this reference into a closed-loop policy in simulation. Experiments spanning single-hand, bimanual, articulated, and dynamic tasks demonstrate that DexLink provides fine-grained control over task timing, grasp types, and high-level execution. The modular interaction templates transfer reliably across object scales, instances, and categories; without requiring object-specific human demonstrations, the framework also matches the performance of dense reference-tracking methods. Finally, zero-shot deployment of policies trained purely in simulation shows promise for simulation-based generation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Task Representation, Dexterous Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3vwGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DexLink, a framework for generating dexterous manipulation policies from sparse hand-object interaction templates and object-state waypoints. It uses object-aware retargeting to construct approximate reference trajectories, followed by residual reinforcement learning to obtain executable manipulation policies. Experiments demonstrate control over grasp types, task timing, and manipulation strategies, as well as template reuse across objects and preliminary sim-to-real transfer.

Critical strengths
  1. Meaningful problem and intuitive formulation. The paper addresses the high cost of collecting dexterous manipulation demonstrations. Using reusable hand-object interaction templates provides a flexible and interpretable way to specify different manipulation behaviors.

  2. Diverse simulation experiments. The authors evaluate multiple manipulation tasks, including grasp transitions, articulated-object manipulation, and bimanual interactions. The ablation studies provide useful evidence for the effectiveness of the proposed components.

Critical weaknesses
  1. Unclear reduction in the overall data generation cost. Although DexLink reduces the need for dense demonstrations, it introduces additional manual effort in specifying object-state waypoints, selecting interaction templates, and designing task-specific trajectories. Moreover, each task still requires separate RL training. The paper does not quantify these costs or compare the overall effort against existing demonstration-based approaches. Therefore, the claimed scalability and cost advantages remain insufficiently justified.

  2. Insufficient baseline comparisons. The quantitative comparison with existing dense reference-tracking methods mainly focuses on DexMachina. More comparisons with relevant dexterous manipulation and demonstration-generation approaches would strengthen the evaluation. It is currently difficult to determine whether the observed advantages come from the proposed interaction representation or established retargeting and residual RL techniques.

  3. Limited real-world validation. The real-world experiments are restricted to grasp-and-lift tasks on four objects, which are considerably simpler than the complex manipulation tasks demonstrated in simulation. Although the paper provides static rollout images and mentions project videos, the presented evidence is insufficient to assess the reliability of dynamic contact-rich manipulation in real-world settings. This weakens the claim that DexLink can generate broadly deployable dexterous behaviors.

Questions for authors

Q1. How much manual effort is required to construct the task specifications, including waypoint design and interaction-template selection? Can the authors provide a comparison of total human effort and computational cost against dense demonstration-based methods?

Q2. Could the authors provide additional comparisons or clarify the differences from closely related retargeting and residual RL approaches?

Q3. Can the authors provide more detailed evaluation results for real-world deployment? In particular, can the framework transfer more complex behaviors, such as in-hand grasp transitions or bimanual manipulation, beyond simple grasp-and-lift tasks?

1 comment

Thank you for recognizing the intuitive task formulation, the flexibility of reusable interaction templates, and the diverse simulation experiments. We have added measurements of task-authoring effort and further comparisons to address your questions on generation cost and the contribution of the proposed representation.

W1 / Q1. Manual effort and total cost, compared to dense baselines

The General Response comparison table, last column, summarizes the cost of generation for prior methods, including DexImit and ManipTrans, which require dense demonstrations. And the Task authoring study reports our measured human effort.

DexLink provides flexibility in task authoring: it can (re)use the demonstration-derived inputs without manual effort, or rely on an agentic harness to assemble tasks with minimal human effort (<1 min/task), providing flexibility when dense demos are not available. While dense demos, or videos, could be easier to collect, scaling collection to large object diversity becomes challenging. In contrast, DexLink can capture broad object diversity at a very low manual cost.


W2/Q2. differences from retargeting + residual RL

Conceptual clarification: DexLink employs the already established retargeting + RL recipe.

  • However, in contrast to existing works, it shows that the recipe can be applied outside the domain of dense demonstrations.
  • Emphasizes how RL-based generation (i) can be made efficient (as shown by ablations in Table 2), and (ii) can cover a broad diversity of behaviors -- all with a low-cost expressive task representation.

Retargeting Objective Ablations Please see the General response, “Isolating the relational retargeting objective.” With the downstream RL pipeline fixed, relational retargeting achieves higher task success (on 2/3 tasks and tied for 1/3). Also, the references constructed with relational retargeting are closer to the RL-grounded policies, which can explain improved learning efficiency.

Comparison with Scaffolder.

  • Compared to Scaffolder, we emphasize that DexLink results are across a broader and more complex task set (in simulation), specifically: bimanual tasks, sequential multi-object manipulation tasks, and diverse object geometries.

  • Both methods use RL, but provide different guidance for learning. Scaffolder only provides wrist guidance, while DexLink relies on constructed reference trajectories to add guidance for both wrist and finger motion.

    • In our ablations, we provide controlled experiments that emphasize the utility of complete hand references compared with wrist-only guidance: a complete hand reference provides both finger-motion guidance and valid hand–object states for training resets, which contribute to learning efficiency, especially in long-horizon interactions. For the wrist-only reference, we disable intermediate resets because the finger states are unspecified (resetting the fingers to nominal joints can cause them to penetrate objects, or, if reset arbitrarily, the object falls from the hands and the episode terminates with no useful transition).
  • We adapted Scaffolder (replacing Gemini with GPT-6.1 Sol planner) on two tasks that specifically require long-horizon interactions. Scaffolder results uses its own hand, simulation, and learning setup.

Taskgpt-assisted ScaffolderDexLink
Cup inversion and placement0/5040/50
Three-block stacking0/5048/50

Additional baseline numbers for reference, reported by DexMachina. We update Figure 8 (Section 4.1), in the revised pdf, with additional baseline numbers.

  • We do not attribute the small gains in Figure 8 to better retargeting alone, as the action space also differs (DexMachina predicts absolute joint positions, while DexLink predicts residual finger joints), which may contribute to the difference. Instead, our focus in these experiments is to show that DexLink achieves performance on par with established dense-tracking methods when the target object geometry is unchanged.

W3 / Q3. Real-world validation

Thank you for emphasizing the importance of stronger real-world validation. Although DexLink demonstrates a diverse range of behaviors in simulation, we acknowledge that our hardware evidence remains limited to grasp-and-lift.

We are actively working to deploy more diverse tasks on hardware, addressing challenges around latency and occlusion in our deployment pipeline. As noted in original lines 462–463, we present the submitted hardware results as preliminary evidence of sim-to-real transfer.

Reviewer_CReuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DexLink synthesizes dexterous manipulation policies from object-state waypoints and reusable static hand–object interaction templates. Object-aware retargeting and interpolation construct wrist and finger references, which per-task residual RL refines into executable behaviors; experiments examine template transfer, behavioral control, dense-reference tracking, and preliminary sim-to-real deployment.

Critical strengths

The task representation provides an intuitive way to specify grasp types, object motion, timing, and execution strategies while reusing interaction templates across objects. Table 1 and Figure 6 demonstrate useful flexibility beyond reproducing a fixed demonstration.

The experimental suite covers several manipulation regimes, includes results across three training seeds, and provides preliminary hardware validation. The residual-action and reset ablations offer meaningful evidence for the effectiveness of the reference-guided training design.

Critical weaknesses

The contribution of object-aware retargeting is not isolated. Figure 5 compares retargeters only on unchanged object geometry, while the “No hand trajopt” ablation simultaneously removes reference conditioning, reference-relative actions, and trajectory-wide resets (Table 14). These experiments do not establish whether the proposed relational objective improves geometry transfer over simpler anchor-generation alternatives.

The success metrics do not fully verify the requested behaviors. Table 15 defines stacking through vertical center offsets without specifying alignment, support, or stability after release. For grasp change, Table 2 reports lift success and grasp similarity, but not the success rate of the ordered airborne transition defined in Table 15.

Questions for authors

With reference construction and downstream RL otherwise fixed, how does removing the relational retargeting term affect task success and interaction fidelity on the geometry-transfer tasks? This would directly clarify the contribution of the proposed objective.

Are all evaluation episodes initialized at the first task state, with intermediate-reference resets disabled? Please clarify the implemented stacking checks and report success rates for the complete ordered grasp transition.

For the existing template-density study in Figure 7, can the authors report full-episode evaluation success under a common training budget? Figure 8 uses dense references, so clarifying which results support comparable performance from sparse specifications would help establish the scope of the contribution.

3 comments

Thank you for the precise questions.

Q1. Isolating relational retargeting objective

The contribution of object-aware retargeting is not isolated. ... "how does removing the relational retargeting term affect task success and interaction fidelity on the geometry-transfer tasks?"

Please refer to the General response, Isolating the relational retargeting objective.

  • task success: With the downstream RL pipeline fixed, relational retargeting achieves higher task success (on 2/3 tasks and tied for 1/3).
  • interaction-fidelity: we evaluate interaction fidelity for the above ablations:

Effect on Interaction fidelity

We measure the deviation between each policy’s evaluation rollouts and its supplied reference trajectory, separately for wrist position, wrist orientation, and finger configuration. Wrist pose is measured relative to the manipulated object (and the moving notebook cover), and finger error is averaged over 12 robot joints.

Bottle lift:

RetargetingWrist position (mm)Wrist orientation (deg)Finger RMSE (deg)
Geometric wrist18.47.113.3
Surface attraction23.38.913.0
Force closure21.57.914.8
Relational14.24.78.1

Notebook opening:

RetargetingWrist position (mm)Wrist orientation (deg)Finger RMSE (deg)
Geometric wrist43.115.312.9
Surface attraction56.434.818.0
Force closure55.838.515.8
Relational42.17.513.6

Banana handover:

RetargetingWrist position (mm)Wrist orientation (deg)Finger RMSE (deg)
Geometric wrist13,357.3 ± 1,241.4123.7 ± 0.221.3 ± 5.1
Surface attraction331.6 ± 348.557.3 ± 36.022.2 ± 0.5
Force closure322.1 ± 31.283.9 ± 10.721.2 ± 2.7
Relational186.1 ± 142.555.3 ± 35.217.0 ± 1.4

(We use 2 seeds for banana and 1 seed for the others. and lower is better across all columns. The high values for banana are due to banana falling out of hand, leading to larger deviations)

Relational retargeting gives the lowest mean wrist-position and orientation errors on all three tasks, and the lowest finger error in banana handover and bottle lift. The smaller error between the constructed reference and the behavior realized by RL suggests that relational retargeting provides references closer to the physical dynamics, supporting higher interaction fidelity and stronger task performance under the same training budget.

Q2a. Evaluation from the first task state

Are all evaluation episodes initialized at the first task state, with intermediate-reference resets disabled?

Yes. All reported evaluation episodes start at t = 0, with no intermediate-reference resets. This also holds for the new results we report.

Intermediate-reference resets are used only during training. We will clarify the distinction between training-time reset sampling and full-task evaluation in the final revision.

W2 / Q2b. Stacking: implemented additional checks

Please clarify the implemented stacking checks

Thank you for pointing out that block height alone does not verify a complete stack. We have updated the checks to require alignment, support, hand release, and post-release stability. The table below reports the updated success rates; the relative ranking across ablations remains unchanged.

MethodHeight onlyComplete stack
Initial-only resets33.3%30.0%
Full method100.0%86.7%

The revised pdf reports both metric criteria, including a breakdown of the failure cases in stacking under the stricter criterion (Appendix E.3, Figure 20) compared to the height-only test, explaining the difference of 13 points in the results. Specifically, the 8% drop is due to the stricter alignment criterion: each upper block’s center must lie at least 5 mm inside the lower block’s edges.

W2 / Q2b. Mug: complete ordered airborne grasp transition

and report success rates for the complete ordered grasp transition.

Thank you for asking us to evaluate the ablations under the stricter criterion of supporting ordered grasp.

New criterion: the hand pose in the earlier 3-frame window should be closer to the top-grasp pose from the hand template than to the side-grasp pose, and the average hand pose in the later grasp window should be closer to the side-grasp template. The closeness is computed with distance functions to compare (i) mug-relative wrist pose (position and orientation) difference, and (ii) finger configuration (ten skeleton angles) difference between the avg grasp pose and template. Closeness should hold for both wrist pose and finger skeleton for the rollout to be counted as an ordered grasp change. We retain the requirement that the mug remain airborne throughout the span of windows and intermediate transitions.

Under this stricter criterion

MethodLift success reported in Table 2Ordered success rate criterion
No reference trajectory00.0 ± 0.0%
Delta actions100%33.3 ± 57.7%
Initial-only resets100%33.3 ± 57.7%
No interaction rewards100%0.0 ± 0.0%
Full method100%92.0 ± 12.2%

These results further strengthen the comparisons: the full method achieves the highest success under this ordered wrist-and-finger pose criterion.

Q3. Template density at a common training budget (Figure 7)

can the authors report full-episode evaluation success under a common training budget?

Sparse templates can support complete tasks at the same training budget. We add the matched-budget, full-episode comparison below to test whether reducing input guidance merely shifts the cost to additional RL training.

TaskDense (%)Stride 6 (%)Start/end only (%)
Notebook opening68.0 ± 55.427.3 ± 29.176.0 ± 31.7
Banana handover94.0 ± 10.460.7 ± 53.032.7 ± 56.6

The results reinforce the paper’s task-dependent density finding (lines 428–431): start/end templates support notebook opening without a larger training budget, while banana handover benefits from denser temporal guidance. We have replaced the Figure 7 (pg 8) in the revised pdf, with the full-episode evaluations on the same training budget.

Thank you for the detailed rebuttal. The controlled retargeting ablations, stricter stacking and ordered grasp-transition evaluations, and clarification of full-episode evaluation largely address my main concerns. The matched-budget study also clarifies that the effectiveness of sparse templates is task-dependent.

Although the limited training seeds leave some uncertainty, the additional evidence substantially strengthens the paper. I am therefore raising my rating. Please retain the revised evaluation criteria and clearly qualify the sparse-versus-dense comparison in the final manuscript.

Reviewer_ve4vOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DexLink studies the synthesis of dexterous robot demonstrations from object-state trajectories and reusable human hand–object interaction templates. Object-aware retargeting creates robot interaction anchors, which are linked into a reference trajectory. A reference-conditioned residual RL policy then learns corrections to execute the behavior in simulation. Experiments use XHand and evaluate interaction geometry, template transfer across objects, control of grasp choice and task timing, and sensitivity to template density. The four-task study reports full-method mean lift success of 100% on mug grasp change, and task success of 100% on notebook opening and block stacking and 97.3% on banana handover. The paper also compares dense object tracking with DexMachina and reports grasp-and-lift trials on four physical objects.

Critical strengths
  1. DexLink lets users edit hand-object interactions separately from the object's path. Figure 6 demonstrates changes to grasp, strategy, goal, and timing, while Table 1 shows template reuse across objects. The mug and stacking examples use sparse specifications without complete target-specific human motion sequences.

  2. The ablations reveal different hand-guidance needs across the tested tasks. In Table 2, wrist-residual/absolute-finger control matches the full method on notebook opening (100% each), but fails on stacking (0% versus 100%). Figure 7 similarly shows that notebook opening tolerates sparse anchors, whereas handover benefits from denser guidance.

Critical weaknesses
  1. The experiments provide limited evidence for DexLink's scalability across tasks and interaction combinations. Table 1 evaluates cross-object template reuse in lifting, while Figure 6 and the four-task study demonstrate useful edits and behaviors on a limited set of task specifications. These results leave unclear how reliably a reusable template library can support a broader range of multi-stage behaviors. Please evaluate more task and interaction combinations, reporting success rates and the additional manual specification and policy-training cost. This would substantiate the framework's value for scalable demonstration generation.
Questions for authors
  1. When extending a template library to a new task, which parts of the task specification and reward require manual changes? Please distinguish reusable inputs from task-specific preparation.

  2. Relative to DexImit's grasp/motion composition and Scaffolder's object/wrist planning, which additional interaction choices can DexLink control independently? Please clarify this distinction and qualify lines 124–129's characterization of demonstration augmentation.

1 comment

Thank you for the review and the concrete requests.

W1. Scalability across tasks and interaction combinations

Please evaluate more task and interaction combinations, reporting success rates and the additional manual specification and policy-training cost.

Thank you for asking for broader evidence of template reuse and its associated costs. We have added a 15-task authoring study and an additional 40-task set covering more interactions and multi-stage behaviors reporting human involvement, task success, and policy-training costs in the General response.

Q1. What changes when extending to a new task

which parts of the task specification and reward require manual changes

The general response describes the human role in authoring different task variations. For new tasks, the changed specification results in a changed reference trajectory; but the policy architecture and dense reward formulas are reused. The binary task-completion criterion is defined per-task as needed.

ComponentReused across tasksSupplied or checked per task
Retargeting + reference construction✓ same procedureRun automatically on the new inputs
Policy architecture, observations, actions✓—
Dense rewards (object tracking, hand/interaction tracking)✓ same formulasTargets come from the new obtained reference
Sparse completion check~ (Commonly used criterion is thresholding on object poses)Specify new thresholds or success criterion if needed.

Dense RL rewards. We reuse the same reward terms, while their targets and weights are derived automatically from each reference trajectory. For example, tracking rewards are disabled for entities at rest in the reference. These adjustments require no manual reward tuning.

Q2. Independent interaction control vs. DexImit / Scaffolder

which additional interaction choices can DexLink control independently

Thank you for asking us to clarify these interaction choices. Below we describe different axes of variation that can be controlled with the task specification, compared to DexImit and Scaffolder, in light of the General Response Clarification.

Object geometry. DexImit only shows scale augmentations. Scaffolder only provides preliminary qualitative transfer to similar unseen shapes.

  • DexLink: scale, within- and cross-category transfers (all the tasks shown in the paper + new results, are transfers to different geometry, except 5/7 arctic tasks and some lift tasks)

Grasp type. DexImit’s grasp selection requires a task video; Scaffolder does not expose finger configurations as specification inputs.

  • DexLink lets users explicitly choose a reusable grasp template independently of the object trajectory.

Bimanual behavior. Both DexImit and DexLink demonstrate bimanual manipulation. Scaffolder only shows single-hand manipulation.

Timing. Could be possible with DexImit and Scaffolder, but neither paper shows how the task timing can lead to different downstream behaviors. We explicitly demonstrate this in Figure 6: changing the timing of the hammer-swing waypoints changes the resulting strike force.

Composition. DexImit demonstrates action sequences reconstructed from task videos, however independent recombination of grasps within and across tasks is not shown. Scaffolder can compose object and wrist waypoints, but interactions are not explicitly composable.

  • DexLink demonstrates composing independently selected templates with new object trajectories and geometries, without a complete demonstration of the composed task.

Lines 124–129. We agree that dense data "leaves limited room for data augmentation" is too broad. We'll update the lines:

DexImit and Scaffolder support task composition from videos and language instructions, respectively. DexImit's reported geometry augmentation is limited to scaling, while Scaffolder provides preliminary qualitative evidence of transfer to similar unseen shapes. DexLink demonstrates template transfer across object scales, instances and categories, including their compositions.

Comments on the paper

9 comments
AuthorsOpenReview ↗

General Response: Clarifications and Additional Experiments

We are grateful that the reviewers recognized the intuitive and flexible task representation, the diverse simulation tasks, and the evidence from our ablations. We also appreciate the requests for additional experiments and clarifications. In our response, we clarify our contributions and generation costs, and provide the following additional experiments and evaluation checks:

  • Task authoring and scalability (Reviewers 3vwG and ve4v): a task-authoring study measuring human effort, an additional 40-task set testing broader template reuse, and task-conditioned policy learning to evaluate reuse across task variations.
  • Additional comparisons (Reviewer 3vwG and CReu): assisted Scaffolder on multi-stage tasks, DexImit-style generation and offline motion planning, additional dense-tracking baseline results, and controlled comparisons that isolate the retargeting objective while keeping the downstream RL pipeline fixed.
  • Evaluation checks (Reviewer CReu): stricter stacking checks, direct evaluation of the ordered airborne grasp transition for the mug task, and full-episode template-density comparisons at a common training budget.
AuthorsOpenReview ↗

Contribution: DexLink extends the established retargeting + RL recipe by loosening its usual data requirements.

Reviewers' concerns:

(3vwG) "currently difficult to determine whether the observed advantages come from the proposed interaction representation or established retargeting and residual RL techniques."

(ve4v) "Relative to DexImit's grasp/motion composition and Scaffolder's object/wrist planning, which additional interaction choices can DexLink control independently?"

DexLink extends the established retargeting + RL recipe by loosening its usual data requirements. Existing retargeting + RL approaches typically assume a dense demonstration on the target object. Instead, DexLink relies on object waypoints and reusable hand-object poses to specify the desired behavior. The contribution is a unified framework that takes a loose but interpretable and controllable specification across a diverse range of tasks.

Comparison with related works.

MethodTask-motion demonstration neededHandling different object geometriesDirect specification of interaction typeScene/task compositionScaling input acquisitionObject typesBimanualTime after inputs are available
DexMachina & ManipTrans (MoCap-based)YesDemo geometryYes, but need demoNoNoRigid + articulatedYesManipTrans: 15 min RL for 60-frame demo
DexImit (video-based)YesScale onlyYes, but need videoYesYesRigidYes3-4 mins for 5–20 s videos.
Scaffolder (image + language)NoSimilar shapes (qualitative evidence)wrist guidance onlyYesYesRigid + articulatedNot evaluated1.5–6 h (task dependent).
DexLink (waypoints + interaction templates)NoScale / instance / categoryYes, with reusable templatesYesYesRigid + articulatedYes1–20 h (task dependent)

Summary of close prior works: DexImit derives grasp guidance from a task video and generates robot motion through grasp optimization and planning. Scaffolder avoids demonstration capture, but its scaffold specifies object/wrist motion while RL learns the finger actions.

DexLink removes the need for a dedicated task video for the target geometry while still making interaction choice explicit. DexImit's grasp guidance depends on video reconstruction, and its pipeline supports rigid objects only. Scaffolder avoids demonstration capture but leaves finger actions to RL, without explicit finger-object pose inputs. DexLink, with low task authoring cost (described below), supports both rigid and articulated tasks, as well as object geometry variations and compositions of interactions explicitly specified.

AuthorsOpenReview ↗

Task authoring and scalability: Summary

Reviewers' concerns:

(3vwG) The paper does not quantify these costs or compare the overall effort against existing demonstration-based approaches.

(ve4v) Please evaluate more task and interaction combinations, reporting success rates and the additional manual specification and policy-training cost.

We evaluate the human effort of creating new task specifications and the compute cost for policies learned from them. Overall this thread covers:

QuestionResponse
How much manual preparation is required?Across the 15-task study, human effort is 7.2–46.2 seconds/task.
Does reuse extend beyond the ablations?The additional 40-task set achieves 76% completion, with ~15 minutes of overall human involvement.
Does each task variation need training from scratch?No, RL can be trained in a multi-task setting to learn across several tasks efficiently, with learning on new variations made 4x more efficient.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.