acceptodds
Under review as a conference paper at ICLR 2027

Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

21 views

Abstract

While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present **Dex-One2Many**, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dexterous Manipulation, Reinforcement Learning, Real-to-Sim-to-Real

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_HTAbOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a method that converts a human demonstration into stage-wise relational graphs, then uses those graphs to generate diverse training resets, observations, and rewards for RL training. The results show that the policies can generalize to configurations beyond source videos.

Critical strengths
  1. The idea is original. Fitting the video into a stage-wise scene graph instead of a motion to imitate is really interesting.
  2. The results are compelling with sufficient real-world task evaluation.
Critical weaknesses
  1. Generalization is overstated. It can only generalize to different configurations but not new tasks or scenes.
Questions for authors
  1. How autonomous is VLM? Does it automatically turn the full sequence into stages and extract the established relations? Or there are more manual curation or procedures involved?
  2. What is the quality requirement for the "single human video" input? It's hard to assess how critical the real-to-sim process is to the success of proposed method.
3 comments
AuthorsOpenReview ↗

Response to HTAb (1/2)

(W1) Generalization is overstated

We thank the reviewer for this comment. We clarify that our generalization claim concerns different configurations of the demonstrated task, rather than transfer to entirely new tasks or scenes.

Importantly, unlike parallel-jaw grippers, dexterous hands may require substantially different grasp strategies as initial or goal object poses change. To address this challenge, we leverage grasp synthesis to generate diverse feasible grasps and incorporate them as prior knowledge into the RL initial-state distribution. Rather than prescribing a fixed grasp or motion, this enables the policy to autonomously discover which grasp strategies are most suitable for different configurations through RL.

Accordingly, our generalization claim focuses on three axes: (i) unseen initial object poses, (ii) unseen goal object poses, and (iii) grasp strategies not shown in the demonstration.

To make this scope explicit, we will revise the abstract to emphasize generalization across configurations of the demonstrated task, replacing broader expressions such as "generalizable policy" and "unseen scenarios." We will also clarify in the introduction that our goal is to generalize beyond the specific execution shown in the video, rather than to entirely new tasks or scenes.

(Q1) How autonomous is VLM, and how does it automatically turn the video into stage-wise scene graph?

Yes, both are automatic. Given a single human demonstration video, the VLM is queried once with a shared prompt template containing the current predicate vocabulary to identify task-relevant objects, their roles, and the ordered relations established in the demonstration. The resulting relations are then automatically converted into stage-wise scene graphs, including intermediate approach stages.

To clarify the degree of automation, we distinguish what the user provides from what our framework constructs automatically when applying Dex-One2Many to a new task.

User-provided inputs.

  1. A single human demonstration video, which specifies the task without requiring a separate textual task description or manual stage annotations.
  2. A new predicate definition, only if needed. The predicate vocabulary and corresponding distance functions are already defined in our framework. Users only need to extend this vocabulary in the VLM prompt if a new task requires an additional relation.

Automatic pipeline.

  1. Stage-wise scene graph abstraction: The VLM identifies objects, functional parts, roles, and established relations. Their ordering determines the task stages using the predefined predicate vocabulary.
  2. Reset generation: For each stage, we use its scene graph to sample diverse reset states, including different grasps, that satisfy the predicates specified by the scene graph and verify their physical validity in simulation.
  3. RL training: The observation, reward, and initial-state distribution are constructed from the scene graphs, and the policy is trained using the same RL pipeline.

Importantly, users do not need to design task-specific rewards, reset distributions, or training procedures. These are constructed using reusable predicate-level definitions. Even when a new predicate is required, its definition can be reused across subsequent tasks without redesigning the RL pipeline.

Thus, applying our framework to a new task primarily requires a single demonstration video, with additional predicate definitions only when the existing vocabulary is insufficient. The remaining task formulation and training procedures are handled by the same reusable pipeline.

AuthorsOpenReview ↗

Response to HTAb (2/2)

(Q2) What is the quality requirement for the single human video input?

In our pipeline, the input video is used for two things: (1) extracting the task’s stage-wise scene graph, and (2) reconstructing the object assets.

For (2), the task-relevant objects need to be clearly visible in the first frame, from which their assets are reconstructed. This is a common requirement in real-to-sim pipelines: CHORD [1], for example, also reconstructs each object mesh from a single reference frame with SAM3D [2]. For (1), however, our method requires less than prior work. CHORD additionally tracks the object and hand throughout the video, whereas our VLM only needs to recognize the relations, i.e., which objects are involved and in what order they are grasped, contacted, or placed. Since the demonstrated motion is not used for policy learning, the demonstration need not be slow or clear enough for hand or object motion to be tracked, and the hand may be partially occluded.

To illustrate this, we added an example figure to the supplementary material (extract_from_video_compare.pdf), using a video with heavy occlusion: a ball is placed in a cup, and the cup in a bucket, so the ball stays hidden by the hand or the cup, and the demonstrator’s sleeve partially covers the hand. When reconstructing this demonstration with CHORD’s pipeline, the tracked ball pose drifts away from the cup once the ball is occluded. In contrast, our VLM correctly extracts the eight-stage scene graph, including the nested inside relations, since it only needs to recognize the relations rather than track the motion.

[1] Zhu et al., “Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration” (2026).

[2] SAM 3D Team, “SAM 3D: 3Dfy Anything in Images” (2025).

Thank you again for your time and thoughtful feedback on our paper.

As the discussion period ends in about two days, we wanted to kindly check whether our response has addressed your concerns. If you have any remaining questions or would like further clarification, please let us know. We would be happy to address them before the deadline.

Reviewer_6QJyOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Dex-One2Many learns generalizable dexterous manipulation from a single human video by extracting stage-wise scene graphs that guide reinforcement learning through task-consistent resets and rewards.

Critical strengths
  1. The paper addresses a meaningful problem: learning from a single human demonstration while generalizing beyond the specific execution shown in the video.

  2. I particularly appreciate the stage-wise scene-graph representation. Preserving task-defining relations while allowing poses, grasps, and motion trajectories to vary is an intuitive and well-motivated design.

  3. The paper presents a coherent and natural progression from motivation to representation and policy learning. The experimental evaluation is comprehensive.

Critical weaknesses
  1. My main remaining uncertainty concerns the precise contribution relative to the baselines. Although the paper explains their high-level differences, a more explicit account of the baseline implementations and the distinctions would help me assess the novelty with greater confidence.

  2. I could not locate a demonstration video in the materials available to me. Given the multiple stages of the framework, an end-to-end walkthrough and continuous real-world execution videos would make the approach and its practical effectiveness easier to assess. and of course, more impactful.

Questions for authors

Please address the concerns above. Overall, I am positive about this paper. If the authors clarify these weaknesses, I would be willing to raise my rating.

3 comments
AuthorsOpenReview ↗

Response to 6QJy

(W1) More detailed baseline comparison

We thank the reviewer for this suggestion.

All methods start from the same single video and differ in what they take from it. Do as I Do [1] takes the retargeted hand-wrist trajectory and replays it; CHORD [2] takes the object trajectory and contacts as an imitation reward; we take only the stage-wise relations and their order, and use them to sample diverse resets and define the rewards and observations. The distinction is therefore what the video fixes: for the baselines, the hand motion, object path, and goal pose; for ours, the relation only, with poses, grasps, and motion left free.

This is what separates the results. On seen configurations the baselines succeed because the extracted execution reproduces that configuration (Fig. 4, top). On unseen configurations the same execution no longer applies, and success drops to at most 15% (Fig. 4, bottom). For ours, an unseen configuration is simply another state satisfying the same relations, which the policy has already trained on through the sampled resets.

baseline implementation details. All baselines are ported from their official code to our UR3 + Wuji 1 setting with the same object meshes, scene calibration, and physical parameters as ours. Do as I Do [1], which models only the manipulated object, is executed open-loop by converting its retargeted wrist trajectory to UR3 joints with IK, as in the original. CHORD [2] keeps its staged curriculum and uses its force-closure contact reward, which it prescribes for reconstructed demonstrations with noisy contacts, instead of contact-wrench matching. Its object-pose reward is also applied to the receiving object (for tool use, to the tool relative to the target and the target relative to the receiving object), and the policy runs at our 10 Hz control rate.

(W2) No demonstration video

We have newly added a project page (HTML) as supplementary material. It includes a video with an overview of our method and more detailed visualizations. It also visualizes our experimental results, including simulation / real-world rollouts and diverse grasps. We would appreciate it if you could take a look.

[1] Paliwal et al., “Do as I Do: Dexterous Manipulation Data from Everyday Human Videos” (2026).

[2] Zhu et al., “Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration” (2026).

Thank you again for your time and thoughtful feedback on our paper.

As the discussion period ends in about two days, we wanted to kindly check whether our response has addressed your concerns. If you have any remaining questions or would like further clarification, please let us know. We would be happy to address them before the deadline.

Thank you for clarifying the baseline implementations and providing the project page. The videos make the full pipeline and real-world results much easier to assess, and the response addresses my main concerns. I will raise my score. Good luck!

Reviewer_StWUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Dex-One2Many, a framework for learning dexterous manipulation from a human video. A VLM extracts scene graphs that define task relations. An RL policy is then learned to transfer to real dexterous hands. Experiments across five manipulation tasks demonstrate generalization capability to out-of-demonstration initial and goal configurations.

Critical strengths

The idea of extracting task structure rather than simply imitating demonstrated motions is interesting. Scene graphs are used to generate diverse reset states and task rewards, allowing RL to explore different ways of completing the same task.

Stage-wise resets have been shown to work well in recent RL research, and this paper makes good use of them for dexterous manipulation.

The real-world results are impressive. The method substantially outperforms the baselines on unseen configurations across five tasks. The simulation results across different dexterous hands also demonstrate the flexibility of the approach.

Critical weaknesses

No supplementary material is provided, which is useful for the task this paper is trying to solve.

The baseline comparisons are insufficient. The compared methods primarily rely on motion imitation, while the proposed method uses goal-directed RL with extensive reset randomization and dense rewards. A comparison with goal-conditioned RL using the same rewards, resets, and training budget would be more convincing. In addition, the method relies on substantial computational resources, including 49,152 parallel environments and 10,000 verified reset states per stage. The training cost and sample efficiency are not sufficiently evaluated.

The generalization remains limited to different configurations of the same task. The five tasks rely on a small, predefined predicate vocabulary, with manually designed rewards and reset samplers. It is unclear how the approach scales to more complex manipulation tasks requiring new relations or interactions.

The novelty is limited. The framework combines existing techniques, including VLM-based task abstraction, grasp synthesis, stage-wise resets, manually defined reward functions, and trained with PPO. The contribution of the scene graph itself is not clearly isolated.

Questions for authors

Is a human demonstration necessary for extracting the task structure? Could the same scene graph be generated from a pure language instruction, given that the demonstrated motion is not used for policy learning?

How much of the improvement comes from the proposed scene-graph abstraction rather than diverse resets, grasp synthesis, and dense rewards? Can the authors provide a comparison with standard goal-conditioned RL under matched training conditions?

How does the method handle tasks that cannot be represented by the predefined predicates? Would new tasks require manually designing additional rewards and reset samplers?

What are the computational costs of training each policy and generating the verified reset states? How does the method compare with existing approaches in terms of training efficiency?

5 comments
AuthorsOpenReview ↗

Response to StWU (1/3)

(W1) No supplementary material is provided

We have newly added a project page (HTML) as supplementary material. It includes a video with an overview of our method and more detailed visualizations. It also visualizes our experimental results, including simulation / real-world rollouts and diverse grasps. To view it, please open project_page_open_with_browser.html in the supplementary material with a web browser.

(W2, Q2) Comparison with goal-conditioned RL and the contribution of the scene graph

The stage-wise resets and predicate-derived rewards are outputs of the scene-graph abstraction, so they cannot be given to a baseline without the graph. We therefore compare against standard goal-conditioned RL on UR5e + Wuji 2 embodiment. The baseline receives the goal pose as an additional input, starts from an empty hand with a random object poses, and is rewarded by the dense 6D pose distance between the object and its goal. We evaluate 500 episodes per task with object poses sampled across the workspace.

TaskGoal-conditioned RLDex-One2Many (Ours)
Doll0.4%97.2%
Hammer0.0%97.6%

Without the scene graph to guide exploration, the baseline must search over diverse object configurations and a 20-DoF action space with no intermediate structure, and it fails to learn within the same budget. The graph-derived resets and rewards are what keep this exploration tractable.

AuthorsOpenReview ↗

Response to StWU (2/3)

(W3, Q3) How can the method handle relations beyond the predicate vocabulary, and how much manual design would that require?

We thank the reviewer for raising this concern. We agree that our framework assumes a predefined predicate vocabulary, and tasks involving new relations require additional predicate definitions. However, our primary contribution is to demonstrate that, given a reusable set of predicates and their corresponding distance functions, diverse dexterous manipulation tasks can be learned through RL across a wide range of configurations. Most geometric relations in our five tasks are represented by simple distance functions, such as Euclidean distances in Cartesian space.

Importantly, once these predicates are defined, our framework automatically constructs the observation, reward, and reset distribution from a single human video, without task-specific reward engineering or reset sampler design. This allows the same predicate definitions and RL pipeline to be reused across different manipulation tasks, rather than requiring manual redesign for each new task.

Extending to a new task: Opening a microwave. To illustrate how our framework handles relations beyond the existing vocabulary, we consider a new task: open a microwave door, pick up an apple, and place it inside. This task combines existing pick-and-place relations with one new relation, open(door, microwave). Extending our framework requires only the following steps:

Extending the predicate vocabulary. Adding a geometric relation amounts to defining a distance function dd and two thresholds (ϵ,ϕ)(\epsilon,\phi) (Table 3). Given these definitions, the framework automatically constructs:

  1. Define the new predicate. We introduce open using a simple distance function:

dopen=max⁡(θgoal−θ,0),d_{\mathrm{open}}=\max(\theta_{\mathrm{goal}}-\theta,0),

where θ\theta is the door joint angle, together with two thresholds (ϵ,ϕ)(\epsilon,\phi) (Table 3).

  1. Reuse existing predicates. The remaining interactions are represented using our existing vocabulary, yielding the following relation sequence:

pre-open(hand, door) → open(door, microwave) → pre-grasp(hand, apple) → grasp(hand, apple) → pre-inside(apple, microwave) → inside(apple, microwave).

  1. Apply the same RL framework. Given these predicates, the framework constructs dense progress rewards from their distance functions (Appendix C.2), samples diverse reset states satisfying the scene-graph constraints (Appendix C.3), and trains the policy using the same RL pipeline.

Result. Following these steps with the UR5e arm and the Sharpa Wave hand, the policy learns the full task from the initial scene: it opens the door, grasps the apple, and places it inside the microwave, with a success rate of 98.4% over 500 evaluation episodes. We included the rollout video in the supplementary material (open_oven_put_apple_demo.mp4).

Overall, our five tasks demonstrate that a compact vocabulary of largely geometric relations can support learning diverse dexterous manipulation behaviors across different configurations. The additional microwave example illustrates how the framework can be extended to new interactions by introducing a predicate-level definition while reusing the existing reward construction, reset generation, and RL training procedures. We will include this example in the appendix.

(W4) The novelty is limited. The contribution of the scene graph itself is not clearly isolated

We agree that the individual components (VLM-based abstraction, grasp synthesis, stage-wise resets, dense rewards, PPO) exist in prior work. The problem we address, however, has not been solved by them: learning from a single video a dexterous policy that generalizes to initial poses, goal poses, and grasps not shown in the video.

Prior single-video methods extract execution-level guidance (object trajectories, retargeted hand motion, or contacts) to make dexterous exploration tractable, which ties the policy to the demonstrated configuration. We instead extract only the task structure: which relations must hold and in what order. The stage-wise scene graph represents this structure and defines the MDP (observations, rewards, initial state distributions), which are assembled from the graph without task-specific design. These graph-derived components are what make RL exploration tractable in place of execution-level guidance, which is why a single video yields a policy that works beyond the configuration it shows.

On isolating the graph’s contribution. Since stage-wise resets and predicate-derived rewards are both instantiated from the graph, our ablation study (Fig. 7(a)) already measures its contribution: removing graph-derived resets drops success from over 90% to 16%, and removing graph-derived rewards yields 0%, since neither can be constructed without the relational abstraction.

AuthorsOpenReview ↗

Response to StWU (3/3)

(Q1) Why human video is needed?

We thank the reviewer for this insightful question. While the demonstrated motion is not used for policy learning, the video provides more than the task structure in our pipeline. First, it is the source of the simulation assets: we reconstruct each task-relevant object from the first RGB-D frame of the demonstration (Appendix B.3). Second, it grounds details that a language instruction often leaves unspecified, such as which part of an object the hand grasps, which part of a tool contacts the target and in which direction, how objects are mounted at the start, and the order in which relations are established (Table 6). These details determine the grasp regions and contact poses used for reset generation. The scene graph itself is symbolic and could be written from language, but these grounded details are what the video supplies and what make it necessary in our setting.

(Q4) What is the computational cost of training the policy and generating the verified reset states? Compare training efficiency with existing approaches.

1. Computational cost

RL training. Training a policy from scratch takes approximately 24 hours on 4 NVIDIA L40S GPUs, using 49,152 parallel environments in total, for most task–embodiment pairs.

Reset-state generation. The complete pipeline takes approximately 4–5 hours per task–embodiment pair on 4 L40S GPUs, including (i) grasp synthesis (1–2 hours), (ii) stage-wise reset candidate sampling (approximately 5 minutes), and (iii) physics-based verification of reset stability (2–3 hours).

2. Training efficiency

Do-As-I-Do [1] is not a learning-based method, so it has no training cost to compare.

We therefore compare against two learning-based baselines, Human2Sim2Robot [2] and CHORD [3] , on the stamp task with the same budget of 262M environment steps.

We report success rate on the unseen evaluation set every ~4.7M environment steps, with 500 episodes per point. For the baselines, we also report success on their own training resets (seen). The table below summarizes the results at representative steps.

MethodEval50M100M150M200M262M
Oursunseen0.411.042.662.894.8
Human2Sim2Robotunseen1.41.21.01.41.8
CHORDunseen0.20.20.00.20.0
Human2Sim2Robotseen12.044.234.052.035.4
CHORDseen3.698.6100.099.699.6

Under the same budget, Dex-One2Many reaches 94.8% on the unseen set, while both baselines stay below 4% throughout training. The baselines do learn their training configurations: CHORD exceeds 98% on its seen resets after about 85M steps, and Human2Sim2Robot fluctuates between 0% and 61%. However, this does not carry over to unseen configurations. Within the same environment-step budget, Thus, under the same environment-step budget on this task, only Dex-One2Many improves its success on unseen configurations.

[1] Paliwal et al., “Do as I Do: Dexterous Manipulation Data from Everyday Human Videos” (2026).

[2] Lum et al., “Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstration” (2025).

[3] Zhu et al., “Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration” (2026).

Comments on the paper

1 comment
Area_Chair_YPa6OpenReview ↗

Reviewers-Authors Discussion Reminder

Dear Reviewers,

The review process for the paper has reached the rebuttal stage, and the authors have submitted their responses. Reviewers are encouraged to engage actively with these responses. Please clarify whether your initial concerns have been addressed and indicate if any new questions have arisen. Your meaningful participation is essential to maintaining a healthy ecosystem for ICLR.

Your AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.