- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents a method that converts a human demonstration into stage-wise relational graphs, then uses those graphs to generate diverse training resets, observations, and rewards for RL training. The results show that the policies can generalize to configurations beyond source videos.
Critical strengths
- The idea is original. Fitting the video into a stage-wise scene graph instead of a motion to imitate is really interesting.
- The results are compelling with sufficient real-world task evaluation.
Critical weaknesses
- Generalization is overstated. It can only generalize to different configurations but not new tasks or scenes.
Questions for authors
- How autonomous is VLM? Does it automatically turn the full sequence into stages and extract the established relations? Or there are more manual curation or procedures involved?
- What is the quality requirement for the "single human video" input? It's hard to assess how critical the real-to-sim process is to the success of proposed method.
3 comments
Response to HTAb (2/2)
(Q2) What is the quality requirement for the single human video input?
In our pipeline, the input video is used for two things: (1) extracting the task’s stage-wise scene graph, and (2) reconstructing the object assets.
For (2), the task-relevant objects need to be clearly visible in the first frame, from which their assets are reconstructed. This is a common requirement in real-to-sim pipelines: CHORD [1], for example, also reconstructs each object mesh from a single reference frame with SAM3D [2]. For (1), however, our method requires less than prior work. CHORD additionally tracks the object and hand throughout the video, whereas our VLM only needs to recognize the relations, i.e., which objects are involved and in what order they are grasped, contacted, or placed. Since the demonstrated motion is not used for policy learning, the demonstration need not be slow or clear enough for hand or object motion to be tracked, and the hand may be partially occluded.
To illustrate this, we added an example figure to the supplementary material (extract_from_video_compare.pdf), using a video with heavy occlusion: a ball is placed in a cup, and the cup in a bucket, so the ball stays hidden by the hand or the cup, and the demonstrator’s sleeve partially covers the hand. When reconstructing this demonstration with CHORD’s pipeline, the tracked ball pose drifts away from the cup once the ball is occluded. In contrast, our VLM correctly extracts the eight-stage scene graph, including the nested inside relations, since it only needs to recognize the relations rather than track the motion.
[1] Zhu et al., “Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration” (2026).
[2] SAM 3D Team, “SAM 3D: 3Dfy Anything in Images” (2025).
Thank you again for your time and thoughtful feedback on our paper.
As the discussion period ends in about two days, we wanted to kindly check whether our response has addressed your concerns. If you have any remaining questions or would like further clarification, please let us know. We would be happy to address them before the deadline.
Response to HTAb (1/2)
(W1) Generalization is overstated
We thank the reviewer for this comment. We clarify that our generalization claim concerns different configurations of the demonstrated task, rather than transfer to entirely new tasks or scenes.
Importantly, unlike parallel-jaw grippers, dexterous hands may require substantially different grasp strategies as initial or goal object poses change. To address this challenge, we leverage grasp synthesis to generate diverse feasible grasps and incorporate them as prior knowledge into the RL initial-state distribution. Rather than prescribing a fixed grasp or motion, this enables the policy to autonomously discover which grasp strategies are most suitable for different configurations through RL.
Accordingly, our generalization claim focuses on three axes: (i) unseen initial object poses, (ii) unseen goal object poses, and (iii) grasp strategies not shown in the demonstration.
To make this scope explicit, we will revise the abstract to emphasize generalization across configurations of the demonstrated task, replacing broader expressions such as "generalizable policy" and "unseen scenarios." We will also clarify in the introduction that our goal is to generalize beyond the specific execution shown in the video, rather than to entirely new tasks or scenes.
(Q1) How autonomous is VLM, and how does it automatically turn the video into stage-wise scene graph?
Yes, both are automatic. Given a single human demonstration video, the VLM is queried once with a shared prompt template containing the current predicate vocabulary to identify task-relevant objects, their roles, and the ordered relations established in the demonstration. The resulting relations are then automatically converted into stage-wise scene graphs, including intermediate approach stages.
To clarify the degree of automation, we distinguish what the user provides from what our framework constructs automatically when applying Dex-One2Many to a new task.
User-provided inputs.
Automatic pipeline.
Importantly, users do not need to design task-specific rewards, reset distributions, or training procedures. These are constructed using reusable predicate-level definitions. Even when a new predicate is required, its definition can be reused across subsequent tasks without redesigning the RL pipeline.
Thus, applying our framework to a new task primarily requires a single demonstration video, with additional predicate definitions only when the existing vocabulary is insufficient. The remaining task formulation and training procedures are handled by the same reusable pipeline.