acceptodds
Withdrawn from ICLR 2027

DemoPrior: Learning Unseen Robot Tasks from a Single Demonstration with Action-Space Priors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We propose DemoPrior, a one-shot video-to-skill framework that enables a learned vision-language-action (VLA) policy to acquire unseen manipulation skills from a single demonstration video. Existing approaches either learn adaptation mechanisms whose transfer remains tied to the tasks and variations covered during training, or use a test-time demonstration mainly as a weak conditioning signal or trajectory initialization. We instead propose to learn a reusable demo-to-skill transformation that converts information extracted from a new demonstration into executable behavior. Our key insight is that a demonstration provides two complementary forms of task information: semantic structure specifying what entities and interactions matter, and task-relative spatial structure specifying how the behavior should unfold in space. DemoPrior extracts these semantic and spatial cues from the video, transfers them to the current scene, and maps them into an action-space skill prior. A flow-based VLA is trained to refine this prior using closed-loop observations, learning how to turn video-derived structure into robot actions rather than generating a new skill from scratch. At test time, a single demonstration can therefore introduce a new skill without any task-specific policy update. Across our unseen-task evaluations, DemoPrior consistently improves demonstration-based skill transfer, and achieves a 46.7% success rate on real-world unseen tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, One-Shot Imitation Learning, Learning from Demonstration, Flow Matching, Robot Manipulation, Skill Transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DPJaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a new method for one-shot/few-shot learning for VLAs. It breaks an expert demo into two information sources: semantics and positional. Semantics are identified using SAM2, CoTracker, and SigLIP, where objects from the demonstration and current scene are matched. Then they can transfer by considering the arc of movement in the demonstration, and accounting for the differences in position of the current scene minus the demonstration scene using a linear interpolation over time. This allows them to create an approximate sequence of states in the new scene, which they somehow(?) covert into an action draft. This action draft is then converted into a viable action via flow matching.

Critical strengths

I like the idea of extracting semantic information from a demonstration and mapping that to information in the new scene. That is a nice way of using that information explicitly rather than implicitly as in prior methods.

The experiments look good. Any one shot performance in real life above 0% is awesome.

Critical weaknesses

Presentation can be hard to following. For example, line 198, Q^D is the demonstration position, Q^E is the scene/execution position I think? This should be clearly stated. I get how \tilde{e} is like a corrected trajectory in the new scene, but how is this converted into an action draft (line 214). The action draft is an approximate action, which you then using flow matching to "correct" into a viable action? Figures 1 and 2 are not interpretable to me.

Questions for authors
  1. How do you convert \tilde{e} into an action draft? Is this just something like IKD from the approximate state trajectory
  2. Is \tilde{e} refined during the trajectory?
  3. How do you choose the stopping times?
  4. How do you know what dimensions cannot be recovered from the monocular view? Do you assume prior knowledge on this?
  5. You only pretrain, there is no training for every new task, correct?
Reviewer_XGhaOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DemoPrior, a framework for transferring manipulation behaviors from a single demonstration without task-specific test-time policy updates. The method extracts semantic correspondences and task-relative motion from the demonstration, transfers them to the execution scene, and constructs an action-space draft. A pretrained flow-based VLA is fine-tuned to use this draft as its source distribution, while dimension-specific early stopping limits refinement to preserve useful demonstration structure. Experiments cover held-out LIBERO tasks, SimplerEnv, human demonstrations, and a physical UR5 robot. DemoPrior reports substantial improvements over the language-only backbone, DemoDiffusion, and open-loop draft execution, reaching 76.1% on held-out LIBERO-90 tasks and 46.7% on three unseen real-world tasks.

Critical strengths
  1. Relevant problem and clear formulation. The paper addresses camera–interaction entanglement in latent action learning through a well-motivated crossed-intervention design.
  2. Strong improvements on controlled evaluations. InterACT reduces Camera-OOD action error from 1.44 to 0.34 and improves controlled cross-view Recall@5 from 3.39% to 29.43%.
  3. Informative evaluation and transparent reporting. The paper jointly assesses alignment, scale retention, and predictive information, while explicitly reporting negative results and limitations.
Critical weaknesses
  1. DemoPrior builds on a pretrained VLA and uses multi-task expert demonstrations to learn draft refinement. However, the LIBERO task split and SimplerEnv instruction-based exclusions do not establish that the underlying manipulation behaviors are absent from training. I find the results convincing as evidence of demonstration-guided task transfer, but less conclusive about acquiring genuinely new skills.
  2. In the LIBERO implementation, rotational dimensions are generated by the VLA rather than initialized from demonstration-derived rotation trajectories. It is therefore unclear how a demonstration communicates a new rotation direction, angle, or orientation sequence to the policy. Successful execution of such tasks may depend on behaviors already learned during pretraining or fine-tuning, and the experiments do not isolate this dependency.
  3. Section 3.1 lists the perception modules but leaves several important steps unclear, including interaction-anchor selection, coordinate alignment, and conversion into temporally aligned actions and gripper commands. For UR5, the conversion from visual motion to absolute joint targets is particularly important, yet Appendix G.4 does not provide the promised inverse-kinematics details. Simulation also uses simulator-provided segmentation, while real-world draft construction uses depth. These choices are reasonable, but the incomplete description and lack of evaluation with predicted segmentation make the pipeline difficult to reproduce and its dependence on reliable preprocessing hard to assess.
  4. The three unseen UR5 tasks—cube-to-drawer, banana-to-plate, and cube-to-plate—are all relatively simple pick-and-place combinations. I would have liked to see tasks requiring sustained contact, precise pose alignment, or a different manipulation primitive to better assess the scope of the claimed skill transfer. The only real-world human-demonstration task is classified as seen in Table 17, and each unseen robot task has only ten evaluation trials. The real-world results are encouraging, but their difficulty and coverage are limited relative to the paper’s broader claims.
Questions for authors
  1. Could you provide a reproducible description of the demonstration-to-draft pipeline. Please also clarify any manual inputs and the use of simulator segmentation and depth
  2. Could you add more challenging real-world experiments, especially tasks involving new manipulation behaviors, precise rotations, or sustained contact? More trials and human demonstrations of genuinely unseen tasks would make the skill-transfer claims more convincing
Reviewer_Tu7dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DemoPrior, a one-shot video-to-skill framework that enables a VLA to adapt to a new task given one demonstration video. It first extracts semantic and task-relative spatial structure from the video and transfers them to the deployment scene to generate an action draft. Then, they use a pretrained flow-based VLA to transform the draft into executable actions. The authors then conduct experiments in both simulation and real-world, showing that DemoPrior allows the model to quickly learn a unseen tasks given only one demonstration.

Critical strengths
  1. The proposed DemoPrior method is well-motivated and intuitive, and it could be useful in many deployment scenarios.
  2. Both simulation and real-world experiments validate DemoPrior's effectiveness.
Critical weaknesses
  1. Missing analysis of why DemoPrior is better than DemoDiffusion. DemoDiffusion is the only baseline this paper compared to that makes use of the demonstration video of the new task. The results in the paper shows that DemoPrior significantly outperforms DemoDiffusion, but it's unclear why DemoDiffusion fails in many scenarios and how DemoPrior overcomes them.

  2. The experiment setup seems to be too easy.

    1. For instance, it seems that the demonstration video and the test-time evaluation scenario are of the same distribution in your evaluation in LIBERO. The human demonstration videos are also very similar to the deployment setting, only with the robot arm replaced by the human arm. Can the system still work when the demonstration video is very different from the deployment setting?

    2. The current evaluated tasks are single-step tasks, while in the real-world scenarios such tasks are very rare. The paper should consider evaluating its system on more realistic and longer tasks.

Questions for authors
  1. How does DemoPrior perform on more realistic and long-horizon tasks?
  2. Can the system still work when the demonstration video is very different from the deployment scenario?
  3. Is there more analysis of why DemoPrior outperforms DemoDiffusion?
  4. Can the authors provide more details on how to fine-tune the VLA to refine the action draft? Where does the fine-tuning data come from? Do we need to fine-tune the VLA in every new scenario?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.