acceptodds
Under review as a conference paper at ICLR 2027

Fine-Tuning VLAs with Self-Demonstrated Generative Control for Multi-Task Manipulation

Abstract

State-of-the-art vision-language-action (VLA) models such as pi0.5 exhibit strong semantic understanding, instruction following and task behavior. However, when deployed on new robots, even minor mismatches in hardware configuration relative to pretraining can cause severe performance drops. Finetuning the VLA on in-domain expert data from the new embodiment improves performance on the expert task but leads to a loss in its original instruction following and behavioral priors. In this paper, we propose a self-supervised method that generates online interaction rollouts from the zero-shot VLA as additional training data for finetuning. Our experiments show this yields strong multi-task manipulation policies on the target robot that (1) inherit pretrained tasks distilled from the zero-shot VLA without expert demos for them, (2) improve success and generalist instruction following beyond expert-only fine-tuning on expert-teleoperated tasks, and (3) mitigate forgetting of held-out tasks from the pretrained policy, not part of fine-tuning. We demonstrate the success of our approach across test sets probing generalization on a real ALOHA robot and a new simulation benchmark in RoboTwin. Video results are available at https://anonymous-sgc-vla.pages.dev.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLAs, post-training, forgetting, self-supervision, generative replay

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gozoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to adapt a pretrained VLA to a new robot with limited expert demonstrations without losing its instruction-following ability or previously learned skills. The method combines expert demonstrations collected from teleoperation with rollouts collected by running the frozen pretrained policy on the target robot. The authors evaluate the method on a physical ALOHA robot and in RoboTwin simulation. It improves performance on expert demonstrated tasks and reduces forgetting of other tasks. For example, real-world pick-and-place success increases from 0% (zero-shot) to 55% without expert placing demonstrations, and success on old simulation tasks increases from 16.6% to 70.6%. Experiments also examine novel objects, task compositions, and held-out skills, though the benefits vary across tasks and depend on the quality of the self-demonstrations.

Critical strengths

The paper tackles a practical problem in adapting pretrained VLAs. These models may understand instructions but struggle to execute them on a new robot. Fine-tuning on expert demonstrations can improve control but also cause the model to forget other skills. Preserving those skills without access to the original pretraining data is useful for researchers working with released models.

Critical weaknesses

The method still requires manual data selection. The authors choose task prompts and filter out rollouts that ignore the instruction or behave unsafely. They also find that poorly filtered self-demonstrations can hurt performance compared with expert-only fine-tuning. It is therefore unclear how much human effort the method needs and whether it would remain effective across a larger set of tasks.

Questions for authors

Clarification Question: How much time did you spend choosing prompts and filtering rollouts, and how many rollouts were discarded? Did you use the same filtering rules for all tasks, or did they need to be adjusted for each task?

1 comment
AuthorsOpenReview ↗

Response to the Review

We thank the reviewer for highlighting the strengths of our practical problem setting, and for providing us valuable feedback.

Clarification Question: How much time did you spend choosing prompts and filtering rollouts, and how many rollouts were discarded?

Self-demo collection time, rejection rates, VLM automation. To quantify the time and effort in the self-demo collection process, we logged 45 self-demos, which took 70.2 minutes in total. Of that time, each self-demo took on average 1.25 min, and 13.0 minutes were spent on filtering which demos to keep. 33% of the self-demos were kept (27 follow the "pick" instruction; of those, 15 also follow "place"). This filtering could be automated. We tested that GPT-6.1 Sol (High) performs the filtering with 89% agreement to the manual filtering. We added these to the main paper Section 6, Appendices A.2 and A.4.

Did you use the same filtering rules for all tasks, or did they need to be adjusted for each task?

Yes, we used the same filtering rules. We filter self-demonstrations based on whether the robot safely attempts to follow the instructions.  E.g., for pick and place, it must: (1) not wedge its gripper against the container or table or get it stuck; (2) attempt to pick up the specified object; and (3) carry out a motion as if moving towards the container for a place. Task success is not a criterion. We add the prompt used for VLM automated filtering with these three criteria to A.4.

They also find that poorly filtered self-demonstrations can hurt performance compared with expert-only fine-tuning.

Thanks to your feedback, we conducted additional analysis for our poorly filtered self-demo experiment in A.3. Our ablations suggest that self-demos with less place behavior can still help a different expert task, but that correct pick attempts alone are insufficient to improve pick-place tasks. For the pick-place ablation previously reported under limitations section, all 15 self-demos attempted to pick correct objects across both sets, but only 53.3% attempted place in the substitute (lower quality semantic filtering set) compared with 86.7% in Our original set and performance fell on pick-place.

In the gear insertion benchmark, the self-demos contained pick attempts in 100% and place attempts in 72.4% of episodes and improved expert gear-insertion success from 40 to 90, even as IF (post-grasp) on pick-and-place fell from 85% to 35%. Hence, a rollout can fail the task and still be useful, but it should contain an attempt at the behavior we want to distill.

Reviewer_B83UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a self-demonstration-based approach for fine-tuning vision-language-action (VLA) models while mitigating catastrophic forgetting. A frozen pretrained VLA generates rollouts on previously learned tasks, which are combined with expert demonstrations of new tasks for fine-tuning. The key finding is that even unsuccessful self-demonstrations can preserve useful behavioral knowledge. Experiments on real robots and RoboTwin demonstrate improved task performance and retention of pretrained capabilities.

Critical strengths
  1. Interesting empirical insight. The paper demonstrates that even unsuccessful VLA rollouts can provide useful supervision for preserving pretrained behaviors.

  2. Promising experimental results. The method achieves substantial improvements over expert-only fine-tuning across real-world and simulation benchmarks, supported by extensive ablations.

Critical weaknesses
  1. Limited methodological novelty. The proposed approach primarily combines established ideas from experience replay and policy distillation: collecting rollouts from a frozen pretrained policy, filtering them, and mixing the resulting data with expert demonstrations for standard imitation learning. While the empirical findings are interesting, the algorithmic contribution appears incremental.

  2. Insufficient details on manual filtering. The method relies on manually selecting task prompts and filtering self-demonstrations, yet the filtering procedure is not sufficiently specified. Although the paper provides qualitative filtering criteria and the appendix lists the selected demonstrations, it does not report the number of candidate rollouts, rejection rates, or human effort required. These details are important for assessing reproducibility and scalability.

Questions for authors
  1. Could the authors clarify the methodological novelty of the proposed approach relative to existing generative replay and policy distillation methods, beyond the use of closed-loop VLA rollouts and manual filtering?
  2. Could the authors clarify the self-demonstration filtering protocol, including candidate rollout counts, rejection rates, and manual effort?
  3. How much of the improvement comes from preserving pretrained capabilities versus introducing additional task-relevant supervision? Since the self-demonstrations include pick-and-place and laundry tasks that closely resemble the evaluation tasks, could the authors provide an ablation using self-demonstrations from less closely related tasks?
1 comment
AuthorsOpenReview ↗

Response to the Review

We thank the reviewer for pointing out our key insights and their comprehensive feedback and questions.

Could the authors clarify the methodological novelty of the proposed approach relative to existing generative replay and policy distillation methods, beyond the use of closed-loop VLA rollouts and manual filtering?

We agree with the reviewer that our method builds on ideas from generative replay (Sin et al., 2017) and policy distillation (Rusu et al., 2016). Our key contribution is to show that the base policy's closed-loop rollouts on the target embodiment, even when unsuccessful, compose with expert demonstrations of a different task to recover pretraining behavior (place: 0% to 55% with no expert place demos), improve the expert tasks themselves (65% to 90% on pick; 40% to 90% on gear insertion), and preserve held-out skills better than an oracle trained on expert demos for all fine-tuning tasks (38% vs 82% correct push motions).

These rollouts are necessary rather than incidental: self-demos provide states along with distillation targets, distilling the same base policy offline on expert-visited states yields 0% success versus 55% with self-demos (Table 9), showing that the policy must be distilled on its own state distribution.

Furthermore, we highlight that our approach only assumes access to the pretrain checkpoint and our target robot. Consequently our findings present an effective way of overcoming the absence to the pretraining data and the pretrain robots, which is an otherwise common source for prefilling replay buffers. We have also added and highlighted these points in the related works and method sections, and added more analysis for offline distillation.

Could the authors clarify the self-demonstration filtering protocol, including candidate rollout counts, rejection rates, and manual effort?

Self-demo collection time, rejection rates, VLM automation. To quantify the time and effort in the self-demo collection process, we logged 45 self-demos, which took 70.2 minutes in total. Of that time, each self-demo took on average 1.25 min, and 13.0 minutes were spent on filtering which demos to keep. 33% of the self-demos were kept (27 follow the "pick" instruction; of those, 15 also follow "place"). This filtering could be automated. We tested that GPT-6.1 Sol (High) performs the filtering with 89% agreement to the manual filtering. We added these to the main paper Section 6, Appendices A.2 and A.4.

Filtering rules. We filter self-demonstrations based on whether the robot safely attempts to follow the instructions. E.g., for pick and place, it must: (1) not wedge its gripper against the container or table or get it stuck; (2) attempt to pick up the specified object; and (3) carry out a motion as if moving towards the container for a place. Task success is not a criterion. We add the prompt used for VLM automated filtering with these three criteria to A.4.

How much of the improvement comes from preserving pretrained capabilities versus introducing additional task-relevant supervision? Since the self-demonstrations include pick-and-place and laundry tasks that closely resemble the evaluation tasks, could the authors provide an ablation using self-demonstrations from less closely related tasks?

Evaluation on completely different held-out task families not part of either expert- or self-demo supervision in post-training. In section 5.5 and Table 7, we compared performance on held-out skill families, "push objects" and "pour/turn over objects", neither method fine-tuned on. Post-training is done on ES (pick-up) and SS (pick-place). In a blind user study we conducted on robot rollouts, Ours was preferred 75% over 11% (Oracle) on “turn over” skill and 68% over 9% on “push” skill. This was a surprise finding since the Oracle here is an expert collecting “pick-and-place” tasks on our robot, showing self-demos preserve held-out motion priors better than the Oracle!

Reviewer_pJ6jOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose training off the shelf policies (like pi0.5) on self-generated demos and expert human demos to retain their pretrained capbilities and learn new skills on new embodiments. The authors demonstrate the success of their method on a new task in robo twin and on ALOHA robots in reality, where, adding self demos to expert data significantly improves performance even through an embodiment gap.

Critical strengths
  • The paper is clear and easy to follow. The results are clearly presented and many important ablations are included.

  • The paper leverages the clever idea of cotraining with data predicted by the model and hence from its pretraining distribution to maintain and remember pre-training capabilities. This is similar to ideas in the VLM/LLM finetuning space and is excellent to see it implemented in a practical way in robot learning.

  • The capability of the method to induce higher success rates on tasks not in pretraining, like the orang gear on peg task, and much higher than just with expert demos, is exciting.

Critical weaknesses

I think a critical weakness is that we need the base policy to have some level of base performance for every new embodiment and task, especially with the diverse progress metrics in figure 4 (and close to full success in some rollouts). Can the authors analyze and mention the level of base capability needed for self demos to have any impact? How many self demos are needed in the finetuning distribution per new task and embodiment in the real world and how does this vary with tasks and embodiments further away from the pretraining distribution? It would also be helpful to include standard deviations in the paper.

Questions for authors

Please see Weaknesses

1 comment
AuthorsOpenReview ↗

Response to the Review

We thank the reviewer for their valuable and nuanced review, and for finding our idea and results exciting.

I think a critical weakness is that we need the base policy to have some level of base performance for every new embodiment and task, especially with the diverse progress metrics in figure 4 (and close to full success in some rollouts). Can the authors analyze and mention the level of base capability needed for self demos to have any impact?

Nature of self-demos. In Figure 4, the arm attempts to grasp the object, misses, and then moves toward the container empty-handed. None of the self-demos have the object in the gripper. Since this can only been seen qualitatively, we share the self-demos used in Table 1 on our anonymous website https://anonymous-sgc-vla.pages.dev/. The annotations in Figure 4 are for detailed analysis and not for filtering. For filtering, we check only that the rollout is safe, that the arm attempts to pick the specified object, and that it moves toward the container for a place.

Base capability. Due to the embodiment gap, pi0.5 has 0% success rate on our robot. The base policy does not need to be successful at the task, but it does need to have some idea how to follow the instruction. We used the IF (pre-grasp) and IF (post-grasp) metrics to quantify base capability. For "pick-up" skill family, this is indicated by 80% and 70% IF (pre-grasp) for picking ID and OOD objects. The caterpillar "gear insertion" task is largely absent as the base policy hardly moves (0% IF). For "pick-and-place", it goes to the correct object for pick 70% and moves towards correct container for place 25% (IF, post-grasp). This indicates whether a task can be distilled through self-demos or not and we distill the pick-place behavior since we observe its priors. A self-demo on a task that is far outside the training distribution is unlikely to be useful. However, the benefit of our approach is still large because it is possible for pretraining to cover a broad distribution of tasks but prohibitive to retrain all those tasks for each new embodiment. If base capability is present for a task, we can distill generalist instruction following and motion priors directly through their self-demos. At the same time, self-demos improve other expert skill families across ID/OOD object test sets. We show this for both "pick-up" where the task is present in pretraining and where it is not, "gear insertion".

No priors exist on embodiment. If no priors exist on the embodiment or hardware changes are made in the future, then self-demos can be helpful in subsequent post-training stages to learn new skills after a first round of expert training as shown in our simulation benchmarks (Table 2).

During rebuttal, we added a new simulation experiment with a controlled embodiment gap, moving the top camera slightly forward and down, E1 (mid-train) -> E2 (post-train) (Figure 12, Table 4). This reduces mid-training performance from 90.8% in E1 to 51.6% zero-shot in E2 (reduced base capability). Here, the expert post-training skills "stacking cubes" are not present in the 10 mid-training skills, causing further drop in mid-train performance from 51.6% to 20.9%. Despite reduced base capability, self-demos in E2 raise mid-train task performance to 38.5%.

How many self demos are needed in the finetuning distribution per new task and embodiment in the real world and how does this vary with tasks and embodiments further away from the pretraining distribution?

In our real-robot experiments, in Appendix G we report 15 self-demos with 30 expert pick-up demos and 29 self-demos with 60 expert gear-insertion demos, using 1:1 batch sampling. Imbalanced sampling leads to erratic grasping (opening and closing) in our real-robot runs (Appendix C.1). Based on several real-robot experiments we conducted, as long as self-demo filtering has sufficient place behavior and number of frames for self-demo dataset is greater than the expert dataset size, pick-place self-demo behavior is likely to be distilled. Sufficient number of expert episodes are also required to learn the new task and bridge the embodiment gap for self-demo tasks. We also plan to release our checkpoints, datasets, and code with simulation seeds.

It would also be helpful to include standard deviations in the paper.

We added mean and standard errors for 3 training seeds across our simulation results in Tables 2, 4, 6 of the main paper. We updated the text with the updated mean numbers.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Revision General Comment

We are grateful to all reviewers for their constructive feedback and valuable suggestions. We have updated the revision, incorporating results added during the rebuttal:

  1. Adding collection time, rejection rates, filtering rules and VLM automation for filtering self-demonstrations.
  2. Adding an additional controlled embodiment gap experiment in simulation.
  3. Made changes to related works and reformatted the Appendix to easily locate details and analysis.

We highlighted all key changes in blue color across the main paper and Appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.