- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies one-demonstration adaptation of pretrained vision-language-action models. The models first undergo fine-tuning on pretext behaviors from the target setup. They then adapt to held-out tasks involving environmental changes, new compositions, or skills absent from the pretext set.
FOSIL updates all model parameters for 1-30 gradient steps using the original training objective. The paper studies the role of robot pretraining and the structure of the parameter updates. It also refines demonstration labels by combining policy-generated motion with demonstrated waypoint and gripper constraints.
Experiments cover LIBERO-PRO, compositional LIBERO tasks, and physical manipulation with and MolmoAct2. FOSIL improves success over the pretext-trained models. It also achieves higher environmental suite averages and compositional success rates than the tested in-context baselines.
Critical strengths
The experiments show substantial improvements from a few updates on one demonstration. For , average success on LIBERO-PRO task perturbations increases from 25% to 49% after one update. LIBERO-10 success increases from approximately 5% to 52% after ten updates. The results cover two pretrained VLAs and include physical robot evaluation. They establish that full-parameter fine-tuning is a strong baseline in this setting.
Critical weaknesses
-
The adaptation gap is small in much of the evaluation. The models first receive 254 demonstrations on the same robot and setup, including 111 throwing demonstrations. Most test tasks modify or combine these trained behaviors. Only scooping and sweeping test skills absent from the pretext data. The results show that standard fine-tuning works well after this preparation, but provide limited evidence for learning substantially different behaviors from one demonstration.
-
A low-rank update preserving performance does not show that the pretrained model already possessed the required skill. The refinement results also do not show whether the policy-generated actions are needed. The method smooths the actions and removes frames near grasps. The authors should check whether simply removing those frames or smoothing the demonstrated actions gives similar improvements.
Questions for authors
- How much does adaptation depend on closely related pretext behaviors? For example, does adaptation to throwing new objects still work if throwing is excluded from pretext training, while the other behaviors are retained?
- Does refinement still help when the baseline removes the same near-grasp frames but trains on the recorded actions? How does it compare with simply smoothing the demonstrated actions?
3 comments
Rebuttal by Authors (2/2)
"Does refinement still help when the baseline removes the same near-grasp frames but trains on the recorded actions?" … "How does it compare with simply smoothing the demonstrated actions?" (W2 / Q2)
Refinement still helps. Removing the same near-grasp frames and training on the recorded actions gives 48.8% and 39.6%, against 76.2% and 75.2% for refinement. App. G.4 also reports a separate oracle pace-matching experiment and comparisons over larger adaptation budgets (Fig. 14).
The unadapted and recorded-demonstration rows are three-seed LIBERO-Object means (Table 8). The original single-seed recorded-demonstration results were approximately 52% / 36%.
| Adapted on (LIBERO-Object, one step) | Task perturbation (%) | Object swap (%) |
|---|---|---|
| Unadapted | 9.5 | 10.3 |
| Recorded demonstration | 47.5 | 37.3 |
| Near-grasp frames removed [new] | 48.8 | 39.6 |
| Self-refined demonstration | 76.2 | 75.2 |
-
Frame removal changes success by 1.3 and 2.3 points over the recorded demonstration, so it does not account for the gain.
-
Pace matching, separate run. App. G.4 reports resampling recorded trajectories at the policy's pace using oracle speed information from similar training demonstrations. This run reports pace-matched versus refined success of 56.0% versus 48.2% for task perturbation (), and 75.2% versus 72.2% for object swap. These results come from a separate run from the Table 8 comparison above. The oracle speed requirement also limits direct application to a new dataset.
-
Suboptimal demonstrations. To examine refinement beyond pace matching, we also evaluate adaptation on successful demonstrations whose execution differs from the training distribution. Appendix G.4 (Fig. 14) shows degradation with more steps on recorded demonstrations, while refinement preserves or improves success. Panel (a) covers five wrist-heavy LIBERO-Object tasks; panel (b) covers all ten LIBERO-Spatial task-perturbation tasks. Fig. 15 shows corresponding wrist-motion failures and refined rollouts.
-
Smoothing. The smoothness prior in App. C acts only on the correction that moves a policy sample onto the demonstrated waypoints; it does not filter the recorded actions. We did not low-pass or spline-smooth the recorded actions. The controls investigate frame removal and execution pace; they do not separately isolate the contribution of wrist orientation. The original schematic has been replaced by the illustration of task constraints and execution properties in revised Fig. 6(a).
Appendix G.4 discusses the contribution of pace and the differences that persist with more adaptation steps.
Official Comment by Authors
We are following up to ask whether our response addresses your concerns. To the best of our ability, we have addressed the questions through new results, additional experiments, and clearer interpretations:
- Dependence on related pretext behaviors (W1, Q1). We added the requested throwing hold-out, new-skill hold-outs in simulation, and compositional pretext ablations. These show that one demonstration can teach behaviors absent from pretext training, while related prior experience substantially improves adaptation speed and robustness. We explicitly distinguish acquiring a new motion from executing it reliably.
- Interpretation of low-rank updates (W2). We clarified that compact updates do not establish that the unadapted model already possessed the target skill. But rather supports the finding that pre-training enables effective adaptation through compact parameter changes, and connect its relation to previous work.
- Refinement controls and smoothing (W2, Q2). Additional experiments show that frame removal alone does not explain the refinement gains. Oracle pace matching can recover the one-step benefit in a separate comparison, while longer-budget experiments show benefits of refinement beyond pace on demonstrations with unusual wrist motion. We also clarified the role of the smoothness prior and replaced the schematic that could suggest filtering of recorded actions.
We posted our detailed responses on November 14 to allow time for further discussion. Is there any concern that remains unresolved? We are very open to running additional experiments during the remaining rebuttal period and providing any further detail.
Your questions led to direct tests of both the method’s capabilities and its limits. We believe this new evidence strengthens the paper’s conclusions and hope you will reconsider your assessment.
Rebuttal by Authors (1/2)
We thank the reviewer for the careful reading, for emphasizing the few-step gains across two VLAs and on the physical robot, and for asking for the throwing hold-out and the refinement controls. Pointers below are to the revised PDF (blue).
With throwing removed from pretext, one demonstration still teaches the throw, but it takes about 30 steps instead of 1, and the soap lands in the strainer in 25% of trials.
Real-world throwing hold-out (App. G.5, Fig. 16). We retrained pretext without throwing and kept the other behaviors.
This dependence is discussed in Sec. 4 and App. G.5, where we highlight how robustness of adaptation for new-skills depends on task difficulty. Thus we hope that future models, with larger pretext behaviors will continue to improve few-step performances on even harder tasks like our more difficult novel skills learning included in the ablation.
New skills in simulation (App. G.3, Table 7, Fig. 13). We trained π0.5 on 78 LIBERO-90 behaviors, excluding all tasks involving turning stove knobs, opening microwave doors, and opening drawers. We then adapted from a single demonstration on the 12 held-out tasks, together with 20 environmental and 19 compositional tasks (shared grid 0, 1, 32, 64, 128 steps).
Compositional pretext hold-out (Table 9; App. G.5). Real-world success rates (%) per sub-task over 20 rollouts, with one demonstration and ten gradient steps from the π0.5 initialization:
With the pretext behavior supplying the second sub-task held out, Zip falls from 55% to 10% and Reach & pick from 80% to 25%, while Handover falls from 60% to 50%.
The reviewer is right on this point: a low-rank update that preserves the adaptation gain is not evidence that the unadapted policy could already execute the skill. What the analysis shows is that robot pretraining makes the update compact:
This is consistent with Aghajanyan et al. (2021), who show that pretraining lowers the intrinsic dimension of downstream adaptation. Section 5.1 explicitly clarifies that the result does not show that the unadapted model already possesses the behavior (lines 408–410).
References: Aghajanyan et al. (2021), "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning"