- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies minimal-data adaptation of pretrained robot policies, where one demonstration initializes subsequent online learning. MiDAS combines behavior cloning, frozen VLA representations, lightweight action editing, PA-RL updates, and success-balanced replay. Experiments on LIBERO-Long, selected RoboCasa tasks, and two bimanual real-world tasks demonstrate improvements over single-demonstration BC, alongside analyses of representation quality, training choices, and generalization limitations.
Critical strengths
- Relevant adaptation setting. Studying whether one demonstration can bootstrap online improvement addresses a useful question about reducing task-specific supervision.
- Useful empirical evidence. The simulation and hardware results demonstrate feasibility, while the representation and distribution-shift analyses help characterize when adaptation succeeds and where it remains limited.
Critical weaknesses
- Limited methodological novelty and insufficient isolation of the proposed modifications. The recipe largely combines established ingredients from residual RL, warm-start learning, and PA-RL. Although such integration can be valuable, the experiments do not sufficiently establish which modifications provide additional insight specifically in the single-demonstration regime. The main design ablations are concentrated on one task, and removing the base-action input nearly matches the default method. More controlled comparisons are needed to distinguish essential principles from interchangeable implementation choices.
- The closest-method comparison is incomplete and does not establish a consistent advantage. On LIBERO-Long, MiDAS achieves 91.2±4.2% versus DICE-RL’s 89.3±1.8% over three seeds, without a confidence interval or statistical analysis of the small difference. Appendix E.1 also reports that DICE-RL is more sample-efficient on five tasks, worse on one, and comparable on four. These results support shared design principles but do not establish consistent superiority. DICE-RL results are also missing on RoboCasa.
- The hardware evaluation lacks a strong online-adaptation baseline. It only compares with BC.
- Hardware task coverage is narrow. Both tasks involve pick-and-place, leaving applicability to precision-critical manipulation or substantial behavioral changes unclear.
- The practical benefit of single-demonstration adaptation remains unclear. The paper aims to enable real-world RL from one demonstration, but online practice remains reset-dependent and requires operator involvement. Starting from one demonstration could therefore require more total human effort than collecting five demonstrations and reducing subsequent practice. Without accounting for reset and intervention costs, it is unclear whether minimizing demonstration count provides a meaningful practical advantage over a few-demonstration alternative.
Questions for authors
- Contribution clarification: Which modifications distinguish MiDAS from existing residual-RL recipes, and what controlled evidence supports their importance specifically under single-demonstration initialization?
- Comparative evidence: Are DICE-RL results available for RoboCasa or hardware? Can the authors characterize uncertainty in the reported LIBERO performance difference?
- Practical-cost clarification: How are hardware resets and success labels handled, and how much active operator time is required? Does the reported training duration include reset and recovery overhead?
5 comments
Rebuttal by Authors - 2/3
Methodological contributions and comparison with DICE-RL
The distinctions from DICE-RL are the most prominent and consequential in how learning proceeds in the low-success regime.
First, MiDaS continually adds successful trajectories from online experience to a success buffer. By repeatedly sampling these trajectories, we help the policy retain and build on each success it discovers. DICE-RL instead samples from a fixed offline demonstration buffer alongside a standard online buffer, reducing the fraction of offline samples as training progresses. When unsuccessful trajectories dominate the online buffer, previously discovered successes become underrepresented in training. Our success buffer addresses this imbalance and helps prevent previously learned successful behavior from being forgotten. For further details, please refer to Appendix G and Figure 11.
Second, DICE-RL regularizes the residual toward the base policy’s actions, with a filter that relaxes this penalty based on the critic’s predictions and observed returns. This regularization can hinder improvement when the base policy itself is brittle, particularly when the critic does not yet reliably recognize better actions and further exploration is needed for improvement. MiDaS does not impose this penalty, allowing the policy to move away from the base-policy actions as it learns.
We additionally evaluated DICE-RL on RoboCasa. MiDaS substantially outperforms it on the evaluated longer-horizon tasks, achieving 72.7% average success compared with 53.50% for DICE-RL over 3 seeds. The tasks have episode horizons ranging from 500 to 1,300 steps and involve different objects and kitchen fixtures. We summarize this variation below:
| Task | Steps |
|---|---|
| Hot dog: counter→cabinet | 500 |
| Banana: fridge drawer→shelf | 500 |
| Mug→coffee maker; start | 800 |
| Select croissant+jam | 1300 |
We summarize our results below:
| Task | Steps | MiDAS | DiceRL |
|---|---|---|---|
| Hot dog: counter→cabinet | 150k | 100.00 ± 0.00 | 93.33 ± 6.11 |
| Banana: fridge drawer→shelf | 150k | 93.33 ± 7.02 | 95.33 ± 1.15 |
| Mug→coffeemaker; start | 200k | 74.67 ± 20.03 | 19.33 ± 33.49 |
| Select croissant+jam | 200k | 22.67 ± 10.07 | 6.00 ± 10.39 |
| Average | – | 72.67 ± 6.79 | 53.50 ± 8.89 |
These results demonstrate stronger performance for the MiDAS recipe on the evaluated longer-horizon tasks. We hypothesize that success balancing and differences in policy regularization contribute to these gains.
Design decision ablations on three tasks
We extend our design decision ablations on two more tasks and add them in Figure 11 in the updated PDF file. The relevant discussion in Appendix G is also updated in light of the new added tasks. We summarize our findings in the table below:
| Task | BC | MiDAS | MiDAS (no success buffer) | MiDAS (pop base actions) | MiDAS (BC reg 0.1) |
|---|---|---|---|---|---|
| Both Moka Pots → Stove | 13.3 ± 3.1 | 88.1 ± 4.6 | 58.3 ± 45.5 | 82.7 ± 7.2 | 68.0 ± 5.4 |
| Black Bowl → Bottom Drawer | 22.7 ± 6.4 | 98.7 ± 1.2 | 96.0 ± 1.6 | 69.3 ± 13.7 | 91.3 ± 8.1 |
| Alph. Soup + Tom. Sauce → Basket | 4.0 ± 3.5 | 82.7 ± 13.3 | 60.7 ± 43.0 | 6.0 ± 8.5 | 56.0 ± 24.7 |
Across all three tasks, the MiDAS configuration with all its design decisions intact, achieves the highest mean success rate, with omission of each decision reducing mean performance. These differences are particularly pronounced on the Alph. Soup + Tom. Sauce → Basket task, where the initial BC policy achieves only 4.0% success. On this task, MiDAS achieves 82.7% success, while the ablated variants range from 6.0% to 60.7%. These results support the importance of the individual design choices within MiDAS and suggest that they are especially valuable for enabling effective adaptation when the initial policy is extremely brittle and unreliable.
Rebuttal by Authors - 3/3
Removing the base policy action input ablation
In our default method, we provide the base policy’s action as an input to the residual actor, following common practice in residual learning. With this ablation (see Figure 11 in the updated PDF file and the table in the previous comment (2/3)), we find that on some tasks, removing the base actor input matches the performance of MiDAS. However, in settings where the initial policy has an extremely low success rate (Alph. Soup + Tom. Sauce → Basket with 4%), this collapses during the online learning phase. We explain these results by providing intuition behind the warmup phase of online adaptation.
During the 10K-step warmup, the actor learns to approximate the base policy’s action distribution with a behavior cloning loss, acquiring its behavioral prior before online RL begins. This initialization is important because the actor needs to preserve at least the base policy’s low but non-zero success rate to generate successful experience for learning. Without this initial competence, successful trajectories can become too scarce to sustain improvement; removing warmup causes learning to collapse in our experiments. Success balancing and the online policy updates then allow the initialized actor to preserve and build on the successes it discovers. The base policy action removal ablation shows that even without the base policy action conditioning, the actor initialized through warmup can collect successful trajectories and enable successful online improvement in settings where the base policy has reasonable initial success. However, in extremely low initial success regimes, conditioning becomes crucial to avoid collapse.
Together with the warmup and replay ablations, this result shows the role of base action conditioning in MiDAS. In the regime where the base policy's success is extremely low, conditioning is crucial for online improvement. Conditioning can, however, match the full MiDAS performance when the base policy is reasonably successful, and we explain this via the warmup phase where the behavior prior of the base policy is distilled into the residual actor.
Hardware setup, human effort and task coverage
The hardware experiments were designed as a proof of concept for minimal-data adaptation on a real robot. They show that online learning can substantially improve a policy initialized from a single demonstration, with each RL run requiring up to six hours including operational overhead.
The evaluation covers two pick-and-place tasks and compares against BC. It therefore establishes hardware feasibility within this scope; advantages over alternative online-adaptation methods are supported by the simulation comparisons, including the new RoboCasa results. We will add additional hardware experiments for the camera-ready version.
In the current setup, scene resets are manual and success labels are binary and human-provided. The reported training duration, up to six hours per RL run, denotes total wall-clock time for the experiment, including policy updates, robot execution, resets, and recovery overhead. Future systems could reduce human effort overhead through VLM-based success annotations and by alternating between related or cyclic tasks, where executing one task helps establish the starting conditions for another.
Practical motivation
Our motivation is to study adaptation in deployment settings where obtaining a reliable base policy is difficult, the initial success rate can be extremely low, and collecting additional demonstrations is costly or repeatedly required. For example, a remotely deployed robot such as a rover may encounter unfamiliar conditions that make its existing policy brittle, while obtaining demonstrations for each new situation is difficult. Our minimal-data adaptation setting provides an experimental proxy for these scenarios: the policy has some initial task competence but remains unreliable, and we study which properties and design choices enable it to improve through interaction.
In standard controlled environments, collecting additional demonstrations is often feasible and may reduce the amount of subsequent online interaction. In unstructured deployment, however, robots encounter a long tail of conditions that are difficult to anticipate and cover through demonstrations. The practical trade-off depends on the relative costs and availability of demonstrations, resets, recovery, and success feedback.
If your concerns are addressed, we'd be grateful if you could raise your score.
References
[1] RoboTTT: Context Scaling for Robot Policies URL https://arxiv.org/pdf/2607.15275
[2] INSTANT POLICY: IN-CONTEXT IMITATION LEARNING VIA GRAPH DIFFUSION URL https://arxiv.org/pdf/2411.12633
Rebuttal by Authors - 1/3
Thank you for your detailed feedback. We would like to first clarify your concern regarding our primary contribution. We then clarify our methodological contributions and provide new comparisons with DICE-RL on RoboCasa. Across four tasks evaluated over three seeds, MiDAS achieves 72.7% average success compared with 53.5% for DICE-RL, with the largest gains on the longer-horizon tasks. We further discuss how success balancing and policy parameterization differences relate to these results. We then address comments regarding the ablation on dropping the base action as input to the residual. Finally, we clarify the scope of our hardware evaluation, describe the associated human involvement and operational overhead, and explain the practical motivation for studying adaptation from minimal demonstrations. We also extend replay and regularization ablations to two additional tasks in the updated PDF and find that MiDAS' design decisions are crucial especially when the base policy is extremely unreliable, and lead to stable online improvement across tasks with varying initial task success.
Contribution of our work
Our primary contribution is to establish and characterize minimal-data adaptation: whether pretrained robot policies can learn reliable task execution from as little as one demonstration followed by online interaction, and what enables this when the initial policy rarely succeeds. To the best of our knowledge, we are the first to demonstrate reliable adaptation of pretrained robot policies from a single task-specific demonstration, followed by online RL. MiDaS provides a concrete adaptation recipe through which we study the roles of pretraining, demonstrations, and online learning, identify effective design choices, and examine the limits of adaptation. That said, MiDaS is not the only recipe that one can build for MDA, and we clearly remark this in the paper (Section 5, Experimental Results Subsection (lines 312-317)).
Regarding comparisons to recent methods: while MiDaS and DICE-RL share ingredients from residual RL, our contribution extends beyond the choice of the adaptation algorithm (PA-RL vs Residual TD3) to understanding what makes this minimal-supervision regime tractable. Our additional RoboCasa experiments also show substantially stronger performance with MiDaS, demonstrating that the choice of adaptation recipe matters on these longer-horizon tasks. We discuss the methodological differences and these results in our response below.
Finally, we clarify that our goal is not to recommend collecting one demonstration plus online experience in place of additional demonstrations. Of course, when many demonstrations are available, one should perhaps run imitation learning as best as possible on that data. Only then should they consider using an adaptation algorithm. Rather, our goal is to study how much improvement is possible when task-specific demonstrations are scarce and the pretrained policy is initially unreliable; i.e., whether learning from minimal demonstrations is even a viable choice or not. Similar to why few-shot learning has been regarded as a promising problem statement in robot learning recently [1,2], we believe MDA is an important problem statement, where we want to improve policy performance with as few task-specific demonstrations as possible. Understanding what enables adaptation under these conditions (either through MiDaS or through other methods) is particularly important for unstructured deployment, where diverse and changing environments make it difficult to train a base policy that is reliable across the full deployment distribution.