acceptodds
Under review as a conference paper at ICLR 2027

What Does It Take for a Generalist Robot Policy to Adapt from Minimal Data?

Abstract

A central goal in robot learning is to move beyond task-specific human data collection toward robots that improve through autonomous interaction. Yet fully autonomous learning remains difficult with current policies: sparse rewards and weak zero-shot exploration often prevent online learning from discovering successful behavior. As a first step towards fully autonomous learning, we propose the minimal-data adaptation regime, in which a pretrained robot policy must learn a new task from as little as one demonstration followed by autonomous online interaction. We establish that, although pretrained policies fail zero-shot on many new tasks, a single demonstration suffices to direct their behavioral prior to coarse task-following behavior, albeit with unreliable performance. From this insight, we isolate the key design principles required to convert coarse task-following into reliable task success. These include (1) the surprising efficacy of pretrained VLA representations to support sample-efficient value learning from online interaction (2) the sufficiency of residual policy learning to learn corrective behavior and (3) the need for careful data-balancing to prevent exploration collapse. As a representative of these principles, we propose MiDAS, a simple offline-to-online RL recipe that combines few-demonstration behavior cloning with value-based residual learning over frozen VLA representations. Across LIBERO and RoboCasa tasks, MiDAS substantially improves performance from single-demonstration initializations, including policies that succeed only rarely, validating the principles highlighted above. On a bimanual YAM platform, starting from a brittle initial policy, MiDAS improves task success after approximately six hours of online interaction.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Few shot adaptation of pretrained policies, RL post-training of vision language action models, Robot foundation models and general-purpose knowledge systems, Reinforcement learning for physical robot control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_56fROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies minimal-data adaptation of pretrained robot policies, where one demonstration initializes subsequent online learning. MiDAS combines behavior cloning, frozen VLA representations, lightweight action editing, PA-RL updates, and success-balanced replay. Experiments on LIBERO-Long, selected RoboCasa tasks, and two bimanual real-world tasks demonstrate improvements over single-demonstration BC, alongside analyses of representation quality, training choices, and generalization limitations.

Critical strengths
  1. Relevant adaptation setting. Studying whether one demonstration can bootstrap online improvement addresses a useful question about reducing task-specific supervision.
  2. Useful empirical evidence. The simulation and hardware results demonstrate feasibility, while the representation and distribution-shift analyses help characterize when adaptation succeeds and where it remains limited.
Critical weaknesses
  1. Limited methodological novelty and insufficient isolation of the proposed modifications. The recipe largely combines established ingredients from residual RL, warm-start learning, and PA-RL. Although such integration can be valuable, the experiments do not sufficiently establish which modifications provide additional insight specifically in the single-demonstration regime. The main design ablations are concentrated on one task, and removing the base-action input nearly matches the default method. More controlled comparisons are needed to distinguish essential principles from interchangeable implementation choices.
  2. The closest-method comparison is incomplete and does not establish a consistent advantage. On LIBERO-Long, MiDAS achieves 91.2±4.2% versus DICE-RL’s 89.3±1.8% over three seeds, without a confidence interval or statistical analysis of the small difference. Appendix E.1 also reports that DICE-RL is more sample-efficient on five tasks, worse on one, and comparable on four. These results support shared design principles but do not establish consistent superiority. DICE-RL results are also missing on RoboCasa.
  3. The hardware evaluation lacks a strong online-adaptation baseline. It only compares with BC.
  4. Hardware task coverage is narrow. Both tasks involve pick-and-place, leaving applicability to precision-critical manipulation or substantial behavioral changes unclear.
  5. The practical benefit of single-demonstration adaptation remains unclear. The paper aims to enable real-world RL from one demonstration, but online practice remains reset-dependent and requires operator involvement. Starting from one demonstration could therefore require more total human effort than collecting five demonstrations and reducing subsequent practice. Without accounting for reset and intervention costs, it is unclear whether minimizing demonstration count provides a meaningful practical advantage over a few-demonstration alternative.
Questions for authors
  1. Contribution clarification: Which modifications distinguish MiDAS from existing residual-RL recipes, and what controlled evidence supports their importance specifically under single-demonstration initialization?
  2. Comparative evidence: Are DICE-RL results available for RoboCasa or hardware? Can the authors characterize uncertainty in the reported LIBERO performance difference?
  3. Practical-cost clarification: How are hardware resets and success labels handled, and how much active operator time is required? Does the reported training duration include reset and recovery overhead?
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors - 1/3

Thank you for your detailed feedback. We would like to first clarify your concern regarding our primary contribution. We then clarify our methodological contributions and provide new comparisons with DICE-RL on RoboCasa. Across four tasks evaluated over three seeds, MiDAS achieves 72.7% average success compared with 53.5% for DICE-RL, with the largest gains on the longer-horizon tasks. We further discuss how success balancing and policy parameterization differences relate to these results. We then address comments regarding the ablation on dropping the base action as input to the residual. Finally, we clarify the scope of our hardware evaluation, describe the associated human involvement and operational overhead, and explain the practical motivation for studying adaptation from minimal demonstrations. We also extend replay and regularization ablations to two additional tasks in the updated PDF and find that MiDAS' design decisions are crucial especially when the base policy is extremely unreliable, and lead to stable online improvement across tasks with varying initial task success.

Contribution of our work

Our primary contribution is to establish and characterize minimal-data adaptation: whether pretrained robot policies can learn reliable task execution from as little as one demonstration followed by online interaction, and what enables this when the initial policy rarely succeeds. To the best of our knowledge, we are the first to demonstrate reliable adaptation of pretrained robot policies from a single task-specific demonstration, followed by online RL. MiDaS provides a concrete adaptation recipe through which we study the roles of pretraining, demonstrations, and online learning, identify effective design choices, and examine the limits of adaptation. That said, MiDaS is not the only recipe that one can build for MDA, and we clearly remark this in the paper (Section 5, Experimental Results Subsection (lines 312-317)).

Regarding comparisons to recent methods: while MiDaS and DICE-RL share ingredients from residual RL, our contribution extends beyond the choice of the adaptation algorithm (PA-RL vs Residual TD3) to understanding what makes this minimal-supervision regime tractable. Our additional RoboCasa experiments also show substantially stronger performance with MiDaS, demonstrating that the choice of adaptation recipe matters on these longer-horizon tasks. We discuss the methodological differences and these results in our response below.

Finally, we clarify that our goal is not to recommend collecting one demonstration plus online experience in place of additional demonstrations. Of course, when many demonstrations are available, one should perhaps run imitation learning as best as possible on that data. Only then should they consider using an adaptation algorithm. Rather, our goal is to study how much improvement is possible when task-specific demonstrations are scarce and the pretrained policy is initially unreliable; i.e., whether learning from minimal demonstrations is even a viable choice or not. Similar to why few-shot learning has been regarded as a promising problem statement in robot learning recently [1,2], we believe MDA is an important problem statement, where we want to improve policy performance with as few task-specific demonstrations as possible. Understanding what enables adaptation under these conditions (either through MiDaS or through other methods) is particularly important for unstructured deployment, where diverse and changing environments make it difficult to train a base policy that is reliable across the full deployment distribution.

AuthorsOpenReview ↗

Rebuttal by Authors - 2/3

Methodological contributions and comparison with DICE-RL

The distinctions from DICE-RL are the most prominent and consequential in how learning proceeds in the low-success regime.

First, MiDaS continually adds successful trajectories from online experience to a success buffer. By repeatedly sampling these trajectories, we help the policy retain and build on each success it discovers. DICE-RL instead samples from a fixed offline demonstration buffer alongside a standard online buffer, reducing the fraction of offline samples as training progresses. When unsuccessful trajectories dominate the online buffer, previously discovered successes become underrepresented in training. Our success buffer addresses this imbalance and helps prevent previously learned successful behavior from being forgotten. For further details, please refer to Appendix G and Figure 11.

Second, DICE-RL regularizes the residual toward the base policy’s actions, with a filter that relaxes this penalty based on the critic’s predictions and observed returns. This regularization can hinder improvement when the base policy itself is brittle, particularly when the critic does not yet reliably recognize better actions and further exploration is needed for improvement. MiDaS does not impose this penalty, allowing the policy to move away from the base-policy actions as it learns.

We additionally evaluated DICE-RL on RoboCasa. MiDaS substantially outperforms it on the evaluated longer-horizon tasks, achieving 72.7% average success compared with 53.50% for DICE-RL over 3 seeds. The tasks have episode horizons ranging from 500 to 1,300 steps and involve different objects and kitchen fixtures. We summarize this variation below:

TaskSteps
Hot dog: counter→cabinet500
Banana: fridge drawer→shelf500
Mug→coffee maker; start800
Select croissant+jam1300

We summarize our results below:

TaskStepsMiDASDiceRL
Hot dog: counter→cabinet150k100.00 ± 0.0093.33 ± 6.11
Banana: fridge drawer→shelf150k93.33 ± 7.0295.33 ± 1.15
Mug→coffeemaker; start200k74.67 ± 20.0319.33 ± 33.49
Select croissant+jam200k22.67 ± 10.076.00 ± 10.39
Average–72.67 ± 6.7953.50 ± 8.89

These results demonstrate stronger performance for the MiDAS recipe on the evaluated longer-horizon tasks. We hypothesize that success balancing and differences in policy regularization contribute to these gains.

Design decision ablations on three tasks

We extend our design decision ablations on two more tasks and add them in Figure 11 in the updated PDF file. The relevant discussion in Appendix G is also updated in light of the new added tasks. We summarize our findings in the table below:

TaskBCMiDASMiDAS (no success buffer)MiDAS (pop base actions)MiDAS (BC reg 0.1)
Both Moka Pots → Stove13.3 ± 3.188.1 ± 4.658.3 ± 45.582.7 ± 7.268.0 ± 5.4
Black Bowl → Bottom Drawer22.7 ± 6.498.7 ± 1.296.0 ± 1.669.3 ± 13.791.3 ± 8.1
Alph. Soup + Tom. Sauce → Basket4.0 ± 3.582.7 ± 13.360.7 ± 43.06.0 ± 8.556.0 ± 24.7

Across all three tasks, the MiDAS configuration with all its design decisions intact, achieves the highest mean success rate, with omission of each decision reducing mean performance. These differences are particularly pronounced on the Alph. Soup + Tom. Sauce → Basket task, where the initial BC policy achieves only 4.0% success. On this task, MiDAS achieves 82.7% success, while the ablated variants range from 6.0% to 60.7%. These results support the importance of the individual design choices within MiDAS and suggest that they are especially valuable for enabling effective adaptation when the initial policy is extremely brittle and unreliable.

AuthorsOpenReview ↗

Rebuttal by Authors - 3/3

Removing the base policy action input ablation

In our default method, we provide the base policy’s action as an input to the residual actor, following common practice in residual learning. With this ablation (see Figure 11 in the updated PDF file and the table in the previous comment (2/3)), we find that on some tasks, removing the base actor input matches the performance of MiDAS. However, in settings where the initial policy has an extremely low success rate (Alph. Soup + Tom. Sauce → Basket with 4%), this collapses during the online learning phase. We explain these results by providing intuition behind the warmup phase of online adaptation.

During the 10K-step warmup, the actor learns to approximate the base policy’s action distribution with a behavior cloning loss, acquiring its behavioral prior before online RL begins. This initialization is important because the actor needs to preserve at least the base policy’s low but non-zero success rate to generate successful experience for learning. Without this initial competence, successful trajectories can become too scarce to sustain improvement; removing warmup causes learning to collapse in our experiments. Success balancing and the online policy updates then allow the initialized actor to preserve and build on the successes it discovers. The base policy action removal ablation shows that even without the base policy action conditioning, the actor initialized through warmup can collect successful trajectories and enable successful online improvement in settings where the base policy has reasonable initial success. However, in extremely low initial success regimes, conditioning becomes crucial to avoid collapse.

Together with the warmup and replay ablations, this result shows the role of base action conditioning in MiDAS. In the regime where the base policy's success is extremely low, conditioning is crucial for online improvement. Conditioning can, however, match the full MiDAS performance when the base policy is reasonably successful, and we explain this via the warmup phase where the behavior prior of the base policy is distilled into the residual actor.

Hardware setup, human effort and task coverage

The hardware experiments were designed as a proof of concept for minimal-data adaptation on a real robot. They show that online learning can substantially improve a policy initialized from a single demonstration, with each RL run requiring up to six hours including operational overhead.

The evaluation covers two pick-and-place tasks and compares against BC. It therefore establishes hardware feasibility within this scope; advantages over alternative online-adaptation methods are supported by the simulation comparisons, including the new RoboCasa results. We will add additional hardware experiments for the camera-ready version.

In the current setup, scene resets are manual and success labels are binary and human-provided. The reported training duration, up to six hours per RL run, denotes total wall-clock time for the experiment, including policy updates, robot execution, resets, and recovery overhead. Future systems could reduce human effort overhead through VLM-based success annotations and by alternating between related or cyclic tasks, where executing one task helps establish the starting conditions for another.

Practical motivation

Our motivation is to study adaptation in deployment settings where obtaining a reliable base policy is difficult, the initial success rate can be extremely low, and collecting additional demonstrations is costly or repeatedly required. For example, a remotely deployed robot such as a rover may encounter unfamiliar conditions that make its existing policy brittle, while obtaining demonstrations for each new situation is difficult. Our minimal-data adaptation setting provides an experimental proxy for these scenarios: the policy has some initial task competence but remains unreliable, and we study which properties and design choices enable it to improve through interaction.

In standard controlled environments, collecting additional demonstrations is often feasible and may reduce the amount of subsequent online interaction. In unstructured deployment, however, robots encounter a long tail of conditions that are difficult to anticipate and cover through demonstrations. The practical trade-off depends on the relative costs and availability of demonstrations, resets, recovery, and success feedback.

If your concerns are addressed, we'd be grateful if you could raise your score.

References

[1] RoboTTT: Context Scaling for Robot Policies URL https://arxiv.org/pdf/2607.15275

[2] INSTANT POLICY: IN-CONTEXT IMITATION LEARNING VIA GRAPH DIFFUSION URL https://arxiv.org/pdf/2411.12633

Reviewer_zE2pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether a pretrained robot policy can learn a task from one demonstration followed by online interaction. The proposed recipe, MIDAS, first uses behavior cloning to obtain a policy that roughly follows the task, then freezes it and learns corrective actions through value-based RL. Pretrained representations and balanced replay of successful experience help make this adaptation effective. Experiments on LIBERO, RoboCasa, and a real robot show substantial improvements over the initial policy. The paper also examines generalization, finding that improved task execution does not consistently transfer to changes in object placement or manipulation strategy.

Critical strengths
  1. The paper provides convincing evidence that one demonstration can be enough to start useful online adaptation, even when the initial policy rarely succeeds. The comparison with the closely related DICE-RL strengthens this finding by showing that it extends beyond the particular MIDAS implementation (Table 1).
  2. The generalization analysis is informative. It distinguishes robustness to visual and language changes from the much weaker transfer to object swaps and larger position changes (Section 7). This helps explain what online adaptation actually improves and where additional experience remains necessary.
Critical weaknesses
  1. Some conclusions are stronger than the supporting analysis. The replay and regularization ablations cover one task, while the analysis of actions outside the base policy’s effective support examines one critical state (Appendices G and K). These are useful observations, but provide limited support for general claims about which ingredients are necessary.
  2. The comparison with Filtered BC does not use a matched interaction budget. It is stopped much earlier based on reported saturation, while MIDAS receives substantially more experience (Appendix J.3; Fig. 9). This weakens the conclusion that supervised learning from successful rollouts is insufficient in this regime.
Questions for authors
  1. How was the single demonstration selected, and do the reported seeds vary the demonstration as well as training randomness?
  2. Does the advantage over Filtered BC remain under a matched interaction budget?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors 1/2

Thank you for your feedback. To address your concerns, we extended Filtered BC training to 3000 trajectories on two tasks to match the interaction budget. Additional interaction does not improve Filtered BC’s mean success rate on either task; value-based RL methods perform substantially better. Critical-state analysis on an additional task shows MiDAS learning a corrective action beyond the base policy’s action prior, while DSRL and Filtered BC stay close to it. We also add additional Robocasa results where MiDAS achieves 72.7% average success compared with 53.5% for DICE-RL, with the largest gains on the longer-horizon tasks. Finally, we clarify demonstration selection details. We also extend replay and regularization ablations to two additional tasks in the updated PDF and find that MiDAS' design decisions are crucial especially when the base policy is extremely unreliable, and lead to stable online improvement across tasks with varying initial task success.

Analysis of actions outside the base policy’s prior

In Figure 13 of the revised PDF, we provide an additional example to broaden the analysis of actions outside the base policy’s effective support. On the LIBERO-Long task “Place Black Bowl to Bottom Drawer,” the base policy, Filtered BC, and DSRL tend not to withdraw the arm at a critical state, preventing the drawer from closing. MiDaS instead produces a withdrawal action that lies far from the sampled base-policy actions and has a substantially higher action value. This provides another example of learning a corrective behavior poorly represented in the base policy’s action samples.

There are two potential limitations of noise steering. First, DSRL learns to select noise initializations for a frozen action expert. If a necessary corrective action cannot be generated by that expert for any allowed noise input, noise steering cannot recover it. There are therefore settings where DSRL cannot solve the task, regardless of the amount of online training. The UMAP plots in Figures 5 and 13 of the revised PDF provide a qualitative illustration of the sampled action distributions. In both examples, DSRL’s sampled actions largely overlap with those of the base policy.

Second, even when the frozen action expert can generate the actions needed to complete the task or recover from errors, the low-success base policy does not select them with sufficient probability to succeed reliably in the MDA setting. DSRL must therefore explore in noise space and learn to assign greater probability to inputs that produce successful behavior. This is a standard off-policy RL problem in the noise space with sparse rewards and a low-success initial policy, retaining the challenges of exploration and value learning from scarce successful experience. Our experiments show that noise steering alone is insufficient for reliable adaptation in the settings studied.

Similarly, Filtered BC can improve performance by emphasizing successful behavior and potentially extend these gains to nearby states and actions. However, trajectory-level filtering discards useful segments of unsuccessful trajectories while reinforcing suboptimal actions within successful ones. It therefore cannot distinguish which actions contributed to success and which were unnecessary or suboptimal. Value-based learning can instead assign credit at the action-chunk level and exploit useful segments across trajectories through stitching.

Related analysis in an offline setting [1] establishes a dependence on task horizon: Filtered BC has O(H)O(H) suboptimality scaling, compared with an achievable O(H)O(\sqrt{H}) scaling for RL under the stated coverage assumptions. A brittle base policy compounds the problem by producing few successful trajectories from which Filtered BC can learn, leaving much of the collected experience unused. These observations motivate using value-based RL to learn corrective behavior from both successful and unsuccessful experience.

Our action-distribution analysis and the observed performance gap between Filtered BC, DSRL and value-based residual RL methods (MiDAS, DICE-RL) are consistent with this analysis.

AuthorsOpenReview ↗

Rebuttal by Authors - 2/2

Filtered BC matched budget

In our experiments, Filtered BC’s success rate plateaued early in training, with no sustained improvement over subsequent iterations. We therefore stopped these runs based on the observed saturation. Similar behavior is reported in ResFiT [2], where online Filtered BC provides limited improvement and quickly saturates. The authors hypothesize that precision errors are difficult to resolve without explicit value maximization. This is consistent with our observation that imitating successful rollouts alone does not reliably recover the fine-grained corrections needed for task completion.

To directly address the interaction-budget mismatch, we extended Filtered BC training on two tasks to match the interaction budget used by MiDaS. We report the final success rates below.

TaskMiDAS (3k trajectories)DICERL (3k trajectories)Filtered BC (paper reported)Filtered BC (3k trajectories)
Black Bowl → Bottom Drawer91.33 ± 5.7786.00 ± 7.2128.70 ± 10.3016.67 ± 6.11
Both Moka Pots → Stove72.67 ± 21.2074.00 ± 3.4619.30 ± 7.008.67 ± 1.15

We see that under the same interaction budget of 3000 trajectories Filtered BC while using significantly more compute performs much worse (>60 pp) than value-based RL methods.

Additional RoboCasa results

TaskSteps
Hot dog: counter→cabinet500
Banana: fridge drawer→shelf500
Mug→coffee maker; start800
Select croissant+jam1300

For training runs, refer to Figure 10 in the revised PDF. Summary of the final performance:

TaskStepsMiDASDiceRL
Hot dog: counter→cabinet150k100.00 ± 0.0093.33 ± 6.11
Banana: fridge drawer→shelf150k93.33 ± 7.0295.33 ± 1.15
Mug→coffeemaker; start200k74.67 ± 20.0319.33 ± 33.49
Select croissant+jam200k22.67 ± 10.076.00 ± 10.39
Average–72.67 ± 6.7953.50 ± 8.89

BC demonstration selection

We initially sample a demonstration at random from the available pool and use it for BC fine-tuning. If the resulting policy has zero observed success, we repeat this process with another randomly sampled demonstration until the policy achieves non-zero success. This criterion provides an initial policy that can occasionally complete the task, allowing online learning to bootstrap from some successful experience.

Design decision ablations on three tasks

We extend our design decision ablations on two more tasks and add them in Figure 11 in the updated PDF file. The relevant discussion in Appendix G is also updated in light of the new added tasks. We summarize our findings in the table below:

TaskBCMiDASMiDAS (no success buffer)MiDAS (pop base actions)MiDAS (BC reg 0.1)
Both Moka Pots → Stove13.3 ± 3.188.1 ± 4.658.3 ± 45.582.7 ± 7.268.0 ± 5.4
Black Bowl → Bottom Drawer22.7 ± 6.498.7 ± 1.296.0 ± 1.669.3 ± 13.791.3 ± 8.1
Alph. Soup + Tom. Sauce → Basket4.0 ± 3.582.7 ± 13.360.7 ± 43.06.0 ± 8.556.0 ± 24.7

Across all three tasks, the MiDAS configuration with all its design decisions intact, achieves the highest mean success rate, with omission of each decision reducing mean performance. These differences are particularly pronounced on the Alph. Soup + Tom. Sauce → Basket task, where the initial BC policy achieves only 4.0% success. On this task, MiDAS achieves 82.7% success, while the ablated variants range from 6.0% to 60.7%. These results support the importance of the individual design choices within MiDAS and suggest that they are especially valuable for enabling effective adaptation when the initial policy is extremely brittle and unreliable.

If your concerns are addressed, we'd be grateful if you could raise your score.

References

[1] WHEN SHOULD WE PREFER OFFLINE REINFORCEMENT LEARNING OVER BEHAVIORAL CLONING? https://arxiv.org/pdf/2204.05618

[2] Residual Off-Policy RL for Finetuning Behavior Cloning Policies https://arxiv.org/pdf/2509.19301

AuthorsOpenReview ↗

Gentle Reminder: Follow-up on Our Rebuttal and Updated Results

Dear Reviewer zE2p,

Thank you again for taking the time to review our paper and for your constructive feedback. With two days remaining in the discussion period, we would be grateful if you could review our rebuttal and the additional results in the updated manuscript. Please let us know if any concerns remain or if further clarification would be helpful. We would be happy to address them before the discussion closes.

Reviewer_5nioOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies minimal-data adaptation of pretrained generalist robot policies, where a policy learns a new task from as little as one demonstration followed by autonomous online interaction. The authors propose MiDAS, a two-stage approach that first fine-tunes a pretrained VLA using BC and then improves it through value-based residual reinforcement learning while keeping the base policy frozen. The method incorporates critic initialization and success-balanced replay to address sparse rewards and unreliable initial policies. Experiments on LIBERO-Long, RoboCasa, and a real bimanual robot demonstrate substantial improvements over one-demonstration behavior cloning.

Critical strengths
  • The paper addresses a meaningful challenge in robot learning: adapting generalist policies to new tasks with minimal task-specific supervision. MiDAS substantially improves task success even when the initial behavior-cloned policy has very low success rates. The evaluation across two simulation benchmarks and a real robot demonstrates the practical potential of combining pretrained VLA policies with online RL.
  • Beyond demonstrating the effectiveness of MiDAS, the authors systematically investigate the factors enabling successful adaptation, including pretrained VLA representations, task-specific demonstrations, value-based residual learning, and replay balancing. The comparisons with alternative representations and adaptation strategies, together with the analysis of robustness under distribution shifts, provide useful insights into the capabilities and limitations of adapting generalist robot policies from minimal data.
Critical weaknesses
  • Limited novelty relative to existing work. The authors refer to DICE-RL as concurrent work, but it was published at ICML 2026 and therefore represents relevant prior work for this submission. Moreover, DICE-RL achieves comparable final performance and better sample efficiency on several tasks. While I appreciate the additional empirical analysis in MiDAS, I would like the authors to clarify the methodological novelty of MiDAS relative to DICE-RL and more explicitly distinguish the contributions of the two papers.
  • Real-world evaluation is also limited. The evaluation is limited to two relatively simple pick-and-place tasks on a single bimanual robot platform, with only 15 evaluation rollouts per task, making statistical conclusions relatively limited.
Questions for authors
  • Given that DICE-RL achieves comparable final performance and better sample efficiency on several tasks, could the authors clarify the main methodological contributions of MiDAS beyond DICE-RL? In particular, which design choices distinguish MiDAS and provide advantages over DICE-RL?
  • Could the authors clarify whether MiDAS offers advantages over existing methods such as DICE-RL in real-world settings?
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors - 1/2

Thank you for your feedback. To address your concerns, we clarify our central contribution of establishing and characterizing minimal-data adaptation, explain the design choices that distinguish MiDAS from DICE-RL, and provide new comparative experiments on RoboCasa. Across four tasks evaluated over three seeds, MiDAS achieves 72.7% average success compared with 53.5% for DICE-RL, with the largest gains on the longer-horizon tasks. We then clarify the scope of real-world evaluations.

Methodological contributions and comparison with DICE-RL

Our work studies whether pretrained robot policies can acquire reliable task execution from a single demonstration followed by online interaction, and which properties make this possible. The contribution is establishing and characterizing the minimal-data adaptation regime: understanding what pretraining provides, what the demonstration must specify, and what online learning can improve. MiDaS builds on prior work to study these questions and identify design choices that support effective adaptation from a low-success initial policy. Our additional RoboCasa comparisons provide further evidence of the effectiveness of these choices on longer-horizon tasks.

We have conducted additional comparisons on RoboCasa, where MiDaS substantially outperforms DICE-RL on the evaluated longer-horizon tasks (see results below). The tasks have episode horizons ranging from 500 to 1,300 steps and involve different objects and kitchen fixtures. We summarize this variation below:

TaskSteps
Hot dog: counter→cabinet500
Banana: fridge drawer→shelf500
Mug→coffee maker; start800
Select croissant+jam1300

RoboCasa Results

For training runs, refer to Figure 10 in the revised PDF. Summary of the final performance:

TaskStepsMiDASDiceRL
Hot dog: counter→cabinet150k100.00 ± 0.0093.33 ± 6.11
Banana: fridge drawer→shelf150k93.33 ± 7.0295.33 ± 1.15
Mug→coffeemaker; start200k74.67 ± 20.0319.33 ± 33.49
Select croissant+jam200k22.67 ± 10.076.00 ± 10.39
Average–72.67 ± 6.7953.50 ± 8.89

The approaches differ in two design choices that are particularly relevant in the low-success regime.

First, MiDaS continually adds successful trajectories from online experience to a success buffer. By repeatedly sampling these trajectories, we help the policy retain and build on each success it discovers. DICE-RL instead samples from a fixed offline demonstration buffer alongside a standard online buffer, reducing the fraction of offline samples as training progresses. When unsuccessful trajectories dominate the online buffer, previously discovered successes become underrepresented in training. Our success buffer addresses this imbalance and helps prevent previously learned successful behavior from being forgotten. For further details, please refer to Appendix G and Figure 11.

Second, DICE-RL regularizes the residual toward the base policy’s actions, with a filter that relaxes this penalty based on the critic’s predictions and observed returns. This regularization can hinder improvement when the base policy itself is brittle, particularly when the critic does not yet reliably recognize better actions and further exploration is needed for improvement. MiDaS does not impose this penalty, allowing the policy to move away from the base-policy actions as it learns.

AuthorsOpenReview ↗

Rebuttal by Authors - 2/2

Scope of real-world evaluation

Our real-world experiments were intended to test whether minimal-data adaptation is tractable on physical hardware. We evaluate each policy over 15 rollouts per task, a count comparable in scale to established real-world evaluation protocols: OpenVLA [1] reports 5–10 trials per task in its generalist evaluations, while IBRL [2] and Diffusion Policy [3] report 20 evaluations in their hardware experiments. Across both tasks, online RL substantially improves success over initialization from a single demonstration, supporting the feasibility of minimal-data adaptation in the real world.

Broader task coverage would further strengthen our findings. However, each real-world RL run takes up to six hours, which limited the scope of our hardware evaluation. The rebuttal timeline and hardware constraints make additional hardware experiments infeasible during this period. We will add additional hardware experiments for the camera-ready version. Although we have not compared online adaptation methods on hardware, our additional RoboCasa experiments show substantially stronger performance than DICE-RL on longer-horizon tasks.

If your concerns are addressed, we'd be grateful if you could raise your score.

References

[1] OpenVLA: An Open-Source Vision-Language-Action Model https://arxiv.org/pdf/2406.09246

[2] Imitation Bootstrapped Reinforcement Learning https://arxiv.org/pdf/2311.02198

[3] Diffusion Policy: Visuomotor Policy Learning via Action Diffusion https://arxiv.org/pdf/2303.04137

AuthorsOpenReview ↗

Gentle Reminder: Follow-up on Our Rebuttal and Updated Results

Dear Reviewer 5nio,

Thank you again for taking the time to review our paper and for your constructive feedback. With two days remaining in the discussion period, we would be grateful if you could review our rebuttal and the additional results in the updated manuscript. Please let us know if any concerns remain or if further clarification would be helpful. We would be happy to address them before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.