acceptodds
Withdrawn from ICLR 2027

VLRLA: Deployment-Time Adaptation of Vision-Language-Action Models through In-Context Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) models connect visual perception and language understanding with robot control, providing a foundation for instruction-conditioned manipulation. However, conventional pretrained VLA policies lack a mechanism for gradual adaptation through accumulated interaction experience during deployment. We therefore propose VLRLA, a general framework that equips existing VLA models with capabilities for deployment-time adaptation. Our approach augments a VLA model with a reinforcement learning module that learns from historical observations, actions, rewards, and termination signals. During deployment, the policy gradually refines its action generation and progressively improves task success while keeping model parameters fixed. Our approach is also designed for general applicability and compatibility with diverse learning history construction strategies and model training pipelines. Finally, through extensive experiments, we systematically investigate the learning properties of VLRLA in simulation and demonstrate its ability to adapt to unseen scenarios during real-world deployment.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Model, In-context Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FPHROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VLRLA, a framework for deployment-time adaptation of pretrained vision-language-action models through interaction history, with all model parameters fixed during deployment. It trains recurrent modules on ordered histories constructed from simulation RL, progressively increased human intervention, or hindsight relabeling. VLRLA-simp predicts actions from frozen visual-language representations, while VLRLA-bridge modifies the KV cache supplied to a frozen action expert. The paper evaluates the framework on RoboTwin and real-world manipulation, reporting improvements across successive episodes, transfer to an unseen pepper-and-pot setting, and reward-conditioned target selection in a two-plate task.

Critical strengths
  1. Practical integration with pretrained action generation. VLRLA-bridge retains both the visual-language backbone and the action expert, using a recurrent module to produce residual KV modifications. This provides a concrete way to incorporate interaction feedback without retraining the full policy or updating parameters during deployment.

  2. Real-world evidence of improvement across attempts. In the unseen pepper-and-pot setting, success increases from 36% in the first episode to 68% in the fourth, across 25 evaluation runs. The seen fruit tasks also improve across episodes. These results provide encouraging evidence that the trained policy can benefit from accumulated deployment history.

Critical weaknesses
  1. Insufficient evidence of reward-driven adaptation. Training histories are ordered by increasing human intervention, so improvement across episodes may reflect the learned progression toward expert behavior rather than adaptation to feedback. A reward ablation on VLRLA-bridge is needed to distinguish these explanations.

  2. Limited evidence of generalizable adaptation rules. The two-plate experiment reproduces the feedback rule explicitly constructed in its training histories: retain a rewarded choice and switch after an unrewarded one. It demonstrates execution of this learned rule, but provides limited evidence that the adaptation mechanism generalizes beyond the training setup.

  3. Limited attribution of unseen-task performance. The supplementary video suggests that fine-tuned π0.5 already follows the instruction and attempts to grasp the unseen pepper. The ability to handle the new object therefore appears to originate from the base model, while VLRLA mainly improves grasping accuracy. The experiment does not clearly separate this inherited generalization from the contribution of deployment-time adaptation.

  4. Dependence on human feedback and repeated attempts. Evaluation relies on human-provided episode-level labels, and improved performance requires multiple attempts on the same task. For example, the reported 68% success on the pepper task is reached in the fourth episode, after three preceding attempts. These feedback and interaction requirements should be accounted for when assessing the claimed deployment-time adaptation efficiency.

Questions for authors

See Weaknesses.

1 comment

Thank you for highlighting the practical value of retaining the pretrained action expert, and for noting the improvements across attempts in the real-world experiments.

  ~~

Q1. Insufficient evidence of reward-driven adaptation

A1. In this work, we mainly demonstrate reward-driven adaptation through downstream experimental results. For example, in Table 1, RoboTTT doesn't have access to reward and termination signals in the training dataset and adapts less effectively than VLRLA. Also, in the two-plate experiment, we deliberately designed a situation where the reward signals in past experience are critical for adapting current behavior.

  ~~

Q2. Limited evidence of generalizable adaptation rules ... in two-plate experiment

A2. The two-plate experiment is deliberately designed to demonstrate the critical role of histories that include rewards in behavior adaptation: if we only collect histories without reward, it is hard for VLA policies to adapt their behavior, as the correct plate is specified only through the reward signal. Hence this experiment is not intended to demonstrate generalization.

We demonstrate generalization through simulation experiments (Table 1) and real-world experiments (fruit placement, where we demonstrate gradual adaptation to unseen fruit-vessel combinations).

  ~~

Q3. Limited attribution of unseen-task performance

A3. We expect the base model to supply the semantic competence needed to recognize the pepper and follow the instruction. The question for VLRLA is whether experience from previous competent attempts could help the robot gradually adapt its behavior and improve task performance.

This is similar to how reinforcement learning fine-tuning works: if the base model cannot perform a task at all, using RL to fine-tune it would be inefficient. The main goal of this work is to pursue gradual, update-free adaptation at deployment time, reflected in increasing task success rates.

Over 25 runs, the base model achieves 24%, and the bridge achieves 36%, 36%, 48%, and 68% over four episodes. Pepper and the metal pot were excluded from the real-world history-training data. This is the type of metric and result commonly used in online RL fine-tuning work such as RLT[1].

[1] RL Token: Bootstrapping Online RL with Vision-Language-Action Models : https://arxiv.org/abs/2604.23073

  ~~

Q4. Dependence on human feedback and repeated attempts.

A4. Thanks for pointing this out. We will revise the manuscript to mention this. However, we think that requiring four trajectories is not a significant drawback for an adaptation method, as online RL fine-tuning usually needs dozens of trajectories.

Regarding the dependence on human feedback, we think this is orthogonal to our work. In future work, we may adopt automatic reward models like RoboReward[2] for this labeling purpose.

[2] RoboReward: General-Purpose Vision-Language Reward Models for Robotics : https://arxiv.org/abs/2601.00675

Reviewer_Hv9zOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes VLRLA, a framework for deployment-time adaptation of Vision-Language-Action models through in-context reinforcement learning. The method introduces an adaptation module that generates action with memory conditioned on historical observations, actions, rewards and termination sign. The authors evaluate the method across simulation and real-world manipulation tasks. While the overall idea is interesting, I have significant concerns regarding the results, the motivation for the proposed memory mechanism and the novelty of this method.

Critical strengths

The paper addresses the important problem of adapting pretrained VLA policies during deployment without parameter updates.

The framework is compatible with different pretrained backbones and training-history construction strategies.

Critical weaknesses

1. Unclear contribution of historical failures.

The paper assumes that accumulating past experiences, including failures, improves future actions but does not explain how the model extracts useful corrective information from unsuccessful attempts. In the original Algorithm Distillation (AD), which does not leverage large-scale pretrained world knowledge, failure histories can help the model discover basic task structure, such as avoiding previously explored incorrect paths. In contrast, pretrained VLA models already possess strong semantic priors. In such cases, it is unclear how simply accumulating past failures enables the model to identify and correct errors. The paper should provide a more convincing explanation or analysis of why failure memory is beneficial in the pretrained VLA setting.

2. Limited method novelty and empirical validation

The core idea of VLRLA is essentially identical to Algorithm Distillation. The proposed extensions, including adapted KV-cache and alternative history construction strategies, appear to be incremental adaptations of existing techniques rather than fundamentally new methodological contributions. Also, on unseen tasks, VLRLA achieves 60.6% success, slightly below GRPO. This does not demonstrate a compelling advantage for in-context adaptation. Moreover, evaluation is limited to 12 selected RoboTwin tasks. The task-selection criteria and broader generalizability remain unclear.

Questions for authors
  1. How were the 12 evaluation tasks selected? Do the results generalize to a full task set?
  2. Can the authors distinguish VLRLA from standard Algorithm Distillation?
5 comments

Hi,

I have a question for the reviewer (or anyone familiar with the topic as well), as I'm interested in the broader research direction.

My understanding is that a central motivation for in-context RL is to replace task-specific RL post-training with a general meta-training phase, enabling few-shot adaptation to unseen tasks without additional gradient updates. If that's the intended setting, how should the paper have framed the benefit?

Hi,

I think your understanding is correct. My main concern is that VLRLA should demonstrate a clear advantage over existing methods to unseen tasks, as this is its primary motivation. However, GRPO slightly outperforms VLRLA on unseen tasks. This is somewhat unexpected given the paper's emphasis is on generalization to unseen tasks. Since the paper does not clearly specify whether GRPO was fine-tuned on the unseen tasks, I thought GRPO was fine-tuned on only 6 seen tasks and evaluate on unseen tasks. If GRPO's performance is based on fine-tuning with 12 tasks(6 seen tasks + 6 unseen tasks), this can be an advantage.

Thanks for the clarification.

Reviewer_2zsEOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose running in-context RL on VLAs, creating a system called VLRLA. They train VLRLA with a mixture of data including human-in-the-loop, RL rollouts, and hindsight experience and employ algorithm distillation. They test VLRLA on a number of capabilities (like cross episode memory, history transfer, and task adaptation) and compare against other baselines.

Critical strengths
  • There seems to be thorough experimental breadth, which I appreciate.
  • The writing is OK, which made reading the paper and understanding its contents more straightforward.
  • I think algorithm distillation is an exciting direction to take VLAs towards to have more adaptive behaviors!
Critical weaknesses
  • This paper is lacking ablation studies as well - there seem to be a lot of unexplained design decisions in the architecture. And also the data collection. Maybe ablations could help us understand why these choices were made and provide some insight.
  • Some of the capabilities in 5.3 and 5.4 could be a bit more sound to truly communicate the capabilities being claimed. I detail this in the Questions section.
  • The results aren’t particularly strong. Table 1 shows this and is also a bit hard to parse. For example, handover_mic seems to be the only reason that VLRLA_bridge outperforms GRPO on the average seen tasks, as there’s a 40% delta. And on the unseen tasks, GRPO is better. Overall, a more robust metric (the number of wins) shows GRPO as a stronger method. It wins 8 times, compared to VLRLA's 7 wins. The rest of the figures show comparisons with VLRLA itself, which is good, but doesn’t convince me as a reviewer that this design and method are performant. Table 1 also seems cut off.
Questions for authors
  • Regarding section 5.3, was the model trained on examples with no history? And trained with repeated history? Or is there at least some indicator of the information being stale/zeroed out? I ask because it might be trivial as to why the model doesn’t work with fresh information, as it was only ever trained on a distribution of history that’s induced by your dataset. TO better understand how it’d work without history, you would want to compare it with something trained on the same exact data but made memoryless (perhaps via controlling the recurrent rule, etc.) This also extends to 5.4 to ensure that the reason it works well is the information itself, and not just the fact that you’re feeding in some in-distribution conditioning. Or the fact that it has overfit to having good performance later from mimicry of AD.
2 comments

Thank you for recognizing the breadth of our experiments and for finding the paper easy to follow. We also appreciate your encouragement of AD as a direction for more adaptive VLAs.

  ~~

Q1. What motivates the architectural and data-collection choices?

A1. For architecture, we choose 2 typical variants: a simple architecture that outputs action tokens directly and a Pi0.5-style architecture that includes an additional action expert. We think these represent 2 VLA model variants for generating actions (other example is OpenVLA's approach to generating stochastic discrete actions, but we consider it neither as simple as VLRLA-simp nor as powerful as VLRLA-bridge).

For data-collection choice, the original AD data-collection method does not readily fit the VLA setting: 1. collecting large-scale datasets in simulation introduces a Sim2Real gap; 2. collecting data directly in the real world requires substantial human labor. It is also unclear what constitutes a dataset of gradually improving behavior and how to obtain it efficiently for real-world VLA tasks.

In this work, we tried both settings and achieved effective results. Notably, for real-world cases, we design and test 2 ways to obtain training data showing gradual improvement: 1. HITL collection, where the base model starts execution and the human intervention ratio gradually increases. This clearly defines a dataset of gradually improving behavior in real-world settings and could reduce human operators' labor. 2. Hindsight reward relabeling, where we can assign reward labels after collecting the dataset to avoid collecting it again.

As discussed in General Response 1, this work focuses on integrating the principle of AD into a downstream application domain (VLA).

  ~~

Q2. The results aren’t particularly strong. Most of the advantage over GRPO comes from handover_mic?

A2. Your observation is correct: removing handover_mic removes the mean advantage over GRPO. Including ties, GRPO has eight row-best entries and VLRLA-bridge seven.

However, we would like to highlight the significant differences between VLRLA-bridge and GRPO, as described in General Response 2. In this work, we view the GRPO results as an upper reference rather than a directly comparable baseline.

  ~~

Q3. Were no-history and repeat-history used during training? Could the observed drop reflect distribution shift?

A3. They were used only at test time. We did not deliberately train the model with these history interventions. So for no history, the distribution shift is a plausible explanation for part of the drop.

For repeat-history, we think this is still an illustrative experiment in which we feed the same episode (the first finished one) repeatedly. Its performance does not improve as it does with in-context learning.

Also, we use RoboTwin's random setting which already uses disjoint seeds for training (including dataset collection) and evaluation. In this sense, the concrete evaluation environment instances (object placement, background, lighting, table height, etc.) are not seen by VLRLA during training, even for seen tasks.

Also, could the reviewer elaborate on the suggestion to compare it with something trained on the same exact data but made memoryless? We think this seems to refer to the baseline π0.5\pi_{0.5} which is known to lack adaptation capability.

  ~~

Q4. Table 1 is hard to read

A4. Thank you for pointing this out. We will revise the table to make the comparisons between methods clearer. For more clarification, please refer to General Responses 2 and 3.

Thanks for the response.

After your response, I still do not feel motivated to move this paper into acceptance range. And I do not feel like my concerns regarding system/data design were properly addressed.

If GRPO is not to be compared to since it requires compute, then this story of VLRLA should be one of memory efficiency - but the paper isn't really positioned this way. Also, generally, "real-world" settings constrain the deployment efficiency, not the training efficiency. But sure, I can understand how you'd like to consider your method separately as an in-context approach.

However, the results and limited novelty beyond AD also keep me from moving to the accept range.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General Response

We thank all reviewers for their careful reading and valuable comments. We first provide several General Responses, then address the individual comments.

  ~~

1. What VLRLA adds to Algorithm Distillation

We follow the principle of Algorithm Distillation (AD) and study how to effectively integrate it into the downstream VLA domain. The main contribution of our work is to provide multiple effective ways to integrate AD into VLA.

For example, the original AD data-collection method does not readily fit the VLA setting: 1. collecting large-scale datasets in simulation has Sim2Real gap; 2. collecting data directly in the real world requires substantial human labor. It is also unclear what constitutes a dataset of gradually improving behavior and how to obtain it efficiently for real-world VLA tasks.

In this work, we tried both settings and achieved effective results. Notably, for real-world cases, we design and test 2 ways to obtain training data showing gradual improvement: 1. HITL collection, where the base model starts execution and the human intervention ratio gradually increases. This clearly defines a dataset of gradually improving behavior in real-world settings and could reduce human operators' labor. 2. Hindsight reward relabeling, where we can assign reward labels after collecting the dataset to avoid collecting it again.

Also, in VLRLA-simp and VLRLA-bridge, we tried 2 types of VLA backbones: 1. a basic version where the transformer outputs action tokens directly and 2. a variant like Pi05 that includes an additional action expert and outputs continuous actions. These are 2 popular ways to generate actions from vision-language embeddings.

In summary, this work aims to integrate established principles into a downstream application domain, obtaining novel results (gradually improving success rates at deployment time without parameter updates) across multiple scenarios.

  ~~

2. How to read the GRPO comparison

GRPO was fine-tuned separately on all 12 evaluation tasks (we run RLinf's GRPO fine-tuning for each individual task. It uses 256 simulated environments, 8 Nvidia A100-80GB cards, and 25 epochs with 25600 total episodes, making it difficult to apply in real-world settings). In contrast, VLRLA-bridge is trained on histories collected from six seen tasks and evaluated on the unseen tasks (single environment instance, 25 sequential episodes and no parameter updates). Our RoboTTT implementation follows the same protocol as VLRLA, with reward and termination signals removed from the dataset. We understand that the current table may be hard to read and will revise it in the manuscript.

Evaluation groupPi05RoboTTTVLRLA-bridgeTask-specific GRPO
Six seen tasks48.657.866.259.2
Six unseen tasks53.358.660.660.8
All 12 tasks50.958.263.460.0

We regard GRPO as a task-specific upper reference in this setting. The result we find most useful is the unseen comparison: VLRLA-bridge reaches 60.6%, close to the 60.8% of GRPO policies, which require 256 simulated environments, 8 Nvidia A100-80GB cards, and 25 epochs with 25600 total episodes, making this training difficult to conduct in real-world settings. It shows the potential of deployment-time update-free adaptation mechanisms.

  ~~

3. Why those tasks in Table 1

Typical RLinf fine-tuning requires substantial computational and memory resources: fine-tuning the π0.5\pi_{0.5} model in simulation requires 256 simulated environments, 8 Nvidia A100-80GB cards, 25 epochs with 25600 total episodes and 13-15 hours of wall-clock time. Also, the π0.5\pi_{0.5} base model (SFT-trained using RoboTwin's published dataset) already achieves different initial success rates across tasks. If its initial success rate is high enough, there is little room for later method comparison. If its initial success rate is too low, RLinf fine-tuning may not yield meaningful improvements. Hence, we choose 10 tasks where the π0.5\pi_{0.5} base model achieves a moderate initial success rate and perform RLinf fine-tuning on them. We find that success rates improve on only 6 tasks using RLinf's default configurations. In summary, we use these 6 seen tasks because 1. computational resources are limited, and 2. training datasets showing gradual improvement are not readily available, so we need to filter RLinf's results.

For the 6 unseen tasks, we reuse the 4 tasks above where RLinf does not yield significant success-rate improvements, plus 2 tasks, place_empty_cup and adjust_bottle, which are extensively used in other experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.