- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VLRLA, a framework for deployment-time adaptation of pretrained vision-language-action models through interaction history, with all model parameters fixed during deployment. It trains recurrent modules on ordered histories constructed from simulation RL, progressively increased human intervention, or hindsight relabeling. VLRLA-simp predicts actions from frozen visual-language representations, while VLRLA-bridge modifies the KV cache supplied to a frozen action expert. The paper evaluates the framework on RoboTwin and real-world manipulation, reporting improvements across successive episodes, transfer to an unseen pepper-and-pot setting, and reward-conditioned target selection in a two-plate task.
Critical strengths
-
Practical integration with pretrained action generation. VLRLA-bridge retains both the visual-language backbone and the action expert, using a recurrent module to produce residual KV modifications. This provides a concrete way to incorporate interaction feedback without retraining the full policy or updating parameters during deployment.
-
Real-world evidence of improvement across attempts. In the unseen pepper-and-pot setting, success increases from 36% in the first episode to 68% in the fourth, across 25 evaluation runs. The seen fruit tasks also improve across episodes. These results provide encouraging evidence that the trained policy can benefit from accumulated deployment history.
Critical weaknesses
-
Insufficient evidence of reward-driven adaptation. Training histories are ordered by increasing human intervention, so improvement across episodes may reflect the learned progression toward expert behavior rather than adaptation to feedback. A reward ablation on VLRLA-bridge is needed to distinguish these explanations.
-
Limited evidence of generalizable adaptation rules. The two-plate experiment reproduces the feedback rule explicitly constructed in its training histories: retain a rewarded choice and switch after an unrewarded one. It demonstrates execution of this learned rule, but provides limited evidence that the adaptation mechanism generalizes beyond the training setup.
-
Limited attribution of unseen-task performance. The supplementary video suggests that fine-tuned π0.5 already follows the instruction and attempts to grasp the unseen pepper. The ability to handle the new object therefore appears to originate from the base model, while VLRLA mainly improves grasping accuracy. The experiment does not clearly separate this inherited generalization from the contribution of deployment-time adaptation.
-
Dependence on human feedback and repeated attempts. Evaluation relies on human-provided episode-level labels, and improved performance requires multiple attempts on the same task. For example, the reported 68% success on the pepper task is reached in the fourth episode, after three preceding attempts. These feedback and interaction requirements should be accounted for when assessing the claimed deployment-time adaptation efficiency.
Questions for authors
See Weaknesses.
Thank you for highlighting the practical value of retaining the pretrained action expert, and for noting the improvements across attempts in the real-world experiments.
Q1. Insufficient evidence of reward-driven adaptation
A1. In this work, we mainly demonstrate reward-driven adaptation through downstream experimental results. For example, in Table 1, RoboTTT doesn't have access to reward and termination signals in the training dataset and adapts less effectively than VLRLA. Also, in the two-plate experiment, we deliberately designed a situation where the reward signals in past experience are critical for adapting current behavior.
Q2. Limited evidence of generalizable adaptation rules ... in two-plate experiment
A2. The two-plate experiment is deliberately designed to demonstrate the critical role of histories that include rewards in behavior adaptation: if we only collect histories without reward, it is hard for VLA policies to adapt their behavior, as the correct plate is specified only through the reward signal. Hence this experiment is not intended to demonstrate generalization.
We demonstrate generalization through simulation experiments (Table 1) and real-world experiments (fruit placement, where we demonstrate gradual adaptation to unseen fruit-vessel combinations).
Q3. Limited attribution of unseen-task performance
A3. We expect the base model to supply the semantic competence needed to recognize the pepper and follow the instruction. The question for VLRLA is whether experience from previous competent attempts could help the robot gradually adapt its behavior and improve task performance.
This is similar to how reinforcement learning fine-tuning works: if the base model cannot perform a task at all, using RL to fine-tune it would be inefficient. The main goal of this work is to pursue gradual, update-free adaptation at deployment time, reflected in increasing task success rates.
Over 25 runs, the base model achieves 24%, and the bridge achieves 36%, 36%, 48%, and 68% over four episodes. Pepper and the metal pot were excluded from the real-world history-training data. This is the type of metric and result commonly used in online RL fine-tuning work such as RLT[1].
[1] RL Token: Bootstrapping Online RL with Vision-Language-Action Models : https://arxiv.org/abs/2604.23073
Q4. Dependence on human feedback and repeated attempts.
A4. Thanks for pointing this out. We will revise the manuscript to mention this. However, we think that requiring four trajectories is not a significant drawback for an adaptation method, as online RL fine-tuning usually needs dozens of trajectories.
Regarding the dependence on human feedback, we think this is orthogonal to our work. In future work, we may adopt automatic reward models like RoboReward[2] for this labeling purpose.
[2] RoboReward: General-Purpose Vision-Language Reward Models for Robotics : https://arxiv.org/abs/2601.00675