acceptodds
Under review as a conference paper at ICLR 2027

Surpassing VLA Performance: Online RL Training of Lightweight Policies with Minimal Human Interventions

Abstract

Pre-trained vision-language-action (VLA) models often exhibit low success rates and task completion speeds when deployed without fine-tuning in new environments. The standard approach to improve the performance of a pre-trained VLA is to collect new demonstrations and fine-tune the VLA for every new task encountered during deployment. However, collecting human demonstrations during deployment can be costly and time-consuming, which is a barrier to the widespread adoption of VLAs in industry and daily life. In this work, we propose a framework based on reinforcement learning for surpassing the performance of pre-trained VLAs by learning a lightweight policy (LWP) with no human demonstrations and minimal human intervention. By reinforcement learning only from online experience, our proposed framework RELAY transfers task-specific robot behavior from the VLA to the LWP and improves the behavior over time. Our experiments on three real-robot manipulation tasks show that RELAY-based agents autonomously improve their performance and achieve higher success rates and task completion speeds than the pre-trained VLA within six hours of interaction time.

Primary area:
reinforcement learning
Keywords:
Robot Learning, Online Reinforcement Learning, Vision-Language-Action Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_epPGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents RELAY, a framework that improves the performance of pretrained Vision-Language-Action (VLA) models through online reinforcement learning. The method freezes the pretrained VLA and trains a lightweight policy (LWP) using its visual encoder. A two-stage online RL process gradually transfers control from the VLA to the LWP. Experiments on three real-world robotic manipulation tasks demonstrate that RELAY improves task success rates and execution efficiency without requiring additional human demonstrations.

Critical strengths

Practical motivation and real-world validation:

This paper addresses the practical problem of improving pretrained VLA performance without collecting additional human demonstrations. The authors conduct real-world online RL experiments and demonstrate substantial improvements in task success rates and execution efficiency.

Efficient and practical framework:

RELAY reduces inference and training costs by freezing the pretrained VLA and training an independent lightweight policy. The two-stage learning process, combined with automated environment resets, enables real-world robot learning with minimal human intervention.

Critical weaknesses

Limited novelty: The proposed framework mainly combines existing components, including a pretrained VLA, a lightweight policy network, and Cal-QL, with a staged online learning procedure. The approach appears more like an engineering integration of established techniques than a fundamentally new algorithm or learning mechanism. Its unique contribution compared to existing lightweight policy adaptation and online RL methods remains unclear.

Insufficient baseline comparisons and ablation studies: This is my primary concern. The paper mainly compares the trained LWP against the original pretrained VLA without downstream adaptation. However, showing that a task-specific policy trained with additional online RL outperforms an unadapted VLA is insufficient to establish the effectiveness of the proposed RELAY framework.

The paper lacks comparisons with standard online RL methods, behavior cloning using VLA-generated trajectories, and existing lightweight policy adaptation approaches under matched interaction budgets. Furthermore, key components, including the two-stage training procedure, replay sampling ratios, and the choice of Cal-QL, are not systematically ablated. It is therefore difficult to determine which components are responsible for the reported improvements.

Limited task diversity and generalization:

The experiments involve only three relatively simple real-world manipulation tasks, two of which involve the same object. Moreover, the method assumes that the pretrained VLA already achieves a non-zero success rate and trains a separate lightweight policy for each task. Consequently, the experiments mainly demonstrate task-specific performance improvements, rather than generalization to more complex or unseen tasks and environments.

Evaluation protocol and presentation: The final LWP is evaluated over 30 episodes per task, while the VLA baseline performance is estimated from episodes collected during online training. Although the VLA remains frozen, it is unclear whether both policies are evaluated under identical initial-state distributions and evaluation conditions. A matched evaluation protocol would make the comparison more rigorous. Regarding presentation, some framework diagrams and experimental figures contain small text, and the figure layouts and excessive whitespace on certain pages reduce overall readability.

Questions for authors

Could the authors compare RELAY against standard Cal-QL, SAC, or behavior cloning using VLA-generated trajectories under the same online interaction budget? It remains unclear whether the improvements primarily come from RELAY's specific design or simply from additional task-specific training.

How much do the two-stage training procedure and replay sampling strategy individually contribute to the final performance? Could the authors provide corresponding ablation studies?

The framework relies on task-specific success detectors and automated reset procedures. Can it still operate with minimal human intervention on more complex tasks or in environments where automated resets are difficult?

2 comments

We thank the reviewer for their constructive feedback.

The approach appears more like an engineering integration of established techniques than a fundamentally new algorithm or learning mechanism. Its unique contribution compared to existing lightweight policy adaptation and online RL methods remains unclear.

Combining existing techniques and demonstrating its effectiveness on real-robot tasks can be a valuable contribution on par with a fundamentally new algorithm or learning mechanism. Furthermore, much of the value in our contribution comes from constraints such as not collecting human demonstrations and not relying on a VLA forward pass for generating actions, which are often violated in related work, and proposing a framework and a learning system that satisfies such constraints.

Insufficient baseline comparisons and ablation studies

Since the submission date, we have collected LIBERO simulation results for comparing the performance of DSRL (Wagenmaker et al. 2025) and a behavior-cloned LWP with RELAY. Furthermore, we have performed ablation experiments of different design choices of RELAY such as the two-stage learning process, Cal-QL, replay sampling ratio, and VLA recovery. The following two tables summarize the results. Numbers in parentheses show the minimum and maximum values over three seeds. Additional details and results are available in section 5 and appendices A.4 and A.5 of the updated paper.

Baseline success rate is averaged over the last 10K steps and 10 tasks. As shown, RELAY can achieve higher average success rate than the baselines.

MetricRELAYDSRLOffline BCPi0 VLA
Success rate % (min-max)68 (67-70)51 (49-53)28 (28-30)35 (33-36)

Ablation success rates are averaged over the last 10K steps. The RLPD-style ablation uses SAC with a critic ensemble, high UTD, and no entropy backup term in the critic target. The results show that using the Cal-QL algorithm and the two-stage learning process are crucial to the performance of RELAY.

Design choiceVariantTask 28Task 38Task 43Task 46
BaselineRELAY100 (99-100)41 (36-46)94 (92-95)76 (59-89)
Cal-QLRLPD-style97 (96-98)47 (42-52)23 (21-28)8 (4-12)
Cal-QLRLPD-style + entropy backup98 (96-100)24 (20-32)22 (21-24)8 (8-8)
Two stagesSkip stage 199 (98-100)0 (0-0)0 (0-0)0 (0-0)
Sampling ratioStage 1: 100% VLA99 (97-100)41 (24-50)88 (87-89)14 (12-18)
Sampling ratioFixed 50%/50%99 (99-100)43 (24-54)88 (87-91)57 (15-83)
Sampling ratioStage 2: 100% LWP100 (99-100)35 (28-42)86 (81-89)74 (47-88)
VLA recoveryEnabled95 (91-97)47 (35-54)77 (58-89)71 (64-80)

Limited task diversity and generalization

We were limited in our choice of tasks by the capabilities of the pre-trained π0 VLA fine-tuned on the DROID dataset since we did not want to collect new demonstrations and fine-tune the VLA. We experimented with a variety of different tasks including folding a towel, wiping the table, pressing a button, etc., on which the DROID-fine-tuned π0 VLA did not have a reliable starting success rate, e.g., >20%, in our physical setup. Generalization to unseen tasks and environments is out of the scope of our work if the pre-trained VLA cannot perform the task at least poorly out of the box.

Evaluation protocol and presentation

While the evaluation protocol is sensible since the VLA and LWP both use the same reset script and the LWP was evaluated after training on the same day, we will re-evaluate the VLA and LWP in immediate succession for a fairer comparison and update Figure 5. We will also address readability concerns.

The framework relies on task-specific success detectors and automated reset procedures. Can it still operate with minimal human intervention on more complex tasks or in environments where automated resets are difficult?

Human interventions can be reduced along multiple dimensions including demonstrations, corrective interventions during learning, success labels, and resets. Our work is primarily concerned with learning without human demonstrations or corrective interventions during learning. Although our implemented success detectors and automated reset procedures work well in practice on our tasks, their effectiveness on more complex tasks is not guaranteed.

We appreciate the other questions and will address them when results become available.

Wagenmaker, A., Nakamoto, M., Zhang, Y., Park, S., Yagoub, W., Nagabandi, A., Gupta, A., Levine, S. (2025). Steering your diffusion policy with latent space reinforcement learning. In Proceedings of the 9th Conference on Robot Learning.

We would like to remind the reviewer of our response as we are approaching the discussion deadline.

Reviewer_de4FOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes RELAY, a framework for improving a pretrained vision-language-action (VLA) model by training a separate lightweight policy (LWP) through online reinforcement learning. The pretrained VLA and its visual encoder remain frozen, while the LWP operates on image embeddings and proprioceptive information. The training procedure has two stages: initially, the VLA generates most of the experience, after which the LWP progressively takes over. The method uses Calibrated Q-Learning (Cal-QL), separate replay buffers for VLA- and LWP-generated experience, and occasional VLA-assisted recovery. The authors also automate task success detection and environment resets to reduce human intervention.

The framework is evaluated on three real-robot manipulation tasks using a Franka Panda robot. The authors report that the trained LWP achieves higher success rates and shorter successful-episode durations than the pretrained π0 baseline, attributing the improvements to both faster inference and more efficient behavior learned through RL.

The problem is practically relevant, and training an independent lightweight policy could reduce inference overhead. However, the paper's methodological contribution beyond the combination of existing components is insufficiently established. The evaluation is restricted to three relatively short manipulation tasks, the baseline comparisons and ablations are inadequate to isolate the source of the improvements, and the dependence on task-specific success detection raises questions about scalability. Overall, the evidence does not support the broad claims implied by the paper's title.

Critical strengths
  1. The paper demonstrates online RL training on a physical robot rather than relying exclusively on simulation.

  2. The method might show potential reduction in inference overhead and human intervention.

Critical weaknesses

1. Insufficient methodological novelty and inadequate experimental controls.

RELAY combines a frozen pretrained visual representation, a separate lightweight policy, off-policy RL with Cal-QL, staged policy-generated experience, mixed replay buffers, and VLA-assisted recovery. These components form a plausible engineering pipeline, but the paper does not adequately establish what is methodologically novel or why the proposed combination is necessary. The absence of key ablations is a major issue. The authors do not isolate the contributions of Cal-QL, the two-stage training schedule, replay-buffer mixing, and VLA-assisted recovery. Nor do they provide sufficient comparisons against simpler alternatives, such as SAC under a matched setup or behavioral cloning from successful trajectories. Consequently, the reported improvements cannot be confidently attributed to the proposed RL design rather than to faster inference, the availability of VLA-generated experience, or other components of the pipeline. The baseline evaluation protocol also raises concerns. The paper reports LWP evaluation over 30 episodes after training, while the VLA results are drawn from episodes collected during training. These are not clearly matched evaluation conditions. Both policies should be evaluated using the same protocol and comparable initial-state distributions. Without matched comparisons and targeted ablations, the paper demonstrates that the pipeline can work but does not establish that it is superior to simpler alternatives.

**2. Narrow experimental scope and insufficient evidence of generality. **

The evaluation covers only three relatively short manipulation tasks: closing a drawer, picking up a banana, and placing a banana on a plate. These tasks do not sufficiently test the complex, long-horizon, and instruction-dependent behaviors that motivate generalist VLAs. For example, the paper does not investigate multi-stage task execution, flexible task decomposition, or transfer to substantially different tasks and environments. This is particularly important given the broader capabilities associated with generalist VLA systems, such as π0.5. Moreover, the concrete LWP architecture is described as operating on image embeddings and proprioception, without an explicit language input. The paper therefore does not establish whether the framework preserves language-conditioned generalization, supports multiple instructions with a shared policy, or requires a separately trained policy for each task. The results support a much narrower conclusion than the title suggests: the LWP can outperform the selected π0 baseline on three particular tasks, not that the approach generally surpasses VLA performance. The claimed generality of the training pipeline is also insufficiently substantiated. Although the reward formulation has a largely shared structure, task completion still depends on task-specific success detection. The paper does not adequately quantify the engineering required to implement and validate these detectors for new tasks, nor does it evaluate the effects of false-positive or false-negative success signals. This is a meaningful scalability concern, especially for long-horizon tasks with multiple intermediate goals. A shared reward formula alone does not establish that the full training pipeline is task-agnostic or robust.

Questions for authors
  1. Can the authors provide a more rigorous and statistically reliable comparison between RELAY and the pretrained VLA baseline?

  2. Can the authors include experiments in a standard robotic manipulation simulator, such as RoboCasa, and compare RELAY against representative baselines?

  3. How much task-specific engineering and human intervention are required in practice?

4 comments

We appreciate the reviewer’s constructive comments.

The absence of key ablations is a major issue. The authors do not isolate the contributions of Cal-QL, the two-stage training schedule, replay-buffer mixing, and VLA-assisted recovery. Nor do they provide sufficient comparisons against simpler alternatives, such as SAC under a matched setup or behavioral cloning from successful trajectories.

Since the paper was submitted, we have gathered results comparing the performance of DSRL (Wagenmaker et al. 2025) and a behavior-cloned LWP with RELAY on LIBERO simulation tasks. In addition, we have performed ablation experiments of different design choices of RELAY such as the two-stage learning process, Cal-QL, replay sampling ratios, and VLA recovery. The following two tables summarize the results of the baseline and ablation experiments. Numbers in parentheses denote minimum and maximum values over three seeds. Additional results and details are in section 5 and appendices A.4 and A.5 of the updated paper.

Baseline success rate is averaged over the last 10K steps and 10 tasks. RELAY achieves higher average success rate than other baseline methods.

MetricRELAYDSRLOffline BCPi0 VLA
Success rate % (min-max)68 (67-70)51 (49-53)28 (28-30)35 (33-36)

Ablation success rates are averaged over the last 10K steps. The RLPD-style ablation uses SAC with a critic ensemble, high UTD, and no entropy backup term in the critic target. As shown in the results, the two-stage learning process and the Cal-QL algorithm are crucial to the performance of RELAY.

Design choiceVariantTask 28Task 38Task 43Task 46
BaselineRELAY100 (99-100)41 (36-46)94 (92-95)76 (59-89)
Cal-QLRLPD-style97 (96-98)47 (42-52)23 (21-28)8 (4-12)
Cal-QLRLPD-style + entropy backup98 (96-100)24 (20-32)22 (21-24)8 (8-8)
Two stagesSkip stage 199 (98-100)0 (0-0)0 (0-0)0 (0-0)
Sampling ratioStage 1: 100% VLA99 (97-100)41 (24-50)88 (87-89)14 (12-18)
Sampling ratioFixed 50%/50%99 (99-100)43 (24-54)88 (87-91)57 (15-83)
Sampling ratioStage 2: 100% LWP100 (99-100)35 (28-42)86 (81-89)74 (47-88)
VLA recoveryEnabled95 (91-97)47 (35-54)77 (58-89)71 (64-80)

The baseline evaluation protocol also raises concerns. The paper reports LWP evaluation over 30 episodes after training, while the VLA results are drawn from episodes collected during training.

While the evaluation protocol is sensible since the VLA and LWP both used the same reset script and were evaluated on the same day, we will re-evaluate VLA and LWP in immediate succession and matching initial states to ensure a fairer comparison and update Figure 5.

The evaluation covers only three relatively short manipulation tasks: closing a drawer, picking up a banana, and placing a banana on a plate. These tasks do not sufficiently test the complex, long-horizon, and instruction-dependent behaviors that motivate generalist VLAs. For example, the paper does not investigate multi-stage task execution, flexible task decomposition, or transfer to substantially different tasks and environments.

The goal of our paper, as described in the problem setup, is to surpass the performance of a pre-trained VLA without collecting any human demonstrations on tasks where the pre-trained VLA has non-zero success rate. We instantiate our proposed RELAY framework using the pre-trained π0 VLA that has been fine-tuned on the DROID dataset. The DROID-fine-tuned π0 VLA is incapable of achieving non-zero success rates on complex and long-horizon tasks out of the box. Therefore, to test RELAY on complex and long-horizon tasks, we would need to collect human demonstrations, which is not part of our problem setup and is out of the scope of our work. Alternatively, we would need to use a more capable VLA, which may not be publicly available. Our picking, pick and place, and drawer closing tasks cover common general task types similar to the types of tasks used in DSRL.

The paper therefore does not establish whether the framework preserves language-conditioned generalization, supports multiple instructions with a shared policy, or requires a separately trained policy for each task.

The proposed RELAY framework in our paper does not have language-conditioned generalization in the learned lightweight policy. Our paper is concerned with surpassing the performance of a pre-trained VLA on a single task at a time, and we leave multi-task generalization to future work. Conditioning the LWP on language or VLM embeddings or using continual RL methods could be potential approaches for making the LWP generalize to multiple tasks.

The results support a much narrower conclusion than the title suggests: the LWP can outperform the selected π0 baseline on three particular tasks, not that the approach generally surpasses VLA performance.

Our LIBERO simulation results indicate that RELAY is able to surpass the performance of π0 on the majority of tasks, which serves as evidence toward the generality of our approach. We will add the simulation results to the paper.

The paper does not adequately quantify the engineering required to implement and validate these detectors for new tasks, nor does it evaluate the effects of false-positive or false-negative success signals.

In practice, implementing the success detector with the help of LLM coding agents took a few hours for each task. The following results show the accuracy and confusion matrix for the success detector of each task. For each task, we executed 30 VLA episodes until timeout. After an episode timeout, a human was asked whether the episode became successful at any point during the episode without observing the prediction from the success detector.

Close the red drawer: Accuracy: 100% (30/30)

Actual \ PredictedPositiveNegative
Positive4 (TP)0 (FN)
Negative0 (FP)26 (TN)

Put the banana on the plate: Accuracy: 83.33% (25/30)

Actual \ PredictedPositiveNegative
Positive6 (TP)5 (FN)
Negative0 (FP)19 (TN)

Pick up the banana: Accuracy: 90.00% (27/30)

Actual \ PredictedPositiveNegative
Positive7 (TP)2 (FN)
Negative1 (FP)20 (TN)

How much task-specific engineering and human intervention are required in practice?

In practice, automated success detection and automated reset were implemented once before any experiments with the help of LLM coding agents in a few hours to less than a day. During experiments, human interventions were needed when an object or the robot entered an unrecoverable state maybe once an hour as a rough estimate. We will add such details to the paper, which are helpful in gauging what we mean by minimal human interventions.

We acknowledge and will address the remaining questions once further results are available.

Wagenmaker, A., Nakamoto, M., Zhang, Y., Park, S., Yagoub, W., Nagabandi, A., Gupta, A., Levine, S. (2025). Steering your diffusion policy with latent space reinforcement learning. In Proceedings of the 9th Conference on Robot Learning.

  1. The problem setting in this paper is fundamentally unrealistic. In practical VLA training, a scenario in which expert demonstrations are entirely unavailable simply does not arise. The authors’ reference to their use of the π0.5 DROID checkpoint therefore amounts to an unconvincing attempt to justify this unrealistic setting.

  2. The LIBERO experiments are insufficiently challenging, and the evaluation is further limited to a small subset of tasks selected by the authors.

For these reasons, I do not believe this paper meets the standard for acceptance at ICLR, and I will maintain my score of 1.

The problem setting in this paper is fundamentally unrealistic. In practical VLA training, a scenario in which expert demonstrations are entirely unavailable simply does not arise. The authors’ reference to their use of the π0.5 DROID checkpoint therefore amounts to an unconvincing attempt to justify this unrealistic setting.

There are many tasks in which human demonstrations are available or can be collected readily. However, it is also valuable to understand how the performance of pre-trained models can be improved when demonstrations are unavailable or hard to obtain. For instance, a household robot may need to adapt when occupants are unavailable and remote teleoperation is restricted for privacy. A critical benefit of this capability is that robot deployment can scale more easily to end-user tasks if the dependence on human demonstrations is minimal.

Further, this potential benefit has motivated improving pre-trained VLA performance without task-specific demonstrations in several prior works such as DSRL (Wagenmaker et al. 2025), V-GPS (Nakamoto et al. 2024), Robometer (Liang et al. 2026), InSight (Wang et al. 2026), EVOLVE-VLA (Bai et al. 2025), and the work of Nurtdinov et al. (2026).

The LIBERO experiments are insufficiently challenging, and the evaluation is further limited to a small subset of tasks selected by the authors.

The LIBERO tasks used in our simulation experiments were selected as follows. We evaluated the publicly-available π0 VLA fine-tuned for LIBERO on LIBERO-90 tasks, which were not used for π0 fine-tuning. We then selected tasks that had success rates between 20% and 50%, resulting in 11 tasks in total. We used one of the tasks for hyper-parameter tuning and the remaining 10 tasks for our simulation experiments. The ablation experiments will be extended to all of the 10 tasks.

Bai, Z., Gao, C., Shou, M. Z. (2025). EVOLVE-VLA: Test-time training from environment feedback for vision-language-action models. arXiv preprint arXiv:2512.14666.

Liang, A., Korkmaz, Y., Zhang, J., Hwang, M., Anwar, A., Kaushik, S., Shah, A., Huang, A. S., Zettlemoyer, L., Fox, D., Xiang, Y., Li, A., Bobu, A., Gupta, A., Tu, S., Biyik, E., Zhang, J. (2026). Robometer: Scaling general-purpose robotic reward models via trajectory comparisons. In Proceedings of Robotics: Science and Systems XXII.

Nurtdinov, D., Kornaev, A., Maloletov, A. (2026). RL bootstrapping of OpenVLA-OFT for a novel robot embodiment. arXiv preprint arXiv:2608.01013.

Wang, M., Osterberg, L., Tian, S., Shorinwa, O., Bowkett, J., Pastor, D., Wu, J., Schwager, M. (2026). InSight: Self-guided skill acquisition via steerable VLAs. arXiv preprint arXiv:2606.24884.

Reviewer_zT6ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RELAY, which pairs a frozen π0-DROID VLA with a lightweight policy (LWP). The LWP is a feedforward head on π0's frozen SigLIP embeddings plus proprioception. It is trained with online Cal-QL using a 10-critic ensemble. Training runs in two stages. In Stage 1, the VLA collects most episodes, the LWP is evaluated every fifth episode, and minibatches are split 50/50 between the two buffers. In Stage 2, the LWP acts, the VLA steps in for recovery after a timeout, and sampling decays to all-LWP data. Success detection (SAM3 masks plus thresholds) and resets are scripted per task, with help from LLM coding agents. On three Franka tasks (close drawer, pick banana, banana on plate), the final LWP beats π0 on success rate and success duration within 2.8 to 5.8 hours of interaction. A timestep/inference decomposition (Eq. 1) attributes 29 to 55% of the speedup to behavior learned through RL.

Critical strengths
  • At deployment the LWP never needs a VLA forward pass, only image embeddings. Steering, residual and value-guided methods (Nakamoto et al., 2024; Wagenmaker et al., 2025; Xu et al., 2026) still query the VLA every step. This is a real practical distinction.
  • Eq. 1 cleanly splits the speedup into fewer timesteps versus faster inference. Assigning the interaction term to inference makes the RL share a conservative lower bound. Table 1 shows the behavioral share is not trivial, especially on the drawer task (55%).
  • The unsafe-behavior penalty equals the return of a timed-out episode, so ending an episode early by acting unsafely can never beat failing. This is a small but well-reasoned reward choice.
  • The training loop runs end to end on real hardware without demonstrations or human success labels, on a single RTX 3090 with asynchronous updates.
Critical weaknesses
  1. The only baseline is zero-shot π0. None of the closest methods in Section 5 are compared: DSRL, RL Token, residual RL (Xiao et al.), EXPO, V-GPS, IBRL. Simpler baselines that would isolate the contribution are also missing:

    • filtered BC (distillation) of the LWP on successful VLA rollouts, with and without RL afterward;
    • online RL on the same SigLIP features without the VLA buffer;
    • RLPD (Ball et al., 2023), which is not cited. The last omission matters most. Symmetric 50/50 sampling of prior and online data, a critic ensemble with a random-subset min (REDQ), and a high update-to-data ratio (20 critic updates per step) together are close to the RLPD recipe. As it stands, the paper cannot say whether the gains come from the two-stage handoff, from Cal-QL, or simply from running any reasonable online RL on top of VLA-generated successes.
  2. Mechanistic claims about Cal-QL lack direct evidence. Section 3.1 says SAC on VLA data produces unsafe, random behavior because of extrapolation error. No SAC or RLPD ablation, Q-value diagnostic, or learning curve supports this. Cal-QL is also used outside its intended offline-to-online setting: its calibration term uses Monte Carlo returns from a mixture of two behavior policies whose mixture changes over time. Why calibrating against this nonstationary reference is the right lower bound is never discussed.

  3. The VLA and LWP are evaluated under different protocols. VLA numbers are pooled from all of its training-time episodes, with stochastic sampling, across hours of possible scene drift. LWP numbers come from 30 deterministic episodes at the end. A same-session evaluation of both policies on matched initial states is needed before the Figure 5 gaps can be read as policy differences. Each task is also a single training run, with no seeds or repeats, so run-to-run variance is unknown.

Questions for authors
  • How does RELAY compare with filtered BC of the LWP on successful VLA rollouts, and with RLPD using the VLA buffer as prior data?
  • What happens if Cal-QL is replaced with SAC (the failure Section 3.1 describes), and if the VLA buffer is removed after Stage 1?
  • How accurate are the success detectors, measured against human labels on a held-out set?
  • How many human interventions and reset failures occurred per task, and what was the total wall-clock time including resets?
  • How does the VLA perform when evaluated in the same session, on the same initial states, as the final LWP?
  • How much of the VLA's failure rate goes away if π0 is run with a gripper matching the DROID setup?
3 comments

We thank the reviewer for their detailed and constructive feedback.

The only baseline is zero-shot π0.

Since the submission date, we have performed new experiments to compare the performance of RELAY with DSRL (Wagenmaker et al. 2025) and a behavior-cloned LWP in LIBERO simulation tasks. Regular online RL is not a valid baseline as it would not learn to perform manipulation tasks from sparse rewards. The following table shows the summary of the results. Success rates are averaged over the last 10K steps and 10 tasks. Numbers in parentheses denote minimum and maximum over three seeds. Relay achieves higher success rates than the baseline methods.

MetricRELAYDSRLOffline BCPi0 VLA
Success rate % (min-max)68 (67-70)51 (49-53)28 (28-30)35 (33-36)

Further, we have compared RELAY with RLPD (Ball et al. 2023) through ablation experiments of different design choices of RELAY such as the two-stage learning process, Cal-QL, replay sampling ratio, and VLA recovery. The following table shows average success rates over the last 10K steps. The RLPD-style ablation uses SAC with a critic ensemble, high UTD, and no entropy backup term in the critic target. The results show that the Cal-QL algorithm and the two-stage learning process are crucial to the performance of RELAY. Additional baseline and ablation results are in section 5 and appendices A.4 and A.5 of the updated paper.

Design choiceVariantTask 28Task 38Task 43Task 46
BaselineRELAY100 (99-100)41 (36-46)94 (92-95)76 (59-89)
Cal-QLRLPD-style97 (96-98)47 (42-52)23 (21-28)8 (4-12)
Cal-QLRLPD-style + entropy backup98 (96-100)24 (20-32)22 (21-24)8 (8-8)
Two stagesSkip stage 199 (98-100)0 (0-0)0 (0-0)0 (0-0)
Sampling ratioStage 1: 100% VLA99 (97-100)41 (24-50)88 (87-89)14 (12-18)
Sampling ratioFixed 50%/50%99 (99-100)43 (24-54)88 (87-91)57 (15-83)
Sampling ratioStage 2: 100% LWP100 (99-100)35 (28-42)86 (81-89)74 (47-88)
VLA recoveryEnabled95 (91-97)47 (35-54)77 (58-89)71 (64-80)

Other mentioned works either did not have publicly available code or did not match our problem setup. RL token (Xu et al. 2026) and PLD (Xiao et al. 2026) do not provide publicly-available code and their real-robot experiments involve an offline pre-training stage on human demonstrations. EXPO-FT (Dong et al. 2026) released their codebase after the ICLR 2027 submission deadline. V-GPS (Nakamoto et al. 2024) is an offline approach with no online improvement in performance. IBRL (Hu et al. 2024) does not use a VLA. However, the proposed improvements to the interaction and training phases in IBRL are orthogonal to and can be incorporated into RELAY. All of these methods, RL token, PLD, EXPO-FT, V-GPS, and IBRL, rely on one or more VLA forward passes to generate one action chunk. In contrast, RELAY learns online without any offline stages or human demonstrations, and the learned lightweight policy does not rely on VLA forward pass.

Section 3.1 says SAC on VLA data produces unsafe, random behavior because of extrapolation error. No SAC or RLPD ablation, Q-value diagnostic, or learning curve supports this.

In this work, we propose the RELAY framework and study a particular instantiation of RELAY that uses Cal-QL as the learning algorithm. The goal of the work was to show that our particular RELAY-based agent can surpass the performance of the pre-trained π0 VLA without human demonstrations. For this purpose, an ablation study comparing RELAY with SAC or RLPD is not essential. However, we added ablation results on LIBERO tasks in section 5 and appendix A.5 in the updated paper.

[Cal-QL’s] calibration term uses Monte Carlo returns from a mixture of two behavior policies whose mixture changes over time. Why calibrating against this nonstationary reference is the right lower bound is never discussed.

We performed an ablation of the Cal-QL calibration term on two LIBERO tasks in appendix A.6. Compared with RELAY, not using the calibration term did not increase or decrease the success duration and may slightly increase the success rate. We will perform our real-robot runs without the calibration term.

The VLA and LWP are evaluated under different protocols.

While the evaluation protocol is sensible since VLA and LWP both used the same reset procedure and the LWP was evaluated right after training on the same day, we will perform evaluations of the VLA and LWP again in close succession to ensure a more accurate comparison and will update Figure 5.

Each task is also a single training run, with no seeds or repeats

Our single training run and multi-trial evaluations of the final policy on a real robot are in line with contemporary papers that also improve the performance of VLAs with online interaction and do not report multiple training runs. Nonetheless, we agree with the value of running experiments with multiple seeds and aim to add more real-robot runs.

How accurate are the success detectors, measured against human labels on a held-out set?

We measured the accuracy of the automated success detector by executing 30 VLA episodes for each task. Each episode ran until task-specific timeout. After an episode timeout, we asked a human whether the episode became successful at any point during the episode and compared the result to the predictions from the success detector. The human provided a label without observing the predictions. The following results show the accuracies and confusion matrices for the success detector of each task.

Close the red drawer: Accuracy: 100% (30/30)

Actual \ PredictedPositiveNegative
Positive4 (TP)0 (FN)
Negative0 (FP)26 (TN)

Put the banana on the plate: Accuracy: 83.33% (25/30)

Actual \ PredictedPositiveNegative
Positive6 (TP)5 (FN)
Negative0 (FP)19 (TN)

Pick up the banana: Accuracy: 90.00% (27/30)

Actual \ PredictedPositiveNegative
Positive7 (TP)2 (FN)
Negative1 (FP)20 (TN)

We appreciate the reviewer’s list of questions and will address them when the results for our additional experiments are ready.

Ball, P. J., Smith, L., Kostrikov, I., Levine, S. (2023). Efficient online reinforcement learning with offline data. In Proceedings of the 40th International Conference on Machine Learning.

Dong, P., Hung, K.-H., Gao, T., Sadigh, D., Finn, C. (2026). EXPO-FT: Sample-efficient reinforcement learning finetuning for vision-language-action models. arXiv preprint arXiv:2605.25477.

Hu, H., Mirchandani, S., Sadigh, D. (2024). Imitation bootstrapped reinforcement learning. In Proceedings of Robotics: Science and Systems XX.

Nakamoto, M., Mees, O., Kumar, A., Levine, S. (2024). Steering your generalists: Improving robotic foundation models via value guidance. In Proceedings of the 8th Conference on Robot Learning.

Wagenmaker, A., Nakamoto, M., Zhang, Y., Park, S., Yagoub, W., Nagabandi, A., Gupta, A., Levine, S. (2025). Steering your diffusion policy with latent space reinforcement learning. In Proceedings of the 9th Conference on Robot Learning.

Xiao, W., Lin, H., Peng, A., Xue, H., He, T., Xie, Y., Hu, F., Wu, J., Luo, Z., Fan, L., Shi, G., Zhu, Y. (2026). Self-improving vision-language-action models with data generation via residual RL. In Proceedings of the 14th International Conference on Learning Representations.

Xu, C., Springenberg, J. T., Equi, M., Amin, A., Esmail, A., Levine, S., Ke, L. (2026). RL token: Bootstrapping online RL with vision-language-action models. In Proceedings of the 10th Conference on Robot Learning.

We would like to remind the reviewer of our response as we are approaching the discussion deadline.

Comments on the paper

1 comment

Dear Reviewers,

We thank you for your valuable feedback and have revised the paper with additional experimental results in section 5 and appendices A.4 to A.6.

Kind regards, Authors of submission 54042

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.