- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents RELAY, a framework that improves the performance of pretrained Vision-Language-Action (VLA) models through online reinforcement learning. The method freezes the pretrained VLA and trains a lightweight policy (LWP) using its visual encoder. A two-stage online RL process gradually transfers control from the VLA to the LWP. Experiments on three real-world robotic manipulation tasks demonstrate that RELAY improves task success rates and execution efficiency without requiring additional human demonstrations.
Critical strengths
Practical motivation and real-world validation:
This paper addresses the practical problem of improving pretrained VLA performance without collecting additional human demonstrations. The authors conduct real-world online RL experiments and demonstrate substantial improvements in task success rates and execution efficiency.
Efficient and practical framework:
RELAY reduces inference and training costs by freezing the pretrained VLA and training an independent lightweight policy. The two-stage learning process, combined with automated environment resets, enables real-world robot learning with minimal human intervention.
Critical weaknesses
Limited novelty: The proposed framework mainly combines existing components, including a pretrained VLA, a lightweight policy network, and Cal-QL, with a staged online learning procedure. The approach appears more like an engineering integration of established techniques than a fundamentally new algorithm or learning mechanism. Its unique contribution compared to existing lightweight policy adaptation and online RL methods remains unclear.
Insufficient baseline comparisons and ablation studies: This is my primary concern. The paper mainly compares the trained LWP against the original pretrained VLA without downstream adaptation. However, showing that a task-specific policy trained with additional online RL outperforms an unadapted VLA is insufficient to establish the effectiveness of the proposed RELAY framework.
The paper lacks comparisons with standard online RL methods, behavior cloning using VLA-generated trajectories, and existing lightweight policy adaptation approaches under matched interaction budgets. Furthermore, key components, including the two-stage training procedure, replay sampling ratios, and the choice of Cal-QL, are not systematically ablated. It is therefore difficult to determine which components are responsible for the reported improvements.
Limited task diversity and generalization:
The experiments involve only three relatively simple real-world manipulation tasks, two of which involve the same object. Moreover, the method assumes that the pretrained VLA already achieves a non-zero success rate and trains a separate lightweight policy for each task. Consequently, the experiments mainly demonstrate task-specific performance improvements, rather than generalization to more complex or unseen tasks and environments.
Evaluation protocol and presentation: The final LWP is evaluated over 30 episodes per task, while the VLA baseline performance is estimated from episodes collected during online training. Although the VLA remains frozen, it is unclear whether both policies are evaluated under identical initial-state distributions and evaluation conditions. A matched evaluation protocol would make the comparison more rigorous. Regarding presentation, some framework diagrams and experimental figures contain small text, and the figure layouts and excessive whitespace on certain pages reduce overall readability.
Questions for authors
Could the authors compare RELAY against standard Cal-QL, SAC, or behavior cloning using VLA-generated trajectories under the same online interaction budget? It remains unclear whether the improvements primarily come from RELAY's specific design or simply from additional task-specific training.
How much do the two-stage training procedure and replay sampling strategy individually contribute to the final performance? Could the authors provide corresponding ablation studies?
The framework relies on task-specific success detectors and automated reset procedures. Can it still operate with minimal human intervention on more complex tasks or in environments where automated resets are difficult?
2 comments
We would like to remind the reviewer of our response as we are approaching the discussion deadline.
We thank the reviewer for their constructive feedback.
Combining existing techniques and demonstrating its effectiveness on real-robot tasks can be a valuable contribution on par with a fundamentally new algorithm or learning mechanism. Furthermore, much of the value in our contribution comes from constraints such as not collecting human demonstrations and not relying on a VLA forward pass for generating actions, which are often violated in related work, and proposing a framework and a learning system that satisfies such constraints.
Since the submission date, we have collected LIBERO simulation results for comparing the performance of DSRL (Wagenmaker et al. 2025) and a behavior-cloned LWP with RELAY. Furthermore, we have performed ablation experiments of different design choices of RELAY such as the two-stage learning process, Cal-QL, replay sampling ratio, and VLA recovery. The following two tables summarize the results. Numbers in parentheses show the minimum and maximum values over three seeds. Additional details and results are available in section 5 and appendices A.4 and A.5 of the updated paper.
Baseline success rate is averaged over the last 10K steps and 10 tasks. As shown, RELAY can achieve higher average success rate than the baselines.
Ablation success rates are averaged over the last 10K steps. The RLPD-style ablation uses SAC with a critic ensemble, high UTD, and no entropy backup term in the critic target. The results show that using the Cal-QL algorithm and the two-stage learning process are crucial to the performance of RELAY.
We were limited in our choice of tasks by the capabilities of the pre-trained π0 VLA fine-tuned on the DROID dataset since we did not want to collect new demonstrations and fine-tune the VLA. We experimented with a variety of different tasks including folding a towel, wiping the table, pressing a button, etc., on which the DROID-fine-tuned π0 VLA did not have a reliable starting success rate, e.g., >20%, in our physical setup. Generalization to unseen tasks and environments is out of the scope of our work if the pre-trained VLA cannot perform the task at least poorly out of the box.
While the evaluation protocol is sensible since the VLA and LWP both use the same reset script and the LWP was evaluated after training on the same day, we will re-evaluate the VLA and LWP in immediate succession for a fairer comparison and update Figure 5. We will also address readability concerns.
Human interventions can be reduced along multiple dimensions including demonstrations, corrective interventions during learning, success labels, and resets. Our work is primarily concerned with learning without human demonstrations or corrective interventions during learning. Although our implemented success detectors and automated reset procedures work well in practice on our tasks, their effectiveness on more complex tasks is not guaranteed.
We appreciate the other questions and will address them when results become available.
Wagenmaker, A., Nakamoto, M., Zhang, Y., Park, S., Yagoub, W., Nagabandi, A., Gupta, A., Levine, S. (2025). Steering your diffusion policy with latent space reinforcement learning. In Proceedings of the 9th Conference on Robot Learning.