- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PF-RL learns a goal-conditioned value function from VLA features and uses value changes to estimate action progress. This signal supports offline policy training and online refinement. Experiments cover LIBERO, four RoboTwin tasks, and four real-world bimanual manipulation tasks.
Critical strengths
-
PF-RL improves over the reproduced GRPO baseline under matched initialization, data, and interaction budgets. The improvement on RoboTwin, from approximately 79% to 92%, provides useful evidence beyond the nearly saturated LIBERO benchmark.
-
Comparisons with alternative value functions and ablations of the training objectives support the proposed design. Experiments on four real-world tasks also show improvements over EvoRL, extending the evaluation beyond simulation.
Critical weaknesses
-
Comparisons with strong existing RL methods are incomplete. SimpleVLA-RL and SRPO report similar LIBERO success rates, but direct comparisons are missing. SRPO is particularly relevant because it also uses successful trajectories to provide progress-related feedback for failed attempts. Comparisons under comparable training budgets would better establish the practical benefit of PF-RL.
-
Real-world training includes human corrections, but their frequency and duration are not reported. Following the same collection protocol does not necessarily provide comparable amounts of corrective supervision. This makes it difficult to assess how much of the improvement comes from the proposed method and how much depends on human assistance.
Questions for authors
-
How does PF-RL compare with SimpleVLA-RL and, particularly, SRPO under comparable data and interaction budgets, accounting for additional computation?
-
How much human intervention was required by each method in the real-world experiments, and how was the corrective supervision controlled across methods?
3 comments
Thank you for the detailed response and additional experiments. The authors have adequately addressed my main concerns, and I have increased my rating accordingly.
Thank you for your positive feedback and for updating your score. We sincerely appreciate your time and constructive suggestions, which have helped us improve our paper.
[W1 & Q1: Incomplete comparisons with strong existing RL methods.]
Thank you for the constructive comments. To provide a more comprehensive comparison with existing RL methods, we additionally evaluate SimpleVLA-RL and SRPO on LIBERO. Both methods are initialized from the same 30K-step SFT checkpoint used by RLinf-GRPO and trained for 100 online RL updates, with 64 groups per update, 8 trajectories per group, and 512 trajectories in total per update. For PF-RL, we retain our original offline-to-online training pipeline. All methods are evaluated under the same protocol with the same training budget. As shown in Table R1, PF-RL achieves an average success rate of 99.1%, outperforming RLinf-GRPO (97.1%), SimpleVLA-RL (97.5%), and SRPO (98.0%). In particular, PF-RL improves over SRPO by 1.4 and 2.0 percentage points on LIBERO-Goal and LIBERO-Long, respectively, both of which demand stronger long-horizon decision-making capabilities. These results provide further evidence of the effectiveness of PF-RL compared with strong existing RL methods. We have included the additional experimental details and results in Appendix D.
Table R1. Comparison with recent RL baselines on LIBERO under matched data and training budgets.
We additionally report the online training time of each method using the same cluster of 8 NVIDIA A800 GPUs, as summarized in Table R2. Despite differences in their implementation pipelines, PF-RL incurs only a modest additional computational cost compared with RLinf-GRPO, while requiring less training time than SimpleVLA-RL and SRPO.
Table R2. Online training time comaprison of different online RL methods under matched data and training budgets.
[W2 & Q2: Real-world training includes human corrections, but their frequency and duration are not reported. ]
Thanks for raising this important concern. We additionally report the human intervention details for Evo-RL and PF-RL across four real-world tasks and two rounds of RL training (40 rollouts per task), as summarized in Table R3. All methods follow the same intervention protocol: human operators take over when the robot exhibits execution errors or becomes unable to make meaningful task progress, and return control to the policy once execution recovers to a stable state. As shown in Table R3, the corrective-supervision budgets are broadly comparable between PF-RL and Evo-RL, with overall human-control durations of 4.6 and 5.3 minutes and human-control ratios of 6.9% and 7.2%, respectively, where the human-control ratio represents the percentage of training timesteps involving human intervention. These results support the fairness of the real-world comparison by showing that PF-RL's improvements are not attributable to receiving more human corrective supervision. We have added the human intervention details to the revised Appendix B.2.
Table R3. Human intervention details during real-world RL training (40 rollouts per task).