- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents DRIVE, a post-training framework designed to refine flow-based VLA policies using mixed robot experience including expert demonstrations, autonomous successes, autonomous failures, and human interventions. The method adapts a vision-language progress estimator on successful rollouts to provide dense chunk-level rewards, which are distilled into an action-chunk critic via implicit Q-learning. Under the same state condition, the critic evaluates candidate action chunks to produce relative soft labels that guide a negative-aware flow-matching objective, reinforcing high-value actions while suppressing inferior ones. The critic then provides a negative-aware objective that reinforces high-value actions and penalizes low-value alternatives Experiments report 98.25% average success on LIBERO and 62.5% on four real-world tasks.
Critical strengths
The paper solves the problem of how to effectively utilizing mixed robot execution experience especially failures and human interventions. It avoids the problem of negative weight in SFT will cause the model to collapse. The method is simple and effective, and the experiments show that it can significantly improve the performance of the policy.
Critical weaknesses
- some core components are not novel, such as the vision-language progress estimator and the negative-aware fine-tuning.
- The effectiveness of the method is heavily dependent on the quality of the vision-language progress estimator and the critic.
Questions for authors
- The paper mentions that the method is post-training, but unlike other online RL methods, it does not update the critic during training. Will this cause potential reward hacking?
- The dense reward is generated by a learned progress estimator, which may be inaccurate. How does the quality of the progress estimator affect the performance of the method?
- How is the failure and human intervention data collected in LIBERO? The base model of DRIVE is already 97.2% on LIBERO, so the benchmark is a bit saturated.