- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper describes a method for improving the sample efficiency of an RL framework. The main idea is to make use of a learned world model, along with a VLM that can identify failures and propose corrective actions. The world model is used to verify that the proposed corrections are reasonable, and these corrections are then used for fine-tuning the policy. Concretely, rollouts are split into chunks and a VLM is prompted to identify chunks that contain mistakes and a goal that should have been reached. This procedure can be repeated iteratively. Experiments are primarily constructed on the MineRL domain, which contains demonstration data as well as a simulator. RLBench is also considered. Experiments show that the method achieves comparable performance to a longer PPO baseline and outperforms a DREAMER baseline on Minecraft, and is reasonable on RLBench.
Critical strengths
The paper is written cleanly and the method is proposed well. The intuition to make use of a learned prior in both the correction VLM and also the world model makes sense and is a strong way to achieve sample efficient improvement.
Experimentally, there is a fair comparison against a recent strong baseline in the domain (Dreamer) as well as a naive RL method that uses more compute. Error bars and analysis of the method are provided. Ablations show that repeated correction during online learning is additionally beneficial.
Critical weaknesses
In general this method does make the assumption that a reasonable strong correction-proposal VLM as well as an accurate world model are present. This makes it hard to attribute whether the performance gains come from a proper algorithmic procedure, or rather come largely by means of expert supervision in a roundabout way.
Additionally, the idea of planning in world model space is not new, e.g. Dreamer studied this technique in the same domain setting.
Questions for authors
Is it possible to apply ReVision-E/W in the RLBench setting?
3 comments
Thank you for the detailed response. As I have already recommended acceptance, I will keep my score.
Thank you for your follow-up, we sincerely appreciate your feedback and positive assessment of our work.
We sincerely thank you for your thoughtful comments and expert feedback. We appreciate your careful evaluation of our work, and provide some further clarifications and results below
We agree that ReVision relies on the ability of the generative models to propose useful corrections and predict their consequences, similar to DAgger and other corrective imitation learning methods relying on experts. Through the online learning procedure, we attempt to improve its search and robustness capabilities in generating corrections. Our experiments also show benefits beyond distilling the pretrained teacher alone.
Section 4.2 lists ablations that hold the teacher fixed while comparing online correction against additional BC, VLM demonstration distillation, and frozen-policy correction. Full ReVision-E achieves the strongest performance, showing that the correction loop does provide additional performance gain. Additionally, section 4.3 further evaluates world model dependence by comparing ReVision-E, ReVision, and ReVision-W, as well as different world-model sizes and VLM correctors. Generally, the experiments show that correction generation may require capable correctors and simulators, but shows additional benefit beyond regular expert supervision. We will clarify that teacher quality and world-model reliability remain important limitations.
We agree that the novelty of ReVision is not planning inside a world model, as studied by Dreamer and earlier model-based RL methods. Our main algorithmic contribution is the full automation of a corrective training loop like DAgger. The world model is necessary for implementing the search/branching components for irreversible or physical environments.
Additionally, Dreamer primarily uses imagined trajectories to optimize reward-based policy objectives. In ReVision, the world model is not used to directly provide predicted return/credit assignment supervision as in reward-based methods, and simply produces the frames and state for candidate corrective trajectories. Hence, we view it as correction-based planning in world models, rather than inference-time planning or reward-based RL in world models.
Yes, all ReVision variants can be run on RLBench. We run additional experiments to report ReVision-E and ReVision-W performance on RLBench, with results provided in Table 2. We note that performance is within expected variance.
Thank you for the insightful comments. We hope these new results and clarifications help resolve most of your concerns.