acceptodds
Under review as a conference paper at ICLR 2027

Generative Models as Online Policy Correctors

Abstract

How can an agent learn efficiently online? Reward-based reinforcement learning can require costly exploration to discover better behaviors in open-world environments with sparse rewards. We propose an alternative in which an agent learns not only by reinforcing successful behavior but also by correcting mistakes in imagination online. We introduce \method, a learning algorithm that fully closes the loop in online corrective learning using pretrained generative models as online correctors. Given a policy rollout, \method identifies failures and generates counterfactual rollout continuations that correct them. The policy is then finetuned on these corrected rollouts and redeployed in the environment to collect new experience, repeating this cycle online. On Minecraft ObtainDiamond, \method substantially improves both behavior-cloned and vision-language-action policies. By applying pretrained priors at the learner's own failures, ReVision achieves returns comparable to PPO using 5 times fewer online environment steps (6M versus 30M). Controlled comparisons show that repeatedly correcting the updated learner yields greater improvements than distilling teacher demonstrations or correcting a frozen learner. Further experiments on manipulation tasks like RLBench also demonstrate improvements in pretrained manipulation policies.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reinforcement learning, robotics, planning, policy learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bNYhOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper describes a method for improving the sample efficiency of an RL framework. The main idea is to make use of a learned world model, along with a VLM that can identify failures and propose corrective actions. The world model is used to verify that the proposed corrections are reasonable, and these corrections are then used for fine-tuning the policy. Concretely, rollouts are split into chunks and a VLM is prompted to identify chunks that contain mistakes and a goal that should have been reached. This procedure can be repeated iteratively. Experiments are primarily constructed on the MineRL domain, which contains demonstration data as well as a simulator. RLBench is also considered. Experiments show that the method achieves comparable performance to a longer PPO baseline and outperforms a DREAMER baseline on Minecraft, and is reasonable on RLBench.

Critical strengths

The paper is written cleanly and the method is proposed well. The intuition to make use of a learned prior in both the correction VLM and also the world model makes sense and is a strong way to achieve sample efficient improvement.

Experimentally, there is a fair comparison against a recent strong baseline in the domain (Dreamer) as well as a naive RL method that uses more compute. Error bars and analysis of the method are provided. Ablations show that repeated correction during online learning is additionally beneficial.

Critical weaknesses

In general this method does make the assumption that a reasonable strong correction-proposal VLM as well as an accurate world model are present. This makes it hard to attribute whether the performance gains come from a proper algorithmic procedure, or rather come largely by means of expert supervision in a roundabout way.

Additionally, the idea of planning in world model space is not new, e.g. Dreamer studied this technique in the same domain setting.

Questions for authors

Is it possible to apply ReVision-E/W in the RLBench setting?

3 comments

We sincerely thank you for your thoughtful comments and expert feedback. We appreciate your careful evaluation of our work, and provide some further clarifications and results below

1. Dependence on a strong VLM and accurate world model

We agree that ReVision relies on the ability of the generative models to propose useful corrections and predict their consequences, similar to DAgger and other corrective imitation learning methods relying on experts. Through the online learning procedure, we attempt to improve its search and robustness capabilities in generating corrections. Our experiments also show benefits beyond distilling the pretrained teacher alone.

Section 4.2 lists ablations that hold the teacher fixed while comparing online correction against additional BC, VLM demonstration distillation, and frozen-policy correction. Full ReVision-E achieves the strongest performance, showing that the correction loop does provide additional performance gain. Additionally, section 4.3 further evaluates world model dependence by comparing ReVision-E, ReVision, and ReVision-W, as well as different world-model sizes and VLM correctors. Generally, the experiments show that correction generation may require capable correctors and simulators, but shows additional benefit beyond regular expert supervision. We will clarify that teacher quality and world-model reliability remain important limitations.

2. Planning in world-model space is not new

We agree that the novelty of ReVision is not planning inside a world model, as studied by Dreamer and earlier model-based RL methods. Our main algorithmic contribution is the full automation of a corrective training loop like DAgger. The world model is necessary for implementing the search/branching components for irreversible or physical environments.

Additionally, Dreamer primarily uses imagined trajectories to optimize reward-based policy objectives. In ReVision, the world model is not used to directly provide predicted return/credit assignment supervision as in reward-based methods, and simply produces the frames and state for candidate corrective trajectories. Hence, we view it as correction-based planning in world models, rather than inference-time planning or reward-based RL in world models.

3. Can ReVision-E and ReVision-W be applied to RLBench?

Yes, all ReVision variants can be run on RLBench. We run additional experiments to report ReVision-E and ReVision-W performance on RLBench, with results provided in Table 2. We note that performance is within expected variance.

RLBench Results (Success Rate %, Mean ± Std)

MethodInsert PegPlace CupsSort ShapeStack BlocksStack CupsAvg.
RVT-2 (base)40.0 ± 0.038.0 ± 4.535.0 ± 7.180.0 ± 2.869.0 ± 5.952.4
RVT-2 + ReVision-E44.8 ± 3.045.6 ± 6.547.2 ± 6.487.2 ± 4.779.2 ± 3.060.8
RVT-2 + ReVision43.2 ± 3.046.4 ± 7.044.8 ± 8.286.4 ± 2.078.4 ± 2.059.8
RVT-2 + ReVision-W44.0 ± 5.747.2 ± 5.343.2 ± 4.785.6 ± 4.876.0 ± 2.559.2
RVT-1 (base)11.2 ± 3.04.0 ± 2.536.0 ± 2.528.8 ± 3.926.4 ± 8.221.3
RACER9.6 ± 4.86.4 ± 4.125.6 ± 4.115.2 ± 3.041.6 ± 5.419.7
RVT-1 + ReVision-E21.6 ± 4.118.4 ± 5.429.6 ± 4.146.4 ± 8.237.6 ± 4.130.7
RVT-1 + ReVision20.0 ± 5.715.2 ± 3.932.0 ± 3.636.8 ± 3.934.4 ± 5.427.7
RVT-1 + ReVision-W20.0 ± 2.517.6 ± 4.140.0 ± 2.540.0 ± 7.235.2 ± 6.930.6

Thank you for the insightful comments. We hope these new results and clarifications help resolve most of your concerns.

Thank you for the detailed response. As I have already recommended acceptance, I will keep my score.

Thank you for your follow-up, we sincerely appreciate your feedback and positive assessment of our work.

Reviewer_dZ6WOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReVision, a DAgger-style training loop that replaces the human expert with an off-the-shelf VLM teacher (gpt/gemini), and uses a learned world model in place of environment resets. During the process, a VLM picks a failure segment and writes a local goal, a VLM corrector proposes primitive-level actions, the world model imagines the continuation from the failure state, and a VLM judge either accepts it or sends it back for a retry. Accepted corrections are mixed 1:1 with the original demonstrations for behavior cloning, and the loop repeats. On MineRL ObtainDiamond, ReVision raises a BC policy's mean return from 36.5 to 102.3 using 6M environment steps.

Critical strengths

A teacher that can't finish the whole task can still fix the learner's specific failures or say improve the students.

motivated for irreversible envs

Critical weaknesses
  • Using a world model to branch in irreversible envs is a well-established idea, applied to many data generation processes and many domains. So imo the main novelty rests only on the online, VLM-driven correction loop (tho a bit complex, and with frontier off the shelf models).
  • The world model's own contribution is left isolated:
    • All experiments are in simulators, where resets are available. No real-robot or truly irreversible digital experiment (e.g., web) to support the main motivation.
    • The deployable alternative is missing as a baseline: an HG-DAgger-style setup in which the VLM takes over in the real environment at the moment of failure, with no reset and no world model. A natural question is: whether branching in the world model beats this baseline approach.
    • The world model appears to stay fixed during training. Table 7 shows up to 20% optimistic errors, and a larger world model with better FVD doesn't improve return (Fig. 4b).
  • Much of the knowledge comes from the teacher, which makes the headline efficiency claim not apples-to-apples. The open-loop VLM with the same primitives already reaches 88.6 without any training, and ReVision's margin over it is still limited. Imo the contribution is better framed as efficiently distilling a strong teacher into a cheap policy, which is still useful.
Questions for authors

Please see the weaknesses above.

3 comments

We sincerely thank you for your thoughtful comments and expert feedback. We appreciate your careful evaluation of our work, and provide some further clarifications and results below

1. Whether branching in the world model beats an HG-DAgger style baseline with no reset

We've run additional experiments with an adaption of HG-DAgger using our VLM scorer and corrector, instead of a human expert. Compared with the ReVision variants, HG-DAgger helps test the benefit of retrospective failure scoring and correction search. Details are updated in the main results table and appendix.

After 6M steps, the HG-DAgger policy scores 77.4 ± 8.8 on BC and 80.2 ± 10.5 on VLA, compared with 102.3 ± 11.1 on BC and 110.5 ± 13.5 on VLA for ReVision. Qualitatively, we observe these limitations with the HG-DAgger formulation:

  1. Failure detection: Since the expert must intervene during rollout, it cannot use long-horizon outcomes. This introduces credit assignment problems common for long-horizon tasks, and without hindsight, HG-DAgger sometimes intervenes prematurely. For example, it progresses to stone collection too early, collecting approximately half as many logs and experiencing more wood-shortage failures (17 vs. 7). Also, once a failure is identified, the policy may already have moved beyond the state where an effective correction could be generated. In contrast, ReVision scores whole rollouts at once, which can mitigate this issue by giving the scorer access to outcomes.

  2. Correction quality and safe branching: The VLM or expert may not demonstrate a proper correction on the first try. Hence, accepting corrections without search or gating may allow lower-quality supervision to enter the training data. Additionally, a failed teacher intervention changes the real environment, affecting the rollout. In contrast, ReVision can evaluate and revise multiple candidate corrections from the same past state without affecting the rollout.

Supporting this, Table 3 also contains an ablation for search and acceptance gating. This ablation still retains retrospective scoring, and hence uses resets. Our comparisons show that this ablation decreases mean return from 91.6 to 60.5 after 500k updates, showing that search in correction generation can be effective compared to single pass correction generation.

2. Much of the knowledge comes from the teacher, which makes the headline efficiency claim not apples-to-apples…

We agree that ReVision leverages substantial pretrained knowledge. Our comparison intended to measure the efficiency of online environment interactions, rather than end-to-end compute efficiency under equal knowledge. We added modifications in section 4.1 to clarify, and report the associated training costs in the appendix.

We also agree that ReVision and DAgger can both be understood as a form of teacher distillation. However, correction-based learning can provide supervision that is online and directly on failure states, rather than regular demonstration distillation. Importantly, the performance gains are not explained by pretrained teacher knowledge alone. Table 3 ablates this with a fixed teacher, testing direct distillation (61.0), offline corrections (65.2), and online corrections (91.6) at matched steps.

Also, the use of short-horizon corrections can allow the teacher to provide useful local supervision without needing to reliably solve the full long-horizon task. The ReVision-trained BC policy slightly outperforms with 102.3 mean return compared to the open-loop VLM’s 88.6 baseline, without requiring VLM or world model inference during deployment.

3. Use of world models is an established idea, novelty rests on online VLM-driven correction loop

We agree that planning and branching in learned world models are established techniques, and our contribution is not necessarily the use of world models. Instead, we aim to fully automate a DAgger-style online corrective learning loop, while improving how corrective supervision is generated and refined. ReVision requires no human or scripted expert, allowing online corrections to be generated at scale. To effectively use an imperfect generative teacher, Revision retrospectively identifies failures and local goals, searches over alternative corrections, and gates successful corrections for policy training. The world model is used purely to allow state resets, which is necessary for search in correction generation for expensive or irreversible environments. ReVision thus removes human or scripted experts, while providing a more effective online learning procedure, as shown by our VLM-based HG-DAgger and fixed-teacher distillation comparisons. We provide some clarifications in the introduction section.

4. No real-robot or truly irreversible digital experiment

Yes, our experiments do not validate ReVision on a physical robot or another fully irreversible environment. Our comparison between ReVision-E and ReVision was made to examine the effects of the world model, providing a proxy for its feasibility for simulating real environments.

On those experiments, ReVision achieves mean return comparable to ReVision-E on Minecraft, showing that branching in a world model can maintain much of the benefit of corrections branched in the environment. However, we do not fully show a successful real-world deployment. We will modify the text to clarify this in the limitations section of the discussion.

5. Clarifications on the world model’s impact on training

Yes, our world model is frozen during ReVision training. Appendix A.2 evaluates 320 paired episodes across 16 tasks. As pointed out, we observe that optimism reaches 20% on some tasks, although the negative controls are correctly rejected. Hence, the predictions are not always fully reliable, which may explain the performance gap between ReVision-E and ReVision.

Additionally, we examine the lack of policy improvement from a larger world model with lower FVD. FVD measures overall video fidelity, whereas corrective learning relies on accurate prediction of dynamics and goal completion. Our world-model scaling study in Figure 4a shows that improved visual prediction quality does not necessarily translate to higher downstream policy return. We interpret this as evidence that better visual quality may not translate directly into better corrective supervision. Instead, high quality corrections may be the bottleneck. Improved dynamics accuracy, or a better corrector expert, may improve performance more. We provide clarification in appendix A2.

Thank you again for the valuable comments and suggestions. We hope these new results and clarifications help resolve most of your concerns.

Dear reviewer dZ6W, thank you again for taking the time to review and provide your initial feedback. If possible, we were wondering if you wouldn't mind taking a look at our rebuttal responses. We believe that our responses and paper revision help address the concerns raised in the initial review. In particular, we've clarified our main focus, added the HG-DAgger baseline with live intervention to showcase improved scoring/correction search with resets, and provided additional details on our use of the world model. Please let us know your thoughts, and we are very happy to answer any further questions.

Reviewer_XHcUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ReVision, a DAgger-like approach that uses a VLM to judge online policy rollouts and identify states deemed failures. The VLM then proposes corrections and interacts with a World Model to generate a trajectory, continuing until the trajectory is scored as acceptable and judged to have achieved the local goal. The resulting corrected trajectory is stored in a replay buffer for policy learning. The paper presents experiments on Minecraft ObtainDiamond, five RLBench tasks, and MetaWorld, reporting 5× the sample efficiency of PPO on Minecraft and higher performance across all evaluated environments.

Critical strengths

This paper presents a complete closed-loop design: a VLM judges rollouts and determines the local goal; a corrector then performs action correction and interacts with a world model; finally, a scorer determines from its score whether the corrected trajectory can be used for policy training. The method is well designed and of practical value: it offers a feasible solution for non-simulated settings where human expert correction is unrealizable. In addition, the paper's exposition is logical and its experimental design is persuasive; the ablation studies further demonstrate the necessity of the judge, local goals, and the acceptance gate.

Critical weaknesses
  1. The performance gains on RLBench are limited, and performance even drops slightly on the Sort Shape task; these limitations and their causes are not adequately explained by the experiments.

  2. The idea is similar to DAgger, but the paper does not provide a direct comparison with DAgger on these tasks.

  3. The paper claims higher efficiency in online environment interaction, but does not provide an analysis of the overall learning cost, such as the additional training data used by the world model and the extra inference time of the VLM.

Questions for authors
  1. The gains on RLBench are limited, and RVT-1 even degrades on the Sort Shape task. Could the authors provide a failure-case analysis to determine whether this reflects a limitation of the method itself or some other issue? In addition, would it be possible to try a stronger base policy on this task to verify that ReVision remains effective?

  2. There is no direct comparison with DAgger. In environments where an oracle corrector is feasible, such as MetaWorld, a direct comparison with DAgger could be provided to show the performance upper bound of DAgger in such settings, or to provide evidence that the proposed method is more sample-efficient than DAgger.

  3. The paper states that, compared with PPO, ReVision achieves comparable performance with fewer environment interactions. However, other cost analyses still need to be provided, such as the training data for the world model and the inference cost of the VLM. A more detailed comparison would better demonstrate the practical value of this method.

3 comments

We sincerely appreciate your expert feedback and thoughtful comments. We provide some clarifications to your questions with additional results below.

1. The performance gains on RLBench are limited…

Thank you for highlighting the uneven gains for RLBench tasks. We note that while RVT-1's success on Sort Shape decreases, we also evaluate ReVision with the stronger RVT-2 architecture in Table 2, where performance on Sort Shape improves from 35.0% to 44.8%. Averaged across all five tasks, ReVision improved for RVT-1 from 21.3 to 27.8 and RVT-2 from 52.4 to 59.7 mean success rate.

We conducted an additional failure case analysis to provide a qualitative angle. Common to corrective learning methods, we find that some performance drops may be caused by correction action distributions that substantially differ from actions the policy was originally trained on. On Sort Shape, VLM-generated corrections use smaller movements on average (33 mm versus 74 mm in the original demonstrations). After fine-tuning, RVT-1's zero-displacement predictions increase from 10% to approximately 60%, potentially causing the policy to stall more often. We note that while we use a replay buffer to help mitigate distribution shifts. This effect is task dependent, as ReVision improves Stack Blocks and other tasks, where the effect is less substantial. Additionally, RVT-2 does not exhibit the same degradation, so the degradation is not universal across base policies. We have included this analysis in Appendix C.1.

2. Direct comparison with DAgger

Thank you for the feedback. As you correctly mentioned, scripted oracles are available for MetaWorld. We additionally run a new experiment using the oracle in a more standard DAgger formulation with matched settings and budget. The DAgger baseline uses the oracle to provide correction labels at every state, in contrast to using a VLM to find failure states and generate correction continuations through search.

MetaWorld Results

MethodSoccerBasketballPush backPick-place
Pixel BC (200 demos)43.3 ± 0.757.3 ± 4.464.0 ± 4.075.3 ± 7.1
Pixel BC (250 demos)41.3 ± 4.465.3 ± 1.865.3 ± 4.182.7 ± 1.8
ReVision (VLM corrector)50.0 ± 2.068.0 ± 2.072.0 ± 3.180.7 ± 2.9
ReVision (Oracle corrector)50.7 ± 1.868.0 ± 2.371.3 ± 6.787.3 ± 4.1
DAgger (Oracle)46.7 ± 2.959.3 ± 7.568.0 ± 2.082.7 ± 1.8

At matched online environment steps, oracle ReVision achieves numerically higher success on all four tasks, while VLM ReVision does so on three tasks. We note that MetaWorld tasks have relatively short horizons, where scripted oracles can provide effective one-step action labels. Long-horizon tasks such as Minecraft may involve multi-step behaviors and delayed consequences, making effective one-step supervision more difficult to generate. The updated MetaWorld experiments are in section C.2 of the appendix.

Additionally, a core goal with ReVision beyond improving the generation of corrections is to automate the DAgger loop to scale corrective training with an imperfect generative teacher. For long-horizon environments such as Minecraft, getting expert action labels at the scale of the experiments needs immense human effort or an existing ground-truth expert. Hence, it is difficult to provide a true DAgger comparison at matched scales, and we instead compare against other reward-based methods. To provide a DAgger-style comparison, we’ve added an adapted version of HG-DAgger using our VLM corrector, shown in the main Minecraft results table.

3. Cost analysis and environment interaction efficiency

We agree that environment interaction efficiency should be distinguished from total computational cost. Our claimed 5× improvement is with regard to online environment interactions, not a 5× reduction in end-to-end compute. We have edited the text to make the distinction clearer.

We provide cost accounting in Appendix A.1 and Appendix B.3:

(1) Regarding online training costs, tables 14 and 15 report VLM usage and compute, while Table 15 compares costs for ReVision with PPO and Dreamer-style RL. Any online interaction with the world model and VLM latency is accounted for in the provided total GPU hours, since they extend the total runtime.

(2) Regarding offline training costs, Table 6 reports world-model training data, which shows the world model is pretrained on 46.5M VPT frames and fine-tuned on 8.1M task-specific frames. This training constitutes additional offline resources and are not included in the online environment training budget.

We sincerely appreciate your constructive feedback and hope that these additional results and clarifications address your concerns.

Thank you for the detailed rebuttal and the new experiments. The RVT-1 failure analysis on Sort Shape and the RVT-2 improvement on the same task explain the RLBench result well. The MetaWorld oracle-DAgger comparison and the VLM-based HG-DAgger baseline on Minecraft also give the direct DAgger comparison. My concerns have been addressed, and I will maintain my positive score.

Thank you for your feedback. We are happy that our responses/additional experiments were helpful, and we greatly appreciate your positive rating of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.