acceptodds
Under review as a conference paper at ICLR 2027

VETO: Reward-Supervised World Models as Action Verifiers for Sample-Efficient RL

12 views

Abstract

Fine-tuning pre-trained manipulation policies with reinforcement learning (RL) is a promising route to real-world adaptation, but its sample efficiency remains the limiting factor on real robots. A key source of wasted interactions comes from credit-assignment lag, i.e., RL policies repeatedly reattempting failure modes they have already observed because observed outcomes must be propagated through a slowly converging temporal-difference (TD) critic, and then through the policy, before behavior changes. Model-based RL methods are often used to improve sample efficiency since they rely on supervised learning, but they can introduce model-error bias into policy gradients. Our key insight is that action ranking need not rely on TD learning: a latent world model trained with *supervised reward prediction* can provide useful rankings with fewer transitions than TD-trained -functions. We introduce Verifying Exploratory Trajectories Online (), which uses a continually learning, lightweight world model as an online action verifier. The policy proposes candidate action chunks, and the verifier scores them by rolling out latent dynamics and predicting rewards to veto poor action candidates. Because policy and critic targets use real transitions rather than imagined rollouts, world-model errors do not enter the Bellman backup, allowing for asymptotic optimality. Across 7 tasks in 2 simulation suites and on real-world YAM and Franka platforms, adapts in fewer environment interactions than off-policy and model-based baselines and improves pre-trained policies in the real world.

Primary area:
reinforcement learning
Keywords:
adaptation, latent world model, real-world reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.50 over 2 reviews, above 99% of the others.
Reviewer_euoUOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Reinforcement learning (RL) post-training is beneficial for adapting large pre-trained policies to the real world, but it is sample-inefficient. One of the main sources of this inefficiency is lagged credit assignment: the policy repeatedly re-attempts actions whose outcomes have already been observed, because this information reaches the policy only slowly. To address this problem, the authors propose using a pre-trained and continually updated world model to filter action chunks before they are executed in the environment. To this end, a small world model with strong text and image encoders is trained to predict latent dynamics and rewards given the current observations and instructions. Reward supervision comes from large reward models from the literature or from the simulator’s dense rewards. During RL post-training, the world model scores the action-chunk candidates proposed by the RL policy based on their predicted rewards and keeps the most promising ones. One of the retained candidates is then executed in the environment, leading to better data collection. This reduces repeated low-reward attempts in the replay buffer and helps RL learn faster. As the experiments show, the proposed method, VETO, significantly improves recent RL post-training approaches, demonstrating strong applicability across different RL methods and tasks.

Critical strengths
  • The idea is very interesting, well explained, and well motivated.
  • Sampling uniformly among the top-K candidates scored by the world model is a nice design choice that provides robustness to verifier errors and noise. The experiments on K empirically show how helpful it is.
  • Both the online RL and the ranking-accuracy experiments are strong and well presented.
  • The ablations are informative; in particular, the ablation on the effect of explicit dynamics modeling is a valuable addition.
  • Overall, the idea is interesting and the paper is strong, with demonstrated practical usefulness. I believe it would be very beneficial for ongoing post-training efforts.
Critical weaknesses
  • The authors state that the world model is intentionally kept small to enable continual learning during post-training, but this choice is not ablated. An ablation showing the performance decrease and compute-efficiency gains relative to much larger world models, such as Cosmos-Predict2.5 or Cosmos 3, would be interesting.
  • Relatedly, the world model is continually updated during post-training. An ablation showing what happens if the model is kept frozen after pre-training would be very helpful.
Questions for authors
  • Did you evaluate larger world models within VETO? Could you show the performance gains and efficiency costs when a much larger world model is used instead of a small one?
  • Could you show what happens if the world model is frozen after pre-training, and how much this decreases performance?
3 comments
AuthorsOpenReview ↗

Thank you for your review!

We thank the reviewer for their thoughtful comments. Here are the responses to their questions.

Show performance decrease and compute efficiency gains relative to much larger WMs like Cosmos-Predict2.5/3?

Great point – we now directly compare compute efficiency against larger WMs like Cosmos-Predict2.5 – see general comment #4.

We are currently running additional experiments measuring sample-efficiency with larger world models for VETO’s backbone, but, due to compute requirements for these large world models, they likely will not finish before the rebuttal period ends.

Freeze the WM after pre-training - what happens then?

VETO’s ability to adapt is important. We have run a new experiment freezing the world model after pretraining, and it performs worse (Appendix Section F.4, Fig 21).  As the policy explores unseen state-action regions, a frozen world model cannot adapt its predictions to newly collected experience. Continual online updates allow VETO to maintain more accurate action rankings as the policy evolves.

Reviewer_euoUOpenReview ↗

Response to Rebuttal

Thanks to the authors for the response and the new experiments. The frozen world-model ablation fully addresses my second question. The latency comparison with Cosmos-Predict2.5 answers the efficiency half of my first question. I understand the sample-efficiency experiments with a larger backbone will not finish in time. Please include them in the final version, along with the cost of online updates for the larger model, since that is the stated reason for keeping the model small. I continue to see this as a clear accept and I am willing to support it in the discussion.

Please include them in the final version, along with the cost of online updates for the larger model, since that is the stated reason for keeping the model small.

We will definitely include the sample-efficiency experiments with a larger model in the final version; we're currently running Cosmos-backbone VETO experiments.

Thank you for this suggestion and for supporting the paper in the discussion!

Reviewer_bwTZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VETO uses a small latent-space world model with pure supervised learning and continuous online updates; it filters low-reward action chunks during the data collection phase. This enables reinforcement learning (RL) to obtain high-quality data at an earlier stage, thereby mitigating the lag in credit assignment; meanwhile, the policy and critic are only trained on real data, preventing model errors from being introduced into Bellman backups.

Critical strengths
  1. The insights are clear and verifiable. The division of labor of "using supervised learning for ranking and TD learning for value" is very natural. The result on PointMaze that "the exact match rate is 44% yet it achieves 87% of the oracle gain" intuitively illustrates why filtering works.

  2. The usage of the model is prudent. The world model only performs filtering and does not generate training data, which structurally prevents model errors from entering the learning objectives of the policy and critic. The Dyna baseline implemented with the same world model precisely puts "filtering with the model" and "generating data with the model" into a direct comparison.

  3. Lightweight and versatile. With only 4.5–12M trainable parameters, it supports continuous online retraining and can be directly applied on top of DSRL, TMRL, and iterative BC without any modifications to them.

Critical weaknesses

The weaknesses of this work can be summarized as follows:

  1. I think the ground-truth ranking for PointMaze is biased toward VETO: Figures 2,4, 10, and 15 all use the H-step block return as the ground-truth for ranking, which happens to be the training objective of VETO, whereas the TD critic estimates the full discounted return, so this set of comparisons inherently favors VETO. The ranking experiments on LIBERO use the full Monte Carlo return, which is fairer, but there are only 5 candidates per state, no error bars, and no specification of how many states were evaluated.

  2. The validator is short-sighted, and the theory only covers H steps. The scoring only accumulates the predicted rewards for H steps, with no terminal value term; both propositions also target the H-step truncated return rather than the true objective of RL. Therefore, the performance relies on dense shaped rewards or progress-based reward models, while sparse rewards and long-horizon tasks have not been tested.

  3. The theoretical foundation is relatively weak. Proposition 1 only requires integrability, which is essentially an accounting identity. Proposition 2 requires that the probability of top-K exact matching reaches 1−δ; based on the paper’s own reported 44% consistency rate, this bound will be extremely loose. The claim of "asymptotic optimality preservation" is an argument rather than a theorem, and the filtering process itself alters the distribution of behavioral data.

  4. Some baselines have different starting points. Newt and FOWM start with almost 0 success rate on Robomimic, which is much lower than the pre-trained flow policy shared by other methods. This comparison mainly reflects the difference in initialization, rather than the role of the verifier.

  5. The amount of real-device evidence is small. Each condition was only run once, with 10–20 trials per evaluation point. If the endpoint for DROID was only evaluated 10 times as stated in the appendix, which is roughly 6/10 vs. 2/10, the p-value from a two-sided Fisher's exact test is approximately 0.17, which is not significant. The reward model used by DROID is an unpublished work in submission.

  6. The computational overhead is not quantified. For each decision step, 16 samples are drawn from the 3B–6B model, but the wall-clock time per step or per episode is not reported.

  7. I do not think that the "Credit assignment lag" was directly measured. The paper did not statistically count the number of times the strategy repeated the same failure, nor did it directly quantify the length of the lag; this motivational mechanism was inferred backwards from the final results.

Besides, there are multiple inconsistencies/typos in the text:

◦ Number of tasks: The abstract and experimental setup mention a total of 7 tasks, consisting of 3 Robomimic tasks and 4 LIBERO tasks; however, only 2 Robomimic tasks are presented in the results, and Section 4.2 also states "six tasks".

◦ Task Name: Write Object-9 for the main text of Iteration BC, and Spatial-9 for the figures and appendices.

◦ K value: Q-BoN states top-K= 5 in the main text of the appendix, while Table 2 indicates 16/4.

◦ Number of trials: The appendix states that DROID evaluates 10 rollouts per assessment, yet the vertical axis of Figure 9 is labeled as 20 trials.

◦ Model names: Flex-π and Flexi-π are used interchangeably.

Questions for authors

Please refer to the weaknesses above. Besides, the remaining questions are as follows:

  1. Can the ranking experiment of PointMaze use the full discounted return as the ground truth? Can the LIBERO ranking plot add error bars and the number of evaluation states?

  2. How does VETO perform on tasks with sparse binary rewards or longer time horizons? Would incorporating terminal value (e.g., the estimate from a critic) into the scoring help?

  3. Can Newt and FOWM be initialized from the same pre-training streaming strategy? How does Dyna perform at lower synthetic data ratios, such as 0.5 and 0.25?

  4. If the world model does not undergo offline pre-training, or if the critic of Q-BoN also uses the same offline data, how much of the gap between the two will remain?

  5. Can the real machine experiment provide the exact number of trials, confidence interval, repeated-run results, and the wall-clock overhead of each decision step?

4 comments
AuthorsOpenReview ↗

Thank you for the review!

We thank the reviewer for their thoughtful comments. Here are the responses to their questions.

Using H-step return as ground truth for ranking is biased towards VETO in Maze Analysis?

We agree – we’ve rerun experiments in Figures 2, 4, 9, and 17 using the full discounted return as the ground truth instead of H-step return. VETO still improves ground-truth action ranking significantly more quickly, as determined by full discounted return, than the TD Critic across all updated figures.

LIBERO Ranking Evaluation: add error bars? How many states are evaluated?

We have added error bars to the LIBERO ranking plot by running new experiments across 3 seeds, and VETO remains more accurate than Q-TD.

As stated in the caption of Fig. 6, we sampled 50 trajectories under the policy. Each trajectory was executed until the task succeeded or the maximum horizon was reached (220 for shorter and 520 for longer trajectories), after which we computed the Q-values for ranking. We’ve updated Appendix D.3 to mention the max task horizon.

The verifier only covers H-step returns; thus performance relies on progress-based reward models?

The reviewer is right in that VETO assumes access to reward models, which are consistently improving and demonstrating useful rewards for difficult real-world tasks. We’ve updated the “problem setting” paragraph to clearly state this assumption. We emphasize that VLM-based progress signals are increasingly used in robot learning, including reinforcement learning [1, 3, 6] **and behavior cloning **[2, 4, 5, 7], supporting the practical relevance of this assumption.

[1] Robometer: Scaling general-purpose robotic reward models via trajectory comparisons, Liang et al., R:SS 2026

[2] Topreward: Token probabilities as hidden zero-shot rewards for robotics, Chen et al., CoRL 2026

[3] Roboreward: General-purpose vision-language reward models for robotics, Lee et al., arXiv 2026

[4] Liv: Language-image representations and rewards for robotic control, Ma et al., ICML 2023

[5] Vision language models are in-context value learners, Ma et al., ICLR 2025

[6] D&A: Decoupling Description and Assessment Enables VLMs as Zero-Shot Robotic Reward Models, Memmel et al., arXiv 2026

[7] WARP-RM: A Warp-Augmented Relative Progress Reward Model for Data Curation, Wu et al., arXiv 2026

Asymptotic optimality preservation is an argument rather than a theorem?

The reviewer is correct – please refer to general comment #1.

Theory: the theory only covers H-step actions?

Please refer to general comment #2.

Theory: Prop 1 is an accounting identity, not a true theorem?

Proposition 1 is not meant to serve as a theorem, nor do we claim it as one. It sets up a regret decomposition: VETO’s filtering helps when expected return gain exceeds expected selection regret. This means that “VETO does not need to exactly rank top-K candidates the same as the oracle” to provide performance improvement (Section 3.4), which is verified in the updated Figure 4.

Theory: Prop 2 requires exact Top-K matches, but only happens 44% of the time?

We agree that Proposition 2’s bound is not tight: Appendix B explicitly states that Proposition 2 is a “Worst-case guarantee” to provide a lower bound, and we use the true top-K match of VETO vs. the oracle to construct this bound.

As stated in the above response, the revised Figure 4 (changed to full-return instead of H-step return) shows that 13% exact top-4 agreement still yields 75% of the oracle gain! We have updated Appendix B to clearly state that Proposition 2 serves purely as a worst-case bound.

Some baselines have different starting success rates (Newt, FOWM)?

This has been fixed with new experiments. See general response #3.

Some real-world results used 10 trials per task?

We performed new real-robot experiments with additional sampling points, more DROID evaluations (20 instead of 10), and an additional baseline (Q-BoN). We now perform 540 total real-world evals (3 baselines, 4-5 evaluation points per task, 2 tasks, 20 evals per task), clarified in Section 4.3.

The DROID reward model is unpublished work in submission?

Building on work currently in submission is acceptable and not prohibited by the ICLR guidelines. We would like to emphasize **that VETO is agnostic about the choice of reward model, as long as it is reasonably accurate. Hence, our experiments use rewards from 3 sources: ground truth (Robosuite in sim), Robometer, and the in-submission work. We used the in-submission reward model for DROID experiments because we found it to be better than Robometer.

Computational overhead is not quantified?

Good point – this is now fixed with new experiments; see general response #4.

AuthorsOpenReview ↗

Continuation of our responses.

“Credit assignment lag” was not directly measured?

Thank you for raising this point. We include new experiments in Section 4.4, titled “How does VETO directly affect credit-assignment lag?”:

We track whether each method corrects wrong actions, as deemed by optimal action ranking, after refitting on data that contain them. VETO corrects 92% of these mistakes after just 1 update iteration, 2.6× as many as Q-TD, and still corrects 1.3× more than Q-TD at convergence. VETO also repeats already-observed failures about 12× less often.

Inconsistencies/typos: number of tasks, task name, K-value for Q-BoN, number of trials for Droid, Flex-\pi versus Flexi-\pi

Thanks for pointing these out! We have fixed these inconsistencies. Additionally, the top-K for Q-BoN being different from VETO’s was a typo, and DROID has been updated with a new experiment with 20 trials per evaluation.

How does VETO perform on tasks with sparse binary rewards or longer time horizons?

VETO specifically assumes access to dense rewards (see response to "performance relies on progress-based reward models?”).

Meanwhile, VETO is effective for longer-horizon tasks: the real-world Flex-π\pi experiment times out after 3000 timesteps, and longer LIBERO tasks are 500+ timesteps.

Can Newt and FOWM be initialized from the same pre-training strategy?

Newt/FOWM’s worse initial performance was not due to the pre-training strategy – Please refer to general comment #3.

How does Dyna perform at lower synthetic data ratios, such as 0.5 and 0.25?

Thank you for this suggestion. Our new experiment in Figure 19 evaluates Dyna with synthetic sampling fractions of 0.5 and 0.25, compared to the default 0.95 (as described in the MBPO paper we based the implementation on). Both settings barely improve upon the default Dyna configuration but still underperform VETO.

If the world model does not undergo offline pre-training, or if the critic of Q-BoN also uses the same offline data, how much of the gap between the two will remain?

See General Comment #5

Reviewer_bwTZOpenReview ↗

That works

I thank the authors for their clarification. Most of my concerns are addressed. I believe these details can be integrated into the final version of the essay to provide stronger support. I raise my score and wish the authors good luck.

AuthorsOpenReview ↗

thank you for the prompt response

We thank the reviewer for raising the score and especially for the quick reply!

These details have all already been integrated into the updated PDF, with most changes highlighted in blue text.

Please let us know what other concerns we can address that are precluding a higher score. Thank you!

Comments on the paper

4 comments
AuthorsOpenReview ↗

[Rebuttal] General Comments

We thank the reviewers for their helpful feedback and detailed reviews! We’re happy the reviewers find the paper interesting and well-presented (euoU, bwTZ), the method useful for RL adaptation (ourH, euoU), and the experiments strong (ourH, euoU). In response to the reviews, we have added new experiments, updated baselines, and clarified parts of the paper.

Rebuttal experiments not included in the main paper, currently organized in a separate appendix section for easy reference, will be better integrated into existing sections upon acceptance. Our general response to shared reviewer concerns follows:

1. VETO’s theory does not guarantee asymptotic optimality? (bwTZ, ourH)

VETO “preserves asymptotic optimality” is indeed stronger than what we prove. We have removed the asymptotic optimality argument from the abstract and introduction, instead emphasizing the original claim that VETO’s verifier does not introduce model bias into policy gradients and thus performs better than other model-based approaches that simulate training transitions.

We empirically demonstrate this in Fig. 20a, with a new experiment showing that Dyna’s early modeling error corresponds with its lower performance. We have updated Section F.5 with this result and reference it in the introduction with our model bias claims.

2. Theory shows VETO only provides improvement over H-step actions? (bwTZ, ourH)

We’ve now added a clarification in Section 3.4 to remind readers that, although our experiments demonstrate substantial empirical gains over episodic success, even on long-horizon tasks, the theory specifically covers local action selection.

3. FOWM and Newt baselines have lower starting success rates? (bwTZ, ourH)

Newt uses a “constrained planning” hyperparameter that biases MPPI's initial action distribution toward the pretrained policy, which we initially turned off because it prevented the overall RL policy from improving.

We reran all experiments with Newt and FOWM with constrained planning turned on (Figure 16), and both methods now achieve higher initial success rates than TMRL/DSRL/VETO, but do not improve significantly during online RL. Thus, VETO still achieves higher final success rates.

Section 4.2 now includes a footnote citing these additional experiments. We thank the reviewers for pointing this out!

4. How much overhead does VETO’s world model add? Compared to large models like Cosmos? (bwTZ, ourH, euoU)

We have added new computational analysis in Section 4.4, Table 1. VETO adds little overhead: because it is a small, lightweight world model, it incurs only 13% time overhead over the default best-of-N (BoN) Q-guided selection on a real robot. Using a large model like Cosmos2.5 doubles action selection time, justifying the use of a latent world model. We conducted all tests on a stand-alone NVIDIA RTX 5090.

MethodPer-action chunk action sampling + action ranking timeTotal episode time (incl. hardware execution)
π0\pi_0 (1 action, no ranking)1.91 ± 0.19 s102.03 ± 0.81 s
BoN (16 actions queried against Q)2.24 ± 0.11 s119.10 ± 0.95 s
VETO (16 actions queried against small world model)2.55 ± 0.46 s135.17 ± 8.68 s
Cosmos2.5 Predict (16 actions queried against 2B param world model)5.83 ± 0.20 s271.00 ± 8.39 s

5. Train Q in Q-BoN and VETO WM with the same data? (bwTZ, ourH)

Our original experiments did not pre-train the Q function for Q-BoN on the offline data used for VETO training because we found it did not improve performance. To demonstrate this, new experiments in Fig 18 demonstrate that pre-training Q does not change Q-BoN’s performance, and VETO still far outperforms it in sample efficiency.

This is due in particular to the pretraining data being off-policy; the on-policy (s,a)(s, a) are still out-of-distribution (this is also observed in several other works, such as [1, 2, 3]).

[1] Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning? Perry Dong et al., arXiv 2026.

[2] Post-Training, Finetuning, and Steering: A Field Guide for Curating Training Data, Maria Attarian et al., arXiv 2026

[3] Conservative q-learning for offline reinforcement learning, Aviral Kumar et al., NeurIPS 2020.

AuthorsOpenReview ↗

Thank you for your review!

We thank the reviewer for their thoughtful comments. Here are the responses to their questions.

The asymptotic optimality claim is unsupported by the presented theory?

This claim has been fixed – Please refer to General Comment #1.

Short-horizon progress ranking can conflict with long-horizon success?

We agree this is a limitation of H-step scoring, and we have added it to the Limitations (Sec. 5). Overall, our experiments demonstrate VETO helps, even on longer-horizon tasks (3000 timesteps for the real-world Flex-π\pi task, 500+ timesteps for longer LIBERO tasks) compared against Q-BoN.

Inconsistencies between values of K for BoN action selection in VETO and Q-BoN?

All methods use K=4 (Table 2); the original statement of Q-BoN using K=5 in the text was a typo. We have fixed this in the revision.

Comparison between VETO-TMRL vs. Q-BoN-TMRL instead of Q-BoN-DSRL?

Thank you for the suggestion. We agree that Q-BoN-TMRL is a stronger and important baseline. We’ve now updated Figure 7 with the additional baseline: Q-BoN (TMRL) for the LIBERO suite of tasks. The method generally performs better than TMRL itself, but VETO-TMRL still outperforms Q-BoN (TMRL).

WM pretraining uses broader data?

Fixed with new experiments – please refer to general comment #5.

Direct-return and TD-ranking controls should receive comparable offline data?

Direct-return and TD-ranking analysis experiments all in fact receive identical offline data and training budgets.

End-to-end latency and robot waiting time during RL training?

New real-world experiments compare overall per-step and total experiment time in general response #4.

DROID warmup counts differ between main text and hyperparameter table?

Thank you for pointing out this typo – we have fixed it.

Iterative BC supervises training on unexecuted action suffixes?

Our LIBERO SFT experiments (Appendix D.5)  in fact do not supervise on unexecuted action suffixes – the original text was from an earlier experiment that was replaced by one which does not supervise on unexecuted suffixes. Appendix D.5 now states “The flow-matching loss covers only the executed prefix of each chunk.” Thank you for pointing out this writing issue.

Need to soften claim: the evidence supports improved early action-ranking but not the general necessity of dynamics prediction?

Our paper does not claim that dynamics prediction is necessary for VETO. However, Figures 2 and 9 show that explicitly modeling and rolling out dynamics improve early action ranking (see H-return vs. VETO). In fact, without dynamics prediction (i.e., H-return), action-ranking performance relative to Q-TD is very similar.

Local theory should not be used to imply unrestricted long-term guarantees?

This is fixed – Please refer to General Comment #1.

AuthorsOpenReview ↗

Summary of our Rebuttal Updates

Dear AC, Reviewers, SACs, and PCs,

For your convenience, we summarize the major revisions and additional experiments conducted in response to the reviews:

  1. Clarified theoretical claims: Removed the unsupported asymptotic-optimality claim, clarified the local scope of our theoretical guarantees and the reward-model assumption, and updated ranking evaluations to use full discounted returns.
  2. Strengthened baseline comparisons: Reran Newt/FOWM with constrained planning and added Q-BoN-TMRL, Q-BoN critic-pretraining, and Dyna synthetic-data fraction ablations. VETO retains its advantage during online adaptation.
  3. Added mechanistic analyses: Measured the relationship between Dyna's model-prediction error and task performance, demonstrated the importance of online world-model updates, and quantified credit-assignment improvements.
  4. Expanded real-world and efficiency evaluations: Increased real-robot evaluations to 540, added Q-BoN comparisons, and measured inference latency.

We substantially strengthened the empirical evaluation with 10 new/rerun experimental studies, including stronger baselines, model-bias diagnostics, ablations, and expanded real-world and computational evaluations.

We thank the reviewers for their constructive feedback and hope these additional results and clarifications address their concerns.

AuthorsOpenReview ↗

Response request for VETO Paper Rebuttal

Hi reviewer ourH -- given that there is 1 day left in the rebuttal period, could you please let us know if our rebuttal addressed your concerns, and if so, consider raising your score?

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.