- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Reinforcement learning (RL) post-training is beneficial for adapting large pre-trained policies to the real world, but it is sample-inefficient. One of the main sources of this inefficiency is lagged credit assignment: the policy repeatedly re-attempts actions whose outcomes have already been observed, because this information reaches the policy only slowly. To address this problem, the authors propose using a pre-trained and continually updated world model to filter action chunks before they are executed in the environment. To this end, a small world model with strong text and image encoders is trained to predict latent dynamics and rewards given the current observations and instructions. Reward supervision comes from large reward models from the literature or from the simulator’s dense rewards. During RL post-training, the world model scores the action-chunk candidates proposed by the RL policy based on their predicted rewards and keeps the most promising ones. One of the retained candidates is then executed in the environment, leading to better data collection. This reduces repeated low-reward attempts in the replay buffer and helps RL learn faster. As the experiments show, the proposed method, VETO, significantly improves recent RL post-training approaches, demonstrating strong applicability across different RL methods and tasks.
Critical strengths
- The idea is very interesting, well explained, and well motivated.
- Sampling uniformly among the top-K candidates scored by the world model is a nice design choice that provides robustness to verifier errors and noise. The experiments on K empirically show how helpful it is.
- Both the online RL and the ranking-accuracy experiments are strong and well presented.
- The ablations are informative; in particular, the ablation on the effect of explicit dynamics modeling is a valuable addition.
- Overall, the idea is interesting and the paper is strong, with demonstrated practical usefulness. I believe it would be very beneficial for ongoing post-training efforts.
Critical weaknesses
- The authors state that the world model is intentionally kept small to enable continual learning during post-training, but this choice is not ablated. An ablation showing the performance decrease and compute-efficiency gains relative to much larger world models, such as Cosmos-Predict2.5 or Cosmos 3, would be interesting.
- Relatedly, the world model is continually updated during post-training. An ablation showing what happens if the model is kept frozen after pre-training would be very helpful.
Questions for authors
- Did you evaluate larger world models within VETO? Could you show the performance gains and efficiency costs when a much larger world model is used instead of a small one?
- Could you show what happens if the world model is frozen after pre-training, and how much this decreases performance?
3 comments
Response to Rebuttal
Thanks to the authors for the response and the new experiments. The frozen world-model ablation fully addresses my second question. The latency comparison with Cosmos-Predict2.5 answers the efficiency half of my first question. I understand the sample-efficiency experiments with a larger backbone will not finish in time. Please include them in the final version, along with the cost of online updates for the larger model, since that is the stated reason for keeping the model small. I continue to see this as a clear accept and I am willing to support it in the discussion.
Please include them in the final version, along with the cost of online updates for the larger model, since that is the stated reason for keeping the model small.
We will definitely include the sample-efficiency experiments with a larger model in the final version; we're currently running Cosmos-backbone VETO experiments.
Thank you for this suggestion and for supporting the paper in the discussion!
Thank you for your review!
We thank the reviewer for their thoughtful comments. Here are the responses to their questions.
Show performance decrease and compute efficiency gains relative to much larger WMs like Cosmos-Predict2.5/3?
Great point – we now directly compare compute efficiency against larger WMs like Cosmos-Predict2.5 – see general comment #4.
We are currently running additional experiments measuring sample-efficiency with larger world models for VETO’s backbone, but, due to compute requirements for these large world models, they likely will not finish before the rebuttal period ends.
Freeze the WM after pre-training - what happens then?
VETO’s ability to adapt is important. We have run a new experiment freezing the world model after pretraining, and it performs worse (Appendix Section F.4, Fig 21). As the policy explores unseen state-action regions, a frozen world model cannot adapt its predictions to newly collected experience. Continual online updates allow VETO to maintain more accurate action rankings as the policy evolves.