acceptodds
Withdrawn from ICLR 2027

World-Gymnast: Training Robots with Reinforcement Learning in a World Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Robot learning is bottlenecked by costly real-world interaction. Supervised finetuning (SFT) is limited by expert demonstration coverage, while reinforcement learning (RL) in software simulators faces a sim-to-real gap in manipulation. We ask whether training policies in world models learned from real-world video-action data can improve real-robot performance more effectively than either alternative. We propose World-Gymnast, a framework for improving vision-language-action (VLA) policies through reinforcement learning in a pretrained world model. Given an initial image and a language instruction, the policy learns through imagined interactions, with a zero-shot vision-language model (VLM) judging task completion. We show that a single policy trained across multiple tasks in this way improves performance on real-robot tasks and environments held out from RL training. On the Bridge robot setup, World-Gymnast achieves up to 18× the success rate of supervised fine-tuning and 2× that of RL in a software simulator. More importantly, World-Gymnast demonstrates intriguing capabilities of RL with a world model, including training on diverse language instructions and novel scenes from the world model, test-time training in a novel scene, and online iterative world model and policy improvement. Our results suggest learning a world model and training robot policies in the cloud could be the key to bridging the gap between robots that work in demonstrations and robots that can work in anyone’s household.

Primary area:
reinforcement learning
Keywords:
World Models, Model-Based Reinforcement Learning, Vision-Language-Action Models, Robot Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oETWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes World-Gymnast, which fine-tunes an OpenVLA-OFT policy inside a pretrained action-conditioned video world model (WorldGym). It uses GRPO, with binary task-completion rewards from GPT-4o. Each training run starts from a single frame plus a language instruction. The authors also try four extensions:

  • adding distractors to frames with an image editor
  • training on new instructions proposed by a VLM
  • test-time RL starting from the evaluation frames
  • a Dyna-style loop that fine-tunes the world model on real AutoEval rollouts

Real-robot evaluation covers four Bridge tasks in AutoEval. Baselines are SFT, iterative SFT, RL in SIMPLER, and adapted versions of WMPO and VLA-RFT.

Critical strengths
  • Real-robot results come from AutoEval, a third-party platform, not an in-house setup. Outside groups can check the numbers.
  • The SIMPLER baseline is trained on digital twins of the evaluation scenes (Table 1, Appendix B.2). This directly tests whether a learned world model can replace a physics simulator on the same tasks.
  • The world-model scaling experiment (Section 4.6) evaluates policies in a separate 7.5B model, not the model used for training. This avoids grading a policy with the same model it was trained in.
Critical weaknesses
  1. Little new method. Every component comes from prior work:

    • The environment and VLM-judge pipeline are WorldGym's policy-evaluation protocol (Quevedo et al., 2026; Appendix D.1).
    • The optimizer is GRPO, with the SimpleVLA-RL changes adopted as-is (Section 3.1).
    • The iterative loop follows Dyna (Sutton, 1991).

    The only difference from WMPO, VLA-RFT, and WoVR that the paper claims (Section 5) is a zero-shot VLM reward instead of a trained classifier or an expert-matching reward. In practice, that means swapping WorldGym's evaluation step for a policy-gradient update. Sections 3.2 to 3.4 describe new training data setups (new frames, new instructions, edited images), not new algorithms. A systems paper could still stand on strong results, but see points 2 and 3.

  2. The evaluation tasks are in-distribution for every pretrained component. The four AutoEval tasks come from the Bridge V2 distribution (Appendix C.5). The base policy is fine-tuned on Bridge V2. WorldGym is pretrained on Open X-Embodiment, which includes Bridge. The abstract calls these "tasks and environments held out from RL training." That is true only for the RL stage, not for the SFT data or the world-model pretraining. Nothing is evaluated outside this distribution, so the claims about household deployment (Abstract, Section 3.2) are unsupported.

Questions for authors
  1. How does unmodified OpenVLA-OFT (with proprioception, wrist camera, and L1 head) do on the same four AutoEval tasks? Does RL still help when starting from that stronger policy?
  2. What is GPT-4o's false-positive rate on WorldGym-generated rollouts, measured against human labels of those same videos?
  3. Were the Distract, Language, and Scaled variants run on the real robot? If not, can they be evaluated on AutoEval?
  4. What does the Dyna-style loop achieve on the other three AutoEval tasks?
  5. Can WMPO be rerun with a reward model that at least beats a constant guess? Alternatively, can Table 3 be presented explicitly as a reward ablation?
  6. What is the variance across RL training seeds for the main results in Table 2?
Reviewer_hzYQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper trains a VLA policy (OpenVLA-OFT) with RL entirely inside a pretrained video world model. A frame and an instruction are enough to start a training episode, and GPT-4o watches the imagined rollout and says whether the task succeeded. One policy is trained on 17 Bridge tasks and then tested on four real-robot tasks on AutoEval that it never saw during RL. It does clearly better than SFT, RL in the SIMPLER simulator, and two earlier world-model RL methods on three of the four tasks. The paper also explores what this setup makes easy: adding distractors by image editing, writing new instructions, adding tasks, adapting at test time, and refreshing the world model with real data.

Critical strengths

The question is the right one. Everyone says world models should be useful for training robots, but there is not much evidence that RL inside a general-purpose video model actually transfers to a real robot. This paper gives some, on a public third-party platform, with tasks held out from RL training. The jump on the two eggplant tasks (from under 10% to over 70%) is big enough that I don't think it is noise.

The setup is also attractive in a practical way. Removing the trained reward model and the expert references makes the pipeline much lighter than WMPO or VLA-RFT. And the observation that "anything you can describe with a frame and a sentence becomes a training task" is a nice way to frame why this matters compared to hand-built simulators. The SIMPLER comparison makes that point well: the simulator policy was trained on digital twins of the test tasks and still transfers worse.

The paper is easy to read. The structure is clear, the figures help, and the limitations section is honest about reward reliability and hacking.

Critical weaknesses

My main concern is that the experiments are too narrow for the claims the paper makes. The paper is pitched as evidence that RL in a world model is a general way to improve generalist robot policies. The real-robot evidence, though, is four tasks on one platform with one robot. Only three of those tasks show a gain, and two of those three are the same eggplant pick-and-place in slightly different scenes. Everything else is held fixed: one base policy, one world model, one reward model. So I can't tell which parts of the result come from the method and which come from this particular combination. I would want to see at least some of the following: a second policy family, a second robot or scene set, or real-robot results on more of the 17 training tasks. Without them the headline result is promising but thin.

The rest of the paper doesn't widen that base much. The distractor, language and scaled-task experiments are scored only inside the world model, by a VLM judge, and the gains over plain World-Gymnast are a few points. Test-time training and the Dyna loop are shown on close drawer only, and the paper notes that test-time training hurts the other tasks. These feel more like demos than results, yet the abstract and conclusion present them as capabilities of the method. Nothing in the paper connects WorldGym scores to real-robot scores, so I don't know how much weight to give the world-model-only numbers.

There is also no ablation of the method itself. The paper never isolates what matters: the zero-shot judge vs. a trained one under a fair setup, world-model size on the real robot, or the SimpleVLA-RL tricks. The reward is validated only on real videos, never on the generated rollouts it actually scores during training, where reward hacking would show up. Finally, the WMPO-style baseline uses a reward classifier that looks weak (Table 5), so beating it says little.

The idea of RL inside a learned video simulator is not new. Given that, the paper has to make its case with breadth and analysis, and right now it has neither.

Questions for authors
  1. Can you report real-robot results beyond these four tasks? This could be more AutoEval-style scenes, or real rollouts on some of the 17 training tasks. A broader real-robot table is the single thing that would most change my score.
  2. Does the gain hold with a different base policy, for example π0 or another VLA? If it only works for OpenVLA-OFT with the modified action head, the claim gets a lot narrower.
  3. Which component actually drives the gain? I'd like an ablation that swaps the reward (zero-shot VLM vs. a properly trained classifier) and the world model size while keeping everything else fixed, measured on the real robot.
  4. Do WorldGym success rates rank your policies the same way AutoEval does? If they do, the world-model-only experiments become much more meaningful.
Reviewer_xmqAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes World-Gymnast, which performs RL post-training of a VLA policy (OpenVLA-OFT) inside a pretrained video world model (WorldGym). A zero-shot VLM (GPT-4o) provides binary task-completion rewards on imagined rollouts, and GRPO optimizes the policy. The authors also explore distractor / language augmentation, test-time policy adaptation, and Dyna-style iterative world-model updating. On four real-robot AutoEval tasks it reports 58-78% success vs. 40/62/4/8% for SFT and 34/74/32/40% for SIMPLER RL, including an 18x gain on Eggplant -> Sink (4% -> 72%).

Critical strengths
  1. Meaningful problem with encouraging real-robot results: RL without real-world interaction or hand-built simulators, and the two pick-and-place tasks show large genuine gains (4% -> 72%, 8% -> 78%).
  2. Broad coverage of an emerging recipe: distractor augmentation, language augmentation, scaled task sets, world-model scaling, test-time adaptation, and Dyna-style co-update are all at least probed.
  3. Controlled comparison against WMPO and VLA-RFT with matched world model, policy init, rollout budget, and GRPO implementation, plus a reward-model quality analysis (GPT-4o F1 0.913 vs. WMPO 0.601).
Critical weaknesses
  1. Limited methodological novelty relative to existing world-model-based RL. The framework integrates existing components: WorldGym (world model), OpenVLA-OFT (policy), GPT-4o (zero-shot reward), and GRPO. WMPO (Zhu et al., ICLR 2026) and VLA-RFT (Li et al., 2025, arXiv:2510.00406) already do world-model-based VLA policy optimization with imagined rollouts; zero-shot VLM rewards and language-conditioned multi-task RL also have clear precedents (Rocamonde et al., ICLR 2024, arXiv:2310.12921; RL-VLM-F, Wang et al., ICML 2024, arXiv:2402.03681; ReWiND, Zhang et al., CoRL 2025, arXiv:2505.10911). The Table 3 comparison controls the world model and optimizer, so it isolates the reward formulation, not a new learning mechanism. More generally, neither the zero-shot VLM reward nor the multi-task language conditioning is presented with evidence that it departs from what is now common practice in this space.
  2. Broader generalization claims are evaluated mostly inside the world model itself. Distractor, language-augmentation, and scaled-task gains (Table 4) are measured by WorldGym rollouts + VLM reward — the same class of model used for RL training — with no corresponding real-robot transfer results. Test-time training improves one task (62% -> 100%) but degrades others; Dyna-style iteration is demonstrated on a single real task.
  3. Real-world evaluation is narrow: 4 tasks, 2 scenes, 1 embodiment (WidowX / Bridge setting). Sufficient for a proof of concept, not for the generality claims in the framing.
  4. The reward reliability analysis uses human-labeled real-world trajectories. VLM reward accuracy on the world-model-generated rollouts actually used for RL is never independently validated, leaving open whether reward errors on imagined videos bias policy learning.
Questions for authors
  1. Relative to WMPO and VLA-RFT, what is the primary methodological novelty? How much of the Table 3 gain is attributable to the stronger zero-shot VLM reward rather than to a new training mechanism?
  2. Are there real-robot evaluations for the distractor, novel-language, and scaled-task variants, or are those gains only observed inside WorldGym?
  3. How accurate is the VLM reward on world-model-generated rollouts (especially after RL shifts the policy off-distribution), and how sensitive are results to reward-model errors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.