- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes World-Gymnast, which fine-tunes an OpenVLA-OFT policy inside a pretrained action-conditioned video world model (WorldGym). It uses GRPO, with binary task-completion rewards from GPT-4o. Each training run starts from a single frame plus a language instruction. The authors also try four extensions:
- adding distractors to frames with an image editor
- training on new instructions proposed by a VLM
- test-time RL starting from the evaluation frames
- a Dyna-style loop that fine-tunes the world model on real AutoEval rollouts
Real-robot evaluation covers four Bridge tasks in AutoEval. Baselines are SFT, iterative SFT, RL in SIMPLER, and adapted versions of WMPO and VLA-RFT.
Critical strengths
- Real-robot results come from AutoEval, a third-party platform, not an in-house setup. Outside groups can check the numbers.
- The SIMPLER baseline is trained on digital twins of the evaluation scenes (Table 1, Appendix B.2). This directly tests whether a learned world model can replace a physics simulator on the same tasks.
- The world-model scaling experiment (Section 4.6) evaluates policies in a separate 7.5B model, not the model used for training. This avoids grading a policy with the same model it was trained in.
Critical weaknesses
-
Little new method. Every component comes from prior work:
- The environment and VLM-judge pipeline are WorldGym's policy-evaluation protocol (Quevedo et al., 2026; Appendix D.1).
- The optimizer is GRPO, with the SimpleVLA-RL changes adopted as-is (Section 3.1).
- The iterative loop follows Dyna (Sutton, 1991).
The only difference from WMPO, VLA-RFT, and WoVR that the paper claims (Section 5) is a zero-shot VLM reward instead of a trained classifier or an expert-matching reward. In practice, that means swapping WorldGym's evaluation step for a policy-gradient update. Sections 3.2 to 3.4 describe new training data setups (new frames, new instructions, edited images), not new algorithms. A systems paper could still stand on strong results, but see points 2 and 3.
-
The evaluation tasks are in-distribution for every pretrained component. The four AutoEval tasks come from the Bridge V2 distribution (Appendix C.5). The base policy is fine-tuned on Bridge V2. WorldGym is pretrained on Open X-Embodiment, which includes Bridge. The abstract calls these "tasks and environments held out from RL training." That is true only for the RL stage, not for the SFT data or the world-model pretraining. Nothing is evaluated outside this distribution, so the claims about household deployment (Abstract, Section 3.2) are unsupported.
Questions for authors
- How does unmodified OpenVLA-OFT (with proprioception, wrist camera, and L1 head) do on the same four AutoEval tasks? Does RL still help when starting from that stronger policy?
- What is GPT-4o's false-positive rate on WorldGym-generated rollouts, measured against human labels of those same videos?
- Were the Distract, Language, and Scaled variants run on the real robot? If not, can they be evaluated on AutoEval?
- What does the Dyna-style loop achieve on the other three AutoEval tasks?
- Can WMPO be rerun with a reward model that at least beats a constant guess? Alternatively, can Table 3 be presented explicitly as a reward ablation?
- What is the variance across RL training seeds for the main results in Table 2?