acceptodds
Under review as a conference paper at ICLR 2027

GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation

14 views

Abstract

World-action models (WAMs) commonly adapt video generators pretrained on natural video, leaving open how their manipulation capability scales when they are pretrained from scratch on manipulation data. To study this question, we introduce Genie Envisioner Act 2.0 (GE-Act 2.0), a world-action model that combines a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM). CoAE learns compact representations for robot control, while the SVP and IDM are pretrained from scratch on manipulation data. Knowledge-aligned selective optimization (KASO) jointly trains the SVP and IDM using generated futures compatible with recorded actions. These components generate a 52-action chunk in 104 ms on a single RTX 5090, supporting real-time control at 30 Hz. We study zero-shot out-of-distribution (OOD) generalization without per-task fine-tuning on 100 real-robot tasks across 20 manipulation skill groups and two embodiments. Scaling co-training data from 300 to 30,000 hours substantially improves success rates on both G1-OP and G2-90D, despite the latter contributing less than 2% of the training mixture. Beyond total data scale, our skill-level coverage analysis reveals a strong correlation between skill-specific training hours and zero-shot OOD success. We further study fine-grained instruction following, achieving ≥90% Follow Score for object identity, color, shape, and position. On RoboTwin-Clean2Rand and GenieSim-Instruction, GE-Act 2.0 outperforms strong recent baselines including π0.5, further demonstrating its generalization ability. We will release pretrained checkpoints and code.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world-action model, robot manipulation, robot foundation model, robot learning, visual planning, inverse dynamics model, co-training, data scaling, zero-shot generalization, real-robot evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zP4WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GE-Act 2.0, a world action model that is trained from scratch instead of being initialized from a pretrained video diffusion model. Specifically, they train a new autoencoder called CoAE, a single-step visual planner (SVP; analogous to a video diffusion model), and an IDM (analogous to a policy/action predictor). For CoAE, they regularize the latent space to be aligned with pretrained visual representations; for SVP, they adopt MeanFlow to enable one-step generation; and for IDM, they predict actions from the current and predicted future frames. Thanks to the compact latents and one-step generation, they achieve real-time inference. They first independently train the IDM using GT video-action pairs and the SVP using GT videos, and then co-train both models by feeding generated rollouts to the IDM and pairing them with GT actions. However, generated rollouts sometimes do not match the GT actions, so the authors propose KASO, which selects the rollout that best matches the GT actions among multiple rollouts from the SVP. The selection is based on the IDM’s velocity differences (i.e., energy) between the GT recorded future and each generated candidate. By doing so, they mitigate this issue and show zero-shot OOD scaling and fine-grained instruction-following capabilities, as well as competitive benchmark performance.

Critical strengths
  • The paper is generally well-written and easy to follow.
  • Instead of initializing from the pretrained video diffusion models, the paper tries to do more comphrehensive study by training every model from scratch.
Critical weaknesses
  • While I know the terms “WAM” and “VAM” are used interchangeably these days, I think the term “WAM” may be misleading in this paper. Specifically, many WAMs jointly predict future video frames and corresponding actions during both training and inference, in which case the validity gap described in the paper may not arise. I think this issue mainly stems from the fact that the paper trains a video model and a decoupled action predictor. In this respect, the paper should clarify that the problem is specific to decoupled architectures, and ideally discuss or compare with joint-denoising approaches.
  • I think 10 trials per target is not enough, especially for the KASO ablation in Table 6, and more trials are needed to reduce the noise. For example, with 10 trials, a 40% success rate has a 95% confidence interval of roughly 17–69%, which makes the differences between methods hard to interpret.
  • For the scaling experiment in Table 1, I think the number of training samples seen should be identical across models, rather than training each model for one epoch. With the current setup, the number of iterations for the 0.3k h model is 100× smaller than for the 30k h model, so the effects of data scale and training compute are confounded.
Questions for authors
  • Could the authors provide real-world experiments with more trials to reduce the noise?
5 comments

Dear Reviewer zP4W,

We sincerely thank you for the careful and constructive review! Your comments pointed us to the places where the paper needed more evidence and clearer wording. Below, we address your concerns point by point.

W1. The term "WAM" and joint-denoising approaches. Thank you for this precise observation on terminology. You are right that many WAMs jointly predict future video frames and actions, and that the validity gap may not arise there. Following your suggestion, we added a clarification in Section 2.3 that the problem is specific to decoupled architectures.

On joint-denoising approaches: our related work (Appendix A) separates decoupled two-stage designs from joint designs. Joint-denoising models avoid the gap because video and action are denoised against the same recorded sample. In a decoupled architecture, the gap appears once the video model and the action predictor are co-trained: the action predictor receives a generated future but is supervised with the recorded action. This is the setting we study.

W2 and Q1. More trials to reduce the noise. Thank you for this helpful suggestion. We agree that 10 trials per target is not enough, especially for the KASO ablation in Table 6. We added real-world experiments with more trials to reduce the noise: a second full batch of the whole ablation under the same protocol. Together with the original trials, this gives 50 single-object trials and 20 trials per target in the four-object scene, which is 130 pick trials per method.

MethodSingle-objectSpongeRed appleMouseRemoteFour-objectPooled, 130 trials
E2E9/505/2014/203/201/2023/8024.6%
E2E+PT7/508/2010/201/202/2021/8021.5%
KASO22/5011/2012/205/203/2031/8040.8%

With the added trials, the differences between methods are now clear to interpret. The 95% interval for KASO narrows from the 17 to 69 you computed for 10 trials to 33 to 49. KASO is 19.3 points above E2E+PT and 16.2 points above E2E in pooled pick success, and both differences are significant (Cochran-Mantel-Haenszel test stratified by protocol, p = 0.0008 and p = 0.006).

W3. Data scale and training compute in Table 1. Thank you for looking so closely at our scaling protocol. Separating the effect of data from the effect of compute is exactly the right question to ask of a scaling study, and we are glad to explain our choice. On whether the number of training samples seen should be identical across models: we follow the common convention in data-scaling studies, where training length is set by the data and a larger dataset naturally receives more training steps. Large language models are trained in about a single pass over their data, so more data means more steps [1]. In robot learning, data-scaling studies likewise give larger datasets more training steps so that every size converges [2].

We train each model for one epoch because our internal evaluation showed that, at around one epoch, the model has converged and is well trained without overfitting. Making the number of samples seen identical would mean about 100 passes over the 0.3k-hour data. Repeating data that many times is known to bring no further gain and introduces overfitting at the current data scale [3]. We hope this explanation clarifies our choice of scaling setup.

Again, thank you very much for your suggestions for improvements! We would deeply appreciate it if you could let us know whether your concerns have been addressed properly.

References

[1] Hoffmann et al. Training Compute-Optimal Large Language Models. NeurIPS 2022.

[2] Lin et al. Data Scaling Laws in Imitation Learning for Robotic Manipulation. ICLR 2025.

[3] Muennighoff et al. Scaling Data-Constrained Language Models. NeurIPS 2023.

AuthorsOpenReview ↗

An Additional Zero-shot Comparison

Dear Reviewer zP4W,

In addition to the experiments in our first reply, we added an experiment that directly compares the zero-shot ability of GE-Act 2.0 with a VLA trained under matched conditions. We trained StarVLA [4] on the same 30,000 hours of co-training data as GE-Act 2.0, with the same Qwen3.5-2B vision-language backbone. Both models were evaluated zero-shot, without task-specific fine-tuning, on the same G1-OP robot in the same unseen environments. To cover every skill while keeping real-robot evaluation tractable during the rebuttal period, we evaluated both models on a subset of 47 of our 100 tasks, taking about two tasks from each skill category (all tasks where a category has two or fewer), so that all 20 skill groups are represented. This subset is not easier for GE-Act 2.0: its success on these 47 tasks (36.0%) is below its average over all 100 tasks (44.1%). Skill groups follow Table 1.

Skill group (tasks)StarVLAGE-Act 2.0Skill group (tasks)StarVLAGE-Act 2.0
Pick (5)54.076.0Close (1)70.090.0
Place (2)80.0100.0Separate (2)15.025.0
Sweep + Wipe (2)80.040.0Press (2)15.030.0
Stack (2)20.020.0Pull (3)0.013.3
Pass (2)80.085.0Hang (2)35.00.0
Fold (2)20.025.0Tear (2)0.05.0
Straighten (2)50.035.0Flip (3)0.030.0
Pour (2)0.045.0Zip (2)0.00.0
Push (4)27.530.0Shake/Stir (4)0.012.5
Insert (3)43.333.3All 47 tasks29.136.0

Over all 47 tasks, GE-Act 2.0 reaches 36.0% zero-shot success (95% CI 32 to 40) against 29.1% for StarVLA (25 to 33). The 6.8-point difference is significant under a Cochran-Mantel-Haenszel test stratified by skill group (p = 0.008). GE-Act 2.0 is higher on 13 compared groups and lower on 4. With 470 trials per model, this comparison also adds to the larger-scale real-robot evidence you asked for.

Again, thank you very much for your suggestions! We would deeply appreciate it if you could let us know whether these additional results address your concerns.

References

[4] StarVLA Community. StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing. arXiv:2604.05014, 2026.

Thanks for the rebuttal. I still have a concern in data scale and training compute: I think the authors should provide 100 passes over the 0.3k-hour data for better comparison. I think in recent imitation-earning-based polices in robotics behave quite differently from other domains like LLM that you mentioned, e.g. https://seohong.me/blog/behavioral-cloning-mystery/ . So I think the data scaling claim may be too strong.

Reviewer_1aHnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GE-Act 2.0, a world–action model pretrained from scratch for robotic manipulation. It combines a Control-Oriented Autoencoder (CoAE), a Single-step Visual Planner (SVP), and an Inverse Dynamics Model (IDM). The SVP predicts future visual states in one pass, while the IDM converts those predictions into actions. The authors also propose KASO, which selects generated futures that the IDM judges compatible with the demonstrated action, aiming to reduce mismatched visual and action supervision. The main result is a scaling study on 100 real-robot tasks across 20 skill groups. As co-training data increases, zero-shot OOD success rises, without task-specific fine-tuning.

Critical strengths
  1. The paper connects compact visual representations, fast future prediction, and action generation in a coherent architecture. In particular, one-step generation makes it practical to backpropagate the action loss through the predicted future and to pretrain the SVP and IDM separately on different kinds of data.

  2. A demonstration records one valid way to complete a task, while a generated future may depict another. Pairing that future with the demonstration’s action can therefore give the IDM inconsistent supervision. Selecting futures based on action compatibility is a sensible response to this problem.

Critical weaknesses
  1. The comparison to other approaches is not yet decisive. The strongest evidence is the paper’s own scaling curve. Its simulation comparisons involve benchmark-specific SFT, so they do not establish a direct real-robot advantage over competing WAMs or VLAs under matched conditions. A comparison using the same hardware, tasks, and data budget would make the case for GE-Act 2.0 much stronger.

  2. The source of the scaling gains is not fully isolated. More training hours also bring changes in data diversity, skill coverage, and source composition. The results show that scaling helps, but do not clearly separate the effect of sheer data volume from the effect of broader coverage or different data mixtures.

  3. The coverage analysis is suggestive, but not causal. A strong relationship between skill coverage and OOD success is useful evidence, but it does not show that coverage itself drives the gains. Task difficulty, data quality, and embodiment-specific factors could also contribute. A targeted experiment that adds data for selected low-coverage skills would help resolve this.

Questions for authors
  1. How much does KASO add beyond simply training on more generated futures? The ablation compares a few alignment strategies, but it would be useful to know how results change with different candidate counts and selection ratios, or with a simpler compatibility score. Is KASO’s gain robust to these choices, or does it depend on careful tuning?

  2. What is the inference-time cost of the complete policy?

  3. What is the advantage of a WAM in the Astra era? As real-time multimodal agents become more capable, it is not clear whether a WAM is a distinct policy paradigm, or better understood as a training and control architecture inside a more general agent. GE-Act 2.0 offers plausible reasons WAMs may still matter: it exposes an explicit predicted future, and its components can learn from action-free video as well as instruction-free robot trajectories. But the paper does not show that these features provide an advantage over a sufficiently capable real-time multimodal agent on the same manipulation tasks. It would help to clarify whether the intended long-term role of WAMs is better physical prediction, more data-efficient robot learning, more interpretable control, or some combination—and to evaluate at least one of those advantages directly.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1aHn (1/2)

Dear Reviewer 1aHn,

We sincerely thank you for the thoughtful and encouraging review! We are glad that you found the one-step interface and the action-compatible selection sensible. Below, we address your concerns point by point.

W1. A comparison using the same hardware, tasks, and data budget. Thank you for this suggestion. Following it, we added a real-robot comparison with StarVLA [1], which we trained on the 30,000 hours of training data as GE-Act 2.0. We chose StarVLA because it builds on Qwen3.5-2B, the same vision-language model that GE-Act 2.0 uses for understanding, so the two models share both their data and their language backbone. Both were evaluated zero-shot, without task-specific fine-tuning, on the same G1-OP robot in the same unseen environments and with the same instructions. To cover every skill while keeping real-robot evaluation tractable during rebuttal, we used a subset of 47 of our 100 tasks, taking about two tasks from each skill category so that all 20 skill groups are represented. This task subset does not favor GE-Act 2.0: its success on these 47 tasks (36.0%) is below its average over all 100 tasks (44.1%). Skill groups in the table follow Table 1.

Skill group (tasks)StarVLAGE-Act 2.0Skill group (tasks)StarVLAGE-Act 2.0
Pick (5)54.076.0Close (1)70.090.0
Place (2)80.0100.0Separate (2)15.025.0
Sweep/Wipe (2)80.040.0Press (2)15.030.0
Stack (2)20.020.0Pull (3)0.013.3
Pass (2)80.085.0Hang (2)35.00.0
Fold (2)20.025.0Tear (2)0.05.0
Straighten (2)50.035.0Flip (3)0.030.0
Pour (2)0.045.0Zip (2)0.00.0
Push (4)27.530.0Shake/Stir (4)0.012.5
Insert (3)43.333.3All 47 tasks29.136.0

GE-Act 2.0 achieves higher success on 13 groups and lower on 4. Over all 47 tasks, it reaches 6.8 points higher success than StarVLA.

W2. Separating data volume from coverage and data mixtures. Thank you for raising this. We designed the scaling ladder to keep the data mixture fixed. The four training pools are nested, and each smaller pool preserves the proportion of each data source and each embodiment in the larger ones as closely as possible. The gains therefore come from more data with the same source and embodiment mixture, not from a different mixture. Following your comment, we re-wrote the description in Section 3.1 to state explicitly that the proportion of every data source and every embodiment is kept fixed across the four pools.

W3. Whether coverage itself drives the gains. Thank you for this careful point. We agree that the coverage analysis shows an association, not a causal effect, and the paper presents it as such. A fully controlled test would add data for one low-coverage skill while holding everything else fixed, then retrain the model, and repeat this for each skill of interest. We see this targeted experiment as an important direction for future work.

References

[1] StarVLA Community. StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing. arXiv:2604.05014, 2026.

AuthorsOpenReview ↗

Response to Reviewer 1aHn (2/2)

Q1. Sensitivity of KASO to candidate count, selection ratio, and score. Thank you for this question. In internal runs we used 3, 4, 6, and 8 candidates, selecting one each time, so the selection ratio ranged from 1/3 to 1/8. Larger candidate groups were generally better, but the cost of generating candidates grows with the group size. Under our compute budget, we use 4 candidates as the balance between quality and cost, and keep this setting fixed in all experiments.

On the score: the compatibility score itself is lightweight. It needs no extra network or training, only one forward pass of the current IDM per candidate. A simpler visual score, such as latent distance to the recorded future, would reject futures that look different but support the same action, which is exactly what KASO is designed to keep.

Q2. Inference cost of the complete policy. Thank you for asking. The complete policy runs end to end, from raw three-camera observations to a 52-action chunk, in 104 ms on a single RTX 5090. This includes every component: CoAE, the frozen Qwen3.5-2B conditioner (2.72 B parameters), the SVP (2.51 B), and the IDM (0.56 B). It supports real-time control at 30 Hz (Section 2, Appendix E).

Q3. The role of WAMs in the era of multimodal robotic agents. Thank you for this thought-provoking question. GE-Act 2.0 was actually developed before robotic agents such as GPT-6 Astra were released, but we are really glad to share our thoughts!

We see WAMs as complementary to agentic multimodal models, not competing with them, because the two operate at different time scales. Agentic models, now approaching trillions of parameters, excel at understanding, reasoning, and declarative memory. A general robot, however, also has to respond to a changing environment within a fraction of a second, and an agent that can act only through explicit reasoning is like a person without a cerebellum: every movement has to pass through deliberate thought. Faster VLM models alone would not close this gap. Unlike a chatbot, which runs only when a user sends a message, a robot operates continuously, 24 hours a day, so routing every control decision through a trillion-parameter model would be too costly to sustain. A robot-native model can instead respond at control rate using edge devices like RTX 5090: as noted in Q2, GE-Act 2.0 produces each action chunk in 104 ms, fast enough for closed-loop control at 30 Hz.

Much as you suggest, we therefore see two ways the two can work together:

  • The agent calls the WAM as a skill. The agent decomposes a task into instructions such as "pick up the red apple", and the WAM executes each one zero-shot. This happens to be the setting our atomic skill evaluation tests on.
  • In a more ideal setup, both run concurrently at two rates. A fast model, such as a WAM, keeps running all the time, predicting the short-horizon future and controlling the robot continuously. It reacts on its own to fast changes in the world and to accidents, and is responsible for preventing harmful outcomes before they happen. The high-level agent stays idle until a signal or surprise wakes it, such as a task failure or a change of instruction. This is an emerging direction that the community and we are actively pursuing.

This division of labor is also why we think robot-native models deserve more investment than they receive today. Agentic VLMs currently attract most of the field's attention and resources, while WAMs and other robot-native alternatives receive a small fraction. Given comparable resources, we believe they could become both faster and more efficient (both compute-efficient and data-efficient).

Again, thank you very much for your suggestions for improvements! We are eager to hear your feedback, and we would deeply appreciate it if you could let us know whether your concerns have been addressed.

Reviewer_yNAjOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the pretraining and scaling properties of World-Action Models (WAMs) trained from scratch entirely on robotic manipulation data, rather than fine-tuning video generators pretrained on natural internet video. The authors present GE-Act 2.0 (Genie Envisioner Act 2.0), an integrated system combining three core components: (1) a Control-Oriented AutoEncoder (CoAE) that compresses multi-view visual frames with a 64x64 spatial downsampling factor into 512-channel compact latents while aligning representations with frozen SigLIP 2, V-JEPA 2.1, and DINOv3 features; (2) a Single-step Visual Planner (SVP) using a conditional MeanFlow Diffusion Transformer (DiT) that predicts future visual latents in a single forward pass; and (3) an Inverse Dynamics Model (IDM) that generates 52-action chunks from current and predicted latents. To bridge the "validity gap" during co-training—where generated visual futures depict valid but alternative trajectories incompatible with recorded demonstration actions—the authors propose Knowledge-Aligned Selective Optimization (KASO), which probes candidate futures in IDM action space and selects compatible futures for end-to-end updates. The system outputs 52 actions in 104 ms on a single RTX 5090, supporting closed-loop 30 Hz real-time control. The authors conduct a massive scaling study from 300 to 30,000 hours of co-training data and evaluate zero-shot out-of-distribution (OOD) generalization across 100 real-robot manipulation tasks spanning 20 skill groups and two distinct embodiments (G1-OP and G2-90D; over 4,000 real-robot rollouts). Scaling data yields massive overall gains (suite mean SR rising from 17.1% to 44.1% on G1-OP), and a strong log-linear correlation (r = 0.74) is identified between skill-specific training hours and zero-shot OOD success. Simulated benchmarks (RoboTwin Clean-to-Random, GenieSim-Instruction, and LIBERO-Plus) further confirm superior generalization over strong baselines including π0.5.

Critical strengths
  1. Unprecedented Scale and Empirical Rigor in Real-World Embodied AI: The paper presents arguably the most extensive empirical evaluation of a world-action model to date. Training from scratch on up to 30,000 hours of manipulation data and conducting zero-shot OOD evaluations across 100 real-robot manipulation tasks spanning 20 distinct skill groups on two disparate robotic embodiments (G1-OP and G2-90D) represents a massive milestone. The statistical revelation of skill-coverage scaling laws (1.90 logit units per decade of training hours, Pearson r = 0.74) provides concrete, foundational guidance for the embodied scaling community.

  2. Real-Time High-Frequency Execution via Single-Step Visual Planning and KASO: The authors effectively eliminate the primary practical bottleneck of world models—slow, multi-step autoregressive generation. By pairing the 64x64 CoAE compression with a single-step MeanFlow DiT visual planner, GE-Act 2.0 generates a 52-action chunk in 104 ms on a single RTX 5090 GPU, achieving 30 Hz closed-loop control. Furthermore, KASO offers a principled, action-compatible solution to the fundamental validity gap when co-training visual generative planners with inverse dynamics models.

Critical weaknesses
  1. Fundamental Capability Ceiling on Contact-Rich, Deformable, and Fine-Dexterity Skills: Despite scaling demonstration data by two orders of magnitude (to 30,000 hours), the model exhibits persistent failure on contact-rich and deformable manipulation tasks (Table 1 and Table 9). Tasks such as Zip achieve 0.0% success across all data scales (0.3k to 30k hours), Tear achieves only 5.0%, Sweep achieves 3.3%, and Fold towel with two arms reaches only 10.0%. This severe plateau demonstrates that open-loop visual latent prediction combined with inverse dynamics cannot resolve millimeter-scale contact mechanics, friction, or deformable material states without tactile/haptic feedback. The paper tends to generalize its scaling narrative while understating this critical operational boundary.

  2. Mixed Robustness on LIBERO-Plus Relative to π0.5: On the standardized LIBERO-Plus benchmark (Table 4), GE-Act 2.0 achieves 80.4% overall success, which is noticeably lower than π0.5 (84.4%). Specifically, GE-Act 2.0 degrades severely under robot initial state perturbations (50.7% vs. 73.6% for π0.5) and background shifts (60.5% vs. 94.1% for π0.5). This indicates that the 64x64 downsampled CoAE latents may suffer from perceptual brittleness when facing out-of-distribution visual backgrounds and proprioceptive state deviations.

Questions for authors

No additional questions.

1 comment

Dear Reviewer yNAj,

We sincerely thank you for the generous and encouraging review, and for your strong support of our work! It means a lot to us that you found our large-scale real-robot evaluation and skill-coverage analysis valuable for the embodied scaling community.

W1. Capability ceiling on contact-rich, deformable, and fine-dexterity skills. We agree that these skills remain a clear boundary of GE-Act 2.0: Zip stays at 0% across all scales, and Tear, Sweep, and Fold remain low even at 30,000 hours. Fine control of contact, friction, and deformable material state likely requires tactile or force feedback beyond vision alone, or requires more post-training like reinforcement-learning and we see this as an important direction for future work.

W2. Mixed robustness on LIBERO-Plus relative to π0.5. Thank you for this careful reading of Table 4. We agree that GE-Act 2.0 trails π0.5 overall, especially under background perturbations, and improving robustness to these shifts under the latency constraints is an important next step for our model.

Thank you again for your thoughtful comments and kind support!

Comments on the paper

1 comment

Dear Reviewers,

Please read the authors’ response and let us know whether it addresses your concerns. If any concerns remain, please briefly indicate which ones.

Thank you for your time and for your thoughtful reviews.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.