HORAE-WM: Action Conditioned RGBD Keyframe World Modeling for Long Horizon Robot Manipulation

Action conditioned video world models have made progress in robot manipulation, but with a limited number of output frames, uniform sampling often covers only part of a long-horizon manipulation trajectory, while larger sampling intervals can extend coverage at the cost of missing brief, critical interaction events. We introduce HORAE-WM, an RGBD keyframe world model designed to balance coverage of long manipulation trajectories with the capture of brief critical interactions under a limited output budget. An action-aware sampling selects key frames to be predicted based on arm motion and gripper state changes. Our curriculum progressively introduces prediction of condition videos representing robot actions, metric depth videos capturing scene geometry, and RGB videos depicting robot manipulation under frame-wise time and action condition, using sequences uniformly sampled at different frame rates, followed by training on keyframes. Keyframe prediction evaluations on simulated and real world manipulation show that HORAE-WM outperforms the compared baselines in visual fidelity, arm region consistency, and metric depth accuracy. Sampling analysis shows that our sampler achieves higher event coverage than uniform sampling while allocating more frame budget to events that are both task critical and challenging to capture.