HiLo-WAM: Decoupling Long-Horizon Task Evolution and Local Dynamics in World Action Models
Abstract
World Action Models (WAMs) improve robotic manipulation by jointly modeling visual dynamics and robot actions. However, existing WAMs primarily predict dense videos over short temporal windows, providing strong local interaction modeling but limited understanding of long-horizon task evolution. Extending visual prediction over longer horizons can provide richer task context, but also increases prediction uncertainty and computation, while forcing task-level evolution and fine-grained local dynamics to be modeled within a single temporal process. More importantly, this entangles different temporal dynamics, obscuring their dependencies and distinct roles in action generation. We propose HiLo-WAM, which decouples long-horizon task evolution from local visual dynamics while jointly modeling their interaction for action generation. A long-horizon visual stream captures historical task context and coarse future evolution, whereas a local visual dynamics stream models dense short-horizon robot–object interactions. The two streams operate at different temporal scales within a unified dual-DiT architecture and jointly guide robot actions through structured cross-stream interaction. Progressive stream-wise inference further reduces unnecessary world-model computation. Experiments on simulation and real-world manipulation tasks demonstrate strong long-horizon manipulation performance. HiLo-WAM achieves a 93.70% average success rate on RoboTwin 2.0 and a 17.43 long-horizon progress score on RoboDojo, while also maintaining robust performance on the Fourier GR-3 robot under both in-domain and OOD settings. Ablations further verify the complementary contributions of long-horizon task evolution and local interaction dynamics to action generation.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.